613b4d44b7
Previous version had a hard-coded loop for updating the dataset. This change relenquishes control over to google benchmarks. Running it from the command-line will iterate each sub-benchmark a different number of iterations depending on how long each individual test takes. The number of iterations can be increased (but not controlled) by increasing the minimum time for each test. Alternatively, using the "repetitions" argument, you can control exactly how many times the benchamrks are run. See the README for InSitu that throughly documents the arguments.
1.9 KiB
This document describes how to use the command-line options for Google Benchmarks (GBench) to control the behavior of BenchmarkInSitu.
Generally, "BenchmarkInSitu --help" will provide the list of standard benchmarks along with the associated ones from GBench.
As a refresher, GBench iterates a test defined in the application, in this case, Contour, Streamlines, ..., a number of times until two criteria are met, a statistically stable set of samples have been generated, and the test ran for a specified minimum amount of time (by default, 0.5 seconds).
There are three ways to run the InSitu benchmark that control the number of iterations run by GBench. These are independent of the "standard" arguments passed to the benchmark (we'll define the standard arguments as: --vtkm-device, --size, --image-size, plus other not defined by GBench).
-
BenchmarkInSitu
- Under this scenario, the iterations are controlled completely by GBench. Generally, each test will be run between 1 and N iterations depending on how long each test runs.
-
BenchmarkInSitu --benchmark_min_time=<min_time>
- This will ensure that the test will run for at least <min_time> seconds. You will set this option if you don't care about the actual number of iterations, but only that each test runs for at least a specified time.
-
BenchmarkInSitu --benchmark_repetitions=
- The purpose of this option is to exactly control the number of iterations performed by GBench. Internally, this does two things:
- Sets the minimum time to a very small value ("--benchmark_min_time=0.00000001")
- Sets the output to only report aggregate statistics for each test, e.g., mean, median, standard deviation (--benchmark_report_aggregates_only=true) Both of these arguments can be overridden by providing different values on the command-line. With the current setting, all test runs have resulted in only repetitions being executed.
- The purpose of this option is to exactly control the number of iterations performed by GBench. Internally, this does two things: