vtk-m/benchmarking

BENCHMARKING VTK-m

TL;DR

When configuring VTM-m with CMake pass the flag -DVTKm_ENABLE_BENCHMARKS=1 . In the build directory you will see the following binaries:

$ ls bin/Benchmark*
bin/BenchmarkArrayTransfer*  bin/BenchmarkCopySpeeds* bin/BenchmarkFieldAlgorithms*
bin/BenchmarkRayTracing* bin/BenchmarkAtomicArray*    bin/BenchmarkDeviceAdapter*
bin/BenchmarkFilters* bin/BenchmarkTopologyAlgorithms*

Taking as an example BenchmarkArrayTransfer, we can run it as:

$ bin/BenchmarkArrayTransfer -d Any

---

Parts of this Documents

0. TL;DR
1. Devices
2. Filters
3. Compare with baseline
4. Installing compare.py

---

Choosing devices

Taking as an example BenchmarkArrayTransfer, we can determine in which device we can run it by simply:

$ bin/BenchmarkArrayTransfer
...
Valid devices: "Any" "Serial"
...

Upon the Valid devices you can chose in which device to run the benchmark by:

$ bin/BenchmarkArrayTransfer -d Serial

Run a subset of your benchmarks

VTK-m benchmarks uses Google Benchmarks which allows you to choose a subset of benchmaks by using the flag --benchmark_filter=REGEX

For instance, if you want to run all the benchmarks that writes something you would run:

$ bin/BenchmarkArrayTransfer -d Serial --benchmark_filter='Write'

Note you can list all of the available benchmarks with the option: --benchmark_list_tests.

Compare with baseline

VTM-m ships with a helper script based in Google Benchmarks compare.py named compare-benchmarks.py which lets you compare benchmarks using different devices, filters, and binaries. After building VTM-m it must appear on the bin directory within your build directory.

When running compare-benchmarks.py:

• You can specify the baseline benchmark binary path and its arguments in --benchmark1=
• The contender benchmark binary path and its arguments in --benchmark2=
• Extra options to be passed to compare.py must come after --

Compare between filters

When comparing filters, we only can use one benchmark binary with a single device as shown in the following example:

$ ./compare-benchmarks.py --benchmark1='./BenchmarkArrayTransfer -d Any
--benchmark_filter=1024' --filter1='Read' --filter2=Write -- filters

# It will output something like this:

Benchmark                                                                          Time             CPU      Time Old      Time New       CPU Old       CPU New
---------------------------------------------------------------------------------------------------------------------------------------------------------------
BenchContToExec[Read vs. Write]<F32>/Bytes:1024/manual_time                     +0.2694         +0.2655         18521         23511         18766         23749
BenchExecToCont[Read vs. Write]<F32>/Bytes:1024/manual_time                     +0.0212         +0.0209         25910         26460         26152         26698

Compare between devices

When comparing two benchmarks using two devices use the option benchmark after -- and call ./compare-benchmarks.py as follows:

$ ./compare-benchmarks.py --benchmark1='./BenchmarkArrayTransfer -d Serial
--benchmark_filter=1024' --benchmark2='./BenchmarkArrayTransfer -d Cuda
--benchmark_filter=1024' -- benchmarks


# It will output something like this:

Benchmark                                                              Time             CPU      Time Old      Time New       CPU Old       CPU New
---------------------------------------------------------------------------------------------------------------------------------------------------
BenchContToExecRead<F32>/Bytes:1024/manual_time                     +0.0127         +0.0120         18388         18622         18632         18856
BenchContToExecWrite<F32>/Bytes:1024/manual_time                    +0.0010         +0.0006         23471         23496         23712         23726
BenchContToExecReadWrite<F32>/Bytes:1024/manual_time                -0.0034         -0.0041         26363         26274         26611         26502
BenchRoundTripRead<F32>/Bytes:1024/manual_time                      +0.0055         +0.0056         20635         20748         21172         21291
BenchRoundTripReadWrite<F32>/Bytes:1024/manual_time                 +0.0084         +0.0082         29288         29535         29662         29905
BenchExecToContRead<F32>/Bytes:1024/manual_time                     +0.0025         +0.0021         25883         25947         26122         26178
BenchExecToContWrite<F32>/Bytes:1024/manual_time                    -0.0027         -0.0038         26375         26305         26622         26522
BenchExecToContReadWrite<F32>/Bytes:1024/manual_time                +0.0041         +0.0039         25639         25745         25871         25972

Installing compare-benchmarks.py

compare-benchmarks.py relies on compare.py from Google Benchmarks which also relies in SciPy, you can find instructions here regarding its installation.