cb34cbbaf2
Previously, most of the benchmarks just measured time spent reading or writing the array on the device. The transfer only happened on the first iteration and was then cached on the device. This change clears out the array every iteration so that the array has to be transferred afresh. |
||
---|---|---|
.. | ||
BenchmarkArrayTransfer.cxx | ||
BenchmarkAtomicArray.cxx | ||
BenchmarkCopySpeeds.cxx | ||
BenchmarkDeviceAdapter.cxx | ||
Benchmarker.h | ||
BenchmarkFieldAlgorithms.cxx | ||
BenchmarkFilters.cxx | ||
BenchmarkRayTracing.cxx | ||
BenchmarkTopologyAlgorithms.cxx | ||
CMakeLists.txt | ||
README.md |
BENCHMARKING VTK-m
TL;DR
When configuring VTM-m with CMake pass the flag -DVTKm_ENABLE_BENCHMARKS=1
. In the build directory you will see the following binaries:
$ ls bin/Benchmark*
bin/BenchmarkArrayTransfer* bin/BenchmarkCopySpeeds* bin/BenchmarkFieldAlgorithms*
bin/BenchmarkRayTracing* bin/BenchmarkAtomicArray* bin/BenchmarkDeviceAdapter*
bin/BenchmarkFilters* bin/BenchmarkTopologyAlgorithms*
Taking as an example BenchmarkArrayTransfer
, we can run it as:
$ bin/BenchmarkArrayTransfer -d Any
Parts of this Documents
Choosing devices
Taking as an example BenchmarkArrayTransfer
, we can determine in which
device we can run it by simply:
$ bin/BenchmarkArrayTransfer
...
Valid devices: "Any" "Serial"
...
Upon the Valid devices you can chose in which device to run the benchmark by:
$ bin/BenchmarkArrayTransfer -d Serial
Run a subset of your benchmarks
VTK-m benchmarks uses Google Benchmarks which allows you to choose a subset
of benchmaks by using the flag --benchmark_filter=REGEX
For instance, if you want to run all the benchmarks that writes something you would run:
$ bin/BenchmarkArrayTransfer -d Serial --benchmark_filter='Write'
Note you can list all of the available benchmarks with the option:
--benchmark_list_tests
.
Compare with baseline
VTM-m ships with a helper script based in Google Benchmarks compare.py
named compare-benchmarks.py
which lets you compare benchmarks using different
devices, filters, and binaries. After building VTM-m
it must appear on the
bin
directory within your build
directory.
When running compare-benchmarks.py
:
- You can specify the baseline benchmark binary path and its arguments in
--benchmark1=
- The contender benchmark binary path and its arguments in
--benchmark2=
- Extra options to be passed to
compare.py
must come after--
Compare between filters
When comparing filters, we only can use one benchmark binary with a single device as shown in the following example:
$ ./compare-benchmarks.py --benchmark1='./BenchmarkArrayTransfer -d Any
--benchmark_filter=1024' --filter1='Read' --filter2=Write -- filters
# It will output something like this:
Benchmark Time CPU Time Old Time New CPU Old CPU New
---------------------------------------------------------------------------------------------------------------------------------------------------------------
BenchContToExec[Read vs. Write]<F32>/Bytes:1024/manual_time +0.2694 +0.2655 18521 23511 18766 23749
BenchExecToCont[Read vs. Write]<F32>/Bytes:1024/manual_time +0.0212 +0.0209 25910 26460 26152 26698
Compare between devices
When comparing two benchmarks using two devices use the option benchmark
after --
and call ./compare-benchmarks.py
as follows:
$ ./compare-benchmarks.py --benchmark1='./BenchmarkArrayTransfer -d Serial
--benchmark_filter=1024' --benchmark2='./BenchmarkArrayTransfer -d Cuda
--benchmark_filter=1024' -- benchmarks
# It will output something like this:
Benchmark Time CPU Time Old Time New CPU Old CPU New
---------------------------------------------------------------------------------------------------------------------------------------------------
BenchContToExecRead<F32>/Bytes:1024/manual_time +0.0127 +0.0120 18388 18622 18632 18856
BenchContToExecWrite<F32>/Bytes:1024/manual_time +0.0010 +0.0006 23471 23496 23712 23726
BenchContToExecReadWrite<F32>/Bytes:1024/manual_time -0.0034 -0.0041 26363 26274 26611 26502
BenchRoundTripRead<F32>/Bytes:1024/manual_time +0.0055 +0.0056 20635 20748 21172 21291
BenchRoundTripReadWrite<F32>/Bytes:1024/manual_time +0.0084 +0.0082 29288 29535 29662 29905
BenchExecToContRead<F32>/Bytes:1024/manual_time +0.0025 +0.0021 25883 25947 26122 26178
BenchExecToContWrite<F32>/Bytes:1024/manual_time -0.0027 -0.0038 26375 26305 26622 26522
BenchExecToContReadWrite<F32>/Bytes:1024/manual_time +0.0041 +0.0039 25639 25745 25871 25972
Installing compare-benchmarks.py
compare-benchmarks.py
relies on compare.py
from Google Benchmarks which also
relies in SciPy
, you can find instructions here regarding its
installation.