6f5f654878
We've been having problems with PerformanceTestBenchContour. In the last few iteration, the runtime goes way up. We cannot find any reason for this in the source code. There don't appear to be any particular problems with memory or tables. The best we can figure is an issue with the device hardware in the container. The easy solution should be to break the benchmark into smaller peices to avoid the problem. |
||
---|---|---|
.. | ||
BenchmarkArrayTransfer.cxx | ||
BenchmarkAtomicArray.cxx | ||
BenchmarkCopySpeeds.cxx | ||
BenchmarkDeviceAdapter.cxx | ||
Benchmarker.h | ||
BenchmarkFieldAlgorithms.cxx | ||
BenchmarkFilters.cxx | ||
BenchmarkInSitu.cxx | ||
BenchmarkODEIntegrators.cxx | ||
BenchmarkRayTracing.cxx | ||
BenchmarkTopologyAlgorithms.cxx | ||
CMakeLists.txt | ||
README_insitu.md | ||
README.md | ||
vtkm.module |
BENCHMARKING VTK-m
TL;DR
When configuring VTM-m with CMake pass the flag -DVTKm_ENABLE_BENCHMARKS=1
. In the build directory you will see the following binaries:
$ ls bin/Benchmark*
bin/BenchmarkArrayTransfer* bin/BenchmarkCopySpeeds* bin/BenchmarkFieldAlgorithms*
bin/BenchmarkRayTracing* bin/BenchmarkAtomicArray* bin/BenchmarkDeviceAdapter*
bin/BenchmarkFilters* bin/BenchmarkTopologyAlgorithms*
Taking as an example BenchmarkArrayTransfer
, we can run it as:
$ bin/BenchmarkArrayTransfer -d Any
Parts of this Documents
Choosing devices
Taking as an example BenchmarkArrayTransfer
, we can determine in which
device we can run it by simply:
$ bin/BenchmarkArrayTransfer
...
Valid devices: "Any" "Serial"
...
Upon the Valid devices you can chose in which device to run the benchmark by:
$ bin/BenchmarkArrayTransfer -d Serial
Run a subset of your benchmarks
VTK-m benchmarks uses Google Benchmarks which allows you to choose a subset
of benchmaks by using the flag --benchmark_filter=REGEX
For instance, if you want to run all the benchmarks that writes something you would run:
$ bin/BenchmarkArrayTransfer -d Serial --benchmark_filter='Write'
Note you can list all of the available benchmarks with the option:
--benchmark_list_tests
.
Compare with baseline
VTM-m ships with a helper script based in Google Benchmarks compare.py
named compare-benchmarks.py
which lets you compare benchmarks using different
devices, filters, and binaries. After building VTM-m
it must appear on the
bin
directory within your build
directory.
When running compare-benchmarks.py
:
- You can specify the baseline benchmark binary path and its arguments in
--benchmark1=
- The contender benchmark binary path and its arguments in
--benchmark2=
- Extra options to be passed to
compare.py
must come after--
Compare between filters
When comparing filters, we only can use one benchmark binary with a single device as shown in the following example:
$ ./compare-benchmarks.py --benchmark1='./BenchmarkArrayTransfer -d Any
--benchmark_filter=1024' --filter1='Read' --filter2=Write -- filters
# It will output something like this:
Benchmark Time CPU Time Old Time New CPU Old CPU New
---------------------------------------------------------------------------------------------------------------------------------------------------------------
BenchContToExec[Read vs. Write]<F32>/Bytes:1024/manual_time +0.2694 +0.2655 18521 23511 18766 23749
BenchExecToCont[Read vs. Write]<F32>/Bytes:1024/manual_time +0.0212 +0.0209 25910 26460 26152 26698
Compare between devices
When comparing two benchmarks using two devices use the option benchmark
after --
and call ./compare-benchmarks.py
as follows:
$ ./compare-benchmarks.py --benchmark1='./BenchmarkArrayTransfer -d Serial
--benchmark_filter=1024' --benchmark2='./BenchmarkArrayTransfer -d Cuda
--benchmark_filter=1024' -- benchmarks
# It will output something like this:
Benchmark Time CPU Time Old Time New CPU Old CPU New
---------------------------------------------------------------------------------------------------------------------------------------------------
BenchContToExecRead<F32>/Bytes:1024/manual_time +0.0127 +0.0120 18388 18622 18632 18856
BenchContToExecWrite<F32>/Bytes:1024/manual_time +0.0010 +0.0006 23471 23496 23712 23726
BenchContToExecReadWrite<F32>/Bytes:1024/manual_time -0.0034 -0.0041 26363 26274 26611 26502
BenchRoundTripRead<F32>/Bytes:1024/manual_time +0.0055 +0.0056 20635 20748 21172 21291
BenchRoundTripReadWrite<F32>/Bytes:1024/manual_time +0.0084 +0.0082 29288 29535 29662 29905
BenchExecToContRead<F32>/Bytes:1024/manual_time +0.0025 +0.0021 25883 25947 26122 26178
BenchExecToContWrite<F32>/Bytes:1024/manual_time -0.0027 -0.0038 26375 26305 26622 26522
BenchExecToContReadWrite<F32>/Bytes:1024/manual_time +0.0041 +0.0039 25639 25745 25871 25972
Installing compare-benchmarks.py
compare-benchmarks.py
relies on compare.py
from Google Benchmarks which also
relies in SciPy
, you can find instructions here regarding its
installation.