Blender startup time and shader compilation is a big factor when running
hundreds of tests, so now all renders in the same ctest run in the same
process.
This was previously reverted due to skipping other tests when one test
crashed. Now if a test crashes, Blender is re-run with the remaining
tests so we get results from them still.
This makes finding the crashing tests harder, reverting until there is a
better solution.
This reverts commit 93901e7f0a05ba471f3b4c0201500d9dfcd68c2c.
Blender startup time and shader compilation is a big factor when running
hundreds of tests, so now all renders in the same ctest run in the same
process. If a test crashes, the remaining tests in the same category will
be marked as skipped.
Benchmarked on a quad core with ctest -j8.
cycles: 118.1s -> 94.3s
eevee: 66.2s -> 29.2s
workbench: 31.7s -> 8.6s
Being able to compare Eevee reference images is useful for refactoring I'm
working on so might as well add them now, even if we can still improve them.
Workbench tests are just rendering the same files as Cycles and Eevee. This
doesn't really tests many workbench settings until we add tests specifically
for them, but does cover how it it handles the different object types.
This adds Eevee render tests using the Cycles files. Currently it must
be enabled by setting WITH_OPENGL_RENDER_TESTS=ON. Once we have reference
images we can enable it by default.
Some of the Cycles and Eevee tests are also currently broken due to
modifier and particle changes.
Differential Revision: https://developer.blender.org/D3182
Shows new, reference and diff renders, with mouse hover to flip between
new and ref for easy comparison. This generates a report.html in
build_dir/tests/cycles, stored along with the new and diff images.
Differential Revision: https://developer.blender.org/D2770
Works similar to regular Cycles tests, just does OpenGL render to
get output image.
Seems to work fine with the only funny effect: Blender window will
pop up for each of the tests. This is current limitation of our
OpenGL context. Might be changed in the future.
Made them closer to how GTest shows the output, so reading test logs
is easier now (at least feels more uniform).
Additionally now we know how much time tests are taking so can tweak
samples/resolution to reduce render time of slow tests.
It is now also possible to enable colored messages using magic
CYCLESTEST_COLOR environment variable. This makes it even easier to
visually grep failed/passed tests using `ctest -R cycles -V`.
This reverts commit d390e24c49ba8543b56948402622563cae6e860f.
Forcing tests to success is really bad idea. It'll only lead to cases when
you see PASSED and will think everything is OK.
Long story short: never force tests to pass!
The idea is to use the set of really small images from the lib folder
and run Cycles render on them comparing render output to reference
images in the tests repository.
For sure same thing could become more generic for BI or Freestyle
render engines.
Thanks Campbell for review and code tweaks!
