Commit Graph

51 Commits

Author SHA1 Message Date
Hans Goudey
bec350ba6e Tests: Simple automated sculpt brush stroke performance test
As an initial step to creating automated regression tests for sculpt brushes,
make our existing performance test script into an automated performance
test. The test uses the brush active in each file and runs the brush stroke
operator on a large generated grid. The time is just for the brush evaluation,
it doesn't include building the PBVH, drawing, etc.

I'm not sure about the consequences of conditionally disabling
`view3d_operator_needs_opengl`, but it was needed to make the test
work in background mode.

Pull Request: https://projects.blender.org/blender/blender/pulls/123148
2024-07-01 16:12:02 +02:00
Brecht Van Lommel
fe4c28f2dc Fix: Benchmarks auto build not working with new lib directories
Symlinking the lib folder doesn't work anymore with submodules.
2024-06-06 19:52:55 +02:00
Campbell Barton
4f8db2ee67 Cleanup: use static sets for contains checks, remove f-string use 2024-02-28 11:02:49 +11:00
Sergey Sharybin
396348eea9 Merge branch 'blender-v4.1-release' 2024-02-22 13:51:48 +01:00
Sergey Sharybin
3dc832a904 Switch SVN to Git submodules using Git-LFS
This change makes it so build system and update utilities for Blender builds
are using pre-compiled libraries and other resources attached as Git modules
instead of using checkout of SVN repositories in the parent folder.

The directory layout:
```
  * release/datafiles/
    * assets/        -> blender-assets.git
      * publish/
      * ...
      * README.txt
  * lib/
    * darwin_x64/    -> lib-darwin_x64.git
    * darwin_arm64/  -> lib-darwin_arm64.git
    * linux_x64/     -> lib-linux_x64.git
    * windows_x64/   -> lib-windows_x64.git
  * tests/
    * data/         -> blender-test-data.git
```

The changes about configuring the actual Git sub-modules are not included
into this patch, as those require repository to actually exist before it
can be used.

The assets submodule is enabled by default, and the rest of them are
disabled. This means that if someone runs `git submodule update --init`
they will not get heavy libraries. The platform-specific and tests
related submodules are enabled when using `make update` or `make test`.

All the submodules are tracked: this means that when new commits are
done to the submodule, the blender.git repository is to be updated to
point them to the new hash. This causes some extra manual work, but it
allows to more easily update Blender and its dependencies to known good
state when performing operations like bisect.

Ref #108978

Pull Request: https://projects.blender.org/blender/blender/pulls/117946
2024-02-22 13:50:55 +01:00
Brecht Van Lommel
0f2064bc3b Revert changes from main commits that were merged into blender-v4.1-release
The last good commit was 4bf6a2e564ad2.
2024-02-19 15:59:59 +01:00
Brecht Van Lommel
d287f17803 Tests: Tweak benchmarks for automation
* Non-zero exit code when test failed
* Ignore GPU devices for tests that don't use them
* Support config option to only run background mode tests
2024-02-19 09:25:24 +01:00
Brecht Van Lommel
9c8a9662c0 Tests: Cache revision builds in benchmark script
In the mode where the script automatically builds Blender at given git
hashes, cache the builds instead of rebuilding them before running.
2024-01-18 21:28:23 +01:00
Campbell Barton
e955c94ed3 License Headers: Set copyright to "Blender Authors", add AUTHORS
Listing the "Blender Foundation" as copyright holder implied the Blender
Foundation holds copyright to files which may include work from many
developers.

While keeping copyright on headers makes sense for isolated libraries,
Blender's own code may be refactored or moved between files in a way
that makes the per file copyright holders less meaningful.

Copyright references to the "Blender Foundation" have been replaced with
"Blender Authors", with the exception of `./extern/` since these this
contains libraries which are more isolated, any changed to license
headers there can be handled on a case-by-case basis.

Some directories in `./intern/` have also been excluded:

- `./intern/cycles/` it's own `AUTHORS` file is planned.
- `./intern/opensubdiv/`.

An "AUTHORS" file has been added, using the chromium projects authors
file as a template.

Design task: #110784

Ref !110783.
2023-08-16 00:20:26 +10:00
Campbell Barton
89811cb96f Cleanup: unused-imports in tests/ 2023-07-25 21:43:53 +10:00
Campbell Barton
932a85a24f Fix tests failing from errors in the users startup file
Tests should never depend on the users startup.blend which can have
settings that interfere with tests.

Failure to load the user startup.blend for e.g. prevented
bl_alembic_io_test from passing.
2023-07-25 21:20:10 +10:00
Sergey Sharybin
4f9a9abcf1 Merge branch 'blender-v3.6-release' 2023-06-15 13:40:38 +02:00
Sergey Sharybin
2d3c9bcee8 Benchmark: Pack python expression into a single line
There should be no functional changes for the typical usecase,
but it allows to have more tricky setups like pointing to a
BAT script to override some configuration.

The issue is that BAT scripts do not support new lines in the
command line arguments. That's where single-line python expression
helps.

For example, it is possible to point benchmark script to a blender.bat
which contains

    blender.exe --python-expr "import bpy; bpy.context.preferences.addons['cycles'].preferences.use_oneapirt = False" %*

to have side-by-side numbers of oneAPI with and without HW RT.

Without this change the %* is which did not work: the BAT
script did not "see" part of the command line past the new line.

Pull Request: https://projects.blender.org/blender/blender/pulls/109006
2023-06-15 13:40:17 +02:00
Campbell Barton
65f99397ec License headers: use SPDX-FileCopyrightText in all sources 2023-06-15 13:35:34 +10:00
Sergey Sharybin
fc440c4fa0 Fix monster performance benchmark run on Windows
The scene contains some interesting names, which requires to be
written as utf-8. And on Windows file descriptor is not guaranteed
to be using utf-8. Or, will error out if the invalid utf-8 sequence
is written.

This change makes it so running benchmarks on windows it fully successful.

Pull Request: https://projects.blender.org/blender/blender/pulls/108982
2023-06-14 18:11:22 +02:00
Sergey Sharybin
5115afaca2 Rename benchmark to benchmark.py
Windows does not really have an idea of shebangs, and it needs to
go via a file extension to see that the script is to be executed
by Python.

This change simplifies execution from `python3 benchmark ...`
to `benchmark.py ...`.

Pull Request: https://projects.blender.org/blender/blender/pulls/108971
2023-06-14 12:41:24 +02:00
Patrick Mours
8a32d56056 Tests: Fix device list of benchmark script only showing a single GPU
Pull Request #104583
2023-02-10 19:38:37 +01:00
Sergey Sharybin
bd6b0bac88 Update references to the new projects platform and main branch 2023-02-07 14:18:19 +01:00
Jacques Lucke
d49b850399 Tests: run geometry nodes benchmarks multiple times to reduce noise
Ideally, we would also get variance information out of the test, but that
seems a bit more complex to implement. For now just run the test a couple
of times and average the timings.

The test now runs between 5 and 100 times, depending on how long it
to run the test once.
2023-01-06 13:04:49 +01:00
Jacques Lucke
006e905957 Tests: support running benchmark when Blender prints unrelated text
I noticed that sometimes the geometry nodes benchmark would not work
if there were some left-over debug prints in the code. The reason it did
not work was because the prints were mixed with the test output print.
I also tried using explicit flusing and `atexit` for printing the test output,
but that didn't solve it. Just printing additional newlines to better separate
the test output from other printed output helped though.
2023-01-06 13:04:49 +01:00
Hans Goudey
3a41e0f611 Tests: Automated geometry nodes benchmark
Add a script for a very simple object evaluation benchmark.
There could be more advanced ways of measuring the time
per-node or per modifier, but this just loads the file, tags
the active object for a reevaluation, and times how long
that takes.

Differential Revision: https://developer.blender.org/D16604
2022-11-26 17:15:55 -06:00
Brecht Van Lommel
cc12f3e0ba Cleanup: fix various typos
Contributed by luzpaz.

Differential Revision: https://developer.blender.org/D15588
2022-08-03 19:28:05 +02:00
Campbell Barton
feeb8310c8 Cleanup: format 2022-06-30 12:14:23 +10:00
Jeroen Bakker
66f826ae85 Benchmark: Add eevee viewport playback tests.
This commit adds the ability to test Eevee viewport playback performance tests.

Tests should be placed in `lib/benchmarks/eevee/*/*.blend`. {rBL62962} added
initial test files. See https://wiki.blender.org/wiki/Tools/Tests/Performance how
to set it up.

To record the playback performance the test start the viewport playback, and adds
a post frame change handler.

This handler will go over the next steps:

* Ensures the viewport is set to rendered mode.
* Wait for shaders to be compiled. Utilizes `bpy.app.is_job_running` function when
  available (v3.3) to wait for shader compilation to finish. When not available will wait
  for one minute.
* Draw several warmup frames
* Record for 10 seconds tracking the number of frames drawn and performance counters.
* When ready print the result to the console. The results will be extracted when the
  benchmark has run.

## Example report
```
                                         master               v3.0                 v3.1                 v3.2
T88219                                   0.0860s              0.0744s              0.0744s              0.0851s
blender290-fox                           1.3056s              0.8744s              0.7994s              1.2809s
```

{F13232387}

Reviewed By: brecht, fclem

Maniphest Tasks: T99136

Differential Revision: https://developer.blender.org/D15302
2022-06-29 07:55:51 +02:00
Campbell Barton
bba757ef81 Cleanup: various minor changes
- Add missing doxy-section for Apply Parent Inverse Operator
- Use identity for None comparison in Python.
- Remove newline from operator doc-strings.
- Use '*' prefix multi-line C comment blocks.
- Separate filenames from doc-strings.
- Remove break after return.
2022-04-24 13:41:03 +10:00
Campbell Barton
65a1fcdaf7 Cleanup: run autopep8 on tests/performance/benchmark
This file was skipped by source/tools/utils/autopep8_clean.py
since it doesn't have a .py extension, running the autopep8 tool
recursively detects Python scripts without extensions.
2022-04-20 17:07:46 +10:00
Campbell Barton
3035235def Cleanup: run autopep8 on tests/ 2022-04-19 15:25:55 +10:00
Campbell Barton
c434782e3a File headers: SPDX License migration
Use a shorter/simpler license convention, stops the header taking so
much space.

Follow the SPDX license specification: https://spdx.org/licenses

- C/C++/objc/objc++
- Python
- Shell Scripts
- CMake, GNUmakefile

While most of the source tree has been included

- `./extern/` was left out.
- `./intern/cycles` & `./intern/atomic` are also excluded because they
  use different header conventions.

doc/license/SPDX-license-identifiers.txt has been added to list SPDX all
used identifiers.

See P2788 for the script that automated these edits.

Reviewed By: brecht, mont29, sergey

Ref D14069
2022-02-11 09:14:36 +11:00
Brecht Van Lommel
8ae6995e98 Tests: show more precise numbers in benchmark charts 2022-01-14 15:17:22 +01:00
Brecht Van Lommel
7d5ef64bfb Cleanup: fix typos in comments and docs
Contributed by luzpaz.

Differential Revision: https://developer.blender.org/D13264
2021-11-19 12:46:49 +01:00
Brecht Van Lommel
1b55b911f2 Merge branch 'blender-v3.0-release' 2021-11-12 20:04:05 +01:00
Brecht Van Lommel
71131b4969 Tests: disable audio system for performance tests 2021-11-12 20:03:46 +01:00
Hans Goudey
4648c4990c Merge branch 'blender-v3.0-release' 2021-11-09 13:08:36 -06:00
Ray Molenkamp
7c25399576 tests/benchmarks: Fix operation on windows
The test script did not work on windows
since it had some trouble importing the
api module on the blender side of things.

turning the file path to the module into
a raw string literal sidesteps the
backslash issue in the path.

Differential Revision: https://developer.blender.org/D13163

Reviewed by: brecht
2021-11-09 09:58:48 -07:00
Campbell Barton
27b37517f8 Cleanup: use static sets 2021-11-08 17:00:36 +11:00
Campbell Barton
c516659b5e Cleanup: use static sets 2021-11-08 14:52:08 +11:00
Brecht Van Lommel
fc36772b06 Tests: minor updates to benchmark script for running on buildbot
* graph command accepts folder of json files as input
* reset command clears log files
2021-10-28 22:41:40 +02:00
Brecht Van Lommel
335f40ebfa Tests: include device type in benchmark graph labels 2021-10-06 14:21:26 +02:00
Brecht Van Lommel
cc653c9b02 Fix potential render tests error with invalid utf-8 characters
In general should not happen, but better to report the actual error instead
of the Python test code failing.
2021-09-28 20:50:38 +02:00
Brecht Van Lommel
0803119725 Cycles: merge of cycles-x branch, a major update to the renderer
This includes much improved GPU rendering performance, viewport interactivity,
new shadow catcher, revamped sampling settings, subsurface scattering anisotropy,
new GPU volume sampling, improved PMJ sampling pattern, and more.

Some features have also been removed or changed, breaking backwards compatibility.
Including the removal of the OpenCL backend, for which alternatives are under
development.

Release notes and code docs:
https://wiki.blender.org/wiki/Reference/Release_Notes/3.0/Cycles
https://wiki.blender.org/wiki/Source/Render/Cycles

Credits:
* Sergey Sharybin
* Brecht Van Lommel
* Patrick Mours (OptiX backend)
* Christophe Hery (subsurface scattering anisotropy)
* William Leeson (PMJ sampling pattern)
* Alaska (various fixes and tweaks)
* Thomas Dinges (various fixes)

For the full commit history, see the cycles-x branch. This squashes together
all the changes since intermediate changes would often fail building or tests.

Ref T87839, T87837, T87836
Fixes T90734, T89353, T80267, T80267, T77185, T69800
2021-09-21 14:55:54 +02:00
Brecht Van Lommel
e43ecca016 Tests: measure time per frame in animation performance benchmark
Instead of a fixed number of frames, so that benchmarking takes about the
same time on any machine.
2021-09-20 19:03:22 +02:00
Brecht Van Lommel
42215d7cb8 Tests: more graceful handling of keyboard interrupting benchmarks
Leave current test result unchanged and stop executing immediately,
so it can be continued.
2021-09-10 16:48:26 +02:00
Brecht Van Lommel
5bfc3a3421 Fix error running benchmark script with environment variables for builds
Ref D12434
2021-09-08 19:58:27 +02:00
Brecht Van Lommel
6fc94d1848 Tests: updates for performance benchmarking
* Make "run" command (re-)run all tests, add "update" command to only
  run queued and outdated tests equivalent to the old "run" command.
* Support specifying environment variables for revisions, to easily
  compare multiple parameter values.
* Better sorting of revisions in graph.
2021-09-08 16:40:58 +02:00
Brecht Van Lommel
214e4aac97 Fix Python error in ./benchmark init after recent changes 2021-08-19 17:45:37 +02:00
Brecht Van Lommel
22b03e1c68 Fix Python error in benchmark executable detection after recent changes 2021-07-26 18:04:40 +02:00
Brecht Van Lommel
320f34af86 Tests: continue running benchmarks if some tests fail to build or run
Convenient when testing many revisions where some might be broken.
2021-07-22 16:35:00 +02:00
Brecht Van Lommel
19b597c55d Tests: improve finding of Blender executables in benchmarking
* Allow specifying a folder and automatically setting the proper executable
  name depending on the operating system
* Use executables from configs for listing devices instead of a blender
  command being available
2021-07-22 16:34:57 +02:00
Brecht Van Lommel
37a5ff4a84 Tests: support graphing peak memory in Cycles performance tests
The general graphing mechanism will create one graph for each output
variable. So it's not limited to time and memory, but that is what the
Cycles tests now output.
2021-07-14 16:12:10 +02:00
Campbell Barton
432bfbf7a3 Cleanup: pep8 2021-07-06 12:05:27 +10:00