Blender was not configured to exit with non-zero return code on Python errors.
A bunch of tests worked around this but not all. This removes the need for such
workarounds.
This uses the same framework as automated modifier tests. It adds a physics
modifier, bakes and compares vertex coordinates on the end frame.
Differential Revision: https://developer.blender.org/D7017
Submitting on behalf of Jesse Y (deadpin).
In test harness for modifier testing, now run mesh validation
on output mesh. Also, fix printing so it interleaves properly.
CentOS on the buildbot still runs Python 3.6, which is also used for the
unit tests. This means that the tests can't use language features that
are available to Blender itself. And testing with a different version of
Python than will be used by the actual code seems like a bad idea to me.
This commit adds `TEST_PYTHON_EXECUTABLE` as advanced CMake option. This
will allow us to set a specific Python executable when we need it. When
not set, a platform-specific default will be used:
- On Windows, the `python….exe` from the installation directory. This is
just like before this patch, except that this patch adds the
overridability.
- On macOS/Linux, the `${PYTHON_EXECUTABLE}` as found by CMake.
Every platform should now have a value (configured by the user or
detected by CMake) for `TEST_PYTHON_EXE`, so there is no need to allow
running without. This also removes the need to have some Python files
marked as executable.
If `TEST_PYTHON_EXE` is not user-configured, and thus the above default
is used, a status message is logged by CMake. I've seen this a lot in
other projects, and I like that it shows which values are auto-detected.
However, it's not common in Blender, so if we want we can either remove
it now, or remove it after the buildbot has been set up correctly.
Differential Revision: https://developer.blender.org/D7395
Reviewed by: campbellbarton, mont29, sergey
* Rename template parameter N to InlineBufferCapacity
* Expose InlineBufferCapacity parameter for Set and Map
* Add some comments
* Fixed an error that I introduced recently
This is not currently used and will take some work to support with TBB, so
remove it until we have a new implementation based on TBB.
Fixes T76005, parallel range pool tests failing.
Ref D7475
Instead of building on top of `BLI::Vector`, just use a raw array
and handle the growing in `BLI::VectorSet`.
After this change, the existing `EdgeSet` can be reimplemented using
`BLI::VectorSet` without performance regressions.
This was used to investigate T73840. Since the armature math is far from
simple, I thought it would be a good idea to start writing some unit
tests for it.
No functional changes in Blender itself.
Submitting on behalf of Jesse Y (deadpin).
In test harness for modifier testing, now run mesh validation
on output mesh. Also, fix printing so it interleaves properly.
In preparation of TBB we need to split the finalize function into reduce
and free. Reduce is used to combine results and free for freeing any
allocated memory.
The reduce function is called to join user data chunk into another, to reduce the
result to the original userdata_chunk memory. These functions should have no side
effects so that they can be run on any thread.
The free functions should free data created during execution (TaskParallelRangeFunc).
Original patch by Brecht van Lommel
{rB61f49db843cf5095203112226ae386f301be1e1a}.
Reviewed By: Brecht van Lommel, Bastien Montagne
Differential Revision: https://developer.blender.org/D7394
Tasks: move priority from task to task pool {rBf7c18df4f599fe39ffc914e645e504fcdbee8636}
Tasks: split task.c into task_pool.cc and task_iterator.c {rB4ada1d267749931ca934a74b14a82479bcaa92e0}
Differential Revision: https://developer.blender.org/D7385
The test failure in T75491 only showed up in debug builds because
`BLI_assert()` is a no-op in release builds. This is now replaced by a
proper GTests call to `ADD_FAILURE()`, ensuring that the test fails
regardless of build mode.
NOTE: While most of the milestone 1 goals are there, a few smaller features and
improvements are still to be done.
Big picture of this milestone: Initial, OpenXR-based virtual reality support
for users and foundation for advanced use cases.
Maniphest Task: https://developer.blender.org/T71347
The tasks contains more information about this milestone.
To be clear: This is not a feature rich VR implementation, it's focused on the
initial scene inspection use case. We intentionally focused on that, further
features like controller support are part of the next milestone.
- How to use?
Instructions on how to use this are here:
https://wiki.blender.org/wiki/User:Severin/GSoC-2019/How_to_Test
These will be updated and moved to a more official place (likely the manual) soon.
Currently Windows Mixed Reality and Oculus devices are usable. Valve/HTC
headsets don't support the OpenXR standard yet and hence, do not work with this
implementation.
---------------
This is the C-side implementation of the features added for initial VR
support as per milestone 1. A "VR Scene Inspection" Add-on will be
committed separately, to expose the VR functionality in the UI. It also
adds some further features for milestone 1, namely a landmarking system
(stored view locations in the VR space)
Main additions/features:
* Support for rendering viewports to an HMD, with good performance.
* Option to sync the VR view perspective with a fully interactive,
regular 3D View (VR-Mirror).
* Option to disable positional tracking. Keeps the current position (calculated
based on the VR eye center pose) when enabled while a VR session is running.
* Some regular viewport settings for the VR view
* RNA/Python-API to query and set VR session state information.
* WM-XR: Layer tying Ghost-XR to the Blender specific APIs/data
* wmSurface API: drawable, non-window container (manages Ghost-OpenGL and GPU
context)
* DNA/RNA for management of VR session settings
* `--debug-xr` and `--debug-xr-time` commandline options
* Utility batch & config file for using the Oculus runtime on Windows.
* Most VR data is runtime only. The exception is user settings which are saved
to files (`XrSessionSettings`).
* VR support can be disabled through the `WITH_XR_OPENXR` compiler flag.
For architecture and code documentation, see
https://wiki.blender.org/wiki/Source/Interface/XR.
---------------
A few thank you's:
* A huge shoutout to Ray Molenkamp for his help during the project - it would
have not been that successful without him!
* Sebastian Koenig and Simeon Conzendorf for testing and feedback!
* The reviewers, especially Brecht Van Lommel!
* Dalai Felinto for pushing and managing me to get this done ;)
* The OpenXR working group for providing an open standard. I think we're the
first bigger application to adopt OpenXR. Congratulations to them and
ourselves :)
This project started as a Google Summer of Code 2019 project - "Core Support of
Virtual Reality Headsets through OpenXR" (see
https://wiki.blender.org/wiki/User:Severin/GSoC-2019/).
Some further information, including ideas for further improvements can be found
in the final GSoC report:
https://wiki.blender.org/wiki/User:Severin/GSoC-2019/Final_Report
Differential Revisions: D6193, D7098
Reviewed by: Brecht Van Lommel, Jeroen Bakker
Test specs are read from strings, and there was a comma instead
of a decimal point, and then an extra decimal point in the Quad0 test.
This test has been flaky on Windows buildbot. Perhaps this is why.
Fixed memory leak that showed up after the original issue (crash) had been fixed in 93ac4709ebe8. The fix ensures that light cache bakes free up GPU smoke textures and the smoke domain list correctly.
This commit also removes the workaround (f3a33a92987f) that disabled light cache bakes for fluid objects.
This allows fast access to various arrays in the Python API.
Most notably, `image.pixels` can be accessed much more efficiently now.
**Benchmark**
Below are the results of a benchmark that compares different ways to
set/get all pixel values. I do the tests on 2048x2048 rgba images.
The benchmark tests the following dimensions:
- Byte vs. float per color channel
- Python list vs. numpy array containing floats
- `foreach_set` (new) vs. `image.pixels = ...` (old)
```
Pixel amount: 2048 * 2048 = 4.194.304
Byte buffer size: 16.8 mb
Float buffer size: 67.1 mb
Set pixel colors:
byte - new - list: 271 ms
byte - new - buffer: 29 ms
byte - old - list: 350 ms
byte - old - buffer: 2900 ms
float - new - list: 249 ms
float - new - buffer: 8 ms
float - old - list: 330 ms
float - old - buffer: 2880 ms
Get pixel colors:
byte - list: 128 ms
byte - buffer: 9 ms
float - list: 125 ms
float - buffer: 8 ms
```
**Observations**
The best set and get speed can be achieved with buffers and a float image,
at the cost of higher memory consumption. Furthermore, using buffers when
using `pixels = ...` is incredibly slow, because it is not optimized.
Optimizing this is possible, but might not be trivial (there were multiple
attempts afaik).
Float images are faster due to overhead introduced by the api for byte images.
If I profiled it correctly, a lot of time is spend in the `[0, 1] -> {0, ..., 255}`
conversion. The functions doing that conversion is `unit_float_to_uchar_clamp`.
While I have an idea on how it can be optimized, I do not know if it can be done
without changing its functionality slightly. Performance wise the best solution
would be to not do this conversion at all and accept byte input from the api
user directly, but that seems to be a more involved task as well.
Differential Revision: https://developer.blender.org/D7053
Reviewers: JacquesLucke, mont29
This moves the `alembic`, `avi`, `collada`, and `usd` modules into a common
`io` directory.
This also cleans up some `#include "../../{somedir}/{somefile}.h"` by
adding `../../io/{somedir}` to `CMakeLists.txt` and then just using
`#include "{somefile}.h"`.
No functional changes.
Introduce new IDTypeInfo structure.
Each ID type will have its own, with some minimal basic common info,
and ID management callbacks.
This patch only does it for Object type, for demo/testing purpose.
Moving all existing IDs is a goal of next "cleanup Friday".
Note that BKE_idcode features should then be merged back into BKE_idtype -
but this will have to be done later, once all ID types have been properly
converted to the new system.
Another later TODO might be to try and add callbacks for file read/write,
and lib_query ID usages looper.
This is part of T73719.
Thanks to @brecht for initial idea, and reviewing the patch.
Differential Revision: https://developer.blender.org/D6966
As explained in T65568 by @LucaRood, the self collision system should exclude triangles that are connected by sewing springs.
Differential Revision: https://developer.blender.org/D6911
The 'random' unit tests and some examples from the new boolean code
triggered asserts and crashes. This fixes those.
There is a new flag in the input that optionally disables a pass
over input to snap segment edges to other segments.
This fixes {T70269}.
Before this commit there was complicated code to try and compute the
correct parent inverse matrix for the 'Child Of' and 'Object Solver'
constraints outside the constraint evaluation. This was done mostly
correctly, but did have some issues. The Set Inverse operator now defers
this computation to be performed during constraint evaluation by just
setting a flag. If the constraint is disabled, and thus tagging it for
update in the depsgraph is not enough to trigger immediate evaluation,
evaluation is forced by temporarily enabling it.
This fix changes the way how the inverse matrix works when some of the
channels of the constraint are disabled. Before this commit, the channel
flags were used to filter both the parent and the inverse matrix. This
meant that it was impossible to make an inverse matrix that would
actually fully neutralize the effect of the constraint. Now only the
parent matrix is filtered, while inverse is applied fully. As a result,
pressing the 'Set Inverse' matrix produces the same transformation as
disabling the constraint. This is also reflected in the changed values
in the 'Child Of' unit test.
This change is not backward compatible, but it should be OK because the
old way was effectively unusable, so it is unlikely anybody relied on
it.
The change in matrix for the Object Solver constraint is due to a
different method of computing it, which caused a slightly different
floating point error that was slightly bigger than allowed by the test,
so I updated the matrix values there as well.
This patch was original written by @angavrilov and subsequently updated
by me.
Differential Revision: https://developer.blender.org/D6091
The 'Clear Inverse' operator didn't properly update the constraint, so
it didn't do anything until the entire depsgraph was updated. It's now
properly tagged for update.
In the collections unit test file developers can now disable layer
collections and declutter the 3D Viewport while working in
`constraints.blend`, without influencing the actual unit tests themselves.
Currently this only tests the Child Of constraint. My aim is to cover
constraints with tests before they are refactored/altered.
No functional changes.
There are two issues solved in this commit:
- Our Windows buildbot has slightly different floating point errors than
the Linux one, which meant a larger delta was required for float
comparisons.
- The test performs an export to a temporary Alembic file and
subsequently imports it. Deleting the temporary file was impossible on
Windows because it was still in use. This is now resolved by first
loading the default blend file before deleting the Alembic file.
The Alembic importer now works with local coordinates. Previously, the
importer converted transformations from Alembic to world coordinates
before processing them further; this processing often included
re-converting to local coordinates. This change made it possible to
remove some code that assumed that a child transform was only read after
its parent transform.
Blender's Alembic code follows the Maya convention, where in the zero
orientation the camera looks forward instead of down. This extra
rotation is now handled more consistently, and now also properly handles
children of cameras. This fixes T73269.
Unit tests were added to at least ensure that the importer and exporter
are compatible with each other, and that static and animated camera
transforms are handled in the same way.
This rename is to prepare for a future addition to the unit test file.
Currently it's named "import" and I will add an export test as well. The
rename is a separate commit to easily see the difference between the
rename and the addition of another test.
No functional changes.
The changes come from the `functions` branch, where I'm using
these structures a lot.
This also includes a new `BLI::Optional<T>` type, which is similar
to `std::Optional<T>` which can be used when Blender starts using
C++17.
A big rework of the code now uses exact predicates for orientation
and incircle. Also switched the main algorithm to use a faster
divide and conquer algorithm, which is possible with the exact
predicates.
This 'fixes' T68554: 'API mathutils.geometry.tessellate_polygon returns
bad results sometimes' by documenting the limitations of the current
implementation.
I've also added a unit test for the function, so that any change in this
behaviour will get noticed.
No functional changes.
The dependency graph has to know whether a driver must be re-evaluated
every frame due to a dependency on the current frame number. For python
drivers it was using a heuristic based on searching for certain sub-
strings in the expression, notably including '('.
When the expression is actually evaluated using Python, this can't be
easily improved; however if the Simple Expression evaluator is used,
this check can be done precisely by accessing the parsed data.
Differential Revision: https://developer.blender.org/D6624
This is a more correct fix to the issue Brecht was fixing in D6600.
While the fix in that patch worked fine for linking it broke ASAN
runtime under some circumstances.
For example, `make full debug developer` would compile, but trying
to start blender will cause assert failure in ASAN (related on check
that ASAN is not running already).
Top-level idea: leave it to CMake to keep track of dependency graph.
The root of the issue comes to the fact that target like "blender" is
configured to use a lot of static libraries coming from Blender sources
and to use external static libraries. There is nothing which ensures
order between blender's and external libraries. Only order of blender
libraries is guaranteed.
It was possible that due to a cycle or other circumstances some of
blender libraries would have been passed to linker after libraries
it uses, causing linker errors.
For example, this order will likely fail:
libbf_blenfont.a libfreetype6.a libbf_blenfont.a
This change makes it so blender libraries are explicitly provided
their dependencies to an external libraries, which allows CMake to
ensure they are always linked against them.
General rule here: if bf_foo depends on an external library it is
to be provided to LIBS for bf_foo.
For example, if bf_blenkernel depends on opensubdiv then LIBS in
blenkernel's CMakeLists.txt is to include OPENSUBDIB_LIBRARIES.
The change is made based on searching for used include folders
such as OPENSUBDIV_INCLUDE_DIRS and adding corresponding libraries
to LIBS ion that CMakeLists.txt. Transitive dependencies are not
simplified by this approach, but I am not aware of any downside of
this: CMake should be smart enough to simplify them on its side.
And even if not, this shouldn't affect linking time.
Benefit of not relying on transitive dependencies is that build
system is more robust towards future changes. For example, if
bf_intern_opensubiv is no longer depends on OPENSUBDIV_LIBRARIES
and all such code is moved to bf_blenkernel this will not break
linking.
The not-so-trivial part is change to blender_add_lib (and its
version in Cycles). The complexity is caused by libraries being
provided as a single list argument which doesn't allow to use
different release and debug libraries on Windows. The idea is:
- Have every library prefixed as "optimized" or "debug" if
separation is needed (non-prefixed libraries will be considered
"generic").
- Loop through libraries passed to function and do simple parsing
which will look for "optimized" and "debug" words and specify
following library to corresponding category.
This isn't something particularly great. Alternative would be to
use target_link_libraries() directly, which sounds like more code
but which is more explicit and allows to have more flexibility
and control comparing to wrapper approach.
Tested the following configurations on Linux, macOS and Windows:
- make full debug developer
- make full release developer
- make lite debug developer
- make lite release developer
NOTE: Linux libraries needs to be compiled with D6641 applied,
otherwise, depending on configuration, it's possible to run into
duplicated zlib symbols error.
Differential Revision: https://developer.blender.org/D6642
`posix_memalign` requires the `alignment` to be at least `sizeof(void *)`.
Previously, `MEM_mallocN_aligned` would simply return `NULL` if a too small
`alignment` was used. This was an OS specific issue.
The solution is to use a minimal alignment of `8` for all aligned allocations.
The unit tests have been extended to test more possible alignments (some
of which were broken before).
Reviewers: brecht
Differential Revision: https://developer.blender.org/D6660
This add a basic sanity check that validates
the features we use from ffmpeg are actually
available
Differential Revision: https://developer.blender.org/D5999
Reviewed By: sybren
Patch from Habib Gahbiche (zazizizou) moves the "run operator and
compare mesh to a golden" paradigm used in bevel and boolean tests
into a general framework that separates the test specs from the
blend files. Then adds some other operator and modifier tests using
the new framework. Diff D5357.id20724.diff was applied.
New .blend files, modifiers.blend and operators.blend are needed
in the tests/modeling svn directory; those were separately committed.
There are deeper issues than just updating the regression test .blend file
and the solution is dragging for far too long.
Considering this a known broken feature, which will either be fixed next week
or completely removed from the interface for the coming release.
E.g. "Cube" would be placed after "Cube.001", which is not what you'd
expect. 2.80 handled this correctly.
Loosely based on D6525 by @radcapricorn, but found a bug in that and
prefered to do some further adjustments.
Also activates test for this case.
Adds tests covering a good amount of common cases and corner cases for
`BLI_strcasecmp_natural()`.
Could of course always add more tests for more cases, but don't want to
spend too much time on this.
Those tests are here mostsly to ensure ID name management is working as
expected (the code ensuring we never have two ilocal data-blocks of the
same type with the same name in a .blend file).
Note: Currently fails in some cases, fixes are incoming.
Note: Ideally this would be in C, but we already have too many tests
linking the whole Blender and its libraries, this is becoming a real
pain to link debug + ASAN + tests build these days... So until we find a
better way to handle those dependencies, sticking to simple python
scripts.
This commit introduces the first version of an exporter to Pixar's
Universal Scene Description (USD) format.
Reviewed By: sergey, LazyDodo
Differential Revision: https://developer.blender.org/D6287
- The USD libraries are built by `make deps`, but not yet built by
install_deps.sh.
- Only experimental support for instancing; by default all duplicated
objects are made real in the USD file. This is fine for exporting a
linked-in posed character, not so much for thousands of pebbles etc.
- The way materials and UV coordinates and Normals are exported is going
to change soon.
- This patch contains LazyDodo's fixes for building on Windows in D5359.
== Meshes ==
USD seems to support neither per-material nor per-face-group
double-sidedness, so we just use the flag from the first non-empty
material slot. If there is no material we default to double-sidedness.
Each UV map is stored on the mesh in a separate primvar. Materials can
refer to these UV maps, but this is not yet exported by Blender. The
primvar name is the same as the UV Map name. This is to allow the
standard name "st" for texture coordinates by naming the UV Map as such,
without having to guess which UV Map is the "standard" one.
Face-varying mesh normals are written to USD. When the mesh has custom
loop normals those are written. Otherwise the poly flag `ME_SMOOTH` is
inspected to determine the normals.
The UV maps and mesh normals take up a significant amount of space, so
exporting them is optional. They're still enabled by default, though.
For comparison: a shot of Spring (03_035_A) is 1.2 GiB when exported
with UVs and normals, and 262 MiB without. We probably have room for
optimisation of written UVs and normals.
The mesh subdivision scheme isn't using the default value 'Catmull
Clark', but uses 'None', indicating we're exporting a polygonal mesh.
This is necessary for USD to understand our normals; otherwise the mesh
is always rendered smooth. In the future we may want to expose this
choice of subdivision scheme to the user, or auto-detect it when we
actually support exporting pre-subdivision meshes.
A possible optimisation could be to inspect whether all polygons are
smooth or flat, and mark the USD mesh as such. This can be added when
needed.
== Animation ==
Mesh and transform animation are now written when passing
`animation=True` to the export operator. There is no inspection of
whether an object is actually animated or not; USD can handle
deduplication of static values for us.
The administration of which timecode to use for the export is left to
the file-format-specific concrete subclasses of
`AbstractHierarchyIterator`; the abstract iterator itself doesn't know
anything about the passage of time. This will allow subclasses for the
frame-based USD format and time-based Alembic format.
== Support for simple preview materials ==
Very simple versions of the materials are now exported, using only the
viewport diffuse RGB, metallic, and roughness.
When there are multiple materials, the mesh faces are stored as geometry
subset and each material is assigned to the appropriate subset. If there
is only one material this is skipped.
The first material if any) is always applied to the mesh itself
(regardless of the existence of geometry subsets), because the Hydra
viewport doesn't support materials on subsets. See
https://github.com/PixarAnimationStudios/USD/issues/542 for more info.
Note that the geometry subsets are not yet time-sampled, so it may break
when an animated mesh changes topology.
Materials are exported as a flat list under a top-level '/_materials'
namespace. This inhibits instancing of the objects using those
materials, so this is subject to change.
== Hair ==
Only the parent strands are exported, and only with a constant colour.
No UV coordinates, no information about the normals.
== Camera ==
Only perspective cameras are supported for now.
== Particles ==
Particles are only written when they are alive, which means that they
are always visible (there is currently no code that deals with marking
them as invisible outside their lifespan).
Particle-system-instanced objects are exported by suffixing the object
name with the particle's persistent ID, giving each particle XForm a
unique name.
== Instancing/referencing ==
This exporter has experimental support for instancing/referencing.
Dupli-object meshes are now written to USD as references to the original
mesh. This is still very limited in correctness, as there are issues
referencing to materials from a referenced mesh.
I am still committing this, as it gives us a place to start when
continuing the quest for proper instancing in USD.
== Lights ==
USD does not directly support spot lights, so those aren't exported yet.
It's possible to add this in the future via the UsdLuxShapingAPI. The
units used for the light intensity are also still a bit of a mystery.
== Fluid vertex velocities ==
Currently only fluid simulations (not meshes in general) have explicit
vertex velocities. This is the most important case for exporting
velocities, though, as the baked mesh changes topology all the time, and
thus computing the velocities at import time in a post-processing step
is hard.
== The Building Process ==
- USD is built as monolithic library, instead of 25 smaller libraries.
We were linking all of them as 'whole archive' anyway, so this doesn't
affect the final file size. It does, however, make life easier with
respect to linking order, and handling upstream changes.
- The JSON files required by USD are installed into datafiles/usd; they
are required on every platform. Set the `PXR_PATH_DEBUG` to any value
to have the USD library print the paths it uses to find those files.
- USD is patched so that it finds the aforementioned JSON files in a path
that we pass to it from Blender.
- USD is patched to have a `PXR_BUILD_USD_TOOLS` CMake option to disable
building the tools in its `bin` directory. This is sent as a pull
request at https://github.com/PixarAnimationStudios/USD/pull/1048
This new test class minimally sets up Blender so that it can load blend
files and construct a depsgraph without crashing.
Note that it hasn't been tested on very complex blend files, so it may
still crash when the loaded blend file references/requires uninitialised
data structures.
The test will certainly crash with Blend files created with Blender
older than 2.80, as the versioning code requires space types to be
registered. This is normally done by initialising the window manager,
which is not done in this test. The WM requires Python to run, which in
turn requires that Blender finds the release directory in the same
directory that contains the running executable, which is not the case
for GTest tests (they are written to `bin/tests/executablename`.
Reviewed By: sergey, mont29
Differential Revision: https://developer.blender.org/D6246
We still had a few deprecated assignements of `bpy.props.xxx` to class
members in our API documentation and one of our py tests. Annotations
are to be used now.
Also remove the section about `register_module` utils, this has been
removed in 2.8.
Fix T71877: Python API overview sample code warning: class MyMaterialProps contains a property which should be an annotation!
Fix T71876: Python API overview references old bpy.utils.register_module function
This code allows to push a set of different operations all based on
iterations over a range of indices, and then process them all at once
over multiple threads.
This commit also adds unit tests for both old un-pooled, and new pooled
task_parallel_range family of functions, as well as some basic
performances tests.
This is mainly interesting for relatively low amount of individual
tasks, as expected.
E.g. performance tests on a 32 threads machine, for a set of 10
different tasks, shows following improvements when using pooled version
instead of ten sequential calls to BLI_task_parallel_range():
| Num Items | Sequential | Pooled | Speed-up |
| --------- | ---------- | ------- | -------- |
| 10K | 365 us | 138 us | 2.5 x |
| 100K | 877 us | 530 us | 1.66 x |
| 1000K | 5521 us | 4625 us | 1.25 x |
Differential Revision: https://developer.blender.org/D6189
Note: Compared to previous commit yesterday, this reworks atomic handling in
parallel iter code, and fixes a dummy double-free bug.
Now we should only use the two critical values for synchronization from
atomic calls results, which is the proper way to do things.
Reading a value after an atomic operation does not guarantee you will
get the latest value in all cases (especially on Windows release builds
it seems).
This reverts commit f9028a3be1f77c01edca44a68894e2ba9d9cfb14.
This is giving weird heisenbug crash on only Windows release builds...
Reverting until we understand to issue.
This code allows to push a set of different operations all based on
iterations over a range of indices, and then process them all at once
over multiple threads.
This commit also adds unit tests for both old un-pooled, and new pooled
`task_parallel_range` family of functions, as well as some basic
performances tests.
This is mainly interesting for relatively low amount of individual
tasks, as expected.
E.g. performance tests on a 32 threads machine, for a set of 10
different tasks, shows following improvements when using pooled version
instead of ten sequential calls to `BLI_task_parallel_range()`:
| Num Items | Sequential | Pooled | Speed-up |
| --------- | ---------- | ------- | -------- |
| 10K | 365 us | 138 us | 2.5 x |
| 100K | 877 us | 530 us | 1.66 x |
| 1000K | 5521 us | 4625 us | 1.25 x |
Differential Revision: https://developer.blender.org/D6189
This new function is part of the 'parallel for loops' functions. It
takes an iterator callback to generate items to be processed, in
addition to the usual 'process' func callback.
This allows to use common code from BLI_task for a wide range of custom
iteratiors, whithout having to re-invent the wheel of the whole tasks &
data chuncks handling.
This supports all settings features from `BLI_task_parallel_range()`,
including dynamic and static (if total number of items is knwon)
scheduling, TLS data and its finalize callback, etc.
One question here is whether we should provide usercode with a spinlock
by default, or enforce it to always handle its own sync mechanism.
I kept it, since imho it will be needed very often, and generating one
is pretty cheap even if unused...
----------
Additionaly, this commit converts (currently unused)
`BLI_task_parallel_listbase()` to use that generic code. This was done
mostly as proof of concept, but performance-wise it shows some
interesting data, roughly:
- Very light processing (that should not be threaded anyway) is several
times slower, which is expected due to more overhead in loop management
code.
- Heavier processing can be up to 10% quicker (probably thanks to the
switch from dynamic to static scheduling, which reduces a lot locking
to fill-in the per-tasks chunks of data). Similar speed-up in
non-threaded case comes as a surprise though, not sure what can
explain that.
While this conversion is not really needed, imho we should keep it
(instead of existing code for that function), it's easier to have
complex handling logic in as few places as possible, for maintaining and
for improving it.
Note: That work was initially done to allow for D5372 to be possible... Unfortunately that one proved to be not better than orig code on performances point of view.
Reviewed By: sergey
Differential Revision: https://developer.blender.org/D5371
Problem was twofold
1) `GENERATOR_IS_MULTI_CONFIG` is a property not a variable so
the test for it would always be false, unless you set a custom
CMAKE_INSTALL_PREFIX (like the buildbot does) the unit tests
would have a wrong working directory and complain about missing
dlls or blender executable
2) Tests added outside of `/test` (like libmv) would have no working
folder set since the variable would not be visible for them.
consulted @sergey who voiced the opinion that duplicating the code
to the test macro was slightly less evil than moving it to the main
CMakeLists.txt
The problematic video from T68091 clearly has an invalid stream duration
(it would be 55 centuries long if interpreted at 30 FPS, and given that
it was recorded with an Android 9 device, it's unlikely that recording
started that long ago). I've added a heuristic to check the stream
duration against the container duration; if the stream is more than 4x
longer than the container, Blender now falls back to the container
duration.
We could use MIN(stream duration, container duration), but there might
be video files out there where the container duration is less precise
than the stream duration; they are measured in different units of time
(microseconds for the container vs. frames for the stream).
Includes a unit test for the above heuristic.
Reviewed by: jbakker
Differential revision: https://developer.blender.org/D5853
The function now allows custom return types defined
by the callbacks. This can be useful when a user of the
data structure has to implement some custom behavior.
This commit adds some new hashing based data structures to blenlib.
All of them use open addressing with probing currently.
Furthermore, they support small object optimization, but it is not
customizable yet. I'll add support for this when necessary.
The following main data structures are included:
**Set**
A collection of values, where every value must exist at most once.
This is similar to a Python `set`.
**SetVector**
A combination of a Set and a Vector. It supports fast search for
elements and maintains insertion order when there are no deletes.
All elements are stored in a continuous array. So they can be
iterated over using a normal `ArrayRef`.
**Map**
A set of key-value-pairs, where every key must exist at most once.
This is similar to a Python `dict`.
**StringMap**
A special map for the case when the keys are strings. This case is
fairly common and allows for some optimizations. Most importantly,
many unnecessary allocations can be avoided by storing strings in
a single buffer. Furthermore, the interface of this class uses
`StringRef` to avoid unnecessary conversions.
This commit is a continuation of rB369d5e8ad2bb7.
These two data structures reference strings somewhere in memory.
They do not own the referenced string. The string is considered
const.
A string referenced by StringRefNull can be expected to be
null-terminated. That is not the case for StringRef.
This commit is a continuation of rB369d5e8ad2bb7c2.
Many generic C++ data structures have been developed in the
functions branch. This commit merges a first chunk of them into
master. The following new data structures are included:
Array: Owns a memory buffer with a fixed size. It is different
from std::array in that the size is not part of the type.
ArrayRef: References an array owned by someone else. All elements
in the referenced array are considered to be const. This should
be the preferred parameter type for functions that take arrays
as input.
MutableArrayRef: References an array owned by someone else. The
elements in the referenced array can be changed.
IndexRange: Specifies a continuous range of integers with a start
and end index.
IntrusiveListBaseWrapper: A utility class that allows iterating
over ListBase instances where the prev and next pointer are
stored in the objects directly.
Stack: A stack implemented on top of a vector.
Vector: An array that can grow dynamically.
Allocators: Three allocator types are included that can be used
by the container types to support different use cases.
The Stack and Vector support small object optimization. So when
the amount of elements in them is below a certain threshold, no
memory allocation is performed.
Additionally, most methods have unit tests.
I'm merging this without normal code review, after I checked the
code roughly with Sergey, and after we talked about it with Brecht.
Currently unused, but will allow to keep of an owner of the depsgraph.
Could also simplify other APIs in the future by avoiding to pass bmain
explicitly to relation update functions and things like that.
Blender can only be run correctly from the install path since it requires Python
scripts, dynamic libraries and other files to be present. By default the install
path is the same as the build path, so it works anyway. But on the buildbot it
isn't. There was a workaround but it failed on Windows and macOS.
Now tests run from the install path. Detecting that path for ctest is more
complicated than I would like, but I couldn't find a better solution.
Ref T69541.
Bugs were: (1) needed an epsilon test in CCW test in order to
handle new costraint edge that intersects an existing point
but only within epsilon; (2) the "valid bmesh" output mode
sometimes left a face that included outside frame point.
- Remove use_screen_refraction as it conflict with SSR and SSS
- Increase GTAO distance
- Add a simple lightprobe setup that works well in most cases
- Enable soft shadows
Baking the lightprobes adds some overhead to the test time (+33%).
Reviewers: brecht
Differential Revision: https://developer.blender.org/D5507
The most common use of the text editor seems to be for scripting. Having
line numbers and syntax highlighting enabled by default seems sensible.
Syntax highlighting is now enabled by default, but is automatically
disabled when the datablock has a non-highlighted extension.
Highlighting is enabled for filenames like:
- Text
- Text.001
- somefile.py
and is automatically disabled when the datablock has an extension for
which Blender has no syntax highlighter registered.
Reviewers: billreynish, campbellbarton
Subscribers: brecht, billreynish
Differential Revision: https://developer.blender.org/D5472
See Design task T68277, and patch D5423.
This commit includes edits by @ideasman42 to patch in
branch temp-D5423-update, plus responses to his comments.
Blender startup time and shader compilation is a big factor when running
hundreds of tests, so now all renders in the same ctest run in the same
process.
This was previously reverted due to skipping other tests when one test
crashed. Now if a test crashes, Blender is re-run with the remaining
tests so we get results from them still.
Fix things to make test actually fail as expected (one cannot compare
functions to strings, so no more sorting for now).
Not sure how to actually fix the test though, not even sure test make
any sense anymore actually, with all those weirdo gizmos and tools
keymaps thingy...
Using a capitalized app name fits the platform guidelines. Since macOS file
systems are case insensitive by default this should not break scripts that
assume lowercase.
Nothing special to mention about regression test itself, it basically
mimics the one for `BLI_task_parallel_mempool()`...
Basic performances benchmarks do not tell us much, besides the fact that
for very light processing of listbase, even with 100k items,
single-thread remains an order of magnitude faster than threaded code.
Synchronization is just way too expensive in that case with current
code. This should be partially solvable with much bigger (and
configurable) chunk sizes though (current ones are just ridiculous
for such cases ;) )...
The task_scheduler was not being explicitly freed, leading to
unpredictable behavior when the process was exiting. The test
would pass, but would sometimes segfault at process shutdown.
There is now a checkbox to enable/disable depth of field per camera. For Eevee
this replace the scene level setting. For Cycles there is now only an F-Stop
value, no longer a Radius.
Existing files are converted based on Cycles or Eevee being set in the scene.
Differential Revision: https://developer.blender.org/D4882
This is to simplify the usage of Volumetrics.
Now it automatically detect if there is any Volumetric material in the
view and allocate the needed buffer if any.
One of the usecases is to create mesh from an object is a manner similar to
how Apply Modifiers does it, and have it in the bmain so it can be referenced
by other objects.
This usecase is something what went unnoticed in the previous API changes, so
here is a followup.
Summary of changes:
* bpy.meshes.new_from_object() behaves almost the same as before this change.
The difference now is that it now ensures all referenced data-blocks are
original (for example, materials referenced by the mesh).
* object.to_mesh() now creates free-standing Mesh data-block which is outside
of any bmain. The object owns it, which guarantees the memory never leaks.
It is possible to force free memory by calling object.to_mesh_clear().
Reviewers: brecht
Reviewed By: brecht
Differential Revision: https://developer.blender.org/D4875
This makes finding the crashing tests harder, reverting until there is a
better solution.
This reverts commit 93901e7f0a05ba471f3b4c0201500d9dfcd68c2c.
Main goal here is to make it obvious and predictable about
what is going on.
Summary of changes.
- Access to dependency graph is now only possible to a fully evaluated
graph. This is now done via context.evaluated_depsgraph_get().
The call will ensure both relations and datablocks are updated.
This way we don't allow access to some known bad state of the graph,
and also making explicit that getting update dependency graph is not
cheap.
- Access to evaluated ID is now possible via id.evaluated_get().
It was already possible to get evaluated ID via dependency graph,
but that was a bit confusing why access to original is done via ID
and to evaluated via depsgraph.
If datablock is not covered by dependency graph it will be returned
as-is.
- Similarly, request for original from an ID which is not evaluated
will return ID as-is.
- Removed scene.update().
This is very expensive to update all the view layers.
- Added depsgraph.update().
Now when temporary changes to objects are to be done, this is to
happen on original object and then dependency graph is to be
updated.
- Changed object.to_mesh() to behave the following way:
* When is used for original object modifiers are ignored.
For meshes this acts similar to mesh-copy, not very useful but
allows to keep code paths similar (i.e. for exporter which has
Apply Modifiers option it's only matter choosing between original
and evaluated object, the to_mesh() part can stay the same).
For curves this gives a mesh which is constructed from displist
without taking own modifiers and modifiers of bevel/taper objects
into account.
For metaballs this gives empty mesh.
Polygonization of metaball is not possible from a single object.
* When is used for evaluated object modifiers are always applied.
In fact, no evaluation is happening, the mesh is either copied
as-is, or constructed from current state of curve cache.
Arguments to apply modifiers and calculate original coordinates (ORCO,
aka undeformed coordinates) are removed. The ORCO is to be calculated
as part of dependency graph evaluation.
File used to regression-test (a packed Python script into .blend):
{F7033464}
Patch to make addons tests to pass:
{F7033466}
NOTE: I've included changes to FBX exporter, and those are addressing
report T63689.
NOTE: All the enabled-by-default addons are to be ported still, but
first want to have agreement on this part of changes.
NOTE: Also need to work on documentation for Python API, but, again,
better be done after having agreement on this work.
Reviewers: brecht, campbellbarton, mont29
Differential Revision: https://developer.blender.org/D4834
The choices are now World, View and 3D Cursor.
This breaks Python API compatibility, add-ons that add objects with this
parameter will need to be updated.
Differential Revision: https://developer.blender.org/D4706
Blender startup time and shader compilation is a big factor when running
hundreds of tests, so now all renders in the same ctest run in the same
process. If a test crashes, the remaining tests in the same category will
be marked as skipped.
Benchmarked on a quad core with ctest -j8.
cycles: 118.1s -> 94.3s
eevee: 66.2s -> 29.2s
workbench: 31.7s -> 8.6s
Being able to compare Eevee reference images is useful for refactoring I'm
working on so might as well add them now, even if we can still improve them.
Workbench tests are just rendering the same files as Cycles and Eevee. This
doesn't really tests many workbench settings until we add tests specifically
for them, but does cover how it it handles the different object types.
It makes no sense to load add-ons here, we already do that (in a more
complete way) in load_addons test, this is only adding overhead and
doubling code to maintain).
Also do not try to load-as-modules add-ons that are not 2.8-ready, and
some other misc fix.
load_py_modules test should be passing again now.
Thanks to @sergey who did part of the work here as well.
Use CMake's target_link_libraries instead of manually maintaining
library dependencies in a single list.
In practice adding new libraries often ended up being guess-work,
now each library lists the libraries it uses.
This was used for the game player executable so libraries
could optionally link to stubs.
If we need this functionality it can be done using target-properties
as described in T46725.
No functional change, this adds LIB definition and args to cmake files.
Without this it's difficult to migrate away from 'BLENDER_SORTED_LIBS'
since there are many platforms/configurations that could break when
changing linking order.
Manually add and enable WITHOUT_SORTED_LIBS to try building
without sorted libs (currently fails since all variables are empty).
This check will eventually be removed.
See T46725.
While it looks more longer, but also contains more comments
about what's going on.
Surely, this function almost never breaks and investing time
into maintaining its tests is not that important, but we should
have a good, clean, understandable tests so they act as a nice
example of how they are to be written. Especially important to
show correct language usage, without old school macros magic.
Doing this at a lunch breaks, so will be series of some updates
in the area.
The `BLI_path_frame_strip` function was completely broken, unless the
number of digits in the sequence number was the same as the length of
the extension. In other words, it would work fine for `file.0001.abc` (4
digit `0001` and 4 char `.abc`), but other combinations would truncate
to the shortest (`file.001.abc` would become `file.###.ab` and
`file.00001.a` would become `file.##.a`). The dependency between the
sequence number and the file extension is now removed.
The behaviour has changed a little bit in the case where there are no
numbers in the filename. Previously, `path="filename.abc"` would result
in `path="filename.abc"` and `ext=""`, but now it results in
`path="filename"` and `ext=".abc"`. This way `ext` always contains the
extension, and the behaviour is consistent regardless of whether there
were any numbers found.
Furthermore, I've removed the `bool set_frame_char` parameter, because
it was unclear, probably also buggy, and most importantly, never used.
I've also added a unit test for the `BLI_path_frame_strip` function.
To keep running these tests relatively fast and practical to run often,
running it on all .blend files is a bit much. So now we only run it on
files from this directory.
Additionally this adds supports for following symlinks, so that you can
easily symlinks to other directories if you want to tests extra files
which may have linked libraries.
