This change makes it so build system and update utilities for Blender builds
are using pre-compiled libraries and other resources attached as Git modules
instead of using checkout of SVN repositories in the parent folder.
The directory layout:
```
* release/datafiles/
* assets/ -> blender-assets.git
* publish/
* ...
* README.txt
* lib/
* darwin_x64/ -> lib-darwin_x64.git
* darwin_arm64/ -> lib-darwin_arm64.git
* linux_x64/ -> lib-linux_x64.git
* windows_x64/ -> lib-windows_x64.git
* tests/
* data/ -> blender-test-data.git
```
The changes about configuring the actual Git sub-modules are not included
into this patch, as those require repository to actually exist before it
can be used.
The assets submodule is enabled by default, and the rest of them are
disabled. This means that if someone runs `git submodule update --init`
they will not get heavy libraries. The platform-specific and tests
related submodules are enabled when using `make update` or `make test`.
All the submodules are tracked: this means that when new commits are
done to the submodule, the blender.git repository is to be updated to
point them to the new hash. This causes some extra manual work, but it
allows to more easily update Blender and its dependencies to known good
state when performing operations like bisect.
Ref #108978
Pull Request: https://projects.blender.org/blender/blender/pulls/117946
People running `make update` first time, will not have a lib
folder yet, and get a warning about python being missing, this
sends some of them off onto somewhat of a wild goose chase
installing python when this is really not needed since it's
included in our libraries.
This change, changes the warning to only emit when the lib
folder exists, but python is missing in the lib folder and
there actually is likely an issue with the lib folder.
Blender had a very limited (only uncompressed or MJPEG frames) .avi file
support, for both reading and writing. This is something that ffmpeg can
fully do.
This removes all of that. 3500 lines of code gone, primary motivations being:
- ffmpeg can read and write .avi files just fine, including ones with
uncompressed or MJPEG frames.
- Blender's ffmpeg integration could also be taught to produce uncompressed or
MJPEG .avi files, but TBH I don't see a particular reason to do that. Modern
formats like H264 are better in every way, and already support "lossless"
option if needed.
- The "Lite" blender build configuration was excluding both ffmpeg and avi
anyway, so that config is something that can't read nor write any movies.
User visible changes:
- In scene image output type, under Video section now there's only Ffmpeg Video
(AVI Raw and AVI JPEG are gone)
- Whenever loading an existing file, if output was one of AVI Raw / AVI JPEG,
it is set to Ffmpeg Video.
Pull Request: https://projects.blender.org/blender/blender/pulls/118409
Main improvements:
* Spatial structure (Kd-tree) build is now multithreaded.
* Kd-tree switched to use cache-friendlier TreeLets.
* Field fixed some non-deterministic behavior when spatial cache does
not receive any training data during a training iteration due to a
large number of training iterations.
* Fixed build problems on (non-Mac) ARM systems.
Pull Request: https://projects.blender.org/blender/blender/pulls/118328
This was already the minimum requirement for Intel and Apple Silicon
GPUs. It is required for the Metal backend to work correctly.
Previously the minimum for AMD GPUs was 10.15.
Pull Request: https://projects.blender.org/blender/blender/pulls/118287
Turns out we were not building OSL with OptiX enabled anymore.
Also check now if the OSL builds has OptiX support and if not
disable it in Cycles.
Building OSL with support for this (still) does not require
either the OptiX SDK or CUDA, it only needs LLVM.
Pull Request: https://projects.blender.org/blender/blender/pulls/118234
- Commands which have arguments split over multiple lines use
indented lines.
- Wrap lines where multiple commands run using "&&".
- Blank lines between multiple commands helps the text from becoming
too dense.
This is not a hard requirement to be able to build the libraries, but
these versions should be written down somewhere. So compare compiler
versions as part of make deps setup.
Pull Request: https://projects.blender.org/blender/blender/pulls/117457
Flex's bundled configure depended on aclocal-1.15 which has been
updated to 1.16.
Resolve by regenerating configure files on Linux which in turn adds a
new dependency on texinfo.
Flex's bundled configure depended on aclocal-1.15 which has been
updated to 1.16.
Resolve by regenerating configure files on Linux which in turn adds a
new dependency on texinfo.
This is supported on Apple Silicon GPUs and macOS 13.0+.
Co-authored-by: Stefan Werner <stefan.werner@intel.com>
Co-authored-by: Attila Afra <attila.t.afra@intel.com>
Pull Request: https://projects.blender.org/blender/blender/pulls/116124
NOTE: Blender does not seem to build with older OpenImageIO versions
(2.4.x and before) anymore. Since this library is now mandatory, it
means that Blender cannot be built without using the pre-compiled libs
on most Debian system, currently.
Ref. #113157.
Since the addition of Meteor Lake binaries, prebuilt GPU binaries
are now stored as fatbinaries. When running on a platform for which
prebuilt binaries are lacking or considered incompatible, the DPC++
SYCL runtime caching logic failed storing the (re)compiled
compatible version. This patch to DPC++ SYCL runtime fixes it.
Pull Request: https://projects.blender.org/blender/blender/pulls/117844
Add silently fail option to GPU based render tests. This is a pre-requisite to enable
render tests on the buildbot. By default these render tests will pass silently.
* Test will pass when using the `--pass-silently` arguments.
* Only crashes will be reported as failed tests.
* To find out failing test, review the test reports.
`WITH_GPU_RENDER_TESTS_SILENT` compile option can be used to let tests pass (default)
or fail (default for developers).
Although some tests fail, they still passed. In the generated render report,
the silently passed failures are correctly reported to be failures.
Pull Request: https://projects.blender.org/blender/blender/pulls/117629
This change fixes confusion situation when the render output
is an RGBA image: the difference in color was not visible in
the report because alpha channel was all zeros. This is due
to idiff performing per-channel difference.
The solution to this problem is to have separate images for
color and alpha difference, which makes it clear where the
difference actually is coming from.
It's not clear what these are used for as they seem like recent
additions and are set to "NOT-FOUND" even when tiff/freetype are found.
Marking as advanced instead of removing so as not to interfere with
freetype/tiff finding logic.
Exceptions:
* Links to personal wiki pages
* Pages that are not in the new developer docs yet (like Human Interface Guidelines)
* tools\check_wiki\check_wiki_file_structure.py needs a refactor
Previous code was using the 'nuclear' SAN weapon `exitcode=0`, which
essentially makes testing ignore all sanitizer errors.
This commit removes this option, and instead uses the suppression
list from Blender source repository (`tools/config/analysis/lsan.supp`)
for the leak sanitizer.
It also enables the `allocator_may_return_null` for the address sanitizer.
This should only be needed for one test (the `guardedalloc` one), but for
now it is simpler to define it for all tests. This can be refined at a
later time.
Pull Request: https://projects.blender.org/blender/blender/pulls/116635
Along with regular fixes and improvements,
this removes a warning on startup for Linux, see:
https://github.com/kcat/openal-soft/issues/554
Some deprecated CMake variables were removed and needed to be renamed.
Note that the previous URL from http://openal-soft.org/
is no longer responsive, using the GITHUB URL instead.
Ref !116877
The current handling had a fairly bad issue: multiple calls to
`set_tests_properties` to set envvars of a same test.
This does not work, only the last call is effective, all previous
ones have absolutely no effect.
This has been addressed by moving all 'set envvar for test' logic into a
single CMake function, `blender_test_set_envvars`.
This function takes optional extra envvars if needed, and define a set
of default ones (currently, `PATH` from `PLATFORM_ENV_INSTALL` if
defined, and the 'nuke' `exitcode=0` `LSAN_OPTIONS` if relevant).
NOTE: The way `blender_test_set_envvars` handles extra envvars passed to
it as parameter is fairly basic and unsafe, in that there is no check
whether a same envvar is defined more than once. Think for now this is
an acceptable limitation.
NOTE: Although this commit _should_ be a non-functional change one, the
unification of the handling of all envvars makes it hard to ensure there is no
side effects.
The `PATH` envvar e.g. was set to either `PLATFORM_ENV_INSTALL` if defined,
or a copy of that variable's definition, but only in Windows case. So technically,
the behavior for this envvar is changed.
This commit adds a new helper to define expected properties when a
target needs to use the unity build feature.
That new helper does what was already done for existing cases, and in
addition add the target to the Ninja 'heavy' pooljobs if relevant.
Pull Request: https://projects.blender.org/blender/blender/pulls/116791
Bundling many tests in a single binary reduces build time and disk space
usage, but is less convenient for running individual tests command line
as filter flags need to be used.
This adds WITH_TESTS_SINGLE_BINARY to generate one executable file per
source file. Note that enabling this option requires a significant amount
of disk space.
Due to refactoring, the resulting ctest names are a bit different than
before. The number of tests is also a bit different depending if this
option is used, as one uses gtests discovery and the other is organized
purely by filename, which isn't always 1:1.
Co-authored-by: Sergey Sharybin <sergey@blender.org>
Pull Request: https://projects.blender.org/blender/blender/pulls/114604
The warning was introduced with XCode 15 and conflicted with some
assumptions in the CMake. Even without actual cyclic dependency
between targets CMake might decide to pass library multiple times
to the linker, to ensure all its users find symbols from it.
This behavior is expected to be tweaked in the upcoming CMake
version, but until it is released and became widely used by all
Blender macOS developers silence the warning.
Pull Request: https://projects.blender.org/blender/blender/pulls/116718
* Different fix for Mantaflow linker warnings that works with new OpenVDB.
* Use new linker for arm64 as it no longer produces warnings with latest
Xcode. Still use the old one for x86_64 as some warnings remain.
* Fix wrong x86_64 build target in deps builder.
For the upcoming 4.1 libraries.
Ref #113157
Pull Request: https://projects.blender.org/blender/blender/pulls/116708
OIIO was including imaths half.h when it did not
need to, leading to build errors in lite builds.
This issue had been fixed already upstream but
was not yet included in the version we use.
- OIDN doesn't need to be build for debug builds on windows, while
it's mostly C++ code its public API is C
- Python was not found by the HIP build
- Level Zero was downloaded during the build.
New include directory variable from [0] was only set for WIN32,
causing openimageio to fail to build with libraries updated for 4.1.
[0]: 802ac5ba5afc04a2c4bdd72ef52df281d9065824
When storing multiple binaries in a fatbinary, ocloc stores Intel
Graphics binaries for specific devices and these don't get found when
the device doesn't match fully.
Blender compiles one binary per generation rather than per device, to
gain space and compile time.
We introduce a patch at ocloc level to work around this behavior that
otherwise leads to binaries not being found and recompilation being
attempted.
Pull Request: https://projects.blender.org/blender/blender/pulls/115750
The build scripts are still referring to gpu tests as being opengl.
Although they can also use Metal or Vulkan. This PR only replaces
the work `opengl` with `gpu` for build options.
Special note is that the windows argument `with_opengl_tests` is
also replaced with `with_gpu_tests` for consistency.
Pull Request: https://projects.blender.org/blender/blender/pulls/116030
The NanoVDB headers are not compatible with Metal due to missing address
space qualifiers. We currently have a big patch for NanoVDB header
files, which is difficult to update for OpenVDB 11. Instead extract a
few hundred lines of code from NanoVDB to do just what we need.
Pull Request: https://projects.blender.org/blender/blender/pulls/115992
Simple string replacement failed when one compiler was the exact
prefix of another (`clang` & `clang++` in this case),
resolve by performing string replacement on the list items.
We disabled SSE for OIIO headers in the days we still supported
32 bit builds, as it was giving build errors there. OIIO handles
this internally now so we no longer need to explicitly disable it,
also we stopped shipping 32 bit quite a while ago
Asserts historically only have been on in debug builds, however
running the tests in a debug configuration takes a while especially
if Cycles is enabled.
This adds WITH_ASSERT_RELEASE to enable asserts in release
configurations. It is enabled for "make developer" , and will also
be used for CI.
Co-authored-by: Brecht Van Lommel <brecht@blender.org>
Pull Request: https://projects.blender.org/blender/blender/pulls/115071
vcpkg somewhat forcefully injects itself into the build system
causing some work stations to use the vcpkg headers/libs rather than
the headers from our lib folder,we added some mitigation against this
in 34b3a9583ac2433dd8d223d2bd098adc30bc72bd but still the occasional
user with build issues showed up, this should nip that in the bud once
and for all.
Harfbuzz and FriBiDi are included in our external libraries for all
platforms. This PR adds the glue to make them available as optional
build components (off by default).
Pull Request: https://projects.blender.org/blender/blender/pulls/114947
Previously the cmake code would try to run the LIBDIR specific "findX.cmake" files for both vulkan and shaderc.
However these would pick up system headers and libraries when LIBDIR were not present. This would lead to compilation errors as the system library configurations were not taken into account or queried.
This changes it so that if LIBDIR is not present, the proper pkgconfig files will be used instead.
Pull Request: https://projects.blender.org/blender/blender/pulls/114639
Using /etc/ssl only makes sense when the versions of SSL on the system
is compatible with the version Blender uses.
Failure to load the configuration for e.g. causes SSL to fail entirely
(causing downloading over HTTPS to fail).
Recently [0] de facto standard directory `/etc/ssl` was used however we
can't guarantee files in this path are compatible with Blender's SSL.
Use a known invalid path to resolve#114452.
Ref !114569
[0]: 60a8ae7830acd09c3d8e9ab52630d48ecd4d281b
WESTON is used when WITH_UI_TESTS is enabled.
Note that the system WESTON installation can be used by setting
WESTON_BIN to a path on the users system.
Ref !114164.
In order to speedup compilation, we upgrade IGC to 1.0.14828.26 along
with ocloc and the associated dependencies.
We also bump min-driver version accordingly to 26918.
Ref !114341
Configure failed to find vorbis & vorbisenc on Linux with the errors:
ERROR: vorbis not found using pkg-config
ERROR: vorbisenc not found using pkg-config
the issue was caused by configures test program failing to link the test
code against the static library because functions in libm weren't found.
Resolve by adding `$libm_extralibs` to libvorbis.
Note that it's strange this issue didn't come up before,
since it occurs on a clean build it seems necessary.
The last good commit was 8474716abb0db3b06838a57f7217bc945638d8df.
After this commits from main were pushed to blender-v4.0-release. These are
being reverted.
Commits a4880576dc from to b26f176d1a that happend afterwards were meant for
4.0, and their contents is preserved.
Resolve an error with SSL using a hard coded path to certificates on
Linux causing HTTPS access to fail.
request.urlopen(..) couldn't access any HTTPS URL's.
The first public Windows driver version with a higher number is
101.4824, so we bump the min-required driver version on Windows to this
one to ensure compatibility.
Enable dynamic sound library loading so that "make release" produces
binaries that are as portable as buildbot builds.
Remove unnecessary linker flags that are already set by default, and
don't specifiy explicit lib directoy since it is also found by default.
Pull Request: https://projects.blender.org/blender/blender/pulls/113360
This was only used for accessing cursor themes which only worked
with gnome and wasn't used in official releases.
Use the default theme or the theme defined by XCURSOR_THEME.
Eventually wp_cursor_shape_manager_v1 can be supported which avoids
having to access the theme.
When WITH_GHOST_SDL or WITH_HEADLESS were used, the message didn't make
much sense, especially since the features warned about weren't
necessarily enabled or even supported by the platform.
Replace with a `set_and_warn_incompatible` macro which only reports
configuration changes based on incompatible features.
Printing that a library is found every time CMake runs isn't helpful.
Restrict these messages for the first execution so messages are limited
to information developers may need to know such as features being
disabled because of incompatible configurations.
This PR enables vulkan backend as experimental option.
It will only be available in alpha builds on Linux and Windows.
This option is highly experimental and enabled to get some insight
on supported platforms. Don't expect a fully working Blender
yet. Also don't expect it to have usable performance.
**What is known to not work?**
* OCIO textures are not supported on Intel and AMD GPUs. sRGB/Standard is supported
on those platforms.
* AMD Polaris based GPUs on Linux will generate a crash when drawing the 3d cursor as it
doesn't support the needed vertex format. Comment out `DRW_draw_cursor` in `DRW_draw_region_info`.
* The colors in the node editor and sequencer are of as sRGB viewports aren't detected correctly.
* The image / UV editor isn't working as many texture formats haven't been tested yet. Some
tweaks are also needed to do correct depth testing.
* 3D Viewport is known to be flickering. Sometimes workbench doesn't display anything.
* 3D Viewport wireframe will crash as it uses a framebuffer with gaps between color attachments,
which isn't supported yet. (#113141)
* Rotate the view widget is partially drawn due to incompatible depth clipping.
* GPU Selection isn't working. It is expected to be solved when Overlay-Next will become the
default engine. For now disable GPU depth picking in the preferences.
* Cycles/EEVEE are known to not work with Vulkan yet. Cycles requires Vulkan Pixel Buffer.
Cuda <-> Vulkan interop might require a different approach than OpenGL as Vulkan doesn't allow
importing memory from a Cuda context. EEVEE uses features that aren't available yet in the backend
* Workbench is working, except Workbench shadows.
* EEVEE-Next basics are working. Shadows, lights are known to be not working. Materials/Shading
works in simple scenes. Changes are expected in EEVEE-Next that will break Vulkan compatibility
in the near future.
* Systems with multiple GPUs is not expected to work.
* Wayland support is in development and requires some iterations. You can start Blender, but
the protocols are not aligned yet.
* OpenXR hasn't been modified and is expected to fail.
* The backend is very strict when mis-using the GPU module. In debug builds it may crash
on asserts.
* Older drivers/GPUs might not have all the features that we require. The workarounds
for the missing features still need to be implemented.
**A word about performance**
In the project planning we focus first on stability and platform support. The performance of Vulkan is
around 20% of what we want to achieve. The reason is that each command sent to the
GPU is done one at a time. The implementation even waits until we have feedback that the GPU
is idle again.
Geometry is currently stored in System RAM. The GPU will read and cache the data when
accessing geometry. This slows down when using objects with much geometry.
Some performance features like MDI (Multi-Draw-Indirect) hasn't been implemented and
falls back to Single Draw Indirect.
**Why enable it is an experimental option?**
* Ensures that new features are being tested with Vulkan
* Ensure that building with Vulkan is possible on supported platforms
* Get feedback from developers if Vulkan can run on their system or that
there are special cases that we are not aware of. Main development
environment has been Linux/X11 with occasionally testing using Windows.
* Validate Add-ons that use the `gpu` module.
* Possible to enable GLSL validation on the buildbot. (Needs more work).
* Does it compile on all machines or does it require more changes to cmake
config. We expect it to be able to compile without installing the Vulkan SDK.
The Vulkan SDK is a very powerful tool, but only when actually doing GPU
development. Otherwise it is an overhead which slows down other
activities.
**How can the backend be enabled?**
Currently the Vulkan backend can be enabled per Blender session by starting
using the command line argument `--gpu-backend vulkan`. In the future, when
the backend is more mature, we will add a user preference to switch between
OpenGL and Vulkan.
Pull Request: https://projects.blender.org/blender/blender/pulls/113057
Previously this was the double the CPU count because:
- Modern CPU's from AMD & Intel support SMT/hyper-threading which
present twice as many cores, doubling again has little to no benefit.
- Using 2x or 4x the number of physical cores number can use a lot of
memory on systems with many cores which are becoming more common.
CMake decided to link the Debug libs for MaterialX because no specific
RelWithDebInfo configuration was provided, strange default imho but
not difficult to fix.
webp 1.3 changed the filenames on windows to include a `lib` prefix
(ie libwebp.lib rather than webp.lib) now this is a common thing
on linux and cmake has a `CMAKE_FIND_LIBRARY_PREFIXES` variable that
has a list of prefixes to look for during a `find_library` call.
`CMAKE_FIND_LIBRARY_PREFIXES` gets set during the call to the
`project` method in the main CMakeLists of a project. Now for windows
`lib` is *not* a common prefix by CMake, and it doesn't add "lib" to
CMAKE_FIND_LIBRARY_PREFIXES during that call.
so find library doesn't look for it, the libs are not found and an
unhappy time is had by all. Now the most obvious solution would be to
pass `-DCMAKE_FIND_LIBRARY_PREFIXES=lib` to CMake to sidestep this
however, the `project` call will set the variable overwriting
anything you passed through the CLI.
So the fix here is to have `find_library` counter-intuitively look
for both `libwebp` and `webp`
The last webp update changed the filenames of the webp libraries
on windows causing oiio not to find them and oiio silently build
without webp support, which only came to light after all of
blender was build and a test failed.
This change makes the OIIO validate and error out if certain
dependencies are not found at configure time so these mistakes
are caught early.
