While SCons building system was serving us really good for ages it's no longer
having much attention by the developers and started to become quite a difficult
task to maintain.
What's even worse -- there started to be quite serious divergence between SCons
and CMake which was only accumulating over the releases now. The fact that none
of the active developers are really using SCons and that our main studio is also
using CMake spotting bugs in the SCons builds became quite a difficult task and
we aren't always spotting them in time.
Meanwhile CMake became really mature building system which is available on every
platform we support and arguably it's also easier and more robust to use.
This commit includes:
- Removal of actual SCons building system
- Removal of SCons git submodule
- Removal of documentation which is stored in the sources and covers SCons
- Tweaks to the buildbot master to stop using SCons submodule
(this change requires deploying to the server)
- Tweaks to the install dependencies script to skip installing or mentioning
SCons building system
- Tweaks to various helper scripts to avoid mention of SCons folders/files
as well
Reviewers: mont29, dingto, dfelinto, lukastoenne, lukasstockner97, brecht, Severin, merwin, aligorith, psy-fi, campbellbarton, juicyfruit
Reviewed By: campbellbarton, juicyfruit
Differential Revision: https://developer.blender.org/D1680
Runtime costs were horrible. On gooseberry in some sequencer edits using
proxies of small size, a cache with about 2000 elements would slow to
about 6 fps once cache was full and system tried to find smallest
element available.
There are still improvements to be done here, like requesting a number
of good candidates to avoid rerunnung through the list, or even using
some heap or ring buffer scheme to sort data, but nothing suits all
needs so for now that should bring the cache back to usable state (25fps
here at the studio)
Issue was caused by recent image cache rewrite and root of
the issue goes to the fact that blender player doesn't
initialize cache limiter and it uses 32meg of memory only.
This leads to infinite image loading/freeing.
For now disabled cache limiter in game engine, this brings
back old behavior.
In theory we might be smarter here, but better caching
policy is to be discussed.
This callback is used when cache limiter needs to remove
some cached objects when running out of limit.
From blender side it's used to keep painted images always
in memory.
This fixes issue when painted images were removing from
the memory after image cache rewrite.
This commit basically implements frames prefetching for
movie clip datablock.
Number of frames to be prefetched is controlled in User
Preferences, System tab, Prefetch Frames option.
Currently prefetching is destructive-less for movie cache,
meaning mo frames will be removed from the cache when while
prefetching. This is because it's half of simplier to
implement, but it also makes sense from tracking point of
view -- we could want to playback in both directions and
removing frames from behind time cursor is not always a
good idea.
Anyway, smarter prefetching strategy could be developed
later.
Some implementation notes:
- Added MEM_CacheLimiter_get_memory_in_use function to get
memory usage of specified memory limiter.
- Fixed prototype of MEM_CacheLimiter_get_maximum which
was simply wrong (used wrong data type for output).
- Added some utility functions to movie clip and movie
cache for direct cache interaction and obtaining cache
statistics.
- Prefetching is implemented using general jobs system.
which is invoking from clip draw function.
- Prefetcing will stop as soon other job or playback starts.
This is done from performance point of view. Jobs will
likely require lots of CPU power and better to provide
whole CPU to it.
Playback is a bit more complicated case. For jpeg sequence
playback prefetching while paying back is nice. But trying
to prefetch heavy exr images and doing color space
conversion slows down both playback and prefetching.
TODO:
- Think of better policy of dealing with already cached frames
(like when cached frames from other clips prevents frames
from current clip to be prefetched)
- Currently a bit funky redraw notification happens from
prefetch job. Perhaps own ND_ is better to have here.
- Hiding clip while prefetch is active in theory shall stop
prefetching job.
- Having multiple clips opened on file load will prefetch
frames for only one of them.
* MEM_CacheLimitier - Size type to int conversion, should be safe for now (doing my best Bill Gates 640k impression)
* OpenNL CMakeLists.txt - MSVC and GCC have slightly different ways to remove definitions (DEBUG) without the compiler complaining
* BLI_math inlines - The include guard name and inline option macro name should be different. Suppressed warning about not exporting any symbols from inline math library
* BLI string / utf8 - Fixed some inconsistencies between declarations and definitions
* nodes - node_composite_util is apparently not used unless you enable the legacy compositor, so it should not be compiled in that case.
Leaving out changes to BLI_fileops for now, need to do more testing.
Memory limitor's queue could be affected when it's being iterated
on enforcing limits -- that's because iteration could free color
managed image buffers.
Fixed by getting least priority element after every element was
freed. Could be optimized a bit, but it anyway shouldn't be so
slow due to specific of cache limiting and limit enforcing finish
condition.
Replace pseudo-LRU approach of determining which buffer
to remove when running out of space allowed for cache
with approach which would remove the frame which is most
far away from newly added frame.
This is still a bit tricky because it's impossible to
distinguish which frame to delete in situation of:
CCCC...CC
^
it's either user wants to extend left segment of cached
frames and buffers from right segment should be removed
or he wants to join this two segments and in that case
buffers from right segment should be removed.
Would need a bit more investigation which situation
is more common in general usecase.
Additional changes:
- Cleanup some memutil files (which are familiar to cache limiter)
- Add option to make moviecache verbose. If DEBUG_MESSAGES is
defined in moviecache.c detailed logs would be printed to the
console.
- Movie caches are now named which helps reading debug messages.
"The Blender Foundation also sells licenses for use in proprietary software under the Blender Licens"
also remove NaN references from files that have been added since blender went opensource.
blender_add_lib now takes a separate include argument to suppress warnings in system includes (mostly ffmpeg & python).
also only build wm_apple.c on apple+carbon configuration.
globbing vs explicit is discussed here.
http://www.cmake.org/pipermail/cmake/2008-December/025694.html
Practical implications are:
- developers need to keep CMakeLists.txt files up to date.
- Users wont get strange linking errors if they build after a file is added, since CMake detects CMakeLists.txt is modified and automatically reconfigure.
