- Add support for refining the camera's intrinsic parameters
during a solve. Currently, refining supports only the following
combinations of intrinsic parameters:
f
f, cx, cy
f, cx, cy, k1, k2
f, k1
f, k1, k2
This is not the same as autocalibration, since the user must
still make a reasonable initial guess about the focal length and
other parameters, whereas true autocalibration would eliminate
the need for the user specify intrinsic parameters at all.
However, the solver works well with only rough guesses for the
focal length, so perhaps full autocalibation is not that
important.
Adding support for the last two combinations, (f, k1) and (f,
k1, k2) required changes to the library libmv depends on for
bundle adjustment, SSBA. These changes should get ported
upstream not just to libmv but to SSBA as well.
- Improved the region of convergence for bundle adjustment by
increasing the number of Levenberg-Marquardt iterations from 50
to 500. This way, the solver is able to crawl out of the bad
local minima it gets stuck in when changing from, for example,
bundling k1 and k2 to just k1 and resetting k2 to 0.
- Add several new region tracker implementations. A region tracker
is a libmv concept, which refers to tracking a template image
pattern through frames. The impact to end users is that tracking
should "just work better". I am reserving a more detailed
writeup, and maybe a paper, for later.
- Other libmv tweaks, such as detecting that a tracker is headed
outside of the image bounds.
This includes several changes made directly to the libmv extern
code rather expecting to get those changes through normal libmv
channels, because I, the libmv BDFL, decided it was faster to work
on libmv directly in Blender, then later reverse-port the libmv
changes from Blender back into libmv trunk. The interesting part
is that I added a full Levenberg-Marquardt loop to the region
tracking code, which should lead to a more stable solutions. I
also added a hacky implementation of "Efficient Second-Order
Minimization" for tracking, which works nicely. A more detailed
quantitative evaluation will follow.
- Remove the overly-conservative use of libmv's re-track tracker. The re-track
tracker would take a normal tracker such as TRKLT or KLT or pyramid KLT, and
track from frame 1 to 2, then back from the position found in 2 back to 1.
Then, when the reverse-track doesn't match the original track with high
precision, the track is considered "failed". This is a good approach for
fully automatic reconstruction, but is too conservative for supervised
tracking.
The retrack-tracker will return when fully automatic tracking is added.
- Always solve for (dx, dy) in the TRKLT loop even if the linear system is
ill-conditioned. The client (Blender in this case) can still use the solved
position, even though it is less reliable.
- Expose the pyramid level setting to the tracking UI when in KLT tracking
mode. While it was tempting to hide this detail from the user, in reality it
does more harm than good, since the way tracking fails depends on the pyramid
level. For now, exposing the pyramid details is a good compromise.
- Move the settings for the tracking algorithm, including search window,
tracking algorithm, pyramid levels, etc, into the track object instead of a
global movie clip object.
===========================
Initial implementation of graph view for movie tracking data.
Used the same UI-side approach as preview region for sequencer:
create region for graph-related information inside clip editor.
It's easier and nicer than trying to hack graph editor which is
currently designed to work with AnimData only. Trying to make it
more abstract to deal with any kind of data doesn't seem be real
benefit for now.
Currently supported displaying per-frame average error and
selected tracks' movement speed (pixels per frame).
Additional changes:
- Collect per-frame average error after solving.
- Split space clip drawing code into different files.
- Added per-frame average solving error.
===========================
Added option to choose which method to use for feature detector: FAST or Moravec
Moravec was supposed to be more "stable", but it collected worse feedback.
before returning back to FAST detector added option to easier toggle between
different detector to compare it's result and find out if we can leave only FAST.
===========================
Support for (un)distortion of images with overscan. Needed for
easier keeping this up-to-date until proper overscan support is
implemented in renderer. Not exposed into UI yet at all.
===========================
Fixed threading issues used to happen when compositor tree
and rendering happened in the same time. Problem was caused
by cached camera intrinsics stored in MovieTracking structure
and access to which wasn't thread-safe.
Now each movie distoriton mode stores it's own camera intrinsics
in it's storage which makes compositor (un)distoriton be totally
thread-safe and this also makes compositor tree faster when
(un)distortion happens on images with different resolution.
===========================
- Merge Movie Distort and Movie Undistort nodes into one node
called Movie Distortion where action (distort/undistort is specified).
- Implemented more proper distortion/undistortion for scaled images.
NOTE: Please, delete distortion nodes with current blender first
before opening files with blender versions from this commit/
===========================
Lookup grids used for image distortion/undistortion are now
caching and re-building if distortion settings are changing.
This speedups compositor node for cases distortion model isn't
changing, but first run and runs after changing distortion
coefficients is still slow.
===========================
Bundle new libmv to fix crash caused by some errors in detector.
This commit makes SAD tracking much slower because now it supports
afgine tracking. Not implemented in Blender yet to keep commits
more clear.
===========================
- Bundling new libmv, issues with undefined uint and C99 functions should gone.
- Changes to Detect operator to match new API.
===========================
- Bundle new version of libmv.
- Use image from nearest track's keyframe as reference for tracking.
This should reduce drifting.
- Added option to switch to new SAD tracker.
This tracker doesn't support pattern with variable size (yet?)
and always uses pattern 16x16px.
Tracker could be choosed in Tracking Settings panel.
===========================
- Distortion/undistortion were accidentally swapped in both of grease
pencil calibration and grid.
This was fixed but distortion coefficients should be re-adjusted.
There are some issues with high distortion coefficients but it
came from libmv and until it'll be fixed there nothing really
can be done here.
- Fixed displayed aspect ratio for muted footage when pixel/display
aspect ratio is differ from 1.0 or proxy size is not Full Render,
- Added c-api for image distortion/undistortion, Not used yet.
===========================
- Added function to calculate per-track average error.
Made this in libmv c-api, probably should be ported
to libmv pipeline.
- MovieTrackingTrack now holds average reprojection error
which is displayed in Active Track panel and information
text in clip editor (if displaying of names are enabled).
- Make "Clean Tracks" take average error on account.
===========================
- Made naming in libmv-api more uniform.
- Move aspect ration detection of clip to movieclip.c.
- Remove unused width and height from camera intrinsics function.
- Reconstruction should now deal correct with aspect ratio.
===========================
- Fixed occasional crash caused by conflicting access
to non-threadsafe moviecache. Added lock on movie clip
ibuf acquire level.
- A bit nicer feedback from libmv when doing reconstruction.
Now if all tracks and frames were reconstructed average
error would be reported in "notification area".
If there's something failed to be reconstructed warning
message would appear and failed items would be printed to
the console.
===========================
- Fixed crash when holding down S and R keys.
- Fixed deleting of first marker.
- Tracking should now work fine with float buffers.
- Added option to show distortion grid. It's most probably
temporary solution until [auto]calibration tools aren't
merged.
This grid defines straight lines on the image.
===========================
- Bundle re-newed libmv library.
- Update API in libmv-capi to match new library.
- Initial implementaition of feature detection.
- Fixed bug with incorrect user sync when tracking
for first frame.
===========================
- Changed libmv api to use doubles instead of floats.
No real benefit rather than keeping API uniform.
- Optimized reconstructed camera search. It's optimized for
playback, not random access.
- Added option to show projection of bundles into footage.
To see bundles "Show Bundles" from Display panel should
be enabled. Used very rough limit of 3 px to consider projection
is fine. Colors are still hard-coded. Not sure it could be useful
to make them configurable.
- Added option to mute footage. It could be useful to check
if markers/bundles are moving smoothly.
- Added selector for focal length units.
===========================
- ColaMD moved from OpenNL to extern/.
It'll be needed for libmv. Also, it's a bit updated from
year 1999 to 2007.
Need to be tested for regressions.
- Updated bundling script for libmv. Now it uses fuller
subset of this library.
- Bundled new libmv.
- Request from Keir: add command line argument to toggle logging
stuff on. Currently, if Blender is launched with -d argument
libmv would start printing logging messages. There's no
argument to increase verbosity, but there's API in libmv-capi,
so it'll be easy to add.
- Finally fixed crash when ibuf is acquiring with user=NULL.
- Added ActiveClip property to the scene. This clip is used
as default value for new match-moving constraints.
- Added some flags to Display panel of View3D. Related on
displaying match-moving stuff.
- Internal change: bundles data moved inside to MovieTrackingTrack.
- Initial implementation of 3d reconstruction.
- Added constraint "Camera Solver". This constraint is supposed
to be used to make camera follow the reconstructed camera path.
- Added "reference" property to "Follow Track" constraint.
Now object could be "parented" to 2D track position or to
3D bundle position.
The very quick guide:
To use reconstruction you should have footage with tracked markers,
choose two keyframes in "Tracking settings" panel. There should be
quite noticeable parallax effect between this two frames. This
is used to initialize reconstruction stuff.
Camera data (focal length and optical center) should be filled in
"Camera Data" panel. Optical center is often the center of image,
so it'll be filled in automatically.
You should also set values for undistortion (K1, K2 and K3). Currently,
there's no any visualization for this parameters and approach of
"change value -> reconstruct -> see what've changed" is the only way
for now.
Libmv team is working on auto-calibration tool, so it should be
easier to gather this coefficients in nearest (i hope) future.
There's also no scene orientation stuff.
Basic workflow:
- Open footage.
- Set markers and track them.
- Fill in camera data and keyframes.
- Hit "Solve Camera" button.
- Add "Camera Solver" constraint to camera in scene.
- Choose movieclip in that constraint.
- To see bundles in 3D viewport active clip should be set
in scene buttons.
===========================
- Corrected copyrights in my new files header.
- Added debug option to dump as PNG image content of search
areas tracking between which was failed.
- Removed internal tracking settings like pyramid level and
tolerance. Weren't useful neither for me nor for artists.
- Fixed bug in BKE_tracking_get_marker which could lead
to failure of getting marker for current frame when
there's marker for requested frame.
- Fixed bug with disappearing markers after tracking for
cases when marker was added at frame different from
tracking start frame and tracking start frame is
equal to tracking end frame.
- Stop tracking when no markers could be tracked.
- Changed marker selection rules:
* If marker isn't selected, all it's areas are getting
selected with mouse.
* If marker is selected and selection type doesn't have
"extend" mode (SHIFT isn't holded down) the nearest
to mouse cursor area would be selected. All the reast areas
would loose selection.
- Limit number of frames which would be used in "Track Markers"
operator. So now you could easily track along 10/20/etc frames.
- Change scene current frame after "Track Markers" operator.
- Implemented speed limitors. Now tracking could happen:
* As fast as possible
* With reatime speed (scene FPS)
* With half of realtime speed
* With quarter of realtime speed.
- Got rid of orig_user in MovieTrackingContext and user
in TrackMarkersJob. This prevents crashes when user
joins some space to SpaceClip from which tracking was started.
- Made keyframes on path bigger and do not draw keyframes for
non-selected markers.
TODO: need to get rid of storing main, scene and screen
in TrackMarkersJob. This is needed to set scene's
current frame after tracking (such behavior is comfortable
for artists) but this could lead to crash if scene is
getting removed when tracking job is running.
Render animation operator also stores scene and main,
so such solution could be fine for first time before
we've got something smarter.
===========================
- itasc adopted for Eigen3 library.
It compiles well, but need deeper testing for regressions.
- Removed Eigen2 library.
- Added settings to tracker which could be changed from UI.
- Pattern area is now affects on tracker.
Currently libmv supports square patterns which are centered
to marker's position. Maximal pattern dimensions is sending
to libmv as pattern size. Would be changed when libmv would
support non-centered and non-square patterns.
- Fixed bug with syncing pattern's flags when tracking.
- Current frame in cache line became a bit more visible.
It's useful for me to when debugging.
- Changed behaviour of "Add Marekr" operator: not it's non-modal
and places marker on mouse position at click.
- Added macro "Add Marekr and Move" which is used to place
markers from toolbar button.
- Added some utility functions to get image buffer under search
and pattern area which also returns relative position of
marker center for this images.
Generated images are more "correct" from coords rounding
POV, but re-calculation of marker position back to
frame coords is more complicated and not implemented yet,
so old not very accurate logic is still used.
- Added preview widget with content of pattern area.
NOTE: files saved in previous versions of this branch could
easily crash on tracking. Use "Reset To Settings" button
from Tracking Settings before tracking selected markers
for such files.
TODO:
- Implement adjusting marker position from marker's
preview widget.
- We've got an idea of sliding marker after click before
releasing mouse button.
===========================
- Added missed stub for blenderplayer, so it should
compile again fine.
- libmv is added to extern/.
Script for bundling needed subset of sourcses from
libmv repo is included there.
- Eigen3 is also added to extern/.
libmv depends on Eigen3 (which isnt't compatible with Eigen2
from API point of view). So both of versions are going to
be in worktree for time (until code which used Eigen2 would be
ported to Eigen3).
- Added some functions to libmv-capi, so some things from
this library could already be used.
- Added the very basic implementation of 2D tracking.
It should be treated as draft for tracking architecture
which probably would be cleaned up, changed or whatever else.
Current implementation was supposed to demonstrate that our
structures and understading is correct to interact with libmv easily.
