Commit Graph

23 Commits

Author SHA1 Message Date
Sergey Sharybin
25bb441301 Planar tracking support for motion tracking
===========================================

Major list of changes done in tomato branch:

- Add a planar tracking implementation to libmv
  This adds a new planar tracking implementation to libmv. The
  tracker is based on Ceres[1], the new nonlinear minimizer that
  myself and Sameer released from Google as open source. Since
  the motion model is more involved, the interface is
  different than the RegionTracker interface used previously
  in Blender.

  The start of a C API in libmv-capi.{cpp,h} is also included.

- Migrate from pat_{min,max} for markers to 4 corners representation

  Convert markers in the movie clip editor / 2D tracker from using
  pat_min and pat_max notation to using the a more general, 4-corner
  representation.

  There is still considerable porting work to do; in particular
  sliding from preview widget does not work correct for rotated
  markers.

  All other areas should be ported to new representation:

  * Added support of sliding individual corners. LMB slide + Ctrl
    would scale the whole pattern
  * S would scale the whole marker, S-S would scale pattern only
  * Added support of marker's rotation which is currently rotates
    only patterns around their centers or all markers around median,

    Rotation or other non-translation/scaling transformation of search
    area doesn't make sense.

  * Track Preview widget would display transformed pattern which
    libmv actually operates with.

- "Efficient Second-order Minimization" for the planar tracker

  This implements the "Efficient Second-order Minimization"
  scheme, as supported by the existing translation tracker.
  This increases the amount of per-iteration work, but
  decreases the number of iterations required to converge and
  also increases the size of the basin of attraction for the
  optimization.

- Remove the use of the legacy RegionTracker API from Blender,
  and replaces it with the new TrackRegion API. This also
  adds several features to the planar tracker in libmv:

  * Do a brute-force initialization of tracking similar to "Hybrid"
    mode in the stable release, but using all floats. This is slower
    but more accurate. It is still necessary to evaluate if the
    performance loss is worth it. In particular, this change is
    necessary to support high bit depth imagery.

  * Add support for masks over the search window. This is a step
    towards supporting user-defined tracker masks. The tracker masks
    will make it easy for users to make a mask for e.g. a ball.

    Not exposed into interface yet/

  * Add Pearson product moment correlation coefficient checking (aka
    "Correlation" in the UI. This causes tracking failure if the
    tracked patch is not linearly related to the template.

  * Add support for warping a few points in addition to the supplied
    points. This is useful because the tracking code deliberately
    does not expose the underlying warp representation. Instead,
    warps are specified in an aparametric way via the correspondences.

- Replace the old style tracker configuration panel with the
  new planar tracking panel. From a users perspective, this means:

  * The old "tracking algorithm" picker is gone. There is only 1
    algorithm now. We may revisit this later, but I would much
    prefer to have only 1 algorithm. So far no optimization work
    has been done so the speed is not there yet.

  * There is now a dropdown to select the motion model. Choices:

        * Translation
        * Translation, rotation
        * Translation, scale
        * Translation, rotation, scale
        * Affine
        * Perspective

  * The old "Hybrid" mode is gone; instead there is a toggle to
    enable or disable translation-only tracker initialization. This
    is the equivalent of the hyrbid mode before, but rewritten to work
    with the new planar tracking modes.

  * The pyramid levels setting is gone. At a future date, the planar
    tracker will decide to use pyramids or not automatically. The
    pyramid setting was ultimately a mistake; with the brute force
    initialization it is unnecessary.

- Add light-normalized tracking

  Added the ability to normalize patterns by their average value while
  tracking, to make them invariant to global illumination changes.

Additional details could be found at wiki page [2]

  [1] http://code.google.com/p/ceres-solver
  [2] http://wiki.blender.org/index.php/Dev:Ref/Release_Notes/2.64/Motion_Tracker
2012-06-10 15:28:19 +00:00
Sergey Sharybin
59ef51aa27 Initial Ceres integration into Blender
Currently only put sources of Ceres library into extern/libmv/third_party and
setup CMake and SCons building systems.

Integration details:

- Even CMake build files are not re-used from Ceres's trunk: they're using some
  automatic stuff detection like glog, pthreads, protobuf and so and it's not
  so clear how to re-use that files without modifications.
  And IMO it's easier if build files are getting re-generated automatically to
  match Blender-specific setup rather than keeping changes made locally in
  Blender in sync when re-bundling Ceres library. Especially in case when it's
  already needed to support SCons build system.
- Integrated only actual sources, all tests were stripped. Probably it'll be nice
  to have them, but they'll need clear integration with current module test stuff
  in Blender.
- Suitesparse was disabled. It'll help a lot having it, but there are some difficulties
  making cholmod working fine on windows. Would be added in future
- collections_port.cc was also stripped. It's not used by Ceres's upstream and
  it gives compilation error (undefined uint32 -- looks like namespace issue).
- Currently all schur eliminators are included. Not sure if it makes sense,
  also not sure if it makes sense having them switchable on and off -- IMO better
  to have single configuration which works and does not require special tweaks
  after everything was set up.

To bundle updated version of Ceres:
- Go to extern/libmv/third_party/ceres folder
- Run ./bundle.sh

This will checkout fresh Ceres snapshot of Windows branch (which is currently
most interesting from integration into Blender POV), apply all patches listed
in patches/series and copy needed files into Blender's working copy. This will
also re-generate CMake/SCons build rules.

If you'll need extra files from Ceres repository which are not present in
Blender, you'll need to copy them manually and then run ./mkfiles.sh from
extern/libmv/third_party/ceres folder which will update list of files used
by Blender.

Thanks to Leir Mierle and Sameer Agarwal (and all others who helped developing
Ceres) this library and thanks to Keir Mierle with help integrating it into Blender!
2012-06-10 15:27:41 +00:00
Sergey Sharybin
efb9b6c1c3 libmv: bundle new upstream version of libmv from own branch
Added modal solver module which is needed for tripod solving.
For details of this solver read changelog of libmv.
2012-04-14 12:02:39 +00:00
Sergey Sharybin
e6c45cc1de libmv: bundle new upstream version from own branch with rigid registration implementation
Currently not used in blender code but is needed for some current work.
2012-04-12 11:37:51 +00:00
Sergey Sharybin
ce95b0be1e libmv: bundle new upstream version from own branch which should support
compilation of glog on hurd platform.
2012-04-11 13:51:37 +00:00
Sergey Sharybin
81e3db364d Camera tracking refactoring:
- Deduplicate patetrn sampling used in esm and lmicklt trackers and
  move SamplePattern to image/sample.h - Move computation of Pearson
  product-moment correlation into own function in new file image/correlation.h
  so all trackers can use it to check final correlation.
- Remove SAD tracker. It's almost the same as brute tracker, with only two differences:
  1. It does brute search of affine transformation which in some cases helps to track
     rotating features 2. It didn't use common tracker api which probably gave some
     speed advantage, but lead to a real headache to use it together with other
     trackers leading to duplicated code in blender side.
- Switch blenedr to use brute tracker instead of sad tracker which tracking made
  source code much more simple to follow.
2012-03-30 10:37:39 +00:00
Campbell Barton
87681b46ea fix for cmake error on non osx systems 2012-03-23 10:39:59 +00:00
Campbell Barton
f127a0cd6f modify osx CMake hack so the CMake files can be parsed more easily by naive my error checker. 2012-03-21 12:15:02 +00:00
Sergey Sharybin
42b3463030 Bundle new upstream version of libmv from own branch
This version of libmv includes new gflags and glog libraries which makes
it possible to compile libmv with clang compiler.

Also remove code from CMakeLists which was disabling libmv if using clang.

Tested on linux with gcc-4.6 and clang-3.0, windows cmake+msvc and scons+mingw.
Could be some issues with other platforms/build system which shall be simple to resolve.
2012-03-11 19:52:25 +00:00
Jens Verwiebe
fa169fdcb6 OSX/libmv: restrict the unwind.h workaround to 10.6.sdk only 2012-03-11 12:15:39 +00:00
Jens Verwiebe
2382afa968 OSX/libmv: added a ahck to fix compiling with 10.6.sdk, unwind.h related 2012-03-11 11:23:30 +00:00
Sergey Sharybin
17f6f7e2af Camera tracking: switch to own repo of libmv where most of patches are applied
and which includes latest changes from Keir's branch.

Hopefully it'll make backporting of changes back to main libmv repo easier.
2012-02-17 15:39:32 +00:00
Campbell Barton
758c228263 update gpl header in cmake files 2012-02-11 04:05:00 +00:00
Sergey Sharybin
0effb45d58 Fixed typo in comments for libmv build rules files 2012-01-11 07:19:44 +00:00
Sergey Sharybin
42ae315aef Added note that cmake/scons rules are automatically generated for extern/libmv
Also updated generation scripts and templates
2011-12-15 12:44:05 +00:00
Campbell Barton
9b762a51b1 missing config_freebsd.h from cmake file list 2011-12-11 00:53:37 +00:00
Sergey Sharybin
d261623800 Camera tracking: merge hybrid tracker from tomato branch
Comment from Keir's commit:

Add a new hybrid region tracker for motion tracking to libmv, and
add it as an option (under "Hybrid") in the tracking settings. The
region tracker is a combination of brute force tracking for coarse
alignment, then refinement with the ESM/KLT algorithm already in
libmv that gives excellent subpixel precision (typically 1/50'th
of a pixel)

This also adds a new "brute force" region tracker which does a
brute force search through every pixel position in the destination
for the pattern in the first frame. It leverages SSE if available,
similar to the SAD tracker, to do this quickly. Currently it does
some unnecessary conversions to/from floating point that will get
fixed later.

The hybrid tracker glues the two trackers (brute & ESM) together
to get an overall better tracker. The algorithm is simple:

1. Track from frame 1 to frame 2 with the brute force tracker.
   This tries every possible pixel position for the pattern from
   frame 1 in frame 2. The position with the smallest
   sum-of-absolute-differences is chosen. By definition, this
   position is only accurate up to 1 pixel or so.
2. Using the result from 1, initialize a track with ESM. This does
   a least-squares fit with subpixel precision.
3. If the ESM shift was more than 2 pixels, report failure.
4. If the ESM track shifted less than 2 pixels, then the track is
   good and we're done. The rationale here is that if the
   refinement stage shifts more than 1 pixel, then the brute force
   result likely found some random position that's not a good fit.

svn command used: svn merge -r 42375:42376 -r 42377:42379 ^/branches/soc-2011-tomato
2011-12-04 13:26:11 +00:00
Sergey Sharybin
c74f6a51b6 Fix #29469: Latest motion tracking not working.
It was error in CMakeLists.txt caused by automatic bundling script which
expanded variables instead of substituting them as-is.

Fixed both of bundling script and CMakeLists.txt
2011-12-01 10:25:16 +00:00
Sergey Sharybin
9f3c921957 Camera tracking: moved camera solver into it's own job
In some cases solving can take a while (especially when refining is used)
and keeping interface locked is a bit annoying. Now camera solver is moved
to job system and interface isn't locking.

Reporting progress isn't really accurate, but trying to make it more linear
can lead to spending more effort on it than having benefit. Also, changing
status in the information line helps to understand that blender isn't hang
up and solving is till working nicely.

Main changes in code:
- libmv_solveReconstruction now accepts additional parameters:
  * progress_update_callback - a function which is getting called
    from solver algorithm to report progress back to Blender.
  * callback_customdata - a user-defined context which is passing
    to progress_update_callback so progress can be updated in needed
    blender-side data structures.

  This parameters are optional.

- Added structure MovieTrackingStats which is placed in MovieTracking
  structure. It's supposed to be used for displaying information about
  different operations (currently it's only camera solver, but can be
  easily used for something else in the future) in clip editor.
  This statistics structure is getting allocated for time operator is
  working and not saving into .blend file.

- Clip Editor now displays statistics stored in MovieTrackingStats structure
  like it's done for rendering.
2011-11-28 13:49:42 +00:00
Sergey Sharybin
6fbc4186fd Assorted camera tracker improvements
- Add support for refining the camera's intrinsic parameters
  during a solve. Currently, refining supports only the following
  combinations of intrinsic parameters:

    f
    f, cx, cy
    f, cx, cy, k1, k2
    f, k1
    f, k1, k2

  This is not the same as autocalibration, since the user must
  still make a reasonable initial guess about the focal length and
  other parameters, whereas true autocalibration would eliminate
  the need for the user specify intrinsic parameters at all.

  However, the solver works well with only rough guesses for the
  focal length, so perhaps full autocalibation is not that
  important.

  Adding support for the last two combinations, (f, k1) and (f,
  k1, k2) required changes to the library libmv depends on for
  bundle adjustment, SSBA. These changes should get ported
  upstream not just to libmv but to SSBA as well.

- Improved the region of convergence for bundle adjustment by
  increasing the number of Levenberg-Marquardt iterations from 50
  to 500. This way, the solver is able to crawl out of the bad
  local minima it gets stuck in when changing from, for example,
  bundling k1 and k2 to just k1 and resetting k2 to 0.

- Add several new region tracker implementations. A region tracker
  is a libmv concept, which refers to tracking a template image
  pattern through frames. The impact to end users is that tracking
  should "just work better". I am reserving a more detailed
  writeup, and maybe a paper, for later.

- Other libmv tweaks, such as detecting that a tracker is headed
  outside of the image bounds.

This includes several changes made directly to the libmv extern
code rather expecting to get those changes through normal libmv
channels, because I, the libmv BDFL, decided it was faster to work
on libmv directly in Blender, then later reverse-port the libmv
changes from Blender back into libmv trunk. The interesting part
is that I added a full Levenberg-Marquardt loop to the region
tracking code, which should lead to a more stable solutions. I
also added a hacky implementation of "Efficient Second-Order
Minimization" for tracking, which works nicely. A more detailed
quantitative evaluation will follow.

Original patch by Keir, cleaned a bit by myself.
2011-11-14 06:41:23 +00:00
Campbell Barton
f086201518 cmake & pep8 tidy up, also some style edits.
remove unneeded collection length function.
2011-11-08 01:32:34 +00:00
Sergey Sharybin
ae06ebcd95 Hopefully compilation with MinGW will work again.
Tested with official MinGW and SCons, not sure how ti check it when using CMake.
2011-11-07 18:14:50 +00:00
Sergey Sharybin
27d42c63d9 Camera tracking integration
===========================

Commiting camera tracking integration gsoc project into trunk.

This commit includes:

- Bundled version of libmv library (with some changes against official repo,
  re-sync with libmv repo a bit later)
- New datatype ID called MovieClip which is optimized to work with movie
  clips (both of movie files and image sequences) and doing camera/motion
  tracking operations.
- New editor called Clip Editor which is currently used for motion/tracking
  stuff only, but which can be easily extended to work with masks too.

  This editor supports:
  * Loading movie files/image sequences
  * Build proxies with different size for loaded movie clip, also supports
    building undistorted proxies to increase speed of playback in
    undistorted mode.
  * Manual lens distortion mode calibration using grid and grease pencil
  * Supervised 2D tracking using two different algorithms KLT and SAD.
  * Basic algorithm for feature detection
  * Camera motion solving. scene orientation

- New constraints to "link" scene objects with solved motions from clip:

  * Follow Track (make object follow 2D motion of track with given name
    or parent object to reconstructed 3D position of track)
  * Camera Solver to make camera moving in the same way as reconstructed camera

This commit NOT includes changes from tomato branch:

- New nodes (they'll be commited as separated patch)
- Automatic image offset guessing for image input node and image editor
  (need to do more tests and gather more feedback)
- Code cleanup in libmv-capi. It's not so critical cleanup, just increasing
  readability and understanadability of code. Better to make this chaneg when
  Keir will finish his current patch.

More details about this project can be found on this page:
    http://wiki.blender.org/index.php/User:Nazg-gul/GSoC-2011

Further development of small features would be done in trunk, bigger/experimental
features would first be implemented in tomato branch.
2011-11-07 12:55:18 +00:00