Commit Graph

229 Commits

Author SHA1 Message Date
Sergey Sharybin
f32615653b Update libmv from own branch
Fixes compilation error on win32 and shall give few
cpu ticks boost by passing vectors by reference rather
than by value.
2013-06-01 10:30:46 +00:00
Sergey Sharybin
719c2f8163 Fixed compilation error after recent keyframes commit. 2013-05-30 12:05:28 +00:00
Sergey Sharybin
c5b4983eca Remove files unused by blender's tracker
They were giving compilation error with msvc2012,
and instead of having workaround in them let's
drop them away form blender (they're not used even).
2013-05-30 12:05:20 +00:00
Sergey Sharybin
cf5e979fb4 Motion tracking: automatic keyframe selection
Implements an automatic keyframe selection algorithm which uses
couple of approaches to find out best keyframes candidates:

- First, slightly modifier Pollefeys's criteria is used, which
  limits correspondence ration from 80% to 100%. This allows to
  reject keyframe candidate early without doing heavy math in
  cases there're not much common features with first keyframe.

- Second step is based on Geometric Robust Information Criteria
  (aka GRIC), which checks whether features motion between
  candidate keyframes is better defined by homography or
  fundamental matrices.

  To be a good keyframe candidate, fundamental matrix need to
  define motion better than homography (in this case F-GRIC will
  be smaller than H-GRIC).

  This two criteria are well described in this paper:
  http://www.cs.ait.ac.th/~mdailey/papers/Tahir-KeyFrame.pdf

- Final step is based on estimating reconstruction error of
  a full-scene solution using candidate keyframes. This part
  is based on the following paper:

  ftp://ftp.tnt.uni-hannover.de/pub/papers/2004/ECCV2004-TTHBAW.pdf

  This step requires reconstruction using candidate keyframes
  and obtaining covariance matrix of 3D points positions.
  Reconstruction was done pretty much straightforward using
  other simple pipeline routines, and for covariance estimation
  pseudo-inverse of Hessian is used, which is in this case
  (J^T * J)+, where + denotes pseudo-inverse.

  Jacobian matrix is estimating using Ceres evaluate API.

  This is also crucial to get rid of possible gauge ambiguity,
  which is in our case made by zero-ing 7 (by gauge freedoms
  number) eigen values in pseudo-inverse.

  There're still room for improving and optimizing the code,
  but we need some point to start with anyway :)

  Thanks to Keir Mierle and Sameer Agarwal who assisted a lot
  to make this feature working.
2013-05-30 09:03:49 +00:00
Campbell Barton
8982439383 quiet warnings from r56825
- using vars before checking they are defined
- /* within comment
2013-05-16 03:55:47 +00:00
Jürgen Herrmann
8a89d27b87 Libmv internal compiler error workaround for VC2012.
Will be reverted as soon as the x64 compiler is fixed.
For now it shouldn't have an impact on tracking performance. My test have shown no significant speed difference to official VC2008 build of 2.67.
2013-05-15 17:00:53 +00:00
Campbell Barton
ff1a587e1c update libmv stub. 2013-05-14 03:06:58 +00:00
Sergey Sharybin
d4c6ac9a60 Cleanup and small improvements to libmv
- Added const modifiers where it makes sense and
  helps keep code safe.
- Reshuffled argument to match <inputs>,<outputs>
  convention on parameters.
- Pass values to ApplyRadialDistortionCameraIntrinsics
  by a constant reference.
  This will save lots of CPU ticks passing relatively
  heavy jet objects to this function when running
  bundle adjustment.
2013-05-13 14:39:06 +00:00
Sergey Sharybin
4e5541a9a6 Headers cleanup. 2013-05-13 09:51:36 +00:00
Sergey Sharybin
91b659d174 Forgot this in one of previous commits
Idea here is to be able to have all files opened in IDE
and not confuse it by the same function implemented in
multiple files.
2013-05-12 22:40:12 +00:00
Sergey Sharybin
4dba8307f2 Compilation fix for recent commit. 2013-05-12 22:30:34 +00:00
Sergey Sharybin
32faec1a53 Remplace bunch of annoying ifdefs in tracking.c with a libmv-capi_stub.cc
Makes code in tracking.cc much easier to understand and modify,
without worring to breck compulation with Libmv disabled.

It is still possible compilation will break due to libmv-capi
changes, but that's not happening so much often.
2013-05-12 22:17:37 +00:00
Sergey Sharybin
5637b0d39b Update bundled version of libmv
- Ensures fix for msvc2012 is applying correct.
- Some code cleanup to match libmv's code style.
- Do not include points which were intersect
  behind the camera to a reconstruction.
- Includes changes needed for keyframe selection.
2013-05-12 17:06:00 +00:00
Sergey Sharybin
beb73831f6 Documentation for functions inside tracking.c
Additional changes:

- Cleaned up sources to reduce mess in some
  big functions.
- Removed unused function from libmv c-api.
- Made functions naming more consistent.
- Use bool for internal stuff in tracking.c.

Shall be no functional changes :)
2013-05-12 16:04:08 +00:00
Brecht Van Lommel
309db8032c Fix libmv build with visual studio 2012.
Patch #35158 by Jurgen Herrmann.
2013-05-11 13:10:39 +00:00
Sergey Sharybin
2cf1f1f4d6 Left extra logging print by accident. 2013-05-10 07:26:55 +00:00
Sergey Sharybin
4648ec3e82 Remove check for zero focal length in BA cost functor
This check is actually redundant, because empty intrinsics
will have focal length of 1.0, which means original comment
about BundleIntrinsics was not truth.

It is possible that external user will send focal length of
zero to be refined, but blender prevents this from happening.
2013-05-10 06:48:47 +00:00
Sergey Sharybin
3a309a8313 Synchronize changes between bunded libmv and own libmv branch. 2013-05-10 06:26:01 +00:00
Sergey Sharybin
ed131e67c4 Add check for points behind camera in euclidan BA cost functor
In cases keyframes are no so good, algebraic two frames construction
could produce result, for which more aggressive Ceres-based BA code
will fall to a solution for which points goes behind the camera,
which is not so nice.

Seems in newer Ceres returning false from cost functor wouldn't
abort solution, but will restrict solver from moving points behind
the camera.

Works fine in own tests, but requires more tests.
2013-05-09 16:38:58 +00:00
Sergey Sharybin
4f5f97254c Reconstructed scene scale ambiguity improvement
Made it so reconstructed scene always scaled in a way
that variance of camera centers is unity.

This solves "issues" when different keyframes will
give the same reprojection error but will give scenes
with different.scale, which could easily have been
considered as a bad keyframe combination.

This change is essential for automatic keyframe
selection algorithm to work reliable for user.
2013-05-09 16:38:43 +00:00
Sergey Sharybin
d2f61b36f9 Revert 56319, misunderstood API
This was completely skipping BA step when point
goes behind the camera.
2013-04-26 18:25:49 +00:00
Sergey Sharybin
126ca24dbf Fix for BA possible move bundles behind the camera
In some cases (was noticed on not good enough keyframe
pair) bundle adjuster could have moved bundles behind
the camera.

This could indeed lead to lower rewprojection error but
this is just pointless thing to do.

Now added check to residuals functor which will return
false to Ceres in cases point moved behind the camera
to prevent such issues.
2013-04-26 15:43:20 +00:00
Sergey Sharybin
d05f5da111 Update Ceres to current upstream version
This brings a fixes for threading issue in BLAS
making BA step more robust (there were some in-detemrinacy
caused by this threading issue).

Also brings some optimizations, which does not directly
affect on blender.
2013-04-22 09:25:37 +00:00
Sergey Sharybin
79ca1bd004 Make it so FATAL messages from libmv are print to the console
even when --debug-libmv is not passed via command line.
2013-04-15 10:02:33 +00:00
Sergey Sharybin
8cca2fec82 Fix for bundle adjusting with motion restricted
This commit bundles new libmv version from own branch
which brings fix for wrong parameter block used for
modal solver parameterization.

Fixes #34985: Crash with Motion tracker (Tripod Motion)
2013-04-14 23:43:00 +00:00
Sergey Sharybin
d2b55d1171 Bundle adjustment improvements
- Get rid of rotation matrix parameterization,
  use angle-axis instead.

  Also Joined rotation and translation into a
  single parameter block.

  This made minimization go significantly faster,
  like 1.3x times in average.

- Fix first camera when bundling. This is to
  address orientation ambiguity.

  Reconstruction result could still vary in
  size, but that's another issue to be addressed
  later.

Additional change:

Split EuclideanBundleCommonIntrinsics into
smaller functions, so it's now a bit easier
to follow.
2013-04-08 17:05:52 +00:00
Sergey Sharybin
8e8823be12 Update ceres to current upstream version
Brings fixes for BLAS.
2013-04-08 17:05:48 +00:00
Sergey Sharybin
f62d2669de Revert change made to bilinear sampler in libmv
This made preview working but that broke internals
of tracking.

Namely, BlurredImageAndDerivativesChannels is giving
much more blurred image because it was assuming pixel
center is an integer position.

Guess other parts of libmv used to suffer because of
this issue.

Now pixel centering happens in blender side, and
libmv assumes integer position is a pixel center.
2013-04-06 14:47:45 +00:00
Sergey Sharybin
898ba93a12 Update libmv to current upstream version
- Solves some strict compilation warning
- Style/code cleanup
2013-04-05 09:23:20 +00:00
Sergey Sharybin
43b61fb8bd Bundle current master of ceres-solver
Thins brings up some speed improvements:

SPARSE_SCHUR is approx 1.3-1.5x times faster
ITERATIVE_SCHUR is approx 1.2x times faster

For blender this means camera solution go a bit
faster now. Would not have affect on tracking
speed.
2013-04-05 09:22:54 +00:00
Sergey Sharybin
729b76eb79 Update ceres to current git version.
Mainly to support msvc2010, bit also includes some minor fixes.
2013-03-21 18:07:03 +00:00
Sergey Sharybin
6dc4ea34e4 Multi-threaded frame calculation for movie clip proxies
This commit implements multi-threaded calculation of frames
when building proxies. Both scaling and undistortion steps
are now threaded.

Frames and proxy resolution are still handled one-by-one,
saving files after every single step. So if HDD is not so
fast, this commit could have not so much benefit.

Internal changes:

- Added IMB_scaleImBuf_threaded which scales given image
  buffer in multiple threads and uses bilinear filtering.

- libmv's camera intrinsics now have SetThreads() method
  which is used to specify how many OpenMP threads to use
  for buffer distortion/undistortion.

  And yeah, this code is using OpenMP for threading.

- Reshuffled a bit libmv-capi calls and added function
  BKE_tracking_distortion_set_threads to specify number
  of threads used by intrinscis.
2013-03-15 11:59:46 +00:00
Sergey Sharybin
f81b594790 Switch from DENSE_NORMAL_CHOLESKY to DENSE_QR
DENSE_QR is better behaved numerically and after recent
changes from Sameer there's no big difference in speed.
2013-03-05 09:38:50 +00:00
Sergey Sharybin
3a0d62f4f4 Update Ceres to 1.5RC3
It brings optimization of DENSE_QR and DENSE_SCHUR solvers.
2013-03-05 09:38:43 +00:00
Sergey Sharybin
9da36d3ab1 Updates to libmv's bundle.sh to make sure libmv upstream
is copying properly to our sources.
2013-03-01 11:52:38 +00:00
Sergey Sharybin
1a7ab1da09 Hrm, gcc became broken after recent commit.
Hope now gcc and 32bit msvc would both work.
2013-03-01 08:56:04 +00:00
Sergey Sharybin
a2027237ed Pass data to modal reprojection cost functor by reference
Seems it solves alignment error reported by 32bit MSVC.
2013-03-01 08:20:13 +00:00
Sergey Sharybin
aada0e4a29 Header cleanup in libmv c-api 2013-02-28 14:25:40 +00:00
Sergey Sharybin
d68e766c8c This lines are also not so much useful for now. 2013-02-28 14:25:33 +00:00
Sergey Sharybin
3e320a67e3 Remove unused rigid registration code
There're some features planned which would
require rigid registration, but this code
would need to be re-done anyway to use new
minimizer and solving some issues with ICP
algorithm there.
2013-02-28 14:24:52 +00:00
Sergey Sharybin
52f34f017d Modal (aka tripod) solver rework
Several major things are done in this commit:

- First of all, logic of modal solver was changed.
  We do not rely on only minimizer to take care of
  guessing rotation for frame, but we're using
  analytical rotation computation for point clouds
  to obtain initial rotation.

  Then this rotation is being refined using Ceres
  minimizer and now instead of minimizing average
  distance between points of point of two clouds,
  minimization of reprojection error of point
  cloud onto frame happens.

  This gives quite a bit of precision improvement.

- Second bigger improvement here is using bundle
  adjustment for a result of first step when we're
  only estimating rotation between neighbor images
  and reprojecting markers.

  This averages error across the image sequence
  avoiding error accumulation. Also, this will
  tweak bundles themselves a bit for better match.

- And last bigger improvement here is support of
  camera intrinsics refirenment.

  This allowed to significantly improve solution
  for real-life footage and results after such
  refining are much more usable than it were before.

Thanks to Keir for the help and code review.
2013-02-28 14:24:42 +00:00
Sergey Sharybin
2a5ed5293d Increase verbosity level for reprojected markers info
This information is useful, but in cases when you, say,
working on a bundler it's annoying to scroll all the
information up.

Now behavior would be:
- running `./blender --debug-libmv` will print all the
  debug messages
- running `./blender --debug-libmv --verbose 0` will
  print only debug messages from solvers, recosntruction
  and so, but will bypass final reprojection bunch of
  messages
- running `./blender --debug-lib,v --verbose 1` will
  include final reprojection messages.
2013-02-28 14:24:25 +00:00
Sergey Sharybin
9bfe99281c Move radial distortion code to own templated function
This shall not lead to any functional changes, just
avoids radial distortion code duplicated in camera
intrinsics and bundling code.

For fancier bundle adjustment supprting different
distortion models this is not actually enough and
would need to make some bigger changes, but this
changes makes code a bit easier to maintain already.
2013-02-28 14:24:17 +00:00
Sergey Sharybin
1d20f2496a Use threaded cost function and jacobian computation
Also made it theraded linear solver, seems it makes
sense for iterative schur with inner iterations
enabled.

Use OpenMO's max therads called from bundler code
to detect how many threads to use. Could be changed
in a way that number of threads is passing in options
from blender side in the future.

Also removed redundant V3D definition from compiler's
flags.
2013-02-26 17:52:10 +00:00
Sergey Sharybin
0ec75be1c6 Camera tracking: support refining radial K1, K2 only
This commits adds extra refirenment entry in the menu which is
"K1, K2" and which will apparently refine only this distortion
coefficients.

This would be useful in cases when you know for sure focal length
(which could be obtained from lens, EXIF and so) but not sure
about how good you manual calibration is.

Be careful tho, there're no internal constraints on this
coefficients so distortion model could just screw up into insane
values.
2013-02-26 11:46:38 +00:00
Sergey Sharybin
05a5cbbda2 Camera tracking: switch euclidean intersection code to use Ceres
Would not expect any significant changes in solver behavior, but
it could be more accurate in some cases.

Switching projective intersection to ceres is marked as a TODO
for now.
2013-02-26 11:46:27 +00:00
Sergey Sharybin
b7af3207cc Bye-bye, SSBA!
With new bundle adjustment based on Ceres we don't need
SSBA library anymore. This also means we don't need ldl
library and libmv is no longer depends on colamd as well.
2013-02-25 20:15:42 +00:00
Keir Mierle
2833994a71 Switch motion tracker bundle adjustment to Ceres.
Patch originally written by me, then finished by Sergey. Big
thanks to Sergey for troopering through and fixing the many issues
with my original (not compilable) patch.

The Ceres implementation uses 2 parameter blocks for each camera
(1 for rotation and 1 for translation), 1 parameter block for
common intrinsics (focal length etc) and 1 parameter block for
each track (e.g. bundle or 3D point).

We turn on some fancy optimizer options to get better performance,
in particular:

  options.preconditioner_type = ceres::SCHUR_JACOBI;
  options.linear_solver_type = ceres::ITERATIVE_SCHUR;
  options.use_inner_iterations = true;
  options.use_nonmonotonic_steps = true;
  options.max_num_iterations = 100;

Special thanks to Sameer Agarwal of Ceres fame for splitting out
the SCHUR_JACOBI preconditioner so that it didn't depend on
CHOLMOD.  Previously we could not use that preconditioner in
Blender because CHOLMOD is too large of a dependency for Blender.

BundleIntrinsicsLogMessage:
- Moved bunch of if(foo) LG << "bar" into this function, to make
  EuclideanBundleCommonIntrinsics a little bit easier to follow.

EuclideanBundle:
- Fix RMSE logging.
2013-02-25 20:00:48 +00:00
Sergey Sharybin
ff6ca7dcd0 code cleanup: remove unused and unsupported functions from libmv-capi
---
svn merge -r52855:52856 ^/branches/soc-2011-tomato
2013-02-25 09:38:59 +00:00
Sergey Sharybin
2df945d8ff Camera tracking: libmv distortion API now also uses camera intrinsics
structure instead of passing all the parameters to every function.

Makes it much easier to tweak distortion model.

---
svn merge -r52854:52855 ^/branches/soc-2011-tomato
2013-02-25 09:36:52 +00:00