There're some features planned which would
require rigid registration, but this code
would need to be re-done anyway to use new
minimizer and solving some issues with ICP
algorithm there.
Several major things are done in this commit:
- First of all, logic of modal solver was changed.
We do not rely on only minimizer to take care of
guessing rotation for frame, but we're using
analytical rotation computation for point clouds
to obtain initial rotation.
Then this rotation is being refined using Ceres
minimizer and now instead of minimizing average
distance between points of point of two clouds,
minimization of reprojection error of point
cloud onto frame happens.
This gives quite a bit of precision improvement.
- Second bigger improvement here is using bundle
adjustment for a result of first step when we're
only estimating rotation between neighbor images
and reprojecting markers.
This averages error across the image sequence
avoiding error accumulation. Also, this will
tweak bundles themselves a bit for better match.
- And last bigger improvement here is support of
camera intrinsics refirenment.
This allowed to significantly improve solution
for real-life footage and results after such
refining are much more usable than it were before.
Thanks to Keir for the help and code review.
This information is useful, but in cases when you, say,
working on a bundler it's annoying to scroll all the
information up.
Now behavior would be:
- running `./blender --debug-libmv` will print all the
debug messages
- running `./blender --debug-libmv --verbose 0` will
print only debug messages from solvers, recosntruction
and so, but will bypass final reprojection bunch of
messages
- running `./blender --debug-lib,v --verbose 1` will
include final reprojection messages.
This shall not lead to any functional changes, just
avoids radial distortion code duplicated in camera
intrinsics and bundling code.
For fancier bundle adjustment supprting different
distortion models this is not actually enough and
would need to make some bigger changes, but this
changes makes code a bit easier to maintain already.
Also made it theraded linear solver, seems it makes
sense for iterative schur with inner iterations
enabled.
Use OpenMO's max therads called from bundler code
to detect how many threads to use. Could be changed
in a way that number of threads is passing in options
from blender side in the future.
Also removed redundant V3D definition from compiler's
flags.
This commits adds extra refirenment entry in the menu which is
"K1, K2" and which will apparently refine only this distortion
coefficients.
This would be useful in cases when you know for sure focal length
(which could be obtained from lens, EXIF and so) but not sure
about how good you manual calibration is.
Be careful tho, there're no internal constraints on this
coefficients so distortion model could just screw up into insane
values.
Would not expect any significant changes in solver behavior, but
it could be more accurate in some cases.
Switching projective intersection to ceres is marked as a TODO
for now.
With new bundle adjustment based on Ceres we don't need
SSBA library anymore. This also means we don't need ldl
library and libmv is no longer depends on colamd as well.
Patch originally written by me, then finished by Sergey. Big
thanks to Sergey for troopering through and fixing the many issues
with my original (not compilable) patch.
The Ceres implementation uses 2 parameter blocks for each camera
(1 for rotation and 1 for translation), 1 parameter block for
common intrinsics (focal length etc) and 1 parameter block for
each track (e.g. bundle or 3D point).
We turn on some fancy optimizer options to get better performance,
in particular:
options.preconditioner_type = ceres::SCHUR_JACOBI;
options.linear_solver_type = ceres::ITERATIVE_SCHUR;
options.use_inner_iterations = true;
options.use_nonmonotonic_steps = true;
options.max_num_iterations = 100;
Special thanks to Sameer Agarwal of Ceres fame for splitting out
the SCHUR_JACOBI preconditioner so that it didn't depend on
CHOLMOD. Previously we could not use that preconditioner in
Blender because CHOLMOD is too large of a dependency for Blender.
BundleIntrinsicsLogMessage:
- Moved bunch of if(foo) LG << "bar" into this function, to make
EuclideanBundleCommonIntrinsics a little bit easier to follow.
EuclideanBundle:
- Fix RMSE logging.
structure instead of passing all the parameters to every function.
Makes it much easier to tweak distortion model.
---
svn merge -r52854:52855 ^/branches/soc-2011-tomato
- Moved keyframes and refirement flags into reconstruction options structure
- Moved distortion coefficients and other camera intrinsics into own structure
- Cleaned up reconstruction functions in libmv c-api
---
svn merge -r52853:52854 ^/branches/soc-2011-tomato
This is helpful because it brings CHOLMOB-free ITERATIVE_SCHUR and
SCHUR_JACOBI which is really nice for new upcoming bundle adjustment.
If also includes all the local fixes we made locally.
There're lots of other improvements/fixed which are not currently
would be so much visible in Blender, but which opens doors for some
great improvements in future.
SSBA seemed to be working OK last time i've checked it
with MSVC and optimization enabled.
Also, we'll likely replace it with own BA soon, which
works fine with MSVC anyway.
find ldl symbols because order of libraries seems to be critical
for gcc linker.
A bit stupid, but that's how linker works..
Both CMake and SCons shall work fine on linux now.
Root of the issue goes to SSBA library which didn't work
properly when using optimization in MSVC. It was worked
around by disabling optimization for libmv, which is in
fact shame and shouldn't have been done.
It seems after some changes optimization does not affect
on SSBA code, but enabling optimization could be risky so
close to release.
For now solved by splitting SSBA to separate CMake/SCons
library, disabling optimization only for this particular
library and enabling optimization for rest of libmv.
Tested on all files which used to fail with optimization
enabled in SSBA and all of them works the same as before.
Tracking speed is significantly higher now.
After release we'll enable optimization for SSBA as well,
so there'll be no crappy build setup. Later we'll replace
old SSBA library with new BA code based on Ceres.
Bundle script would be broken for until then, so better
not to use it.
libmv still requires optimization switched off because
of some incompatibility of SSBA and MSVC optimizer which
makes bundle adjustment work just wrong.
This shall not be an issue for Ceres and no need to
disable optimization for extern_ceres
- Moved keyframes and refirement flags into reconstruction options structure
- Moved distortion coefficients and other camera intrinsics into own structure
- Cleaned up reconstruction functions in libmv c-api
This fixes some "regressions" introduced in rev50781 which lead to much
worse solution in some cases. Now it's possible to bring old behavior back.
Perhaps it's more like temporal solution for time being smarter solution is
found. But finding such a solution isn't so fast, so let's bring manual
control over reprojection usage.
But anyway, imo it's now nice to have a structure which could be used to
pass different settings to the solver.
- move object_iterators.c --> view3d_iterators. (ED_object.h had to include ED_view3d.h which isn't so nice)
- move projection functions from view3d_view.c --> view3d_project.c (view3d_view was becoming a mishmash of utility functions and operators).
- some some cmake includes as system-includes.
The Euclidean resection code had a magical constant, 1e-3, used to
compare the results of solving an equation. This failure detection
was well-intended, trying to prevent poor solutions from getting
made without notifying the caller. Unfortunately in practice, this
threshold is too conservative. Furthermore, it is not clear the
threshold should exist at all; the purpose of the Euclidean
resection is to come up with the best solution it can; other
methods (e.g. reprojection error) should be used to compare
whether the method succeeded.
This commit changes the Euclidean EPnP code to always succeed,
causing the previous fallback to projective resection to never
run. In most cases, this will result in better reconstructions.
This should, in most cases, fix the dreaded "flipping" problem.
The planar tracker uses Ceres for the refinement stage. During
refinement, Ceres iteratively updates the parameters with the
latest best guess. If the change in the parameters falls below a
threshold, Ceres will abort successfully ("converged").
For the case of pure translation tracking, the parameters are
exactly the two pixel shifts (dx, dy), and measuring the change in
these parameters gives a meaningful termination criterion.
However, for all the other parameterizations like affine, where
the parameterization involves affine parameters that have no
physical interpretation, Ceres is left with no way to terminate
the solver early. With the existing code, often many iterations
are run long after Ceres has found a solution sufficiently
accurate for all tracking needs. No one needs tracking with
a quadrillionth of a pixel accuracy; that time is wasted.
This patch extends the existing iteration callback that is passed
in to Ceres to check if the pattern has fallen out of the search
window, to also check if the optimizer has made a tiny step. In
particular, if the maximum shift of any patch corner between two
successful optimizer steps is less than a threshold (currently
0.005 pixels), the track is declared successful and tracking
is terminated.
This leads to dramatic speed increases in some cases, with little
to no loss in track quality. This is especially apparent when
tracking patches with affine or perspective motion models. For
example, on some tracking cases I tried, the iterations Ceres took
went from 50 to 3.
The planar tracker did not detect very skinny patches which have
effectively zero area and are untrackable. This adds detection and
rejection of patterns with zero area. This fixes a crash found by
during Mango production.
It was an Abort() caused by check for solver result not equal to USER_ABORT.
In some cases solver returns USER_ABORT due to BoundaryCheckingCallback
detects coordinates does not belong to image.
Somehow this callback wasn't called in previous version of Ceres and
in the same case marker was jumping. Now when the callback is called
it seems we could simply return failure of tracking without aborting
Blender.
Probably this is in fact some issue somewhere else, would double
check with Keir about this.
