commit cfabdfe48df2add3d1f30cf4370efd0b31990ab0 Author: Sergey Sharybin Date: Thu Dec 20 05:46:53 2012 +0600 Assorted fixes for keyframe selection: - Biggest error was in cost functors used for F and H refirement, they were just wrong. - Use natural logarithms, since it's actually makes sense from math papers point of view and error is somewhere else. - Disabled rho for GRIC, for now use non-clamped error for tests. - Made SymmetricEpipolarDistance returning non-squared distance Keyframe selection is currently the only used of this function and it seems using non-squared distance makes much more sense. Also would think to append suffix "Squared" to functions which returns squared distances. - Removed templated version of SymmetricEpipolarDistance, since it's not needed actually. This is actually even worse working than previous implementation, but commit it needed for further review. commit 35d8c57626ad74818f155e6e5960c663ea84e032 Author: Sergey Sharybin Date: Thu Dec 20 03:00:40 2012 +0600 Euclidean resection cost function didn't use correct constructor It was storing a reference to initial rotation passed by value, leading to pointer being pointing to a stack variable, leading to wrong memory access in residuals computing. Apparently was visible in optimized builds only with inline substitution allowed. commit 0798d3162bb49cee7e1c423ceccbca1326ad5650 Author: Sergey Sharybin Date: Thu Dec 20 02:50:52 2012 +0600 Automatic keyframe selection based on Pollefeys's criteria This commit implements automatic keyframe selection algorithm based on Pollefeys's criteria (F-GRIC is smaller than H-GRIC and correspondence ratio is more then 90%). It is implemented as a part of simple pipeline and returns vector of keyframe images for a given Tracks structure. For simple pipeline reconstruction two best keyframes are expected to be selected from all detected candidates. Criteria for this selection could be reprojection error of solution from two candidate keyfames. Unfortunately, it's not fully workable yet, hopefully would be fixed soon. commit e943985552f0598ae122252876f305d72c25c2f9 Author: Sergey Sharybin Date: Thu Dec 6 17:47:11 2012 +0600 Camera Tracking: allow fallback to reprojection resection by user demand This fixes some "regressions" introduced in previous commit which lead to much worse solution in some cases. Now it's possible to bring old behavior back. Perhaps it's more like temporal solution for time being smarter solution is found. But finding such a solution isn't so fast, so let's bring manual control over reprojection usage. But anyway, imo it's now nice to have a structure which could be used to pass different settings to the solver. commit 5a23d01dd531d1e0798298d17ba42a3397effb82 Author: Keir Mierle Date: Thu Sep 20 18:55:44 2012 +0000 Make Euclidean resection "always" succeed. The Euclidean resection code had a magical constant, 1e-3, used to compare the results of solving an equation. This failure detection was well-intended, trying to prevent poor solutions from getting made without notifying the caller. Unfortunately in practice, this threshold is too conservative. Furthermore, it is not clear the threshold should exist at all; the purpose of the Euclidean resection is to come up with the best solution it can; other methods (e.g. reprojection error) should be used to compare whether the method succeeded. This commit changes the Euclidean EPnP code to always succeed, causing the previous fallback to projective resection to never run. In most cases, this will result in better reconstructions. This should, in most cases, fix the dreaded "flipping" problem. commit 57dad861d2a7f9d058c6d8edde1a2d51d7225a51 Author: Keir Mierle Date: Thu Sep 20 02:27:34 2012 +0000 Fix variable naming in the planar tracker. commit e9392fd3b46f5668662935696e7d9afac3390ca4 Author: Keir Mierle Date: Thu Sep 20 02:10:33 2012 +0000 Add smarter tolerance checking in the planar tracker. The planar tracker uses Ceres for the refinement stage. During refinement, Ceres iteratively updates the parameters with the latest best guess. If the change in the parameters falls below a threshold, Ceres will abort successfully ("converged"). For the case of pure translation tracking, the parameters are exactly the two pixel shifts (dx, dy), and measuring the change in these parameters gives a meaningful termination criterion. However, for all the other parameterizations like affine, where the parameterization involves affine parameters that have no physical interpretation, Ceres is left with no way to terminate the solver early. With the existing code, often many iterations are run long after Ceres has found a solution sufficiently accurate for all tracking needs. No one needs tracking with a quadrillionth of a pixel accuracy; that time is wasted. This patch extends the existing iteration callback that is passed in to Ceres to check if the pattern has fallen out of the search window, to also check if the optimizer has made a tiny step. In particular, if the maximum shift of any patch corner between two successful optimizer steps is less than a threshold (currently 0.005 pixels), the track is declared successful and tracking is terminated. This leads to dramatic speed increases in some cases, with little to no loss in track quality. This is especially apparent when tracking patches with affine or perspective motion models. For example, on some tracking cases I tried, the iterations Ceres took went from 50 to 3. commit 36729c19bf90cb767e9adb96ba7dd48a5ace2be1 Author: Keir Mierle Date: Wed Sep 19 22:25:02 2012 +0000 Detect too-small planar tracking patches. The planar tracker did not detect very skinny patches which have effectively zero area and are untrackable. This adds detection and rejection of patterns with zero area. This fixes a crash found by during Mango production. commit 5cf2bae255a5a0f2e36ea0516670782cb88b589d Author: Sergey Sharybin Date: Thu Dec 6 17:33:53 2012 +0600 Real fix for previous commit from Keir. He's comment; Cleanup for when trackers fall out of the search window. Sergey originally left a TODO() here, but his fix is the correct one. I removed the TODO and fixed some comment issues. commit a11533918720e5b43dc1e95895db0eb36c8c06aa Author: Sergey Sharybin Date: Thu Dec 6 17:31:16 2012 +0600 Fix crash when tracking in planar motion model (and maybe some other) It was an Abort() caused by check for solver result not equal to USER_ABORT. In some cases solver returns USER_ABORT due to BoundaryCheckingCallback detects coordinates does not belong to image. Somehow this callback wasn't called in previous version of Ceres and in the same case marker was jumping. Now when the callback is called it seems we could simply return failure of tracking without aborting Blender. Probably this is in fact some issue somewhere else, would double check with Keir about this. commit 4be2306bcc664b259aaf7068b9f32ab60124a509 Author: Sergey Sharybin Date: Thu Dec 6 17:29:39 2012 +0600 Resolved some compilation warnings (missed prototypes) In some cases it was missed include of header file, in some other cases symbol could be static. commit bef729ba5c12683d13584d2a728b8b6506b7ca90 Author: Sergey Sharybin Date: Thu Dec 6 17:27:17 2012 +0600 Code cleanup: silence some -Wnarrowing warnings from C++11 commit add1415d896818367087c784a3013dd8f1bb2095 Author: Sergey Sharybin Date: Thu Dec 6 17:25:18 2012 +0600 Changes to SamplePlanarPatch to support mask input and added output for pattern center. commit daa354c0735b954b0cd7725626e9a3d67416d46b Author: Keir Mierle Date: Sat Jun 9 19:22:39 2012 +0000 Change libmv's bilinear sampling to assume the same pixel conventions as Blender. This fixes the preview widget in Blender, and should make tracking slightly more accurate. commit 99b6222873fbfbe248316316956720376a58f438 Author: Keir Mierle Date: Sat Jun 9 18:58:51 2012 +0000 Add new warp regularization scheme for planar tracking. This adds a new term to the tracking cost function that restricts how much the optimizer can warp the patch (as opposed to merely adjusting the translation). This should reduce the "jumpiness" that is sometimes seen when doing non-"Loc" tracks. It is disabled in this commit; a subsequent commit will add controls to the tracking dialog for this. commit a1c5a70badd11cba0470700bad2eac2b2bd30c86 Author: Keir Mierle Date: Sat Jun 9 06:55:21 2012 +0000 Planar tracker polish. - Fixes the correlation checking code that was broken in the previous commit. The bug was a transpose error. - Fixes a memory leak of the warp functor, found by Sameer. - Various cleanups done at Sameer's suggestion. Thanks to Sameer Agarwal for a code review. commit 2cb784caa854a77cdd43620ab133f26b87ed0d83 Author: Keir Mierle Date: Fri Jun 8 17:42:17 2012 +0000 Make planar tracking much faster. - This makes planar tracking around 2-3x or more faster than before, by rearranging how the sampling is done. Previously, the source patch was sampled repeatedly on every optimizer iteration; this was done for implementation speed, but was wasteful in computation. - This also contains some additions to Ceres to help deailing with mixed numeric / automatic differentation. In particular, there is now a "Chain::Rule" operator that facilitates calling a function that takes Jet arguments, yet does numeric derivatives internally. This is used to mix the numeric differentation of the images with the warp parameters, passed as jets by Ceres to the warp functor. There is also a new "JetOps" object for doing operations on types which may or may not be jets, such as scaling the derivative part only, or extracting the scalar part of a jet. This patche is aimed at Ceres upstream. - A new function for sampling a patch is now part of the track_region.h API; this will get used to make the preview widget properly show what is getting tracked. Currently the preview widget does not handle perspective tracks. Known issues: This patch introduces a bug such that the "Minimum Correlation" flag does not work; if it is enabled, tracking aborts immediately. The workaround for now is to disable the correlation checking, and examine your tracks carefully. A fix will get added shortly. commit 81d028f13738ebe2304287dfce90e91bc782e2cf Author: Keir Mierle Date: Fri May 18 20:04:43 2012 +0000 Remove an unnecessary template<> line in libmv. Convert debug logs to LG. commit 238aaba241ef99995d254aadc974db719da04b96 Author: Keir Mierle Date: Fri May 18 12:05:10 2012 +0000 Support normalization in the tracking prepass The last tracker commit added normalized tracking. This makes tracking patches undergoing uniform illumination change easier. However, the prepass which computes a quick translation-only estimate of the warp did not take this into account. This commit fixes that. This works reasonably well but in some examples the brute initialization fails. I suspect this is due to the warped template estimate in the current frame being too different from the original, so there are multiple peaks in the normalized-SAD correlation function. The solution is to use the previous frame for the brute initialization and the keyframe for refinement, but that requires architecture changes. commit 981ca4f6a679cd9ac3d086eae3cd946ce72ca8a5 Author: Keir Mierle Date: Fri May 18 02:12:47 2012 +0000 Add light-normalized tracking to the planar tracker This commit adds the ability to normalize patterns by their average value while tracking, to make them invariant to global illumination changes. To see this in action, check out the "Lobby" scene from Hollywood VFX. If you track the markers that are shadowed by the actress, previously they would not track. With the scale adaption on, the tracker would shrink the area to compensate for the changed illumination, losing the track. With "Normalize" turned on, the patch is correctly tracked and scale is maintained. A remaining problem is that only the Ceres cost function is updated to handle the normalization. The brute translation search does not take this into account. Perhaps "Prepass" (see below) should get disabled if normalization is enabled until I fix the prepass to normalize as well. There are a few other changes: - Bail out of the sampling loop early if the mask is zero; this saves expensive samples of the image derivatives. - Fix a bug where the mask was ignored when sampling in the cost functor. commit e9384b15fb2a6a5b81346d5758fa136f0911e945 Author: Keir Mierle Date: Thu May 17 23:53:32 2012 +0000 Implement support for affine tracking in the planar tracker; cleanups. commit 021d41eed8b4ce6a4e37786ccd357ed5dc83a13f Author: Keir Mierle Date: Thu May 17 21:26:06 2012 +0000 For the planar tracker, initialize the warp from the four correspondences after brute force translation search. commit 003d1bf6145cfd30938b35f6e10d43708dbf916c Author: Sergey Sharybin Date: Thu Dec 6 16:56:01 2012 +0600 Correction to region tracker options initialization. Based on patch from Keir to Blender: https://svn.blender.org/svnroot/bf-blender/branches/soc-2011-tomato@46743 commit 6af47b218cfdf5219f0ebb3cb95459817cf9abf2 Author: Keir Mierle Date: Thu May 17 02:31:52 2012 +0000 Add new planar tracker features and use the new planar API This commit removes the use of the legacy RegionTracker API from Blender, and replaces it with the new TrackRegion API. This also adds several features to the planar tracker in libmv: - Do a brute-force initialization of tracking similar to "Hybrid" mode in the stable release, but using all floats. This is slower but more accurate. It is still necessary to evaluate if the performance loss is worth it. In particular, this change is necessary to support high bit depth imagery. - Add support for masks over the search window. This is a step towards supporting user-defined tracker masks. The tracker masks will make it easy for users to make a mask for e.g. a ball. - Add Pearson product moment correlation coefficient checking (aka "Correlation" in the UI. This causes tracking failure if the tracked patch is not linearly related to the template. - Add support for warping a few points in addition to the supplied points. This is useful because the tracking code deliberately does not expose the underlying warp representation. Instead, warps are specified in an aparametric way via the correspondences. - Remove the "num_samples_xy" concept and replace it with automatic determination of the number of samples. This makes the API easier for users. - Fix various bugs in the parameterizations. There remains a bug with subpixel precision tracking when in "keyframe" mode; this will get fixed shortly. commit 16a46db104468cec80bd31ca9d5f8bffbe3e003e Author: Keir Mierle Date: Mon May 14 12:15:38 2012 +0000 "Efficient Second-order Minimization" for the planar tracker This implements the "Efficient Second-order Minimization" scheme, as supported by the existing translation tracker. This increases the amount of per-iteration work, but decreases the number of iterations required to converge and also increases the size of the basin of attraction for the optimization. commit 23243b1b1f3e1ab3ef862b47bca06ee876ac2cf4 Author: Keir Mierle Date: Sun May 13 23:08:56 2012 +0000 Add a planar tracking implementation to libmv This adds a new planar tracking implementation to libmv. The tracker is based on Ceres[1], the new nonlinear minimizer that myself and Sameer released from Google as open source. Since the motion model is more involved, the interface is different than the RegionTracker interface used previously in Blender. The ESM tracker, also known as the KLT tracker in the UI, is temporarily changed to use the new Ceres-based planar tracker in translation-only mode. Currently it is a bit slower than ESM and also doesn't have all the bells and whistles implemented. Those will come soon. Longer term, both trackers will remain since Ceres is unlikely to be as fast as ESM for pure translation solving, due to its generality. The next step is to implement a new tracking UI. The current UI assumes a translational motion model; the new one must support arbitrary perspective transforms of the pattern regions. [1] http://code.google.com/p/ceres-solver commit 52be92b53eb4decb1a316690b162196f227cc441 Author: Sergey Sharybin Date: Thu Dec 6 16:06:08 2012 +0600 Initial Ceres integration Currently only put sources to src/third_party/ceres and made sure they're not giving compilation issues. Used Ceres upstream version 1.3.0. Needed to make some modifications to it's CMakeLists.txt also to glog and fglags. They're described in README.libmv of this libraries. Basically: - Added -fPIC to glog/gflags, so shared ceres library could be linked statically against this libraries. - Tweaked Ceres's build rules to use needed libraries from libmv's third_party folder. commit b13f9d13122e091cb85855c2094386ccdef6e5a4 Author: Sergey Sharybin Date: Wed Dec 5 19:05:34 2012 +0600 Update Eigen to version 3.1.2 Mainly because of lots of warnings generating by gcc-4.7 which are resolved in newer eigen version. commit 1f0dd94e8e37d3fe2df89282ec16a6a685fdde0b Author: Sergey Sharybin Date: Fri May 25 16:36:44 2012 +0600 - Added avutil to qt-tracker linking when building with FFmpeg support. On some platforms it seems to be required - Synchronized QT Creator project for qt-tracker with changes in sources, so no it might be compiled from QT Creator. commit b813dbe3f46bbbc7e73ac791d4665622e4fc7ba5 Author: Sergey Sharybin Date: Wed May 9 19:01:10 2012 +0600 Modal solver: Detect rigid transformation between initial frame and current instead of detecting it between two neighbour frames. This prevents accumulation of error and seems to be working better in footages i've tested. commit 9254621c76daaf239ec1f535e197ca792eea97b6 Author: Sergey Sharybin Date: Wed May 9 18:57:00 2012 +0600 Backport changes made by Keir in Blender: - Enhance logging in libmv's trackers. - Cleanups in brute_region_tracker.cc. commit d9c56b9d3c63f886d83129ca0ebed1e76d9c93d7 Author: Sergey Sharybin Date: Fri Apr 27 16:20:41 2012 +0600 Fixes for MinGW64 support by Caleb Joseph with slight modifications by Antony Riakiotakis - Functions snprintf and sincos shouldn't be redefined for MinGW64 - Type pid_t shouldn't be re-defined for MinGW64 commit e1902b6938676011607ac99986b8b140bdbf090e Author: Sergey Sharybin Date: Fri Apr 27 16:04:19 2012 +0600 Fixes for Qt calibration tool - Passing directory with images via command line argument now isn't required -- it there's no such directory specified standard open dialog might be used for this (before application used to abort due to accessing to non-existing list element). - Conversion of source images to grayscale now happens correct. It was needed to build grayscale palette for 8bit indexed buffer. commit 05f1a0a78ad8ff6646d1e8da97e6f7575b891536 Author: Sergey Sharybin Date: Sat Apr 14 17:21:29 2012 +0600 Make QtTracker compilable again porting it to recent API change and code cleanup: - It was using SAD tracker with own API, now it's using standard RegionTracker API which should make it easier to switch between different trackers. - Restored LaplaceFilter from old SAD module which convolves images with the discrete laplacian operator. commit a44312a7beb2963b8e3bf8015c516d2eff40cc3d Author: Sergey Sharybin Date: Thu Apr 12 13:56:02 2012 +0600 Added solver for modal camera motion, currently supports only tripod solving This solver is intended to deal with such camera motions as tripod and panning, where it's impossible to reconstruct exact position of markers in 3d view. It projects markers onto sphere and uses rigid registration of rotation to find rotation angles which makes bundles from previous and current frame be as closest as it's possible. commit fa3842e472e3b9c789e47bf6d8f592aa40a84f16 Author: Sergey Sharybin Date: Thu Apr 12 12:32:48 2012 +0600 implementation of some basic algorithms for point cloud orientation: - Implementation of rigid registration algorithm which searches transformation form one point cloud to another assuming that points in this clouds are already paired (points with the same index in different clouds belongs to the same pair) which minimizes average distance between points in pairs. Algorithm uses Levenberg-Marquardt solver to find such transformation. Supports registration of rotation-scale-transform (which is probably most common usage) and rotation only (which might be useful for basic modal tripod solver). - Implementation of Iterative-Point-Clouds algorithm which searches transformation from one arbitrary point cloud to another making points as closest to each other as possible. This algorithm doesn't require points be initially paired, but for good result clouds should have rough initial orientation. If they're arbitrary oriented from the very beginning, algorithm might fail producing good resold. Iteration is based on building pairs of closest to each other points and registering rigid transformation between them which incrementally constructs final result. TODO: building pairs might be speedup a lot using data structures like AABB trees, K-D trees or so. commit 9618d9a1d48bb3c28da605d9027f57a74f462785 Author: Sergey Sharybin Date: Wed Apr 11 14:17:14 2012 +0600 Added configuration file for glog to compile smooth on Hurd platform. Patch by Pino Toscano , thanks! commit 63b2bd20237c8599fa73ce42556e4fb99b9f7cca Author: Sergey Sharybin Date: Thu Mar 22 17:03:34 2012 +0600 Trackers refactoring: - Deduplicate pattern sampling used in esm and lmicklt trackers and move SamplePattern to image/sample.h - Move computation of Pearson product-moment correlation into own function in new file image/correlation.h so all trackers can use it to check final correlation. - Remove SAD tracker. It's almost the same as brute tracker, with only two differences: 1. It does brute search of affine transformation which in some cases helps to track rotating features 2. It didn't use common tracker api which probably gave some speed advantage, but lead to a real headache to use it together with other trackers leading to duplicated code in 3d-party software. commit 9fe49c32e990f28c83f2bbb1d18057aed8879af7 Author: Sergey Sharybin Date: Mon Mar 12 09:36:15 2012 +0600 Code cleanup: convert line endings to Unix style (native on my platform) so it wouldn't confuse other versioning systems used for project where libmv is bundled to, Also change mode to +x for glog's windows-related script. commit fe74ae2b53769389b0ed9d7e604c8e60be81077d Author: Sergey I. Sharybin Date: Sun Mar 11 20:34:15 2012 +0600 Replace "third_party/glog/src/glog/logging.h" with It was needed because of how build systems is setup in Blender but think this will be helpful change for other applications too because it makes it's easier to move libraries around and even use libraries installed on the operation system. commit 37fc726701479f2d321d6af878fa93f3176278d5 Author: Sergey I. Sharybin Date: Sun Mar 11 19:27:41 2012 +0600 Upgrade gflags and glog libraries - stage 2 (final) Changes to upstream code which are needed to make libmv compile smooth on all platforms * Replace with "third_party/gflags/gflags/gflags.h" which is easier to setup things in libmv and also helps with setting up building libmv into external applications. * Replace "glog/logging.h" and "glog/logging.h" with and which is needed on Windows platform because otherwise files like logging.cc will be using relative path which points to headers used by linux instead of headers need to be used on Windows. * Replace _asm int 3 with __debugbreak(). Such assembler code is obsolete and doesn't work with 64bit versions of MSVC compilers. * Do not use stacktrace for MinGW and FreeBSD because it leads into issues accessing some specific data on this platforms. * Define HAVE_LIB_GFLAGS for Windows builds. * Do not define __declspec(dllimport) for MinGW platforms. * Setup proper includes and datatypes for int32, uint32, int64 and uint64 for MinGW * Do not define va_copy for MinGW platforms (it's already defined there). * Patch localtime_r to be working fine with MinGW, disable strerror_r for MinGW because of lack of needed functions. commit 8ed07abfa49d1e0511752021c972e0715e5a1383 Author: Sergey I. Sharybin Date: Sun Mar 11 19:06:33 2012 +0600 Upgrade gflags and glog libraries - stage 1 This commit copies sources from latest original release of gflags and glog over currently bundled versions of this libraries without any modifications. This revision can't b compiled, all needed changes to make new libraries working fine will be done with next commit to make it clear which changes were necessary for easier bundling further newer version and extract patches and put them to gflags/glog upstream repo. Such upgrade of libraries is needed to make it able to compile libmv with clang compilers. Currently used versions: - gflags is version 2.0 - glog is version 0.3.2 commit 75b9af405964ff2c7d3f0a44500e27e63b37c91b Author: Sergey Sharybin Date: Fri Feb 17 23:29:11 2012 +0600 _USE_MATH_DEFINES is needed to define constants like M_E when building with msvc Occasionally was removed, but now added comment about this so hopefully it wouldn't removed again. commit f85b1232a9b929f69443b5eed6e7a39908cd6551 Author: Sergey Sharybin Date: Fri Feb 17 21:34:40 2012 +0600 Picky edit: corrected mode for ssba readme file. commit f8c2b223f01551fd81a85f6d5221646165147035 Author: Sergey Sharybin Date: Fri Feb 17 21:32:05 2012 +0600 Picky edits: corrected EOL commit 3f2a4205ec5adadcdfa306b161c705c868a7be93 Author: Sergey Sharybin Date: Fri Feb 17 21:30:07 2012 +0600 Fixed incorrect access to ucontext on linux. Caused by incorrect merge conflict resolve. commit d360a21a5aa125cf9e83dd26b302508688ff7007 Author: Sergey Sharybin Date: Fri Feb 17 20:54:13 2012 +0600 More Windows -> Unix EOL conversions commit 18aeda58bec9556140ba617724e31ada6f5b67c0 Author: Sergey Sharybin Date: Fri Feb 17 20:15:42 2012 +0600 Looks like this debug output was removed accidentally. commit 189dc0cacdee3c1eab68c43263ecb038ed244c09 Author: Sergey Sharybin Date: Fri Feb 17 20:11:56 2012 +0600 Made V3D verbose again by default commit 8b3422d3eec5e450d76243886bf07fb0a3e83a81 Author: Sergey Sharybin Date: Fri Feb 17 20:08:01 2012 +0600 SAD tracker now can deal with pattern size any size, Very quick implementation came from Blender before Hybrid tracker was added. Better to be replaced with brute tracker.