commit 03cbc88ce7f51aa26ba503acea2e984bcb78873c Author: Sergey Sharybin Date: Mon Apr 15 05:35:33 2013 +0600 Fix for bundle adjusting with motion restricted Was a bug introduced in previous commit, which was trying to set parameterization for non-existing camera->t parameter block. Replaced with subset parameterization. Also added basic synthetic unit test for modal solver. commit c78a68f980e778d40ce836fa1d7471cb7264d4a0 Author: Sergey Sharybin Date: Mon Apr 8 23:33:20 2013 +0600 Bundle adjustment improvements - Get rid of rotation matrix parameterization, use angle-axis instead. Also Joined rotation and translation into a single parameter block. This made minimization go significantly faster, like 1.3x times in average. - Fix first camera when bundling. This is to address orientation ambiguity. Reconstruction result could still vary in size, but that's another issue to be addressed later. Additional change: Split EuclideanBundleCommonIntrinsics into smaller functions, so it's now a bit easier to follow. commit 28c7566629c2cf5b03a787c9509856e87472eb2f Author: Sergey Sharybin Date: Mon Apr 8 23:31:57 2013 +0600 Update Ceres to current HEAD Brings up some noticeable speed improvements. In particular the automatic differentiation and bundle adjustment solvers. commit efde9faa21e70b031d3cbcb2dcdcd38e597bf56e Author: Sergey Sharybin Date: Mon Apr 8 02:21:26 2013 +0600 Corrected path to gflags Currently tools/track.cc is not used, but let's keep things a bit more up-to-date :) commit f8b5ea196fb00ab07d577e9738a60cdd1de16509 Author: Sergey Sharybin Date: Mon Apr 8 02:17:16 2013 +0600 Re-enable tests for multiview and image For as long code is in repo and used by some tools better to have it covered by tests. Some of them are failing tho, but that's completely different story to be addressed later. commit d2a7ee60a5845738f76b88bfc373eefc2cc8501a Author: Sergey Sharybin Date: Mon Apr 8 02:10:07 2013 +0600 Do not modify cache's CMAKE_CXX_FLAGS_RELEASE when configuring Ceres Otherwise you'll have infinite appending of Ceres-specific flags on every saving of any CmakeLists.txt. commit 45edb507bf46194dd55b7fc46a7d90ee3853834d Author: Sergey Sharybin Date: Sun Apr 7 21:53:23 2013 +0600 Fixed compilation with BUILD_TOOLS enabled This commit mainly reverts parts of following commits: 0eebc21db831211738acc938566bbc29d68d45db d8109b7a4fede1660e0dbd73735f1a9e3fd79eec e59595806c045916ab4ef15ef7047c1a728b2da9 2d6cd58ee1cd7c5073980f358c71b2898ad16b4c They declared lots of stuff deprecated, but in fact it's not deprecated just a bit different usage pipeline. Anyway, deprecation shall not happen spontaneously as a part of other changes. And for sure shall not break anything. commit 7a9c83d3ccaa2f0015f88b9156d7662c46244b4a Author: Sergey Sharybin Date: Sat Apr 6 20:49:05 2013 +0600 Revert "Change libmv's bilinear sampling to assume the same" Revert changes to bilinear sampler which were originally aimed to match blender's pixel center (where integer coord is a left-bottom corner, x.5 coords are centers. The reason of revert is changing this assumption in only sampler didn't work well and lead to wrong results of BlurredImageAndDerivativesChannels for example. Discovered when was doing unit-tests for brute region tracker. This reverts commit daa354c0735b954b0cd7725626e9a3d67416d46b. commit 15f3bb00340933ce753a1a55e9cde9383352e259 Author: Sergey Sharybin Date: Sat Apr 6 18:37:37 2013 +0600 Added basic test for brute region tracker It is failing at this moment and this is caused because of how SampleLinear works - seems it's assumption about pixel center is not correct for internal sampling. commit 1146602972c07e99a9e4ab37d35ac83aec490e60 Author: Sergey Sharybin Date: Sat Apr 6 16:54:08 2013 +0600 Tweak to KLT region tracker test KLT is usually used to track relatively small motions, and in this case motion almost equals to half window size. This confuses math and leads to not so much expected result. Further, not actually sure this is nice idea to use KLT in such synthetic case. commit 1e22cbcac480863b6b5abc5c85f23dc70748933a Author: Sergey Sharybin Date: Sat Apr 6 16:40:59 2013 +0600 Pyramid tracker unit test fix Issue was caused by trackers modifying guessed point location even in case of failure. So made id so both level 0 and level 3 of pyramid are starting from the same initial guessed location. Modifying locations in case tracker returns false is not actually a bug - someone could still want to use that location. False in this case means more like "returned location is not so much accurate". commit 3ad5e0efa071f202ee7c2034d70dd97aa62b13aa Author: Sergey Sharybin Date: Sat Apr 6 00:38:40 2013 +0600 Intersect unit test fix EuclideanIntersect is not aware of camera calibration matrix yet and always assumes it to be an identity. So using non-identity matrix to construct sample case leads to wrong projection results. For now made it so test case uses identity matrix for calibration. Also fixed variable shadowing which lead to wrong markers positions (were either zero or undefined). commit 3a153c2b65f38653a36c36975018f68d42d60670 Author: Sergey Sharybin Date: Sat Apr 6 00:12:12 2013 +0600 Camera intrinsics unit tests fixes - Existing test ApplyIsInvertibleSimple was not doing right thing - distortion model used there was ininvertible. Tweaked parameters in a way model is invertible now. - Added some own tests which tests; * Principal point always maps from pixel space to zero normalized position. * Some basic tests to check whether individual apply/invert works correct. commit e3b2bccba6145290738a6677c14f7369ec7a38cd Author: Sergey Sharybin Date: Thu Apr 4 02:59:58 2013 +0600 Suppress strict compiler warnings in glags/glog libraries commit 5fca459adcf0a3419fa9cd8d983dc2c952d02647 Author: Sergey Sharybin Date: Thu Apr 4 01:20:18 2013 +0600 Lint cleanup, mostly white space and line width. Also moved own includes to the top of files. Should be no functional changes :) commit 9a9dd458a622928b91dbd3c79900577923283838 Author: Sergey Sharybin Date: Fri Mar 29 00:20:29 2013 +0600 Fix for TransformTracks in uncalibrated pipeline Transformation matrix was completely ignored by TransformTracks() and final marker coordinate exactly matched it's source coordinates. Seems to be just a typo in vector usage: need to use "b" (which is transformed one) instead of "a" when converting projective coordinate to 2D space. commit d35766cc9901609e32f4d80faba715695bea3c40 Author: Sergey Sharybin Date: Fri Mar 29 00:19:11 2013 +0600 Revert part of e2eb58c4230f94ef0c72fb4005e5434088d52e80 That commit included one change which shall have been go as separate commit with more detailed description. commit e8d71b4e96fd78eb60773b6557d66da672e65753 Author: Sergey Sharybin Date: Wed Mar 27 20:37:05 2013 +0600 Silenced more warnings - Added includes of own header to fast implementation files. - Camera intrinsics wouldn't complain about unknown pragma when building without OpenMP support. TODO: Make it a CMake option to build libmv with OpenMP support. Currently multi-threaded intrinsics only available when using custom CMake rules for bundled libmv version (as it's done in Blender). commit ad442812654f270dc088394410fda1b81b8dc450 Author: Sergey Sharybin Date: Wed Mar 27 20:18:51 2013 +0600 Multithreaded camera intrinsics Implemented multithreaded buffer (un)distortion for camera intrinsics using OpenMP. By default, (un)distortion is single-threaded, but it is possible to as CameraIntrinsics to use more threads by calling SetThreads method. commit c88b4881096174a16a9f9e6fc2c9dcad3e255b25 Author: Sergey Sharybin Date: Wed Mar 27 18:45:09 2013 +0600 Movie functions implementation from panography header into own CC implementation file. Before this all panography functions were declared as static, which is not so much useful from re-useability point of view. commit 2d2faf9104bc035722cff6775e1b8e7c93143aba Author: Sergey Sharybin Date: Wed Mar 27 18:37:36 2013 +0600 Build shared Ceres library only if BUILD_SHARED_LIBS is enabled commit daa3ddd3260ccaf2bf9c72eadb89213d91e549ec Author: Sergey Sharybin Date: Wed Mar 27 18:21:52 2013 +0600 Update Ceres to upstream version 1.5.0 commit cf5dc678878345ea3f221ce50cb2b9e539c2ab38 Author: Sergey Sharybin Date: Wed Mar 27 15:06:24 2013 +0600 Code cleanup: removed more deprecated FFmpeg API usage This time in qt-tracker application. commit e2eb58c4230f94ef0c72fb4005e5434088d52e80 Author: Sergey Sharybin Date: Tue Mar 26 17:19:51 2013 +0600 Code cleanup: silent unused variables warnings commit af89bb24667e39b7e655173ea807fdcfbeef4422 Author: Sergey Sharybin Date: Tue Mar 26 16:54:14 2013 +0600 Code cleanup: no need to declare empty body for ProgressUpdateCallback:invoke Make force this method to be overridden by derivative classes. Also removed currently unneeded callbacks.cc. commit 0441d4ee06fad0219256a5704f931eec916a3868 Author: Sergey Sharybin Date: Tue Mar 26 16:37:27 2013 +0600 Code cleanup: silent type narrowing in qt-tracker commit cd4b61c976448d0fdedefb3ed4b21d70e078f94b Author: Sergey Sharybin Date: Tue Mar 26 16:26:39 2013 +0600 Changes to unit testing - Move ceres test binaries to ${LIBMV_TESTS_OUTPUT_DIR}/ceres, so they don't mess with libmv's application binaries and tests. - Removed ceres_ prefix from ceres unit tests, only use this prefix for targets (targets need to be unique name). - Added unit tests data for ceres, otherwise system_test fails. - Restored "test" makefile target. commit cf704ada08acc8b26167e7bfb3e1e88fd278de23 Author: Sergey Sharybin Date: Tue Mar 26 15:01:15 2013 +0600 Code cleanup: use rw-rw-r-- mode for source files commit 64b31e3e43acb52aaf6f591b9d1c2449bf6ef3bd Author: Sergey Sharybin Date: Tue Mar 26 14:57:46 2013 +0600 Code cleanup: don't use deprecated FFmpeg API functions commit 2a3676499548ad5dba5a5c5eadf3bb71e640b612 Author: Sergey Sharybin Date: Tue Mar 5 17:40:52 2013 +0600 Switch from DENSE_NORMAL_CHOLESKY to DENSE_QR DENSE_QR is better behaved numerically and after recent changes from Sameer there's no big difference in speed. commit bcb920df02133da5b7e55fbc74edb9222004eecc Author: Sergey Sharybin Date: Tue Mar 5 17:15:43 2013 +0600 Update Ceres to 1.5RC3 It brings optimization of DENSE_QR and DENSE_SCHUR solvers. commit 473996468a4e67e7c860169181a4ff31ce9b8c80 Author: Sergey Sharybin Date: Fri Mar 1 17:44:54 2013 +0600 Fixed incorrect order of arguments passing to EXPECT_EQ in keyframe selection tests. commit d38ebb74693fdf5b8f0fecf62a3d8c9c53b0b84a Author: Sergey Sharybin Date: Fri Mar 1 17:40:38 2013 +0600 Modal (aka tripod) solver rework Several major things are done in this commit: - First of all, logic of modal solver was changed. We do not rely on only minimizer to take care of guessing rotation for frame, but we're using analytical rotation computation for point clouds to obtain initial rotation. Then this rotation is being refined using Ceres minimizer and now instead of minimizing average distance between points of point of two clouds, minimization of reprojection error of point cloud onto frame happens. This gives quite a bit of precision improvement. - Second bigger improvement here is using bundle adjustment for a result of first step when we're only estimating rotation between neighbor images and reprojecting markers. This averages error across the image sequence avoiding error accumulation. Also, this will tweak bundles themselves a bit for better match. - And last bigger improvement here is support of camera intrinsics refirenment. This allowed to significantly improve solution for real-life footage and results after such refining are much more usable than it were before. Thanks to Keir for the help and code review! commit 5d6c2e7a27bdd1a1b23bf289d70a9b8f62514c9a Author: Sergey Sharybin Date: Fri Mar 1 17:37:35 2013 +0600 Increase verbosity level for reprojected markers info This information is useful, but in cases when you, say, working on a bundler it's annoying to scroll all the information up. commit ac252bb1250b3028b9c94736b644e7ab4e7b14b8 Author: Sergey Sharybin Date: Fri Mar 1 17:36:19 2013 +0600 Move radial distortion code to own templated function This shall not lead to any functional changes, just avoids radial distortion code duplicated in camera intrinsics and bundling code. For fancier bundle adjustment support of different distortion models this is not actually enough and would need to make some bigger changes, but this changes makes code a bit easier to maintain already. commit c253b794612dd529e1d3a9bd7a7c41c32c9a9abb Author: Sergey Sharybin Date: Fri Mar 1 17:33:27 2013 +0600 Use threaded cost function, jacobian and linear solver computation, so bundling is as fast as it could be with current parameter block structure. commit 931fe37a10212b91b525d4f6eb753990a338b471 Author: Sergey Sharybin Date: Fri Mar 1 17:29:21 2013 +0600 Fixed comment for euclidean bundling, which is now supports raidal bundling independently from other intrinsics. commit 217d8e6edc3de1a853fb84275d2d2dd898e7529c Author: Sergey Sharybin Date: Tue Feb 26 18:19:01 2013 +0600 Allow K1,K2 refirement combination It is now possible to refine only radial distortion with new Ceres based bundler and this new combination is already used in Blender. commit d8850addc944d400f7a9c358396c437d9e4acc70 Author: Sergey Sharybin Date: Tue Feb 26 18:17:09 2013 +0600 Switch euclidean intersection code to use Ceres Would not expect any significant changes in solver behavior, but it could be more accurate in some cases. Switching projective intersection to ceres is marked as a TODO for now. commit 6990b7946ec96b3cb2dcfc8a1beaaba9538b0802 Author: Keir Mierle Date: Mon Feb 25 20:00:48 2013 +0000 Switch motion tracker bundle adjustment to Ceres. Patch originally written by me, then finished by Sergey. Big thanks to Sergey for troopering through and fixing the many issues with my original (not compilable) patch. The Ceres implementation uses 2 parameter blocks for each camera (1 for rotation and 1 for translation), 1 parameter block for common intrinsics (focal length etc) and 1 parameter block for each track (e.g. bundle or 3D point). We turn on some fancy optimizer options to get better performance, in particular: options.preconditioner_type = ceres::SCHUR_JACOBI; options.linear_solver_type = ceres::ITERATIVE_SCHUR; options.use_inner_iterations = true; options.use_nonmonotonic_steps = true; options.max_num_iterations = 100; Special thanks to Sameer Agarwal of Ceres fame for splitting out the SCHUR_JACOBI preconditioner so that it didn't depend on CHOLMOD. Previously we could not use that preconditioner in Blender because CHOLMOD is too large of a dependency for Blender. BundleIntrinsicsLogMessage: - Moved bunch of if(foo) LG << "bar" into this function, to make EuclideanBundleCommonIntrinsics a little bit easier to follow. EuclideanBundle: - Fix RMSE logging. commit 1696342954614b54133780d74d6ee0fbcbe224f0 Author: Sergey Sharybin Date: Tue Feb 26 18:10:33 2013 +0600 Upgrade ceres to latest upstream version This is needed because of some features of new Ceres for further development. commit 575336f794841ada90aacd783285014081b8318c Author: Sergey Sharybin Date: Mon Jan 7 15:58:40 2013 +0600 Fixed for keyframe selection - Calculate residuals for GRIC in pixel space rather than in normalized space. This seems to be how it's intended to be used. Algebraic H and F will still use normalized coordinates which are more stable, after this matrices are converted to pixel space and Ceres refinement happens in pixel space. - Standard deviation calculation was wrong in GRIC. It shouldn't be deviation of residuals, but as per Torr it should be deviation of measurement error, which is constant (in our case 0.1) Not sure if using squared cost function is correct for GRIC, but cost function is indeed squared and in most papers cost function is used for GRIC. After some further tests we could switch GRIC residuals to non-squared distance. - Bring back rho part of GRIC, in unit tests it doesn't make sense whether it's enabled or not, lets see how it'll behave in real-life footage. - Added one more unit test based on elevator scene and manual keyframe selection. commit 24117f3c3fc5531beb6497d79bb6f1780a998081 Author: Sergey Sharybin Date: Sun Jan 6 19:07:06 2013 +0600 Added test for keyframe selection based on manual selection Additional changes: - Reduce minimal correspondence to match real-world manually tracked footage - Returned back squares to SymmetricEpipolarDistance and SymmetricGeometricDistance -- this is actually a cost functions, not distances and they shall be squared. commit 770eb0293b881c4c419c587a6cdb062c47ab6e44 Author: Sergey Sharybin Date: Fri Dec 21 00:43:30 2012 +0600 Improvements for keyframe selection - Changed main keyframe selection cycle, so in cases there're no more next keyframes for current keyframe could be found in the image sequence, current keyframe would be moved forward and search continues. This helps in cases when there's poor motion in the beginning of the sequence, then markers becomes occluded. There could be good keyframes in the middle of the shot still. - Extended keyframe_selection_test with real world cases. - Moved correspondences constraint to the top, so no H/F estimation happens if there's bad correspondence. Makes algorithm go a bit faster. Strangely, but using non-squared distances makes neighbor frames test fail, using squared distances makes this tests pass. However, using non-squared distances seems to be working better in real tests i've been doing. So this requires more investigation/ commit 7415c62fbda36c5bd1c291bc94d535a66da896d0 Author: Sergey Sharybin Date: Thu Dec 20 18:46:09 2012 +0600 Cosmetic change to correspondences reports in keyframe selection commit ceaf80c987ec0338e7e83965bc808411453eb755 Author: Sergey Sharybin Date: Thu Dec 20 18:08:03 2012 +0600 Various fixes: - That was a typo in symmetric geometric cost functor, which computed inverse distance in a wrong way. - Fixed compilation of unit tests - Added simple test for keyframe selection. Currently only covers case that neighbor frames with only translation (homography should be better than fundamental) are not considered a keyframes. Still need to be investigated why it only works if tracks are in pixel space and why doesn't work in normalized space. commit cfabdfe48df2add3d1f30cf4370efd0b31990ab0 Author: Sergey Sharybin Date: Thu Dec 20 05:46:53 2012 +0600 Assorted fixes for keyframe selection: - Biggest error was in cost functors used for F and H refirement, they were just wrong. - Use natural logarithms, since it's actually makes sense from math papers point of view and error is somewhere else. - Disabled rho for GRIC, for now use non-clamped error for tests. - Made SymmetricEpipolarDistance returning non-squared distance Keyframe selection is currently the only used of this function and it seems using non-squared distance makes much more sense. Also would think to append suffix "Squared" to functions which returns squared distances. - Removed templated version of SymmetricEpipolarDistance, since it's not needed actually. This is actually even worse working than previous implementation, but commit it needed for further review. commit 35d8c57626ad74818f155e6e5960c663ea84e032 Author: Sergey Sharybin Date: Thu Dec 20 03:00:40 2012 +0600 Euclidean resection cost function didn't use correct constructor It was storing a reference to initial rotation passed by value, leading to pointer being pointing to a stack variable, leading to wrong memory access in residuals computing. Apparently was visible in optimized builds only with inline substitution allowed. commit 0798d3162bb49cee7e1c423ceccbca1326ad5650 Author: Sergey Sharybin Date: Thu Dec 20 02:50:52 2012 +0600 Automatic keyframe selection based on Pollefeys's criteria This commit implements automatic keyframe selection algorithm based on Pollefeys's criteria (F-GRIC is smaller than H-GRIC and correspondence ratio is more then 90%). It is implemented as a part of simple pipeline and returns vector of keyframe images for a given Tracks structure. For simple pipeline reconstruction two best keyframes are expected to be selected from all detected candidates. Criteria for this selection could be reprojection error of solution from two candidate keyfames. Unfortunately, it's not fully workable yet, hopefully would be fixed soon. commit e943985552f0598ae122252876f305d72c25c2f9 Author: Sergey Sharybin Date: Thu Dec 6 17:47:11 2012 +0600 Camera Tracking: allow fallback to reprojection resection by user demand This fixes some "regressions" introduced in previous commit which lead to much worse solution in some cases. Now it's possible to bring old behavior back. Perhaps it's more like temporal solution for time being smarter solution is found. But finding such a solution isn't so fast, so let's bring manual control over reprojection usage. But anyway, imo it's now nice to have a structure which could be used to pass different settings to the solver.