Camera tracking integration

===========================

Fixed threading issues used to happen when compositor tree
and rendering happened in the same time. Problem was caused
by cached camera intrinsics stored in MovieTracking structure
and access to which wasn't thread-safe.

Now each movie distoriton mode stores it's own camera intrinsics
in it's storage which makes compositor (un)distoriton be totally
thread-safe and this also makes compositor tree faster when
(un)distortion happens on images with different resolution.
This commit is contained in:
Sergey Sharybin 2011-10-07 19:21:29 +00:00
parent f8b081153b
commit 92ddc03126
16 changed files with 308 additions and 283 deletions

206
extern/libmv/ChangeLog vendored

@ -1,3 +1,15 @@
commit 531c79bf95fddaaa70707d1abcd4fdafda16bbf0
Author: Matthias Fauconneau <matthias.fauconneau@gmail.com>
Date: Sat Aug 20 00:00:42 2011 +0200
Display warped pattern in marker preview.
commit bb5c27e671b6f8eb56ddf490f0795d59bede591b
Author: Matthias Fauconneau <matthias.fauconneau@gmail.com>
Date: Fri Aug 19 18:37:48 2011 +0200
Fix CMake build.
commit 2ac7281ff6b9545b425dd84fb03bf9c5c98b4de2
Author: Matthias Fauconneau <matthias.fauconneau@gmail.com>
Date: Fri Aug 19 17:34:45 2011 +0200
@ -298,197 +310,3 @@ Author: Matthias Fauconneau <matthias.fauconneau@gmail.com>
Date: Wed Jul 27 10:27:07 2011 +0200
Image Distortion: Bilinear sampling, Optimization, Instantiate all variants (Distort/Undistort, float/ubyte, 1-4 channels).
commit 91916db921e1f2818f0aa2be823bf92c50ad4de9
Author: Matthias Fauconneau <matthias.fauconneau@gmail.com>
Date: Tue Jul 26 22:51:21 2011 +0200
New Undistortion API in CameraIntrinsics.h.
Implement Undistortion API in CameraIntrinsics.cc.
commit 8c47a26072cfa9cf216771e5ae7a1dc60a770f82
Author: Matthias Fauconneau <matthias.fauconneau@gmail.com>
Date: Tue Jul 26 22:16:06 2011 +0200
Qt Calibration fixes. Image undistortion experiments.
commit b575d9f68856b4e95a6b0a92ecc4e7d635342f95
Author: Matthias Fauconneau <matthias.fauconneau@gmail.com>
Date: Tue Jul 26 22:14:54 2011 +0200
Qt Calibration fixes.
Image undistortion experiments.
commit fac2b3e88ef8f14fa62149f6fc929c623d73fe39
Author: Matthias Fauconneau <matthias.fauconneau@gmail.com>
Date: Mon Jul 25 11:44:17 2011 +0200
Merge uncalibrated reconstruction.
UI fixes.
commit d04071ee210baef5ff657441c8c5284c235e93a3
Merge: 795e50f c4c67db
Author: Matthias Fauconneau <matthias.fauconneau@gmail.com>
Date: Sat Jul 23 12:18:58 2011 +0200
Merge branch 'master' of git://github.com/keir/libmv
Conflicts:
src/libmv/simple_pipeline/initialize_reconstruction.h
src/ui/tracker/main.cc
src/ui/tracker/main.h
src/ui/tracker/scene.cc
src/ui/tracker/scene.h
commit 795e50fa1ca9ca4373ad9b5432916edf2f1940a0
Author: Matthias Fauconneau <matthias.fauconneau@gmail.com>
Date: Sat Jul 23 12:12:53 2011 +0200
Support creating FloatImage without copying data.
commit c4c67db84cc6e972be19c3e0f495477a1419200e
Author: Keir Mierle <mierle@gmail.com>
Date: Thu Jul 21 10:24:06 2011 -0700
Add an uncalibrated reconstruction pipeline to libmv.
Note: The pipeline doesn't actually work yet! It runs, but the resulting
reconstruction is wildly broken. I have a number of theories as to why this is,
and will write tests to track this down, but this change has grown out of
control in the meantime.
commit 584e0ddc4058a6a4e41b1fd9665654097af177d4
Author: Matthias Fauconneau <matthias.fauconneau@gmail.com>
Date: Wed Jul 20 20:21:55 2011 +0200
Latest working revision of simple optimized tracker.
commit 7983f86ff93f1ccd06f78439fb87387aecdfe49e
Author: Matthias Fauconneau <matthias.fauconneau@gmail.com>
Date: Wed Jul 20 20:10:12 2011 +0200
Revert "Incremental optimization: Change API to take float*."
This reverts commit 922dc5c31022afc7fd14b2ead32491c079565d7b.
commit 922dc5c31022afc7fd14b2ead32491c079565d7b
Author: Matthias Fauconneau <matthias.fauconneau@gmail.com>
Date: Wed Jul 20 17:11:20 2011 +0200
Incremental optimization: Change API to take float*.
commit 9d9fab4165aabad6474bb879b5f418c1b7a7862e
Author: Matthias Fauconneau <matthias.fauconneau@gmail.com>
Date: Wed Jul 20 17:05:14 2011 +0200
Incremental optimization.
commit 739fe5a118a2a5c90cf2c6d66c776231da0fb92b
Author: Matthias Fauconneau <matthias.fauconneau@gmail.com>
Date: Wed Jul 20 16:57:37 2011 +0200
Incremental optimization.
commit c48ba3b9f49e8fc5ef45ab1b6753f70bfdef0c34
Author: Matthias Fauconneau <matthias.fauconneau@gmail.com>
Date: Wed Jul 20 15:05:11 2011 +0200
Inline.
commit 2b7e704d947cafd12d67b3904365feb7c2b3e89a
Author: Matthias Fauconneau <matthias.fauconneau@gmail.com>
Date: Wed Jul 20 14:59:12 2011 +0200
Keep relative marker position in tracker.
commit a92e1fb70c2be05f9808bc43b87d3949790ef7ed
Author: Matthias Fauconneau <matthias.fauconneau@gmail.com>
Date: Wed Jul 20 10:56:52 2011 +0200
Avoid unecessary pyramid copy.
commit 31908b280e8862f15c3f67f697727dd692db9997
Author: Matthias Fauconneau <matthias.fauconneau@gmail.com>
Date: Wed Jul 20 10:45:54 2011 +0200
tracker.cc: Extract float image only when necessary.
klt.cc: Remove retrack which wasn't useful when we already use robust time-reversible KLT (+40% speed)
klt.cc: Factorize bilinear sampling
We are probably limited by bandwidth because of the float image filtering.
Next optimization step would require reducing filter sizes and use integer filtering.
Though, it will be necessary to verify the impact of tracking quality.
commit 433cd976f6047324ba27f21b3cafe3ecfbcb5aa1
Author: Matthias Fauconneau <matthias.fauconneau@gmail.com>
Date: Tue Jul 19 23:21:47 2011 +0200
Really add the simplified and optimized tracker.
commit caf938781bcd3e695f127a548678a1cf0a426b8f
Author: Matthias Fauconneau <matthias.fauconneau@gmail.com>
Date: Tue Jul 19 23:19:24 2011 +0200
Replace tracking/* trackers with the new simplified and optimized version.
This first optimization pass should result in a 4x speedup.
TODO: Integrating optimized filtering and optimized bilinear sampling should provide additionnal performance.
commit 13f5707573757a37d72b3d5be21a019049de9523
Author: Matthias Fauconneau <matthias.fauconneau@gmail.com>
Date: Tue Jul 19 23:14:19 2011 +0200
Update documentation.
commit a002fcf0ee42bd15991df34832cf557a5653c48e
Author: Matthias Fauconneau <matthias.fauconneau@gmail.com>
Date: Tue Jul 19 23:07:05 2011 +0200
Incremental Optimization #4: Keep last filtered pyramid.
This avoid computing the pyramid downsampling twice (as new patch and then as old patch).
API was changed as this require the client to keep our state.
commit 6772d7916126e710179370a5412d9380f05995a9
Author: Matthias Fauconneau <matthias.fauconneau@gmail.com>
Date: Tue Jul 19 22:54:30 2011 +0200
Incremental Optimization #3: Move Filtering in MakePyramid.
This avoid computing each pyramid filter twice (i.e for each call to TrackImage).
commit eec24f8c8b21ec286d12990b6e63772d7559bccc
Author: Matthias Fauconneau <matthias.fauconneau@gmail.com>
Date: Tue Jul 19 22:42:35 2011 +0200
Incremental Optimization #2: Move pyramid downsampling up to Track.
This avoid computing the pyramid downsampling twice (forward and backward).
commit 35cd70d321113956f2413084102a3a76ca35186f
Author: Matthias Fauconneau <matthias.fauconneau@gmail.com>
Date: Tue Jul 19 22:37:01 2011 +0200
Incremental optimization #1: Convert retrack/pyramid/KLT tracker objects to respective methods in a single Tracker object.
This is necessary to allow retrack to reuse pyramids and avoid filtering images multiple times.
commit 84341da380c585645f38e67dce5a8e1fd58242ac
Author: Matthias Fauconneau <matthias.fauconneau@gmail.com>
Date: Tue Jul 19 22:17:58 2011 +0200
Revert to working tracker in Qt Tracker.
TODO: The optimization changes shall be done incrementally.
commit 1d15e8572e8922c2a7aa8927f6f82c13ed6c3fc8
Author: Matthias Fauconneau <matthias.fauconneau@gmail.com>
Date: Tue Jul 19 21:18:21 2011 +0200
Fix missing deallocation.
commit a3ca18db4dec90414e658d480ea5f19b31bb8a77
Author: Matthias Fauconneau <matthias.fauconneau@gmail.com>
Date: Tue Jul 19 21:00:37 2011 +0200
Progress in debugging optimized KLT.

@ -13,7 +13,8 @@ tmp=`mktemp -d`
git clone $repo $tmp/libmv
git --git-dir $tmp/libmv/.git --work-tree $tmp/libmv log --since="1 month ago" > ChangeLog
#git --git-dir $tmp/libmv/.git --work-tree $tmp/libmv log --since="1 month ago" > ChangeLog
git --git-dir $tmp/libmv/.git --work-tree $tmp/libmv log -n 50 > ChangeLog
for p in `cat ./patches/series`; do
echo "Applying patch $p..."

@ -558,6 +558,14 @@ struct libmv_CameraIntrinsics *libmv_CameraIntrinsicsNew(double focal_length, do
return (struct libmv_CameraIntrinsics *) intrinsics;
}
struct libmv_CameraIntrinsics *libmv_CameraIntrinsicsCopy(struct libmv_CameraIntrinsics *libmvIntrinsics)
{
libmv::CameraIntrinsics *orig_intrinsics = (libmv::CameraIntrinsics *) libmvIntrinsics;
libmv::CameraIntrinsics *new_intrinsics= new libmv::CameraIntrinsics(*orig_intrinsics);
return (struct libmv_CameraIntrinsics *) new_intrinsics;
}
void libmv_CameraIntrinsicsDestroy(struct libmv_CameraIntrinsics *libmvIntrinsics)
{
libmv::CameraIntrinsics *intrinsics = (libmv::CameraIntrinsics *) libmvIntrinsics;

@ -82,6 +82,8 @@ void libmv_destroyFeatures(struct libmv_Features *libmv_features);
struct libmv_CameraIntrinsics *libmv_CameraIntrinsicsNew(double focal_length, double principal_x, double principal_y,
double k1, double k2, double k3, int width, int height);
struct libmv_CameraIntrinsics *libmv_CameraIntrinsicsCopy(struct libmv_CameraIntrinsics *libmvIntrinsics);
void libmv_CameraIntrinsicsDestroy(struct libmv_CameraIntrinsics *libmvIntrinsics);
void libmv_CameraIntrinsicsUpdate(struct libmv_CameraIntrinsics *libmvIntrinsics, double focal_length,

@ -23,8 +23,32 @@
namespace libmv {
struct Offset { signed char ix,iy; unsigned char fx,fy; };
struct Grid { struct Offset *offset; int width, height; };
struct Offset {
signed char ix, iy;
unsigned char fx,fy;
};
struct Grid {
struct Offset *offset;
int width, height;
};
static struct Grid *copyGrid(struct Grid *from)
{
struct Grid *to = NULL;
if (from) {
to = new Grid;
to->width = from->width;
to->height = from->height;
to->offset = new Offset[to->width*to->height];
memcpy(to->offset, from->offset, sizeof(struct Offset)*to->width*to->height);
}
return to;
}
CameraIntrinsics::CameraIntrinsics()
: K_(Mat3::Identity()),
@ -38,6 +62,20 @@ CameraIntrinsics::CameraIntrinsics()
distort_(0),
undistort_(0) {}
CameraIntrinsics::CameraIntrinsics(const CameraIntrinsics &from)
: K_(from.K_),
image_width_(from.image_width_),
image_height_(from.image_height_),
k1_(from.k1_),
k2_(from.k2_),
k3_(from.k3_),
p1_(from.p1_),
p2_(from.p2_)
{
distort_ = copyGrid(from.distort_);
undistort_ = copyGrid(from.undistort_);
}
CameraIntrinsics::~CameraIntrinsics() {
FreeLookupGrid();
}
@ -155,8 +193,8 @@ void CameraIntrinsics::ComputeLookupGrid(Grid* grid, int width, int height) {
double src_x = x / aspx, src_y = y / aspy;
double warp_x, warp_y;
WarpFunction(this,src_x,src_y,&warp_x,&warp_y);
warp_x *= aspx;
warp_y *= aspy;
warp_x = warp_x*aspx;
warp_y = warp_y*aspy;
int ix = int(warp_x), iy = int(warp_y);
int fx = round((warp_x-ix)*256), fy = round((warp_y-iy)*256);
if(fx == 256) { fx=0; ix++; }
@ -195,14 +233,14 @@ static void Warp(const Grid* grid, const T* src, T* dst,
void CameraIntrinsics::FreeLookupGrid() {
if(distort_) {
delete distort_;
delete distort_->offset;
delete distort_;
distort_ = NULL;
}
if(undistort_) {
delete undistort_;
delete undistort_->offset;
delete undistort_;
undistort_ = NULL;
}
}

@ -31,6 +31,7 @@ struct Grid;
class CameraIntrinsics {
public:
CameraIntrinsics();
CameraIntrinsics(const CameraIntrinsics &from);
~CameraIntrinsics();
const Mat3 &K() const { return K_; }

@ -1,18 +1,59 @@
diff --git a/src/libmv/simple_pipeline/camera_intrinsics.cc b/src/libmv/simple_pipeline/camera_intrinsics.cc
index f9888ff..e1e6117 100644
index f9888ff..110a16d 100644
--- a/src/libmv/simple_pipeline/camera_intrinsics.cc
+++ b/src/libmv/simple_pipeline/camera_intrinsics.cc
@@ -24,6 +24,7 @@
@@ -23,7 +23,32 @@
namespace libmv {
struct Offset { signed char ix,iy; unsigned char fx,fy; };
+struct Grid { struct Offset *offset; int width, height; };
-struct Offset { signed char ix,iy; unsigned char fx,fy; };
+struct Offset {
+ signed char ix, iy;
+ unsigned char fx,fy;
+};
+
+struct Grid {
+ struct Offset *offset;
+ int width, height;
+};
+
+static struct Grid *copyGrid(struct Grid *from)
+{
+ struct Grid *to = NULL;
+
+ if (from) {
+ to = new Grid;
+
+ to->width = from->width;
+ to->height = from->height;
+
+ to->offset = new Offset[to->width*to->height];
+ memcpy(to->offset, from->offset, sizeof(struct Offset)*to->width*to->height);
+ }
+
+ return to;
+}
CameraIntrinsics::CameraIntrinsics()
: K_(Mat3::Identity()),
@@ -38,8 +39,7 @@ CameraIntrinsics::CameraIntrinsics()
@@ -37,9 +62,22 @@ CameraIntrinsics::CameraIntrinsics()
distort_(0),
undistort_(0) {}
+CameraIntrinsics::CameraIntrinsics(const CameraIntrinsics &from)
+ : K_(from.K_),
+ image_width_(from.image_width_),
+ image_height_(from.image_height_),
+ k1_(from.k1_),
+ k2_(from.k2_),
+ k3_(from.k3_),
+ p1_(from.p1_),
+ p2_(from.p2_)
+{
+ distort_ = copyGrid(from.distort_);
+ undistort_ = copyGrid(from.undistort_);
+}
+
CameraIntrinsics::~CameraIntrinsics() {
- if(distort_) delete[] distort_;
- if(undistort_) delete[] undistort_;
@ -20,7 +61,7 @@ index f9888ff..e1e6117 100644
}
/// Set the entire calibration matrix at once.
@@ -146,11 +146,17 @@ void CameraIntrinsics::InvertIntrinsics(double image_x,
@@ -146,11 +184,17 @@ void CameraIntrinsics::InvertIntrinsics(double image_x,
// TODO(MatthiasF): downsample lookup
template<typename WarpFunction>
@ -35,12 +76,12 @@ index f9888ff..e1e6117 100644
double warp_x, warp_y;
- WarpFunction(this,x,y,&warp_x,&warp_y);
+ WarpFunction(this,src_x,src_y,&warp_x,&warp_y);
+ warp_x *= aspx;
+ warp_y *= aspy;
+ warp_x = warp_x*aspx;
+ warp_y = warp_y*aspy;
int ix = int(warp_x), iy = int(warp_y);
int fx = round((warp_x-ix)*256), fy = round((warp_y-iy)*256);
if(fx == 256) { fx=0; ix++; }
@@ -162,10 +168,10 @@ void CameraIntrinsics::ComputeLookupGrid(Offset* grid, int width, int height) {
@@ -162,10 +206,10 @@ void CameraIntrinsics::ComputeLookupGrid(Offset* grid, int width, int height) {
if( iy >= height-2 ) iy = height-2;
if ( ix-x > -128 && ix-x < 128 && iy-y > -128 && iy-y < 128 ) {
Offset offset = { ix-x, iy-y, fx, fy };
@ -53,7 +94,7 @@ index f9888ff..e1e6117 100644
}
}
}
@@ -173,11 +179,11 @@ void CameraIntrinsics::ComputeLookupGrid(Offset* grid, int width, int height) {
@@ -173,11 +217,11 @@ void CameraIntrinsics::ComputeLookupGrid(Offset* grid, int width, int height) {
// TODO(MatthiasF): cubic B-Spline image sampling, bilinear lookup
template<typename T,int N>
@ -67,27 +108,27 @@ index f9888ff..e1e6117 100644
const T* s = &src[((y+offset.iy)*width+(x+offset.ix))*N];
for (int i = 0; i < N; i++) {
dst[(y*width+x)*N+i] = ((s[ i] * (256-offset.fx) + s[ N+i] * offset.fx) * (256-offset.fy)
@@ -188,8 +194,17 @@ static void Warp(const Offset* grid, const T* src, T* dst,
@@ -188,8 +232,17 @@ static void Warp(const Offset* grid, const T* src, T* dst,
}
void CameraIntrinsics::FreeLookupGrid() {
- if(distort_) delete distort_, distort_=0;
- if(undistort_) delete undistort_, undistort_=0;
+ if(distort_) {
+ delete distort_;
+ delete distort_->offset;
+ delete distort_;
+ distort_ = NULL;
+ }
+
+ if(undistort_) {
+ delete undistort_;
+ delete undistort_->offset;
+ delete undistort_;
+ undistort_ = NULL;
+ }
}
// FIXME: C++ templates limitations makes thing complicated, but maybe there is a simpler method.
@@ -211,11 +226,50 @@ struct InvertIntrinsicsFunction {
@@ -211,11 +264,50 @@ struct InvertIntrinsicsFunction {
}
};
@ -142,7 +183,7 @@ index f9888ff..e1e6117 100644
if(channels==1) Warp<float,1>(distort_,src,dst,width,height);
else if(channels==2) Warp<float,2>(distort_,src,dst,width,height);
else if(channels==3) Warp<float,3>(distort_,src,dst,width,height);
@@ -224,10 +278,7 @@ void CameraIntrinsics::Distort(const float* src, float* dst, int width, int heig
@@ -224,10 +316,7 @@ void CameraIntrinsics::Distort(const float* src, float* dst, int width, int heig
}
void CameraIntrinsics::Distort(const unsigned char* src, unsigned char* dst, int width, int height, int channels) {
@ -154,7 +195,7 @@ index f9888ff..e1e6117 100644
if(channels==1) Warp<unsigned char,1>(distort_,src,dst,width,height);
else if(channels==2) Warp<unsigned char,2>(distort_,src,dst,width,height);
else if(channels==3) Warp<unsigned char,3>(distort_,src,dst,width,height);
@@ -236,10 +287,7 @@ void CameraIntrinsics::Distort(const unsigned char* src, unsigned char* dst, int
@@ -236,10 +325,7 @@ void CameraIntrinsics::Distort(const unsigned char* src, unsigned char* dst, int
}
void CameraIntrinsics::Undistort(const float* src, float* dst, int width, int height, int channels) {
@ -166,7 +207,7 @@ index f9888ff..e1e6117 100644
if(channels==1) Warp<float,1>(undistort_,src,dst,width,height);
else if(channels==2) Warp<float,2>(undistort_,src,dst,width,height);
else if(channels==3) Warp<float,3>(undistort_,src,dst,width,height);
@@ -248,10 +296,7 @@ void CameraIntrinsics::Undistort(const float* src, float* dst, int width, int he
@@ -248,10 +334,7 @@ void CameraIntrinsics::Undistort(const float* src, float* dst, int width, int he
}
void CameraIntrinsics::Undistort(const unsigned char* src, unsigned char* dst, int width, int height, int channels) {
@ -179,10 +220,10 @@ index f9888ff..e1e6117 100644
else if(channels==2) Warp<unsigned char,2>(undistort_,src,dst,width,height);
else if(channels==3) Warp<unsigned char,3>(undistort_,src,dst,width,height);
diff --git a/src/libmv/simple_pipeline/camera_intrinsics.h b/src/libmv/simple_pipeline/camera_intrinsics.h
index 29bc8a1..c89832f 100644
index 29bc8a1..f525571 100644
--- a/src/libmv/simple_pipeline/camera_intrinsics.h
+++ b/src/libmv/simple_pipeline/camera_intrinsics.h
@@ -26,7 +26,7 @@ typedef Eigen::Matrix<double, 3, 3> Mat3;
@@ -26,11 +26,12 @@ typedef Eigen::Matrix<double, 3, 3> Mat3;
namespace libmv {
@ -191,7 +232,12 @@ index 29bc8a1..c89832f 100644
class CameraIntrinsics {
public:
@@ -123,7 +123,9 @@ class CameraIntrinsics {
CameraIntrinsics();
+ CameraIntrinsics(const CameraIntrinsics &from);
~CameraIntrinsics();
const Mat3 &K() const { return K_; }
@@ -123,7 +124,9 @@ class CameraIntrinsics {
int width, int height, int channels);
private:
@ -202,7 +248,7 @@ index 29bc8a1..c89832f 100644
void FreeLookupGrid();
// The traditional intrinsics matrix from x = K[R|t]X.
@@ -140,8 +142,8 @@ class CameraIntrinsics {
@@ -140,8 +143,8 @@ class CameraIntrinsics {
// independent of image size.
double k1_, k2_, k3_, p1_, p2_;

@ -40,6 +40,7 @@ struct MovieClip;
struct MovieClipScopes;
struct MovieClipUser;
struct MovieTrackingTrack;
struct MovieDistortion;
void free_movieclip(struct MovieClip *clip);
void unlink_movieclip(struct Main *bmain, struct MovieClip *clip);
@ -62,7 +63,8 @@ void BKE_movieclip_update_scopes(struct MovieClip *clip, struct MovieClipUser *u
void BKE_movieclip_get_cache_segments(struct MovieClip *clip, struct MovieClipUser *user, int *totseg_r, int **points_r);
void BKE_movieclip_build_proxy_frame(struct MovieClip *clip, int cfra, int *build_sizes, int build_count, int undistorted);
void BKE_movieclip_build_proxy_frame(struct MovieClip *clip, struct MovieDistortion *distortion,
int cfra, int *build_sizes, int build_count, int undistorted);
#define TRACK_CLEAR_UPTO 0
#define TRACK_CLEAR_REMAINED 1

@ -40,6 +40,7 @@ struct MovieTrackingMarker;
struct MovieTracking;
struct MovieTrackingContext;
struct MovieClipUser;
struct MovieDistortion;
struct Scene;
struct Object;
@ -94,6 +95,13 @@ void BKE_tracking_stabilization_data(struct MovieTracking *tracking, int framenr
struct ImBuf *BKE_tracking_stabilize(struct MovieTracking *tracking, int framenr, struct ImBuf *ibuf, float loc[2], float *scale, float *angle);
void BKE_tracking_stabdata_to_mat4(int width, int height, float loc[2], float scale, float angle, float mat[4][4]);
/* Distoriton/Undistortion */
struct MovieDistortion *BKE_tracking_distortion_create(void);
struct MovieDistortion *BKE_tracking_distortion_copy(struct MovieDistortion *distortion);
struct ImBuf *BKE_tracking_distortion_exec(struct MovieDistortion *distortion, struct MovieTracking *tracking,
struct ImBuf *ibuf, int width, int height, int undistort);
void BKE_tracking_distortion_destroy(struct MovieDistortion *distortion);
struct ImBuf *BKE_tracking_undistort(struct MovieTracking *tracking, struct ImBuf *ibuf, int width, int height);
struct ImBuf *BKE_tracking_distort(struct MovieTracking *tracking, struct ImBuf *ibuf, int width, int height);

@ -523,7 +523,7 @@ static ImBuf *get_undistorted_cache(MovieClip *clip, MovieClipUser *user)
return cache->undistibuf;
}
static ImBuf *get_undistorted_ibuf(MovieClip *clip, ImBuf *ibuf)
static ImBuf *get_undistorted_ibuf(MovieClip *clip, struct MovieDistortion *distoriton, ImBuf *ibuf)
{
ImBuf *undistibuf;
@ -531,7 +531,10 @@ static ImBuf *get_undistorted_ibuf(MovieClip *clip, ImBuf *ibuf)
otherwise, undistorted proxy can be darker than it should */
imb_freerectfloatImBuf(ibuf);
undistibuf= BKE_tracking_undistort(&clip->tracking, ibuf, ibuf->x, ibuf->y);
if(distoriton)
undistibuf= BKE_tracking_distortion_exec(distoriton, &clip->tracking, ibuf, ibuf->x, ibuf->y, 1);
else
undistibuf= BKE_tracking_undistort(&clip->tracking, ibuf, ibuf->x, ibuf->y);
if(undistibuf->userflags|= IB_RECT_INVALID) {
ibuf->userflags&= ~IB_RECT_INVALID;
@ -555,7 +558,7 @@ static ImBuf *put_undistorted_cache(MovieClip *clip, MovieClipUser *user, ImBuf
if(cache->undistibuf)
IMB_freeImBuf(cache->undistibuf);
cache->undistibuf= get_undistorted_ibuf(clip, ibuf);
cache->undistibuf= get_undistorted_ibuf(clip, NULL, ibuf);
if(cache->stableibuf) {
/* force stable buffer be re-calculated */
@ -948,7 +951,8 @@ static void movieclip_build_proxy_ibuf(MovieClip *clip, ImBuf *ibuf, int cfra, i
IMB_freeImBuf(scaleibuf);
}
void BKE_movieclip_build_proxy_frame(MovieClip *clip, int cfra, int *build_sizes, int build_count, int undistorted)
void BKE_movieclip_build_proxy_frame(MovieClip *clip, struct MovieDistortion *distortion,
int cfra, int *build_sizes, int build_count, int undistorted)
{
ImBuf *ibuf;
MovieClipUser user;
@ -962,7 +966,7 @@ void BKE_movieclip_build_proxy_frame(MovieClip *clip, int cfra, int *build_sizes
int i;
if(undistorted)
tmpibuf= get_undistorted_ibuf(clip, ibuf);
tmpibuf= get_undistorted_ibuf(clip, distortion, ibuf);
for(i= 0; i<build_count; i++)
movieclip_build_proxy_ibuf(clip, tmpibuf, cfra, build_sizes[i], undistorted);

@ -59,6 +59,10 @@
#include "libmv-capi.h"
#endif
typedef struct MovieDistortion {
struct libmv_CameraIntrinsics *intrinsics;
} MovieDistortion;
/*********************** common functions *************************/
void BKE_tracking_clamp_track(MovieTrackingTrack *track, int event)
@ -464,10 +468,8 @@ void BKE_tracking_free(MovieTracking *tracking)
if(tracking->stabilization.scaleibuf)
IMB_freeImBuf(tracking->stabilization.scaleibuf);
#ifdef WITH_LIBMV
if(tracking->camera.intrinsics)
libmv_CameraIntrinsicsDestroy(tracking->camera.intrinsics);
#endif
BKE_tracking_distortion_destroy(tracking->camera.intrinsics);
}
/*********************** tracking *************************/
@ -1897,80 +1899,111 @@ void BKE_tracking_stabdata_to_mat4(int width, int height, float loc[2], float sc
mul_serie_m4(mat, lmat, smat, cmat, rmat, icmat, NULL, NULL, NULL);
}
ImBuf *BKE_tracking_undistort(MovieTracking *tracking, ImBuf *ibuf, int width, int height)
MovieDistortion *BKE_tracking_distortion_create(void)
{
MovieDistortion *distortion;
distortion= MEM_callocN(sizeof(MovieDistortion), "BKE_tracking_distortion_create");
return distortion;
}
MovieDistortion *BKE_tracking_distortion_copy(MovieDistortion *distortion)
{
MovieDistortion *new_distortion;
new_distortion= MEM_callocN(sizeof(MovieDistortion), "BKE_tracking_distortion_create");
#ifdef WITH_LIBMV
new_distortion->intrinsics= libmv_CameraIntrinsicsCopy(distortion->intrinsics);
#endif
return new_distortion;
}
void BKE_tracking_distortion_update(MovieDistortion *distortion, MovieTracking *tracking, int width, int height)
{
ImBuf *resibuf;
MovieTrackingCamera *camera= &tracking->camera;
float aspy= 1.f/tracking->camera.pixel_aspect;
resibuf= IMB_dupImBuf(ibuf);
#ifdef WITH_LIBMV
if(camera->intrinsics == NULL) {
camera->intrinsics= libmv_CameraIntrinsicsNew(camera->focal,
if(!distortion->intrinsics) {
distortion->intrinsics= libmv_CameraIntrinsicsNew(camera->focal,
camera->principal[0], camera->principal[1] * aspy,
camera->k1, camera->k2, camera->k3, width, height * aspy);
} else {
libmv_CameraIntrinsicsUpdate(camera->intrinsics, camera->focal,
libmv_CameraIntrinsicsUpdate(distortion->intrinsics, camera->focal,
camera->principal[0], camera->principal[1] * aspy,
camera->k1, camera->k2, camera->k3, width, height * aspy);
}
#endif
}
ImBuf *BKE_tracking_distortion_exec(MovieDistortion *distortion, MovieTracking *tracking,
ImBuf *ibuf, int width, int height, int undistort)
{
ImBuf *resibuf;
BKE_tracking_distortion_update(distortion, tracking, width, height);
resibuf= IMB_dupImBuf(ibuf);
if(ibuf->rect_float) {
#ifdef WITH_LIBMV
libmv_CameraIntrinsicsUndistortFloat(camera->intrinsics,
ibuf->rect_float, resibuf->rect_float,
ibuf->x, ibuf->y, ibuf->channels);
if(undistort) {
libmv_CameraIntrinsicsUndistortFloat(distortion->intrinsics,
ibuf->rect_float, resibuf->rect_float,
ibuf->x, ibuf->y, ibuf->channels);
} else {
libmv_CameraIntrinsicsDistortFloat(distortion->intrinsics,
ibuf->rect_float, resibuf->rect_float,
ibuf->x, ibuf->y, ibuf->channels);
}
#endif
ibuf->userflags|= IB_RECT_INVALID;
} else {
#ifdef WITH_LIBMV
libmv_CameraIntrinsicsUndistortByte(camera->intrinsics,
(unsigned char*)ibuf->rect, (unsigned char*)resibuf->rect,
ibuf->x, ibuf->y, ibuf->channels);
if(undistort) {
libmv_CameraIntrinsicsUndistortByte(distortion->intrinsics,
(unsigned char*)ibuf->rect, (unsigned char*)resibuf->rect,
ibuf->x, ibuf->y, ibuf->channels);
} else {
libmv_CameraIntrinsicsDistortByte(distortion->intrinsics,
(unsigned char*)ibuf->rect, (unsigned char*)resibuf->rect,
ibuf->x, ibuf->y, ibuf->channels);
}
#endif
}
return resibuf;
}
void BKE_tracking_distortion_destroy(MovieDistortion *distortion)
{
#ifdef WITH_LIBMV
libmv_CameraIntrinsicsDestroy(distortion->intrinsics);
#endif
MEM_freeN(distortion);
}
ImBuf *BKE_tracking_undistort(MovieTracking *tracking, ImBuf *ibuf, int width, int height)
{
MovieTrackingCamera *camera= &tracking->camera;
if(camera->intrinsics == NULL)
camera->intrinsics= BKE_tracking_distortion_create();
return BKE_tracking_distortion_exec(camera->intrinsics, tracking, ibuf, width, height, 1);
}
ImBuf *BKE_tracking_distort(MovieTracking *tracking, ImBuf *ibuf, int width, int height)
{
ImBuf *resibuf;
MovieTrackingCamera *camera= &tracking->camera;
float aspy= 1.f/tracking->camera.pixel_aspect;
resibuf= IMB_dupImBuf(ibuf);
if(camera->intrinsics == NULL)
camera->intrinsics= BKE_tracking_distortion_create();
#ifdef WITH_LIBMV
if(camera->intrinsics == NULL) {
camera->intrinsics= libmv_CameraIntrinsicsNew(camera->focal,
camera->principal[0], camera->principal[1] * aspy,
camera->k1, camera->k2, camera->k3, width, height * aspy);
} else {
libmv_CameraIntrinsicsUpdate(camera->intrinsics, camera->focal,
camera->principal[0], camera->principal[1] * aspy,
camera->k1, camera->k2, camera->k3, width, height * aspy);
}
#endif
if(ibuf->rect_float) {
#ifdef WITH_LIBMV
libmv_CameraIntrinsicsDistortFloat(camera->intrinsics,
ibuf->rect_float, resibuf->rect_float,
ibuf->x, ibuf->y, ibuf->channels);
#endif
ibuf->userflags|= IB_RECT_INVALID;
} else {
#ifdef WITH_LIBMV
libmv_CameraIntrinsicsDistortByte(camera->intrinsics,
(unsigned char*)ibuf->rect, (unsigned char*)resibuf->rect,
ibuf->x, ibuf->y, ibuf->channels);
#endif
}
return resibuf;
return BKE_tracking_distortion_exec(camera->intrinsics, tracking, ibuf, width, height, 0);
}

@ -1253,6 +1253,7 @@ void blo_end_image_pointer_map(FileData *fd, Main *oldmain)
void blo_make_movieclip_pointer_map(FileData *fd, Main *oldmain)
{
MovieClip *clip= oldmain->movieclip.first;
Scene *sce= oldmain->scene.first;
fd->movieclipmap= oldnewmap_new();
@ -1263,6 +1264,15 @@ void blo_make_movieclip_pointer_map(FileData *fd, Main *oldmain)
if(clip->tracking.camera.intrinsics)
oldnewmap_insert(fd->movieclipmap, clip->tracking.camera.intrinsics, clip->tracking.camera.intrinsics, 0);
}
for(; sce; sce= sce->id.next) {
if(sce->nodetree) {
bNode *node;
for(node= sce->nodetree->nodes.first; node; node= node->next)
if(node->type==CMP_NODE_MOVIEDISTORTION)
oldnewmap_insert(fd->movieclipmap, node->storage, node->storage, 0);
}
}
}
/* set old main movie clips caches to zero if it has been restored */
@ -1271,6 +1281,7 @@ void blo_end_movieclip_pointer_map(FileData *fd, Main *oldmain)
{
OldNew *entry= fd->movieclipmap->entries;
MovieClip *clip= oldmain->movieclip.first;
Scene *sce= oldmain->scene.first;
int i;
/* used entries were restored, so we put them to zero */
@ -1283,6 +1294,15 @@ void blo_end_movieclip_pointer_map(FileData *fd, Main *oldmain)
clip->cache= newmclipadr(fd, clip->cache);
clip->tracking.camera.intrinsics= newmclipadr(fd, clip->tracking.camera.intrinsics);
}
for(; sce; sce= sce->id.next) {
if(sce->nodetree) {
bNode *node;
for(node= sce->nodetree->nodes.first; node; node= node->next)
if(node->type==CMP_NODE_MOVIEDISTORTION)
node->storage= newmclipadr(fd, node->storage);
}
}
}
@ -2271,7 +2291,11 @@ static void direct_link_nodetree(FileData *fd, bNodeTree *ntree)
link_list(fd, &node->inputs);
link_list(fd, &node->outputs);
node->storage= newdataadr(fd, node->storage);
if(node->type == CMP_NODE_MOVIEDISTORTION) {
node->storage= newmclipadr(fd, node->storage);
} else
node->storage= newdataadr(fd, node->storage);
if(node->storage) {
/* could be handlerized at some point */
if(ntree->type==NTREE_SHADER && (node->type==SH_NODE_CURVE_VEC || node->type==SH_NODE_CURVE_RGB))

@ -713,6 +713,8 @@ static void write_nodetree(WriteData *wd, bNodeTree *ntree)
write_curvemapping(wd, node->storage);
else if(ntree->type==NTREE_TEXTURE && (node->type==TEX_NODE_CURVE_RGB || node->type==TEX_NODE_CURVE_TIME) )
write_curvemapping(wd, node->storage);
else if(ntree->type==NTREE_COMPOSIT && node->type==CMP_NODE_MOVIEDISTORTION)
/* pass */ ;
else
writestruct(wd, DATA, node->typeinfo->storagename, 1, node->storage);
}

@ -44,6 +44,7 @@
#include "BKE_library.h"
#include "BKE_movieclip.h"
#include "BKE_sound.h"
#include "BKE_tracking.h"
#include "WM_api.h"
#include "WM_types.h"
@ -862,6 +863,7 @@ static void proxy_startjob(void *pjv, short *stop, short *do_update, float *prog
ProxyJob *pj= pjv;
Scene *scene=pj->scene;
MovieClip *clip= pj->clip;
struct MovieDistortion *distortion= NULL;
int cfra, undistort;
short tc_flag, size_flag, quality, build_flag;
int sfra= SFRA, efra= EFRA;
@ -891,12 +893,15 @@ static void proxy_startjob(void *pjv, short *stop, short *do_update, float *prog
if(size_flag&IMB_PROXY_75) build_sizes[build_count++]= MCLIP_PROXY_RENDER_SIZE_75;
if(size_flag&IMB_PROXY_100) build_sizes[build_count++]= MCLIP_PROXY_RENDER_SIZE_100;
if(undistort)
distortion= BKE_tracking_distortion_create();
for(cfra= sfra; cfra<=efra; cfra++) {
if(clip->source != MCLIP_SRC_MOVIE)
BKE_movieclip_build_proxy_frame(clip, cfra, build_sizes, build_count, 0);
BKE_movieclip_build_proxy_frame(clip, NULL, cfra, build_sizes, build_count, 0);
if(undistort)
BKE_movieclip_build_proxy_frame(clip, cfra, build_sizes, build_count, 1);
BKE_movieclip_build_proxy_frame(clip, distortion, cfra, build_sizes, build_count, 1);
if(*stop || G.afbreek)
break;
@ -904,6 +909,9 @@ static void proxy_startjob(void *pjv, short *stop, short *do_update, float *prog
*do_update= 1;
*progress= ((float)cfra)/(efra-sfra);
}
if(distortion)
BKE_tracking_distortion_destroy(distortion);
}
static int sequencer_rebuild_proxy_exec(bContext *C, wmOperator *UNUSED(op))

@ -47,6 +47,7 @@
#include "BKE_global.h"
#include "BKE_main.h"
#include "BKE_node.h"
#include "BKE_tracking.h"
#include "BKE_utildefines.h"
#include "node_exec.h"
@ -170,6 +171,17 @@ static void local_merge(bNodeTree *localtree, bNodeTree *ntree)
BKE_image_merge((Image *)lnode->new_node->id, (Image *)lnode->id);
}
}
else if(lnode->type==CMP_NODE_MOVIEDISTORTION) {
/* special case for distortion node: distortion context is allocating in exec function
and to achive much better performance on further calls this context should be
copied back to original node */
if(lnode->storage) {
if(lnode->new_node->storage)
BKE_tracking_distortion_destroy(lnode->new_node->storage);
lnode->new_node->storage= BKE_tracking_distortion_copy(lnode->storage);
}
}
for(lsock= lnode->outputs.first; lsock; lsock= lsock->next) {
if(ntreeOutputExists(lnode->new_node, lsock->new_sock)) {

@ -60,17 +60,20 @@ static void exec(void *UNUSED(data), bNode *node, bNodeStack **in, bNodeStack **
if(ibuf) {
ImBuf *obuf;
MovieClipUser *user= (MovieClipUser *)node->storage;
MovieTracking *tracking= &clip->tracking;
int width, height;
ibuf->rect_float= cbuf->rect;
BKE_movieclip_acquire_size(clip, user, &width, &height);
BKE_movieclip_acquire_size(clip, NULL, &width, &height);
if(!node->storage)
node->storage= BKE_tracking_distortion_create();
if(node->custom1==0)
obuf= BKE_tracking_undistort(&clip->tracking, ibuf, width, height);
obuf= BKE_tracking_distortion_exec(node->storage, tracking, ibuf, width, height, 1);
else
obuf= BKE_tracking_distort(&clip->tracking, ibuf, width, height);
obuf= BKE_tracking_distortion_exec(node->storage, tracking, ibuf, width, height, 0);
stackbuf->rect= obuf->rect_float;
stackbuf->malloc= 1;
@ -111,6 +114,20 @@ static const char *label(bNode *node)
return "Distortion";
}
static void storage_free(bNode *node)
{
if(node->storage)
BKE_tracking_distortion_destroy(node->storage);
node->storage= NULL;
}
void storage_copy(bNode *orig_node, bNode *new_node)
{
if(orig_node->storage)
new_node->storage= BKE_tracking_distortion_copy(orig_node->storage);
}
void register_node_type_cmp_moviedistortion(ListBase *lb)
{
static bNodeType ntype;
@ -121,6 +138,7 @@ void register_node_type_cmp_moviedistortion(ListBase *lb)
node_type_init(&ntype, init);
node_type_label(&ntype, label);
node_type_exec(&ntype, exec);
node_type_storage(&ntype, NULL, storage_free, storage_copy);
nodeRegisterType(lb, &ntype);
}