One of the basic tasks in Computer Stereo Vision is to calibrate the stereo camera in order to obtain the parameters that will allow you to calculate 3D information of the scene.
Now, I could tell you a lot of stuff about camera projection models, stereoscopy, lens distortion, etc... but there is a lot of information available about such topics out there. So, this post is for those who simply need to calibrate a stereo camera system and start calculating 3D stuff right away by using OpenCV.
Anyway, I strongly recommend you to read the book: Learning OpenCV: Computer Vision with the OpenCV Library by Gary Bradski and Adrian Kaehler, Published by O'Reilly Media, October 3, 2008.
So... what do I need to calibrate my stereo camera? A chessboard like this:
Why a chessboard? Because its corners are very easy to find by using computer vision algorithms and its geometry is very simple. In order to find out the position of any corner you only need to know how many horizontal and vertical squares there are in the chessboard and the size of a square. The chessboard in the image is a 9x6 chessboard and if you print it in a paper of size A4 the size of the squares would be more or less 2.5cm.
OK, I've printed my chessboard and I have measured the real size of the squares, now what?
Now you just take multiple views of the chessboard in different positions and orientations with your stereo camera using your favorite software (maybe your own software, software provided by your camera manufacturer or some other free software like Coriander). The images should look like this:
 |
| (Yeah, that is me in Hawaiian shorts on a summer day :P) |
The more variety of positions and orientations of the checkerboard in the images the better.
Great, you have taken a lot of shots of the chessboard in different positions, now create a text file with the paths to the images. For example:
images/left01.ppm images/right01.ppm images/left02.ppm images/right02.ppm images/left03.ppm images/right03.ppm images/left04.ppm images/right04.ppm images/left05.ppm images/right05.ppm images/left06.ppm images/right06.ppm images/left07.ppm images/right07.ppm images/left08.ppm images/right08.ppm images/left09.ppm images/right09.ppm images/left10.ppm images/right10.ppm images/left11.ppm images/right11.ppm images/left12.ppm images/right12.ppm images/left13.ppm images/right13.ppm images/left14.ppm images/right14.ppm images/left15.ppm images/right15.ppm images/left16.ppm images/right16.ppm images/left17.ppm images/right17.ppm images/left18.ppm images/right18.ppm images/left19.ppm images/right19.ppm images/left20.ppm images/right20.ppm
Now download this software and compile it.
It is just one of the examples of the book mentioned above that I modified to accept some configuration parameters and store the results of the calibration. The usage of the software is as follows:
It is just one of the examples of the book mentioned above that I modified to accept some configuration parameters and store the results of the calibration. The usage of the software is as follows:
USAGE: ./stereo_calibrate imageList nx ny squareSize
imageList : Filename of the image list (string). Example : list.txt
nx : Number of horizontal squares (int > 0). Example : 9
ny : Number of vertical squares (int > 0). Example : 6
squareSize : Size of a square (float > 0). Example : 2.5
imageList : Filename of the image list (string). Example : list.txt
nx : Number of horizontal squares (int > 0). Example : 9
ny : Number of vertical squares (int > 0). Example : 6
squareSize : Size of a square (float > 0). Example : 2.5
So, in this example the call to the program stereo_calibrate would be:
./stereo_calibrate list.txt 9 6 2.5
The program will start showing the detected chessboards, calculate the calibration parameters and store them in a bunch of xml files:
D1.xml D2.xml
M1.xml M2.xml
mx1.xml mx2.xml
my1.xml my2.xml
P1.xml P2.xml
R1.xml R2.xml
Q.xml
M1.xml M2.xml
mx1.xml mx2.xml
my1.xml my2.xml
P1.xml P2.xml
R1.xml R2.xml
Q.xml
Congratulations! You have calibrated your stereo camera!! Now you can load this parameters into any other program that uses that stereo camera and play with them:
CvMat *Q = (CvMat *)cvLoad("Q.xml",NULL,NULL,NULL);
CvMat *mx1 = (CvMat *)cvLoad("mx1.xml",NULL,NULL,NULL);
CvMat *my1 = (CvMat *)cvLoad("my1.xml",NULL,NULL,NULL);
CvMat *mx2 = (CvMat *)cvLoad("mx2.xml",NULL,NULL,NULL);
CvMat *my2 = (CvMat *)cvLoad("my2.xml",NULL,NULL,NULL);
CvMat *mx1 = (CvMat *)cvLoad("mx1.xml",NULL,NULL,NULL);
CvMat *my1 = (CvMat *)cvLoad("my1.xml",NULL,NULL,NULL);
CvMat *mx2 = (CvMat *)cvLoad("mx2.xml",NULL,NULL,NULL);
CvMat *my2 = (CvMat *)cvLoad("my2.xml",NULL,NULL,NULL);
Each of the files contains a matrix, if you would like to know the meaning of each matrix, please refer to the book at the beginning of this post. Right now, the useful stuff is contained on the files mx1.xml, my1.xml, mx2.xml, my2.xml and Q.xml.
The files m*.xml are the distortion models of the cameras. So we will need these matrices to undo the distortion of the images caused by the lens. Using the cvRemap() function:
cvRemap(imgLeftOrig, imgLeftUndistorted, mx1, my1);
cvRemap(imgRightOrig, imgRightUndistoreted, mx2, my2);
cvRemap(imgRightOrig, imgRightUndistoreted, mx2, my2);
The goal of all this is to be able to calculate the 3D position (in meters, cm, mm or whatever magnitude you chose) of a point given its position (in pixels) on the left image and its correspondent on the right image. We are almost there, but for that we need the matrix Q. Given the position an interest point in the left and right image, its 3D position can be calculated as follows:
d = pointRightImage.X - pointLeftImage.X;
X = pointLeftImage.X * Q[0, 0] + Q[0, 3];
Y = pointLeftImage.Y * Q[1, 1] + Q[1, 3];
Z = Q[2, 3];
W = d * Q[3, 2] + Q[3, 3];
X = X / W;
Y = Y / W;
Z = Z / W;
X = pointLeftImage.X * Q[0, 0] + Q[0, 3];
Y = pointLeftImage.Y * Q[1, 1] + Q[1, 3];
Z = Q[2, 3];
W = d * Q[3, 2] + Q[3, 3];
X = X / W;
Y = Y / W;
Z = Z / W;
And thats pretty much it, now you know how to calculate 3D positions from 2 images using OpenCV. How to find interest points in one image and its correspondent on the other is an art that will be explained another day ;)
EDIT [2011/06/16]: Many people asked me about a good book to get started in OpenCV, so take a look at this: OpenCV 2 Computer Vision Application Programming Cookbook
EDIT [2011/08/22]: What next? Check out the following post: OpenCV: Stereo Matching
EDIT [2011/08/27]: I changed the Makefile of the software for one much simpler and without hard-coded stuff. Also hosted the software in my own server (it was in megaupload, sorry about that), the link to the software in this post has been updated. Here it is, just in case: http://www.martinperis.com/stereocalibration/OpenCV-Stereo-Calibration.tgz
EDIT [2012/01/05]: My hosting bandwidth was not enough to handle the traffic so I had to host the software on googlecode, the link to the software in this post has been updated. Here it is, just in case: http://opencv-stereo-calibration.googlecode.com/files/OpenCV-Stereo-Calibration-1.0.tgz
EDIT [2011/06/16]: Many people asked me about a good book to get started in OpenCV, so take a look at this: OpenCV 2 Computer Vision Application Programming Cookbook
EDIT [2011/08/22]: What next? Check out the following post: OpenCV: Stereo Matching
EDIT [2011/08/27]: I changed the Makefile of the software for one much simpler and without hard-coded stuff.
EDIT [2012/01/05]: My hosting bandwidth was not enough to handle the traffic so I had to host the software on googlecode, the link to the software in this post has been updated. Here it is, just in case: http://opencv-stereo-calibration.googlecode.com/files/OpenCV-Stereo-Calibration-1.0.tgz
206 comments:
1 – 200 of 206 Newer› Newest»good job, thanks! but i can see some bugs. in BOUGUET'S method dont save xml, dont show windows, etc. and the HARTLEY'S method cant save, dont have the code :)
Hi gatito1979!
Thanks for your comment, when I modified the source code of the calibration example in the book "Learning OpenCV" I was only interested in saving the results of Bouguet's method, that is why Hartley's method doesn't have the code to save the xml files.
About the bugs in Bouguet's method, it saves the xml and show the windows correctly for me, sorry maybe I misunderstood you :S
Hi Martin!
It's a good job!
I'd like to know what linux distribution do you use. I'm a enthusiast opencv developer and I'm having some trouble using windows.
Thanks.
Hi Bernardo!
Thanks a lot!
Currently I use Ubuntu 10.04 but I could use OpenCV without problems on previous releases of Ubuntu and Debian.
I hope Linux could solve your problems :)
Hi Martin,
This is great. I am also developing based on OpenCV's stereo camera functions for 3D position measuring using cameras. Have you tried testing to see if the calculated 3D position is accurate? For example measure the 3D position from the camera and see if the calculated values corresponds. It'll be interesting to know how accurate the Q matrix is.
Thanks and good work.
Hi cc!
Thanks a lot for your comment! I have tested (a little bit) the accuracy of the calibration obtained in the real world and also in a simulated environment. For both I used images of 320x240 pixels (with higher resolution images one can expect higher accuracy)
For the real world I measured (by hand) the distance to a target object at several locations and then compared it to the measurements provided by the Q matrix and I got an accuracy around +/- 3cm (quite big, but you have to take into account the error introduced by myself measuring distances with a ruler).
For the simulated world (I use a robot simulated in Microsoft Robotics Developer Studio) where I can control the exact position of the camera and the target, the accuracy was +/- 1cm
These accuracy results is more than enough for my purposes.
Anyway, one should keep in mind that depth and disparity are inversely related, so fine-grain depth measurement is restricted to nearby objects.
Best regards,
Martin.
Hi Martin,
Nice job.
I have a question about point transformation. Supposing there is a point P1 in the original imge, how can I calculate the coressponding point P2 in the udistorted image (mx and my are already calculated using "cvRemap").
Thanks a lot.
Hi HWT!
Thanks for your comment!
Regarding your question, if I understood it correctly, you can figure it out by knowing what cvRemap really does: cvRemap()
mx and my are not anything else but a couple of matrices of the same size as your images that contains the correspondent coordinate in the undistorted image for a point in the original image.
So what you are looking for is just this:
P2.x = mx(P1.x, P1.y)
P2.y = my(P1.x, P1.y)
I hope this helps :)
Best regards,
Martin
Hi Martin,
This is Divya from India. We are carrying out a similar project as part of our graduation studies.
As far as we have understood, the values of reprojection matrix Q obtained using the stereorectify function is in terms of the square size of the chessboard we are using to calibrate the camera.Also, the intrinsics and distortion matrices found using cameracalibrate2 function are in terms of the square size. In the final equation to find the 3D coords, its Q*{transpose of[x,y,d,1]}={transpose of[X Y Z W]} where Q is in terms of square size which we can adjust to our unit and x,y,d are in terms of pixels. So, 3D coords matrix elements- X,Y,Z are found to be in which units?
Thanks.
Hi Divya,
The 3D coords matrix elements (X,Y,Z) are in the same units as the chessboard's square size.
For example, if you define the square size to be 1cm then the 3D coords will be in cm as well.
Best regards,
Martin.
Hi Martin,
Thanks a Lot.
:)
No problem Divya!
I'm glad I could help :)
Hi Martin:
This is Weimin from Singapore. I would like to share some of my current camera calibration problem see if you could help me.
We are doing a research project to get depth map. We now have two high-resolution Chameleon camera (both 1280*960) mounted together with principle line almost parallel and we have to do the calibration for them in order to get the intrinsic and extrinsic parameters . I have been using matlab camera calibration toolbox to do the job but the results are too bad. The focal length, principle point, and distortion coefficients all have very high uncertainties, some results even do not make sense, where their example online gives a quite good result.And when I switched back to my cheap webcams, they all worked quite well, with bearable uncertainties.
So I am quite confused now. My expensive high-resolution cameras with little distortion can not be calibrated well, while my cheap webcams, based on the exactly same methods, can. So what could be the possible reasons?
Regards,
Weimin
Hi Weimin!
Thank you for sharing your experience, how is the weather in Singapore? I haven't been there yet :)
About your problem, I understand perfectly your frustration. It is very annoying when things doesn't work like they are supposed to. I have never used Matlab camera calibration toolbox, but if with cameras of lower resolution works then I am pretty sure that the problem is not on that toolbox.
I haven's used Chamelon cameras either, but from my experience with other high-resolution cameras the first thing that comes into my mind is: Synchrony.
Have you checked the synchrony between both cameras? (Maybe you could check the timestamp on your images when you retrieve them)
As far as I know, Chamelon cameras are connected using USB, like cheap webcams. The thing with cheap webcams is that, as they have low resolution, the bandwidth that they consume is low. And also, usually cheap webcams send the images to the computer in jpg format, so the bandwidth that they use is even lower.
Now, expensive high-resolution cameras use a lot more of bandwidth, and they can send the image in raw format, which means even more bandwidth consumption. That is a lot of data to be transfered using USB connection. And there is two of those cameras connected to your computer, which can result in delays between frames.
So, if you are moving the calibration target around the cameras by hand, it would be likely that the left image has been taken in a slightly different moment than the right image.
That slight difference of time between left and right frames could perfectly be the reason why your calibration algorithm is becoming crazy.
So, I would try to check the synchrony between left and right images.
I hope this could help you. If you are so kind, please, comment back any findings on this matter. I am curious about it :)
Best regards,
Martin
Hi Martin:
Rain season has delayed until Feb. this year, and it has been raining all day long. Anyway, the weather here is quite nice most of the time:)
Thank you so much for help!
You are right that the image is sent in raw format. And then I have to copy the data from raw images into IplImage in order to use OpenCV to process the them. And the image format casting should be no problem because I can display and process them by calling OpenCV functions.
But I have to say that the problem is not in the synchrony. Because by using the Matlab toolbox, the first thing I do is calibrate each camera separately. But neither of them could give good result.The focal length, principle point position and distortion coefficients all have huge uncertainties.
The way I do is take around 20 images for each camera from different viewpoints as the online example images, then I manually label the chessboard corners. The images look quite clear and almost no distortion can be seen. If everything worked well, I would call another matlab function: stereo_gui to combine two cameras' intrinsic I got in the calibration stage in order to get the final rotation and translation matrix between two cameras.And only at this time would I use synchronized images. But before I could get there, I had to stop at the calibration stage.
I already sent email to the Chameleon company tech support. They replied that high resolution may have slight variety and instabilities. I am not quite sure about this since I am not profession in cameras.
I will try to use your program to do the calibration tomorrow and see if it works well.
Again, I much appreciate your kind help.
Best Regards,
Weimin
Dear Weimin,
Thank you very much for your update. Your problem reminds me to a similar problem that I had with a High Resolution FireWire camera some years ago. The FireWire lost some data packages (due to the bad quality of the cable that I was using) and the high resolution images arrived with some corruption (the missing parts of the image were filled with parts of the previous image as the same memory chunk was used as a storage buffer, very "funny", it took me almost a month to find out the problem).
Anyway if the problem is "the slight variety and instabilities" as the tech support suggest, then you should be able to appreciate some "deformations" or "corruption" in the images with your bare eyes, otherwise the calibration algorithm should be flexible enough as to generate a valid calibration, even if the uncertainty is a bit higher than it should be.
Did you get any better result with OpenCV calibration method?
I am sorry I could not be more helpful.
Best regards,
Martin
Hi Martin:
Thank you very much again for your reply.
I have good news for you that by using your opencv program, the result I got was good enough. The error I got is around 0.6(compared with yours which is 0.4)
The images from the two cameras can be rectified almost perfectly (even though some slight unrectification still can be seen by bare eyes)
But now I am facing another problem. Because by using OpenCV's matching algorithm, the block-matching one, I can get a roughly good result of depth map of the object, but it's not that accurate, and I need to do facial animation by using this depth map of human face.
So is there any other algorithm that I can use which may get better result than cvFindStereoCorrespondenceBM()do? Is there any source code available now?
Best Regards,
Weimin
Dear Weimin,
I am very happy to hear that you achieved some progress :)
Regarding your question, cvFindStereoCorrespondenceBM() is based on [Konogige97] method. It works by using small sliding windows of "sum of absolute difference" (SAD) to find matching points between left and right images. This method works well and fast for points with high texture, for example an outdoor scene of a forest. But for low textured scenes (such as a face) very few points might register depth.
There is plenty of alternative algorithms for stereo matching, but they will usually be more computationally expensive. You could check the work of [Yang et al. 2009] which is a bit newer. Sorry have no source code for that yet, maybe you could find something in google :)
Anyway do you need the dense depth map of the human face for your facial animation? Isn't it enough to calculate depth information for a few points specific points (eyes, nose, lips...)?
Best regards,
Martin
Sorry, the correct spelling of "[Konogige97]" is [Konolige97] :p
Hi Martin,
I have a question for you! You wrote:
" Given the position an interest point in the left and right image, its 3D position can be calculated as follows: ..."
Is the position in the undistorted or in the original images?
thanks!
Hi Anonymous reader!
Thanks for your question. It is the position in the undistorted image :)
Best regards,
Martin
Hi Martin,
I'm past the stage of calibration for now, I used your code and it gave me a slew of xml files. I also made a program that finds the features points in consecutive frames, and calculates the z value for each of those points, but i assumed focus x baseline = 1000 for that(which gave me decent values for z but not that nice). Now that I've calibrated my stereo setup i need to find the actual f and b from the xml files(or the depth values directly as you suggested), but i keep running into some errors where it says that I cannot assign CvMat values (from Q) to float (X, Y, Z). I cant find a way to plug the values in (except opening up the xml and copy-pasting the numbers directly, which is quite dirty). Any ideas what could be wrong here?
Thanks
Ravish
Hi Ravish!
It depends on which programming langauge you use, but for example in C++ you can check CvMat::At or if you use C check CvMat->data->fl to access the data in the matrix Q.
I hope this helps.
Best regards,
Martin
Hey Martin,
Thanks on the tip, I'm using C, and after some googling, the simplest solution was to use cvmGet(M,i,j), which fixed it for me. Now I got my twin cameras calibrated, feature points detected, corresponding points matched and x, y & z values calculated for the 3d feature points (I'm still running into some exceptions but I'll figure 'em out!). Now I intend to find the rotation and translation matrices, somehow convert those into their ogre3d equivalents, invert them and directly apply them onto my 3d models. This will implement some form of markerless AR. So here are the questions:
1. I'm using cvPOSIT() for pose estimation and calculation of rotation and translation matrices, but I need to plug in the focal length of the cameras. How do I extract the focal length out of the generated calibration xml's?
2. Someone suggested that I should incline towards cvFindExtrinsicCameraParams2() instead of cvPOSIT(), for my purposes, which one should I choose(if you have any experience on the matter)?
3. Any suggestions on the way forward?
I know its asking a lot, but my deadline is just a week away, and I'm nothing but desperate.
Thanks
Ravish
hey martin..
my name is pram,im from indonesia.. i have a problem, how to calibrate the camera/ stereo camera with opencv1.1.. can you help me the step by step.. and what source code, i must to read? i'm sorry if my english is bad.. thanks a lot martin
Hey Ravish!
Sorry for taking so long to reply. I am really busy lately, actually I didn't have the time to post any new content since February xD
Anyway, as Jack the Ripper said: Let's go by parts.
1. If you want the focal length of the cameras you only have to look at the files M1.xml and M2.xml. They contain the intrinsic parameters of the cameras in this fashion:
Fx 0 Cx
M = 0 Fy Cy
0 0 1
Where Fx and Fy are the focal lenghts on the X and Y axis of the camera (the pixels are not actually square, that is why they are different and you don't have only one value for the focal length, but for some practical applications it could be safe to assume that Fx = Fy) and Cx, Cy are the displacement of the center of coordinates on the projection screen. There you go.
2. If someone recommended you to use cvFindExtrinsicCameraParams2() instead of cvPOSIT() for your purposes, he or she did the right recommendation. Let me quote a part of the book "Learning OpenCV":
"In some cases you will already have the intrinsic parameters of the camera and therefore need only to compute the location of the object(s) being viewed. This scenario clearly differs from the usual camera calibration, but is nonetheless a useful task to be able to perform."
Now look at the parameters for cvFindExtrinsicCameraParams2():
cvFindExtrinsicCameraParams2(
const CvMat* object_points,
const CvMat* image_points,
const CvMat* intrinsic_matrix,
const CvMat* distortion_coeffs,
CvMat* rotation_vector,
CvMat* translation_vector
);
The distortion coeffs are given in the files D1.xml and D2.xml, so you have everything you need! And this function does exactly what you want! Life is great, ain't it?
3. I could only suggest you to share the knowledge you acquired while doing this project with the rest of the world! We want videos in youtube and source code if possible ;)
I hope this helps, good luck with your deadline!
Best regards,
Martin.
Just to clarify... there should be some blank spaces when I talked about the M1.xml and M2.xml files.
The format of the matrix is like this:
M =
Fx 0 Cx
0 Fy Cy
0 0 1
@Pram
Maybe you could read the source code for the calibration procedure on the post, it should not be very much different for version 1.1 you can check the API for version 1 of OpenCV here
http://www.cs.indiana.edu/cgi-pub/oleykin/website/OpenCVHelp/
Sorry, due to lack of time I can not provide you a step by step explanation.
Best regards,
Martin.
ofcourse! I've got some of the initial code on github, but I never committed for a long while, but when its done, I'm deifinitely going to commit the entire thing on github and post some videos aswell.
Thanks man, you saved my ass more way than you think.
Hey Martin,
Sorry to bother you again, but problems never seem to cease. I tried using cvFindExtrinsicCameraParams2 but it accepts all const cvMats while my object_points and image_points are in cvPoints3d32f and cvPoints2d32f respectively. I tried converting my object points into a cvMat but then the compiler complained about it not being a const variable. After some frustration, I moved on back to using cvPOSIT after plugging the focal length into it, but I still get some aweful matrices with values that are not even related to the actual movement of the cameras. It would really help if you could just look at my code and suggest me a way forward. btw, there was a sample that showed how to convert the rotation and translation matrices into their equivalent ogl ones, but after a LOT of googling, I found no sample/example demo-ing the usage of cvFindExtrinsicCameraParams2.
Here it is: http://pastebin.com/mEcNAkw0
On a side note, do you know of any opencv IRC channel other than the freenode one?
Thanks
Ravish
Hei,
wassup?
I'm currently developing a 3D real-time tracking system, and I am now able to reconstruct in 3D the chessboard in many ways thanks to your hints!!
Anyway, now I am trying to solve the problem of the perspective error, and I have no clues of where to start. Is there any functions in opencv that can help me with this?
Here my plots in matlab of the chessboard:
using SVD (from hartley's):
http://tinypic.com/r/1zyj61d/7
using yours (from this article):
http://tinypic.com/r/30muzvs/7
@Ravish
Hi man! Sorry for taking so long to reply, crazy busy as usual :(
Regarding the issue with cvFindExtrinsicCameraParams2, you should really try to fix the compiler problem, maybe if you post the error and some code I could help you.
I took a look at your code, and it seems correct to me. The reason for your bad results could be the following:
To use POSIT you should provide at least 4 non-coplanar points on the surface of your object and it is assumed that the points on the object are all at effectively the same depth. That is, it is assumed that your object is far enough from the camera that you can neglect any internal depth differences within the object.
Are the points that you selected on your object non-coplanar?
Is the internal depth of the object small compared to the distance away from the camera?
If both assumptions are not true, then the algorithm will not work properly.
In my humble opinion you should try to fix the compiler complains with cvFindExtrinsicCameraParams2. It might work better for your application than POSIT.
Regarding the opencv IRC channel, sorry, I am not aware of any :(
I would suggest you http://tech.groups.yahoo.com/group/OpenCV/
I hope this could help.
Best regards,
Martin.
@Albatroz Jeremias
Hi! Thanks for tour comment and sorry for taking so long to reply.
I saw the pictures of the chessboard that you posted. You should be aware that it might be difficult to get more accurate results.
Keep in mind that the lens in the camera introduce some distortion. That distortion is supposed to be corrected by calibrating the camera, but the accuracy of your results will depend a great deal on the quality of your calibration.
Also, keep in mind that the corner detection is not 100% accurate, this will introduce more error in your measurements.
And also, keep in mind that the depth and disparity are inversely related, so fine-grain depth measurement is restricted to nearby objects. This means that you will have errors when trying to estimate the depth on objects far away from the camera.
So if you want to improve the accuracy of your measurements you should invest in having a good camera and good lens, get the best calibration you can and detect your interest points as accurately as possible.
I don't know if this answers your question. But anyway, if you want to work with perspective transformation you could have a look at the functions:
cvPerspectiveTransform()
cvWarpPerspective()
cvGetPerspectiveTransform()
I hope this could help you some way.
Best regards,
Margin
Hello Martin.
Firstly thanks for the valuable information. Currently I want to compute the depth for a sparse set of points obtained with Lucas-Kanade Pyramid tracker over the left/right stereo remapped images, I need to compute the coordinates related with the right camera coordinates system so in this case I would neglect the translation and obtain:
d = Pr.x - Pl.x;
X = Pr.x * Q[0,0];
Y = Pr.y * Q[1,1];
Z = Q[2,3];
W = d * Q[3,2];
X = X / W;
Y = Y / W;
Z = Z / W;
Is this approach correct? Could you give me some other links where I can read more about this topic?
Thank you in advance for any help, and thumbs up for your blog.
Greetings from Krakow - Poland.
Hi Martin,
Has anyone raised an issue regarding disparity maps? With the example pictures provided with OpenCV my code works.
The images appear to be rectified, using the chessboards, but the disparity map.... is frankly, terrible.
So this makes me think it's a cam issue. However all of the cvGetCaptureProperty() values are the same for each cam.
The cams are approximately 6inches apart and are of the same make and model. They are basic logitech webcams.
I have not precalibrated them as i believe this is dealt with in cvUndistortRecifiedMap(). Have you heard of this problem? Or experienced issues?
Dan
Apologies..... i HAVE used stereocalibrate THEN Unidistorectifymap. :-D
Dan
Hi Dan!
If you want to get better disparity maps I suggest you to use the function cvFindStereoCorrespondenceGC. As far as I remember the program described in this post uses cvFindStereoCorrespondenceBM to calculate the disparity map, it is fast but the disparity map you get is not that good.
I hope this helps.
Best regards,
Martin.
Thanks Martin. The GC does provide a bit a of a improved result, but not much. I'm certain it's my setup. Would it be ok to email you some files? Images of the output. Just it's driving me a bit mad. :-/
Dan
Hi Dan,
Sure, go ahead. You can send the files to the e-mail address that you can find at http://www.martinperis.com/
Best regards,
Martin.
hi martin..
i have succeded to calibrate camera with online webcam, but when my program run the rectified image, and their disparity,, it looks not good.. may you help me please? i'll send you my source, if you permitted.. sorrry if my english get bad..
thanks for your attention
pram - indonesia
Hi Pram!
No problem, send me your source (maybe you could use http://pastebin.com/ or send it by e-mail) I will try to take a look at it :)
If you could send some images it would help too.
Best regards,
Martin.
Hi Martin,
stereoCalibrate() or calibrateCamera() functions require 3D world coordinates of the corners. For a single view you can arbitrary set up world coordinate to be aligned with a chess board. But for any additional views you have to take into account rotation of the chessboard and supply physical x, y, z of the corners for joint optimization. How come you never do that?
Dear Vlad,
Take into account that for cvCalibrateCamera2 the 3D coordinates of the corners points are always located in the coordinate frame attached to the object. So the 3D coordinates required are not "World" coordinates but "object" coordinates.
If you provide the 3D object coordinates and the image coordinates of the chess board corners to cvCalibrateCamera2 then this function will take care of the rest.
I hope this answers your question.
Best regards,
Martin.
Hi Martin,
I think I found a bug in corner extraction function. Sometimes the order of corners is opposite to what you expect, see the image here:
https://docs.google.com/leaf?id=0B3XyhwMHiqMlYzI4NzcxNzctNWZlMS00YjczLWE2YjAtOGQ1Y2FhNDVkNjM3&hl=en_US
It doesn't affect calibration but does affect rectification. Have you encountered something like this? To fix this I have to do the following:
if (cornersR[0].y < cornersR[Pattern_total-1].y)
image_pointsR[i].push_back(Point2f(cornersR[k]));
else
image_pointsR[i].push_back(Point2f(cornersR[Pattern_total-k-1]));
Best,
iVlad
Hi Vlad,
It only happened to me when using a square chessboard (same number of rows and columns) but never using a non-square chessboard like in the images that you posted.
Very strange indeed.
Best regards,
Martin
Hi
I am working on V-SLAM and just need some help as far as depth estimation using multiple cameras is concerned.Any good reference to get the basic idea of doing cameras calibration.
Hi there!
I would recommend you the book "Learning OpenCV"
(http://www.amazon.com/Learning-OpenCV-Computer-Vision-Library/dp/0596516134/ref=sr_1_1?ie=UTF8&qid=1313806213&sr=8-1)
there is a complete chapter about stereo camera calibration.
If you are working with more than 2 cameras you can check out the book "Multiple View Geometry in Computer Vision" (http://www.amazon.com/Multiple-View-Geometry-Computer-Vision/dp/0521540518/ref=sr_1_1?s=books&ie=UTF8&qid=1313806337&sr=1-1)
I hope this helps :)
Best regards,
Martin
Hi Martin,
Thank you for your post on stereo calibration.
I am new to OpenCV and stereo vision.I may be stupid in some of my question so please bare with me.Hope you can understand.
We made our custom stereo camera.We are planning to use it for depth calculation.
From your post what i understood is for depth calculation we first need to do calibration of stereo camera.
So my first step is calibrating the cameras by using chessboard image samples.Once the calibration is done we will have some parameters from the calibration.
My question is what i have do with these parameters? should i need to use it during depth calculation?
Should we need to calibrate our camera setup each time for different environment before calculating the depth?
Your code looks to take sample images from file?can we change this to take live stream from our stereo cameras?
If we are using live images from stereo cameras capturing chessboards,when or how we can finalize that calibration is complete?
Regards,
Prabu
Hi Prabu!
Thank you for your comment. We all have been new to OpenCV before, so don't worry, your questions are not stupid at all :)
>>>>>My question is what i have do with these parameters? should i need to use it during depth calculation?
The parameters that you calculate during calibration are used for two things:
1 - Undistort and rectify the images (check out cvRemap)
2 - Calculate the 3D position (relative to the camera) of any point that appears in the left and right image
>>>>>>Should we need to calibrate our camera setup each time for different environment before calculating the depth?
No, you only need to calibrate the camera once. You would only need to re-calibrate if the cameras move one respect the other (In the case of custom stereo cameras, it will be likely that the cameras become lose and they move away from their calibrated position, so you will need to re-calibrate)
>>>>>>Your code looks to take sample images from file?can we change this to take live stream from our stereo cameras?
Absolutely, you can do anything you want. If I remember correctly, you can take the code from OpenCV examples or from the code examples of the book "Learning OpenCV")
>>>>>>If we are using live images from stereo cameras capturing chessboards,when or how we can finalize that calibration is complete?
You should search each image for a valid chessboard, if a valid chessboard is found store the image. Repeat the search until you have enough valid chessboards (20? 30? 40? it is up to you)
Once you collected enough valid chessboard images then call the calibration function.
I am sure that there is already code out there that does what you need. Sorry, I don't remember exactly where... it should be on the OpenCV samples or the samples of the book that I mentioned before.
I hope this helps.
Best regards,
Martin
Hi Martin,
My name is Mike from Canada. I have been using your sample code with great success but am having one problem with it. When I extract the real XYZ coordinates from an image my accuracy drops drastically for points that are greater than approximately 16" from the cameras, do you have any idea why this could be?
I have tried using different sets of webcams with different levels of resolution and this error is still present.
Thanks for you help
Hi Mike!
Thanks for your comment. Regarding your question, well, you should take into account that depth is inversely proportional to disparity (see this figure taken from the book "Learning OpenCV" p.417: http://www.martinperis.com/images/distance_vs_disparity.jpg). So, you can expect drops of accuracy on distant objects.
But the fact that it is only 16" and happens with different levels of resolution intrigues me. Probably it is a calibration problem. When you calibrated the cameras using the chessboard pattern, did you took images from more than 16" away? What is the disparity value for objects at more than 16" away from the cameras? If this value is around 0 then the previous figure would explain your problem. Otherwise, try re-calibrating the cameras using a wider range of distances between the cameras and the chessboard.
I hope this could help you.
Best regards to Canada, haven't been there yet but I bet it is a really nice place.
Martin.
Hi Martin,
Sorry for being a total noob (I've never used linux before) but when I try to use the "make" command in the stereo_camera_calibrate folder. It returns the following error:
g++ -L. -L./root/lib -o stereo_calibrate stereo_calibrate.o -lcv -lhighgui
/usr/bin/ld: cannot find -lcv
/usr/bin/ld: cannot find -lhighgui
collect2: ld returned 1 exit status
make: *** [stereo_calibrate] Error 1
What should I do? Do I need to find the -lcv or -lhighgui? Your help is much appreciated. THanks!
Flynn
Hi Flynn,
Sorry about that, the Makefile on that software is more complex than it should, and has some hard-coded dependencies. I have fixed it. You can download the software again (from here: http://www.martinperis.com/stereocalibration/OpenCV-Stereo-Calibration.tgz) and execute "make" again.
Or if you don't want to download it, this command should work:
g++ -o stereo_calibrate stereo_calibrate.cpp `pkg-config --cflags --libs opencv`
Please, let me know if it works for you, otherwise we will try to find out what is the problem :)
Best regards,
Martin.
Hi Martin,
I had a quick question about the reprojection-error returned by cvStereoCalibrate(). What are the units of this error? If the square-size that I use for my chessboard is 5mm and the reprojection-error is 1.48. Is the 1.48 in mm or pixels?
Thank you,
Ankur
Hi Ankur,
Since the reprojection-error is defined as the total sum of squared distances between feature points and the projected object points, the units should be pixels^2
Best regards,
Martin
Hi Martin,
I am a newbie in OpenCV :D
The equations for calculating 3D position. can we use it to calculate distance between object of interest with our camera? or maybe there is another method to do so ?
Thank you
Best regards,
Moris
Hi Martin,
Great work! And thanks for sharing this knowledge with public. Thanks for contributing to the community.
I've a question. I've six cameras say numbered as 1-6. If I use your method and find 3d points with camera no. 4 and 5 how do i rotate and translate to camera no. 1 on which the all the 3d points will be based upon?
Again thank you for lending a helping hand. Keep up the good work.
-Tom
@Moris
Hi there! Yes, the equations for calculating 3D position can be used to find out the distance between your point of interest and your camera. Actually this distance is the Z component of the equations.
I hope this helps.
@Tom
Hi Tom! Thank you very much for your kind comment. Regarding your question. If you find the 3D points using camera 4 and 5 (I assume that camera 4 is left and camera 5 is right) then the 3D coordinates are in the frame of coordinates of camera 4.
You need to calculate the rotation and translation between camera 1 and 4. If camera 1 and 4 have overlapping views then there is a simple solution:
Run the calibration again using camera 1 and 4, if you look at the source code in this post you will see that there is a call to a function named cvStereoCalibrate. Among the output parameters of this function there is two matrices "_R" and "_T". These matrices will contain the Rotation and Translation between camera 1 and camera 4.
You can store those matrices and use them to get your 3D points in camera 1 coordinate frame.
But if the cameras has no overlapping views then you will have to measure the rotation and translation manually somehow :S
I hope this could help.
Best regards,
Martin.
Hi Martin,
Thank you Martin for answering my question.
I see what you're saying. I just got an idea(that I need assurance of) and not sure if it would do the same trick.
Let's say I feed the 2d coordinates of the center of a glowing ball(in a area) on a wand instead of chessboard corners for calibration that is seen by all the cameras. Then find intrinsic and extrinsic parameters and use your method to find 3d coordinates with respect to camera 4. And after that multiply the 3d coordinates with rotation and translation matrices of camera 1.
After this process will the 3d coordinates be found with respect to camera 1's coordinate system?
I want to avoid the chessboard corners as you can see it's not efficient for my case('cause I've 6 cameras) so use a wand instead.
Can I find the 3d coordinates with a wand using the process just explained?
--Best regards
Tom
Hi Tom,
I have another question if you don't mind ..
Here we can calculate the distance between real object and our camera from Z/W, where W=d*Q[3,2]+Q[3,3] right..
can we compute d = pointRightImage.X - pointLeftImage.X by use of matrix we got during our stereo calibration process? or we need another way to do so?
Thank you for sharing your knowledge.
Best regards,
moris
@Tom
Hi there!
Sorry for my late response, I don't know if I understood you correctly, but do you mean that you want to use a glowing ball instead of a chessboard as a calibration object?
If it is so, then I am afraid that this approach would'n work, because (if I am not mistaken) your calibration object should have at least 5 points in order for the calibration algorithm to work.
Maybe you could build a wand with several glowing balls(?). Any calibration object will work as long as it has a minimum number of points and you know the exact 3D position of all points (relative to the calibration object's coordinate frame). In that case, the process that you explained should work.
I hope this could help you.
@Moris
Hi Moris, to compute "d" (this value is called "disparity") you need to identify a point(for example a corner on a chessboard) in the left image (lets call it pointLeftImage) and then find the corresponding point on the right image (the exact same chessboard corner on the right image. And lets call it pointRightImage).
So, to find "d" you need to subtract the X coordinate of pointRightImage and the X coordinate of pointLeftImage
Is this what you are asking?
Best regards,
Martin.
Hi Martin,
I was able to make and run the program after you modified the uploaded file. Thank you very much!
Currently I would really appreciate your expert opinion on another issue. Our lab is studying video clips of the upper extremety to extract wrist biomechanical data. In second last part of your example you used:
cvRemap(imgLeftOrig, imgLeftUndistorted, mx1, my1);
to correct for lens distortions. I believe the cvRemap function works with images. Currently in our lab we have another software it will track points of interest (bright reflective markers) on a video clip and output 2D coordinates frame by frame (the default origin is the upper left cornor of every frame). We are using two synchronized cameras so for any given marker there is an x_left, y_left, x_right and y_right (recorded at 60 Hz for both cameras). These coordinates apprently are not corrected for lens distortion or perspective distortion. My question is: instead of using cvRemap, which transforms pictures by using your m*.xml files, is there any function to use so we can directly work with the numerical coordinates we got from our lab's software (possibly still using the m*.xml files)? This way we can integrate openCV calibrations with our own software's output.
Any suggestions from you is more than welcome. Thank you!
Flynn
Hi Flynn!
I am glad to hear that you could successfully make and run the program.
Regarding your question, the thing is quite simple actually. The m*.xml files are nothing but matrices of the same size as the images that you used to calibrate the cameras and contain the mapping of each point in the original image to the undistorted image.
So, given x_left, y_left, x_right and y_right you can get the undistoreted coordinates as follow:
undistorted_x_left = mx1[x_left,y_left]
undistorted_y_left = my1[x_left,y_left]
undistorted_x_right = mx2[x_right, y_right]
undistorted_y_right = my2[x_right, y_right]
You can find more information about how cvRemap works here: http://opencv.willowgarage.com/documentation/c/geometric_image_transformations.html#remap
I hope this could be useful. Good luck with your experiments :)
Best regards,
Martin.
hi Martin,
good post!
I dont have a stereo camera yet, so I'm looking for a set of chessboard images taken from a stereo camera, for running through openCV and finding the depth values.
I plan to get a rough estimation of depth this way, show the results to my professor and convince him to buy a stereo camera.
any idea where I might find such a database?
many thanks!
Hi Samarth!
Thanks for your comment. You can use the set of images that comes with the software in this same post (the ones with me swinging around a chessboard in Hawaiian shorts xD) or you can get the sample chessboard images that come with the OpenCV package.
In this website you can find more stereo scenes http://vision.middlebury.edu/stereo/data/ they are not chessboards, though.
I hope this helps.
Best regards,
Martin.
>Martin said:
>Maybe you could build a wand with several glowing >balls(?). Any calibration object will work as long as >it has a minimum number of points and you know the >exact 3D position of all points (relative to the >calibration object's coordinate frame). In that case, >the process that you explained should work.
Hi Martin,
Sorry for late reply. Thanks for your response. Say I have a wand with two reflective tennis ball glued at opposite ends of that wand. And the distance between the balls is 2 feet. Knowing this information is there any way to figure out the 3d coordinates of this moving wand in all 6 cameras before using cvStereoCalibrate() function?
--Best Regards
Tom
Hi Martin,
It's me again. I've an idea regarding camera calibration.
Suppose I have made a huge cube with chessboard pattern printed on the all 4 sides of the cube and I place them at the center of all 6 cameras.
And find chessboard corners using your method. Does it really matter which side of the cube's chessboard pattern any two cameras detect during
calibration?
What I mean to say is i want to reconstruct all 3d coordinates relative to camera1. Say a 3d point is detected and it's coordinates are calculated
between camera3 and camera4. I want to transfer this point with respect to camera1 by multiplying 3d coordinates by extrinsic parameter of camera1.
Do camera1 and camera3 need to be calibrated looking at the same side of the large cube I created?
Does it really matter what side of the cube is facing camera1 and camera3 during stereo cablibration if the detected chessboard pattern is same?
Best regards
-Tom
Hi Tom,
The problem with the cube idea that you propose is that probably, at calibration time, one camera will be able to see two sides (two chessboards) of the cube at the same time. And that will complicate your calibration.
From your description I guess that the 6 cameras that you are using are distributed in circle. If this is the case, why don't you try the following?
Perform 5 different calibrations with a regular planar chessboard. Like this:
Calibration1: using camera1 and camera2
Calibration2: using camera2 and camera3
Calibration3: using camera3 and camera4
Calibration4: using camera4 and camera5
Calibration5: using camera5 and camera6
Having all these calibrations you can calculate the transformation from any camera to camera1, for example if you calculate a 3D point using camera3 and camera4 (using the parameters you got from Calibration3), that means that the 3D point will be in the coordinate frame of camera3 and you can transpose it to the coordinate frame of camera1 using the parameters from Calibration2 and Calibration1 (from camera3 to camera2 using Calibration2 and from camera2 to camera1 using Calibration1).
Of course you can pre-calculate the transpose from any camera to camera1 so that you won't have to do unnecessary operations when transforming points from one coordinate frame to another coordinate frame.
Maybe there will be accumulation of errors when transforming from camera 5 to camera1, but it is the simplest way of getting it working that I can think about right now.
Best regards,
Martin.
Hi Martin,
Hmmmm That's a great idea. Thank you for answering all my questions. Sorry to bug you so many times. I'll keep you posted what happens. Again thanks a lot.
Bye!
--Tom
Hi Martin,
I have to ask this question. Sorry to bother you again. May be it's a stupid question but I can't help it.
Say there are three wooden sticks. Three sticks are separated horizontally exactly by 1 feet and each stick is of height 6 feet. Now on each stick there are reflective tape coated six tennis balls(I use white reflective tape so that I can detect and separate them from other objects in the scene). Each ball is on
each stick is separated vertically by exactly 1 feet.
Now my 6 cameras are mounted on a circular area with radius of 7 feet and let's suppose camera1 is located at 0 degree, camera2 at 60 degree, camera 3 at 120 degree,
camera4 at 180 degree, camera 5 at 240 degree and camera6 at 300 degree of the circular area. So camera1 and camera4 are located at opposite sides.
Now my question is: The model of 3 wooden sticks is placed at the center and I want to calibrate camera1 and camera4 using cvStereoCalibrate(). But these cameras are viewing the same tennis balls but seeing them from opposite sides. Can I supply the same 3d object points for camera1 and camera4's corresponding detected 2d coordinates of tennis balls like this from bottom to top:
lowest horizontal tennis balls as (0,0,7), (1,0,7), (2,0,7)
then 2nd lowest horizontal tennis balls as (0,1,7), (1,1,7), (2,1,7)
the 3rd lowest horizontal tennis balls as (0,2,7), (1,2,7), (3,2,7) and so forth
even when the model of three sticks are slanted horizontally toward camera1 or camera4?
Does it matter if I supply the same above 3d points shown above for the corresponding 2d points in camera1 and camera4 as long as they are corresponding and lowest to highest?
I am asking this because camera1 and camera4 are seeing the same thing from two different views. Just want to make sure I'm doing things right.
--Best regards
Tom
Hi Martin,
You gave such an informative article. Good Job Martin:-)
I'll explain steps that I performed in calculating distance of object.
1. Calibrated stereo camera with chessboard 8x6 cm with 2.8cm square size.
2. My calibration was success with rms error 0.206 and rectified images was good.
3. I found my interest point on left-right image. I tried to determine distance of object using method you specified. There is an error in distance measured from 3cm for closer object to 12cm or more for distant object.
Say, actual - 27cm ; measured - 24cm.
actual - 60cm ; measured - 48cm.
My queries are,
- why does there comes this much big varition is distance measurement?
- What may be reason/solution for this error?
Is there any mistake in my procedure or do i miss parameters?
Regards
Sathya Kumar
Hi Tom!
Sorry for taking so long to reply :S
Your question is quite interesting, actually I have never tried to calibrate a stereo pair of cameras that are pointing at each other.
In any case, you should provide the 3D object points in the right order as seen in each camera. If the cameras are pointing at each other, yes the 3D points will be the same... but maybe in inverse order? I don't know I didn't thought that through.
BUT, mi stereo-sense (something like the spider-sense for Spider-Man :P) tells me that something weird will happen... you see... if the cameras are pointing at each other, the epipolar lines are not "lines" any more. Instead of appearing as lines on both images, they will appear as points. And I have no idea if the calibration procedure will be robust against that...
The best thing you can do... is just try :D
Good luck!
Martin
@Sathya
Hi there! Thanks for your comment :) Allow me to answer your question with a post:
http://blog.martinperis.com/2011/10/opencv-stereo-3d-reconstruction-error.html
Best regards
Hello Martin,
It's Tom again. Thanks for your comment. You're right. If the detected 2d coordinate coincides with epipole and camera1 and camera2 are located at opposite side of an area then the epipolar line will be a point. And that will be a problem.
I also agree with what you suggested earlier. You said that:
---------Martin said:------------------------
Perform 5 different calibrations with a regular planar chessboard. Like this:
Calibration1: using camera1 and camera2
Calibration2: using camera2 and camera3
Calibration3: using camera3 and camera4
Calibration4: using camera4 and camera5
Calibration5: using camera5 and camera6
Having all these calibrations you can calculate the transformation from any camera to camera1, for example if you calculate a 3D point using camera3 and camera4 (using the parameters you got from Calibration3), that means that the 3D point will be in the coordinate frame of camera3 and you can transpose it to the coordinate frame of camera1 using the parameters from Calibration2 and Calibration1 (from camera3 to camera2 using Calibration2 and from camera2 to camera1 using Calibration1).
-------Martin said ends----------------------------
But I'm worried about the margin of error in transferring 3d points say from camera5 to camera1 because all the image points will be relative to camera1.
Thanks for your help. I really appreciate it. I think I'll find a way. Again thanks.
--Best regards
Tom
Hello Martin,
It's Tom again. You're right. If the detected 2d coordinate aligns with epipoles(epipolar point) of two cameras then there will be a problem.
But suppose you've two cameras which are not located at opposite end of a region but side by side and call them camera1 and camera2. Doesn't there will be a point in camera1 that lies in the epipoles of camera1 and camera2? So why this is not a problem for this case?
Are you sure if cameras are mounted at opposite side to each other there'll be a reconstruction error?
--Best regards
Tom
Hello Martin,
Great article.Anyway, I found out the latest OpenCV version already has a function called cvReprojectImageTo3D() that returns a 3D vector with the coordinates on the disparity map.
Best regards,
Frederico
@Frederico
Thanks for your comment and the observation. Yes you are right, cvReprojectImageTo3D() has been out there for a while but in this post I didn't use it as it reprojects the whole image to 3D and I was interested in explain how to reproject a single point to 3D (I could have used cvPerspectiveTransform for that too).
Best regards.
@Tom
Hi Tom! Sorry for taking some time to reply. I have been thinking about your problem, and I realized that what I told you in my previous reply was not completely accurate. I said that if your cameras are pointing at each other then the epipolar lines would become points instead of lines... well that only happens for the epipolar line that connects the centers of projection of each camera.
The rest of the epipolar lines will intersect in the center of the images (they will be distributed radially around the center of the image). The problem with that is that the calibration algorithm will try to transform the image so the epipolar lines become parallel, thus the result of the rectification procedure will heavily distort your original images.
Anyway, I could be wrong (there is a big chance that I'm wrong :P), I guess that the only way to make sure is just trying it :)
Best regards,
Martin.
Hi Martin,
That is all I needed to know. Thanks for answering all my questions. I'll let you know what happens next. Again thanks for everything.
Bye for now.
--Best Regards
Tom
Hi Martin,
Thanks for the great post.
Which version of OpenCV should your code compile against?
Thanks,
Mike
Hi Martin
- I did the calibration and got the matrices. I'm wondering about the 3D position computation, you gave the equation to calculate the X,Y,Z as follows :
d = pointRightImage.X - pointLeftImage.X;
X = pointLeftImage.X * Q[0, 0] + Q[0, 3];
Y = pointLeftImage.Y * Q[1, 1] + Q[1, 3];
Z = Q[2, 3];
W = d * Q[3, 2] + Q[3, 3];
X = X / W;
Y = Y / W;
Z = Z / W;
My question is:
how we get pointRightImage.X and pointLeftImage.X , from which matrix of the matrices we extract them?
Hi Martin
- In this function, what's the minimum number of image pairs are advised to get better results. I used 18 images(9pairs of left and right images ) and I got all the required matrices , are these sufficient to get accurate 3d?
@mike
Thanks a lot for your comment! Currently I use the default version of OpenCV that comes with Ubuntu 10.04. If I am not mistaken that is OpenCV 2.0.0
@Heider
Hi! What you have to do is select a point on the left image and find its corresponding point on the right image. That is called "stereo matching". Once you have the coordinates of the matching points on the left and right image then you can calculate "d" as in the formulas that you mentioned.
"d" is called stereo disparity, and there is already functions in OpenCV than can calculate that. You can find more information about that in this other post: http://blog.martinperis.com/2011/08/opencv-stereo-matching.html
Regarding the number of image pairs that you should use for calibration, you should use at least 5. But the more image pairs (with the chessboard in different positions and inclinations) the more accurate calibration results you will get.
I hope this could help!
Best regards
Hi Martin
I did the camera calibration, I got the required matrices, but for focal length I got a strange number from M1 matrix, the obtained focal length is: 3.4985766391386223e+003 ! The used camera calibration focal length 16mm ! as I remember. Is there something wrong? Or this number has to be interpreted in another way?
Hi Martin,
Thanks for your previous answer.
Will it be possible for you to post the stereo configuration you've used (i.e. which cameras and what was was the distance between the cameras).
Thanks,
Mike
Hi Martin,
Excellent and useful blog! I am working on camera calibration using the OpenCV 2.2 binding for python (not the ctypes or pyopencv versions). There are only two examples of this online and both were a bit buggy and incomplete. My new code works fine, and I can read in image files and get plausible calibration matrices. It also produces "undistorted" images from my wide-angle lens images. The problem is that the undistorted images are wildly off along the edges. The central part of the image itself looks pretty good. I've played with all the calibration flags already; it's not them. Do you have any sample calibration images to share so we blog readers can test out our code (to be sure it's our images and not the code itself?) I've also read that you need to collect balanced images with the checkerboard equally in all sectors of the image. Any feedback or help would be very welcome! (I'll be posting my code and my own set of sample images once this project is done!)
Thanks,
Suzanne
Hi Martin,
Quick update: I finally found Bouguet's Matlab toolbox and its set of sample images. The hunch that the problem was the balancing of where the images were appears to be correct, though it showed up in a strange way. Using the code that finds my own chessboard corners from actual video just fine, I couldn't detect more than 1/4 of Bouguet's images. This is just the usual Learning OpenCV checkerboard corners finding code. When that happens, indeed the edges are distorted, just as in my own experience. I'm going to try to get some more "balanced" images of my own later, but now I'm confused about why Bouguet's very clear images don't get located. And other readers might wonder too, since it's good to have a set to learn on. They are located at:
http://www.vision.caltech.edu/bouguetj/calib_doc/htmls/example.html
Suzanne
Hi All!
Sorry for my delay answering your questions:
@Anonymous - I am not sure right now, but that value might be in pixels. Anyway it is normal that the calculated focal length differs from the local length in the specifications of the camera. Have you tested the accuracy of your calibration?
@Mike - I used a Videre Design stereo camera, if I recall correctly the baseline (separation between cameras) was around 10cm.
@Suzanne - Thanks for your kind comment. Regarding your first question, if you download the source code on this same post (big blue button that says "Download Code") you will find a folder with the image set that I used as calibration example. In my experience, the distortion that wide-angle lenses introduce in the images is quite large, so don't expect a perfect rectification, but as you say, trying to get images more "balance" will help.
Regarding the question of why your code couldn't detect more than 1/4 of the images in the dataset that you posted, it happens the same to me, looks like there is not enough resolution for so many corners, maybe playing with the corner detector parameters could improve something. Anyway, the image set that you can download from this post should work just fine.
I hope this helps.
Best regards,
Martin
Hi Martin,
Thanks for the helpful comments. I downloaded your images and was quickly able to use them to verify that my python code works. Also, my distortion matrix values are close to those quoted for the examples at Vladimir Vezhnevets's page at:
http://www.graphicon.ru/oldgr/en/research/calibration/opencv.html
I'll be posting the python code soon on my website links page, since there doesn't seem to be a working version of the camera calibration available for python + OpenCV. Onto trying out the OpenCV stereo calibration code...
Suzanne
hi martin , first of all i want to thanks for your really good effort. its very informative to newcomer in the field of computer vision , using opencv.
i am also doing work on stereo vision. i am using Axis ip cameras. on ubuntu.i have done the stereo-calibration.
what i did , i use the black circle paste on the board and got the Hough circles in both left and right images and find the center coordinates in both left and right image.with the values of xleft and xright i calculate the disparity d.and find the Z using formula
Z=f*(distance between cameras)/disparity.
up to certain distance it give almost accurate depth but beyond this gives nonlinear error.
i m not using undistorted and rectify images .should i use the undistorted rectify images for finding center of circles.
with regards
pankaj upadhyay
Hi Martin and readers,
One more question on units now that I'm able to correct my images nicely. I understand the units X,Y,Z and how they depend on properly inputing object_points and have read the OpenCV books. What's confusing me is the effective units of the distortion coefficients k1,k2, etc. These seem to come up on the order of k1= +/-0.1 or higher for the corrections I have and for other example image sets taken off the web. However, in the equation in the book, this large a value will lead to absolutely huge correction terms, since the corrections are all multiplied by r^2, if x, y and hence radius r really are in pixels and k1 is in units of (1/pixels)^2. It would appear that x,y and r must instead be in homogeneous coordinates, hence so must k1, etc. but how is that possible? Learning OpenCV is dodgy on this topic. On pages 373-374, they say that q = (x y w) = (q1,q2,q3 ) has "proportional values" and requires dividing through by q3 before you get real pixel coordinates (bottom of 373). They then point out that q3 = w = Z on the next page, but Z varies from image to image and depends on real world coordinates--so how can the units of k1 depend on real world coordinates? The distortion coefficients must be independent of the calibration images used! On pages 391-392 and 376, it sounds like x,y and r must be in pixels when we apply k1, etc. for undistortion corrections, though. If k1 = 0.1, then we get simply colossal correction factors at the image edges where x, y and z are hundreds of pixels! (I've done this fixing k2=k3=0 and verified these values of k1 work just fine for undistorting the images! So it's not being offset by those other terms.)
Anyone out there figured this one out? I care because I have older values for pixels locations that I'd like to undistort outside of OpenCV, so I need k1 in proper units of 1/pixel^2 to do this!
Suzanne
Hi Pankaj!
Thank you very much for your kind comment. First of all, you should keep in mind that the relation between disparity and depth is non-linear (more information here: (http://blog.martinperis.com/2011/10/opencv-stereo-3d-reconstruction-error.html). Also, to get results as accurate as possible you should use the undistorted and rectified images (more information here: http://blog.martinperis.com/2011/08/stereo-vision-why-camera-calibration-is.html)
I hope this could help you.
Best regards,
Martin.
Dear Suzanne,
I like your challenging questions :) Actually, I never had the need to consider the effective units of distortion coefficients so I never explored that area. Sorry I didn't have the time to check deeply all the math, but if I am not mistaken (and as usual it is very likely that I am :P), k1 is not depending on real world coordinates but on calibration-object-coordinates. I don't know if this could shed any light into your problem.
Maybe I misunderstood you, but you are saying that you need k1 to undistort older values for pixel locations. I assume that those values for pixel locations where acquired using the same camera and same lens as now, so how about using cvUndistortPoints to undistort them?
Oh! when you upload your code please let me know, I would like to add a link in this post :)
Best regards,
Martin
I'm using Ubuntu Linux as well, having problems just displaying my camera capture. I'll get a grey window. Which cameras do you use? Cheese is capturing the camera with no further problems. In other system there no problems with the same cam.
Andi
Hi Martin,
I have posted my python adaptation of the OpenCV camera calibration code (just for undistortion-not the stereo code so far) here:
http://www.haverford.edu/physics-astro/Amador/links/
Also, I think I figured out the issue of how to think about the camera undistortion units, thanks to a posting at this website:
http://imagej.588099.n2.nabble.com/RE-LENS-DISTORTION-CORRECTION-td631005.html
See second posting at this website.
The idea is that you wish to define these values of k1, k2, etc. to be independent even of the pixel #, so he shows the way you rescale the distortion parameters to guarantee this. Makes sense with my own values too.
In answer to your very sensible suggestion about why not just use cvUndistortPoints to undistort old values: these values now live in a highly processed spreadsheet, not in image files. So it makes more sense to embed my undistortion in my analysis code for this older data, though now I'm streaming new data through the calibration correction code posted here.
Regards,
Suzanne
Hi Martin,
very good and useful job!
Long time I have a problem with calibration and 3D reconstruction and I hope you could help me?
Here are my images http://mystereo.xobor.net/g1-Burgi.html, i use this cameras http://www.stemmer-imaging.co.uk/en/products/article/15213 with folowing optic http://www.sourcesecurity.com/technical-details/cctv/image-capture/lenses/tamron-12vm412asir.html. I have test the pictures with my, your and OpenCV sample code but my disparity images and 3d reconstruction are always bad.
What could be the possible problem?
Thanks!
Burgi
Hi Martin,
I'm not able to find one mystery in OpenCV stereo calibration.
Under same lighting condition, undisturbed camera position, plane as well as orientation, my calibration results (data obtained from calibration) varies a lot.
Say, I take bunch of chessboard images and perform stereo calibration. I got some focal length(f), optic centre(Cl, Cr) of L/R camera. Even rectification is good.
Next, I'm repeating process of calibration by taking another set of chessboard images. Now calibration data gets deviated lot. And rectification also gets worst.
Of-course I followed almost same positioning, tilt, distance and inclination of chessboard.
I've read that recalibration has to be done for following conditions.
1. When lens position is varied. (mine is fixed lens camera).
2. When there is heavy mechanical disturbance to stereo jig.
Why is this such unexpected variation in calibration result? And why i'm not getting same or atleast closer calibration results for each time.
Is there any specific sequence to be followed to take chessboard images?
Please help to solve these queries Martin.
Regards,
Sathya Kumar
@Burgi,
Thanks a lot for your comment :) I took a look at your pictures, they look pretty good to me, it looks like your optic is not introducing too much distortion, so probably your calibration is not bad. But I have noticed something, it looks like there is a slight difference of illumination between the images of the left and right camera, maybe that is causing problems to find the stereo matches. Have you tried playing with the parameters of the stereo matching algorithm? You can find more information about it here: http://blog.martinperis.com/2011/08/opencv-stereo-matching.html
I hope this could be useful for you.
@Sathya
In principle there is not any specific sequence to be followed to take chessboard images. There could be several reasons for getting very different calibration results with images that at first sight look very similar. You should be aware that if your place the calibration pattern completely parallel to the camera then you are not introducing so much information about the depth of the scene so the approximation of the focal length becomes a bit unstable. The only thing I could recommend you is that you take as many images of the calibration chessboard in as many different positions, orientations, tilting, inclinations and distances as possible. Also the calibration chessboard should not be square (for example 6x6 squares) should be uneven (for example 9x6 squares).
I hope this could help you.
Best regards,
Martin
Hi Martin,
I did the calibration and got the depth z, how can I draw the 3D image of the chessboard after I got the depth z? Is there a code in openCV, MATLAB, or other software?
Hi Anonymous reader!
Thanks for your comment, if you need to draw the 3D image that you get with your stereo camera set, you can use Point Cloud Library (http://www.pointclouds.org/). You will need to fill a point cloud with the 3D coordinates of each pixel in your image and use the library's functions to draw it.
Soon I will post an example on how to do it.
Stay tuned!
Martin
I am unable to download the source code.
Thanks.
Hi Anonymous,
I am really sorry, the bandwidth of my hosting service has reached its monthly limit. The downloads will be available again from 1st of next month.
If you would like to help me avoid this in the future, please read this http://blog.martinperis.com/2011/12/please-help-me-help-you.html
Thank you.
Hi, thank you for your hard work.
I have question about the (x,y,z) cordinates i get from your calculation for finding the depth:
How the origin of axes is determined?
In my results, the Y origin is in the middle of the image in contrast to the X and Z origin.
Another thing, my X origin is from left to right unlike your example.
Thanks again,
Barak.
Hi Barak,
Thanks for your comment :) According to the documentation the camera coordinates are relative to the left camera's center of projection (which is, ideally, aligned with the center of the image plane). With +x pointing to the right side of the image plane, +y pointing to the bottom side of the image plane and +z pointing outside the image plane.
For more details, please, check the book "Learning OpenCV: Computer Vision with the OpenCV Library", Chapter 12, Figure 12-6.
I hope this could be useful.
Best regards,
Martin.
Hi,
I have several questions about the calibration process:
1. If i change one pair of images in other pair i get much different Q matrix is this normal?
2. What should be acceptable avg error? i get 1 cm.
Thanks
Hi,
Depending on the number and positions/orientations of your images it is possible that changing one pair for another pair of images affects highly to the values of the Q matrix. If you want to get more stable results you should use as much images as possible in as much as possible positions/orientations/distances respect to the camera.
In my experience 1cm is an acceptable avg error.
Best regards,
Martin.
Hi Martin! It's Tom again. My question is:
Suppose I have four cameras. And I stereo calibrated cam1 and cam2 using your method. Again done the same thing with cam2 and cam3. Also for cam3 and cam4. Now I use your method to find 3d coordinates of a point not displayed in cam1 and cam2 using the info of stereo calibration of cam3 and cam4(as you demonstrated in this blog).
Now you can see that this 3d coordinate is respect to cam3 and cam4's world coordinate system.
How do I transform this 3d coordinate with respect to cam1 and cam2 because I want all the 3d coordinates with respect to cam1 and cam2's world coordinate system?
Also the 3d coordinates that you got in this blog, is that in world coordinate or camera coordinate?
How do I convert 3d coordinates captured in cam3 and cam4 to be in relative to cam1 and cam2(according to cam1 and cam2's world coorinate system)?
Thanks a lot for your help.
--Best regards
Tom
Hi Martin,
It's Tom.
Okay I think I found a solution. Is it possible to kindly confirm if I'm right or wrong?
Suppose there are 4 cameras. Now camera3 and camera4 see a point say [x1, y1, z1] in world coordinate.
So if i transform this point's coordinate with respect to camera1 is the procedure something like this?
-------------------------------------------------------------------------------------------------
Step1: Find the camera coordinate of Transpose([x1, y1, z1]) which is [P1] with respect to camera3 using the calibration of cam2 and cam3
[ P1 ] = Transpose([u1, v1, w1]) = ( [ R1 ] * Transpose([x1, y1, z1]) ) + [ T1 ]
where
[ P1 ] = column vector of camera coordinate of Transpose([x1, y1, z1]) with respect to cam3
[ R1 ] = Camera3 Rotation matrix(from calibration of cam2 and cam3)
[ T1 ] = column vector of cam3 translation matrix (from calibration of cam2 and cam3)
------------------------------------------------------------------------------------------------
Step2: Find the world coordinate of [P1] which is [Q1] with respect to cam2 and cam3 using the calibration of cam2 and cam3
[ Q1 ] = Transpose([x2, y2, z2]) = [ R2^(-1)] * ( Transpose([u1, v1, w1]) - [ T2 ] )
where
[ Q1 ] = column vector of world coordinate of Transpose([u1, v1, w1]) with respect to cam2 and cam3
[ R2^(-1) ] = inverse of Rotation matrix of Cam2 (from calibration of cam2 and cam3)
[ T2 ] = column vector of cam2 translation matrix (from calibration of cam2 and cam3)
-------------------------------------------------------------------------------------------------
Step3: Find the camera coordinate of [Q1] which is [P2] with respect to cam2 using calibration of cam1 and cam2
[ P2 ] = Transpose([u2, v2, w2]) = ( [ R3 ] * Transpose([x2, y2, z2]) ) + [ T3 ]
where
[ P2 ] = column vector of camera coordinate of Transpose([x2, y2, z2]) with respect to cam2
[ R3 ] = Camera2 Rotation matrix(from calibration of cam1 and cam2)
[ T3 ] = column vector of cam2 translation matrix (from calibration of cam1 and cam2)
---------------------------------------------------------------------------------------------------
Step4: Finally find the world coordinate of [P2] with respect to cam1 using the calibration of cam1 and cam2
[ Q2] = Transpose([x3, y3, z3]) = [ R4^(-1) ] * ( Transpose([u2, v2, w2]) - [ T4 ] )
where
[ Q2 ] = column vector of world coordinate of Transpose([u2, v2, w2]) with respect to cam1 and cam2
[ R4^(-1) ] = inverse of Rotation matrix of Cam1 (from calibration of cam1 and cam2)
[ T4 ] = column vector of cam1 translation matrix (from calibration of cam1 and cam2)
-----------------------------------------------------------------------------------------------------
Am I right?
--With regards
Tom
hi martin i am using your code for stereo camera calibration.at the time of compilation it give no error or warning but when i am running this it gives segmentation fault. so please help me out.
thanks
with regards
pankaj upadhyay
@Tom
Hi Tom! Sorry for taking so long to reply. Yep, your procedure looks right to me :) Have you already implemented it?
Good luck!
@Pankaj Upadhyay
I am afraid I can not help you if you don't provide some more detail, for example: Operative System, OpenCV version, at what point you get the segmentation fault, etc...
Best regards,
Martin.
hi martin thanks for reply, i am using ubuntu 10.04 operating system opencv 3.2.
I have resolved the problem of segmentation but i am getting avg error more than 4.0 and i am using more than 25 images from different positions and orientation.
so where is the problem.
thanks
with regards
pankaj upadhyay
Hi Martin
Could you please define exactly the average error we got in the calibration process? Is it in cm? What represents?
Why the images set with binning process is less average error compared with images set without binning process?
Thanks
Hi Martin, thanks for hard work.
I have question about Q matrix:
When i have (x1,y1),(x2,y2) image cordinates,
I can calculate (X,Y,Z) according to Q with your calculation.
Now, i want to test my 3d and show it in opengl - so i need translate (X,Y,Z) to (x,y,z) of the camera.
How can i do that?
Thanks,
Barak
@Pankaj
Hi! Sorry for my late reply. Well, there could be many reasons why you get an error of more than 4.0. It can be due to the distortion introduced by the lenses (if they are cheap), or because the chessboard pattern is not attached to a completely plane surface (like a carton board or similar) and hence when you move it the calibration pattern gets deformed, or any other reason.
@Anonymous
Hi, please read the previous comments, I already answered that question, the average error from the calibration process should be in pixels^2
@Barak
Hi there, thanks for your comment. Maybe I misunderstood you, but if you calculated (X,Y,Z) that is already in camera coordinates. Anyway I would recommend you to read this post: http://blog.martinperis.com/2012/01/3d-reconstruction-with-opencv-and-point.html
Or visit the page of my friend Pablo Fernandez, he already has some work on OpenGL that might be useful for you.
Best regards,
Martin
Hi Martin,
Thanks for such a informative blog.
What is self-calibration? and where this can be used?
Thanks,
Veeru.
Hi Martin, thanks again.
I have question about the whole process of getting 3d model from 2 images:
As i understood, those are the main steps:
- Calibrate the stereo camera.
- Find correspondence between the two images.
- Find 3D points by using Q matrix.
So my question is:
Undistortion and Rectification are required in order to see good results? or I can just add it
later?
I am working on android platform so it is little different from the regular OpenCV.
Thanks alot,
Barak
I want to add another question:
If I calculate the correspondence at different way:
Using Surf algorithm for finding features and thier match , The 3D calculation is same?
Thanks,
Barak
Hi Barak,
You got it right, the main steps are: camera calibration, stereo matching and 3D reconstruction.
Undistortion and rectification is unavoidable if you want to get the best results, here is why: http://blog.martinperis.com/2011/08/stereo-vision-why-camera-calibration-is.html
Also, it doesn't matter how you find the features and their match, the 3D calculation remains the same.
I hope this helps.
Martin.
Hi, for some reason i get very bad results for x,y,z.
I will try to describe my process in detail:
- I made calibration on my stereo camera on HTC 3D Evo device.
- After, in real time, I take each frame that consists of left and right image merged together (half of the frame is the left and the other half is the right). Then, I split the frame to right and left and stretch each image by 2.
Then i search for correspondence between both images and make the 3D reconstruction calculation with Q matrix.
May be if I change the resolution of the images, the results get wrong?
Thanks alot,
Barak
Hi, I would like to have the relative pose estimation of cameras. I mean, the rotation matrix R and translation vector T between two cameras. Any idea?
@Barak
Yup. If you change the resolution of the images then the Q matrix is not valid any more. You can do two things: 1) Use same resolution for calibration and real time, 2) Modify the Q matrix (i don't remember right now, but I think you should multiply or divide Q by the same factor that you are scaling your images).
@Rasoul
Look at the function cvStereoCalibrate() two of the output parameters (R and T) are exactly what you are looking for.
Best regards,
Martin
Hi Martin,
I'm trying to build a system that uses two digital video cameras to track the motion of a ball and to record 3D data about that flight. Thanks for putting up some OpenCV programs that provide steps I'll need to do this. What language are they in? I'm eager to read the code if they're C++. Even if they are, though, I am unable to see them without a Tar translator. I'll buy one if I must but could you email me the OpenCV code for programs you've written instead?
Thanks if you do.
Charles Mennell
Charles@Mennell.us
P.S. The answer to the language question is probably in your site or in the many comments. I'll start to search for an answer in them now.
Hi Charles!
Thanks for your comment and sorry for the late reply, I was attending a workshop last week.
To answer your question, some of the examples are in C and others are in C++, I am trying to move to C++ as libraries such OpenCV and PCL use it by default.
All the examples are compressed using tar and gzip (.tgz) if you use Linux you should have no problem decompressing it, and under windows you can use 7-zip which is free.
Best regards,
Martin.
Hi Martin,
Could you please tell me where to add my files containing the paths to the images in your code. I have tried a lot but could'nt find it.
I am using openCV 1.1pre with Microsoft Visual Studio 2005 in Windows XP platform.
Also please inform me how to make use of the matrix for determining the x,y,z co-ordinates
When I debug your code it returns 1.
Could you tell the reason for it please....
I am not clear with the command lines....
Please inform me at the earliest as it is very urgent.......
Hello Martin! This is Tom again. Sorry for very late reply. Thanks a lot for assuring me that my method of transforming points from camera4's world coordinate system to camera1's world coordinate system is correct.
>Martin said:
>Have you already implemented it?
No I've not implemented it yet. The process is still at the planning stage.
>Martin said:
>Maybe there will be accumulation of errors when transforming from camera 5 to camera1, but it is the simplest way of getting it working that I can think about right now.
My question: why'll there be an error? I thought it is pretty much a straight forward process without error.
Can you kindly tell me why the error will arise in this process? And even if the error is generated how can I minimize or completely remove it?
Again thanks a lot for extending a helping hand.
--Best regards
Tom
Hi Martin! It's Tom again. I've another question that I didn't mention in my last post.
Suppose I've calibrated cameras with the chessboard pattern that is located at the same position when pictures are taken using 2 cameras. So that even when the 2 cameras are not synchronized I've the location of chessboard pattern same (not moved between cameras) on both cameras as if the 2 cameras are synced. Then I've calculated extrinsic and intrinsic parameters using your method.
Now as you know that:
pixel_coordinate_of_a_point_in_right_image = [Rotation_Matrix] * ( pixel_coordinate_of_a_point_in_left_image - [Translation_Matrix] )
symbolically which is:
p_r = R(p_l - T)
I found this equation in a pdf file on page 3 at the top. It's url is:
http://www.cse.unr.edu/~bebis/CS791E/Notes/EpipolarGeonetry.pdf
This rotation and translation matrix: is it the rotation matrix and translation matrix of camera1 or camera2 that you showed you found using opencv's stereocalib() function or is it a different rotation and translation matrix? Can you kindly confirm?
I'm asking this because the cameras I'm using cannot be synced at the same time. So I want to project pixel coordinate of a point from camera1's image to pixel coordinate of camera2's image and then find the 3d coordinates using the process you showed.
So can you kindly tell me if rotation and translation matrix mentioned in the above pdf file is of camera1 or camera2's extrinsic parameter or not?
Again thanks.
--With best regards
Tom
Hi Martin,
really nice work. Thank you very much for this tutorial!! Finally a real practical and usefull implementation for all the theoretical stuff =)
Just have a question, too. How big are your average error values after calibrating your cams? In the OpenCV algorithm a value (stands for "calibration quality check") is calculated. Got values from 0.88 till 1.39 ... Can you please explain what these values stand for?
Is this just the distance between two epipolar points or does it really stands for the grade/quality of the calibration process? Maybe a small value is better than a bigger value?!?!??
Kind regards,
Ronald
Hi Martin,
just want to tell you that I now reached an error value of 0.23 ... it seems to me that it's really a trait of quality. But I still don't know what it stands for =(
Kind regards,
Ronald
Hi there! Thanks for your comments.
@Tom
The calibration process is not 100% accurate, so when you transform, for example, the coordinates from camera 1 to camera 5 you will be accumulating errors from camera 2, 3 and 4. The only way to minimize this is getting a good calibration for all cameras.
Regarding your second question, it looks like the rotation and translation matrices are expressed in terms of the left camera. So assuming that camera 1 is left, then they correspond to the projection of camera 1 into camera 2.
@Ronald
Hi! Thanks a lot for your kind comments. The values you get represent the calibration error in pixels^2. So the lower the value the better the calibration :)
I hope this could be helpful.
Best regards,
Martin.
Hi Martin,
How can i use this code for estimating 3d position of an object(here chessboard). For example, by using 2 camera i wanna have 3d position of chessboard object(Of course simultaneous 3d position)? How could i do that?
Hi Martin:
Your code is very useful and great. I thought I'd pass on a lesson I just learned the hard way...
when making your checkerboard, DO NOT make the number of intersections in x and y the same. The corner finding algorithm will occasionally recognize the wrong corners,and instead of making the "rows" horizontal, will make them vertical on one of the images in a pair. Really screws up the calibration when the wrong pairs get correlated!
Hi Martin very usefull sample.
Just to know how can i achieve the pose of a 3 or 4 point object captured as stereo pair blobs , i have research, but i am very confused, sholud i try homographies, or is there any other solution to get de 6 DOF( ROT, Translation)
Thank you.
Hi Martin,
This is regarding Remap function.One of your replies says that
P2.x = mx(P1.x, P1.y)
P2.y = my(P1.x, P1.y)
gives the the point in undistorted image, given x,y of the disotorted image.Isnt this the other way.? For a given point(x,y) in undistorted image, this finds the source point and fills it.
I would like to know for the other case. Given a point x,y in distorted image, is there any way to find where it lands up in undistorted image? Like finding some inverse map of mx,my ? Could you please help?
Hi all,
Here I am with another batch of comment answers, kind of late, as usual... sorry about that :P
@Anonymous I would recommend you to read this post and this other post. Maybe it can help you.
@jch Very important lesson indeed :) Thanks for the comment.
@Alvaro Take a look at cvPOSIT, if you have the focal length of your stereo camera (is one of the results of the calibration step) you can use this to get the 6DOF pose of your object.
@VP Hi! mx and my are maps to transform points from the distorted image to points in the undistorted image. This is:
Given a point (P1.x, P1.y) in distorted (original) image, it tells you where the point lands in the undistorted image (P2.x, P2.y).
So what you are looking for is exactly this:
P2.x = mx(P1.x, P1.y)
P2.y = my(P1.x, P1.y)
Anyway, this is useful only if you are only interested in getting the undistorted coordinates of specific points in the distorted image. If you want to undistort a complete image this function does it for you:
cvRemap(imgOrig, imgUndistorted, mx, my);
Sorry, maybe I was unclear in the previous replies. It might get a bit confusing :P
Thank you all for reading!
Hi,
I have problem of accurity at the calibration process.
I will be glad if it is possible to send you
my images for the calibration so you can tell us what might be the reason.
My mail : barak1412@gmail.com
Thanks alot,
Barak
Hi,
I found the problem.
Thanks anyway.
Barak David
Hi Martin,
This is Rajaram M from Bangalore, I am currently working on Obstacle's Distance Calculation using Custom Made Stereo CAM setup.
I am using dual Microsoft Lifecam vx700 for the setup.
While calibrating the Stereo setup i am getting the return value from the function cvStereoCalibrate(). That value is keep on changing from 1.5, 3.4, 5.6, 8.5 and 2.3 for every trials. I know the good calibration will get only 0.1 pixels to 1 pixel variation.
I am doing the calibration through the livefeed from camera.
What could cause to get these type large pixel calibration error.
With Regards,
Rajaram M
Hi Rajaram,
How is the weather in Bangalore?
Well, without seeing your calibration images there is not much advice I can give you. But, the most important things to keep in mind to obtain good calibration results are:
-Calibration pattern (chessboard) must be rigid. If, for example, you print a chessboard pattern on a piece of paper, then you should attach it to a rigid surface (piece of wood, carton or similar) to avoid deformations on the calibration pattern.
-Get images of the calibration pattern from as many positions and orientations as possible.
I hope this helps.
Best regards,
Martin.
Thanks for your comments Martin.
Here weather is not like earlier days, too sunny...
Come to the point:
Yes, I have not used rigid pattern, used the paper.
Let me just try on the same with the rigid pattern and i'll let you know the status of the same.
And I've some doubts about the following things,
1. Return value of cvStereoCalibrateCamera()
2. Finding reprojection err using Epilines.
Can you give some hints about the 2 things which i said above?
With Regards,
Rajaram M
Hi Martin,
How can I attach my sample images in your blog, so that you can see those images.
Thanks and Regards,
Rajaram M
Hi Martin, i capture images using a stereo camera Fujifilm W3.
i have question about remapping matrix mx1, my1, mx2, and my2.
Somehow, the remapping is wrong and as you can see here, http://140.118.127.117/files/rectified.png,
the right image is distorted especially in the four corners.
i am still trying to figure out what's wrong but still no luck.
ah never mind, i think i've found the solution. Then i used bigger chess pattern and more images, before i uses 9x6 with 11 pair images
and later use 15x8 26 pair of images
and the result is better :D
http://140.118.127.117/files/rectified_02.png
one more question, i got bigger avg err = 99,3
what does this value mean?
Hy Martin,
I have a question. Is it ok if i use your camera parameters matrix of your camera calibration like _M1 ,_M2 , _T, mx1 ,mx2 ,my1, my2 and use it for my work like calculating disparity map.
Can we use other's stereo Calibration files into other's stereo camera work for calculating disparity map ?
Hi Martin,
I am facing lot of problem when doing modification to your code of stereo calibration. Can you provide the code for stereo calibration and 3d reconstruction implementing with cameras and not with just taking images into code..
waiting for your reply
Hi Martin,
Thank you very much for your code.It helps for understanding Opencv. I want to ask you two things.
1)When we calculate the disparity the order of the subtraction have to be as you said or like the book d = pointLeftImage.X - pointRightImage.X, am confused.
2)The XYZ that we find for an object the coordinate system is from the center of the stereo rig or from the left camera position?
Hi everybody!
Excuse me, I want to ask you about a question. I using OpenCv to process image. Now I want to output coordinates (in plane 2D) of an object (example: table tennis is being motion). How can I program for this project? I use webcam on my laptop.
Best regards,
Victor Lee
This is an useful and interesting post.
I want to know if is possible to pass from the undistorted image to the original because I use my own disparity algorithm and I want the image that it creates, in the original position.
Does it exist an inverse matrix of mx1 and my1 or something like that?
Hy guys,
I calibrated my 2 cam and saved camera calibration parameters .
I just loaded parameters in other program to calculated disparity map .
my disparity map is so different from the disparity(when i calibrater my cams )
Note: I am using logitech cam with fixed distance between cam all the time .
Does it mean each time i have to calibrate camera for disparity map ?
I was interested in knowing why a single planar rig is not sufficient for stereo camera rectification and what rig orientations are optimal. It seems that the answer to this question is based on the estimation of a projection matrix. This matrix is factorized into intrinsic and extrinsic camera matrices and the latter determines relative camera misalignment.
To understand what determines the accuracy of the projection matrix estimation, consider a vanishing point in x direction =[1 0 0 0] in homogeneous coordinates. Is you multiply it from left with a projection matrix P you will end up with the first column of P. This proves that the first three columns of P are coordinates of the projections of the vanishing points in X, Y, Z directions (the forth column is a projection of the origin of the world coordinate system).
Now, even if you slant and tilt a planar rig, its lines will "converge" in 1-2 directions only. Thus one of the columns of the projection matrix will be ill-defined. Moreover, using an image of a slanted pattern will bias estimation process so you need to use an image of an opposite slant too.
Hi Martin,
When I run this line in my terminal
./stereo_calibrate list.txt 9 6 2.5
it shows me this
Segmentation fault(core dumped)
How to fix it?
Hi...Martin
Just had a random querry as to what is the significance of skipping frames while capturing images for camera calibration?
EXCELLENT CODE!
Hi Martin!
When i run the code in visual c++ it only loads the first picture right01 and stops..what is the reason?
thanks
Saab
Hi Martin!
Can anyone help me with this so I did the stereo-calibration, but the avg error is 2.3 , what is the could be the reason, i followed the same steps as above.
Murkus
Hi Martin,
I saw a few implementations of camera calibration. I was not sure what the object points have to be assigned to( Is it coordinates of the corners). In ur case, if square size if 4x4, then the object points become (0,0,0),(0,4,0),(0,8,0)...(4,0,0),(4,1,0) and so on. Is my understanding right.
Thanks
Arvind
Hi,
I am also getting an error "Segmentation Fault"
I used gdb to and I got this as the output:
Program received signal SIGSEGV, Segmentation fault.
0x0306419b in ATL_dgemvT_a1_x1_b0_y1 () from /usr/lib/libblas.so.3gf
I am using OpenCV 2.2 on Ubuntu 11.10
How should I correct the error ??
Thanks!
Hi...i followed the example in the book "Learning OpenCV" by Bradski for the camera calibration technique with chess board..however i am getting different fx and fy values when calibrating the camera at different distances from my webcam...is it correct to get such type of result?i have used the CV_CALIB_FIX_ASPECT_RATIO option for calibration with fx and fy initially set to 1 ..as in the example given in the book..
Thanks for the post and the code, I wonder if you have sometime to look at some images to tell me what is going on. I used your code, the error is low 0.22 but the disparity map is awful to me. I'm new to stereo processing so it might be just my feeling. Since you are an expert, could you take a look to see if the disparity images I've got are fine or not.
hi martin,
i want to know how to create a text file with the paths to the images.
i am using windows 7 and MS visual 2010 32bit..
if possible can you give me the instruction step by step...
Hello,
Great job,
I ran the code, but there is no xml has been saved
So what shall I do???
thanks in advance
Dear Martin,
I am a clarification question regarding the reprojection matrix Q.
If you have a two corresponding points (u1,v1) and (u2,v2), you can compute the vector [u v d 1] where: u=u1, v=v1, d=u1-u2. Then you can compute the 3D coordinates of the point using the reprojection matrix Q: Q*[u v d 1].
My question: do you use the same Q regardless of whether you choose u=u1, v=v1, d=u1-u2 or: u=u2, v=v2, d=u2-u1?
If not, how do you compute the reprojection matrix for the other camera?
Thanks!
Hi Mark,
The matrix Q is defined as the "left camera reprojection matrix". That is, all the parameters on matrix Q are defined using the left camera optical center as coordinate origin.
That is why you use u=u1, v=v1, d=u1-u2. If you want to use u=u2, v=v2, d=u2-u1, then you should transform matrix Q to the right camera optical center. Which is not difficult, if I am not mistaken, just changing the sign on Q[3][4] and Q[4][4] would do the trick.
mmmm forget what I said about Q[3][4] and Q[4][4]... you would need to changes more things. Check page 435 of the book "Learning OpenCV: Computer Vision with the OpenCV Library" by Bradsky and Kaehler for more info.
Got it. I suppose I can just rerun stereoRectify and get another Q (for the right camera).
But, now I have this confusion: if I use the calibration and a set of matching points in the two cameras to compute depth, I notice something strange: the 3D coordinates of a point in the left and right camera are not different by R,T computed by stereoCalibrate.
Here's my question: is this expected only when the pairs of points are actually not a correct pair? or is there something wrong going on?
(because I noticed that when the points are a good match, the 3D coordinates in the left and right frame can be mapped onto each other using the R,T above)
Thanks!
Dear Martin,
Thank you for your wonderful piece of software! I have got several questions. You say we need to load 5 matrixes - Q, mx1, mx2, my1 and my2 in our desired application. After that you show how to use cvRemap. But as we can see it doesn't use Q matrix. I am using cvRemap in my application to undistort images from 2 webcams. But what about Q matrix? How to use it to get full rectification? I mean I want to get epipolar lines working. This is my current image I get:
http://i.imgur.com/C3B9y5D.png
Got it... :D I didn't observed the fact that when I was taking photos for calibration with guvcview I was flipping camera (flipping in guvcview and my app were opposite). Now I have more or less good epipolar lines. But! Corners... Why so bad?
http://i.imgur.com/8D8Esuc.png
hi martin,
i am vineeth,i am a opencv developer,i want to mesure the atual size of rectangular object in a video camera.
Hi Martin:
I have been successfully using a variant of your code for stereo calibration for some time, and have implemented a stereo measurement system using nvidia stereo glasses and open gl stereo windows.
I would like to move a cursor around in 3d object space instead of in pixel space. This would involve reprojecting xyz coordinates into rectified image coordinates. Do you know if the Q matrix should be invertible? If so, I could use a variant of your equation for XYZ computation. Do you know of a better way to do this while working in the openCV environment?
Thanks in advance for any pointers and for your tremendous contribution to ready and practical,use of stereo systems
Jch
Hi Martin,
I have some questions regarding the estimated camera parameters.
I observe that the focal value contained in M1 and M2 are different that the focal value contained in Q. What is the best value to select?
Moreover, I observe that the focal is not in mm. What is the unit of this value? and how can I get it in mm?
What are the differences between the use of the formulas Depth = f * B / d, X=..., Y=... and the use of the function ReprojectImageTo3D? ReprojectImageTo3D
I have looked at the OpenCV book but I don t find convincing information.
Thank you for your assistance.
Best regards, Karim
Dear Martin,
Thank you for the effort. I have the same question as karim, what is the units of the focal length?
Thank you in advance!
Hey,
Am I right that Q puts the reconstructed 3D points in a frame that is centered between the two cameras?
Thanks!
... to clarify: if I reproject the reconstructed 3d points into the left image, they result is translated from where it should be.
I noticed that if I had 0.5*T to the reconstructed 3d points before projecting to 2d, the results are nicely aligned.
Do you agree? Also, if that's the case, I suppose I also need to fix for some rotation between the left frame and the middle frame, but I'm not sure how to extract that from R?
hi Martin,
I have some question about internal parameter & external
parameter, they saved in which file?
thanks for your answering.
Best regards, Taiwan YuJhuLi
Hi all. I am needing help to convert stereo cameras in an app i am developing to measure animals in the wild. We are having trouble with the calibration side of things as we seem to get more accurate length estimates without calibration. But I need these measurements to be within mm. I feel we may have a units conversion issue, or bad code somewhere. I need help asap as the field trip is very soon.
Hey Martin! Thanks for the wonderful blog! Good Work!
I have a question. I am using your calibration source code and its working fine with your set of images, but if I try to use images from my webcam, an error occurs.
but I'm just using the same functions that you used and the same data types for the input and output arrays.
I tried changing the types of the matrices from CvMat* to cv::Mat to IplImage* and there's still the error.
Can you help me figure out the problem?
Thanks :D
-John
Martin,
It s much impressive. I have some other problem similar to it. I would like the read camera attitude parameters from the image inputted. can u help me figure out this problem?
The formula maybe has a little problem.
The d should be defined as d = pointLeftImage.X - pointRightImage.X;
I give an example for detail.
Q:
1. 0. 0. -447.10495758056641
0. 1. 0. -183.65547370910645
0. 0. 0. 745.08625963284953
0. 0. 0.15033768486581800 12.326638373283712
Q:
1 0 0 -Cx
0 1 0 -Cy
0 0 0 f
0 0 -1/Tx (Cx-C'x)/Tx
Q * [x y d 1]' = [(x-Cx) (y-Cy) f (-d+Cx-C'x)/Tx]' = [X Y Z W]'
OpenCV method:
Z = (-Tx*f)/(d-(Cx-C'x)) = (-Tx*f)/(d-Cx+C'x)
= (6.6516921614998751484990356687739 * 745.08625963284953) /(-20 - 447.10495758056641 + 529.09796142578124912672073900634) = 79.945866879035575167439631251721
Tx = -1 / 0.15033768486581800 = -6.6516921614998751484990356687739
f = 745.08625963284953
Cx = 447.10495758056641
C'x = Cx - (Tx * (Q[4, 4]))
= 447.10495758056641 - ((-6.6516921614998751484990356687739) * 12.326638373283712) = 529.09796142578124912672073900634
d = Xleft - Xright
d = 338 - 358 = -20
left = (338 , 192)
right = (358 , 192)
Martin's formula(original):
d = pointRightImage.X - pointLeftImage.X;
X = pointLeftImage.X * Q[0, 0] + Q[0, 3];
= pointLeftImage.X * 1 + (-Cx);
Y = pointLeftImage.Y * Q[1, 1] + Q[1, 3];
= pointLeftImage.Y * 1 + (-Cy);
Z = Q[2, 3];
= f
W = d * Q[3, 2] + Q[3, 3];
= d * (-1/Tx) + (Cx-C'x)/Tx
X = X / W;
Y = Y / W;
Z = Z / W;
= f / (d * (-1/Tx) + (Cx-C'x)/Tx)
= (f * Tx) / (-d + Cx - C'x)
= (-Tx *f) / (d -Cx + C'x)
This formula are same with OpenCV method, but there is a small different with OpenCV.
The definition of d is different.
In OpenCV method, d is defined as (Xleft - Xright).
But in martin's formula, d is defined as (pointRightImage.X - pointLeftImage.X).
martin's formula(match with OpenCV's method):
d = pointRightImage.X - pointLeftImage.X;
= 358 - 338 = 20;
X = pointLeftImage.X * Q[0, 0] + Q[0, 3];
= pointLeftImage.X * 1 + (-Cx);
= 338 * 1 + (-447.10495758056641) = -109.10495758056641
Y = pointLeftImage.Y * Q[1, 1] + Q[1, 3];
= pointLeftImage.Y * 1 + (-Cy);
= 192 * 1 + (-183.65547370910645) = 8.34452629089355
Z = Q[2, 3];
= f
= 745.08625963284953
W = d * Q[3, 2] + Q[3, 3];
= d * (-1/Tx) + (Cx-C'x)/Tx
= 20 * 0.15033768486581800 + 12.326638373283712 = 15.333392070600072
X = X / W;
Y = Y / W;
Z = Z / W;
= f / (d * (-1/Tx) + (Cx-C'x)/Tx)
= (f * Tx) / (-d + Cx - C'x)
= (-Tx *f) / (d -Cx + C'x)
= 745.08625963284953 / 15.333392070600072 = 48.592396007499376081824002716956
martin's formula (Revised):
d = pointLeftImage.X - pointRightImage.X;
= 338 - 358 = -20;
X = pointLeftImage.X * Q[0, 0] + Q[0, 3];
= 338 * 1 + (-447.10495758056641) = -109.10495758056641
Y = pointLeftImage.Y * Q[1, 1] + Q[1, 3];
= 192 * 1 + (-183.65547370910645) = 8.34452629089355
Z = Q[2, 3];
= 745.08625963284953
W = d * Q[3, 2] + Q[3, 3];
= (-20) * 0.15033768486581800 + 12.326638373283712 = 9.319884675967352
X = X / W;
Y = Y / W;
Z = Z / W;
= 745.08625963284953 / 9.319884675967352 = 79.945866879035575167439631251728
Zhao Xiaoyu
Hi Martin,
I calibrated my stereo cameras and rectified my left and right images. I calculated disparity from this formula(d=xL-xR) for each corresponding points. Now, how can I calculate Z(depth) coordinate using disparity values. After my calibration, focal length=778, baseline=53. But using these values, I am getting very large X,Y and Z coordinates and I'm using Z=f *b /d formula. Can you help me about this problem?
Thanks.
Hi Martin, I'm Fivit..
i'm using windows seven and Ms. Visual c++ 2010 (32bit)
i want to ask you, how to compile the stereo_calibrate.cpp file?
and what kind of project it will be creating with? win32 console application, win32 project, or else?
i'm sorry for my stupid question :D
your help is much appreciated, thanks..
i want convert 2d images into 3d using opencv first step im doing is that camera calibration but how i know that camera is calibrated properly and how i go for further please guid me for it
Hello, congrats for your great article!
I've noticed that the input of the program are the pair of photos, the dimensions of the board and the square size.
And what about the camera parameters like focal distance or distance between the cameras? Are they irrelevant to this problem? Or will they be calculated based on the images?
Thanks!
Hi Martin,
This was a really nice tutorial .
I was working on stereo setup made up of same camera bit different focal length of lenses. I have tried to keep camera parallel but due to different focal length I'm not going to have epipoles at infinity .So disparity concept wont work in this case. Do I need to rectify the images but still rectification is valid only if I have same focal length camera .Can you suggest something?
One way is to find Essential Matrix and for each point back project and compute its distance.
Hello,
this is a great article and helped me with my project. However, I'm getting some high avg errors. My program takes the photos that will be used in the calibration and then just fork/exec yours with the images list file and the pattern parameters.
These are the cameras: http://www.baslerweb.com/products/ace.html?model=171
Is there a recommended checkerboard size related to the camera resolution and/or focal distance?
Thanks!
-Gabriel
Hey Martin,
I'm getting the avg = 150!! And the disparity image is wierd! Where am i going wrong?? I tried keeping the chessboard flat too!!
Hello Martin,
As I have mention in your other thread, that I am a beginner so please bare with me. In reference to this calibration, as I have printed the checker board and is ready for the shooting stage. Hence, I have in addition have installed Coriander to my machine - but I am a little confused rather if I will be using both my web cams to take these images of the checker board or just one web cam. I would assume I would need both cams - based on the double images seen on your pics above. Lastly is there any beginner documentation on how to add double webcams to Coriander? Again Thank you for any feedback Maestro!
Saludos,
Luis
Hi Martin!
Is the calibration script working with OpenCV 2.4.8 (Ubuntu 14.04)? I take 2x12 images for calibration with very cheap cameras (not same lenses too). I get avg err = 5 and blank rectified images and disparity (both black only)...
Ok, so here we go - you seem to mean the left image to be from the RIGHT camera - that means you mean LEFT when facing the cameras (not from camera's view). I swapped left and right images and now I get some results already ;-).
Hi Martin,
In your above paper, you said “how to find interest points in one image and its correspondent on the other is an art that will be explained another day“. Have you explained this in somewhere else? How can we find an interest point in the left image and its correspondent on the right image? Can you please give a solution for this?
Thanks for your help!
Yong
Awesome post. I wrote another version of stereo calibration code where you can calibrate a stereo camera in real time. I think it can be helpful for someone else.
Source: https://github.com/upperwal/opencv/blob/master/samples/cpp/stereo_calibrate_real_time.cpp
Sample Video: http://youtu.be/kizO_s-YUDU
Hello Sir,
When I tried to compile your code,, the build is succedded but during execution asome kind of vector assertion error comes?? Please can you help me???
And somewhere in code i found out ,, for loop without the ending contion what is it?? Does this work?? i don't know, this i think is giving me debug assertion error
for(i=0;;i++)
{
char buf[1024];
int count = 0, result=0;
lr = i % 2;
vector& pts = points[lr];
if( !fgets( buf, sizeof(buf)-3, f ))
break;
size_t len = strlen(buf);
while( len > 0 && iss
Hey Martin,
Thanks for the code. I have an error when loading my own calibration images:
"vector subscript out of range". It happens when loading Right images. First it loads "left01.png" correctly, but when loading "right01.png" it returns the above error. I have tried converting my png files to ppm. also renaming files so they start from 00, but still have the problem. it works well on your calibration images though. Is there any special format of calibration images I need to follow?
Hi martin, i am student and interested with the opencv stereo calibration. i have installed visual studio and link it with opencv. i am having a problem to run it.... i already build it and the build is succeed but when i start debugging it say cannot detect pdb file.where did i do wrong.. I am totally a beginner in C++ coding..
i tried to run .exe file using cmd.. but the only thing that pop out is
USAGE: ./stereo_calibrate imageList nx ny squareSize
imageList : Filename of the image list (string). Example : list.txt
nx : Number of horizontal squares (int > 0). Example : 9
ny : Number of vertical squares (int > 0). Example : 6
squareSize : Size of a square (float > 0). Example : 2.5
is there any step that i overlooked..Again Thank you for any feedback
Post a Comment