STEREO VISION: WHY CAMERA CALIBRATION IS NEEDED?

As you might well know, if you have 2 different views from the same scene then you can estimate the 3D coordinates of any point in the scene by finding the position of that point in the left image and in the right image and then apply some trigonometry.

Let's assume that we have no previous information about the relation between the cameras. We can find a point of interest in the left image, but we don't know where that point of interest will appear in the right image. So, what do we do? We have no other option than scan the whole right image looking for our point of interest. 

Now I can hear you say: "But that would be soooo slooooow!!". Yep, you are absolutely right, that brute force approach is really slow. But, if we know the relation between both cameras then we can calculate something called epipolar lines. 

What is so special about these lines? Well, the magic about this lines is that a point in the left image will always have its correspondent point on the right image laying on its correspondent epipolar line! So, now instead of having to scan the whole right image to find our matching point, we only have to look over a single line :)

But wait!! There is more!! If the cameras are completely parallel then something very special happens... the epipolar lines become parallel. This means that the match of a point in the left image will appear in the exact same line on the right image! Isn't that awesome?

That is one of the main reasons to wanting to calibrate your stereo camera. Another good reason is that the lenses of the cameras introduces some distortion. That distortion makes straight lines in the real world appear curved in the image... and you don't like that, do you?

Let me show you a couple of image just to clarify.

The image above shows a couple of images taken with an uncalibrated stereo camera. The cameras are more or less parallel, but they are not perfectly aligned. Do you see the red point? See how it is not in the same line on the left and right images?

Now, almost all the methods used to calculate a dense disparity map rely on calibrated images, so if we try to use this images to calculate the dense disparity map we will get really poor results. You can check it on the next picture:

But now, if we apply the magic of calibration:

The image above has been rectified and undistorted (notice the black borders around the image, they are the result of removing the distortion and aligning the images so the epipolar lines are parallel and appear on the same row in bot images). See the green point? Do you see how it appears in the same row on both images?

Now, if we use this to calculate the dense disparity map:

There it is, much better results!!

To sum up, if you want to get the best out of stereo vision:

1. Make sure that your cameras are as parallel as possible.
2. Calibrate the stereo camera. See this post for instructions: http://blog.martinperis.com/2011/01/opencv-stereo-camera-calibration.html
3. Tune the parameters of your stereo matching algorithm. See this post to get an example: http://blog.martinperis.com/2011/08/opencv-stereo-matching.html
4. Have fun with it!

So you know, questions are welcome and any comments will be appreciated ;)

OPENCV: STEREO MATCHING

In a previous post I talked about how to calibrate a stereo camera using OpenCV. Today, I would like to talk about the next step. Once your stereo camera is calibrated you can estimate the 3D position (relative to the camera) of any object given its position in the left and right image. For that, we need to calculate the stereo disparity for that object (stereo disparity = the difference in image location of an object seen by the left and right camera). If we want to know the 3D position of all points in a stereo pair of images, then we want to compute a dense disparity map. And that is what this post goes about.

A dense disparity map looks like this:

 I am not going to explain the details or the math behind it, I am more of a practical kind of guy. So let's start.

Basically OpenCV provides 2 methods to calculate a dense disparity map:

• cvFindStereoCorrespondenceBM: Fast (can process several images per second), but if parameters not tuned then the results are poor.
• cvFindStereoCorrespondenceGC: Really Slow (takes several seconds, even minutes per image), but gets very accurate results.

In this post I will focus on cvFindStereoCorrespondenceBM, this method is based on Konolige's Block Matching Algorithm. The OpenCV call looks like this:

void cvFindStereoCorrespondenceBM(const CvArr* left, const CvArr* right, CvArr* disparity, CvStereoBMState* state)

The structure CvStereoBMState contains all the parameters that are applicable to the algorithm. There is a bunch of them (pre-filtering, Sum of Absolute Difference windows size, disparity-related, post-filtering...). So, to make it easy, I implemented a small Gtk application that takes 2 images (left image and right image), calculates the disparity map using cvFindStereoCorrespondenceBM and allows you to play with the parameters.

The application is written in C and can be downloaded here: StereoBMTuner-1.0. The application depends on the libraries gtk+-2.0, gmodule-2.0 and opencv. Be sure to have them installed in your system.

Once the file is downloaded just execute:

tar xzvf stereoBMTunner-1.0.tgz
cd StereoBMTunner
make
./main

The last command will execute the application

As you can appreciate, the disparity map generated using the default parameters is hardly similar to the first image on this post. But, you can tune the parameters until you get a clearer disparity map. This video shows the use of the application:

Once the parameters are tuned, the disparity map is much better

It is still not perfect, but it is not so bad either.

Now, to use this application with your own couple of images the only thing you need to do is execute the application like this:

./main -left /path/to/my/image/left -right /path/to/my/image/right

And that's it. Please leave a comment if you found this useful, have any problems, questions, suggestions, impressions, etc...
25JKKNMXU6FE

SVN: EVIL COMMIT

Subversion is a very useful centralized version control system. Is very convenient for Software development projects as it allows to the developer team work together in the same files at the same time, keep a record of all the changes and eventually return to a previous stored version of the project if somebody did something wrong or decide to do things differently. 

I have been using it for many years now in a professional environment and for over a year in my private projects (I use it even in my private life xD). I use it in conjunction with trac (which is a powerful tool for Project Management). You might think I am a bit crazy (or completely crazy), but when you have a lot on your mind it is worth having some kind of organization method and this has worked for me so far.

Anyway, recently I was working on my project, committed some changes and got this:

marpemar@MARTIN-PC:~$ svn ci -m "generate_training_samples_from_monocular_images generate color samples"
Sending generate_training_samples_from_monocular_images/main.cpp
Transmitting file data .
Committed revision 666.

Yay! I've reached 666 commits xD To celebrate it I would like to share this video recorded with gource showing the progress and evolution of my private repository until the moment of the evil commit.  Enjoy it!