THE NEW TSUKUBA STEREO DATASET IS OUT!

Finally! After so much work we released the "New Tsukuba Stereo Dataset" at ICPR2012

This dataset contains 1800 stereo pairs with ground truth disparity maps, occlusion maps and discontinuity maps that will help to further develop the state of the art of stereo matching algorithms and evaluate its performance. It has been generated using photo-realistic computer graphics techniques and modeled after the original "head and lamp" stereo scene released by University of Tsukuba in 1997. 

The dataset is a 1 minute video sequence and also contains the 3D position and orientation of the camera on each frame, so it also can be used to develop and evaluate camera tracking methods.

It can be downloaded freely from the CVLAB website

Enjoy it!

3D RECONSTRUCTION WITH OPENCV AND POINT CLOUD LIBRARY

Hi everybody!

I bring you a sample of how to reconstruct a scene in 3D using OpenCV and Point Cloud Library (PCL) with a simple program and an example scene.

All we need is the left image of our stereo camera:

(You can implement your own cheap stereo webcam following this post: OpenCV Stereo Webcam)

The disparity map generated with your preferred Stereo Matching algorithm:

(For example you can use OpenCV's stereoBM algorithm: OpenCV StereoBM)

And the reprojection matrix (Q) obtained at calibration time:

<?xml version="1.0"?>
<opencv_storage>
<Q type_id="opencv-matrix">
  <rows>4</rows>
  <cols>4</cols>
  <dt>d</dt>
  <data>
    1. 0. 0. -2.9615028381347656e+02
    0. 1. 0. -2.3373317337036133e+02
    0. 0. 0. 5.6446880931501073e+02
    0. 0. -1.1340974198400260e-01 4.1658568844268817e+00
  </data></Q>
</opencv_storage>

(You can get the matrix Q for your own stereo camera following the instructions in this post: OpenCV Camera Calibration)

Now download the source code (I highly recommend to read the source code to understand what is going on, don't worry there is comments :P):

[NOTE]: You will need to have installed OpenCV Library (you can get it here) and Point Cloud Library (you can get it here). Also you will need CMake to generate the Makefiles.

Once you have downloaded the source code and installed the dependencies, just run:

tar xzvf OpenCVReprojectImageToPointCloud-1.0.tgz
cd OpenCVReprojectImageToPointCloud
cmake .
make
./OpenCVReprojectImageToPointCloud rgb-image.ppm disparity-image.pgm Q.xml

You should see something similar to the following video:

I hope you enjoy it!
[UPDATE: 04/02/2012] I have released a bug-fix. Thanks to Chris for pointing it out.

OPENCV: STEREO WEBCAM

Today, an easy one :) A small program in OpenCV to implement a cheap stereo webcam and visualize the left and right images. The only thing you need are two USB webcams plugged in your computer (better if both are same brand and model):

 Download the source code:

Compile it:

tar xzvf OpenCVStereoWebcam-1.0.tgz
cd OpenCVStereoWebcam
make
./OpenCVStereoWebcam

And there you go, your cheap USB stereo webcam.

Have fun!

CLEVERBOT: 59% HUMAN

Cleverbot is an Artificial Intelligence conversation system specially designed to learn from the conversations that it has with other people trying to mimic the human behavior in a conversation.

To find out the level of intelligence of this kind of artificial entities the archi-famous mathematician Alan Turing proposed in 1950 the Turing Test .  Basically, a human judge engages in a natural language conversation with a human and a machine designed to generate performance indistinguishable from that of a human being. All participants are separated from one another. If the judge cannot reliably tell the machine from the human, the machine is said to have passed the test.

Every year the Turing Test  takes place, this year was celebrated on September 3rd at Techniche 2011, IIT Guwahati, India. In this scenario, several human judges had 5 minutes conversations with another entity (they ignored whether the entity was human or not). Some people believe that if 50% of the judges classify the entity they are talking to as human when in reality it is a machine, then the machine has passed the test. 

This year Cleverbot was classified as human 59% of the times and people might say it passed the Turing Test, but we humans still have hope... humans where classified as humans 63% of the time. I will start getting scared when Cleverbot is classified more often as a human than real humans xD

Anyway, it is fun to talk to Cleverbot, you really have to try it! How? You can do it right here! Just enter some text in the text bar above and hit the "Think About It!" button. Have fun!!

EDIT [2011/11/14]: I removed the widget to talk to Cleverbot directly because it was causing undesired behavior on the website. You can still talk to it by clicking on its logo and visiting Cleverbot's homepage.

STEREO VISION: 3D RECONSTRUCTION ERROR

Today I would like to publish the answer to another question in the comments of a previous post that might be worth its own post:

Hi Martin,

You gave such an informative article. Good Job Martin:-)

I'll explain steps that I performed in calculating distance of object.

1. Calibrated stereo camera with chessboard 8x6 cm with 2.8cm square size.

2. My calibration was success with rms error 0.206 and rectified images was good.

3. I found my interest point on left-right image. I tried to determine distance of object using method you specified. There is an error in distance measured from 3cm for closer object to 12cm or more for distant object.
Say, actual - 27cm ; measured - 24cm.
actual - 60cm ; measured - 48cm.


My queries are,
- why does there comes this much big varition is distance measurement?
- What may be reason/solution for this error?

Is there any mistake in my procedure or do i miss parameters?

Regards
Sathya Kumar

Dear Sathya,

First of all, thank you very much for your kind comment. Regarding your queries, I am afraid that what you describe is quite normal. The steps that you followed are correct, so I don't think that you did any mistake in your procedure or missed any parameters.

The thing is, that the relation between the disparity and the depth is non-linear.

Google ads, probably not very well related to the audience of this blog...

For this reason there is such a big variation in the distance measurement error, for close objects a small variation in disparity means a small variation on depth. But, for far objects a small variation in disparity means a big variation in depth.

So, there is no easy solution, it is unavoidable to get bigger error for distant objects, but you can try to mitigate the effects and reduce the error by getting a calibration as good as possible. And to do that you should take captures of the calibration pattern in as many different positions and orientations as possible, far, near, inclined, etc... Also increasing a bit the size of the chessboard could help you to get better accuracy for distant objects.

I hope this solved your doubts.

Best regards,
Martin

OPENCV: PSEUDOCOLOR AND CHROMA DEPTH

Hi everybody!

A couple of days ago one of the readers of the blog asked this on the comments of a previous post:

Hi,

I'm Mike. Nice work! :)

I saw that some people have coloured depth maps and not just shades of grey/black/white. If I want to have coloured depth maps how can I achieve this?

Thanks,
Mike.

Well Mike, some time ago I had the same question and researching a bit I found two approaches. Pseudo Color and Chroma Depth. Let me answer your question with a blog post, so anybody else who could be interested can find it easily.

Basically, what we have is a function that takes as an input a gray scale value and returns the corresponding RGB.  The Pseudo Color approach uses a trigonometric function to assign the RGB color to each gray value. The next picture shows an example:

 As you can see the output is somewhat similar to the images of the luggage scanners in airports and train stations.

But for the purpose of depth visualization, maybe it is more convenient to use Chroma Depth. This method assigns red values to high depth values, magenta values to low depth values and a rainbow color to anything in between. Like in the next picture:

To make it easy, you can download  the simple Gtk application that I made to visualize the output of both methods and play around with some parameters. The source code should be self explanatory, so I will skip the details in this post.

Just execute "make" and then "./GrayToPseudocolor -gray /path/to/your/image.something"

I would like to thank Sarah Martull for letting me use her depth image for this post.

STEREO VISION: WHY CAMERA CALIBRATION IS NEEDED?

As you might well know, if you have 2 different views from the same scene then you can estimate the 3D coordinates of any point in the scene by finding the position of that point in the left image and in the right image and then apply some trigonometry.

Let's assume that we have no previous information about the relation between the cameras. We can find a point of interest in the left image, but we don't know where that point of interest will appear in the right image. So, what do we do? We have no other option than scan the whole right image looking for our point of interest. 

Now I can hear you say: "But that would be soooo slooooow!!". Yep, you are absolutely right, that brute force approach is really slow. But, if we know the relation between both cameras then we can calculate something called epipolar lines. 

What is so special about these lines? Well, the magic about this lines is that a point in the left image will always have its correspondent point on the right image laying on its correspondent epipolar line! So, now instead of having to scan the whole right image to find our matching point, we only have to look over a single line :)

But wait!! There is more!! If the cameras are completely parallel then something very special happens... the epipolar lines become parallel. This means that the match of a point in the left image will appear in the exact same line on the right image! Isn't that awesome?

That is one of the main reasons to wanting to calibrate your stereo camera. Another good reason is that the lenses of the cameras introduces some distortion. That distortion makes straight lines in the real world appear curved in the image... and you don't like that, do you?

Let me show you a couple of image just to clarify.

The image above shows a couple of images taken with an uncalibrated stereo camera. The cameras are more or less parallel, but they are not perfectly aligned. Do you see the red point? See how it is not in the same line on the left and right images?

Now, almost all the methods used to calculate a dense disparity map rely on calibrated images, so if we try to use this images to calculate the dense disparity map we will get really poor results. You can check it on the next picture:

But now, if we apply the magic of calibration:

The image above has been rectified and undistorted (notice the black borders around the image, they are the result of removing the distortion and aligning the images so the epipolar lines are parallel and appear on the same row in bot images). See the green point? Do you see how it appears in the same row on both images?

Now, if we use this to calculate the dense disparity map:

There it is, much better results!!

To sum up, if you want to get the best out of stereo vision:

1. Make sure that your cameras are as parallel as possible.
2. Calibrate the stereo camera. See this post for instructions: http://blog.martinperis.com/2011/01/opencv-stereo-camera-calibration.html
3. Tune the parameters of your stereo matching algorithm. See this post to get an example: http://blog.martinperis.com/2011/08/opencv-stereo-matching.html
4. Have fun with it!

So you know, questions are welcome and any comments will be appreciated ;)