Photography

Neither snow, nor rain, nor gloom of night stays MIT's new TOF camera

Neither snow, nor rain, nor gloom of night stays MIT's new TOF camera
MIT students (left to right) Ayush Bhandari, Refael Whyte and Achuta Kadambi pose next to their "nano-camera" that can capture translucent objects (Photo: Bryce Vickmark)
MIT students (left to right) Ayush Bhandari, Refael Whyte and Achuta Kadambi pose next to their "nano-camera" that can capture translucent objects (Photo: Bryce Vickmark)
View 8 Images
A simple TOF camera includes a modulated illumination source, a detector for the modulated light reflected from an object, and a program to convert these two signals into a phase shift, and from this, a range (Image: B. Dodson)
1/8
A simple TOF camera includes a modulated illumination source, a detector for the modulated light reflected from an object, and a program to convert these two signals into a phase shift, and from this, a range (Image: B. Dodson)
Measuring the phase difference between the outgoing and reflected light only provides a unique result for ranges up to the width of the pulse times the speed of light (Image: B. Dodson)
2/8
Measuring the phase difference between the outgoing and reflected light only provides a unique result for ranges up to the width of the pulse times the speed of light (Image: B. Dodson)
A cartoon describing why multipath interference is bad for TOF cameras. The light rays all start at the same time, but scatter off different features of the camera field (orange from the fog, green and red from different depths in the milky glass); only the black reflects from the front surface of the object. The four light rays impact the same pixel of the sensor (Image: B. Dodson)
3/8
A cartoon describing why multipath interference is bad for TOF cameras. The light rays all start at the same time, but scatter off different features of the camera field (orange from the fog, green and red from different depths in the milky glass); only the black reflects from the front surface of the object. The four light rays impact the same pixel of the sensor (Image: B. Dodson)
MIT's new TOF camera taking a 3D image of a translucent vase in front of a wall (Photo: MIT)
4/8
MIT's new TOF camera taking a 3D image of a translucent vase in front of a wall (Photo: MIT)
TOF picture of a sign behind an optical diffuser. The reflected light looks like the figure on the left, but when only a time delay removes the light reflected and scattered from the diffuser (Photo: MIT)
5/8
TOF picture of a sign behind an optical diffuser. The reflected light looks like the figure on the left, but when only a time delay removes the light reflected and scattered from the diffuser (Photo: MIT)
MIT's new TOF camera as seen from the top and the front (Photo: MIT)
6/8
MIT's new TOF camera as seen from the top and the front (Photo: MIT)
Front-quarter view of MIT's new TOF camera (Photo: MIT)
7/8
Front-quarter view of MIT's new TOF camera (Photo: MIT)
MIT students (left to right) Ayush Bhandari, Refael Whyte and Achuta Kadambi pose next to their "nano-camera" that can capture translucent objects (Photo: Bryce Vickmark)
8/8
MIT students (left to right) Ayush Bhandari, Refael Whyte and Achuta Kadambi pose next to their "nano-camera" that can capture translucent objects (Photo: Bryce Vickmark)
View gallery - 8 images

MIT researchers have developed a new time-of-flight (TOF) 3D "nano-camera" with the ability to work with translucent objects, motion, fog, rain, and other factors in the environment that totally confuse previous TOF cameras, such as Microsoft's second-gen Kinect. The MIT Media Lab team has added these new capabilities by introducing additional information into the illuminating light beam. The resulting camera costs less than US$500 in parts.

Time-of-flight cameras, a subclass of scannerless LIDAR (a portmanteau of light radar), have been around for nearly 25 years and find application in automated manufacturing, image recognition, human-computer interfaces, collision avoidance and automatic parking in cars. The new nano-camera promises to improve on the performance and capabilities of these cameras and many additional niches are expected to open as capabilities expand and prices plummet.

A simple TOF camera includes a modulated illumination source, a detector for the modulated light reflected from an object, and a program to convert these two signals into a phase shift, and from this, a range (Image: B. Dodson)
A simple TOF camera includes a modulated illumination source, a detector for the modulated light reflected from an object, and a program to convert these two signals into a phase shift, and from this, a range (Image: B. Dodson)

A TOF camera takes a 3D photo of a scene by measuring the time required for light to travel from a source (typically a laser or LED), and then to bounce back from objects in the field of view to be detected by the TOF camera. This time delay measurement is calculated pixel by pixel, and is converted into a distance from the object to the camera roughly by multiplying the time delay by the speed of light and a geometrical factor.

A cartoon describing why multipath interference is bad for TOF cameras. The light rays all start at the same time, but scatter off different features of the camera field (orange from the fog, green and red from different depths in the milky glass); only the black reflects from the front surface of the object. The four light rays impact the same pixel of the sensor (Image: B. Dodson)
A cartoon describing why multipath interference is bad for TOF cameras. The light rays all start at the same time, but scatter off different features of the camera field (orange from the fog, green and red from different depths in the milky glass); only the black reflects from the front surface of the object. The four light rays impact the same pixel of the sensor (Image: B. Dodson)

This simple method, however, is not capable of determining distances when the field of view contains objects which are not opaque. Known as multipath interference, the problem arises when a single pixel in the image is illuminated by light returning from a variety of distances.

A simple example would be a clear glass vase in front of a wall, as appears in the lead picture above. The light reflecting from the vase does not travel as far as the light that reflects from the wall behind it and shines through the vase. This one pixel thus contains time delay information for two different surfaces, preventing a simple TOF camera from extracting accurate distance information.

MIT's new TOF camera as seen from the top and the front (Photo: MIT)
MIT's new TOF camera as seen from the top and the front (Photo: MIT)

To sort out the distance information when the light illuminating a pixel comes from paths having different lengths requires considerably more information than is provided by the simple TOF camera. The MIT Media Lab team, led by Associate Professor Ramesh Raskar, has figured out how to insert "considerably more information" into the modulation of the light that is illuminating the 3D scene.

It works like this. A complex, but still periodic, modulation is applied to the light illuminating a 3D scene. Among other things, the modulation of the light intensity changes between the early part and the late part of the periodic cycle. This means that if light arrives at a pixel from reflections at two different distances, the light from the two paths will have different modulations in time, allowing the two signals to be separated from one another.

The output of the TOF will then be that reflections from two different points in space were received at the same pixel, and the phase shift from the illumination for each is unambiguously determined. The spatial resolution for depth is about 2 cm in MIT's demonstration camera, which corresponds to a time resolution of about 700 picoseconds.

This means that the shape, outer dimensions, and inner dimensions of a translucent vase can be determined by such sophisticated TOF cameras. If you picture a wall through a fishtank, you will be able to determine the distance of the fish tank from the wall, as well as the size and shape of the fish tank.

TOF picture of a sign behind an optical diffuser. The reflected light looks like the figure on the left, but when only a time delay removes the light reflected and scattered from the diffuser (Photo: MIT)
TOF picture of a sign behind an optical diffuser. The reflected light looks like the figure on the left, but when only a time delay removes the light reflected and scattered from the diffuser (Photo: MIT)

The MIT TOF camera can even see an object on the far side of an optical diffuser by ignoring the time frame in which light returns from the diffuser, allowing the camera to focus on the object. These and many other feats impossible for a simple TOF camera are now within our grasp. In fact, many existing TOF cameras could adopt this mode of operation with a field software upgrade.

The prototype TOF camera was built in the MIT Media Lab using components on hand. There are now available less expensive versions of the components used, and the team estimates that a DIY'er could probably assemble a version of their advanced TOF camera for about US$500 worth of parts.

The nano-camera is explained in the following video.

Source: MIT Media Lab

Nanophotography/Camera Culture/MIT Video

View gallery - 8 images
1 comment
1 comment
Edgar Castelo
These guys are GIANTS!