MIT's Surround Vision lets you see beyond the edges of your TV set
By Jeff Salton
April 13, 2010
A saying I heard a long time ago that has stuck with me for years (because it’s true) states: Women want to see what’s on TV; men want to see what else is on TV... which pretty much sums up the typical male's reluctance to ever give up control of the TV’s remote. Well now there's a whole new way to see what else is on TV. A new system developed by researchers at the Massachusetts Institute of Technology (MIT) called Surround Vision lets you use a separate handheld device to view additional content that doesn’t fit on the TV’s normal viewing screen.
“If you’re watching TV and you hear a helicopter in your surround sound,” says project leader Santiago Alfaro, a graduate student in the lab, “wouldn’t it be cool to just turn around and be able to see that helicopter as it goes into the screen?”
Surround Vision is intended to work with standard, Internet-connected handheld devices, like some gaming consoles and smart phones. When you want to see what (if anything) is happening off the left edge of the television screen, you can point your device in that direction and the image will appear on its screen.
This technology would work best where multiple cameras are already shooting action from many angles, for instance, live sport. The viewer could, with their handheld device, look further downfield than the rest of the viewers watching the same game.
Getting the ball rolling
Alfaro attached a magnetometer (compass) to a handheld device and wrote software that incorporated its data with that from the device’s other sensors. But Alfaro points out that devices now on the market, including the most recent version of the iPhone, have magnetometers built in, making it even easier to mass produce in the future.
Alfaro and Media Lab research scientist Michael Bove say that if the system were commercialized, the video playing on the handheld device would stream over the Internet and content providers wouldn’t have to modify their broadcasts or their set-top boxes to fit the typical screen size.
“In the Media Lab, and even my group, there’s a combination of far-off-in-the-future stuff and very, very near-term stuff, and this is an example of the latter,” Bove says. “This could be in your home next year if a network decided to do it.”
How they did it
Once Alfaro had fitted his handheld device with the requisite motion sensors, he captured video footage of the street in front of the Media Lab from three angles simultaneously – center, left and right. A television set replays the footage from the center camera. But if you want to watch a vehicle approaching from the left (off screen), you point your motion-sensitive handheld device at the left of the TV for a different perspective and, in this case, watch as the vehicle approaches (on the small screen) then enters the main field of view (on the large screen).
Since many DVDs of commercial films now come with bonus footage of scenes shot from different angles, Alfaro was also able to devise demonstrations of recent movies released on DVD with bonus footage of scenes shot from different angles that allowed him to switch between the final version of a film and alternate takes.
Bove says the next steps are to use content developed in conjunction with a number of partners, such as sports broadcasters who appear a natural fit for the system.