Science

Photo editing tool shows viewers what the camera couldn't see

Photo editing tool shows viewers what the camera couldn't see
An original photo of a paper crane (left) and an artificially-rotated version of it
An original photo of a paper crane (left) and an artificially-rotated version of it
View 1 Image
An original photo of a paper crane (left) and an artificially-rotated version of it
1/1
An original photo of a paper crane (left) and an artificially-rotated version of it

Many people are already annoyed when characters on TV cop shows "zoom in and enhance" on a photo, to reveal a level of detail that could never really have been captured by the camera. Thanks to software developed at Carnegie Mellon University, however, it's now possible to actually turn objects in a photo around ... seemingly revealing sides of them that were facing away from the camera when the picture was taken.

Before you get too excited, the system doesn't actually reveal other sides of the very object that was in the photo. Instead, it utilizes existing third-party stock 3D models of similar objects, found online. This means that it's currently limited to fairly common items, although more and more such models are being created on a daily basis for use in processes such as 3D printing and animation.

The software assesses the geometry of the existing photo, then manipulates the model accordingly, so the resulting blended image is very realistic. This includes not only altering the model's size and shape (along with that of the original image), but additionally changing its color, surface texture and lighting.

Along with its use in photographs, the technology could reportedly also be applied to objects in paintings, or be used to animate one object in an otherwise still photo.

"In the real world, we're used to handling objects – lifting them, turning them around or knocking them over," said Natasha Kholgade, a Ph.D student who is also the lead author of the study. "We've created an environment that gives you that same freedom when editing a photo."

More information is available in the following video.

Source: Carnegie Mellon University

3D Object Manipulation in a Single Photograph using Stock 3D Models: Full Length Video

5 comments
5 comments
Mark A
So how is the area hidden behind the object rendered?
Brian M
Will it be admissible in court! Think this is called have too much time on your hands! Surely there must be more pressing matters for research than this as it adds nothing to the information content, worse could actually give wrong information. For example looking to buy a second hand car? Use the software to view the other sider of the car, you don't get the reality of there being a big dint in the side, just the stock shot of a perfect body.
Goodness knows what would happened if applied to photos on dating sites.......
pmshah
"Many people are already annoyed when characters on TV cop shows "zoom in and enhance" on a photo, to reveal a level of detail that could never really have been captured by the camera." You are right about that. I have been in photographic field for more than 45 years. There in no way in the world you can extract details that were never there in the first place. Like taking a blurred / out of focus picture and enhancing it to identify the person.
But then most of the cop shows are perpetuating fraud all the time. How do you hot wire (under the dash) and drive away a car when it has steering lock released only by steering column mounted ignition switch? The list goes on. But hey, this is Hollywood.
Milton
@ Brian M:
In order for this software to work you need a 3D model of the object being manipulated. And 3D modeling an entire object for the sake of photo manipulation seems like a very elaborate process.
So unless the object being manipulated already had an existing 3D model that was readily available to download, I don't think you have to worry about this level of manipulation.
Also: Photoshop would be a hell of a lot easier to fix a "dent" in a car. You would only use the software in this article if you wanted to re-orient said object. And even still, you may be better off rendering the 3D model in Hypershot and placing it into the scene w/ photoshop than going through this process.
I'd like to see the interface and start-to-finish of photo manipulation for an object that does not already have a 3D model online.
Firehawk70
Deckard: Enhance 224 to 176. Enhance, stop. Move in, stop. Pull out, track right, stop. Center in, pull back. Stop. Track 45 right. Stop. Center and stop. Enhance 34 to 36. Pan right and pull back. Stop. Enhance 34 to 46. Pull back. Wait a minute, go right, stop. Enhance 57 to 19. Track 45 left. Stop. Enhance 15 to 23. Give me a hard copy right there.