New software dramatically simplifies addition of objects to photos


December 12, 2011

Shadows from set geometry are accurately depicted on added objects

Shadows from set geometry are accurately depicted on added objects

Image Gallery (14 images)

For more than twenty years, the software program Photoshop has been the industry standard for seamlessly manipulating images, especially the removal of unwanted items like blemishes, wires and telephone poles. When it comes to adding something to a photo, however, the process is still rather involved. Now, a team from the University of Illinois at Urbana-Champaign (UIUC), led by PhD candidate Kevin Karsch, is presenting a new algorithm at Siggraph Asia 2011 that promises to radically simplify the procedure of believably incorporating new or "synthetic" objects in still shots and the results are quite impressive.

To appreciate the elegance of the UIUC team's contribution, it's helpful to understand just what an improvement it is over current techniques. Look at any effects-heavy feature film and you'll see instances where computer-generated objects, be they giant robots, dinosaurs or futuristic vehicles, were added to previously-shot footage (also called a background plate in the Biz). In order to sell the illusion that the added objects were there in the first place, technicians typically need to gather copious amounts of on-set data (e.g. lens height and focal length, size of on-set objects and direction/color temperature of visible light sources, etc.) from which they recreate a virtual environment. When this is done poorly, it's a dead giveaway that the added object is fake and was inserted after the original image or footage was shot.

The genius of the Karsch team's approach lies in what the user doesn't have to provide in order to create believable additions. (Currently, the software only handles still images, but plans are in the works to tackle film and video sequences before long). Starting with a still photo about which nothing is known, novice users have been able to add virtual subjects in just minutes with results good enough to fool experts. That's great news for graphic artists, but maybe not so thrilling for folks like crime scene investigators who'll soon have to doubt the veracity of every object in frame!

To pull off this digital sleight-of-hand, the process is broken up into intuitive stages, the first being the creation of 3D geometry from 2D data. "The main idea is we want to take our picture and we want to get a 3D model of the scene from it," Karsch explained. "We first apply an automatic estimation procedure which tries to find the boundaries of the scene - it tries to find the walls, the ceiling, the floor." Since that's occasionally inaccurate, the software also includes an interface that allows the user to refine the estimate.

"With two vanishing points, we can recover the focal length of the camera, and then using the rest of the input we can also recover a rough representation of the 3D geometry of the scene: where the floors are at, where the walls are at and where the ceiling is at," Karsch added.

Next, the Lighting Stage requires the user to identify light sources in the image, whether in frame or not. In the traditional computer graphics approach, lighting sources and characteristics are identified by placing a chrome reference ball in front of the camera. The UIUC team's approach eliminates the need for that and a host of other steps as it calculates a decent representation of the actual lighting conditions.

If the object to be added will land behind another opaque item, the Occluding Object Stage (in which the user basically scribbles on the objects that will block part of the object to be added) comes next. Also called a spectral matting procedure, this generates an object mask which, when applied, creates the illusion that the added object is behind the masked one.

Enabling beginners to generate professional results with minimal time and effort is sure to make the Karsch team's algorithm a big hit in the graphics world. Judging from the sample images provided from the team's paper, the approach adeptly handles just about every lighting condition conceivable, even for transparent and shadowed subjects. For those of you itching to get your hands on it, plans are in the works to create an online version that anyone can use. In the meantime, it seems we'll just have to make do with run-of-the-mill reality!

Source: Kevin Karsch

Check out the video below for a look at the software in action:

About the Author
Randolph Jonsson A native San Franciscan, Randolph attended the U.S. Naval Academy at Annapolis, Maryland before finding his way to the film business. Eventually, he landed a job at George Lucas' Industrial Light + Magic, where he worked on many top-grossing films in both the camera and computer graphics departments. A proud member of MENSA, he's passionate about technology, optimal health, photography, marine biology, writing, world travel and the occasional, well-crafted gin and tonic! All articles by Randolph Jonsson

Looks amazing but what is the name of this new software? We have the exhibition details and the developers name, but not the product name!

And price? (Holding my breath here)

Janet Haines

Not gonna lie, but it does look kinda fake.

Each image put into the stills is a 3d model. Of which you can alter easily for lighting and shading and drop ontop of photos.

Not to mention it opens with a video despite the article mentioning they haven\'t been done yet....

Look keyshot up ;)

Mark Penver

I don\'t need this for a second, but it\'s so amazing that I will definitely buy one as soon as it is released, just for the hell of having fun with it.

Krister Knutars now we can\'t believe ANYTHING we see. Amazing tech. I\'m sure it will be put to nefarious use immediately! LOL


Pretty awesome. Now they just need to make it work for video and make a plug in for Premiere and/or After Effects... Or simply sell the tech to Adobe so they can add it.

Dave Andrews

...thus making it easier for criminals to fake documents or photographic evidence. Before this is released, they should develop the algorithms to detect inserted objects just as easily.

William Lanteigne

Guys in future beware what you see in any picture... digital or not!

Leong Hee Chan

First step on the road to criminals producing fake incriminating videos of influential and wealthy people to use for blackmail. Which of course the gullible sorts who\'ll believe absolutely anything - except for the truth - will believe in spite of proof the videos are fake.

Gregg Eshelman

this was demo\'d (? leaked?) months ago ..

at least now we know the team responsible for it ..


I have been doing this for quite a while now using two picture editing programmes. See egsample in videos at youtube at

Donald Thomas
Post a Comment

Login with your Gizmag account:

Related Articles
Looking for something? Search our articles