Computers

Creative AI: The robots that would be painters

Creative AI: The robots that would be painters
040502 is a 2004 painting of pigment on paper by the robotic, artificially intelligent painter AARON
040502 is a 2004 painting of pigment on paper by the robotic, artificially intelligent painter AARON
View 22 Images
A painting by Harold Cohen's robot painter AARON, which sold at auction in May 1995 at a reserve price of US$2,000
1/22
A painting by Harold Cohen's robot painter AARON, which sold at auction in May 1995 at a reserve price of US$2,000
Another painting by AARON that sold in 1995
2/22
Another painting by AARON that sold in 1995
And a third painting by AARON
3/22
And a third painting by AARON
AARON learned to use color in a decorative motif in 1992 (Photo: Becky Cohen)
4/22
AARON learned to use color in a decorative motif in 1992 (Photo: Becky Cohen)
Harold Cohen watches AARON paint in 1995
5/22
Harold Cohen watches AARON paint in 1995
For many years Cohen actually repainted AARON's work, which before the development of his painting machine was done on a much smaller scale (Photo: Becky Cohen)
6/22
For many years Cohen actually repainted AARON's work, which before the development of his painting machine was done on a much smaller scale (Photo: Becky Cohen)
By October 1994, Cohen had figured out how to watch AARON's internal representation of paintings form on the screen; the variations in colored areas indicate the path taken by the simulated brush
7/22
By October 1994, Cohen had figured out how to watch AARON's internal representation of paintings form on the screen; the variations in colored areas indicate the path taken by the simulated brush
Meeting On Gauguin's Beach, from 1988, was featured in the Gordon and Gwen Bell collection (Photo: Becky Cohen)
8/22
Meeting On Gauguin's Beach, from 1988, was featured in the Gordon and Gwen Bell collection (Photo: Becky Cohen)
A far more recent painting by AARON, Wasteland, from 2007
9/22
A far more recent painting by AARON, Wasteland, from 2007
040502 is a 2004 painting of pigment on paper by the robotic, artificially intelligent painter AARON
10/22
040502 is a 2004 painting of pigment on paper by the robotic, artificially intelligent painter AARON
In Zana's Room, an oil over pigment ink on canvas painting by AARON from 2012
11/22
In Zana's Room, an oil over pigment ink on canvas painting by AARON from 2012
Final Approach, one of AARON's most recent paintings, from 2013
12/22
Final Approach, one of AARON's most recent paintings, from 2013
A painting of Auguste Rodin's The Thinker sculpture by robot painter e-David
13/22
A painting of Auguste Rodin's The Thinker sculpture by robot painter e-David
A painting by e-David of the vulture from Franz Kafka's short story Der Geier
14/22
A painting by e-David of the vulture from Franz Kafka's short story Der Geier
Another painting by robot painter e-David
15/22
Another painting by robot painter e-David
The Interactive Robotic Painting Machine responds to sounds in the environment as it paints (Photo: Benjamin Grosser)
16/22
The Interactive Robotic Painting Machine responds to sounds in the environment as it paints (Photo: Benjamin Grosser)
One of The Painting Fool's earliest projects involved producing 222 portraits of the lead actress from the film Amelie, each with a different emotional value
17/22
One of The Painting Fool's earliest projects involved producing 222 portraits of the lead actress from the film Amelie, each with a different emotional value
Another portrait from The Painting Fool's Amelie series
18/22
Another portrait from The Painting Fool's Amelie series
The Painting Fool's The Dancing Salesman Problem is part of a "No Photos Harmed" collection made for a 2011 exhibition; it was printed and assembled across 317 photo-sizes pieces of paper and many visitors believed it had been hand-painted
19/22
The Painting Fool's The Dancing Salesman Problem is part of a "No Photos Harmed" collection made for a 2011 exhibition; it was printed and assembled across 317 photo-sizes pieces of paper and many visitors believed it had been hand-painted
This painting comes from a collage The Painting Fool made with Flickr images that it found after reading a Guardian article on the war in Afghanistan and extracting keywords
20/22
This painting comes from a collage The Painting Fool made with Flickr images that it found after reading a Guardian article on the war in Afghanistan and extracting keywords
A portrait of disgust from The Painting Fool's You Can't Know My Mind exhibition in July 2013
21/22
A portrait of disgust from The Painting Fool's You Can't Know My Mind exhibition in July 2013
Another emotionally-aware portrait, hastily drawn by The Painting Machine
22/22
Another emotionally-aware portrait, hastily drawn by The Painting Machine
View gallery - 22 images

Painting might be the last thing you'd expect computers to excel at. It's abstract, expressive, and tied to cultures, psychology, and subjectivity, whereas computers are objective, precise, and governed by the rules of mathematics. Painting, with its emotional reasoning and unclear meanings, appears to be the antithesis of a feeling, logical computer. But they aren't so far apart as they seem. Painting and other forms of visual art owe much to areas of mathematics such as geometry and perspective, and the algorithms that computers adhere to can in fact be made to generate images as varied and subtle as a human painter.

Much like its musical counterpart, algorithmic art dates back to the time before computers were commonplace and in its purest sense requires no artificial intelligence whatsoever. You've probably seen examples of fractal art, which replicates patterns in a recursive, algorithmic way to often-stunning results that vary in appearance from geometric to organic to alien.

Traditionally, algorithmic art involves a human coming up with a concept that an algorithm then generates or visualizes either from scratch or based on existing material. An extreme example of this is Nagoya University researchers Yasuhiro Suzuki and Tomohiro Suzuki's evolutionary painting algorithm, which takes example paintings of a given style and progressively mutates them – cutting and splicing and flipping elements, throwing out at each evolution any images that don't match the user's initial stylistic choices. But algorithmic art is more commonly used in the sense of images that are generated by computer code written by people like Dextro, who is one of the leading practitioners of algorithmic/generative art.

As with music, game development, and writing, much of the attention from artists and scientists has been placed upon algorithms and intelligent tools that augment the artist's creativity. The Processing programming language was designed as an electronic sketchbook for artists and designers, while some of the better-known apps for algorithmic artists include Ultra Fractal, Scribble, and Fragmentarium.

There are now over a dozen separate kinds of algorithmically-based art, including fractal art, genetic art, cellular automata, proceduralism, and transhumanist art. And there are multitudes of websites such as The Algorists, Algorithmic Worlds, and The compArt database Digital Art that celebrate the work of artists who use algorithms.

Harold Cohen watches AARON paint in 1995
Harold Cohen watches AARON paint in 1995

But there are some who would teach computers to paint like humans, to push them beyond the point of being an extension of the artist and into the territory of artist themselves. The pioneer in this regard is a former artist and University of California San Diego professor called Harold Cohen. He started working on an art-creating program called AARON in 1973, while a visiting scholar at Stanford University's Artificial Intelligence Lab.

AARON's capacity to paint improved year after year as its maker taught it more difficult or complex techniques. It learned to situate objects or people in 3D space in the 1980s, and could paint in color from 1990 onwards. In time its paintings found their way into many of the world's major art museums and onward into the hands of private collectors who paid hundreds or even thousands of dollars for AARON's art.

AARON paints not with pixels, we should note, but with real paint on an actual canvas. Cohen built a painting machine for his painting AI. He taught it to mix paint (fabric dyes, not oil), and even gave it an imagination of sorts. Enough of one, at least, that it can paint still life and portraits of human figures without photos or other human input as reference.

AARON learned to use color in a decorative motif in 1992 (Photo: Becky Cohen)
AARON learned to use color in a decorative motif in 1992 (Photo: Becky Cohen)

"I was always fascinated by the fact that we can look at a drawing and 'see' a face, when we are actually looking at a few dirty marks on a piece of paper," he once told PBS in an interview. It was through thinking about that mystery and how we humans develop the ability to generate those symbolic marks that AARON was born and further developed over a span of four decades.

Cohen never showed AARON any images, but rather taught his robot with lists of object/body elements and the relationships between them. Fundamental rules, essentially, that allow a robot that has never seen a human or a chair or flower to nevertheless paint something that looks like an abstract representation of those things.

AARON paints whatever it wants within the confines of its knowledge, driven only by its limited version of an imagination and not at any point instructed explicitly by Cohen or anybody else. And its end product, as you may have noticed from the images included in this article, has matured from childish lines and squiggles to portraits of people posing awkwardly near pot plants and most recently to abstract explorations of color and shape.

Final Approach, one of AARON's most recent paintings, from 2013
Final Approach, one of AARON's most recent paintings, from 2013

Others have tried to build on Cohen's pioneering work with their own AI painters. In 2011, for instance, Benjamin Grosser debuted his Interactive Robotic Painting Machine, which paints abstract pictures with oil on canvas and, curiously, responds to the sounds in its environment, using those sounds of conversations, background audio, and its own mechanisms to influence the painting as it progresses. You can see a video of it in action below.

Interactive Robotic Painting Machine

Another project, e-David by Oliver Deussen and Thomas Lindemeier from the University of Konztanz in Germany, uses a visual feedback loop to try to mimic how a human painter creates their work (see the video below).

e-David Robot Painting

The idea, they explain on e-David's website, is that the technical aspects of painting are essentially "optimization processes in which color is manually distributed on a canvas until the painter is able to recognize the content, regardless if it is a representational painting or not." E-David does not concern itself with such lofty ambitions as creative intent, but another ambitious project has exactly that in mind.

Goldsmiths College, University of London professor Simon Colton brought The Painting Fool to life back in 2001, with his initial intention simply being to write software that could convert photos into art. The program gradually took on greater sophistication, moving from graphics filters and other elements you'd expect in programs like Photoshop into emotional modelling where photos could be altered according to keywords that were first supplied by an external agent and then self-determined by the AI.

After several years of development, The Painting Fool was granted an imagination – an ability to interpret ideas more loosely and to evolve its scene depictions without necessarily having any specific reference image – and then the ability to paint with intent. Colton describes one example where the software downloaded a news story on the war in Afghanistan and identified keywords that it used to find relevant images on Flickr. The Painting Fool then developed a painterly rendition of these images in a collage, juxtaposing a fighter plane with an explosion, a family, an Afghan girl, and a field of war graves.

This painting comes from a collage The Painting Fool made with Flickr images that it found after reading a Guardian article on the war in Afghanistan and extracting keywords
This painting comes from a collage The Painting Fool made with Flickr images that it found after reading a Guardian article on the war in Afghanistan and extracting keywords

Its intent here was flimsy in that it was more representing the journalist's work visually, but it was to Colton and his colleagues an indication that AI could indeed paint with intent if only you give it the ability to form opinions (especially through natural language processing and analysis of news media).

That's an important step because their long-term goal with The Painting Fool is to have it not be notable because it is a computer pretending to be an artist, as is the case now, but rather because it is a creative artist in its own right. They played with the concept of artists being moody creatures with an exhibition called "You Can't Know my Mind" in which The Painting Fool produced portraits of visitors that were affected by the previous 10 news articles it had read on The Guardian website. (It was reading during the exhibit.)

If the software was reading a lot of articles with a very negative sentiment, it would refuse to paint, while the rest of the time its paintings were directed (by asking the sitter for a particular expression) and styled according to one of a selection of mood-appropriate adjectives. And at the end of each painting it self-assessed how well it went in achieving the goal, whether it was successful in putting intent into practice.

The exhibition got covered on Wired UK and Vice, among other publications, with Colton noting to Vice reporter Nadja Sayej that events like this are important to illustrate that the software embodies "certain intelligent behaviors associated with creativity, such as skill, appreciation, imagination, learning, intention, and reflection."

The Painting Fool is not human and it never will be, but it can appreciate human emotions and develop crude mechanisms for guiding its own painting process by self-defined moods.

As with AI composers, computer programs that paint their own paintings have developed a nuance in recent years that in a child or apprentice we might call magnificent. But in computers there is still a reluctance to see the beauty.

When I spoke to David Cope and Francisco Javier Vico for our creative AI in music composition feature, both noted considerable opposition and suspicion that's been mounted at their work, sometimes through anti-computer sentiment, but more often because of a tendency to downplay the achievement as a poor imitation of a professional human artist or composer or as a joke not to be taken seriously.

One of The Painting Fool's earliest projects involved producing 222 portraits of the lead actress from the film Amelie, each with a different emotional value
One of The Painting Fool's earliest projects involved producing 222 portraits of the lead actress from the film Amelie, each with a different emotional value

In AI-authored art there is the same problem, with many dismissing the work of AARON, The Painting Fool, and others of the sort as uncreative or imitative because they believe that computers are incapable of creative intent. But beyond the layered, unresolved arguments about the nature of creativity, which we will dig into at the end of this series of feature articles, there lies great potential for us to appreciate the artistic collaboration that is possible between humans and computers.

Computers can create subtle, nuanced, original artwork, be it from imaginative interpretations of human instructions or from emergent combinations of mathematical sequences. But the question is: Will we accept them as such? Or will the computer artist always remain an imitator and/or an enabler of the real, human artist?

Next week we tackle a different kind of artistry with a deep dive into the issues surrounding computational architecture and design.

View gallery - 22 images
1 comment
1 comment
TonyLarsson
The idea that even something so "human" as art would be doable by AI really challanges the notion that humans are uniquely capable in some fundamental way.
There is already an online art gallery where you can see all the crazy stuff AI can create today: www.aimakes.com
As we move further into the future those who hold the belief that there is something unreplicatable about the human mind will face hard clashes with reality.