Highlights from Interbike 2014

Eugene Goostman chatbot claimed to have passed Turing Test

By

June 8, 2014

The Eugene Goostman chatbot, which simulates a 13-year old boy, has passed the Turing Test

The Eugene Goostman chatbot, which simulates a 13-year old boy, has passed the Turing Test

It might be time to start being nicer to your laptop, because researchers at the University of Reading are claiming that a supercomputer program has passed the Turing Test for the first time in history. On Saturday, at the Turing Test 2014 organized by the University of Reading’s School of Systems Engineering, the chatbot Eugene Goostman reportedly convinced the judges 33 percent of the time that it was a human being and not a computer.

Devised by the mathematician Alan Turing in 1950 in his paper "Computing Machinery and Intelligence,” the Turing Test is considered the gold standard for gauging how far we’ve come in the field of artificial intelligence. The test is named after Turing, but the roots of it go back to René Descartes in the 17th century. It strikes not only at questions of artificial intelligence, but also at the limits of automata in general, the question of how we know if other people possess consciousness, and even the philosophical basis of materialism.

The idea is that an interrogator communicates with two contestants, one human and the other a machine, solely by text. The interrogator asks questions of each for five minutes and then decides which one is human. If the machine wins more than 30 percent of the rounds, it’s regarded as having passed the test. So far, no computer program has ever achieved this, but on Saturday, Professor Kevin Warwick, a Visiting Professor at the University of Reading and Deputy Vice-Chancellor for Research at Coventry University, says that the chatbot Eugene Goostman scored 33 percent.

There are actually several versions of the test, each with its own rules and criteria of what constitutes success. "Some will claim that the Test has already been passed,” says Professor Warwick. "The words Turing Test have been applied to similar competitions around the world. However this event involved more simultaneous comparison tests than ever before, was independently verified and, crucially, the conversations were unrestricted. A true Turing Test does not set the questions or topics prior to the conversations. We are therefore proud to declare that Alan Turing's Test was passed for the first time on Saturday.”

The test is controversial because of the tendency of interrogators to attribute human characteristics to what is often a very simple algorithm. This is unfortunate because chatbots are easy to trip up if the interrogator is even slightly suspicious. Chatbots have difficulty with follow up questions and are easily thrown by non-sequiturs that a human could either give a straight answer to or respond to by specifically asking what the heck you’re talking about, then replying in context to the answer.

One of five supercomputer programs participating in the event, Eugene Goostman was developed in 2001 by Russian-born Vladimir Veselov, Ukrainian-born Eugene Demchenko, and Russian-born Sergey Ulasen in Saint Petersburg, and is designed to simulate a 13-year old boy in Odessa with an abrasive adolescent personality to match.

"Eugene was 'born' in 2001,” says Veselov. “Our main idea was that he can claim that he knows anything, but his age also makes it perfectly reasonable that he doesn't know everything. We spent a lot of time developing a character with a believable personality. This year we improved the 'dialog controller' which makes the conversation far more human-like when compared to programs that just answer questions. Going forward we plan to make Eugene smarter and continue working on improving what we refer to as 'conversation logic'."

The Turing Test 2014 was held in partnership with RoboLaw, an organization that examines the regulation of robotic technologies, and the judges included Red Dwarf actor Robert Llewellyn and Lord Sharkey.

"Of course the Test has implications for society today," says Warwick. "Having a computer that can trick a human into thinking that someone, or even something, is a person we trust is a wake-up call to cybercrime. The Turing Test is a vital tool for combating that threat. It is important to understand more fully how online, real-time communication of this type can influence an individual human in such a way that they are fooled into believing something is true ... when in fact it is not."

Sources: University of Reading, Eugene Goostman

Update (June 10, 2014): There is some controversy surrounding the claim that Eugene passed the Turing Test. The language of the story has been modified to reflect this.

About the Author
David Szondy David Szondy is a freelance writer based in Monroe, Washington. An award-winning playwright, he has contributed to Charged and iQ magazine and is the author of the website Tales of Future Past.   All articles by David Szondy
17 Comments

It's ironic that the test for gauging humanity in AI requires only 30% to be regarded as successful when so many of human endeavours which are tested for a pass/fail percentage require somewhere around 70%-75% to be deemed passing.

Rt1583
9th June, 2014 @ 05:32 am PDT

And the reds have it !

A true test for this no disrespect is to let the general public at it.

If Eugene was offered the opportunity to talk to thousands where the website recorded random user's votes we would see something better then 33% (which pangs of " we talked three times, and he fooled us once") Hardly a large sample set.

Nairda
9th June, 2014 @ 05:53 am PDT

It flunks if it relies on dodgy responses like "It's interesting that you say that," "Really? Tell me more" and "Why do you ask?" If I were pitted against it, the first thing I'd tell it would be that I won't accept such responses. I'll bet its reply would be precisely the kind of reply I told it I won't accept.

Paul Stregevsky
9th June, 2014 @ 08:18 am PDT

My Aunty Pat would fail the Turing test I reckon.

nutcase
9th June, 2014 @ 08:33 am PDT

The Turing Test should be administered to high school seniors as a requirement for graduation.

Also to politicians to qualify them for candidacy.

Beaugrand_RTMC
9th June, 2014 @ 10:00 am PDT

My problem with the Turing test is not that computers would eventually pass it but that so many of the bipedal hominids already here can't even come close. And yet they're allowed to breed, vote, drive, etc.

rocketride
9th June, 2014 @ 10:19 am PDT

ELIZA was a pseudo-therapist interactive program demonstrated in 1966. It relied quite a bit on reflective sentences, like those Paul Stregevsky noted above, so could be fooled by nonsense comments. e.g. human: "I have a carrot in my ear." ELIZA: "How do you feel about the carrot in your ear?" Some clinical psychology professors used ELIZA to challenge students to think about the nature of "empathy" and psychotherapy. ELIZA is now noted as one of the earliest examples of a chatbot.

http://en.wikipedia.org/wiki/ELIZA

Another somewhat different example is the computer "WATSON" which won the TV game show "Jeopardy" in 2011. WATSON has since been used in optimizing cancer treatment.

http://en.wikipedia.org/wiki/Watson_(computer)

bajessup
9th June, 2014 @ 10:31 am PDT

I think patterning an AI personality after a 13 year old boy is cheating - do you know any who could pass a turing test?

If you think about it, 33% would be pretty good for a real test - remember there is an actual human the bot is being compared to each time - 1/3 the time the judge gets it right, 1/3 it is a tossup (potentially), 1/3 the (human) judge actually gets wrong which the human contact is...

BlueGull
9th June, 2014 @ 12:55 pm PDT

Here's some tips on how to trip up a telemarketing robot which probably aren't on the Turing test:

http://newsfeed.time.com/2013/12/10/meet-the-robot-telemarketer-who-denies-shes-a-robot/

Jay_Wilson
9th June, 2014 @ 01:22 pm PDT

why is the threshold 30% and not 50%?

Abby Normal
9th June, 2014 @ 02:38 pm PDT

This is a non-story:

30% fooled after 5 minutes is a joke.

The headline should read "Everything passes by sufficiently low standards" because that is all that was established by this experiment.

Tommy Maq
9th June, 2014 @ 03:03 pm PDT

Perhaps the passing grade says more about the judge than the AI. Could Sherlock Holmes be fooled?

Don Duncan
9th June, 2014 @ 06:58 pm PDT

Exciting news, although the critical articles are now popping up!

Robin De Lange
9th June, 2014 @ 07:41 pm PDT

Given a "stupid enough" human, even ordinary 1980's Eliza would beat it. Who's the judge, and how qualified are they? Who was the human who got mistaken for a computer 33% of the time? How were they chosen and qualified? What's with the suspicious number 33%? Did they only ask 3 questions? Where's the transcript?!?!?!?!

christopher
9th June, 2014 @ 08:24 pm PDT

My critic on the Turing test is that it was designed with 20th century people in mind. Today we are highly exposed to interacting with computer programs, tv and movie actors, computer game characters, that we come to accept those artificial behaviors as natural. Not only we accept those behaviors but we also adopt it into our own personal behavior. I find that this "numbness" makes younger generations less qualified to be judges on a Turing test.

Joaquim Guerreiro
10th June, 2014 @ 02:40 am PDT

Talk to Eugene Grootsman from your phone:

https://play.google.com/store/apps/details?id=com.theworld.eugenegootsman

レインリクス ラインドウェル
12th June, 2014 @ 12:48 pm PDT
Post a Comment

Login with your gizmag account:

Or Login with Facebook:


Related Articles
Looking for something? Search our 28,500 articles