News Column

Did Eugene the computer program pass Turing test?: Judges believed program was a Ukrainian teenager Other experts say software is entertaining but stupid

June 10, 2014

Ian Sample and Alex Hern

A computer program named Eugene Goostman which imitates a Ukrainian teenager with a quirky sense of humour and a pet guinea pig has won an artificial intelligence competition at the Royal Society in London.

The program convinced 10 out of 30 judges at the nation's most prestigious scientific institution that it was a real person in a series of online chats lasting five minutes each. The event's organisers, from Reading University, claimed Eugene had made history by passing the Turing test, a significant goal in the field of artificial intelligence, though other scientists begged to differ.

Regardless of the program's success, the performance was an improvement on Eugene's attempt to win an AI competition in 2012 when it expressed its love of Eminem and hatred for Britney Spears, and mentioned a pet guinea pig which could squeal Beethoven's Ode to Joy.

Proposed by Alan Turing, the wartime codebreaker and computing pioneer, the Turing test challenges computer scientists to create a program that is indistinguishable from a person in its conversational ability. The goal sidesteps more obscure questions about the nature of the mind, and focuses attention on how it behaves.

Computer scientist Vladimir Veselov began work on Eugene in 2001, a year after leaving his home in Russia for the US.

The program analyses questions it receives, and searches a "knowledge base" for material before compiling a response. Some of the time it will ask a clarifying question, or draw on a stock response from memory. During the tests each judge sat down at a pair of computers and typed in questions. One computer was linked to another with a person at the keyboard, while the other was running a program that provided replies.

The judges included Lord Sharkey, who campaigned for Turing's posthumous pardon over a conviction for homosexuality, and Robert Llewellyn, who played a neurotic robot called Kryten in the television series Red Dwarf.

Declaring that Eugene had passed the Turing test, Prof Kevin Warwick of Reading University said it was fitting that such an important landmark had been reached at the Royal Society.

But one judge, Prof Aaron Sloman, a philosopher and researcher on artificial intelligence at Birmingham University, was unimpressed. Sloman said he took part in the experiment to see how much progress had been made with so-called "chatbots". Speaking about Eugene, he said: "It has kept some - not all - who try it out entertained for more than five minutes. But it is essentially stupid and incompetent, no matter how many people it fools for how long."

Stevan Harnad, professor of cognitive sciences at the University of Quebec in Montreal, said that whatever had happened at the Royal Society, it did not amount to passing the Turing test. "It's nonsense, complete nonsense," he said. "We have not passed the Turing test. We are not even close."

In 1950 Turing predicted that in about 50 years' time computer conversations could pass as human around 30% of the time. But he said that a statistical survey like a Gallup poll to decide if a machine could think was absurd.

"Turing's insight was that the way to explain how the mind works is to design a system that can do whatever the mind can do," Harnad said. "That includes all of our verbal capacity, as well as the sensorimotor, or robotic, capacity in which it is grounded. Not for five minutes, but for a lifetime."

John Denning, who worked with Veselov on Eugene, defended the program. "I think we passed 'a' Turing test, but I don't know if it's 'the' Turing Test," he said. "Is Eugene smarter than a person? No. You're not going to put your life in the hands of a 13-year-old who makes wisecracks and has an odd sense of humour."

News of Eugene's success crashed the server it was hosted on over the weekend. Asked if it marked the rise of the machines that would spell the end of humanity, Denning said: "We have been looking at logs of people chatting with Eugene. What people say does not bode well for the future of humanity. It's pretty startling what people will say to robots. People say paedophilic things, things about Eugene's lineage."

Marvin Minsky, one of the most revered names in artificial intelligence, told the Guardian: "Nothing is learned from poorly designed 'experiments'. Ask the program if you can push a car with a string. And, if not, then, why not?"


AI has yet to behave like HAL in 2001: A Space Odyssey Photo: MGM/Everett/Rex

For more stories covering the world of technology, please see HispanicBusiness' Tech Channel

Source: Guardian (UK)

Story Tools Facebook Linkedin Twitter RSS Feed Email Alerts & Newsletters