The Loebner Prize turned Turing's game into a yearly contest

The conversation

Hugh Loebner agreed with the Cambridge Center for Behavioral Studies in 1990 to fund an annual contest built around Alan Turing's imitation game, and the first contest ran in 1991. The Loebner Prize's own history page, retrieved from its archive because the live site has since changed, states that Loebner "pledged a Grand Prize of $100,000 and a Gold Medal… for the first computer whose responses were indistinguishable from a human's," while "each year an annual prize of $2000 and a bronze medal is awarded to the most human-like computer." Its published list of winners opens with Joseph Weintraub of Thinking Systems Software, who took the prize in 1991, 1992, 1993, and again in 1995.

What the documents show

The contest's own page defines what a win actually means: "the winner of the annual contest is the best entry relative to other entries that year, irrespective of how good it is in an absolute sense." No entry has yet won Loebner's larger, one-time Grand Prize for indistinguishability; the annual bronze medal is a relative ranking among that year's finalists and human confederates, judged by a panel that talks with each and ranks all of them from most to least humanlike. The page itself points readers to a named critique of the contest's format, Stuart Shieber's "Lessons from a Restricted Turing Test", published in Communications of the ACM in 1994.

The system boundary

Shieber's paper describes the boundary the early contests built into the test itself: entries were restricted to a narrow topic and, in some years, to a deliberately "whimsical conversation" style, with judges asked to avoid deceptive tactics of their own. The restriction was meant to make an otherwise impossibly broad task tractable for judges within a short session, not to certify general understanding. Shieber argues the restriction instead let a program's evasive, associative replies pass as cleverness rather than as a documented weakness, because the topic limit removed the judge's ability to press on any one subject.

Where it fails

Shieber's own conclusion is blunt: "people are easily fooled, and are especially easily fooled into reading structure into chaos," and he argues a restricted-topic, time-limited win demonstrates the judges' credulity under those specific constraints more than it demonstrates machine understanding. A reader who encounters "won the Turing test" as a credential should ask what test, exactly.

• Was the win a relative annual ranking among that year's entries, or the unclaimed Grand Prize for unrestricted indistinguishability?
• Was the topic or conversational style restricted, and did the judges know the restriction in advance?
• How long did each judge interact with each entry, and does the source say what a longer session would change?

Both documents, one the contest's own record and one a formal critique the contest links to itself, agree that an annual Loebner win is a claim about relative performance under stated constraints, not a settled verdict on machine intelligence.