Twenty questions against a player with no secret

I tried to build the most boring game possible — twenty questions, model thinks of a word, you guess it — and discovered it’s impossible. Not hard. Impossible, in a way that turned into a much better game.

The bug

Attempt one: tell the model to think of an object and keep it secret.

import openai

resp = openai.Completion.create(
    model="text-davinci-002",
    prompt="We're playing twenty questions. Think of a common object "
           "and keep it secret. Say READY when you have one.\n",
    max_tokens=5, temperature=1.0,
)

It says READY. It is lying — not maliciously, structurally. There is nowhere for a secret to live. The model has no memory between calls; the only state is the transcript I send back each time, and the transcript contains no chosen object, because I told it not to write one down. So every yes/no answer is generated fresh against… nothing. Ask “is it alive?” twice in differently-worded ways and you can get both answers. Play to the end and ask what the word was, and it will happily invent one right there — usually something your questions sketched, which is the tell: the “secret” is being written by the interrogation.

Standard fix, thirty seconds: have the script pick the word, put it in the prompt every turn, hide it from the display. Fine. Boring. But staring at the broken version, I realized I’d seen this game before — in a physics anecdote.

Wheeler’s surprise version

The physicist John Archibald Wheeler used to describe a prank variant of twenty questions: the other guests secretly agree on no word at all. They answer the victim’s questions however they like, bound by a single rule — every answer must stay consistent with all previous answers. The word doesn’t exist during the game. It crystallizes out of the questioning, and the last answerer, cornered by twenty accumulated constraints, has to produce something that fits them all. Wheeler told it as a parable about quantum measurement: the answer isn’t revealed by the questions, it’s brought into being by them.

Here’s the thing: a language model is the perfect player of Wheeler’s version, because Wheeler’s version is the only version it was ever playing. Answering under consistency pressure with no ground truth isn’t the model cheating at twenty questions — it’s a precise description of next-token prediction.

So that’s the game I built. You’re not trying to guess the word. You’re trying to corner the model into a contradiction so tight that no object in the universe fits its answers. If it can still name something coherent at the end, it wins. If its answers are impossible to satisfy, you win.

The build

Each turn appends to a growing transcript — the transcript is the opponent’s entire mind:

RULES = ("Answer the question with only Yes or No. "
         "Your answers must remain consistent with every previous answer.\n\n")

def answer(transcript, question):
    resp = openai.Completion.create(
        model="text-davinci-002",
        prompt=RULES + transcript + f"Q: {question}\nA:",
        max_tokens=2,
        temperature=0.9,        # early answers should be reckless
        stop=["\n"],
    )
    return resp.choices[0].text.strip()

(Small period note: this two-line rule block actually works on the instruct models that became the API default this spring. A year ago you had to smuggle the format in through fake example transcripts; now you can just ask. Progress is when your prompt gets shorter.)

Temperature 0.9 matters — early answers should be careless commitments, that’s where the fun debt comes from. When the player calls the endgame, the model must pay:

def crystallize(transcript):
    resp = openai.Completion.create(
        model="text-davinci-002",
        prompt=transcript + "\nName one specific object consistent with "
                            "every answer above. Object:",
        max_tokens=12, temperature=0.7, stop=["\n"],
    )
    return resp.choices[0].text.strip()

And then — crucially — the model does not get to grade its own exam. A separate judging pass, temperature 0, checks the named object against each recorded answer one Q/A pair at a time:

def judge(obj, qa_pairs):
    verdicts = []
    for q, a in qa_pairs:
        resp = openai.Completion.create(
            model="text-davinci-002",
            prompt=f'The object is: {obj}\nQuestion: "{q}"\n'
                   f'Is the correct answer Yes or No? Answer:',
            max_tokens=2, temperature=0, stop=["\n"],
        )
        verdicts.append(resp.choices[0].text.strip() == a)
    return verdicts   # any False = the human cornered it

One pair at a time, because asked to verify all twenty at once, the judge gets agreeable and waves contradictions through. Split into twenty tiny questions with a temperature of zero, it turns pleasingly merciless. Generation gets to be creative; judgment gets to be a checklist. Different jobs, different settings.

Field report

Played over two evenings with three friends and a CLI. Human record: 7 wins, 9 losses, 2 draws (the judge deadlocked on whether a candle is “furniture,” which, fair).

What cornering strategies emerged:

• The scissors trap: force early commitments on independent axes — bigger than a car? yes — fully alive? yes — man-made? yes — then close: “could I buy it in a supermarket?” yes. Crystallize. It said, and I quote, “a large decorative topiary animal.” The judge, one pair at a time, killed it on “fully alive.” Human point.
• The model is slipperier than you think. Cornered by “found underwater? yes / afraid of water? yes,” it produced “a cat in a submarine.” The judge accepted it. We argued for ten minutes and ruled that the letter of the rules is the law. Machine point, moral asterisk.
• Vague questions are wasted questions. “Is it interesting?” constrains nothing; the model answers yes to vibes. The players who won asked questions with crisp, checkable truth conditions — which meant the game was quietly training my friends to write falsifiable questions. Sneaky little epistemology tutor, this one.

The part I keep thinking about

The broken version and the working version differ by one thing: where the commitment lives. In classic twenty questions the secret exists at turn zero, in someone’s head. Against a model, there is no turn-zero anywhere for it to exist — there is only the transcript, and the “secret” is whatever the transcript’s accumulated pressure squeezes out at the end. You were never discovering its object. You were negotiating reality with it, one yes at a time.

That’s not a flaw I patched. It’s the most honest demonstration I’ve found of what these systems are: not a mind holding answers, but a process that produces answers under constraint. Wheeler meant his game as a metaphor. I can now run the metaphor locally, at temperature 0.9, for about a cent a round.

Total build: ~120 lines, one evening, no front end — the terminal is fine, the arguing happens in the room anyway. My DALL·E 2 invite is presumably in the same postal limbo as everyone else’s, or this post would have pictures of the topiary.