Two liars, one dungeon: making GPT-3 and Stable Diffusion agree on a world

A month ago Stability released the Stable Diffusion weights — actually released them, a real checkpoint you download and run, no waitlist, no API meter. After two years of renting intelligence by the token, having a model on my own GPU feels almost transgressive. It draws anything, in about seven seconds, for the price of electricity.

So naturally I tried to build a game with it. A tiny dungeon crawler: GPT-3 writes each room, Stable Diffusion illustrates it, you click exits and descend. Text model as dungeon master, image model as court painter. Three weekends later I have something playable, and a hard-won thesis:

Generation was never the problem. Agreement is the problem. These two models have never met. They share no world. One of them says “a rusted iron door, slightly ajar” and the other paints three pristine wooden doors and a chandelier. Each is a fluent improviser; together they’re two liars who haven’t compared notes. The entire engineering effort — like, 95% of it — went into building the room where they’re forced to agree.

The setup

Local: a 10GB consumer card, fp16 weights, the diffusers pipeline. 512×512, 30 steps, ~7 seconds a room. Meanwhile davinci writes the dungeon over the API. The naive pipeline is four lines:

room_text = gpt("Describe the next dungeon room.")
image = sd_pipeline(prompt=room_text).images[0]

And the naive pipeline is garbage, for three separate reasons worth dissecting, because they’re the actual lessons.

Failure 1: prose is a terrible interface

GPT-3’s lovely paragraph — moss, dread, a distant dripping — is exactly what SD does not want. Diffusion prompts aren’t sentences; they’re bags of weighted concepts. Feeding narration into the image model gives you mush.

Fix: never let the models talk to each other directly. GPT-3’s real job is to emit a structured scene, and prose and prompt are both renderings of it:

schema = """{
  "name": "...", "mood": "...",
  "key_objects": ["...", "..."],     # max 3, concrete nouns
  "exits": [{"direction": "...", "description": "..."}],
  "palette": "...", "light_source": "..."
}"""

scene = json.loads(gpt(f"Continue the dungeon. Reply with ONLY this JSON:\n{schema}"))

prose  = gpt(f"Narrate this room in 2 atmospheric sentences: {scene}")
prompt = f"{scene['name']}, {', '.join(scene['key_objects'])}, " \
         f"{scene['light_source']}, {scene['palette']}, " \
         "dungeon interior, dark fantasy concept art, highly detailed"

(Yes, json.loads fails sometimes — davinci decorates its JSON with friendly commentary like a labrador bringing you a stick with the tree attached. One retry with the error message pasted back fixes most of it. Ask-for-JSON-and-pray is apparently a load-bearing pattern of our era.)

The JSON is the contract. The text model may only invent within the schema; the image prompt is assembled deterministically from it; the game logic (exits! items!) reads the same object. One source of truth, three renderings. The moment I did this, doors started appearing where doors were described.

Failure 2: the world redrew itself

Second visit to the same room: completely different room. Obvious in hindsight — sampling is stochastic, and a game world that won’t hold still isn’t a world, it’s a fever.

Fix, and this is my favorite line of the project: make every room a pure function of the world seed.

room_seed = int(hashlib.sha256(f"{world_seed}:{room_id}".encode()).hexdigest(), 16) % 2**32
image = pipe(prompt, generator=torch.Generator("cuda").manual_seed(room_seed),
             guidance_scale=7.5, num_inference_steps=30).images[0]

Same seed, same prompt, same sampler → bit-identical image, forever. Room 14 of world snail-7 looks the same on your machine as mine. Determinism isn’t a compromise with these models — it’s available, one manual_seed away, and suddenly you have shareable worlds, cacheable renders (hash the prompt+seed, never draw a room twice), and reproducible bug reports for a generative game. Old-fashioned engineering values, smuggled into the diffusion era.

Failure 3: the hero had forty faces

The player character appeared in room illustrations — as a different person every single time. Same prompt words, different guy. Descriptive text simply cannot pin down an identity; “a pale knight with a copper lantern” is a category, and SD samples a fresh member of it each call.

The fix is the newest tool in the box: textual inversion. You train a new embedding — a made-up token like <snail-knight> — against a handful of images of the character you want, and the concept itself becomes a word the model knows. Twenty minutes of training on my card, and now:

prompt = "<snail-knight> standing in " + room_prompt

…produces the same knight, room after room. This one floors me more than the base model does. The open weights mean the vocabulary is writable — identity as a token, consistency as a training artifact. A month ago the frontier was “what can it draw”; the actual frontier is “what can it draw twice.”

What the toy taught me

Standing back, every fix was the same fix wearing different hats: don’t let the models hold the state. The world lives in a JSON graph my code owns; the seed pins the visuals; the embedding pins the character. The models are renderers — spectacular, alien renderers — at the edges of a boring deterministic core. Creativity at the edges, truth in the middle. I keep arriving at this architecture from different directions, which either means it’s right or I only have one idea.

And it reframes what “multimodal” is going to require. Everyone’s excited that we have a text mind and an image mind; the unglamorous news is that they don’t share a world model, so someone has to build the world and make both of them serve it. The prompt is an API between two minds that have never met — and like every API, the schema matters more than the eloquence.

The dungeon itself? Honestly, it’s a slideshow with delusions. But it’s a slideshow where every room exists twice — once as words, once as light — and they finally match. My daydream post in July wanted NPCs with a mouth on a leash; turns out worlds need the same leash for their eyes. The cage keeps growing new rooms.

Total cost: one warm GPU, three weekends, and whatever davinci charged me for all that JSON it wrapped in apologies.