Three weeks ago OpenAI dropped the GPT-3 paper — “Language Models are Few-Shot Learners”, 31 authors, 75 pages — and then last week announced an API for it. Private beta, waitlist. I’m on the waitlist. This post is me processing the paper while I refresh my inbox.
\nAnd I have to process it, because if the paper’s claims hold up, the two posts I wrote this spring about fine-tuning GPT-2 describe a workflow with an expiration date.
\nGPT-3 is architecturally GPT-2 with the dials turned up. Same decoder-only transformer, more of everything: 175 billion parameters across 96 layers, trained on roughly 300 billion tokens of filtered Common Crawl, WebText, books, and Wikipedia. Context window doubled from 1024 to 2048 tokens.
\nFor scale: the model I fine-tuned on my commit messages was GPT-2 small, 124 million parameters. GPT-3 is over 1,400 of those. The training compute estimates floating around are in the several-million-dollar range, and the weights in half precision would be ~350GB — this thing wouldn’t fit on my SSD, let alone my GPU. Which explains why there’s an API and no download link, and why “Open”AI jokes are back on the menu.
\nNone of that is the interesting part. If “bigger model, better perplexity” were the whole story, this would be a shrug. The Kaplan et al. scaling laws paper from January already showed loss falling smoothly and predictably with model size, data, and compute — GPT-3 is in some sense just OpenAI running that curve way past where anyone else was willing to spend. The interesting part is what showed up along the way.
\nHere’s the thing the paper is really about, and it’s worth being precise because it’s genuinely strange.
\nFine-tuning, the thing I did in March, means updating the model’s weights with gradient descent on your data. The model changes. That’s why my 3,124 commit messages got memorized — with a dataset that small, the cheapest way for the network to reduce loss is to just store the training set. I watched that happen in real time on a free Colab GPU.
\nIn-context learning, the GPT-3 pitch, involves no weight updates at all. You put a few examples in the prompt:
\nEnglish: cheese French: fromage\nEnglish: bread French: pain\nEnglish: snail French:…and the model completes the pattern. The weights are frozen. Nothing is “trained.” And yet on task after task in the paper, performance climbs as you go from zero examples to one to a few dozen — as if the model is learning the task at inference time. The bigger the model, the steeper that climb: the small models in their ablations barely benefit from examples in the prompt, and the 175B model benefits a lot. Whatever this ability is, it’s something that shows up with scale rather than something anyone explicitly built.
\nThe least mystical way I can describe it: after reading a good chunk of the internet, the model has seen so many patterns-with-examples that “infer the pattern and continue it” became part of what next-token prediction means. Conditioning does the work that gradient descent used to do. The examples don’t update the model; they locate the task inside what the model already knows.
\nI’m honestly not sure whether that framing is deep or deflationary. Possibly both.
\nIn the git hook post I wrote that GPT-2 “completes, it doesn’t obey,” and that what these models are missing “isn’t size, it’s some way of being trained to follow intent.”
\nGPT-3’s answer to that is annoying: it doesn’t follow intent either — it’s still a pure completion engine — but at 175B parameters, completion gets so good that arranging examples becomes a usable interface. The paper’s few-shot prompts are exactly the “prompt origami” I was complaining about. They didn’t remove the origami. They made the paper so foldable that origami works.
\nSo: half right, half wrong. Wrong that size wouldn’t matter — size mattered enormously, and I’d have lost money betting on that. But I’ll stand by the other half: nothing in this paper trains the model to want to do your task. You’re still tricking an autocomplete into usefulness, and prompts are a brittle, undebuggable way to program. Someone is eventually going to fine-tune one of these giants on the objective “actually do what the instruction says” rather than “continue the document,” and I suspect that model will be the one non-nerds end up using. The capability is clearly in there now; what’s missing is the steering wheel.
\nThe paper’s title is basically a shot at fine-tuning, so as someone with two fine-tuning posts, let me defend my turf — because I think the honest answer is “fine-tuning is dead for tasks, alive for distributions,” and the difference is exactly what my commit-message toy demonstrates.
\nA task is “translate this,” “summarize this,” “classify this sentiment.” If a handful of demonstrations fit in 2048 tokens, GPT-3 says you may not need a dataset, a GPU, or a training loop at all. That’s a real democratization: collecting 50k labeled examples was always the actual cost of applied NLP, and few-shot moves that cost toward zero.
\nBut my commit-message model wasn’t learning a task. It was learning a distribution — the specific texture of my habits, my vocabulary, my guilt. That’s thousands of examples’ worth of signal. It doesn’t fit in any prompt, and no amount of general internet knowledge contains it, because it’s mine. The whole reason the “commit-message-driven development” trick worked is that the model had my history baked into its weights. Ten examples in a context window is a description of me; fine-tuning was a compression of me. Those aren’t the same object.
\nSo my bet: prompting eats the tasks, fine-tuning keeps the personalization and the domains — your codebase, your support tickets, legal boilerplate, medical notes. And knowing which regime your problem is in becomes the actual skill, more than knowing how to run a training script.
\nThings in the paper that the hype tweets will skip:
\nFirst experiment is obvious: rerun the commit-message generator with zero training. Paste five diffs with their real messages into the prompt as demonstrations, staged diff sixth, let it complete. If a frozen model with five examples beats my lovingly fine-tuned GPT-2 with 3,124 — and after this paper I’d bet it does — that’s the whole regime change reproduced on my own silly benchmark, and post number four writes itself.
\nUntil then I’m a guy with a waitlist confirmation email and a 75-page PDF full of margin notes. Weirdest thing I’ve read all quarantine, and that’s a high bar this year.
","frontmatter":{"title":"GPT-3: my fine-tuning posts might already be obsolete","date":"June 20, 2020","description":"Reading the GPT-3 paper, waiting for API access, and rethinking what fine-tuning is even for"}}},"pageContext":{"slug":"/gpt-3-my-fine-tuning-posts-might-already-be-obsolete/","previous":{"fields":{"slug":"/my-ai-writes-the-commit-message-first-then-i-write-the-code/"},"frontmatter":{"title":"My AI writes the commit message first. Then I write the code"}},"next":{"fields":{"slug":"/so-about-that-docker-swarm-post-i-moved-to-k3s/"},"frontmatter":{"title":"So about that Docker Swarm post: I moved to k3s"}}}}, "staticQueryHashes": ["3435731857","426816048","63159454"]}