Ten months after joining the waitlist, the invite came. So this is the post the June write-up promised: the world’s least rigorous benchmark, rerun. In one corner, my GPT-2 small, 124M parameters, lovingly fine-tuned on 3,124 of my own commit messages, weights shaped by my personal git guilt. In the other corner, GPT-3 davinci: 175 billion frozen parameters that have never heard of me, allowed to see exactly five examples through the 2048-token keyhole of its prompt.
\nThe question from last June, now testable: is fine-tuning dead for tasks?
\nTwenty real staged diffs, pulled from actual work over two weeks. Both models get each diff and must produce a commit message. GPT-2 runs locally, as a pod, for free. GPT-3 gets this:
\nimport openai\n\nFEWSHOT = open(\"five_examples.txt\").read() # 5 diff→message pairs, ~1400 tokens\n\nresp = openai.Completion.create(\n engine=\"davinci\",\n prompt=FEWSHOT + \"\\n===\\ndiff:\\n\" + truncate(diff, 400) + \"\\nmessage:\",\n max_tokens=24,\n temperature=0.7,\n stop=[\"\\n\", \"===\"]\n)Notes from the keyhole: five demonstration pairs plus a truncated diff eats nearly the whole 2048-token context, so “which five examples” and “which 400 tokens of the diff” become the actual engineering. There’s no fine-tuning endpoint for GPT-3 — prompting is the interface — which makes this a genuinely fair fight between the two paradigms. Scoring: a shell script shuffled the outputs so I graded blind, picking the message I’d actually keep.
\nNot close. Davinci 16, my model 4.
\nThe gap is exactly where the paper said it would be: GPT-2 has never seen a diff and is riffing on vibes, while davinci actually reads the thing. Given a diff adding a Redis import and a cache decorator, my model offered “fix readme typo (again)“. Davinci wrote “add redis caching to search results”. That’s not a style difference. One of them did the assignment.
\nThe part that stung more: I assumed the giant would sound like a press release and my model would at least win on voice. Wrong. From five lowercase, slightly self-deprecating examples, davinci picked up my register almost perfectly — lowercase, terse, the occasional parenthetical apology. Style, it turns out, is shallow: it transfers through a keyhole. I’d been treating “sounds like me” as evidence of something deep in my fine-tuned weights, and a frozen model just cold-read it off five samples like a street magician.
\nLast June I bet “prompting eats the tasks, fine-tuning keeps the distributions.” Half survives, and it’s the narrower half.
\nThe four rounds my model won were all the same kind: diffs whose real story wasn’t in the diff. A one-line version bump whose honest message was “update deps and pray” — because that dependency has burned me twice, a fact living in my history, not in the patch. Davinci, reading only the keyhole, wrote the technically-correct-but-alien “bump lodash to 4.17.21”. My model has eaten two years of my scar tissue; it knows prayer is warranted.
\nSo the revised, humbler thesis: fine-tuning doesn’t own style, it owns context that never fits through the keyhole. Prompting rents a mind and slides your world through a 2048-token slot. The slot will widen — everything in this field widens — but there will always be some residue of private history on the wrong side of it. That residue is what your own weights are for. Everything else, the giant does better, today, for money.
\nAnd it is money, which is the quietly radical part. Each suggestion burns ~1,900 davinci tokens; at 6 cents per thousand that’s about 11 cents per commit message. My GPT-2 pod costs literally nothing per thought.
\nSit with how strange that is: for the first time, a function call in my code has a marginal cost per thought. Not per gigabyte, per hour, per request — per act of cognition, metered like electricity. My git hook now has a unit economics section. I caught myself batching commits to save dimes, which means the pricing model is already reshaping my behavior, which is how you know it’s real infrastructure and not a demo. Utility computing finally happened, and the utility is thinking.
\n(Practical corollary: the fine-tuned pet earns a second life as the cheap tier. Local model drafts, giant model only when the diff is gnarly. Cascading by cost is going to be a whole design pattern, I suspect.)
\nOne more thing, filed under “probably terrible.” Everything I’ve built so far is a straight line: diff in, text out, human decides. But the completion API doesn’t care where prompts come from — nothing stops a script from feeding the model’s output back in as the next input and letting it converse with itself. And my k3s manifests are just text in git, and davinci demonstrably reads structured text… you can see the loop closing: model proposes a change, script applies it, reality’s error message goes back through the keyhole, repeat.
\nAn autocomplete with hands. I gave it maybe an hour of daydreaming before remembering that this is a system that confidently invents flags — for rm, presumably — and that my Terraform post has a whole section about beliefs meeting reality under human review. The plan is the product; the loop deletes the plan. So no. Not with this model, not unattended, not on a box I like.
But I’m writing the idea down, because the objection is about this model, and models have a way of getting better while opinions stay still. Ask me again after the next paper.
\nThe GitOps rabbit hole is still scheduled, I promise — the robots that apply git commits deserve their own post, especially now that I’ve spent a paragraph fantasizing about a robot that writes them.
","frontmatter":{"title":"The rematch: my fine-tuned GPT-2 vs. a frozen giant with five examples","date":"April 11, 2021","description":"GPT-3 API access finally arrived, so the commit-message benchmark of exactly one silly task gets its rerun"}}},"pageContext":{"slug":"/the-rematch-my-fine-tuned-gpt-2-vs-a-frozen-giant-with-five-examples/","previous":{"fields":{"slug":"/the-scaling-ladder-a-production-setup-that-scales-by-not-scaling-yet/"},"frontmatter":{"title":"The scaling ladder: a production setup that scales by not scaling yet"}},"next":{"fields":{"slug":"/i-hired-gpt-3-as-my-on-call-intern/"},"frontmatter":{"title":"I hired GPT-3 as my on-call intern"}}}}, "staticQueryHashes": ["3435731857","426816048","63159454"]}