Quarantine lasted long enough. Here’s the promised post about the prepare-commit-msg hook — except the project mutated into something much weirder along the way.
Simple: a git hook grabs the staged diff, sends it to my fine-tuned GPT-2 from last month, and the model suggests a commit message. Since loading a 500MB model on every commit is insane, the model lives in a tiny Flask server that stays warm:
\nfrom flask import Flask, request\n\napp = Flask(__name__)\ngenerator = load_my_finetuned_gpt2() # from last post\n\n@app.route(\"/suggest\", methods=[\"POST\"])\ndef suggest():\n diff = request.form[\"diff\"][:3000]\n return generator(diff)And the hook:
\n#!/bin/sh\n# .git/hooks/prepare-commit-msg\nDIFF=$(git diff --cached)\nMSG=$(curl -s localhost:5000/suggest --data-urlencode \"diff=$DIFF\")\nprintf '# suggested: %s\\n%s' \"$MSG\" \"$(cat \"$1\")\" > \"$1\"It was wrong 100% of the time, but the reasons are more interesting than the result:
\nThe model has never seen a diff. I fine-tuned it on commit messages only. From its point of view, a diff is line noise, so it responds to +import redis with “fix typo in readme”. Training distribution is everything — a lesson people apparently keep having to relearn at every scale.
1024 tokens. GPT-2’s entire context window. My average staged diff is 3–4x that. The model was writing commit messages after reading the first file’s worth of changes, which, to be fair, is also how some humans review PRs.
\nIt completes, it doesn’t obey. This is the big one. GPT-2 is an autocomplete engine. There’s no way to tell it “summarize this diff” — you can only arrange text and hope the most probable continuation happens to be what you wanted. What these models are missing isn’t size, it’s some way of being trained to follow intent. Until someone figures that out, everything built on them is prompt origami.
\nStaring at another wrong suggestion, it hit me: the model can’t describe what I did, but it’s weirdly good at proposing things I could do. It’s been trained on 3,000 of my past intentions.
\nSo I flipped the workflow. Every morning the model deals me a commit message. Then I have to write code that makes the message true.
\nWeek one assignments, verbatim:
\ndelete old feature flags\nfix flaky test\nadd caching to search\nupdate deps and prayHere’s the unsettling part: all four were real work I’d been avoiding. I found three dead feature flags, one genuinely flaky test, and a search endpoint that absolutely needed caching. The model isn’t psychic — it’s a compressed statistical summary of my habits, and my habits include perpetually deferring the same maintenance. It’s like Brian Eno’s Oblique Strategies, except the deck was generated from my own guilt.
\nThe hook was supposed to automate me out of a chore. What I got instead is a slot machine that pays out in chores — and it works because it’s random-but-in-my-distribution. A model too weak to answer questions can still be a fantastic question generator.
","frontmatter":{"title":"My AI writes the commit message first. Then I write the code","date":"April 25, 2020","description":"The GPT-2 commit hook failed exactly as promised, so I turned it around"}}},"pageContext":{"slug":"/my-ai-writes-the-commit-message-first-then-i-write-the-code/","previous":{"fields":{"slug":"/i-fine-tuned-gpt-2-on-my-commit-messages-im-sorry/"},"frontmatter":{"title":"I fine-tuned GPT-2 on my commit messages (I'm sorry)"}},"next":{"fields":{"slug":"/gpt-3-my-fine-tuning-posts-might-already-be-obsolete/"},"frontmatter":{"title":"GPT-3: my fine-tuning posts might already be obsolete"}}}}, "staticQueryHashes": ["3435731857","426816048","63159454"]}