At the end of the last post I threatened to fine-tune GPT-2 on my own commit messages. Well, we’re all locked at home now, so no more excuses.
\nFirst, scrape commit messages out of every repo I could find on my machine:
\nfor repo in ~/projects/*/; do\n git -C \"$repo\" log --all --pretty=format:\"%s\" >> commits.txt\n echo >> commits.txt\ndone\n\nsort -u commits.txt > commits_clean.txt\nwc -l commits_clean.txt\n# 31243,124 unique commit messages. That is a tiny dataset by language model standards, which becomes important later.
\nThe transformers repo ships an example script for this — run_language_modeling.py (it was called run_lm_finetuning.py until a few weeks ago, most tutorials still use the old name). Training on my laptop CPU was hopeless, so like everyone else in 2020 I did it on a free Google Colab GPU:
python run_language_modeling.py \\\n --model_name_or_path gpt2 \\\n --model_type gpt2 \\\n --do_train \\\n --train_data_file commits_clean.txt \\\n --line_by_line \\\n --block_size 64 \\\n --per_gpu_train_batch_size 8 \\\n --num_train_epochs 3 \\\n --output_dir gpt2-commits \\\n --overwrite_output_dirThat’s the small 124M model. Three epochs over 3k lines takes about ten minutes on the Colab K80.
\nSampling from the fine-tuned model:
\nfix bug in fix\nremove console.log for real this time\nfinal fix (final) v2\nupdate readme to match the code that no longer exists\nWIP: everything\nadd tests, disable tests\nrefactor the refactor\nfix typo in previous typoI genuinely cannot tell some of these apart from my real commit history, which says more about my commit history than about the model.
\nThe interesting part wasn’t the jokes, it was watching what a language model does with a dataset this small:
\nIt memorizes. By epoch 3 the training loss was still dropping, but a lot of “generated” samples were my real commit messages, verbatim. With 3k examples the model isn’t learning the style of commit messages so much as compressing my git history into its weights. If I’d trained 10 epochs it would basically be a lossy commits.txt. This is the overfitting/memorization tradeoff everyone hand-waves about, visible in ten minutes on a free GPU.
Temperature is everything. At low temperature it plays it safe (“fix bug”, “update readme” forever). Around 0.9–1.0 you get the creative nonsense above. Higher and it starts inventing words.
\nOne epoch was probably enough. The epoch-1 checkpoint produced messages that were on-style but new. More training made it more accurate and less interesting — for a toy like this, “less overfit” is literally funnier.
\nI have a half-working prepare-commit-msg git hook that asks the model to suggest a message from the diff. It is wrong 100% of the time but with confidence. Maybe a post on that if quarantine lasts long enough.