I fine-tuned GPT-2 on my commit messages (I'm sorry)

At the end of the last post I threatened to fine-tune GPT-2 on my own commit messages. Well, we’re all locked at home now, so no more excuses.

Getting the data

First, scrape commit messages out of every repo I could find on my machine:

for repo in ~/projects/*/; do
  git -C "$repo" log --all --pretty=format:"%s" >> commits.txt
  echo >> commits.txt
done

sort -u commits.txt > commits_clean.txt
wc -l commits_clean.txt
# 3124

3,124 unique commit messages. That is a tiny dataset by language model standards, which becomes important later.

Fine-tuning

The transformers repo ships an example script for this — run_language_modeling.py (it was called run_lm_finetuning.py until a few weeks ago, most tutorials still use the old name). Training on my laptop CPU was hopeless, so like everyone else in 2020 I did it on a free Google Colab GPU:

python run_language_modeling.py \
  --model_name_or_path gpt2 \
  --model_type gpt2 \
  --do_train \
  --train_data_file commits_clean.txt \
  --line_by_line \
  --block_size 64 \
  --per_gpu_train_batch_size 8 \
  --num_train_epochs 3 \
  --output_dir gpt2-commits \
  --overwrite_output_dir

That’s the small 124M model. Three epochs over 3k lines takes about ten minutes on the Colab K80.

The horror, as promised

Sampling from the fine-tuned model:

fix bug in fix
remove console.log for real this time
final fix (final) v2
update readme to match the code that no longer exists
WIP: everything
add tests, disable tests
refactor the refactor
fix typo in previous typo

I genuinely cannot tell some of these apart from my real commit history, which says more about my commit history than about the model.

What I actually learned

The interesting part wasn’t the jokes, it was watching what a language model does with a dataset this small:

It memorizes. By epoch 3 the training loss was still dropping, but a lot of “generated” samples were my real commit messages, verbatim. With 3k examples the model isn’t learning the style of commit messages so much as compressing my git history into its weights. If I’d trained 10 epochs it would basically be a lossy commits.txt. This is the overfitting/memorization tradeoff everyone hand-waves about, visible in ten minutes on a free GPU.

Temperature is everything. At low temperature it plays it safe (“fix bug”, “update readme” forever). Around 0.9–1.0 you get the creative nonsense above. Higher and it starts inventing words.

One epoch was probably enough. The epoch-1 checkpoint produced messages that were on-style but new. More training made it more accurate and less interesting — for a toy like this, “less overfit” is literally funnier.

Next

I have a half-working prepare-commit-msg git hook that asks the model to suggest a message from the diff. It is wrong 100% of the time but with confidence. Maybe a post on that if quarantine lasts long enough.