\n\nArchive note (2026): Reconstructed with April 2023-era models, libraries, and sources as a continuation of the earlier fine-tuning posts. The adapter-size calculation and code are reproducible; sample generations are illustrative rather than contemporaneous measurements.
\n
Three years ago, I fine-tuned GPT-2 on 3,124 of my commit messages. It took ten minutes on a free Colab GPU and produced a half-gigabyte monument to fix typo in previous typo.
A year later, I put that model against GPT-3 with five examples in the prompt. The frozen giant won 16–4. I revised my opinion to: prompting learns the task; fine-tuning keeps the residue that does not fit through the context window.
\nThis month fine-tuning came back through the side door.
\nAlpaca took a 7B base model and 52,000 instruction examples and made something surprisingly chatty. Vicuna used conversations instead and got a different creature. Dolly 2.0 used 15,000 human-written examples and, importantly, released the model and data under terms a company can actually use. Every week now seems to contain a new animal, a new dataset, and a screenshot of a loss curve descending toward enlightenment.
\nThe models are interesting. The stranger thing is the file size.
\nI loaded OPT-6.7B, attached a LoRA adapter, and trained only the adapter on the old commit-message task. The base checkpoint is roughly 13 GB in half precision. The thing I saved afterward was about 8 MB.
\nI assumed the save had failed.
\nIt had not. Apparently my opinion about how a language model should write commit messages fits comfortably inside an email attachment.
\nNormal fine-tuning updates the model’s weight matrices. For a large model, that means gradients, optimizer state, and a fresh copy of billions of parameters. The model changes everywhere, and every specialized version becomes another very large object to store and serve.
\nLoRA takes a smaller route. Freeze the original matrix W, and learn a low-rank update beside it:
W' = W + ΔW\nΔW = B × AIf W is a 4096 × 4096 attention matrix, it contains 16,777,216 parameters. With rank 4, LoRA replaces the trainable update with two skinny matrices:
A: 4 × 4096\nB: 4096 × 4That is 32,768 trainable parameters instead of 16.7 million for one projection. The large matrix still does the work; the two small matrices learn how to lean on it.
\nI attached those updates to the query and value projections in each of OPT’s 32 transformer layers. The arithmetic is pleasingly rude:
\n2 projections\n× 32 layers\n× (4096 × 4 + 4 × 4096)\n= 2,097,152 trainable parametersTwo million sounds large until it is standing next to 6.7 billion. It is about 0.03% of the model. Stored as 32-bit values, it is almost exactly 8 MB.
\nThe base model remains frozen. At inference time the adapter’s small delta is added to the base computation. Remove the adapter and the original model is back. Load another adapter and the same model has another set of habits.
\nThis is not a smaller model. It is a small amendment to a large one.
\nI reused the commit-message experiment because it has become my fruit fly. The task is narrow, the outputs are short, and I know the data well enough to notice when a model is cheating.
\nThis time a training record contains both the patch and the message:
\n### task\nWrite one terse git commit subject for this diff.\nUse lowercase. No period. Describe the change, not the intention.\n\n### diff\n@@ -18,7 +18,10 @@ def fetch_manifest(url):\n- return requests.get(url, timeout=2).json()\n+ for attempt in range(3):\n+ try:\n+ return requests.get(url, timeout=2).json()\n+ except requests.Timeout:\n+ time.sleep(attempt + 1)\n\n### message\nretry manifest fetch after timeouts</s>I extracted pairs from my repositories, dropped merges and generated files, removed patches that would not fit in 512 tokens, and stripped anything that looked remotely like a credential. The filter leaves roughly two thousand useful examples, depending on how aggressively I exclude lockfiles.
\nThe important split is chronological, not random.
\nA random split makes this benchmark look smarter than it is. Real commit histories contain little families: add cache, fix cache key, test cache key, actually fix cache key. Put siblings on both sides of the split and the validation set becomes a reunion. Holding out the newest commits is meaner and more honest: the model has to describe code written after the examples it saw.
The training setup, using Hugging Face’s then-new PEFT library, is short enough to fit in the post:
\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom peft import (\n LoraConfig,\n TaskType,\n get_peft_model,\n prepare_model_for_int8_training,\n)\n\nbase = \"facebook/opt-6.7b\"\n\ntokenizer = AutoTokenizer.from_pretrained(base, use_fast=False)\ntokenizer.pad_token = tokenizer.eos_token\n\nmodel = AutoModelForCausalLM.from_pretrained(\n base,\n load_in_8bit=True,\n device_map=\"auto\",\n)\nmodel = prepare_model_for_int8_training(model)\n\nconfig = LoraConfig(\n task_type=TaskType.CAUSAL_LM,\n r=4,\n lora_alpha=8,\n lora_dropout=0.05,\n target_modules=[\"q_proj\", \"v_proj\"],\n bias=\"none\",\n)\n\nmodel = get_peft_model(model, config)\nmodel.print_trainable_parameters()\n# about 2.1M trainable parameters out of 6.7B: ~0.03%The base weights are loaded in 8-bit, using the same broad trick described in LLM.int8(), to fit comfortably on one GPU. The small trainable matrices remain at higher precision. Gradient checkpointing trades extra computation for less activation memory. Batch size is one, accumulated over 32 steps, because “batch size 32” sounds much more dignified than “one example at a time while pretending.”
\nMy useful settings ended up boring:
\nsequence length: 512\nlearning rate: 2e-4\nmicro batch size: 1\ngradient accumulation: 32\nepochs: 3\nLoRA rank: 4\nLoRA alpha: 8\nLoRA dropout: 0.05The fashionable part is the adapter. The part that determined whether it worked was the labels.
\nThe naïve training record contains a few hundred tokens of instructions and diff, followed by perhaps eight tokens of commit message. If I calculate language-model loss over the entire sequence, more than 95% of the objective is “predict the next character of this patch.”
\nThe loss goes down. The model gets better at braces, plus signs, Python indentation, and the exact phrase ### message. Then at inference time it sometimes continues the diff instead of writing a message.
Nothing is broken. I asked it to become an excellent photocopier and it complied.
\nFor instruction tuning, the prompt should shape the hidden state without becoming a prediction target. The input tokens stay visible to the model, but their labels are replaced with PyTorch’s ignore value:
\nlabels = input_ids.copy()\nlabels[:message_starts_at] = -100\n\nreturn {\n \"input_ids\": input_ids,\n \"attention_mask\": attention_mask,\n \"labels\": labels,\n}Now the gradient comes only from the answer. The model still reads the diff; it is simply not rewarded for reciting it.
\nThis tiny distinction explains a lot of suspiciously beautiful fine-tuning graphs. Training loss is not a product metric. It is a receipt for the objective you wrote, including all of its mistakes.
\nA falling number proves that the optimizer found something easy to learn. It does not prove you chose the right thing.
\nWith the adapter disabled, the base model tends to treat the prompt as a document to continue. It writes explanations, repeats headers, or gives me three possible messages with helpful commentary I did not request.
\nWith the adapter enabled, the surface behavior changes quickly:
\nadd retry logic to manifest fetch\navoid duplicate alerts for recovered pods\nmove token validation before cache lookupLowercase. Imperative. No period. One line. It stops trying to be useful in twelve different directions.
\nThat part feels magical for about five minutes. Then the failure pattern appears.
\nMy history contains far more repairs than new features, so the adapter develops a strong prior for the word fix. Show it a patch that adds a timeout where none existed and it may write fix request timeout, even though nothing was fixed; a constraint was added. Show it a new test and it may infer a bug that the test never mentions. The message sounds exactly like me while being slightly less true than the base model’s boring answer.
This is the dangerous version of personalization: not a wild hallucination, but a wrong answer wearing your clothes.
\nThe adapter did not pour two thousand facts into the model. It changed which completions feel natural. It learned that I prefer add over implement, lowercase over title case, and mild pessimism over celebration. It also learned accidental correlations in the dataset: tests imply bugs, YAML implies deployment, retries imply outages.
Fine-tuning is a prior with excellent branding.
\nThat makes dataset design feel less like collecting documents and more like writing source code. Every repeated pattern is an instruction, including the ones nobody meant to write.
\nif diff contains \"test\" → probably say \"fix\"\nif file ends in .yaml → probably say \"deploy\"\nif author is me → definitely avoid punctuationThe network does not store those rules literally. The effect is worse: they are distributed, soft, and difficult to grep.
\nThe open-model rush is producing a lot of claims that sound like “trained on medicine,” “trained on legal data,” or “trained on our documentation.” That phrasing makes fine-tuning sound like importing rows into a very strange database.
\nIt is the wrong mental model.
\nFine-tuning is good at changing recurring behavior: vocabulary, tone, output shape, taxonomy, and which distinctions the model habitually pays attention to. It can make an existing capability easier to reach. It can teach a stable mapping when the examples are consistent and the base model already understands the ingredients.
\nIt is a bad place for facts that change on Tuesday.
\nIf I need the current deployment version, customer balance, incident status, or API schema, I should fetch it at request time. Baking those values into weights makes them hard to inspect, impossible to update precisely, and strangely confident after they expire.
\nMy current division of labor looks like this:
\n| Need | \nUse | \n
|---|---|
| A one-off task | \nA prompt and a few examples | \n
| Current or private facts | \nRetrieve them and put them in context | \n
| A recurring voice or decision pattern | \nFine-tune an adapter | \n
| Exact syntax or policy | \nA parser, schema, or deterministic validator | \n
| Permission to do something | \nAn authorization system outside the model | \n
A prompt is the request. The adapter is the default. The validator is the law.
\nConfusing those layers is how a style model becomes a source of truth, or a friendly answer becomes permission.
\nThe small file changes the operational shape of fine-tuning more than the cheap training does.
\nI can keep one base model loaded and version the adapters separately. A commit-message adapter, an incident-summary adapter, maybe one trained on a project’s vocabulary. Rollback means selecting the previous directory, not replacing a 13 GB checkpoint. Experiments become cheap enough to branch.
\nBut “small” is not the same as “harmless.” An adapter can systematically change every answer while looking like a rounding error next to the base weights. I now want the same receipts around it that I want around infrastructure:
\ncommit-message-r4/\n├── adapter_config.json\n├── adapter_model.bin\n└── manifest.json{\n \"base_model\": \"facebook/opt-6.7b\",\n \"base_revision\": \"<immutable commit>\",\n \"dataset_sha256\": \"<hash>\",\n \"split\": \"chronological\",\n \"train_on_prompt\": false,\n \"lora_rank\": 4,\n \"targets\": [\"q_proj\", \"v_proj\"],\n \"evaluation\": \"blind-held-out-commits-v1\"\n}The adapter and base revision are one deployable unit. Change either and the evaluation is stale. The dataset needs provenance because it is effectively executable behavior. Sample generations belong in the pull request. So do the embarrassing edge cases.
\nAnd I would not casually stack adapters because the files are small. Two deltas added to the same matrices are not two independent browser extensions. “Writes like me” plus “classifies incidents” may compose beautifully or may produce incident reports that apologize for themselves. The only honest answer is another held-out set.
\nLast month, the text box learned verbs. Plugins decide what the model can do. Fine-tuning changes what it tends to do. Capability needs permissions; tendency needs tests.
\nBoth need a rollback button.
\nIn 2020, I thought fine-tuning meant making a personal copy of a model. In 2021, five examples in a prompt made that look wasteful. My revised revision is this:
\nPrompting describes the task.
\nFine-tuning edits the defaults.
\nThe base model supplies broad capability. The adapter supplies a narrow gravitational field around the outputs you prefer. That field can encode taste, useful domain distinctions, and boring consistency. It can also encode every shortcut, imbalance, private string, and mislabeled example in the dataset.
\nLoRA makes the experiment cheap enough for one person and the result small enough to version. That is genuinely exciting. It also means model behavior is becoming another artifact we can casually copy into production without reading, because there is no meaningful way to read two million floating-point numbers.
\nSo the review has to move outward: inspect the data, freeze the split, test the behavior, record the base revision, and keep deterministic boundaries around anything that matters.
\nThe model is 13 GB. My opinion of it is 8 MB.
\nEight megabytes is not much space.
\nApparently it is enough for a habit.
","timeToRead":10,"frontmatter":{"title":"The model is 13 GB. My opinion of it is 8 MB","date":"April 30, 2023","description":"I fine-tuned a 6.7B language model without changing the 6.7B parameters. LoRA turned behavior into a tiny patch — and made the dataset look uncomfortably like source code."}}},"pageContext":{"slug":"/the-model-is-13-gb-my-opinion-of-it-is-8-mb/","previous":{"fields":{"slug":"/how-a-pdf-button-became-remote-code-execution-and-paid-for-my-year/"},"frontmatter":{"title":"How a PDF button became remote code execution (and paid for my year)"}},"next":{"fields":{"slug":"/the-signature-was-valid-the-company-was-wrong/"},"frontmatter":{"title":"The signature was valid. The company was wrong"}}}}, "staticQueryHashes": ["3037286347","63159454"]}