Most of my infrastructure changes are not engineering. They’re dictation: “add memory limits to the api pod,” “bump that image tag,” “open port 9090 for prometheus.” I know the change before I open the editor; the editor is just where I go to make typos in YAML.
\nLanguage models are extremely good at dictation. They are also confident liars. The whole design question of AI automation is how to get the first property without being destroyed by the second, and I think I’ve landed on an answer I actually trust in production:
\nThe model doesn’t get access to anything. It gets a pen. It writes pull requests. The PR must survive a gauntlet of mechanical validation before a human ever sees it, the human merges, and GitOps applies whatever lands in main — same as any other contributor. The model isn’t an operator. It’s a very fast junior colleague with zero credentials and excellent handwriting.
Five stages, one script, no daemon:
\nrequest ──► context ──► generate ──► gauntlet ──► pull request\n (text) (repo) (codex) (CI tools) (human)The entry point is deliberately dumb — a CLI:
\n./scribe \"add memory limits (512Mi) and cpu limits (500m) to the api deployment\"Models write better infrastructure when they can see the file they’re changing, so the prompt is mostly quotation. One thing I learned early and will fight about: ask for the whole new file, never a diff. Models produce plausible-looking patches with wrong line numbers and off-by-one context — a diff is a positional format and the model has no position, only text. Ask for the full file; compute the diff yourself with tools that can’t imagine things.
\nimport openai, pathlib, sys\n\ndef generate(request: str, path: str) -> str:\n current = pathlib.Path(path).read_text()\n prompt = f\"\"\"# Task: modify the Kubernetes manifest below.\n# Change requested: {request}\n# Rules: output the complete new file, valid YAML, no commentary.\n# Do not add, remove, or rename any resources unless asked.\n\n# Current file ({path}):\n{current}\n\n# Complete new file:\n\"\"\"\n resp = openai.Completion.create(\n model=\"code-davinci-002\", # codex beta: currently free, absurdly\n prompt=prompt,\n max_tokens=1500,\n temperature=0, # dictation, not poetry\n stop=[\"# Task:\"],\n )\n return resp.choices[0].text.strip()Temperature 0 because there is exactly one correct way to add a memory limit and I’d like it every time. Which file to edit is either given on the CLI or picked by a first, tiny completion that sees only git ls-files '*.yaml' — choosing from a real list is a task models are hard to break on, since every possible answer already exists.
Here’s the heart of the design. I never evaluate the model’s output by reading it — reading is how tired humans get fooled by confident text. The output must instead survive tools that are incapable of being charmed:
\n#!/bin/sh\nset -e # any failure kills the PR before it exists\n\nyamllint -d relaxed \"$FILE\" # 1. is it even YAML\nkubeconform -strict -summary \"$FILE\" # 2. is it valid *Kubernetes* —\n # schemas don't negotiate\nkubectl diff -f \"$FILE\" --server-side || true # 3. what would actually change\nkubectl apply -f \"$FILE\" --dry-run=server # 4. would the API server accept it\n\n# terraform changes get the equivalent treatment:\nterraform plan -detailed-exitcode -out=tf.plan # exit 2 = changes, 1 = broken\nterraform show tf.plan | grep -q \"destroy\" && exit 1 # the bot may not destroy. ever.Then the paranoid layer — a blast-radius allowlist enforced on the diff, not the intent:
\ngit diff --name-only main | grep -vE '^k8s/(deployments|services|ingress)/' \\\n && { echo \"touched files outside the sandbox\"; exit 1; }\ngit diff main | grep -qE 'kind:\\s*(Secret|PersistentVolume)' \\\n && { echo \"nice try\"; exit 1; }The model can want whatever it likes. The gauntlet only cares what the bytes do. Every check here existed before language models and will outlive them — that’s the point. Generation is allowed to be creative; validation is a compiler. Never staff both jobs with the same kind of mind.
\nWhen a check fails, the error message goes back into the prompt for exactly one retry:
\nretry_prompt = prompt + generated + f\"\"\"\n\n# The file above failed validation with this error:\n# {stderr}\n# Output the corrected complete file:\n\"\"\"This fixes a surprising share of failures — schema validators write great error messages, and the model reads them better than I do at 9am. But it’s one round, hard-coded. An unbounded fix-it-yourself loop is a self-licking ice cream cone that burns tokens converging on nothing; if two attempts can’t pass a linter, the request was ambiguous and a human should hear about it. Loops need leashes.
\nSurvivors become pull requests, and the PR body carries the receipts:
\ngit checkout -b \"scribe/${SLUG}\"\ngit commit -am \"scribe: ${REQUEST}\" --author=\"Scribe <scribe@ohsnail.com>\"\ngh pr create \\\n --title \"🖋 ${REQUEST}\" \\\n --body \"$(cat <<EOF\n**Request:** ${REQUEST}\n**Server dry-run diff:**\n\\`\\`\\`\n$(kubectl diff -f \"$FILE\" --server-side)\n\\`\\`\\`\n**Gauntlet:** yamllint ✓ kubeconform ✓ dry-run ✓ blast-radius ✓\nEOF\n)\"Review takes seconds, because I’m not reviewing prose or trust — I’m reviewing a machine-verified diff with the evidence attached. Merge, and the GitOps controller ships it like any other commit. The model never held a credential; the merge button is the last mile of intent, and it stays human.
\n38 requests since March. 26 merged untouched. 7 merged after small edits (it loves adding labels nobody asked for — the junior-colleague energy is uncanny). 3 correctly killed by the gauntlet — the best one invented apiVersion: apps/v2, a version that does not exist, delivered with total confidence and executed by kubeconform in four milliseconds. 2 died in ambiguity, as they should have: my request was the bug.
Time per dictation-class change: was ~10 minutes of editor and kubectl archaeology, now ~40 seconds of reading a diff. But the number I care about is different: zero incidents caused, zero possible by construction — the failure domain is “a bad PR gets opened,” and a bad PR is litter, not fire.
\nEveryone building on these models is choosing a point on a line between “autocomplete” and “autonomy,” and most of the debate treats capability as the axis that matters. I don’t think it is. The axis that matters is what artifact the model produces. Give it a terminal and its mistakes are events. Give it a pen and its mistakes are documents — diffable, lintable, rejectable, versioned, reviewed. Pull requests turn out to be a nearly perfect prosthetic for a brilliant liar: all of the leverage, none of the blast radius, and an audit trail as a side effect.
\nDon’t give the model hands. Give it a pen, and make the paper do the vetting.
","frontmatter":{"title":"My infrastructure accepts pull requests from a language model now","date":"June 06, 2022","description":"Don't give the model hands — give it a pen. A pipeline where GPT writes Kubernetes and Terraform changes as PRs that must survive a mechanical gauntlet before a human ever sees them"}}},"pageContext":{"slug":"/my-infrastructure-accepts-pull-requests-from-a-language-model-now/","previous":{"fields":{"slug":"/odd-bot-out-a-party-game-where-gpt-3-hides-among-your-relatives/"},"frontmatter":{"title":"Odd Bot Out: a party game where GPT-3 hides among your relatives"}},"next":{"fields":{"slug":"/twenty-questions-against-a-player-with-no-secret/"},"frontmatter":{"title":"Twenty questions against a player with no secret"}}}}, "staticQueryHashes": ["3435731857","426816048","63159454"]}