My infrastructure accepts pull requests from a language model now

Most of my infrastructure changes are not engineering. They’re dictation: “add memory limits to the api pod,” “bump that image tag,” “open port 9090 for prometheus.” I know the change before I open the editor; the editor is just where I go to make typos in YAML.

Language models are extremely good at dictation. They are also confident liars. The whole design question of AI automation is how to get the first property without being destroyed by the second, and I think I’ve landed on an answer I actually trust in production:

The model doesn’t get access to anything. It gets a pen. It writes pull requests. The PR must survive a gauntlet of mechanical validation before a human ever sees it, the human merges, and GitOps applies whatever lands in main — same as any other contributor. The model isn’t an operator. It’s a very fast junior colleague with zero credentials and excellent handwriting.

The pipeline

Five stages, one script, no daemon:

request ──► context ──► generate ──► gauntlet ──► pull request
 (text)      (repo)     (codex)      (CI tools)      (human)

The entry point is deliberately dumb — a CLI:

./scribe "add memory limits (512Mi) and cpu limits (500m) to the api deployment"

Stage 1–2: context, then generation

Models write better infrastructure when they can see the file they’re changing, so the prompt is mostly quotation. One thing I learned early and will fight about: ask for the whole new file, never a diff. Models produce plausible-looking patches with wrong line numbers and off-by-one context — a diff is a positional format and the model has no position, only text. Ask for the full file; compute the diff yourself with tools that can’t imagine things.

import openai, pathlib, sys

def generate(request: str, path: str) -> str:
    current = pathlib.Path(path).read_text()
    prompt = f"""# Task: modify the Kubernetes manifest below.
# Change requested: {request}
# Rules: output the complete new file, valid YAML, no commentary.
# Do not add, remove, or rename any resources unless asked.

# Current file ({path}):
{current}

# Complete new file:
"""
    resp = openai.Completion.create(
        model="code-davinci-002",   # codex beta: currently free, absurdly
        prompt=prompt,
        max_tokens=1500,
        temperature=0,              # dictation, not poetry
        stop=["# Task:"],
    )
    return resp.choices[0].text.strip()

Temperature 0 because there is exactly one correct way to add a memory limit and I’d like it every time. Which file to edit is either given on the CLI or picked by a first, tiny completion that sees only git ls-files '*.yaml' — choosing from a real list is a task models are hard to break on, since every possible answer already exists.

Stage 3: the gauntlet

Here’s the heart of the design. I never evaluate the model’s output by reading it — reading is how tired humans get fooled by confident text. The output must instead survive tools that are incapable of being charmed:

#!/bin/sh
set -e  # any failure kills the PR before it exists

yamllint -d relaxed "$FILE"                      # 1. is it even YAML
kubeconform -strict -summary "$FILE"             # 2. is it valid *Kubernetes* —
                                                 #    schemas don't negotiate
kubectl diff -f "$FILE" --server-side || true    # 3. what would actually change
kubectl apply -f "$FILE" --dry-run=server        # 4. would the API server accept it

# terraform changes get the equivalent treatment:
terraform plan -detailed-exitcode -out=tf.plan   # exit 2 = changes, 1 = broken
terraform show tf.plan | grep -q "destroy" && exit 1   # the bot may not destroy. ever.

Then the paranoid layer — a blast-radius allowlist enforced on the diff, not the intent:

git diff --name-only main | grep -vE '^k8s/(deployments|services|ingress)/' \
  && { echo "touched files outside the sandbox"; exit 1; }
git diff main | grep -qE 'kind:\s*(Secret|PersistentVolume)' \
  && { echo "nice try"; exit 1; }

The model can want whatever it likes. The gauntlet only cares what the bytes do. Every check here existed before language models and will outlive them — that’s the point. Generation is allowed to be creative; validation is a compiler. Never staff both jobs with the same kind of mind.

Stage 3½: one round of feedback, not a loop

When a check fails, the error message goes back into the prompt for exactly one retry:

retry_prompt = prompt + generated + f"""

# The file above failed validation with this error:
# {stderr}
# Output the corrected complete file:
"""

This fixes a surprising share of failures — schema validators write great error messages, and the model reads them better than I do at 9am. But it’s one round, hard-coded. An unbounded fix-it-yourself loop is a self-licking ice cream cone that burns tokens converging on nothing; if two attempts can’t pass a linter, the request was ambiguous and a human should hear about it. Loops need leashes.

Stage 4: the PR is the audit log

Survivors become pull requests, and the PR body carries the receipts:

git checkout -b "scribe/${SLUG}"
git commit -am "scribe: ${REQUEST}" --author="Scribe <scribe@ohsnail.com>"
gh pr create \
  --title "🖋 ${REQUEST}" \
  --body "$(cat <<EOF
**Request:** ${REQUEST}
**Server dry-run diff:**
\`\`\`
$(kubectl diff -f "$FILE" --server-side)
\`\`\`
**Gauntlet:** yamllint ✓ kubeconform ✓ dry-run ✓ blast-radius ✓
EOF
)"

Review takes seconds, because I’m not reviewing prose or trust — I’m reviewing a machine-verified diff with the evidence attached. Merge, and the GitOps controller ships it like any other commit. The model never held a credential; the merge button is the last mile of intent, and it stays human.

Ten weeks of numbers

38 requests since March. 26 merged untouched. 7 merged after small edits (it loves adding labels nobody asked for — the junior-colleague energy is uncanny). 3 correctly killed by the gauntlet — the best one invented apiVersion: apps/v2, a version that does not exist, delivered with total confidence and executed by kubeconform in four milliseconds. 2 died in ambiguity, as they should have: my request was the bug.

Time per dictation-class change: was ~10 minutes of editor and kubectl archaeology, now ~40 seconds of reading a diff. But the number I care about is different: zero incidents caused, zero possible by construction — the failure domain is “a bad PR gets opened,” and a bad PR is litter, not fire.

The design rule I’d generalize

Everyone building on these models is choosing a point on a line between “autocomplete” and “autonomy,” and most of the debate treats capability as the axis that matters. I don’t think it is. The axis that matters is what artifact the model produces. Give it a terminal and its mistakes are events. Give it a pen and its mistakes are documents — diffable, lintable, rejectable, versioned, reviewed. Pull requests turn out to be a nearly perfect prosthetic for a brilliant liar: all of the leverage, none of the blast radius, and an audit trail as a side effect.

Don’t give the model hands. Give it a pen, and make the paper do the vetting.