I hired GPT-3 as my on-call intern

This blog has been two blogs wearing a trenchcoat: infrastructure posts and GPT posts, politely ignoring each other. This summer they finally met, because I got paged at 3:12am, spent forty minutes spelunking through Loki, and discovered the root cause was — again — the same Redis pod OOMing. Forty minutes, for a diagnosis that fits in a sentence.

A sentence, you say. We have a machine for sentences now.

What it is (and deliberately isn’t)

The April post already litigated the tempting version — the closed loop where the model fixes things itself — and the verdict stands: not unattended, not on a box I like. A system that invents flags does not get kubectl.

But there’s a humbler job that’s still most of the pain: translation. When an alert fires, the on-call intern doesn’t fix anything. The intern gathers context, forms a hypothesis, and hands the human a briefing. So:

Alertmanager webhook → context gatherer → davinci → Slack message.

The Slack message has three parts: a one-sentence hypothesis, the evidence, and a suggested first command — the kubectl describe or LogQL query I’d probably start with anyway. It proposes; I dispose. The plan is still the product.

The keyhole problem, industrial edition

The hard engineering isn’t the model call — it’s that an incident produces megabytes of context and the keyhole is still 2048 tokens. Commit diffs were easy; logs are a firehose of repetition. So most of the code is what I’ve started calling log distillation:

• Pull the alert’s labels, the pod’s k8s events, the last git log entries touching those manifests, and a Loki query for the affected pod.
• Collapse duplicate lines into [x347] connection refused ... — log spam compresses beautifully, because spam is the point of spam.
• Keep the window around the first error, not the last. The end of an incident log is all symptoms; patient zero is at the beginning.
• Budget: ~600 tokens of distilled logs, ~200 of events and deploy history, ~900 of few-shot examples (three real past incidents with the briefings I wish I’d been handed), leaving room for the answer. Temperature 0 — I want the boring completion, this is not poetry night.

The prompt is a crime-scene summary, not the crime scene. Deciding what makes the summary turned out to be 90% of the project, and honestly, building the distiller taught me more about my own logs than the model did.

The lie detector

Now the part I’m actually proud of, because the obvious failure mode here is dangerous: a fluent, confident, wrong briefing at 3am is worse than no briefing — I’ll trust it precisely because I’m too tired not to.

The fix exploits something nice: the model may invent explanations, but the logs are right there. The prompt requires the hypothesis to cite two verbatim log lines as evidence, and then the script — dumb, deterministic Python — checks that those exact lines exist in the real distilled logs before posting anything.

for quote in extract_quotes(completion):
    if quote.strip() not in distilled_logs:
        return post_slack("⚠️ no confident hypothesis (evidence check failed)")

If the model quotes a log line that doesn’t exist, it invented its evidence, and its conclusion is presumptively garbage — discarded, no briefing, just an honest shrug. It’s terraform plan for prose: beliefs don’t ship until they’ve met evidence, and crucially, the verifier isn’t the model. Generation is allowed to be creative; verification is grep. Never ask the suspect to also be the jury.

Six weeks of production numbers

47 alerts since mid-July (a couple were self-inflicted, see: quarterly destroy ritual). Grading each briefing after the fact:

• 31 correct or usefully close — the hypothesis was right, or wrong in a way that still pointed at the right pod.
• 9 confidently wrong — but because the evidence quotes were real, I could see why it was wrong in seconds. A briefing with receipts fails gracefully; you audit the receipts, not the prose.
• 7 discarded by the evidence check. These are the wins nobody sees: seven fluent hallucinated briefings that never reached my phone. The lie detector’s catch rate is the whole reason I trust the other forty.

Best moment: a cascade of timeout alerts where the intern’s briefing led with a deploy from six hours earlier — it had noticed the git timestamp in the context block sat just before the first error, a correlation I’d missed because six hours “felt” too far away. Statistical models don’t have feelings about time. Point, intern.

Worst moment: it diagnosed a disk-pressure alert as a Redis memory issue because two of my three few-shot examples were Redis incidents. My examples had taught it that everything is Redis. Few-shot bias is real: the intern is a mirror, and I’d shown it a very Redis-shaped mirror.

Cost: about $0.14 of davinci per incident. Cheapest coworker I’ve ever had, and it never asks to reschedule the retro.

Why I didn’t fine-tune (although now I could)

OpenAI shipped a fine-tuning API this summer, so the rematch question is live again — and I declined, by my own rulebook. Fine-tuning owns context that never fits through the keyhole, stable distributions like two years of my commit voice. But incidents are the opposite: each one’s crucial context is fresh — today’s logs, this morning’s deploy — and worthless tomorrow. There’s no stable distribution to bake into weights; there’s only the keyhole, refilled per incident. Right tool, and for once I picked it by theory instead of by vibes.

Coda

GitHub spent the summer previewing Copilot — an autocomplete with hands is real now, it lives in editors, and yes, there’s a waitlist, because in this field even the future queues. Meanwhile my little intern suggests, cites, and gets fact-checked by twelve lines of Python.

I’ve stopped thinking the interesting question is “what can the model do.” The interesting question is what shape of harness makes a confident liar useful — distill what goes in, verify what comes out, keep the hands human. That harness cost me a summer of evenings and it’s mostly string handling. Somebody’s going to get very rich selling harnesses.

Next: GitOps, for real this time. The robots that apply commits have waited three posts, and now they have a coworker who reads logs.