This blog has been two blogs wearing a trenchcoat: infrastructure posts and GPT posts, politely ignoring each other. This summer they finally met, because I got paged at 3:12am, spent forty minutes spelunking through Loki, and discovered the root cause was — again — the same Redis pod OOMing. Forty minutes, for a diagnosis that fits in a sentence.
\nA sentence, you say. We have a machine for sentences now.
\nThe April post already litigated the tempting version — the closed loop where the model fixes things itself — and the verdict stands: not unattended, not on a box I like. A system that invents flags does not get kubectl.
But there’s a humbler job that’s still most of the pain: translation. When an alert fires, the on-call intern doesn’t fix anything. The intern gathers context, forms a hypothesis, and hands the human a briefing. So:
\nAlertmanager webhook → context gatherer → davinci → Slack message.
\nThe Slack message has three parts: a one-sentence hypothesis, the evidence, and a suggested first command — the kubectl describe or LogQL query I’d probably start with anyway. It proposes; I dispose. The plan is still the product.
The hard engineering isn’t the model call — it’s that an incident produces megabytes of context and the keyhole is still 2048 tokens. Commit diffs were easy; logs are a firehose of repetition. So most of the code is what I’ve started calling log distillation:
\ngit log entries touching those manifests, and a Loki query for the affected pod.[x347] connection refused ... — log spam compresses beautifully, because spam is the point of spam.The prompt is a crime-scene summary, not the crime scene. Deciding what makes the summary turned out to be 90% of the project, and honestly, building the distiller taught me more about my own logs than the model did.
\nNow the part I’m actually proud of, because the obvious failure mode here is dangerous: a fluent, confident, wrong briefing at 3am is worse than no briefing — I’ll trust it precisely because I’m too tired not to.
\nThe fix exploits something nice: the model may invent explanations, but the logs are right there. The prompt requires the hypothesis to cite two verbatim log lines as evidence, and then the script — dumb, deterministic Python — checks that those exact lines exist in the real distilled logs before posting anything.
\nfor quote in extract_quotes(completion):\n if quote.strip() not in distilled_logs:\n return post_slack(\"⚠️ no confident hypothesis (evidence check failed)\")If the model quotes a log line that doesn’t exist, it invented its evidence, and its conclusion is presumptively garbage — discarded, no briefing, just an honest shrug. It’s terraform plan for prose: beliefs don’t ship until they’ve met evidence, and crucially, the verifier isn’t the model. Generation is allowed to be creative; verification is grep. Never ask the suspect to also be the jury.
47 alerts since mid-July (a couple were self-inflicted, see: quarterly destroy ritual). Grading each briefing after the fact:
\nBest moment: a cascade of timeout alerts where the intern’s briefing led with a deploy from six hours earlier — it had noticed the git timestamp in the context block sat just before the first error, a correlation I’d missed because six hours “felt” too far away. Statistical models don’t have feelings about time. Point, intern.
\nWorst moment: it diagnosed a disk-pressure alert as a Redis memory issue because two of my three few-shot examples were Redis incidents. My examples had taught it that everything is Redis. Few-shot bias is real: the intern is a mirror, and I’d shown it a very Redis-shaped mirror.
\nCost: about $0.14 of davinci per incident. Cheapest coworker I’ve ever had, and it never asks to reschedule the retro.
\nOpenAI shipped a fine-tuning API this summer, so the rematch question is live again — and I declined, by my own rulebook. Fine-tuning owns context that never fits through the keyhole, stable distributions like two years of my commit voice. But incidents are the opposite: each one’s crucial context is fresh — today’s logs, this morning’s deploy — and worthless tomorrow. There’s no stable distribution to bake into weights; there’s only the keyhole, refilled per incident. Right tool, and for once I picked it by theory instead of by vibes.
\nGitHub spent the summer previewing Copilot — an autocomplete with hands is real now, it lives in editors, and yes, there’s a waitlist, because in this field even the future queues. Meanwhile my little intern suggests, cites, and gets fact-checked by twelve lines of Python.
\nI’ve stopped thinking the interesting question is “what can the model do.” The interesting question is what shape of harness makes a confident liar useful — distill what goes in, verify what comes out, keep the hands human. That harness cost me a summer of evenings and it’s mostly string handling. Somebody’s going to get very rich selling harnesses.
\nNext: GitOps, for real this time. The robots that apply commits have waited three posts, and now they have a coworker who reads logs.
","frontmatter":{"title":"I hired GPT-3 as my on-call intern","date":"September 19, 2021","description":"Wiring davinci between Alertmanager and Slack — six weeks of a language model reading my logs at 3am, with a built-in lie detector"}}},"pageContext":{"slug":"/i-hired-gpt-3-as-my-on-call-intern/","previous":{"fields":{"slug":"/the-rematch-my-fine-tuned-gpt-2-vs-a-frozen-giant-with-five-examples/"},"frontmatter":{"title":"The rematch: my fine-tuned GPT-2 vs. a frozen giant with five examples"}},"next":{"fields":{"slug":"/odd-bot-out-a-party-game-where-gpt-3-hides-among-your-relatives/"},"frontmatter":{"title":"Odd Bot Out: a party game where GPT-3 hides among your relatives"}}}}, "staticQueryHashes": ["3435731857","426816048","63159454"]}