I ran terraform destroy on my own production (on purpose)

In the k3s post I claimed Terraform is built on the same declarative idea as Kubernetes. This post is me paying for that sentence, because after a month of actually using Terraform I think it’s only half true — and the half that’s false is the interesting part.

But first, the fun part, where I delete my server.

The problem: my VPS was a snowflake

The Hetzner box running everything — this blog, the side projects, the commit-message bot — was hand-built. SSH in, apt this, curl that, tweak, forget. If it died, “restore from backup” really meant “restore from whatever I remember about April.”

So: everything into Terraform 0.13. The stack is three providers doing a small, honest job:

terraform {
  required_providers {
    hcloud     = { source = "hetznercloud/hcloud" }
    cloudflare = { source = "cloudflare/cloudflare" }
  }
}

resource "hcloud_server" "main" {
  name        = "snail-01"
  server_type = "cx21"
  image       = "ubuntu-20.04"
  location    = "nbg1"
  user_data   = file("cloud-init.yaml")
}

resource "cloudflare_record" "blog" {
  zone_id = var.zone_id
  name    = "ohsnail.com"
  value   = hcloud_server.main.ipv4_address
  type    = "A"
}

The cloud-init.yaml is where the magic compounds: it installs k3s and applies my manifests from git. So Terraform builds the world, cloud-init boots the cluster, and the control loops from last post take it from there. Three layers of “make it so.”

(The new required_providers block with source addresses is a 0.13 thing, by the way — third-party providers like hcloud finally live in the registry instead of “download this binary and put it in a folder like it’s 2003.“)

Adopting the existing server meant a tour through terraform import, which I’d describe as filling out immigration paperwork for infrastructure that already lives here. Tedious, worth it, do it resource by resource and run plan obsessively until it shows no changes.

The ritual

Here’s the thing though: a disaster recovery plan you’ve never executed is a religious belief. So on Friday night, with a beer for courage and a fresh state backup for cowardice:

terraform destroy -auto-approve
# ...12 seconds of watching my entire digital life deallocate
terraform apply -auto-approve

Four minutes and twenty seconds later, everything was back. Server, DNS, cluster, blog, TLS certs re-issued by cert-manager, the bot cheerfully suggesting I “fix flaky test.” The only data that needed restoring was two Docker volumes from a Restic backup — which is exactly the point: the ritual shows you, brutally, what is actually state versus what just looks important. Everything that came back automatically was never state at all. It was opinion, and opinions belong in git.

My server is no longer a pet. It’s cattle with a herd size of one, and I can now hold the entire production environment in a git clone.

I’m making this a quarterly ritual. Nothing keeps infrastructure honest like scheduled death.

The part where I correct last month’s take

Now the confession. Kubernetes and Terraform are both “declarative,” but they reconcile in fundamentally different ways, and conflating them (as I did) hides the most important operational fact about Terraform:

Kubernetes reconciles continuously. Control loops compare desired vs. actual many times a minute, forever. Drift gets corrected while you sleep.

Terraform reconciles episodically. Nothing happens until a human runs plan. Between runs, Terraform isn’t watching anything — it’s not a loop, it’s a snapshot ceremony.

Which leads to the mental model that actually made Terraform click for me: the state file is not a record of your infrastructure. It’s Terraform’s beliefs about your infrastructure. Reality can change behind its back — a colleague clicks something in the console, a provider auto-upgrades a field, you “quickly fix” a firewall rule at 2am — and the beliefs go stale. terraform plan is the moment beliefs are confronted with evidence, and the diff you’re reviewing is really three-way: what you declared, what Terraform believes, what’s actually out there.

Sound familiar? A confident internal model drifting away from reality, generating authoritative statements about things that no longer exist — that’s the same failure mode I spent the spring documenting in GPT-2, which happily invented APIs because nothing ever forced its beliefs to meet the world. Terraform ships the antidote as a first-class command. Half of engineering, I’m starting to suspect, is just deciding when your system’s beliefs get checked against reality: every few seconds (Kubernetes), every deliberate ceremony (Terraform), or never (my old snowflake server, and, well, GPT-2).

Continuous reconciliation is strictly stronger if you can afford it. Terraform’s episodic model is the tradeoff you accept for managing things that are expensive, slow, or dangerous to change — you want a human reading the diff before a database gets replaced. The plan isn’t bureaucracy. The plan is the product.

Honest footnotes

• The state file is sacred and radioactive. It contains everything, sometimes including secrets, and losing it means Terraform forgets your infrastructure exists. Mine lives in Terraform Cloud’s free remote state with locking. Never in git.
• destroy will absolutely also destroy things you forgot depend on each other. Read the destroy plan slower than you’ve ever read anything.
• Some resources are lying about being cattle. Volumes, databases, anything holding user data — mark them prevent_destroy and treat them as pets with paperwork.

Next

The obvious itch: Terraform applies when I run it, but the manifests inside the cluster already live in git… so why am I the one running anything? The word for that itch is apparently “GitOps,” and it’s next quarter’s rabbit hole.

Also my GPT-3 invite is still not here. OpenAI, I have a benchmark of exactly one silly task waiting.