How I turned a server firefight into a method my AI agent runs safely

npx skills add calvincchan/skills --skill server-hardening

A routine check on a client’s production Supabase server turned into a firefight. The recon lit up red: there was no host firewall running at all, the database was bound to every network interface — directly reachable from the public internet — and the disk was one write from full. The perimeter everyone assumed was there simply wasn’t.

Fixing it once wasn’t the interesting part. Making sure I never fixed it ad hoc again was. So I did two things: I stabilized the box, then I turned the fix into a repeatable, safety-gated method — one my AI agent now runs on any server without me babysitting every keystroke.

What the recon surfaced

The box was a self-hosted Supabase stack on a cloud VPS — roughly thirteen Docker containers, 124 days of uptime, and the quiet confidence of a machine nobody had touched in a while. That confidence was the problem.

ufw wasn’t even running — and that’s not the scary part. The scary part is that turning it on wouldn’t have helped: Docker had already routed the database’s port around the firewall entirely. Postgres was bound to every interface, reachable straight off the public internet, and the disk was so close to full that the next log line could have taken the database down. The perimeter everyone assumed was closed had never been there. Those are the gaps that survive for 124 days.

Three landmines worth naming

There were about six landmines. Three earned a place here because each one proves a different kind of judgment, not because they’d teach you the commands.

Docker silently bypasses the firewall. This is the marquee footgun. Even with ufw enabled, a published Postgres port stays reachable from the outside — because Docker publishes ports by writing DNAT rules into the nat table that route through the FORWARD chain. Traffic to a published container port never touches ufw’s INPUT chain, so your host firewall rules simply don’t apply to your containers. The fix lives in the DOCKER-USER chain: drop external hits arriving on the public interface while leaving loopback (nginx → Kong) untouched. If you self-host anything in Docker behind ufw, assume your firewall is not protecting your containers until you’ve proven otherwise.

The false alarm. A TLS certificate looked expired by about ten months — the kind of finding that invites a panicked reissue at 2am. I didn’t reissue it. It turned out to be an orphan: nothing referenced it. The certificate actually serving traffic was a Cloudflare Origin cert valid to 2035. The real fix was disabling a dead renewal timer that was generating noise, not reissuing anything. Verifying what’s actually in the serving path before acting is the difference between a fix and a self-inflicted outage.

The cascade. The full disk hadn’t just threatened the database — it had silently corrupted a half-finished kernel upgrade. That half-installed kernel, missing modules and unconfigurable, then jammed every subsequent apt operation, which meant no security patches could land at all. One failure had quietly created a second that blocked the fix for the first. You untangle that chain deliberately. You do not reach for apt --fix-broken install and hope.

From one-off fix to a gated method

Fixing one server helps one server for one day. What I actually wanted was a procedure that hardens the next box too, and the one after that — without me rediscovering the same footguns each time.

So the method is phased, and every phase is a gate: an objective, the specific safety concern that phase exists to address, and a verification check you do not advance past. It’s deliberately stack- and distro-agnostic. The skill states the concept — package updates, host firewall, SSH daemon, brute-force jail, process manager — and leaves the mapping to the target’s real tooling. apt or dnf, ufw or raw nftables, systemd or something older: the method doesn’t care, because the gates are about outcomes, not commands.

A handful of golden rules hold across every phase, and they’re all scars:

• Keep your current SSH session open through any firewall or SSH change. It’s your escape hatch when a rule locks you out.
• Never enable the firewall before you’ve allowed SSH and accounted for every listening port.
• Re-verify the running applications after every change — hardening that takes the site down is not hardening.
• Prefer offline, FOSS tooling over internet-exposed vendor agents on a box you’re trying to lock down.

The method also ships a small FOSS monitoring wrapper, srvmon, as a template — so the box that just got hardened also starts telling you when it’s about to drift back toward trouble.

Why the agent needs the method, not the reverse

The obvious version of this story is “AI hardened a server.” That’s backwards. The method is the asset — the gates, the golden rules, the failure lore all came out of doing this by hand, on real boxes, with real consequences. The Claude Code agent skill is what makes running that method repeatable and cheap on the next server.

The gates exist precisely because an unsupervised agent on a production box is exactly where “keep your SSH session open” stops being pedantic and starts being the thing that saves you. An agent will happily enable a firewall that locks itself out, because it has no scar tissue. The method is the scar tissue, encoded. That’s the sharp version of the lesson: the agent is only as safe as the method you hand it. Safety here isn’t a step you run at the end — it’s a property designed into every gate.

The skill is public and auditable. You don’t have to take any of this on faith:

npx skills add calvincchan/skills --skill server-hardening

Source, the srvmon templates, and the full SKILL.md are in calvincchan/skills .

Takeaway

The one-off fix helped one server for one day. The method — gated, safe-by-default, agent-runnable — hardens every server since, and each run makes it sharper, because every new footgun gets encoded as failure lore the next run already knows about. The final phase leaves a README and an append-only LOGBOOK on the host itself, so the next session — human or agent — skips the re-recon and picks up where the last one stopped. The system ends up documenting itself. That’s the difference between hiring someone who fixes your emergency and someone who makes sure it can’t quietly come back.