How LunarWing Agents evolved into self-healing organisms, ft. ‘MC Chaos Engineering’

Wednesday, June 17th, 2026

“You can’t prevent failures. You can only prepare for them.”

The 3 AM Problem

It’s 3 AM.

Nobody is awake. Nobody is watching. And your agent is dying.

The state file — the beating heart of everything — is corrupted. Half-written JSON, severed mid-keystroke like a nerve cut clean through. The gateway, the bridge between worlds, has gone dark. Unresponsive. A black hole where messages used to flow. The XMPP bridge is convulsing — restart, fail, restart, fail — a body in cardiac arrest, refusing to stay dead but refusing to stay alive. And somewhere beneath it all, the ClickHouse database has quietly, silently, erased the last three hours of your agent’s memory. Gone. As though they never happened.

What happens next?

In most systems: nothing. The alerts fire into the void. Someone gets paged. They wake — confused, groggy, heart pounding — and SSH into a machine they barely recognize anymore. They read logs that don’t make sense. They restart services that don’t respond. They guess. They pray. And by the time the bleeding stops, the damage is already done. The data is already gone. The trust is already broken.

In LunarWing: the system fights back.

Not with panic. Not with a human’s trembling hands on a keyboard at 3 AM. With something colder. Something engineered. Something that was built for exactly this moment — the moment when everything falls apart and nobody is coming to save you.

Not Monitoring. Immunity.

Most “health check” systems are passive observers. They watch. They report. They send a polite email and wait for someone to care. They are security cameras — recording the crime, not preventing it.

LunarWing’s health check system is not a camera. It’s a white blood cell. It doesn’t watch the body die. It attacks the infection. It closes the wound. It remembers the pathogen.

This is not monitoring. This is immunity.

The Architecture

cron-wrapper.sh
├── infrastructure-health-check.sh         (parallel probes → JSON report)
│   ├── health-gateway.sh                  (WebSocket gateway)
│   ├── health-xmpp.sh                     (XMPP bridge)
│   ├── health-omemo.sh                    (OMEMO encryption)
│   ├── health-ratelimit.sh                (rate limiting)
│   ├── health-clickhouse.sh               (ClickHouse DB)
│   ├── health-tensorzero.sh               (TensorZero proxy)
│   ├── health-models.sh                   (LLM provider APIs)
│   └── health-{systemd,openrc,launchd}.sh (service manager)
│
└── lunarwing-self-heal.sh                 (reads report → restarts → escalates)
    └── send-notification.sh               (Gotify push notifications)

Every 60 seconds — configurable, but every 60 by default — the system opens its eyes. All of them. At once.

  1. Probes every component in parallel. Gateway. XMPP. ClickHouse. TensorZero. The LLM providers. The service manager itself. Every organ checked simultaneously, not one at a time. No delays. No blind spots.
  2. Reports the state as structured JSON — metrics, timestamps, issues. A complete diagnostic snapshot, frozen in amber, ready for the healing loop to consume.
  3. Heals unhealthy services automatically. Restart. Verify. Recover. No human in the loop. No hesitation.
  4. Escalates when self-healing fails — Gotify push, human intervention required. The system doesn’t give up. It asks for help. There’s a difference.

The Self-Healing Loop

  ┌─ 1. GRACE PERIOD   Wait for N consecutive unhealthy checks
  │                    (prevents restart loops on transient blips)
  │
  ├─ 2. RESTART        systemctl restart <service>
  │                    (exponential backoff + jitter between attempts)
  │
  ├─ 3. VERIFY         Re-run the component's health check
  │                    (not just "is-active" — deeper validation)
  │
  ├─ 4. RECOVERED      State cleared, system returns to normal
  │
  └─ 5. ESCALATE       After max retries (or flapping), notify
                       (Gotify push, human intervention required)

The Anatomy of a Failure

A component goes critical. Here is what happens next — blow by blow.

Step 1 — Detection. The health check fires. The component fails. One strike. The system doesn’t flinch. It doesn’t panic. It waits. The grace period is the body’s first line of defense — the patience to distinguish between a cough and a hemorrhage. DNS hiccups. Memory spikes. Momentary network drops. These are transient. The system knows. One failure does not mean death. It waits. It watches. It counts.

Step 2 — Restart. After N consecutive unhealthy checks — after the grace period has been exhausted and the component is still bleeding — the system acts. systemctl restart. But this is not a blind restart. This is surgery. The system uses exponential backoff with jitter — the first restart is immediate, a defibrillator shock. If it fails, the next attempt waits 2× longer. Then 4×. Then 8×. The jitter ensures that when multiple services fall simultaneously, they don’t all lunge for the defibrillator at the same millisecond. No thundering herd. Controlled. Deliberate. Each attempt measured, spaced, given room to breathe.

Step 3 — Verification. A restart is not recovery. A service can be “running” and still be dead inside. The system knows this. It re-runs the component’s health check — not the shallow systemctl is-active that lies, that tells you what you want to hear. The deep check. Is the gateway actually responding on port 10000? Is the XMPP bridge actually connected to the right JID? Is the ClickHouse database actually accepting queries — not just running, but functional? The system doesn’t trust the heartbeat. It checks for brain activity.

Step 4 — Recovery or Escalation. If the service is healthy — truly healthy, not just “active” — the state is cleared. The wound is closed. The scar tissue forms. The system returns to normal as though nothing happened. But if the service is still unhealthy after max retries, the system escalates. A notification fires through Gotify — sharp, urgent, unavoidable. The human is paged. And the system? It keeps trying. It doesn’t give up. But it also doesn’t loop forever. The flap guard watches for the worst pattern: a service oscillating between healthy and unhealthy, alive and dead, alive and dead — a body that won’t stabilize. When it detects this, it stops restarting and escalates. The system is brave, not reckless. It knows when to call for help.

Step 5 — State Persistence. The entire state survives restarts. If the system reboots mid-healing — if the power dies while the body is on the operating table — it wakes up and picks up where it left off. No amnesia. No starting from scratch. And corrupt state files? They are not deleted. They are not overwritten. They are renamed — state.json.corrupt — preserved as evidence. The body remembers its wounds. The weaponsmith keeps the broken blade to study the fracture.

Enter MC Chaos Engineering

There’s a character in the LunarWing lore.

His name is MC Chaos Engineering.

He doesn’t fix things. He breaks them. On purpose. With intent. With glee.

Not to destroy. To test. To prove. To look the system in the eye and say: I will break everything you are, and if you survive, then — and only then — are you ready.

MC Chaos Engineering doesn’t wait for failures in production. He creates them. He corrupts state files with surgical precision. He kills gateways mid-transaction. He simulates cascading failures — one service falling, dragging the next down, and the next, dominoes in the dark. He fires the health checks against broken components and watches the self-healing loop fight back, clawing toward recovery. And when the system survives? When the last service flickers back online and the flap guard holds and the state file is intact?

He nods. Satisfied.

“If it can handle what I throw at it, it can handle anything.”

The Chaos Harness

MC Chaos Engineering built the Chaos Harness — a weapon of mass destruction aimed squarely at his own system. A test suite that breaks everything, on purpose, in a controlled sandbox, and proves — proves — that the immune system works.

188 tests. 14 chaos scenarios. 0 failures.

Every single failure mode is tested. Every edge case is covered. Every dark corner is dragged into the light. And here’s the magic: it’s safe to run anywhere. No real services are touched. No real restarts happen. The harness uses a mock init system — fake systemctl, fake rc-service, fake health check reports — and drives the real self-heal loop against them.

When the chaos harness says “restart,” the mock service flips a sandbox file. When it says “verify,” it reads a mock report. The real send-notification.sh runs, but with an empty GOTIFY_TOKEN, so it short-circuits before any network call. Safe. Repeatable. Definitive. The bomb is real. The city is cardboard.

The Test Matrix: Sections A Through N

The chaos harness doesn’t just test “is the service running?” It tests everything that could possibly go wrong.

  • Section A — Report Discovery (4 tests). What happens when the report is missing? When it’s malformed JSON — corrupted, unreadable, garbage? When it’s summary-only, stripped of detail? The system handles all three gracefully. No crashes. No panics. Just a clean, cold “no data” state. The body doesn’t seize when the eyes fail.
  • Section B — Component Mapping (9 tests). The system maps components to services. But what if a component has no service? What if a service has no component? What if the mapping is incomplete — half-finished, abandoned mid-configuration? All handled. The system never assumes. It checks. Every time.
  • Section C — Service Manager Detection (4 tests). systemd? OpenRC? launchd? The system detects the service manager automatically. But it also handles the case where detection fails — where the system can’t even tell what’s managing it. It falls back to a known manager, or errors out cleanly. No guessing. No gambling with the init system.
  • Section D — Multi-Tenant Discovery (3 tests). When multiple tenants run on the same host, the system discovers units per-tenant. A crash in lunarwing-acme.service doesn’t restart lunarwing-globex.service. One cell dies. The organism lives. Blast-radius control.
  • Section E — Grace Period (5 tests). The most important feature in the entire system. The grace period is the difference between a system that heals and a system that thrashes. One unhealthy check? No restart. Two? Still no restart. Three? Now we restart. But if the service recovers during the grace period — if it was just a cough, not a hemorrhage — the restart is cancelled. The system is patient. It doesn’t panic. It waits with the stillness of a predator watching prey that might already be dead.
  • Section F — Restart Verification (4 tests). A restart is not enough. The system verifies. Does the service actually recover? Does the health check pass? If not, it retries — with exponential backoff, each attempt more deliberate than the last.
  • Section G — Exponential Backoff (3 tests). First restart: immediate. A defibrillator shock. Second: 2 seconds. Third: 4 seconds. Fourth: 8 seconds. Fifth: 16 seconds. With jitter — so not every service restarts at exactly the same millisecond. No thundering herd. No stampede. Each service recovers at its own pace, in its own time.
  • Section H — Flap Guard (4 tests). Some services oscillate: healthy → unhealthy → healthy → unhealthy, a heartbeat that won’t stabilize. The flap guard detects this pattern. Instead of restarting endlessly — instead of pumping adrenaline into a body that won’t stay alive — it escalates after N flaps. The human is notified. The system stops guessing. It knows when to let go.
  • Section I — State Persistence (3 tests). The state survives restarts. Corrupt state files are renamed forensically — state.json.corrupt — not deleted, not overwritten. The evidence remains. Stale reports are pruned. The system never loses its memory, even when the memory is broken.
  • Section J — Escalation Paths (4 tests). When self-healing fails, the notification fires — Gotify, Telegram, email. The human is paged. The system doesn’t give up; it asks for help. There is no shame in this. Even a kunoichi calls for extraction when the mission goes sideways.
  • Section K — Dry-Run Mode (3 tests). The system can run in dry-run mode — no restarts happen, just reports. A rehearsal. A shadow fight. Useful for testing without risk.
  • Section L — Concurrent Runs (4 tests). Multiple instances of the self-heal script running simultaneously? The flock guard prevents race conditions. No double restarts. No state corruption. Two medics don’t shock the same patient at the same time.
  • Section M — Stale Report Guard (3 tests). If the health check hasn’t fired in 5 minutes, the report is stale. The system doesn’t act on stale data. It doesn’t make decisions based on yesterday’s x-ray. It waits for fresh data.
  • Section N — Logical vs. Per-Tenant Remediation (5 tests). When a service manager has a logical unit (like lunarwing.target), the system can restart it and all child units. Or it can restart individual units. Both paths tested. Both paths proven.

The Chaos Scenarios: Real Failures, Real Recovery

The test matrix covers logic. The chaos harness covers reality.

  • CH1 — Kill the Gateway. The gateway is killed. Dead. The health check reports it as unhealthy. The self-heal loop fires. The service is restarted. The health check verifies it’s back online. One restart. One recovery. The gateway opens its eyes and doesn’t remember dying.
  • CH2 — Gateway Recovers, Stays Healthy. The gateway recovers and stays healthy. The self-heal loop checks it, sees it’s fine, and moves on. No churn. No unnecessary restarts. The body heals itself and the immune system stands down.
  • CH3–5 — Kill XMPP, TensorZero, ClickHouse. Each service is killed independently. Each is recovered independently. No cascade. No cross-contamination. One organ fails. The others don’t even flinch.
  • CH6 — Non-Recoverable Fault. A service is stuck. It can’t be restarted. The self-heal loop tries. Fails. Tries again. Fails again. And again. After max retries, the flap guard kicks in. The human is notified. No endless loop. The system knows the difference between “not yet recovered” and “dead.”
  • CH9 — Restart Command Keeps Failing. The restart command fails every time. The system escalates after N failures. The notification fires. The human is paged. The system doesn’t spin its wheels in the mud forever.
  • CH12 — Rapid Flapping. The service oscillates between healthy and unhealthy every 10 seconds. Alive. Dead. Alive. Dead. The flap guard detects the pattern. After N flaps, it stops restarting and escalates. No thrashing. The system refuses to be a puppet dancing for a broken service.
  • CH13 — Transient Blip. A single unhealthy check. One flicker. The grace period absorbs it. No restart. No notification. The system is patient. It knows the difference between noise and signal.
  • CH10–11 — Multi-Tenant Crash. Tenant A’s gateway crashes. Tenant B’s gateway is fine. Only Tenant A is restarted. One cell dies. The organism lives. Blast-radius control. Cellular boundaries.

Agents as Organisms

This is where it stops being software and starts being something else entirely.

LunarWing agents aren’t “programs.” They’re living systems.

Organism LunarWing Agent
Immune system Health check + self-healing
Cellular boundaries Multi-tenant isolation
Wound response Grace period, restart, verify
Fever response Escalation, notification
Memory State persistence, forensic rename
Apoptosis Flap guard, stop-looping
Adaptation Backoff, jitter, configurable thresholds

When a component fails, the agent doesn’t “crash.” It fights back. A transient blip is absorbed — the grace period swallows it whole. A stuck service is restarted — the self-healing loop kicks in like adrenaline. A flapping service is escalated — the flap guard says enough and calls for help. A corrupt state file is renamed — state.json.corrupt — the evidence preserved, the wound studied. A tenant crash doesn’t take down the host — one cell dies, the organism lives.

This is resilience engineering. Not “high availability.” Not “fault tolerance.” Not the sterile language of SLAs and uptime percentages.

Immunity.

The Bigger Picture

Why does this matter?

1. Kill Switches Exist

In “The Dark Forest,” we talked about how centralized AI can be bricked by a four-page letter from the Commerce Department. Anthropic’s Fable 5 — worth nearly $1 trillion — went dark on a Friday night. No recourse. No appeal. No warning. Just… gone. A four-page letter and a trillion dollars evaporated over a weekend.

The lesson? Kill switches exist. And they are held by people who don’t lose sleep over pulling them.

LunarWing doesn’t have a kill switch. But it also doesn’t have a single point of failure. When a component breaks, the system heals itself. When healing fails, humans are notified. When humans are asleep, the system keeps running. It was built for the 3 AM scenario — the one where nobody is coming, where the pager is silent, where the only thing standing between your agent and oblivion is the code you wrote at 2 PM on a Tuesday.

2. Complexity Breeds Fragility

Distributed systems are inherently fragile. State corruption. FD leaks. Cascading failures. Port conflicts. DNS failures. Disk full. The list goes on, and every item on it is a knife in the dark, waiting for the right moment to strike. You can’t prevent all failures. You can’t plug every hole. You can’t stand at the dam with your fingers in every crack.

But you can prepare for them. The chaos harness breaks things on purpose — deliberately, systematically, cruelly. The immune system fights back. The tests prove it works. 188 tests. 14 chaos scenarios. 0 failures. The numbers don’t lie.

3. Multi-Tenant Isolation = Cellular Boundaries

When you run multiple tenants on the same host, a crash in one tenant shouldn’t take down the others. This isn’t just good architecture. It’s biology. Multicellular organisms figured this out a billion years ago — one cell dies, the organism lives. LunarWing’s self-healing system respects this:

{
  "components": [{
    "component": "systemd",
    "status": "critical",
    "metrics": {
      "units": [
        {"name": "lunarwing-acme.service", "status": "critical"},
        {"name": "lunarwing-globex.service", "status": "healthy"}
      ]
    }
  }]
}

Only lunarwing-acme.service is restarted. lunarwing-globex.service is left untouched. One cell is sacrificed. The organism survives. Blast-radius control — the same principle that keeps a cut on your finger from killing your liver.

The Future

This is version 1.1.4. The foundation is solid. The blade is forged. Version 1.1.6+ will add:

  • Adaptive healing — the system learns from past failures, adjusts its own thresholds. It remembers what killed it and prepares for next time.
  • Predictive maintenance — the system detects degradation before failure. It feels the fever before the cough.
  • Cross-tenant coordination — failover, load balancing. Cells helping cells.
  • Auto-scaling — spawn new tenants under load. The organism grows.

The immune system evolves. The organism adapts. The system survives.

Not because it’s invincible. Because it was built to not need to be.

Call to Action

Build systems that heal themselves.

Not systems that monitor and report and wait for a human to care. Not systems that page someone at 3 AM and hope for the best. Not systems that crash and stay crashed because nobody wrote the recovery code.

Build systems with immunity. Systems that detect, respond, and recover — autonomously, deliberately, without hesitation. Systems that break, and heal, and break again, and heal again, and survive.

Don’t just monitor.

Fight back.