A Rebuildable Single-Board Media Appliance

A Raspberry Pi in the living room had been running a media player for years. It worked, and nobody (including me) could have rebuilt it.

Every setting had been made through a remote session at some point, none of it was written down, and the SD card was the only copy. That is an ordinary domestic situation and a completely unacceptable one on any system I would be responsible for professionally. So I treated it as an estate of one and applied the same standard: if it cannot be rebuilt from documentation, it is only working by accident.

Objective

Produce an artefact that reconstructs the device from a blank card, without reference to the running system, and prove it by surveying the live box in full first.

Survey the running device
        ↓
Inventory every deviation from default
        ↓
Write the rebuild as a kit: files + steps + verification
        ↓
Check the kit against the survey: every live setting either
appears in the kit or is recorded as a deliberate omission
        ↓
Fix the defects the survey exposed

The last two steps are where the value was. Writing the guide was clerical. Reconciling the guide against reality found real faults.

What the survey covered

Not “the settings I remember changing”, but every place the device holds state:

• OS image and version, and which settings survive an image upgrade. On this platform exactly one directory persists; everything else is replaced. That single fact determines the entire shape of the rebuild.
• Application configuration, diffed against defaults rather than transcribed, so the kit records the dozen settings that were actually changed instead of the several hundred that were not.
• Installed components, each with its origin and whether it was still in use. Several were not.
• Service flags: which platform services had been enabled or disabled, and why.
• Network configuration, including addressing, DNS and the disabled parts of the stack.
• The tunnel: unit files, scripts, firewall rules as they existed in memory, and the live handshake state.
• Storage use, to know what the rebuild actually needs to carry.

What reconciliation found

Five defects, none of which the device’s behaviour revealed:

A unit-ordering directive that referred to a unit which does not exist. After=network-online.service instead of network-online.target. systemd accepts a dependency on a name that resolves to nothing, so the ordering silently had no effect and the tunnel could be built before the network was up.

A watchdog that had been concealing that fault for weeks. It restarted the failed service within two minutes of every boot, so the tunnel was always up by the time anyone looked. The bug was invisible precisely because the recovery worked. That one changed how I think about automated remediation, and I wrote it up separately in A Fail-Closed VPN on an Appliance-Class Device.

Name resolution outside the tunnel. Payload traffic went through the tunnel; lookups went to the local router. A partial protection presented as a complete one.

Accumulating host routes. Each tunnel restart added a host route for the new endpoint without removing the previous one. Harmless in itself, and a clear sign that the setup path had been written and the teardown path had not.

An unreferenced later revision of the main script, sitting beside the original with improvements that had never been adopted, and nothing indicating which file the service actually ran.

Every one of those is the same class of finding: a system that behaves correctly while carrying a defect, where the only route to discovery is reading the configuration against a written expectation.

The kit

The deliverable is a directory that mirrors the device’s writable path, plus a document that installs it:

files/storage/.config/
  ├── <tunnel scripts>
  ├── <firewall apply / clear scripts>
  ├── <watchdog script>
  └── system.d/
      ├── vpn.service
      ├── vpn-watchdog.service
      └── vpn-watchdog.timer

Four properties made it worth having:

Copy, do not retype. The files are the configuration. The document is the order to apply them in, plus the things that cannot be expressed as a file.

Credentials are referenced, never included. The kit names the credential file, its permissions and its format. The values are not in it and never will be.

Deliberate omissions are recorded. Settings present on the live box that the kit intentionally does not restore are listed with a reason. Silence would otherwise be indistinguishable from an oversight next time.

Verification tests the failure path. Confirming the tunnel is up proves nothing about the kill switch. The checklist brings the interface down and requires that name resolution, HTTPS and IPv6 all fail:

ip link set wg0 down
curl -s --max-time 5 https://ifconfig.me   # must fail
dig +short +time=2 example.com             # must fail
ping6 -c1 2606:4700:4700::1111             # must fail

Three failures is a pass.

What it demonstrated

The technology is modest: one small board, a tunnel, three unit files. The discipline is not, and it transferred straight from professional work:

• A configuration survey diffed against defaults, rather than a list of remembered changes.
• Reconciliation as the actual defect-finding mechanism.
• Recovery documentation that assumes no access to the thing being recovered.
• Verification that exercises the protective control by breaking it.
• Secrets referenced by location and permission, never captured in the artefact.

The device now runs the same as it did before. The difference is that a failed SD card is an afternoon with a written procedure rather than a reconstruction from memory, and five latent faults are fixed rather than waiting.