I wanted a small single-board device to send all of its traffic through a WireGuard tunnel, and to send none of it anywhere else if that tunnel was not up.
That second half is the whole exercise. A VPN that carries traffic when it is working is easy. A VPN that is fail-closed (where the failure mode is no connectivity rather than silently unprotected connectivity) is a design problem, and on an appliance-class device it is a design problem with most of the usual tools removed.
The result works, and it taught me four things I did not expect. One of them is that the self-healing mechanism I added had been hiding a bug for weeks.
The constraints are the interesting part
The device runs a purpose-built embedded Linux image. There is no package manager, no general-purpose filesystem, and the root filesystem is read-only. Exactly one directory persists across reboots and upgrades.
That inverts the usual assumptions:
- Everything is a file you place, not a package you install. The units, the scripts and the credentials all live in the one writable directory.
- Upgrades replace the OS. Anything outside that directory is gone after an image update. That is a strong reason to keep the entire configuration in a small set of files you can copy back, and an equally strong reason not to depend on anything in
/etc. - There is no console. Recovery is over SSH, which is to say recovery depends on the very networking you are about to firewall.
systemd is present, which is the one piece of leverage worth having. The whole design reduces to: one oneshot service that builds the tunnel, and one timer that checks it.
boot
└─ vpn.service (Type=oneshot, RemainAfterExit=yes)
ExecStartPre clear the kill switch ← so the tunnel can be built at all
ExecStartPre disable IPv6
ExecStart bring up the tunnel, replace the default route
ExecStartPost apply the kill switch ← nothing may leave except via the tunnel
every minute
└─ vpn-watchdog.timer → vpn-watchdog.service
interface missing, default route not via the tunnel,
or last handshake older than 120s ⇒ systemctl restart vpn.serviceThe bootstrap paradox
A kill switch says: block all outbound traffic except to the tunnel endpoint and the local network. Build the tunnel first, then close the door.
Except building the tunnel requires reaching the provider’s API over HTTPS, to fetch a token, retrieve a server list, and register this device’s public key with a gateway. All of that is ordinary internet traffic, to addresses you do not know until you have asked. The kill switch from the previous boot is still in place and blocking exactly that.
So the first ExecStartPre step deliberately removes the protection:
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStartPre=/storage/.config/vpn-killswitch-clear.sh
ExecStartPre=/storage/.config/disable-ipv6.sh
ExecStart=/storage/.config/vpn-connect.sh
ExecStartPost=/storage/.config/vpn-killswitch.shThere is a window, every boot, where the device is unprotected. That window is real and cannot be designed away without pre-pinning endpoints. What it can be is short, bounded and honest, and the failure path has to close the door rather than leave it open:
# on any failure: tear down the tunnel and restore the LAN default route
trap 'ip link del wg0 2>/dev/null; ip route replace default via "$GATEWAY" dev eth0' ERRThat trap is the part people leave out. A script that exits halfway through leaves a device with no tunnel, no kill switch and a working internet connection, the exact state the whole design exists to prevent. Deciding what happens on the error path is the design; the happy path is just plumbing.
The resulting rule set is small enough to read in full, which is the point:
-P OUTPUT ACCEPT
-A OUTPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -d 192.168.1.0/24 -j ACCEPT
-A OUTPUT -d <endpoint>/32 -p udp --dport <port> -j ACCEPT
-A OUTPUT -o wg0 -j ACCEPT
-A OUTPUT -j REJECT --reject-with icmp-port-unreachableA default policy of ACCEPT with a terminal REJECT is functionally equivalent to a default DROP, and considerably easier to recover from if you make a mistake while connected over SSH. REJECT rather than DROP because an application that gets a refusal fails immediately and visibly, while one that gets silence sits in a timeout and looks like a performance problem.
The rules are never written to disk. They exist only in memory and are re-applied by ExecStartPost every time the tunnel comes up, with the platform’s own firewall-persistence service left disabled. That is deliberate: persisted rules would be a second source of truth that can disagree with the service, and a stale rule set surviving a failed tunnel build is worse than none.
The ordering typo systemd will not tell you about
The unit originally read:
[Unit]
After=network-online.service
Wants=network-online.serviceThat unit does not exist. The correct name is network-online.target.
systemd does not treat a dependency on a non-existent unit as an error you will notice. The ordering constraint simply has no effect, so the service could (and sometimes did) start before the local network was up. The tunnel build would fail because the provider’s API was unreachable, the error trap would restore the direct route, and the device would sit there unprotected.
This is worth generalising. Ordering directives are strings, not references. After= a typo and you have expressed nothing at all, with no warning at unit load and no error at boot. When a unit’s ordering matters, confirm the graph rather than the file:
systemctl list-dependencies --after vpn.service
systemd-analyze verify /storage/.config/system.d/vpn.serviceThe watchdog was hiding the bug
Here is the part that actually changed how I think about this.
The watchdog runs every minute from two minutes after boot, checks three conditions (the interface exists, the default route goes via it, and the last handshake is under 120 seconds old) and restarts the service if any fails.
It worked. Every time the ordering bug caused a boot-time failure, the watchdog restarted the service a minute or two later, by which point the network was up, and the tunnel came up cleanly. By any external measure the device was fine. The tunnel was always up when I looked.
So a boot-ordering defect survived for weeks inside a system that was measurably healthy, and I only found it by reading the unit file for an unrelated reason.
Self-healing mechanisms convert hard failures into soft ones. That is their job, and it is genuinely valuable. But the soft failure still has a cause, and if nothing records that the healing happened, the cause is now invisible. The fix is to keep the watchdog and make its recovery loud:
logger -t vpn-watchdog "RECOVERED: ${reason} (restart #${count} since boot)"A watchdog that only logs OK tells you the current state. A watchdog that logs why it had to act tells you about the defect. On this device the distinction was the difference between “working” and “working by accident, once a minute, for weeks”.
I would now apply the same test to any automated remediation: can I tell, afterwards, that it fired? If not, what I have built is a filter on my own monitoring.
Fail-closed means every path, not the interesting one
Routing all traffic through the tunnel is the obvious half. Two paths escaped it.
DNS. Name resolution was pointed at the local router, and the kill switch permits all local traffic, so lookups left over the ordinary connection while payload traffic went through the tunnel. Not a leak of content, but a complete log of every name requested, in the one place the design was supposed to remove it from. The fix is to use resolvers reachable only inside the tunnel. That has the pleasing property that when the tunnel is down, name resolution stops, which is exactly what “fail-closed” should mean.
IPv6. The rule set above governs IPv4. A dual-stack device with a working IPv6 path will happily use it, around everything. This build disables IPv6 in three places (in the network configuration, by sysctl at every tunnel start, and with a default DROP policy on the v6 firewall) because it only takes one of those to be missing.
The generalisable point: a kill switch is an assertion about all egress. Anything with its own resolution path, its own address family or its own transport sits outside a rule set written for one of them. Enumerate the paths, then write the rules.
Verification has to test the failure
Confirming the tunnel is up is easy and nearly worthless:
wg show # recent handshake, counters moving
ip route show default # default dev wg0
curl -s https://ifconfig.me # the tunnel's address, not the line'sAll three pass on a device whose kill switch does not work at all. They test the happy path, and the happy path was never the risk.
The test that matters removes the tunnel and checks that nothing gets out:
# take the tunnel down without touching the firewall
ip link set wg0 down
curl -s --max-time 5 https://ifconfig.me # must fail, not return your real address
dig +short +time=2 example.com # must fail
ping6 -c1 2606:4700:4700::1111 # must failThree failures is a pass. If any of them returns an answer, the kill switch has a hole in the exact place the design was supposed to be strongest, and you will never discover it from the working state.
Then restore the tunnel and confirm the watchdog notices within its interval, because the recovery path deserves the same treatment as the protection path.
What this device taught me about bigger systems
The constraints here are extreme (no package manager, one writable directory, recovery only over the network being firewalled) but the lessons scale up without modification:
The error path is the design. The happy path was a few lines of wg and ip route. Everything genuinely considered on this device concerned what happens when a step fails.
A dependency you cannot resolve is a dependency you have not declared. The .service/.target typo cost weeks of undetected boot failures because nothing validates that a name refers to something real.
Automated recovery must be observable, or it becomes concealment. This is the one I would take into any production system. A retry, a restart, a failover or a self-healing job that leaves no trace turns a recurring defect into an invisible one.
Fail-closed is a property of every egress path, and it is only proven by breaking the tunnel. A control validated exclusively in its working state has not been validated.
None of this is specific to a small box in a living room. It is the same reasoning as designing an access path that denies when the policy engine is unreachable, only smaller, cheaper to break, and a good deal easier to test on a Sunday afternoon.
