NixOS Config

Overview

This is the NixOS flake that sets up every computer I run day to day. One repo builds the full system for three NixOS machines and the firmware images for two OpenWrt devices:

ThinkPad T480s, Raspberry Pi 4, OpenWrt One, and Raspberry Pi 1 B+ (photos by SłodkiKarmelek, Laserlicht, BananaPi, and Lucasbosch on Wikimedia Commons, CC BY-SA)

The machines

I focused on:

Motivation

I wanted every machine set up from one place, so that reinstalling or moving to a new drive is one command instead of redoing everything by hand. A lot of the config is workarounds for hardware or software quirks, and I kept forgetting why they were there, so each one has a comment next to it saying what broke and why the fix works.

Architecture

Each machine has a small folder with only the things specific to it, like its hardware, networking, and boot setup. Everything else is shared:

  1. Core: the packages and settings every machine gets
  2. Roles: one per kind of machine (laptop, server, Pi), and each machine picks exactly one
  3. Features: opt-in pieces a role pulls in, like the desktop, audio, virtualization, or laptop power handling
  4. Hardware: quirks for a specific machine, like the T480s lid and dock handling

I also use Home Manager in the same repo to configure the programs I use, so my shell, git, and editor setup is the same on the laptop and the server.

Passwords and keys are stored encrypted in the repo with sops-nix. Each machine decrypts its own secrets using a key derived from its SSH host key, so I never have to copy a private key onto a new machine.

Most of the media stack on the Pi lives in a second, private repo that this one pulls in, so the public repo stays shareable.

Implementation

Laptop and server

The T480s runs the sway desktop along with my dev tools and drivers for the lab equipment I use. A lot of its config is power handling for sleep, the lid, and my Thunderbolt dock.

The server runs headless. Its AMD RX 580 is used for compute on the host until I start the Windows VM, at which point the card gets handed to the VM and comes back to the host when the VM shuts down. I also set up Wake-on-LAN so I can turn the server on remotely instead of leaving it running.

Raspberry Pi 4

I build the Pi's system on my laptop and copy it to the Pi over SSH. My laptop runs the Pi's ARM build tools through an emulator instead of cross-compiling, because cross-compiling hit bugs in unrelated packages that emulation doesn't. The first build took about 36 hours, but after that only the changed parts rebuild.

The Pi's media and backups live on an 8TB encrypted USB drive, which unlocks on boot using the same encrypted secrets as everything else, because the Pi has no TPM to hold the key. The media, photos, and my laptop's backups each get their own volume with its own size limit, and the apps' data has a volume of its own, so one filling up can't stop the others from writing.

OpenWrt

Neither OpenWrt device runs NixOS. The flake uses OpenWrt's ImageBuilder to put together a firmware image from the settings in the repo, so a fresh flash always comes back in the exact same state.

The OpenWrt One connects to the internet over Wi-Fi and shares it over its own Wi-Fi and wired ports, with DNS going through the ad blocker on the Pi.

The Pi 1 B+ runs OpenWrt because NixOS has no prebuilt packages for its ARMv6 CPU, so it would have to compile the whole system from source on a 700MHz single core. OpenWrt already has those packages built, so the whole image is about 21MB. It lets me reach my parents' home network from anywhere on Tailscale.

Tests

I split each shell script into a decision half and an action half. The decision half reads its inputs (like a fake sysfs tree or some JSON) and prints what it would do, and the action half stays tiny and just does it. That way I can test the decision half with fixtures without needing the real hardware, and that's where the bugs actually were.

There are about 25 of these test suites, covering things like the laptop lid logic, display layouts, and the Pi's disk space guards. I run them all with one make test. After writing a suite I put the bug back in and make sure the test actually fails, because I had one guard that was always true and silently did nothing for its whole life.

Challenges

  1. The Pi rebooted with no network and never unlocked its drive
    • Solution: I had told two of the Pi's startup services to run before the drive was mounted, which made a loop in the boot order. systemd breaks a loop by dropping a random job, and it happened to drop the one that brings up the network. It only showed up on a reboot, 19 days and 8 updates after I caused it. I recovered by pulling the SD card and pointing it at the previous version, removed the ordering (it wasn't needed), and added a check that looks for loops on the live machine before I reboot it.
  2. Every download on the Pi stopped while df said there was 151 GiB free
    • Solution: The media volume had hit its size limit, but df shows free space for the whole drive, not the limit. I changed all the disk guards to read the volume's own usage, and moved the app data onto its own volume so a full media library can't block the apps from writing.
  3. The Pi's whole USB 3 bus would die, taking the media drive with it
    • Solution: The drive's USB enclosure and the Pi's USB controller don't get along in the faster UAS mode, and one stalled command could crash the controller. Unplugging the drive did nothing because the controller was the part that was stuck. I forced the drive into the older, slower USB mode, which is still more than fast enough to stream video, and added a guard that reboots the Pi if the bus dies anyway.
  4. Deploys to the Pi would hang forever, which looked like a network problem
    • Solution: Tailscale SSH was answering the connection and waiting for me to approve it in a browser, which a script can never do. I turned off Tailscale SSH on the Pi and kept it on the laptop and server, where I do use it.
  5. The Pi kept running out of memory
    • Solution: It was running about 6 GiB worth of services on 4 GB of RAM. I paused my photo library app, which was the second largest, capped the memory of the other big services, and added a guard that pauses downloads and some background apps when memory pressure gets too high.

Repositories