When to keep troubleshooting, and when to just blow it all away.

I often can’t think of what I should tinker with on my homelab, and then sometimes it decides for me.

Life, uh, gets in the way.

While I always have a to-do list pertaining to my homelab lingering around me like a ghostly specter, for the most part I have battle-tested, mission-critical systems and configurations in place before I go off galavanting around and enjoying real life. Don’t get me wrong, I love this hobby, but it’s not my day job, I’m not an expert. It came out of a place of necessity and admittedly naive hope for career advancement when I first dove head first into the command line. My Tech Career dreams are dashed at this point but that’s a story for another day. What I’m trying to say is that while I am passionate about this hobby, specifically preserving my data, there are times where I get caught up with life and I can’t do daily or weekly sysadmin work/play the way I’d like to do. As such as what happened this past summer.

My schedule changes over the summer, I switch from a night shift to day shift for two months. These two months are early rises, long days with hard work, and early to bed nights. Not much time for sysadmin. Especially now, with a kid on the way much of my evenings were spent reconverting our 2br house into the original 3br configuration so the baby could have her own room. Until my brother picked up a painting job he needed my help with. So I was working after work for these two weeks and was out of the house from 530 to 2130. Over the course of these two weeks we had a summer storm and brief power outage. Unsure of the exact cause, I suspect either the aging UPS providing power to my homelab must finally be crapping out, or my less-than-desirable ZFS pool setup with 2 usb connected arrays overwhelmed the bus in the Mac mini, maybe a combination of both, but something caused the pools to throw I/o errors all over my backups and degraded the pool. Expecting failed disks on more aging hardware, I was too overwhelmed to handle this so simply I shut down the machine.

Checksum before you wreck some (I have no idea I’m so bad at titles)

Once our prioritized projects for our newborn started wrapping up, and the partition wall between our future daughters room and my office was finished, dry walled, mudded, and sanded, I had my office to put back together. I turned the machine back on and ran the ol’ faithful scrub on the degraded pools and they came back repaired. I felt comfortable clearing the errors (as the error message instructed me to do) and the pool was soon off backing up data via syncoid again. After moving my NAS and rearranging the office to better suit new layout, I woke up to my daily zpool status showing my main pool, my source of truth, in the middle of a scrub with a paltry 1-2 checksum errors on 2 disks. No offline or degraded warnings, so the pool was still available. Meaning I could still watch Plex, but I was still concerned. The scrub finished showing repaired but with more checksum errors, smartctl showed nothing in the way of failure, so I ran zpool clear on this pool this time, feeling much more comfortable since these are newer disks. But soon checksum errors once again started propagating. Something was going on and I needed to figure out what, fast.

I immediately took apart the machine and started reseating cables and memory. Anything in easy enough reach. I suspected power or memory issues due to the nature of the kernel logs showing no signs of error before an abrupt shutdown/crash, and I could sort of see how the way ZFS uses memory could lead to these specific types of fanning out checksum errors, though I don’t know enough about the mechanics of the software itself to say for certain. I had a few kernel panics during this troubleshooting, and I have a few logs saved from what might have happened but at this point I was power cycling maybe a little too often and I don’t think they actually mean anything. If I was a better sysadmin giving a real post-mortem I’d share them.

the easy way

If I was a better sysadmin I’d also have done a memtest by now, but I haven’t. What I did first instead, was go the power route. I needed a new UPS anyway, the battery was essentially dead, and I had recently put a window unit AC in that could easily be causing power fluctuations that an aging UPS couldn’t correctly protect from anymore. After setting up a new cyberpower, and getting NUT working for the first time, I was still experiencing persistent and elusive reboot/crash/network-drop issues. dmesg and journalctl logs continued to show nothing of value to me or to Claude, I have successfully continued avoiding a memtest even after everything because I don’t want to deal with the down-time, or buy more RAM. (Prices are even crazier than the crazy they were 6 months ago when I bought these sticks!) We’ll see if that comes back to bite me.

aaaand it’s gone.

Instead, I decided to undo the overclocking on the 32gb DDR5-6000 RAM I set when I built the NAS. It turns out the 6000 in the name on the box is simply ambitious? It is this point in the saga where I’d like to refer you to the earlier statement I made in which I said this isn’t my day job, that I’m not an expert, and that this is simply a hobby I picked up out of necessity and belief in digital sovereignty that I happen to enjoy. I now realize that most of the online guides I followed w/r/t building a computer are geared towards gaming, so when they gave me advice like turn on EXPO and turn the number from the default to the one that matches the one on the box I thought I was doing something sensible. For a NAS running ZFS that should likely have had ECC memory from the start (even though the literal book on ZFS: ZFS Mastery, says the need is overblown) maybe I still shouldn’t have been overclocking it. The jury is still out on this fix, because as soon as I rebooted—and maybe this is something I should have seen coming—proxmox could not import rpool, the root disk for the operating system. I was stuck in initramfs. I somehow borked my system and was stuck troubleshooting my mission-critical custom built 6 month old NAS and installation of proxmox before my 2 year old install on my custom built Arch Linux desktop.

how many times am I gonna have to teach you this lesson old man?

Digging into the issue, I found my big zpool imported fine with all data and snapshots intact following the same -f force flag instruction the status provided for a new host, so it seemed it wasn’t a ZFS issue. I tried many import methods. Eventually I tried changing the host ID anyway after I enlisted the help of Claude because I quickly ran out of ideas, but nothing it suggested or I could think of, would import the zpool. The main error message I kept getting said ‘one or more devices not available’, but the two nvme drives in the mirrored vdev both showed up as block devices with ZFS tags using zbd. Though one of the nvme drives did look dismal in smartctl, so I pulled it thinking maybe the pool would import as degraded if I simulateda failure by pulling the potential problem device. It didn’t. After a few more hours of futile troubleshooting with Claude I decided enough was enough and I needed to just reinstall the OS.

I would have done this much much earlier. Any time I’ve had an issue with an LXC container I pull the last backup and I’m back up in literal seconds. Unfortunately my rpool was not backed up at the time. This was exactly what I wanted to avoid. Luckily all my data on the big zpool was intact, and even in the event of a catastrophe has replicated datasets across a number of targets. Since I’m self aware enough to know I’m not the best sysadmin, the least I can do is make sure I have backups of backups of backups so when I inevitably break things I can restore them. Perhaps this alone makes me at least a decent sysadmin? I just setup NUT on proxmox for the new UPS, and the host node contained all of my sanoid configs, among other things, and again, if I was a better sysadmin I’d have had a golden image of my host node just the way I like it so I could deploy it on the slightest whim like I can with my containers. One of those lessons you just need to learn yourself I guess.

problems, uh, find a way.

I was hoping that after the reinstall there wouldn’t be much left to the story. The NAS still only has a single (but healthy) nvme drive, (have you seen the prices lately?), memory is no longer overclocked, the big zpool imported fine, and since the replication targets pull from the source as soon as sanoid was running again the targets started syncing right where they left off. My LXC containers restored at the click of a button, and all my fun homelab services were available for me to enjoy again. But since I was reconfiguring sanoid anyway and finally getting back into a sysadmin groove after an extended leave of absence, I decided to do some much needed maintenance and poke around my system and found a massive pileup of syncoid snaps on both of my backup pools.

sysadminning never really stops, does it?

Syncoid only prunes its own snapshots, it makes one on the source with its own identifying hostname, and pulls an incremental from the previous one before pruning anything with said identifying hostname, if I understand correctly. My issue, was that I had two machines pulling from the same source and thus, pulling each others snapshots, half of which could not be pruned by the receiving source. Luckily once again, since the community and figureheads working on ZFS and its accompanying sanoid/syncoid tooling are bonafide geniuses, they have a —no-sync-snap flag for just this A>B, A>C replication scenario. I moved the vzdump LXC container backup operation to the local rpool dataset to increase it’s speed instead of sending the tar files over NFS to the backup server, and I added the rpool dataset to the syncoid schedule. Things seemed to be working smoothly, until I checked snapshots again. Both backup pools had the same daily anchor, but differing hourlies. I had configured my proxmox node to only retain 3 hourlies, so by the time the second backup server ran syncoid, the hourlies pulled by the first server had been pruned. While sure they were each updated to the latest hourly snapshot that was present when the backup occurred, I didn’t like the drift. I wanted an intact replication chain between the two. I have such a hard time wrapping my head around this sometimes though. It shows just how tricky this stuff can be, even to someone who is more of a power-user than average. I go from being concerned to this issue like I was earlier, to now during reflection being not so sure it is as much of an issue if the common anchor between two backup servers is only 12-24 hours behind each other. But then I remember about the rollback that happens when each backup server reaches out to the host again, finds different snapshots it cant replicate from, rolls back to the the last common snapshot and incrementals to the new hourly. This can cause a bunch of churn, depending on the dataset. So I increased the snapshot retention to accommodate the overlap and after this afternoons syncoid run they seem to be much closer with a stronger chain forming. So for now, I have nothing to do but sit back and ensure smooth operation before messing with things further, as I clearly have a tendency to do.