moved my ZFS pools around

Remember when I said I wasn’t scared of Arch Linux on my server?

I lied. While I think it’s fine for containers, I’ve had my ZFS pool attached to my Arch desktop since I built it back in March and have been white knuckling the entire experience.

why did I attach it to my desktop in the first place?

Good question. The pool in question is a RAIDz1 4 drive wide vdev with 24TB usable storage nestled snugly inside a Quad Elite Pro USB-C cage from OWC. For the better part of a decade I was using OWC SoftRAID on MacOS, and simply got used to having a large volume attached directly to my computer for my photography use. I’ve spent the last few years migrating to using Linux full time and this habit seems to be one of the last holdovers from those days. A network mount is typical in a true server or workplace environment. Learning how to install ZFS on Arch was useful also; don’t get me wrong. Though terrifying for long term use. I was stuck using the LTS kernel and every DKMS kernel update had me biting my nails.

My previous setup was zpool ‘gamehenge’ attached to my Arch desktop that would replicate using sanoid/syncoid to the zpool ‘catalyst’ that was the main volume for my proxmox homelab. I wanted to move Immich from a docker container on my desktop to the proxmox homelab, but I would have messed up the replication if I was backing up my phones photos to catalyst.

well what are you gonna do about it?

I had an epiphany—why was I using two pools like this? It was causing unnecessary I/O and using two pools in production was going to lead to issues down the line if changes were ever accidentally made on the target instead of the host.

ZFS and proxmox make it really easy to pivot in this situation, essentially all I had to do was zpool export gamehenge from desktop and zpool export catalyst from proxmox, and then zpool import gamehenge on proxmox and update bind mounts for existing containers.

Before I could do any of that, I had to set up a backup server for catalyst. Luckily there’s a stack of Mac minis in my office just waiting for a reason to bootstrap into existence. I grabbed a now ancient 2012 Mac mini from the stack and installed Debian 13 and ZFS. I am using the Minis Thunderbolt port to plug in an equally ancient OWC Thunderbay 4 cage and after a quick ‘zpool import catalyst’ I’m off to the races.

Moving my sanoid/syncoid config to proxmox was also a walk in the park. I followed Jim Salter’s steps on his GitHub for installing his sanoid project on Debian and it was flawless. The man is a genius. I already had all my sanoid configs on my desktop workstation so I just copied the files to the respective locations on the proxmox host and updated the syncoid script with the new local ip addresses. The only thing that tripped me up was having to allow passwordless sudo for zfs commands on the Debian host, which I should have known. I had my eyes on the finish line I guess. Hourly snapshots started piling up as well, so (I thought it was already there) I needed to add —delete-target-snapshots to the script. I also have a second ZFS replication target on a MacOS M4 Mac Mini that uploads data offsite using a cloud service that sanoid/syncoid manages the replication of as well, this didn’t need any changes besides a new SSH key to authorized_keys.

I know backups should have the target pull rather than the host push but I opted to have a single location for sanoid/syncoid to do its job instead of setting it up on both targets and the proxmox host felt like the best place for it. I might switch in the future, but for now since I have snapshots I worry less about the push/pull being an issue. I am but a humble homelabber, if my host ends up getting compromised I will have bigger problems. I am also using the proxmox host to serve the NFS share mounted on my arch desktop. I normally try to limit host jobs, but figured for these two vital services it would be okay. Plus, setting up a whole container for NFS when it only involves a single package and config file ‘sudo zfs set sharenfs’ command felt aggressive. Though if anyone has a different suggestion for what’s typically best practice please let me know!

I have been quietly stressing about this setup for a while. I knew this layout and having two pools in production like this was not best practice. I now feel much better having my main backup pool on a dedicated server rather than using the ‘backup’ to serve homelab content like I was previously. So now instead of having two pools in production I have one pool being used properly across both my homelab services ad attached via NFS to my workstation desktop, plus two local backups that are only active upon replication, and one cloud backup. A now even more truly proper and automated 3-2-1 backup.