2022-04-02

Home Server Tinkering

Weeks ago I  purchased a secondhand machine. Since then I have been tinkering this little box.

The Perfect Media Server site is a good place to start with. Arch Linux Wiki is my go-to learnning resource, even though I use Ubuntu.

Filesystem

I'd be super paranoid and careful, as this is my first time manually configuring a disk array. Basically my optoins include:

  • ZFS
  • btrfs
  • Snapraid (or even combnied ZFS/btrfs)
  • Unraid
My considerations include:
  • Data integrety, which is the most important.
  • Maintenance. I want everything easy to set up and maintain.
  • Popularity. There will be more doc/tutorial/discussions if the technology is more popular.
Eventually I decided to use ZFS with raidz2 on 4 disks. 

I also took this chance to learn configuring disk encryption. I decided to use LUKS beneath ZFS. I could have just used ZFS's built-in encryption, but I thought LUKS is fun to learn. It really was. The commands are way more user-friendly that I had expected.

Hardening SSH

Most popular best practices include:
  • Use a non-guessable port.
  • Use public key authenticatoin and disable password authentication.
  • Optionally use an OTP (e.g. Google Authenticator) authentication.
  • Set up chroot and command restrictions if applicable. E.g. for backup users.

Various Routines

  • Set up remote disk decryption via SSH, with dropbear.
  • Set up mail/postfix, so I will receive all kinds of system errors/warnings. E.g. from cron.
  • Set up ZED. Schedule scrubbing with sanoid.
  • Set up samba and other services.
  • Set up backup routines.

Containers

I also took the chance to learn about Docker, and tried a couple of images. Not all of them are useful, but I found a few very useful:
  • Grafana + Prometheus. Monitoring system, UPS, air quality etc.
  • Photoprism. Managing personal photos
  • Pi-hole. Well I do have it running on my Pi, but I guess it's nice to have another option.
  • Hosting GUI software with web access. E.g. firefox. 
However there may be security concerns. See below.

Security Considerations

While I'd like to run userful software and services, I'd also want to keep my data safe. 

I want to protect my data from two scenarios:
  • Malicious/Untrusted code. I have heard so many news about malicious NPM packages in the last few years.
  • Human/Script errors. It happened with a popular package, where a whitepsace was unintended added into the install script, such that the command became "rm -rf / usr/lib/...". Horrible. For similar reason, I don't trust scripts that I wrote myself either.
At this moment I am not worried about DoS attacks.

User and File Permissions

The easiest option is to use different users for diffrent tasks. Avoid using root when possible. Also limit the resources that each user can access. 

This is a natural choice when I want other devices to back up data to my server. It is also useful when I need to run code in a "sandbox like environment". This is explained well in Gentoo Wiki.

There are two issues with this approach:
  1. It is not really a sandbox. It is straightfoward to prevent a user reading/writing some files, but it'd be trickier to limit other resource, like network, memory etc.
  2. It is tricky to maintain permissions for multiple users, especially when they need to access the same files with different scopes. ACL are better than the classic Linux permission bits, yet it can still become too complicated. I believe that complicated rules equal to security holes.

I created and applied different users for each docker image, but it was not enough. Discussed later below.


Mandatory Access Control (MAC)

Examples include AppArmor and SELinux.

Funny enough, years ago I thought AppArmor was quite annoying, because it kept showing popup messages. And now I proactively write AppArmor profiles.

I decided to write AppArmor profiles for docker images and all my scripts in crontab. I feel more assured knowing that my backup scripts cannot silently delete all my data.

Sandboxing

I had thought that chroot was a nice security tool, until I learned that it isn't. I found a couple of sandboxing options on Arch Wiki

However I don't see them fit well in my case. It should work for my scripts, but dedicated user + MAC sounds simpler to me. I also want to protect against malicious install scripts (e.g. NPM packages), and I feel that firejail/bubblewrap won't help too much here.

Sandboxing seems to be useful for beast software, like web browsers. However I'll also need to access it remotely, so I'd just go for containers or VMs.

I suppose I may find useful scenarios later.

Docker / Container / VM

I use Docker when
  • User + MAC is not enough.
  • I do not trust the code.
  • It is difficult to deploy to the host.
Security best practices include:
  • Do not run as root
  • Drop all unnecessary capabilities. (Most images don't need anything at all)
  • Set no-new-priviledges to true.
  • Apply AppArmor profiles.
[UPDATE: Obviously I did not see the whole story. Added a new section below]
I was quite surprised when I learned that root@container == root@host, unless I'm running rootless docker. What's worse, almost all docker images that I found use root by default.

While I managed to run most containers without root, many GUI-related container really want to use root. I really hate it and I started looking for rootless options.

Instead of mutiple GUI containers, I decided to run an entire OS. This will be my playground which has no access to my data. Docker is not designed for this task, although probably it can still do the job if configured correctly.

VMs (e.g. VirtualBox) are my last resorts. They are quite laggy on my box. It is also tricky to dynamically balance the load, e.g. I have to specify the max CPU/RAM beforehand.

I learned that Kata Container is good for this tast. It is fast and considered very secure. However I didn't find an easy way of depolying it. (Somehow I don't like Snap and disabled it on my machine. Well now snap is almost required by Kata Container, LXD and Firefox, maybe I should give it a go some time?)

Eventually I turned to LXC. It was quite easy to deploy a Ubuntu box. I am very happy with the toolchain and the design choices. For example the root filesystem of the container is exposed as a plain directory tree on the host, instead of (virtual disk) images.

[UPDATE] Containers without root@host

I really dislike it, that processes in containers can be run as root@host. Therefore I was looking for "rootless options". There are in fact two options:

  1. Container daemon run as root. Contaniers run as non-root. 
  2. Both container daemon and containers run as non-root.

#1 means to turn on user namespace mapping for containers in Docker or LXC. #2 means to further configure the daemon of Docker or LXC.

#2 seems more secure, but it requires another kernal feature CONFIG_USER_NS_UNPRIVILEGED, which might have security concerns. So funny enough, it is both "more secure" and "less secure" than #1.

While I don't know anything deeper, I'm slighly leaning towards #1. I will keep an eye on #2 and maybe turn to it when the secury concerns are resolved.

What's Next

Probably I will try to improve the box to reload services/container on failure/reboot. Maybe systemd is enough, or maybe I need something like Kubernetes or Ansible. Or maybe I can live well without them.