How to Build a Minimal ZFS NAS Without Synology, QNAP, TrueNAS (2024)

How to Build a Minimal ZFS NAS without Synology, QNAP, TrueNAS

Published on:

If you need a basic NAS and don't care about GUI features, it is suprisingly simple to set up a ZFS dataset and share it over the network using Samba.

Scope:

| Scope & Requirements | | |---|---| | Raid Level | RAIDZ1 (1 Drive Redundancy) | | Operating System | Debian 12 Bookworm | | Encryption | None | | ZFS Implementation | OpenZFS, zfs-2.1.1 | | CPU | 4 Cores, Xeon Server CPU can be had for cheap | | RAM | ECC RDIMM RAM 16 GB | | Storage | 4x4TB NVMe SSD | | Backups | Not covered, use ZFS Backup Scheduler | | Skills | Basic familiarity with Linux | | Skill Level | Beginner/Easy |

I am using this article to document it for future myself, feel free to adopt it for your needs. Problem with TrueNAS is that it is a full-featured, supposedly enterprise-grade, software suite. While it may be simple to set it up (I've never tried), I just don't need any of the bells and whistles it offers. It's the mismatch between what I need and what it offers; not something inherently wrong with TrueNAS. There is also something to be said about a system you know everything about and not having to rely on yet another thing.

ZFS's best feature that's never explained or written anywhere

ZFS filesystem is self contained. If your OS is nuked suddently, simply take all disks to another machine or install a new OS, install zfs

, run zfs import

and get back your data. This freedom is underrated and not well understood. It is also not explained anywhere.

It's worth emphasizing: All configuration/details about ZFS is stored on the disks themselves. If you've setup a RAIDZ2 (Raid 6) with 6 disks, they are self contained. Move them to a new machine with zfs

tools installed, and simply run zfs import

. Boom, they'll show up as RAIDZ2. This is an amazing feature that no matter what happens to the host OS, machine, etc; as long as the disks are not damaged, your data is fine.

Step 1. Locate and Organize Disks

List all disks on a linux machine using lsblk -d -o TRAN,NAME,TYPE,MODEL,SERIAL,SIZE

command.

[root@sys ~]# lsblk -d -o TRAN,NAME,TYPE,MODEL,SERIAL,SIZE
TRAN NAME TYPE MODEL SERIAL SIZE
sda disk Virtual disk 40G
nvme nvme5n1 disk Samsung SSD 990 PRO 4TB XXXXXXXXXXXXXXX 3.6T
nvme nvme2n1 disk Samsung SSD 990 PRO 4TB XXXXXXXXXXXXXXX 3.6T
nvme nvme6n1 disk Samsung SSD 990 PRO 4TB XXXXXXXXXXXXXXX 3.6T
nvme nvme3n1 disk Samsung SSD 990 PRO 4TB XXXXXXXXXXXXXXX 3.6T

These are brand new NVMe drives from Samsung so they should be completely unallocated.

The disks are also mapped to an ID, running ls -lh /dev/disk/by-id

:

[root@sys ~]# ls -lh /dev/disk/by-id
total 0
lrwxrwxrwx. 1 root root 10 May 10 09:34 dm-name-rhel-root -> ../../dm-0
lrwxrwxrwx. 1 root root 10 May 10 09:34 dm-name-rhel-swap -> ../../dm-1
lrwxrwxrwx. 1 root root 10 May 10 09:34 lvm-pv-uuid-DGBpev-Na0C-tY20-YY6E-tpL3-epNA-8Ts3Y0 -> ../../sda3
lrwxrwxrwx. 1 root root 13 May 10 09:34 nvme-Samsung_SSD_990_PRO_4TB_XXXXXXXXXXXXXXX -> ../../nvme2n1
lrwxrwxrwx. 1 root root 13 May 10 09:34 nvme-Samsung_SSD_990_PRO_4TB_XXXXXXXXXXXXXXX -> ../../nvme5n1
lrwxrwxrwx. 1 root root 13 May 10 09:34 nvme-Samsung_SSD_990_PRO_4TB_XXXXXXXXXXXXXXX -> ../../nvme3n1
lrwxrwxrwx. 1 root root 13 May 10 09:34 nvme-Samsung_SSD_990_PRO_4TB_XXXXXXXXXXXXXXX -> ../../nvme6n1

Notice that they are symlinked to their disk names in /dev/

.

We can create a /etc/zfs/vdev_id.conf

that maps an alias to these IDs:

[root@sys ~]# vim /etc/zfs/vdev_id.conf
# Add these lines in the vdev_id.conf file
alias nvme0 /dev/disk/by-id/nvme-Samsung_SSD_990_PRO_4TB_XXXXXXXXXXXXXXX
alias nvme1 /dev/disk/by-id/nvme-Samsung_SSD_990_PRO_4TB_XXXXXXXXXXXXXXX
alias nvme2 /dev/disk/by-id/nvme-Samsung_SSD_990_PRO_4TB_XXXXXXXXXXXXXXX
alias nvme3 /dev/disk/by-id/nvme-Samsung_SSD_990_PRO_4TB_XXXXXXXXXXXXXXX

Run udevadm trigger

to set the alias (or you can reboot the machine). We can verify that the aliases have been mapped by running ls -lh /dev/disk/by-vdev

:

lrwxrwxrwx. 1 root root 13 May 10 10:28 nvme0 -> ../../nvme2n1
lrwxrwxrwx. 1 root root 13 May 10 10:28 nvme1 -> ../../nvme5n1
lrwxrwxrwx. 1 root root 13 May 10 10:28 nvme2 -> ../../nvme3n1
lrwxrwxrwx. 1 root root 13 May 10 10:28 nvme3 -> ../../nvme6n1

Alias mapping is completely optional, you can if you'd like use the full ID /dev/disk/by-id/nvme-eui.002538414143c248

of the disk when creating the zpool as we will do in the next section. Using an alias makes it nice. However, please don't use /dev/nvme1, /dev/nvme2, ...

as the order is not guaranteed, especially if you mount a new drive to the system. Creating a vdev_id.conf

ensures that the serial number of the drive is tied to the alias.

Remember when we discussed there is no configuration needed on the host OS? /etc/zfs/vdev_id/conf

is not necessary and only used when creating a zpool for convenience. If your OS gets nuked, and you lose vdev_id.conf

, it won't matter at all.

Step 2. Create ZPOOL

For this tutorial, I am creating a RAIDZ1 (RAID 5) zpool. That means 1 drive redundancy in-case of failure. It's up to you if you'd like additional redundancy, RAIDZ2 (RAID 6) would certainly be more risilient.

First, we need to install zfs on the linux machine. Please refer to the OpenZFS documentation on how to install it. It's usually as straight forward as, in my case, dnf install zfs

on RHEL 9.

I recommend setting the ashift=12

option when creating the zpool as this is your last chance to do so. Most disks report 512kB sector size to OS due to backwards compatibility reasons, but large disks such as Samsung 990 Pro has a sector size of 4KB or even 8KB. ashift=12

represents a sector size of 4KB which will substantially improve performance.

[root@sys ~]# ls /dev/disk/by-vdev
nvme0 nvme1 nvme2 nvme3
[root@sys ~]# zpool create -o ashift=12 s16z1 raidz1 nvme0 nvme1 nvme2 nvme3
[root@sys ~]# zpool status s16z1
pool: s16z1
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
s16z1 ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
nvme0 ONLINE 0 0 0
nvme1 ONLINE 0 0 0
nvme2 ONLINE 0 0 0
nvme3 ONLINE 0 0 0
errors: No known data errors
[root@sys ~]#

Perfect! I chose s16z1

as the name as it describes the size and type of raid. Most tutorials will use tank

as the name, as it relates to pool

. Corny, I reject this.

We're not done yet. zpool

is a disk abstraction, and zfs

is the file system. When we ran zpool create

, it created a zfs

file system with it.

List all properties of the zfs file system by running: zfs get all s16z1

. To check whether our zpool is properly configured with ashift=12

, we can run:

[root@sys ~]# zdb | grep ashift
ashift: 12

Before we share the system, configure compression (optionally) as well as the default mount point.

[root@sys ~]# zfs set mountpoint=/mnt/s16z1 s16z1
[root@sys ~]# zfs set compression=lz4 s16z1

Next, let's create a couple of zfs datasets under s16z1

root dataset. We will share them using Samba in the next section.

[root@sys ~]# zfs create s16z1/docs
[root@sys ~]# zfs create s16z1/backups

docs

for documents, backups

for time machine backup. You can create as many datasets as you'd like. Try to keep them at the top level. If you're wondering what is the difference between just a regular filesystem folder and a dataset—a zfs dataset is a way more than just a folder. You can manage zillion properties of a dataset, encrypt it, send and replicate a dataset, take snapshots, etc.—essentially, the entire ZFS feature set. Therefore, it is a good idea to create individual datasets for large categories of your files. docs

or backups

is a good abstraction level for a dataset. If you want to send just docs

to a another remote server as a backup, you can do that without sending the whole s16z1

root dataset.