Running Out of Disk Space in Production

Last night I put up a simple server which allowed customers to download the digital Kanjideck files. This server is hosted on a small Hetzner machine running NixOS, at 4GB of RAM and 40GB of disk space. One of these downloadable files weights 2.2GB.

The matter at hand boils down to a simple Haskell program which serves static files (with some extra steps regarding authorization) plus an nginx reverse proxy which proxies requests to a certain “virtual host” to the Haskell program.

1 First, Panic

Not even minutes after I announced that the files were finally available, hundreds of customers visited my server all at once. As the logs started flying off of my screen with all the accesses, I started noticing a particularly interesting message, repeated over and over again:

Mar 31 20:43:03 mogbit kanjideck-fulfillment[2528300]: user error
(Unexpected reply to: MAIL "<...> at kanjideck.com", Expected reply code: 250, Got this instead: 452 "4.3.1 Insufficient system storage")

Oh no. No one’s able to access their files and I’m already receiving emails about it. I messaged the users explaining the server was having some issues that I was resolving.

Grafana shows 40GB/40GB disk space used up, so does df -h have 100% usage of /dev/sda. I have to clear up space fast. I’m afraid at this point that I’m not even receiving the user complaints anymore since my mail could be getting dropped by lack of space.

I rushed to run du -sh on everything I could. The two larger culprits were /var/lib’s Plausible Analytics, with a 8.5GB database, and the /nix/store with the full server configuration at 15GB.

Clearing the nix store with nix-collect-garbage -d didn’t work because there’s no space left on device to even open a lock file.

OK, let’s kill the logs first: journalctl --vacuum-time=1s. This restored enough space for me to clear the nix store.

2 Mounting the Nix Store on a separate Volume

When in panic, buy more space. I bought more space as a separate Volume. The /nix/store is an immutable store and a perfect candidate for a separate drive. Following the instructions on “Moving the store” in the NixOS Wiki worked flawlessly. After rebooting, the root drive finally had enough space to reply to the users.

3 Finding the root cause in Nginx

This morning I found that the 2.2GB file download was stopping halfway through. With some investigation I found proxy_max_temp_file_size, which defaults to 1024m. Increasing the value allowed the file to be served, but the disk space issue spiked again!

The lsof +L1 command finds unlinked open files. I was greeted by 14.5 GB of deleted files held by nginx!

Nginx was buffering the 2.2GB file from my program to temporary files. When buffering is enabled, a part of the response can be saved to a temporary file. The zero value disables buffering of responses to temporary files, which was the ultimate fix needed.