FreeBSD ate my RAM

Last month I posted about my journey migrating my site server from an old Ubuntu server to FreeBSD. Some people on Hacker News noticed that, when I showed the fastfetch

result, I said I was confused with the RAM usage compared to btop

and commented that fastfetch

is probably more correct. I decided to enter that rabbit hole and try to understand why reporting free or used memory in a modern operating system is more complicated than it seems.

Another user shared Linux ate my RAM, which provide a quick explanation for the same effect on Linux. And if you want a quick answer for FreeBSD too: the usage sometimes look off because the OS will cache everything it can from the disk into the RAM to improve overall performance, but that cache is volatile and will be freed in case it needs more memory. If you want a slightly longer answer, keep reading.

But just a quick disclaimer before: I am not an expert in operating systems internals, especially FreeBSD. This is a writeup of weeks of research in this field on my free time. If you find anything that’s particularly wrong, please comment it: sharing (knowledge) is caring!

RAM usage is hard to define

The whole point of Linux ate my RAM is explaining how unused RAM is wasted RAM. Just like the CPU cache will cache RAM contents because the CPU can access that quicker, the RAM will cache disk data to improve the user’s experience in the system. How that cache works is a bit more complicated, but before that, it’s important to understand how the kernel manages RAM.

Most modern operating systems have a Virtual Memory (VM) system. What it does is basically divide the physical memory into pages of (usually) 4KiB. Each page is then added to different queues, so that the kernel can juggle them around to make sure all the processes have their memory when they need and the whole system will keep working through moments of scarcity. For example: the swap memory. I never thought exactly how the Swap memory was used, except that it’s a space separate in disk that will store temporarily part of the RAM if needed. But in summary, when the OS sees allocated RAM that’s not being used too much, it will set it in a way that it can be stored in disk in case more memory is demanded. When those pages are requested again by the program that owns it, it will then get moved back into RAM.

Every OS has a different set of pages and rules for how to manage them. On FreeBSD, the types of page queues are:

#define PQ_NONE 255
#define PQ_INACTIVE 0
#define PQ_ACTIVE 1
#define PQ_LAUNDRY 2
#define PQ_UNSWAPPABLE 3
#define PQ_COUNT 4

You can find that at sys/vm/vm_page.h. All other unix-based systems will have something similar: Linux, OpenBSD, NetBSD, DragonFlyBSD.

If we check top

, we see that it doesn’t just report memory usage, but divides it into a few categories:

top reports each section of memory, swap and disk cache with a lot of details

active: active pages are pages that are actively being used by (mostly) userland processesinactive: pages that haven’t been accessed by those process in some time will be moved into inactivelaundry: this is the queue of pages to be written to swap. When the system needs to allocate space that is not in the free queue, it will move inactive pages to this queuewired: that’s memory inPQ_NONE

,PQ_UNSWAPPABLE

and memory that the kernel itself is using and is not managed by the VMfree: purely unused memory

When memory that was

inactive, went tolaundry, got written to disk (swap), is requested again by the process that owns it, it will then get retrieved from the disk intoinactiveand finally toactiveagain.

And now we can start to see why it’s not so easy to tell exactly how much memory is being used and how much is free. Memory in the free queue is guaranteed to be free, but we can argue that the one in the inactive queue is too, since it’s reclaimable, because the kernel will free that whenever more memory is demanded. Wired memory is mostly locked, however, that’s where disk cache goes, so part of what’s in wired is also reclaimable, making it “free” too!

Disk Cache

ZFS, the default FreeBSD filesystem nowadays, has ARC, Adaptive Replacement Cache, a specialized system that caches recently used data in memory, improving the repeated reading from disk. That cache shrinks as the system claims more memory. The kernel itself has mechanisms to do this cache, but ARC bypasses that. All the stats from that can be accessed via the kernel parameters kstat.zfs.misc.arcstats.*

. Using sysctl

, we can fetch it all:

sysctl kstat.zfs.misc.arcstats

This will show literally all the parameters available, but now just these are important:

sysctl -n kstat.zfs.misc.arcstats.size
sysctl -n kstat.zfs.misc.arcstats.c_min
sysctl -n kstat.zfs.misc.arcstats.c_max

These will show the current cache as well as the minimum and maximum configured, all in bytes. Using gnumfmt

we can convert to readable units:

$ sysctl -n kstat.zfs.misc.arcstats.size | gnumfmt --to=iec
3.1G

That’s also shown in top

, with even more details.

That’s for ZFS, but you can run other filesystems on FreeBSD.

Why fastfetch

and btop

report differently?

Now we get to the interesting part. Both of these tools, and many others such as htop

, try to report the memory usage so the user (or sysadmin) can have an idea of what’s going on with their systems. For that, they all have to pick a heuristic; effectively decide what they’ll call used memory. And the whole difference comes from the fact that they have different heuristics.

I digged into the source code of each tool to go after how they determine that. fastfetch

does this:

free memory = free + inactive + cache*
used memory = total - free memory

More on that

cache

later!

In my old ThinkPad X230, running FreeBSD 15.0-RELEASE, that looks like:

82% of used memory!

btop

, on the other hand, does:

available memory = total memory - active - wired
free memory = free
used memory = active + wired

Running it at the same time as fastfetch

was giving me this:

Only 7% used?!

And just to make things even more interesting, I checked htop

too, and while it reports the memory categories separately in those bars, it shows the used memory at the end of the bar:

4.49G/5.69G

Using this heuristic:

used memory = wired + active + laundry

Then I wrote a python script that would show me all the heuristics at once. You can find it here.

Pastedimage20260701215700.png

It looks correct, except for btop

, that’s way off. But if you’re looking close, you also noticed that the cache

value in the screenshot I shared earlier is also empty. It seriously took me weeks to realize that. So I started digging further into their code.

btop

memory reporting is pretty wrong on FreeBSD

On their source-code, looking specifically on src/freebsd/btop_collect.cpp, where it fetches the memory information:

int mib[4];
u_int memActive, memWire, cachedMem, freeMem;
size_t len;
len = 4; sysctlnametomib("vm.stats.vm.v_active_count", mib, &len);
len = sizeof(memActive);
sysctl(mib, 4, &(memActive), &len, nullptr, 0);
memActive *= Shared::pageSize;
len = 4; sysctlnametomib("vm.stats.vm.v_wire_count", mib, &len);
len = sizeof(memWire);
sysctl(mib, 4, &(memWire), &len, nullptr, 0);
memWire *= Shared::pageSize;
mem.stats.at("used") = memWire + memActive;
mem.stats.at("available") = Shared::totalMem - memActive - memWire;
len = sizeof(cachedMem);
len = 4; sysctlnametomib("vm.stats.vm.v_cache_count", mib, &len);
sysctl(mib, 4, &(cachedMem), &len, nullptr, 0);
cachedMem *= Shared::pageSize;
mem.stats.at("cached") = cachedMem;
len = sizeof(freeMem);
len = 4; sysctlnametomib("vm.stats.vm.v_free_count", mib, &len);
sysctl(mib, 4, &(freeMem), &len, nullptr, 0);
freeMem *= Shared::pageSize;
mem.stats.at("free") = freeMem;

It uses sysctl