Do you even need a database?

A database is just files. SQLite is a single file on disk. PostgreSQL is a directory of files with a process sitting in front of them. Every database you have ever used reads and writes to the filesystem, exactly like your code does when it calls open().

So the question is not whether to use files. You're always using files. The question is whether to use a database's files or your own. And for a lot of applications, especially early-stage ones, the answer might be: your own.

Now, obviously we love databases. We're building DB Pro, a database client for Mac, Windows, and Linux. But the honest answer to "do you need one?" depends on your scale, and most applications are smaller than people assume. We tested this. We built the same HTTP server in Go, Bun, and Rust, using two storage strategies, and hammered them with wrk. Here's what the numbers look like.

The setup

Three flat files: users.jsonl, products.jsonl, orders.jsonl. The format is newline-delimited JSON (JSONL): one record per line, appended on write. Each file holds one entity type.

Two HTTP endpoints: POST /users to create, GET /users/:id to fetch by ID. We benchmarked the GET path. Reads are where the strategies diverge.

Approach 1: Read the file every time

The simplest thing you can do: when a request comes in for user abc-123, open the file, scan every line, parse each one as JSON, check the ID. Return when you find a match.

This is O(n). Every request reads the entire file from top to bottom, on average scanning half of it before finding the target. The larger the file, the slower every request gets.

Approach 2: Load it into memory

On startup, read the entire file once and store every record in a hash map keyed by ID. Writes go to both the map and the file. Reads are a single map lookup.

The file is the durable backing store. The map is the index. If the process restarts, reload from the file.

Read path is now O(1) at any scale. The sync.RWMutex in Go and RwLock in Rust let multiple readers proceed in parallel, so concurrent requests don't block each other.

Approach 3: Binary search on disk

What if you need reads that don't load everything into RAM, but also don't scan the whole file? The middle ground: sort the data file by ID, build a fixed-width index alongside it, and binary search the index using ReadAt. Each lookup does O(log n) seeks (about 20 for 1M records), then reads exactly one record from the data file.

The index format is simple: one line per record, exactly 58 bytes: <36-char UUID>:<20-digit byte offset in data file>\n. Fixed width means you can jump to any entry with a single ReadAt(buf, entryIndex * 58).

The benchmark

We seeded three datasets (10k, 100k, and 1M records) and used wrk to run 10 seconds of load against each server: 4 threads, 50 concurrent connections, random GET requests picking from a sampled list of real IDs.

All servers ran on the same machine (Apple M1 Mac mini, macOS 15). Go 1.26, Bun 1.3, Rust 1.94 (release build).

The results

Requests per second (higher is better)

| Strategy | 10k records | 100k records | 1M records | |---|---|---|---| | Go: linear scan | 783 | 85 | 23 | | Go: binary search (disk) | 45,742 | 41,661 | 38,866 | | SQLite (Go) | 26,000 | 25,507 | 25,085 | | Go: in-memory map | 97,040 | 98,277 | 97,829 | | Bun: in-memory map | 106,064 | 107,058 | 105,367 | | Rust: in-memory map | 163,687 | 155,364 | 169,106 |

Average latency per request (lower is better)

| Strategy | 10k records | 100k records | 1M records | |---|---|---|---| | Go: linear scan | 84ms | 552ms | 1,010ms | | Go: binary search (disk) | 1.2ms | 1.4ms | 1.4ms | | SQLite (Go) | 2.0ms | 2.0ms | 2.1ms | | Go: in-memory map | 497µs | 571µs | 584µs | | Rust: in-memory map | 231µs | 482µs | 221µs |

Key Takeaways

Binary search beats SQLite. Plain sorted files with a hand-rolled index outperform SQLite's B-tree by about 1.7x. SQLite's overhead is worth it for features, but for pure ID lookups, you're paying for machinery you don't use.

In-memory map is the ceiling. 97k+ req/s with sub-millisecond latency. If your dataset fits in RAM, nothing on disk will match it.

Pick by use case:

• Absolute fastest: Rust in-memory map.
• Fast without RAM bloat: Binary search on disk.
• Need SQL later: SQLite.
• Quickest to ship: Linear scan (for very small data).