Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3

turbolite is a SQLite VFS in Rust that serves point lookups and joins directly from S3 with sub-250ms cold latency.

It also offers page-level compression (zstd) and encryption (AES-256) for efficiency and security at rests, which can be used separately from S3.

turbolite is experimental. It is new and contains bugs. It may corrupt your data. Please be careful.

Object storage is getting fast. S3 Express One Zone delivers single-digit millisecond GETs and Tigris is also extremely fast. The gap between local disk and cloud storage is shrinking, and turbolite exploits that.

The design and name are inspired by turbopuffer's approach of ruthlessly architecting around cloud storage constraints. The project's initial goal was to beat Neon's 500ms+ cold starts. Goal achieved.

If you have one database per server, use a volume. turbolite explores how to have hundreds or thousands of databases (one per tenant, one per workspace, one per device), don't want a volume for each one, and you're okay with a single write source.

turbolite ships as a Rust library, a SQLite loadable extension (.so / .dylib), and language packages for Python and Node.js, plus Github deps for Go. Any S3-compatible storage works (AWS S3, Tigris, R2, MinIO, etc.). It's a standard SQLite VFS operating at the page level, so most SQLite features should work: FTS, R-tree, JSON, WAL mode, etc.

| Query | Type | Cold (S3 Express) | Cold (Tigris) | |---|---|---|---| | Post + user | point lookup + join | 77ms | 259ms | | Profile | multi-table join (5 JOINs) | 188ms | 681ms | | Who-liked | index search + join | 118ms | 384ms | | Mutual friends | multi-search join | 77ms | 201ms | | Indexed filter | covered index scan | 75ms | 159ms | | Full scan + filter | full table scan | 591ms | 921ms |

Benchmarks are organized by cache level:

• none: nothing cached, everything fetched from S3.
• interior: interior B-tree pages cached, index + data pages fetched.
• index: interior + index pages cached, data pages only fetched.
• data: everything cached, equivalent to local SQLite.

turbolite is designed for S3's constraints: minimizing request count, maximizing bandwidth utilization, and handling immutable objects. It uses large 64KB pages, seekable compression, and a manifest file to indirect page lookups. It also features proactive prefetching by intercepting the SQLite query plan to fetch necessary data in parallel before execution.