The startup's Postgres survival guide

The startup's Postgres survival guide

A guide to preventing Postgres from toppling over.

Over the past half year or so, I’ve been writing an internal doc for our engineers trying to distill two years of Postgres battles into a somewhat cohesive document. While I love the Postgres manual, I find it’s hard to turn to when shit hits the fan because it’s just so darn comprehensive. I thought this might be useful for others and would appreciate feedback (or other tidbits that you’ve learned running Postgres in production).

Before starting Hatchet, while I was familiar with SQL, the extent of my knowledge was basically: if a query is slow, you need an index. That’s the starting point for this doc; I’m going to assume you’re familiar with SQL basics, rows, tables, and know roughly what an index is.

And if Claude is writing all of your queries, this might be a waste of time! I recommend supabase/agent-skills

A quick note on ORMs

This guide should still be useful, but you might need to translate some of these tips into your ORM of choice. Lots of optimizations as you scale just aren’t possible with ORMs unless you can break past the abstraction layer and write SQL. You can do this gracefully or non-gracefully; Prisma TypedSQL or equivalents look interesting for this. We use sqlc at Hatchet which gets us very similar behavior; highly recommend if you’re a Go stack.

Table of contents

• The simple stuff: good reads, writes and schemas
• Intermediate: the query planner, bulk updates, and autovacuum
• Some advanced stuff

The simple stuff: good reads, writes and schemas

Let’s start with the basics: queries and schemas at low volume.

Writing a good schema

After you’re deployed, schemas are by far the hardest to change moving forward, so it’s worth spending some time on them. I’d recommend building your schema iteratively: start with a rough approximation for your tables and primary keys, then write some queries on those tables based on your application needs. You can approximate this with some questions: Is this a high-read and/or high-write table? What are the most common filters on reads? Which columns am I updating the most?

If you want to be more formal about it, you can look into database normalization into 1NF/2NF/3NF, but I’ve found normal forms to sometimes be at odds with query efficiency and ease of use, which is critical when you’re moving fast—sometimes it’s just easier to dump data into a jsonb column.

My rules of thumb for schemas are:

• Use identity columns (auto-incrementing integers, slightly more performant than bigserial) or built-in UUIDs for primary keys
• Always use timestamptz
• Always use primary keys
• Use foreign keys with cascading deletes for low-volume tables, particularly where database consistency and correctness are important. Careful at higher volume.

Writing good read queries

Let’s start with SELECT queries. A useful—albeit slightly inaccurate—mental model for fast selects is: under the hood, Postgres is either going to find a single row in a table very quickly, or it’s going to read every single row in your table using something called a sequential scan 😞.

It’s going to find a single row very quickly when you filter by:

• An explicit index
• A unique constraint (just a special case of index)
• A primary key (these are automatically indexed in Postgres)

Indexes by default use a btree implementation. It’s most helpful to think of indexes as just another table in Postgres, with data stored in a specific format which is optimized for lookups (more on this later). These trees are great because finding a single row happens in approximately log(n) time, where n is the number of rows in the table—in other words, really fast.

When Postgres can’t use an index, it’ll use something called a sequential scan, or seq scan. Seq scans are much slower than index lookups, but modern databases are so fast at loading rows into memory that you probably won’t even notice at first: seq scans on tables with less than 20k rows are pretty much instant.

Writing performant joins

For inner joins, there’s rarely an argument for not using primary keys as the inner join; it usually speaks to a schema design or normalization problem. Treat ON clauses with the same respect as a WHERE clause—the same principles apply. Use an index.

Compound indexes and aligning ORDER BY to your indexes

Often the first slow query in your application will be a list query across a large table.

In this case, you can use a compound index.

In more complex cases, a good rule of thumb is: the ORDER BY columns should be the last columns in the index, and you should align columns to the ordering in the ORDER BY. Note that Postgres can scan btrees in both directions, so sometimes the DESC is irrelevant—but for compound indexes it’s good practice.

Writing good write queries

The premise of successful writes is:

Keep transactions short. Don’t go querying an external service in the middle of a transaction unless you have a really good reason to. Be careful of the rows you’re locking for writing; in other words, only lock what you need. Every time you update a row, you’re taking out a lock on that row for a short period of time until the transaction commits.

As your system gets busier, you’re going to start noticing the impact of locks more. In particular, you might try to create an index at some point in the future with a simple CREATE INDEX command: turns out this locks your table and prevents inserts and updates! When creating an index on an existing large table, always use CREATE INDEX CONCURRENTLY.

Migrations

Getting really good at writing migrations is an important technical advantage: it helps you iterate much faster and increases your uptime. As a starting point, try to keep migrations additive (in other words, don’t delete or remove columns) and run them in a transaction wherever possible; this will make rollbacks and partial migrations much easier to deal with. As you get more advanced, you can start looking into expand and contract migrations.

The simplest mental model for good migrations is: does this block all of my writes, or does it not? Creating an index without CONCURRENTLY blocks all your writes, so you might see downtime. Generally, operations which call ALTER TABLE should be worth a second look; for example, adding a new check constraint to a very large table can block your writes as well (unless you add it with the NOT VALID keyword).

Connection management

Every time you execute a transaction or query against your database, you’re utilizing a connection. Connections are expensive in a number of dimensions (cpu and memory), and high connection churn can lead to a lot of unnecessary resource waste, so connections should be long-lived. Connection storms (when you start using up a ton of new connections at the same time) can also lead to very hard to debug edge cases related to internal Postgres locks.

Because of all these connection footguns, external connection poolers like pgbouncer are great! If you can’t add this for whatever reason, in-memory connection poolers are a great second option. For example, because Hatchet is open-source, we don’t assume that all user databases use connection poolers, so we use pgxpool (an in-memory connection pool for Go) for this purpose.

Intermediate: the query planner, bulk updates, and autovacuum

Introducing the leakiest of abstractions, the query planner

At a certain point, your queries might become complex enough that a simple index won’t cut it (and you shouldn’t endlessly add indexes to your tables—they come with overhead). The queries might involve many JOIN statements or different types of joins where the correct path for querying the data isn’t clear.

At this point, you will need to concern yourself with the query planner.