Nobody Got Fired for Uber's $8M Ledger Mistake?

Nobody Got Fired for Uber's $8 Million Ledger Mistake?

LedgerStore became a case study for DynamoDB, and system design publications keep praising it to this day. But the design was abandoned 3 years after it went live.

Uber has rewritten its ledger systems five times in the last ten years. And at least one of those rewrites, if not all, could have been avoided.

That’s because the root of each generation of money software at Uber was driven from bad incentives. Each started with a brand new proposal, approved as the definitive solution; in time, a fatal flaw was surfaced; and finally, a new proposal came along to replace it.

Every rewrite was someone’s promotion project.

At least one of them could’ve been avoided: the one where Uber moved to DynamoDB. In 2017, Uber launched their new payment platform on it, and the critical factor that everyone involved seemed to miss was that DynamoDB is a consumption-priced database.

You pay for every read, and every write.

With each trip generating multiple ledger entries, and Uber as a whole processing 15 million trips per day, it didn’t matter that DynamoDB was great because of high throughput at global scale. The proverbial bean counter should’ve stopped this madness from happening.

Within 2 years, the cost became prohibitely expensive:

At Uber’s scale, DynamoDB became expensive. Hence, we started keeping only 12 weeks of data (i.e., hot data) in DynamoDB and started using Uber’s blobstore, TerraBlob, for older data (i.e., cold data). TerraBlob is similar to AWS S3. For a long-term solution, we wanted to use LSG.

A redesign that gets replaced 2 years later is a catastrophe.

And yet, history remembers Uber’s ledger on top of DynamoDB as a masterpiece. As late as 2024, ByteByteGo has an article praising it.

And that’s what concerns me. Uber’s design was a failure, but nobody seems to remember it that way.

That ends today.

I’m Alvaro Duran, and this is The Payments Engineer Playbook, the only newsletter on Earth tailor-made for engineers of money software. Every week, more than 2,000 subscribers from companies like Stripe, Coinbase and Modern Treasury get a dive deep on how to build software that moves money around. Not to pass interviews, but to do their job exceptionally well.

When money is on the line, stakes are sky high and the margin for error is razor thin.

In The Payments Engineer Playbook, we investigate the technology that transfers money. All to help you become a smarter, more skillful and more successful payments engineer. And we do that by cutting off one sliver of it and extracting insights from it.

Here’s what you can expect in today’s article:

Why DynamoDB works for payments but breaks when you use it as a ledger

The napkin math that would have saved Uber millions of dollars

And one shocking conclusion from all of this

Enough intro, let’s dive in.

The Consistency Trap

But first: is DynamoDB a bad choice for financial software?

Not necessarily. I’ve already covered DynamoDB as a potential data store for payment systems, and it has many features that are worth it: zero-downtime migrations, low latency for a global audience, and built-in replication and failover.

If you’re accepting payments at scale around the globe, DynamoDB is a great choice.

It is because DynamoDB, while not enforcing full linearizability across partitions, can guarantee consistency within a Region. DynamoDB gives you strong consistency on a per-partition basis, but not across partitions — and for a global-scale payments system, that’s a trade-off worth making.

PostgreSQL can; DynamoDB cannot.

This is quite a property when it comes to payments, because they are independent from each other. You can interleave the authentication of one with the capture of another. There’s no need to maintain full linearizability across your data; causal consistency is enough. DynamoDB trades off the linearizability that you don’t actually need for all those nice features I mentioned earlier, which means that for large enterprises that serve customers all over the world in high volume and frequency, DynamoDB is better than PostgreSQL.

But a ledger isn’t a payments system.

A ledger cannot simply say “hey, this account and that account can be dealt with independently”. The scope of a ledger system is The World; a data store that can’t enforce full linearizability isn’t going to cut it, no matter how good at throughput and latency it is.

In other words: DynamoDB works well in payments because payments can give up global consistency for better availability. But ledgers can’t give up global consistency, even if that means they get worse availability.

The Napkin Math (Because Someone Has To Do It At Some Point)

DynamoDB capacity pricing is based on two main models: Provisioned and On-demand. You can buy reads and writes in bulk, buy reads and writes when you need them, or both.

And then, there’s the storage and add-on features. For most, throughput is the real deal, and storage is just the cream on top of the invoice. But for data-heavy applications, such as ledgers, storage can become the dominant cost. That’s why reserving capacity is important, but it demands from you some predictive abilities, or at least making an educated guess about how much you’re going to need in advance, because you’re going to get a discount of more than 50 percent if you do it right.

If you use DynamoDB at scale, you need to do some napkin math.

In 2017, Uber was doing around 11 million trips a day. Assuming 10 entries per trip and 5 WCUs per entry1, that’s 550 million writes per day, and at $1.25 per million writes, that’s $687 per day.

$687 per day doesn’t sound like a lot. But that’s $250K a year, just in writes.

With 3x annual growth, the math is unsustainable: by year 3, we’re talking $2.25 million a year. I don’t have visibility into reads, indexes and global tables, but at Uber’s scale, the read side likely costs as much as the write side.

Which means that Uber was burning 5 million dollars in a freaking ledger.

Based on Uber’s data, by 2020 they had accumulated 1.2 petabytes of data. That, at $0.25 per gigabyte is 300K per month. And assuming the same 3x annual growth from 2017 to 2020 with a final size of 1.2 petabytes, that’s a cumulative cost of $3.5 million.

No wonder they switched to storing only the last 12 weeks of data in Dynamo, and stored the rest on premises.

Add the writes to the storage, and you’re looking at an 8 million dollar bill for a ledger that didn’t need to exist on DynamoDB in the first place.

Somebody Should Have Been Fired For This

What do you do with an 8 million dollar bill? You turn it into a case study.

Since 2020, Uber has migrated away from DynamoDB into their own internal ledger, called LSG (Ledger Store...Gateway?2), built on top of their own internal distributed database called DocStore:

Docstore is a general-purpose multi-model database that provides a strict serializability consistency model on a partition level and can scale horizontally to serve high volume workloads. Features such as Transaction, Materialized View, Associations, and Change Data Capture combined with modeling flexibility and rich query support, significantly improve developer productivity, and reduce the time to market for new applications at Uber.

Why not use an open-source alternative? Because Uber builds in-house. That’s what Uber does.

You could argue that DocStore provided the features they needed in a way no other alternative could. But you would be wrong!

Our homegrown Docstore was a perfect match for our database requirements, except for Change Data Capture (CDC) a.k.a., streaming functionality. [...] We decided to build a streaming framework for Docstore (project name “Flux”) and used that for LedgerStore’s Manifest generation.

So let me get this straight: DynamoDB was a bad choice because it was expensive, which is something you could have figured out in advance.