S3 Files and the changing face of S3

S3 Files and the changing face of S3

Almost everyone at some point in their career has dealt with the deeply frustrating process of moving large amounts of data from one place to another, and if you haven’t, you probably just haven’t worked with large enough datasets yet. For Andy Warfield, one of those formative experiences was at UBC, working alongside genomics researchers who were producing extraordinary volumes of sequencing data but spending an absurd amount of their time on the mechanics of getting that data where it needed to be. Forever copying data back and forth, managing multiple inconsistent copies. It is a problem that has frustrated builders across every industry, from scientists in the lab to engineers training machine learning models, and it is exactly the type of problem that we should be solving for our customers.

In this post, Andy writes about the solution that his team came up with: S3 Files. The hard-won lessons, a few genuinely funny moments, and at least one ill-fated attempt to name a new data type. It is a fascinating read that I think you’ll enjoy.

–W

Part 1: The Changing Face of S3

First, some botany

It turns out that sunflowers are a lot more promiscuous than humans.

About a decade ago, just before joining Amazon, I had wrapped up my second startup and was back teaching at UBC. I wanted to explore something that I didn’t have a lot of research experience with and decided to learn about genomics, and in particular the intersection of computer systems and how biologists perform genomics research. I wound up spending time with Loren Rieseberg, a botany professor at UBC who studies sunflower DNA—analyzing genomes to understand how plants develop traits that let them thrive in challenging environments like drought or salty soils.

The botanists’ joke about promiscuity (the one that started this blog) was one reason why Loren’s lab was so fun to work with. Their explanation was that human DNA has about 3 billion base pairs, and any two humans are 99.9% identical at a genomic level—all of our DNA is remarkably similar. But sunflowers, being flowers, and not at all monogamous, have both larger genomes (about 3.6 billion base pairs) and way more variation (10 times more genetic variation between individuals).

One of my PhD grads at the time, JS Legare, decided to join me on this adventure and went on to do a postdoc in Loren’s lab, exploring how we might move these workloads to the cloud. Genomic analysis is an example of something that some researchers have called “burst parallel” computing. Analyzing DNA can be done with massive amounts of parallel computation, and when you do that it often runs for relatively short periods of time. This means that using local hardware in a lab can be a poor fit, because you often don’t have enough compute to run fast analysis when you need to, and the compute you do have sits idle when you aren’t doing active work. Our idea was to explore using S3 and serverless compute to run tens or hundreds of thousands of tasks in parallel so that researchers could run complex analysis very very quickly, and then scale down to zero when they were done.

The biologists worked in Linux with an analytics framework called GATK4—a genomic analysis toolkit with integration for Apache Spark. All of their data lived on a shared NFS filer. In bridging to the cloud, JS built a system he called “bunnies” (another promiscuity joke) to package analyses in containers and run them on S3, which was a real win for velocity, repeatability, and performance through parallelization. But a standout lesson was the friction at the storage boundary.

S3 was great for parallelism, cost, and durability, but every tool the genomics researchers used expected a local Linux filesystem. Researchers were forever copying data back and forth, managing multiple, sometimes inconsistent copies. This data friction—S3 on one side, a filesystem on the other, and a manual copy pipeline in between—is something I’ve seen over and over in the years since. In media and entertainment, in pretraining for machine learning, in silicon design, and in scientific computing. Different tools are written to access data in different ways and it sucks when the API that sits in front of our data becomes a source of friction that makes it harder to work with.

Agents amplify data friction

We are all aware, and I think still maybe even a little stunned, at the way that agentic tooling is changing software development today. Agents are pretty darned good at writing code, and they are getting better at it fast enough that we’re all spending a fair bit of time thinking about what it all even means (even Werner). One thing that does really seem true though is that agentic development has profoundly changed the cost of building applications. Cost in terms of dollars, in terms of time, and especially in terms of the skill associated with writing workable code. And it’s this last part that I’ve been finding the most exciting lately, because for about as long as we’ve had software, successful applications have always involved combining two often disjointed skillsets: On one hand skill in the domain of the application being written, like genomics, or finance, or design, and on the other hand skill in actually writing code. In a lot of ways, agents are illustrating just how prohibitively high the barrier to entry for writing software has always been, and are suddenly allowing apps to be written by a much larger set of people—people with deep skills in the domains of the applications being written, rather than in the mechanics of writing them.

As we find ourselves in this spot where applications are being written faster, more experimentally, more diversely than ever, the cycle time from idea to running code is compressing dramatically. As the cost of building applications collapses, and as each application we build can serve as a reference for the next one, it really feels like the code/data division is becoming more meaningful than it has ever been before. We are entering a time where applications will come and go, and as always, data outlives all of them. The role of effective storage systems has always been not just to safely store data, but also to help abstract and decouple it from individual applications. As the pace of application development accelerates, this property of storage has become more important than ever, because the easier data is to attach to and work with, the more that we can play, build, and explore new ways to benefit from it.

S3 as a steward for your data

Over the past few years, the S3 team has been really focused on this last point. We’ve been looking closely at situations where the way that data is accessed in S3 just isn’t simple enough—precisely like the example of biologists in Loren’s lab having to build scripts to copy data around so that it’s in the right place to use with their tooling—and we started looking more broadly at places where customers were finding that working with storage was distracting them from working with data. The first lesson that we had here was with structured data. S3 stores exabytes of parquet data and averages over 25 million requests per second to that format alone. A lot of this was either as plain parquet or structured as Hive tables. And it was clear that people wanted to do more with this data. Open table formats, notably Apache Iceberg, were emerging as functionally richer table abstractions allowing insertions and mutations, schema changes, and snapshots of tables. While Iceberg was clearly helping lift the level of abstraction for tabular data on S3, it also still carried a set of sharp edges because it was having to surface tables strictly over the object API.

As Iceberg started to grow in popularity, customers who adopted it at scale told us that managing security policy was difficult, that they didn’t want to have to manage table maintenance and compaction, and