Branimir Lambov from IBM on Cassandra

Branimir Lambov from IBM on Cassandra

This article is part of a series interviewing developers (not founders, not executives) working on software infrastructure to understand their work, how they got here, the projects they’re proud of, the incidents they’ve learned from, and what they’re curious about.

Branimir (LinkedIn; GitHub) is a Cassandra committer, and member of the Project Management Committee (PMC), currently working at IBM (via its acquisition of DataStax). With a PhD (University of Aarhus, 2005) on exact computation of real numbers, and years spent working on digital signals processing, natural language processing, and the last 11 years as a Cassandra committer at IBM, Branimir has an interesting background. One big recent project of his, released in Cassandra 5 (2024), allows users to swap out the Skiplist in the Log-Structured Merge Tree (LSM Tree) for a Trie, improving memory usage and storage efficiency for anyone who opts in.

Starting from the basics... how did you end up working on Cassandra? Were you into databases before DataStax?#

Not at all. In fact, I remember considering them boring in university... Then a recruiter contacted me at the right time and I spoke to people who made me think of concrete problems; this got me interested, landed me in good company and a good company to work for, and I've kept doing it for over a decade now.

Supporting the Trie in the LSM Tree was a big recent project. Can you tell us about some other big projects you've worked on in Cassandra?#

There have been a few. The first one was the deterministic token allocation, where we wanted to find a way to reduce the number of virtual nodes by selecting them algorithmically rather than randomly. I spent a rather long period of time crafting a solution and had it committed. It took a couple of years until it was used in earnest, and turned out to not be that suitable for the way people actually used Cassandra. But it did include a smaller piece, written for tables that don't use replication, that worked great once it was applied for each individual rack, which ended up being both a simpler and better solution.

More recently, I worked with a team of people on modernizing Cassandra's compaction strategies, which resulted in Cassandra 5's Unified Compaction Strategy. We started by looking at the results from academia and compared them to our legacy compaction strategies to figure out how to build a more flexible solution that can cover them. After implementing this, we used it in DataStax's private branch to successfully handle densities an order of magnitude higher than what was typical before. We did not stop there, as the version that was ultimately merged in Cassandra 5 includes further improvements that reduce the need for manual configuration and make it even harder to topple.

As for the Trie project itself, how does a project like that happen end-to-end?#

This has been a long-running project for me, and is still ongoing as we speak. It started almost a decade ago, when I was a relatively fresh contributor to the project, and rather mundanely: we knew there was something to be gained if we took advantage of byte order, i.e. if we used keys that can be compared lexicographically like strings. We did not know what. Some of the more senior contributors at the time asked me to try it, as I had experience working with strings before in the context of natural languages.

This was not a small question, and it had two separate parts:

• How can we take advantage of byte order in some part of Cassandra to get a non-trivial performance gain?
• How do we introduce byte order to the data that's given to Cassandra?

I knew that the primary indexes of SSTables in Cassandra were not as efficient as they could be, so I started developing a trie-based replacement for these. To be able to make the solution generic, I also started working on ways of reinterpreting keys, so that they can become byte-ordered. The result was the trie-indexed bigtable format, i.e. the "BTI" format. Personally, I treat this format as a proof of concept of the usefulness of byte order and tries in the context of an LSM-Tree-based database like Cassandra. It brings a very noticeable improvement in query performance and improves the handling of wide partitions.

This project took about a year, and I was working on it entirely on my own until I had code to review, benchmarks to prove its efficiency as well as documentation. It was merged in DataStax's proprietary version of Cassandra In April 2017 and was delivered in DSE 6, where it has been in use as the only SSTable format for years. Five years later it was contributed to Apache Cassandra and was released (for the second time) in its version 5.

Once the concept was proven, I started working on a much bigger project with a more general approach that can be applied to all of Cassandra's primary storage and retrieval mechanisms, starting from the hardest part – memtables. The trie memtable in Cassandra 5 is the first result from this effort; like the BTI format, it landed first in the closed-source DSE 6.8, and was later contributed to Apache Cassandra 5. I have been working on this project on and off for several years already, mostly on my own with some help for short periods of time. This bigger project is now formalized as CEP-57, and its next stage will soon be up for review.

How has the adoption been since the Cassandra 5 release?#

I don't yet know, to be honest. It takes a couple of years for something new to be picked up by the community at large, and Cassandra's defaults are deliberately chosen to favour backwards compatibility over taking advantage of new features.

The BTI format and the trie memtables, however, have been in heavy use in all of DataStax database products. All customers of DSE 6 and later, Astra Classic and Serverless, as well as HCD, have relied on them for years.

What was your first computer? What got you into programming?#

For me it started with an after-school computer club in the late 80s. Like any kid I played a lot of games; at some point I saw an older student programming some simple animation which made me interested in also making stuff like that. I started reading any programming book I could find and spending even more time in that club experimenting.

The machines there were Apple II clones; I started with Basic, with some Assembler not too long after that. My first personal machine, an Intel 486-based PC, came much later after a friend and I had started writing software for a local business.

Have you ever caused an outage because of a feature someone used that you worked on?#

Even though Cassandra is difficult to break, you can cause more than enough trouble with the wrong kind of change. Last year, nearing the completion of a feature related to compaction parallelization, I added a piece of code to distribute compactions in groups that I treated as not very important. I added an assertion to let us know if the code isn't working correctly, and thought we'd quickly learn if there's something wrong with it as Cassandra is almost always run with assertions enabled.

And we did. A problem was identified and fixed, but meanwhile we had a release, and that release was used with the important assertion disabled. The result was data loss, i.e. some data was not found when requested in certain ways. Thankfully, Cassandra replicates the data and the processing, this bug wasn't deterministic enough to cause the exact same problem in all replicas, and we were able to find the data in others.

Takeaways? You can't assume any code is unimportant; you can't rely on checks that can be switched off; there are things even the best safety nets won't save you from; reducing redundancy even of secondary processes can be a risk not only to availability, but also to correctness. Things we already know, but tend to forget or push aside under pressure.