The unlikely story of Teardown Multiplayer

Multiplayer has been the single most requested feature for Teardown ever since before its initial release. Synchronizing physics over the network is already known to be hard, and on top of that we have a completely dynamic, destructible world with full modding support. For a long time, we considered the whole idea unrealistic.

Despite the scepticism, we did an internal experiment back in 2021, using a naive approach to synchronize moving objects and send altered voxel data as objects were destroyed. It used enormous amounts of bandwidth and completely choked the connection when large objects were destroyed. It was purely a learning project and never reached a usable state, but it taught us where the bottlenecks were.

Around the same time, a community project called TDMP added rudimentary multiplayer support through reverse engineering and DLL injection. Despite being a bit janky, it completely blew my mind. It was an incredible technical achievement by the people involved. The mod mostly synchronized player position and player input, and since the engine isnât deterministic, it could easily get out of sync, especially with destruction.

A semi-deterministic approach

As we started bringing more people on board, we did a more serious investigation into a proper multiplayer implementation in late 2022. We knew we wanted perfect world sync. Anything else would quickly make simulations diverge in the chaotic world of Teardown. Sending large amounts of voxel data wasnât an option because of bandwidth, so we had to rely on determinism. Early on, I dismissed the idea of full determinism for the entire engine (a view I have since reevaluated), so it needed to be a hybrid approach: destruction done deterministically, while most other things use state synchronization.

For the longest time (and for good reasons), floating point operations were considered unsafe for deterministic purposes. That is still true to some extent, but the picture is more nuanced than that. I have since learned a lot about floating point determinism, and these days I know it is mostly safe if you know how to navigate around the pitfalls. I wonât cover them all here, but I hope to do that in another post, because thereâs a lot of confusion around this topic.

At the time, I decided to rewrite the destruction logic in fixed-point integer math, which is fairly straightforward given that weâre dealing with discrete voxel volumes. But thereâs much more to destruction logic than cutting out voxels on a regular grid. Object hierarchies may separate, new objects can be created and joints can be affected or reattached. A lot of this still involves floating point math, so each breakage event is split into a stream of deterministic commands that are replicated on all clients: âcut hole in this shape at voxel coord x,y,zâ, âchange ownership of that shapeâ, âreconnect joint to this shapeâ, etc.

Our implementation does not use dedicated servers. The player hosting a game also acts as server for that session, so all mentions about the server below is really just the player who hosts the session.

Reliable and unreliable

As long as the deterministic commands are applied to the world in exactly the same way, in exacly the same order, the resulting changes will be identical across all machines. The bandwidth requirements are small because commands are the same regardless of object size. Anything that modifies the scene content, such as spawning new objects or recoloring objects, is implemented using the same approach. We put all these commands on a reliable network stream, where everything is guaranteed to arrive in order and nothing is missed, just like a traditional data stream.

For anything that doesnât affect the structure or contents of the scene, such as object transforms, velocities, and player positions, we use state synchronization with eventual consistency. For every update, the server selects a number of objects that should be synchronized and sends their state to the clients. The server keeps a priority queue to ensure everything is eventually sent, prioritizing objects visible to the player while staying within the allowed data budget (in Teardown, around one Mbit per client). Because nearby objects differ per client depending on player position, the server has to maintain this queue and make these decisions per client.

These packets are sent unreliable, meaning they are not guaranteed to arrive, and those that do arrive are not guaranteed to arrive in order. This is what the messy reality of internet packets looks like. Protocols like TCP are layers on top that, responsible for maintain ordering and resend data that never arrived. With unreliable state synchronization, you have to handle some of that yourself, but in many situations you donât need strict ordering. If a packet gets lost, a newer packet with more recent state will arrive soon anyway. Many other games use a similar approach, so this aspect of our implementation is fairly traditional.

Each client runs a local simulation, as it normally would, but once new state arrives from the server the affected objects are corrected to keep everything in sync. In many cases, locally simulated objects are nearly identical to what comes from the server and the correction is invisible. But in complex situations with many simulated objects, the priority queue has to work harder, so that more objects will get corrected at a lower frequency, which can cause visible snapping.

Scripting

We knew from the start that the multiplayer version had to support scripting and modding, but scripts now had to be aware of the new architecture, where scene changes happen on the server and are automatically distributed to clients. Some script parts still have to run on clients, especially UI and overlay graphics. We ran experiments to automate this by running the exact same script on both server and clients, while ignoring certain API calls where they werenât relevant. That turned out rather clunky, and a bad fit for certain use cases, so it was eventually dropped.

We also didnât want to split everything into separate server and client scripts, so we landed somewhere in the middle: client and server parts exist in the same script, with some machinery in place (shared state table and remote calls) to simplify communication between them. Itâs a pretty unusual multiplayer scripting approach, but it has served us well and the modding community seems to get the concept. Thereâs a tutorial available here with more details.

Terminal and UI

The in-game terminals were tricky to get right, because they donât follow the conventional flow. Terminals are part of the scene, not just a UI layer on top, and can be controlled by any player, yet the interaction should be visible for everyone and the resulting actions should happen on the server.

We solved this by running terminal scripts entirely on the server, recording draw commands which are transfered to the clients using delta-compression. Hence, we stream the resulting terminal image to the clients, but not in the form of compressed video. Instead we stream the UI draw commands that build up the image, or to be more specific: the draw command delta from the previous frame to the next. The idea is similar to what the X Window System uses, allowing graphical user interfaces for remote applications on thin clients. The draw commands are often similar frame-to-frame even while animating, so the delta is usually tiny.

Once the system was in place, we started using it on the main menu as well, so that everyone can see what the host is doing when selecting level, game modes and mods for a session.

The big merge

Our initial idea was that multiplayer Teardown should be a separate game. So while implementing the first version, we cleaned up the codebase and made reasonable adaptations to suit the new architecture. Meanwhile, our parent company at the time, Saber Interactive, was hard at work on consol