How We Synchronized Editing for Rec Room's Multiplayer Scripting System

Circuits is Rec Room's technology for programming rooms: a multiplayer-synchronized realtime scripting system that lets players build interactive experiences together. Creating Circuits is one of the top honors of my career. I'd like to share how we solved core networking challenges, for anyone working on similar problems.

Before the Circuits we know today, we had "Circuits V1". Circuits V1 was built in a few weeks by the talented Jamie McDuffee, reusing our shapes synchronization technology. For shapes, the unit of synchronization was a game object, so thousands of scripting elements needed thousands of game objects. This caused memory and CPU concerns that placed a low ceiling on complexity.

The tradeoff was deliberate. We didn't know Circuits would land, so shipping fast and testing the hypothesis was more important than perfect architecture. Circuits V1 met the market with massive success and drove all Rec Room scripting for years. We learned that combining 3D modeling, basic scripting, and an excited community, naturally resulted in interesting 3D games. But after scaling the product to its ceiling, we needed a different strategy, a rewrite we called Circuits V2.

For Circuits V2, a key decision reduced complexity: if objects were the expensive part of synchronization, what if all of Circuits was one game object? Starting from that premise, we built the Circuits V2 synchronization model on top. We made two attempts at this: one we released for MVP; another built on top after discovering issues in retail.

I'll call the first attempt "In-Memory Database". We imagined entities in Circuits, like nodes and edges, as rows in a database. Every row subscribed to network events describing changes it experienced: "Add Node", "Edit Property", "Remove Node". This model caused issues around cross-object operations. What if I connected node N1 and N2 with edge E1, while someone else deleted N1? This touched three rows twice: add edge E1, update N1's edge list, update N2's edge list, delete N1, update N2's edge list, remove E1.

If you weren't careful with row updates you'd see an error. Simple operations could be corrected with null checks but with our peer-to-peer networking model synchronization complexity grew with number of players and operations. Rows could easily get out of sync as messages were sent to every player and arrived in conflicting orders.

We came to a solution inspired by the popular JavaScript library, Redux. Redux's core idea is that your entire application state lives in one object, and the only way to change it is to send an action through a pure function called a "reducer". We kept our Circuits database but reimagined it as one global state. Now, any change to state can be represented by the reducer function f(S1, A) = S2.

We take advantage of the fact that Circuits uses a single game object, and use single RPC method for all networking. We define actions using Google's protocol buffers:

message ActionData {
  oneof payload {
    AddNodePayloadData add_node = 1;
    RemoveNodePayloadData remove_node = 2;
    AddEdgePayloadData add_edge = 3;
    RemoveEdgePayloadData remove_edge = 4;
  }
}

With this setup, we can easily implement Reduce functions. To solve the peer-to-peer latency issue, we opted for a "serializable isolation level", colloquially referred to as the "action funnel". All actions are thrown into a funnel and drip out one-by-one in a consistent order.

These defects disappeared after moving to the action funnel in Circuits V2. Our rate of defects in this component is close to 1 defect every 2 years. Additionally, join-in-progress became trivial. We periodically send an InitializePayloadData action (a snapshot) to the session server. Joining players catch up by loading the latest snapshot and replaying the small queue of subsequent actions.