CRDT benchmarks

How to read the numbers

The benchmark models one specific usage pattern: every operation is its own transaction, and every update is sent to the peer and applied immediately. That is what a collaborative text editor does with keystrokes. It rewards libraries with cheap transactions and cheap per-update application, and it penalises libraries that assume changes will be batched. Read the results with that pattern in mind; a library that is slow here may be fast in an application that batches.

N = 6000 is small. Most B1 to B3 documents are a few kilobytes. B4 (a 105,000 character document built by 260,000 edits) and C1 (200,000 edits per peer, or 30,000 peers) are the only tests at a realistic size, and they are where the libraries separate most.

The numbers are stable: the median coefficient of variation across the ten runs is about 5% for every library. Differences smaller than the confidence intervals of both cells are not differences. Sub-millisecond cells vary by 20% or more and should be read as "very fast", not compared.

json-joy

json-joy is the fastest library in 16 of the 23 timing rows. It replays the real-world trace in 119 ms (Yjs: 2.56 s), merges 1,540 concurrent map writes in 5.8 ms (Yjs: 47 ms), and completes both C1 tests in about two seconds, where the others take between 20 s and 13 minutes.

We checked whether the harness gives it an advantage. It does not. json-joy produces one binary patch per edit (259,778 patches, 4.9 MB, for B4), a peer that applies them one at a time ends with the correct text, and every concurrent test passed the harness's peer-equality assertions. The advantage holds at each layer we could separate on the B4 trace:

Its core is about ten times faster than Yjs's, and its patch encoding costs almost nothing (17 ms for 260,000 patches) where Yjs's costs more than the edits themselves.

What it gives up. json-joy keeps no operation log. It cannot reconstruct past versions or show history, which Loro and Automerge can. Its snapshot of a map keeps only the current value of each key, which is why its B3 map documents are a few dozen bytes. Against Yjs this is a like-for-like comparison, since Yjs has no operation log either; against Loro and Automerge, part of the gap is that trade-off.

Its measured weaknesses: inserting at the front of a text or list is slow (6,000 prepends take 481 ms against 33 ms for Ywasm and 135 ms for Yjs), its documents for arrays of numbers are three times larger than Yjs's (108 kB against 36 kB), and its update messages for random-position inserts are larger (80 kB against 33 kB in B2.2). We did not investigate the cause of the prepend cost.

What we did not check: its algorithmic behaviour on documents much larger than B4, and its implementation. It converged correctly in every test, but we did not read or profile the code.

Automerge

Automerge 3.4.1 is the slowest library in every timing row but one. Four separate causes account for it. We measured each in isolation.

Applying a remote change costs time proportional to the document, inside the WebAssembly core. We split the receiver's loadIncremental into the wasm call and the JavaScript step that rebuilds the document object afterwards. The wasm call grows linearly with the text: 286 ms per 1,500 calls on a small document, 1,727 ms per 1,500 calls once the text reaches 4,500 characters. The JavaScript step stays at 18 ms. The same 6,000 changes applied in one call take 30 ms in total. A stream of small updates therefore pays a document-sized cost on every update, which makes B1, B2, B3 and C1.2 quadratic. For large maps the JavaScript step grows as well, because it materialises the whole map after each patch.

Editing at a random position inside one change also costs time proportional to the text. A single change call containing many splices runs at 63 µs per splice on a 33,000-character text and 365 µs per splice at 260,000 characters. C1.1 performs 200,000 splices in one change per peer with no receiver involved, and takes 451 s. Yjs performs the same splices in about a microsecond each.

Every change is a commit. With one edit per change and no receiver, Automerge costs a constant 30 µs per change: transaction setup, the proxy, hashing the change, and encoding it. Yjs costs 3 to 9 µs for the same transaction. B4 makes 260,000 changes, so this alone is 8 s of its 13.5 s.

Loading rebuilds the state from history. Automerge stores the operation log and reconstructs the document on load, so the B4 document takes 652 ms to open. Loro's snapshot stores the materialised state and opens in 4.5 ms. The same choice gives Automerge the smallest text documents: 129 kB for B4 against 227 kB for Yjs.

The first two are almost certainly new in the 3.x series. The 2024 run of these benchmarks, on slower hardware, had Automerge 2.1.10 completing B1.1 in 343 ms; 3.4.1 takes 4.7 s. Automerge 3 moved to a compressed columnar in-memory representation to reduce memory use. Both costs are consistent with inserting into a compressed structure, but we did not confirm that in the source.

In practice: automerge-repo's sync protocol batches incoming changes, which removes most of the first cost. The second and third remain, and they are what a text editor doing one change per keystroke will experience.

Yjs and Ywasm

Ywasm is the Rust implementation of the Yjs algorithm, used from JavaScript through WebAssembly bindings. The two produce byte-identical documents. Their speed differs by the direction of the cost.

Inside a single transaction, Ywasm is three to four times faster than Yjs: prepending 6,000 items takes 33 ms against 135 ms, and appending 6,000 numbers takes 34 ms against 141 ms.

When every operation is its own transaction, each one crosses the JavaScript–WebAssembly boundary, and Ywasm is much slower: 16.9 s for the B4 trace against 2.56 s, and 1.17 s against 96 ms for merging concurrent inserts and deletes (B2.4). Ywasm's C1.1 is quadratic. Doubling the number of operations multiplied its time by 4.4 in our scaling test, and the full test takes 773 s. Yjs completes the same test in 24 s.

Both take about 20 s for C1.2, applying 30,000 updates from 30,000 distinct clients, and 5 to 6 s to load the resulting document. Loro takes 1.9 s and 0.35 s. We did not investigate the cause; it is the only test where the number of distinct clients is large, so it is likely related to per-client state.

Two settings inherited from the upstream benchmark affect Yjs and Ywasm. Both run with gc: false, which retains deleted content. That costs nothing in time but makes the documents larger (227 kB for B4 instead of 160 kB with garbage collection on, and 7.8 MB in B3.3 where 1,540 clients each wrote a 6 kB string to the same key). Both also encode updates in Yjs's V2 format. The V2 encoder has a fixed cost per update that dominates on single-character edits: Yjs completes B4 in 1.34 s with the default V1 format and 2.3 s with V2. Most applications use V1. We kept V2 to stay comparable with the upstream results, so the Yjs timings for per-keystroke workloads are about 40% higher than a typical application would see.

Loro

Loro loads documents faster than any other library: 4.5 ms for the B4 document, 50 ms for the 6.75 MB C1.1 document (Yjs: 1.27 s), 351 ms for the C1.2 document (Yjs: 6.45 s). Its snapshot format stores both the operation log and the materialised state, so loading does not replay history. The same format makes its documents 1.5 to 3 times larger than Yjs's.

Its timings are the most even. Apart from one pattern it is never more than about 8 times slower than the fastest library in a row, and it is the fastest on B2.2 (concurrent random inserts) and second on both C1 tests. The exception is many clients concurrently inserting text into the same container (B3.4 and B3.5): 350 ms against 41 ms for Yjs.

Its per-edit update messages are about three times the size of Yjs's (88 bytes against 27 for a single character), and exporting an update after every commit is most of its B4 time: replaying the trace takes 0.25 s with no commits, 0.48 s with a commit per edit, and 2.4 s with a commit per edit and an update exported after each. An application that commits per keystroke but exports updates less often would see the lower figure.

One observation we could not explain. When B1, B2 and B3, which create about 8,000 short-lived documents, ran before B4 in the same Node process, Loro's B4 replay took 6.7 s instead of 2.4 s, consistently. Any two of the three suites first had no effect. The campaign therefore runs each suite in a fresh process for every library, and the reported number is the clean one. Applications that create many documents in one process may want to check for this.

Bundle size

The pure-JavaScript libraries ship in 24 kB (Yjs) and 49 kB (json-joy) gzipped. The WebAssembly libraries ship in 350 kB (Ywasm), 1.08 MB (Loro) and 1.14 MB (Automerge). Almost all of the difference is the wasm binary itself.

Choosing

• If you need version history, time travel, or branching, the choice is between Loro and Automerge; Yjs, Ywasm and json-joy do not keep an operation log. Of the two, Loro is faster in every row of this benchmark and loads documents 100 times faster.
• If you send each edit as it happens, json-joy, Yjs and Loro all handle 260,000 single-character edits in under three seconds. Automerge does not, and Ywasm's per-transaction cost makes it the wrong binding for that pattern; use Yjs instead.
• If you use Automerge, batch incoming changes before applying them, and expect large single changes to be slow.
• If bundle size matters, Yjs and json-joy are 20 to 50 times smaller than the WebAssembly libraries.
• json-joy is the fastest library here and produces competitive document sizes, but it is younger and less widely deployed than Yjs, and it has no operation log. Its slow prepend is the one workload to check.

What this benchmark does not measure

• Memory. The memUsed metric only sees the JavaScript heap, so it excludes the WebAssembly memory of Ywasm, Loro and Automerge. It is reported for continuity with upstream and should not be used to compare libraries.
• Rich text formatting, tree and move operations, undo, or presence; only plain text, arrays and maps.
• Sync protocols and networking. Updates are handed from one document to the other in the same process.
• Browser performance. Everything runs in Node 24 on one machine.
• Documents larger than a few hundred kilobytes, apart from C1.