The cheapest byte is the one we don't send
In a realtime database, every write gets multiplied by the number of people watching it. If a client writes 1 KB to a path with 100 subscribers, that single write turns into roughly 100 KB of outbound traffic. Because Lark Cloud bills on outbound bandwidth (and because every event you send also costs CPU and battery on your users' devices to parse and re-render), unnecessary events show up pretty quickly on both your bill and your profiler.
When you look at the traffic coming out of a typical Firebase app, a surprising amount of it is triggered by writes that didn't actually change anything. Presence heartbeats repeatedly call set(true) on a timer. Autosave loops write an entire form object on every keystroke. Games broadcast full state objects every tick, and UI components call update() with every field they manage when only one field was edited.
Over the last few weeks we went through Lark looking for work we could simply not do -- on the network, on disk, and in memory.
Writes that change nothing
Starting with the upcoming release, a write that leaves the data as it already was no longer sends events to any listeners. If /game/hp is 10 and a client sends set("/game/hp", 10), the writer still gets its acknowledgement back so its promise resolves as usual, but no data events are broadcast to subscribers.
The same write also stays out of the write-ahead log on disk. That applies to a set of the value already there, an update where every key already holds the value being written, or a remove of a path that doesn't exist. Skipping the WAL means less disk I/O right now and less background compaction work later.
There is an important catch to how we do this, though, and it comes down to a deliberate trade-off: the comparison only ever looks at data that is already in memory, and never reads from disk.
In Lark, an active database doesn't have to keep its entire JSON tree in RAM; cold subtrees stay on disk in blob.lark behind lazy sentinel nodes until something reads them. If a write lands on a path that isn't currently loaded in memory, reading the old value off disk just to decide whether we can skip sending a few bytes over the network would cost more than simply sending it. So an unchanged write to unloaded data is still written and broadcast as normal. Because that write puts the value into memory, if the client sends the same write a second time a moment later, the second one is skipped. (Volatile paths are also always sent, since volatile sets go straight to listeners in batches without ever touching the stored tree.)
If you read our earlier post on Firebase Gotchas, you might remember a bug we hit where the server acknowledged a write without echoing a data event back to the writer, and the writer's own UI saw the data disappear. That happened because the Firebase SDK applies your write optimistically to its local view, waits for the server's ack, and then drops its optimistic write and recomputes the view from its cached server state.
Why is it safe to skip the data event here without triggering that same bug? Because when a write changes nothing, the canonical server state that the SDK already holds is identical to the optimistic overlay it just applied. When the ack arrives and the SDK throws away the optimistic write, recomputing from its existing server cache produces the exact same state.
Updates send only what changed
Even when a write isn't a complete no-op, it often carries fields that didn't need to be there. It's very common in application code to hold a local object -- say, a character sheet or a settings panel -- and pass the whole thing to update() whenever the user changes a single field.
Previously, if a client called update("/game", {hp: 2, sheet: ...}) where hp changed from 1 to 2 but sheet was identical to what was already stored, Lark broadcast the entire update payload to every listener on /game. Now, the server compares each key in the update against what's in memory and strips out the keys that didn't change, so subscribers on /game only receive {"/hp": 2}.
Just as importantly, any listeners subscribed below the update's path that aren't touched by the remaining keys are spared completely. In that same example, a client listening on /game/sheet or /game/sheet/stats now receives nothing at all, because the only key that actually changed (hp) is elsewhere in the tree.
Patches for nested listeners
While we were looking at how update() interacts with listeners below its path, we fixed a second inefficiency that was much worse on blob-backed databases.
Say a client is subscribed to /rooms/42/players, and someone sends a multi-path update one level higher, at /rooms/42:
{"players/abc/x": 10, "status": "live"}
Previously, whenever an update arrived above a listener's path and touched anything inside it, Lark sent that listener its entire value again as a full put snapshot. Worse still, if parts of /rooms/42/players had gone idle and been evicted from memory, the server had to read those unloaded children back from the blob on disk just to assemble the full /rooms/42/players object and send it out. A tiny coordinate update could trigger both disk reads and a large snapshot broadcast.
Now, as long as none of the keys in the update overwrite /rooms/42/players itself (or a path above it), the listener receives only the keys that write below its path, formatted as a relative patch:
{"/abc/x": 10}
It gets the exact same small patch event it would have received if the writer had called update("/rooms/42/players", {"abc/x": 10}) directly. Nothing else inside /rooms/42/players is read from memory or disk. (If an update does include a key at or above the listener's path that replaces the whole view, the server still loads and sends the full put as before.)
The 30 MB empty database
Bandwidth is one half of the efficiency story; server memory is the other.
One of the patterns we encourage in Lark is using database routing to split a large application into many smaller databases -- for instance, one database per game room, match, or collaborative document -- rather than putting your whole project in a single giant JSON tree. For that pattern to work well, an idle or near-empty database needs to have a very small memory footprint so a single server can comfortably hold thousands of them at once.
When we profiled what a newly loaded database actually allocated before a single byte of user data was written to it, we found that each database was reserving about 30 MB up front.
Almost all of that came from two places:
- Each database's message inbox pre-allocated a ring buffer for 16,384
InboxMessagestructs of roughly 1.5 KB each (about 24 MB), and those memory pages became resident as messages passed through the channel. - Every open blob's read cache (
CachedIO) sized its internal LRU table for its full capacity of 128,000 regions as soon as the file was opened (about 6 MB).
Fixing both was simple and didn't require lowering either limit. Inbox messages are now boxed (Box<InboxMessage>), so each of the 16,384 slots in the ring buffer is an 8-byte pointer rather than a 1.5 KB struct, and the blob read cache's LRU table now grows on demand as regions are cached.
Because fixed per-database overhead is so easy to accidentally regress later when someone adds a field or bumps a buffer constant, we also added allocator-level tests that wrap Rust's global allocator and assert that instantiating a fresh Database allocates well under 1 MB (and a fresh CachedIO under 64 KB). Memory per database is now a tested property right alongside correctness.
The memory we held (in WAL)
There was one more place where active databases were using more RAM than they needed to: the uncompacted write-ahead log.
When writes land in a persistent database, they get appended to a WAL file on disk and folded into blob.lark in the background once the WAL rotates. While those WAL entries are waiting for compaction, the database keeps them in memory so that if a cold path is read from the blob, any newer writes sitting in the WAL can be replayed on top of it.
Previously, we held those pending WAL entries in memory as parsed tree values (ArcValue). Between reference-counted pointers, map nodes, and object headers, a parsed JSON value in memory takes roughly four times as many bytes as the JSON text that produced it.
In truth, most held WAL entries are never read again before the background worker compacts them into the blob; the in-memory tree already has the hot data. So instead of keeping them as parsed values, a database now holds each entry as the raw JSON line that was written to the WAL file, alongside a lightweight path index, and only parses an entry if a blob promotion or a pending-delete check actually needs it. That cut the memory used by held WAL entries by more than half.
It also gave us a nice operational bonus: previously, when an idle database was about to unload, it forced a WAL compaction first so it wouldn't leave uncompacted entries behind. With the leaner WAL tracking and startup replay, a database can now unload immediately without waiting on a compaction pass. (If you're self-hosting and monitoring memory under heavy write load, db_metrics now includes held_wal_bytes and held_wal_entries so you can see exactly how much uncompacted WAL a database is holding.)
Quieter logs
Finally, in the spirit of doing less, we turned down a handful of log lines that were firing on every subtree eviction, promotion, and client auth change. On a busy server with lots of databases and clients coming and going, those lines made up most of the info log.
None of these changes require you to update your client code or change how you write to Lark, and they'll all ship in the next release (and are already live on Lark Cloud). If you've been writing client-side diffing code just to stop update() calls from re-sending unchanged fields, the server will now save you those bytes automatically.