How the board stays consistent
The board on this site is a multi-user app on Netlify Server. Several people can edit it at once over WebSockets, and it stays consistent while its server instances are replaced on deploy, recycled, crash or run side by side. This page explains how, and links to the parts of the Netlify docs it relies on.
Why not keep rooms in memory?
The obvious way to build a multi-user room is to keep its state in the server's memory. Clients connect and send changes, and the server applies them and broadcasts the result. That works as long as there's one long-lived process. On Netlify Server, you can't count on that:
- A deploy starts a new server alongside the current one, moves traffic once it's ready, and then shuts the old instances down gracefully (Deploys). The old instance keeps its WebSockets until it shuts down, and during that time, people in the same room are split across two instances running different code.
- An instance is replaced after 8 hours, even when it's busy (Instance lifetime), and it can crash at any point.
- A WebSocket is closed when it reaches the request duration limit (Request duration), so even a quiet connection doesn't last forever.
- A server can run more than one instance of the same deploy. Netlify adds instances as traffic grows, and sends a new WebSocket to another instance when one is full (Instances, WebSocket connections).
So any state that lives only in an instance's memory would sooner or later drift apart between instances, or get lost. The docs recommend the same thing, since each instance holds its own connections: state that clients share should be in a database (Connections and limits).
The database is the only source of truth
Instances keep no state that matters. Every change goes through the database, and an instance just relays what's in it.
In practice, that comes down to five rules:
- The server confirms a change to the client only after it's committed to the database.
- Every change gets its room's next version number, so each room has an ordered log of changes.
- A client remembers the last version it saw. After any reconnect, whatever the reason, it asks for the events after that version.
- A client keeps the changes that weren't confirmed yet (its outbox) and resends them after a reconnect. Each change has an id, and the database keeps the answer it got, so a change is never applied twice.
- On shutdown, an instance only tells its clients to reconnect. There's nothing to flush.
The database is Netlify Database, which is Postgres, so the server gets transactions, LISTEN/NOTIFY and a schema that's applied on deploy (Migrations).
The life of a change
Let's follow a single change: Ana moves a note while Cara, on the other instance, is watching.
- Ana's tab makes the change. It shows up right away (faded, as pending), and goes into her outbox with a fresh id.
- The tab sends it to instance A, which queues it behind any other changes waiting for the same room. One queue per room is what keeps a connection's changes in the order it sent them ("add a note" can't lose a race to "move that note").
- A commits the room's queue in batches. First, it reads the room's state, version and write count, plus which of the batch's ids were answered before.
- It runs
apply()on each change in order. Then a single statement writes the new events, any rejections and the room's new state, and callspg_notify, but only if the write count is still the one it read. If another instance wrote to the room in between, nothing is written, and the batch starts over. That's two database round trips per batch, with nothing locked in between. - A sends Ana the ack with her change's version, and sends the new event to everyone in the room that's connected to it. It reads the event back from the log, the same way it would for an event that another instance wrote.
- B gets the NOTIFY, reads the events after the last one it relayed, and sends them to Cara. If that notification got lost, B's once-a-second poll of the room's version picks it up instead.
Every change gets exactly one answer
A client may send the same change more than once, since it resends its whole outbox after every reconnect. So the database remembers the answer each change got: committed changes are in room_events (with their version), and rejected ones are in room_rejections (with the reason). Whenever a change comes in again, it gets the same answer as the first time.
| The first time | Every time after that |
|---|---|
| Committed as version 105 | Acked as version 105, and not applied again |
| Rejected ("That note was deleted.") | Rejected with the same reason, even if it would apply by now |
Storing the rejections matters more than it might seem. Otherwise, a change that one instance rejected could be committed later by another one (or by the same instance, later in a batch), once the room had changed.
The commit checks the room's write count rather than its version, since a write that only rejects changes doesn't change the version. That same check also makes versions follow commit order. With a global sequence (bigserial), two inserts can commit out of order, and a reader that already saw version 105 would never see a 104 that committed after it.
The only thing that rejects a change is apply() throwing (or a change the server could never store, like one with a malformed id, which it rejects right away). If a commit fails for any other reason (say the database restarted), the server closes the connections in that batch, and their clients reconnect and resend.
Reconnecting and catching up
Every reconnect looks the same, whatever caused it. The client connects with the last version it saw, the server sends what it missed, then says ready, and the client resends whatever is still in its outbox. Here's how that looks on the wire, for a client that saw version 105 before its connection dropped, with two changes in its outbox:
A client with no version yet, or one that's more than 500 versions behind, gets a snapshot of the room instead of the events. And if a resent change was already committed (say the old instance finished committing it just as the socket dropped), the ack simply carries the version it got the first time.
When to reconnect depends on why the connection closed. Code 1012 means the instance is shutting down: the server's onShutdown hook closes every socket with it, so clients move to a new instance right away rather than waiting out the shutdown. The client comes back within a random 0-1 s, which spreads the instance's clients out a bit. Anything else backs off exponentially, with jitter, up to 10 s, and the backoff starts over once a connection gets to ready. The client also pings every 10 s and drops a connection it hasn't heard from in 20-30 s, since a browser can keep a dead connection "open" for minutes after a network change.
In the docs: Reconnecting recommends the same 1012 handling, and Shutdown describes what happens to open connections when an instance stops.
What happens when...
With all of the above in place, every failure ends the same way: the client reconnects and catches up. Pick a case:
What the client sees
Why nothing is lost or repeated
Presence is soft state
Who's in the room (and whatever the app wants to show about them, like a name) is the kind of state you can afford to lose, so it stays out of the log. Every 300 ms, each instance writes the rows of its clients whose data changed or that it hasn't written in 3 s, all in one statement. Then it reads its rooms' rows seen in the last 8 s, and sends each room's list to its clients if it changed.
When an instance stops, it deletes its rows. When it crashes, nobody deletes them, but nobody needs to either: its clients drop off the list once their rows are 8 s old.
The platform pieces it uses
The pattern needs very little from the platform. Here's each piece, what the board's server does with it, and where it's documented:
| Piece | What it's used for |
|---|---|
upgradeWebSocket | Accepts each room's WebSocket, at /api/live. |
onStart | Starts listening for NOTIFY, and the loops that poll room versions and write presence. |
onShutdown | Runs as soon as the instance starts shutting down. It closes every socket with 1012, deletes the instance's presence rows, and waits for drained (the instance's work in progress) before closing the database pool. There's nothing to save, since every change was committed before it was confirmed. |
onHealthCheck | Reports the number of open sockets as a named metric. That's only for observability: named metrics don't affect load-aware routing. |
getContext() | Gives the deploy id, which the board shows in its header. |
path | Sends /api/* to the server. The pages themselves are static files. |
| Netlify Database | The log, the rooms' state, the rejections and presence, plus LISTEN/NOTIFY between instances. |
region | Not set here, so the server runs in the default region. |
Open sockets also keep an instance from stopping for inactivity (Inactivity), so an instance doesn't stop while people are in a room.
What it doesn't do
This is a sample, so a few things are left out on purpose:
- There are no user accounts or rate limiting. Anyone who can reach the site can watch any room that exists. The server names everyone, and a site can be made read-only for everyone without a shared edit token.
- The log keeps a small row per change forever. Old events aren't needed to rebuild a room (a client that's far behind gets a snapshot), but their ids are what makes a late resend safe. So the server only drops the change itself from events older than a day (except each room's last 1,000), and keeps the ids.
- A room's state is one
jsonbvalue, written with every batch. That's fine for a board. For large state, you'd keep snapshots in Netlify Blobs every N events, with a pointer in the database. - Different versions of
apply()aren't detected. Clients compute the state from events, so every version ofapply()that's running at the same time has to agree. That includes adding a new kind of operation, which old code would reject. - The outbox lives in memory. Unsent changes survive reconnects, deploys and crashes, but not a reload or a closed tab.
How we know it works
There are four layers of tests, from unit tests of the client against a fake WebSocket, through integration tests against Postgres with real instances, to seeded chaos runs (deploys, crashes, database restarts and clients going offline, all while clients keep writing) and a run against a real deploy on Netlify. At the end of each scenario, the tests check that every outbox is empty, every client has the same state as the database at the same version, every change was committed exactly once or rejected, each client's changes were committed in the order it made them, and the log has no gaps. The run against a deploy can't reach the database, so it checks the clients against one that saw every event.
Try it yourself
Go back to the board and open it in a second browser too. If the board is read-only there (like on the live demo), deploy your own copy first. Then:
- Add simulated users from the side panel. Each one has its own connection, so to the server it's just another person. Tidy lines the notes up, Writer types into them, Flaky wifi keeps dropping offline (watch its unsent changes pile up and then go through, with a few rejected), Mover shuffles them around a bit, and Gremlin deletes notes.
- Press Go offline, make some changes (they show faded, with an "unsent" count), and go back online.
- Press Crash instance, which really does crash it 🙂 Everyone reconnects, and nothing that was confirmed gets lost. It's turned off on this site, since anyone can press it, so deploy your own copy and set
ALLOW_CRASH=trueto try it.
The code is in two places: netlify/server/live/ for the server side, and public/live/room.js for the client. The board itself plugs in with a single function, applyOp(state, op).
In the docs: the Netlify Server overview, Real-time, Lifecycle, Limits and the API reference.