GET /api/realtime) and subscribes to topics. The server pushes whenever state changes, so nothing polls.
Events are invalidations, not content: they carry ids and say what changed. The client refetches the existing REST endpoint, which keeps its own access checks. The only payload events are the moderator feed (below) and job_status_changed.
Delivery path
kair:job_status_changed) and the moderator feed (kair:chunk_transcribed, kair:moderator_insight).
Redis is a hard dependency: the API won’t start without it, and there are no “Redis is down” fallbacks. A failed publish is logged rather than failing the request that triggered it. When a replica’s Redis subscription drops and reconnects, it sends resync to every socket it holds (see below), because anything published in the gap is gone.
Topics
Topic keys must be canonical: the lower-case UUID form
key() produces. Anything else (upper-case, braced, simple) is invalid_topic, because events are routed to clients by exact key. The UI helpers (sessionTopic etc.) lowercase ids.
A connection may hold 200 subscriptions. Beyond that, subscribe_denied with reason too_many, and the client doesn’t retry. Known limit: the sidebar’s live indicator takes one workspace:<uuid> per workspace, so an org with more than 200 workspaces hits the cap. The fix is a single org- or user-level topic.
Transcript text is moderator/admin-only. The hub decides per connection when it dispatches: transcript_segment_added is dropped, and pattern_detected loses its trigger_excerpt, for subscribers without transcript access (RealtimeEvent::for_reader).
Events
Sources that just stop
“Live” is time-based: a track, device or bot is live while it heartbeats. When a source dies nothing is written, so nothing would notify.live_state_sweeper runs every 15 s and finds heartbeats that crossed a threshold between the previous tick and now. Each window starts where the last ended, so a slow or skipped tick widens the next window instead of leaving a gap. It uses two thresholds:
- 120 s (
LIVENESS_WINDOW_SECS, the same cutofflive-sourcesuses) for tracks, bots and devices bound to a session. These sendlive_state_changed. devices.heartbeat_timeout_seconds(what the devices pages call “offline”). These senddevices_changedto the owner.
Meeting bots without webhooks
recall_bot_reconciler (only when recall is configured) asks Recall for the status of bots stuck in Pending/Joined, for when webhooks are missed. It pages through every open bot by id, 20 per 30 s tick, and skips bots untouched for 14 days. Status writes (apply_status_code) are compare-and-set with a retry. That stops a slow writer (reconciler, webhook or stream connect) from walking a newer status backwards, and stops a terminal code such as call_ended from being dropped.
Wire protocol
Client to server (JSON text frames):reasonisforbidden,too_manyorinvalid_topic.- A server
resyncmeans the replica’s Redis feed reconnected and events may have been lost; the client resyncs every topic. - The server sends WebSocket ping frames every 25 s. It drops a connection whose client has sent nothing for 75 s, since clients ping every 25 s.
Auth: tickets
Browsers cannot set headers on a WebSocket. So:POST /api/realtime/ticket(normal Bearer auth) stores the caller’s access token in Redis under a random 256-bit ticket for 30 s.GET /api/realtime?ticket=...redeems it withGETDEL(single use), verifies the token throughverify_bearer_token(same code path as every other route) and upgrades. A bad ticket is a plain401before the upgrade.
- API keys have no JWT to park, so they cannot open a socket.
- A token within 60 s of expiry is refused a ticket with
401, which drives the client’s normal refresh-and-retry. The upgrade also refuses an already-expired token. Without these, JWT validation leeway would let a just-expired token open sockets that close immediately with4001, in a tight loop.
Close codes
Backoff (1 s to 30 s, jittered) only resets after a connection has stayed up for 30 s. A server that accepts and then closes straight away therefore still backs off.
The resync contract
A missed event must never leave stale UI. The client:- loads state over REST on mount, then refetches on events;
- receives a synthetic
resyncnotice after every subscribe ack, after every reconnect, after a serverresync, and when a tab that was hidden for 5 s or more becomes visible again. Onresync, refetch; - clears the shared in-flight GET map on every event and resync (
invalidateInFlightGets). A refetch must not join a request that read the database before the change it is reacting to.
reason: "subscribed"; the watch* helpers and stream wrappers already do.
There is no polling fallback. The socket is required infrastructure like the API itself. If it can’t connect, the client keeps reconnecting with backoff, and the UI stops updating until it can.
Stopping a recorder
A moderator’s Stop must reach a browser recorder even if its socket was down when the push went out. The push is the fast path; the stored state is the durable one:- The stop route marks every live capture track
Finalisingand publishessession_stop_requested. - Every chunk upload response carries
stop_requested: trueonce the chunk’s track is no longer Recording/Paused. The recorder uploads every ~10 s, so it notices within one chunk even without a socket. - On every catch-up resync, the recorder (
activeRecording.ts) refetches its own track and stops if it has left Recording/Paused.
Client (ui/src/lib/realtime/)
client.ts: module singleton.subscribe(topic, handler)is reference-counted per topic.- It connects lazily and disconnects 10 s after the last unsubscribe.
- It reconnects with backoff and on
online. - It pings every 25 s; no pong within 10 s forces a reconnect.
- The ticket request times out after 10 s.
- Any identity change (logout, logging in as someone else, starting or leaving impersonation) drops the socket and reconnects as the new user. Handlers are kept, so long-lived consumers like the sidebar stores survive it.
refresh.ts:createCoalescedRefresh(250 ms debounce, never two runs at once) pluswatchSessionLiveState,watchSessionActivity,watchWorkspaceLiveStateandwatchDevices.ui/src/lib/api/client.ts:streamJobEvents,streamWorkspaceReportJobEvents,streamAdminJobEvents,streamModeratorAlertsandstreamJobProgresskeep their callback shapes but ride the socket. There are no SSE endpoints any more.
Adding an event
- Add a variant to
RealtimeEventinapi/crates/core/src/realtime.rs. Carry ids, not content. - Emit it after the write succeeds, with
util::realtime_publish::notify(or a helper such asnotify_live_state). Workers usepublish_realtimedirectly. - Mirror it in
ui/src/lib/realtime/types.tsand subscribe withsubscribe()or awatch*helper inrefresh.ts, refetching the REST endpoint. - If it needs a new topic, add it to
RealtimeTopicand its access check inroutes/realtime/realtime_routes_access.rs.