Skip to main content
The web app keeps one WebSocket per tab (GET /api/realtime) and subscribes to topics. The server pushes whenever state changes, so nothing polls. Events are invalidations, not content: they carry ids and say what changed. The client refetches the existing REST endpoint, which keeps its own access checks. The only payload events are the moderator feed (below) and job_status_changed.

Delivery path

Producers never write to local sockets. Publishing through Redis is what makes this safe with several API replicas. Two sources are bridged locally on every replica instead, because each replica already receives the Redis message: job status (kair:job_status_changed) and the moderator feed (kair:chunk_transcribed, kair:moderator_insight). Redis is a hard dependency: the API won’t start without it, and there are no “Redis is down” fallbacks. A failed publish is logged rather than failing the request that triggered it. When a replica’s Redis subscription drops and reconnects, it sends resync to every socket it holds (see below), because anything published in the gap is gone.

Topics

Topic keys must be canonical: the lower-case UUID form key() produces. Anything else (upper-case, braced, simple) is invalid_topic, because events are routed to clients by exact key. The UI helpers (sessionTopic etc.) lowercase ids. A connection may hold 200 subscriptions. Beyond that, subscribe_denied with reason too_many, and the client doesn’t retry. Known limit: the sidebar’s live indicator takes one workspace:<uuid> per workspace, so an org with more than 200 workspaces hits the cap. The fix is a single org- or user-level topic. Transcript text is moderator/admin-only. The hub decides per connection when it dispatches: transcript_segment_added is dropped, and pattern_detected loses its trigger_excerpt, for subscribers without transcript access (RealtimeEvent::for_reader).

Events

Sources that just stop

“Live” is time-based: a track, device or bot is live while it heartbeats. When a source dies nothing is written, so nothing would notify. live_state_sweeper runs every 15 s and finds heartbeats that crossed a threshold between the previous tick and now. Each window starts where the last ended, so a slow or skipped tick widens the next window instead of leaving a gap. It uses two thresholds:
  • 120 s (LIVENESS_WINDOW_SECS, the same cutoff live-sources uses) for tracks, bots and devices bound to a session. These send live_state_changed.
  • devices.heartbeat_timeout_seconds (what the devices pages call “offline”). These send devices_changed to the owner.

Meeting bots without webhooks

recall_bot_reconciler (only when recall is configured) asks Recall for the status of bots stuck in Pending/Joined, for when webhooks are missed. It pages through every open bot by id, 20 per 30 s tick, and skips bots untouched for 14 days. Status writes (apply_status_code) are compare-and-set with a retry. That stops a slow writer (reconciler, webhook or stream connect) from walking a newer status backwards, and stops a terminal code such as call_ended from being dropped.

Wire protocol

Client to server (JSON text frames):
Server to client:
  • reason is forbidden, too_many or invalid_topic.
  • A server resync means the replica’s Redis feed reconnected and events may have been lost; the client resyncs every topic.
  • The server sends WebSocket ping frames every 25 s. It drops a connection whose client has sent nothing for 75 s, since clients ping every 25 s.

Auth: tickets

Browsers cannot set headers on a WebSocket. So:
  1. POST /api/realtime/ticket (normal Bearer auth) stores the caller’s access token in Redis under a random 256-bit ticket for 30 s.
  2. GET /api/realtime?ticket=... redeems it with GETDEL (single use), verifies the token through verify_bearer_token (same code path as every other route) and upgrades. A bad ticket is a plain 401 before the upgrade.
  • API keys have no JWT to park, so they cannot open a socket.
  • A token within 60 s of expiry is refused a ticket with 401, which drives the client’s normal refresh-and-retry. The upgrade also refuses an already-expired token. Without these, JWT validation leeway would let a just-expired token open sockets that close immediately with 4001, in a tight loop.
Access is checked once, at subscribe time, with the same guards as the REST endpoints.

Close codes

Backoff (1 s to 30 s, jittered) only resets after a connection has stayed up for 30 s. A server that accepts and then closes straight away therefore still backs off.

The resync contract

A missed event must never leave stale UI. The client:
  • loads state over REST on mount, then refetches on events;
  • receives a synthetic resync notice after every subscribe ack, after every reconnect, after a server resync, and when a tab that was hidden for 5 s or more becomes visible again. On resync, refetch;
  • clears the shared in-flight GET map on every event and resync (invalidateInFlightGets). A refetch must not join a request that read the database before the change it is reacting to.
The ack-resync closes the race between the REST load and the server registering the subscription. Callers that just loaded on mount ignore reason: "subscribed"; the watch* helpers and stream wrappers already do. There is no polling fallback. The socket is required infrastructure like the API itself. If it can’t connect, the client keeps reconnecting with backoff, and the UI stops updating until it can.

Stopping a recorder

A moderator’s Stop must reach a browser recorder even if its socket was down when the push went out. The push is the fast path; the stored state is the durable one:
  1. The stop route marks every live capture track Finalising and publishes session_stop_requested.
  2. Every chunk upload response carries stop_requested: true once the chunk’s track is no longer Recording/Paused. The recorder uploads every ~10 s, so it notices within one chunk even without a socket.
  3. On every catch-up resync, the recorder (activeRecording.ts) refetches its own track and stops if it has left Recording/Paused.

Client (ui/src/lib/realtime/)

  • client.ts: module singleton.
    • subscribe(topic, handler) is reference-counted per topic.
    • It connects lazily and disconnects 10 s after the last unsubscribe.
    • It reconnects with backoff and on online.
    • It pings every 25 s; no pong within 10 s forces a reconnect.
    • The ticket request times out after 10 s.
    • Any identity change (logout, logging in as someone else, starting or leaving impersonation) drops the socket and reconnects as the new user. Handlers are kept, so long-lived consumers like the sidebar stores survive it.
  • refresh.ts: createCoalescedRefresh (250 ms debounce, never two runs at once) plus watchSessionLiveState, watchSessionActivity, watchWorkspaceLiveState and watchDevices.
  • ui/src/lib/api/client.ts: streamJobEvents, streamWorkspaceReportJobEvents, streamAdminJobEvents, streamModeratorAlerts and streamJobProgress keep their callback shapes but ride the socket. There are no SSE endpoints any more.

Adding an event

  1. Add a variant to RealtimeEvent in api/crates/core/src/realtime.rs. Carry ids, not content.
  2. Emit it after the write succeeds, with util::realtime_publish::notify (or a helper such as notify_live_state). Workers use publish_realtime directly.
  3. Mirror it in ui/src/lib/realtime/types.ts and subscribe with subscribe() or a watch* helper in refresh.ts, refetching the REST endpoint.
  4. If it needs a new topic, add it to RealtimeTopic and its access check in routes/realtime/realtime_routes_access.rs.