Overview
All heavy processing (transcription, summarisation, topic graph) happens asynchronously. The API never blocks waiting for these — it accepts the upload, creates a job record, publishes to RabbitMQ, and returns immediately. Workers process jobs in the background. Postgres is the source of truth for job state; RabbitMQ owns delivery. A worker acknowledges a message only once the job is done, so a worker that is killed mid-job does not take the job with it — the broker redelivers it to another replica. Redis is still used, but only for what it is good at here: pub/sub fan-out to realtime WebSocket clients (kair:chunk_transcribed, kair:job_status_changed, kair:moderator_insight, per-job progress/status), worker liveness heartbeats, and device command lists.
The pipeline
Job types
The queues
All are quorum queues on thekair.jobs direct exchange.
Summary evaluations share the summarisation queue rather than having their own. One
AMQP consumer cannot strictly prioritise across two queues the way Redis’s multi-key
BLPOP did, so the ordering is expressed with message priority instead — see below.
Each queue has companions: <queue>.dlq (where exhausted messages land),
<queue>.delay (a 5 s TTL ring for work that is merely waiting on its inputs), and
<queue>.retry.1m / .retry.5m / .retry.15m (backoff rings for transient
failures — see below). Nothing consumes the rings; messages leave them by TTL expiry,
which dead-letters them back onto the main queue.
Priorities
Quorum queues support 32 priority levels and default an unset priority to 4.
This replaces the old
LPUSH-to-front trick; AMQP has no push-to-front.
Retries and dead-lettering
x-delivery-limit is set explicitly on every queue — the broker default is 20,
which would silently give every job twenty attempts. Once the limit is exhausted the
message dead-letters and the API reconciler marks the row Failed.
Delivery-limit redelivery is immediate, which is the wrong shape for an upstream that
is down — so transient failures do not use it. They use RetryAfter below.
Transient vs permanent failures
kair_voice_core::retry classifies errors: connect failures, timeouts, 429 and
5xx responses (including gateway errors that only surface as text through rig) are
transient; bad requests, unknown models and config errors are permanent.
-
Per call. Every LLM, embedding and remote-transcription HTTP call retries
transient errors with backoff (
RetryPolicy::LLM: 4 attempts over ~15 s). LLM clients have a 10 s connect timeout and a 15 min request timeout. -
Per job, on the broker. A handler that still sees a transient failure calls
mq::schedule_retry: the row goes tointerruptedwith the error kept andretry_countbumped (the attempt budget), and the handler returnsJobOutcome::RetryAfter(tier). The shared consumer republishes the message unchanged onto<queue>.retry.<tier>(with a publisher confirm), then acks it. The broker holds it for 1, 5 then 15 min and routes it back; no worker sleeps on it, a restart loses nothing, and it does not count againstx-delivery-limit.Permanent failures fail at once.jobs.errorholds the full error chain. The UI treatsinterruptedas still in progress.schedule_retryonly touches rows stillpending/processing/interrupted, so a job cancelled mid-attempt stays cancelled. - Per item. Within a topic merge, one malformed model response costs only that paragraph. The step fails only on a transient error, which goes to the job-level retry instead of silently dropping half the topics.
Deferral vs retry
A summary waiting on a transcription that has not finished is not failing. Those jobs are republished tokair.summarisation.delay, which TTLs them back onto the
main queue, rather than nacked — a nack would count against the delivery limit and
dead-letter a job that was only being patient. Deferral is bounded by
MAX_SUMMARY_DEFERRALS; the pre-broker implementation had no bound and would
re-queue forever against a permanently-stuck transcription.
Automatic summaries are only created once their transcription has settled (below),
so deferral now only covers summaries requested by hand mid-transcription. A summary
whose source transcription has failed or came back empty fails immediately with
Source transcription failed: … instead of waiting out the budget.
Summary trigger
kair_voice_core::summary_trigger::maybe_enqueue_session_summaries runs when a final
transcription job settles (Completed or Failed, from the transcription worker)
and when a track fails or finalises (from the API). Per session, under a Postgres
advisory lock:
- Track sessions — waits until no participating track is still recording and every participating track’s final transcription has settled. It then creates one merged Session summary, plus a Transcript summary per track that produced text when the session has ≥ 2 recording devices (non-failed tracks with recorded chunks); a single-device session gets only the Session summary. A track whose transcription failed simply drops out, so it never blocks siblings. If nothing produced text, no summary is created.
- Single-stream recordings (
finalise_chunks) — one Session summary over the stitched transcription once it completes with text. - A summary that previously failed for the same source is requeued rather than duplicated, so re-running a failed transcription from the admin panel also brings its summary back.
Topic graph job
Runs in the rag worker after a session summary completes. Steps, in order:- Merge the session’s summary paragraphs into the workspace’s topics (skipped for workspaces without an organisation).
- Propose cross-workspace topic links.
- Project the workspace into the Neo4j topic graph (
rag.neo4j_v2). - Run strand enrichment (best effort), then queue
pattern_synthesis(best effort). Pattern synthesis takes its entities from the session summary’s paragraph entities.
Chunk vs final transcription
Sessions are recorded in chunks (short audio segments sent as the recording progresses). Each chunk gets its own transcription job immediately — this gives live feedback in the moderator view. When the session ends, all chunks are stitched into one audio file. A final transcription job runs on the full audio. This final transcript is what gets summarised — chunk transcripts are for live display only.Whisper segments carry no speaker labels. Where a source supplies them (Recall
P{id} labels), they are reconciled onto the final-pass segments.Job states
UPDATE that
flips the row to processing and stamps claimed_by/claimed_at. Whoever wins owns
the job; every other delivery is acked and discarded. That is what makes at-least-once
delivery safe with three replicas: Postgres does the serialising.
claimed_at is re-stamped on each delivery, so a stale stamp is the signal that
nothing owns a row any more. Both the reconciler and the admin “re-run” action use
it to tell an abandoned job from a genuinely long-running one.
Failed jobs are visible in the admin panel. Individual failed chunks can be re-run from the admin UI. If the final transcription fails, summarisation and the topic graph job do not run. Admin re-runs reset the row (status, error, claim, retry_count) before publishing, so a fast worker never sees the old failed status and drops the message. The reset is a single conditional update that only matches failed, interrupted or completed rows, so a job still pending or processing (or a second click on the same re-run) gets 409 instead of a duplicate run.
What the broker does not cover
The API runs a reconciler every 60 s (crates/api/src/util/job_reconciler.rs) for the
gaps redelivery cannot close:
- Lost retries. An
interruptedjob unchanged and unclaimed for 30 min (twice the longest backoff tier) is re-published, at most once per 30 min. - Orphaned
Pendingrows. The job row is committed and then published. Publisher confirms shrink that window but cannot close it, so a crash in between leaves a row nothing will pick up. The reconciler republishes them once they have been Pending for 30 min, at most once per 30 min. Rows Pending and unchanged (byjobs.updated_at, kept current by a trigger) for more than 6 h, or created more than 24 h ago and untouched for 30 min, are aged out toFailedinstead, so a queue with no consumer (worker outage, PR preview) does not collect duplicates for ever. The 30 min grace spares admin re-runs, which keep the originalcreated_at. - Dead-lettered work. Nothing writes to Postgres when a message exhausts its
delivery limit, so the row would sit at
processingfor ever. The reconciler marks itFailedand publishes onkair:job_status_changedso the admin stream updates.
Document-processing jobs are the one case the reconciler cannot rebuild:
documents
has no job_id column, so the document↔job link exists only in the queue message.
Re-upload is the recovery path. Adding documents.job_id would close this.Transcription backends
Two backends are supported, switchable at runtime via the admin panel:
Switching backend takes effect per-job — no worker restart needed (except switching back to
local on a Scaleway-booted worker, which requires a restart to load the Whisper model).
Monitoring
- Admin panel → Jobs: shows transcription jobs (chunk groups, status, progress) plus
topic_graph,document_graphandpattern_synthesisjobs - Failed jobs: visible in admin with a “re-run” action per chunk
- Admin panel → Queues: ready-message depths read live from the broker. Note the summarisation figure counts summaries and evaluations, since they share a queue
- Dead-letter queues: anything in
<queue>.dlqexhausted its retries and needs a human; the matching job row is markedFailed - Logs: worker processes log job pickup and completion with session/job IDs
Configuration
RABBITMQ_URL environment variable, which overrides the
YAML value. Both the API and every worker declare the full topology on connect and
again after a reconnect, so start order does not matter.