Skip to main content

Overview

All heavy processing (transcription, summarisation, topic graph) happens asynchronously. The API never blocks waiting for these — it accepts the upload, creates a job record, publishes to RabbitMQ, and returns immediately. Workers process jobs in the background. Postgres is the source of truth for job state; RabbitMQ owns delivery. A worker acknowledges a message only once the job is done, so a worker that is killed mid-job does not take the job with it — the broker redelivers it to another replica. Redis is still used, but only for what it is good at here: pub/sub fan-out to realtime WebSocket clients (kair:chunk_transcribed, kair:job_status_changed, kair:moderator_insight, per-job progress/status), worker liveness heartbeats, and device command lists.

The pipeline

Job types

The queues

All are quorum queues on the kair.jobs direct exchange. Summary evaluations share the summarisation queue rather than having their own. One AMQP consumer cannot strictly prioritise across two queues the way Redis’s multi-key BLPOP did, so the ordering is expressed with message priority instead — see below. Each queue has companions: <queue>.dlq (where exhausted messages land), <queue>.delay (a 5 s TTL ring for work that is merely waiting on its inputs), and <queue>.retry.1m / .retry.5m / .retry.15m (backoff rings for transient failures — see below). Nothing consumes the rings; messages leave them by TTL expiry, which dead-letters them back onto the main queue.

Priorities

Quorum queues support 32 priority levels and default an unset priority to 4. This replaces the old LPUSH-to-front trick; AMQP has no push-to-front.

Retries and dead-lettering

x-delivery-limit is set explicitly on every queue — the broker default is 20, which would silently give every job twenty attempts. Once the limit is exhausted the message dead-letters and the API reconciler marks the row Failed. Delivery-limit redelivery is immediate, which is the wrong shape for an upstream that is down — so transient failures do not use it. They use RetryAfter below.
x-consumer-timeout is set to 4 h on the transcription and summarisation queues. The broker default is 30 min, and if a job outruns it the broker closes the channel and requeues the message mid-job — producing exactly the duplicate work the acknowledgement model exists to prevent. Raise this before raising job durations.

Transient vs permanent failures

kair_voice_core::retry classifies errors: connect failures, timeouts, 429 and 5xx responses (including gateway errors that only surface as text through rig) are transient; bad requests, unknown models and config errors are permanent.
  • Per call. Every LLM, embedding and remote-transcription HTTP call retries transient errors with backoff (RetryPolicy::LLM: 4 attempts over ~15 s). LLM clients have a 10 s connect timeout and a 15 min request timeout.
  • Per job, on the broker. A handler that still sees a transient failure calls mq::schedule_retry: the row goes to interrupted with the error kept and retry_count bumped (the attempt budget), and the handler returns JobOutcome::RetryAfter(tier). The shared consumer republishes the message unchanged onto <queue>.retry.<tier> (with a publisher confirm), then acks it. The broker holds it for 1, 5 then 15 min and routes it back; no worker sleeps on it, a restart loses nothing, and it does not count against x-delivery-limit. Permanent failures fail at once. jobs.error holds the full error chain. The UI treats interrupted as still in progress. schedule_retry only touches rows still pending/processing/interrupted, so a job cancelled mid-attempt stays cancelled.
  • Per item. Within a topic merge, one malformed model response costs only that paragraph. The step fails only on a transient error, which goes to the job-level retry instead of silently dropping half the topics.

Deferral vs retry

A summary waiting on a transcription that has not finished is not failing. Those jobs are republished to kair.summarisation.delay, which TTLs them back onto the main queue, rather than nacked — a nack would count against the delivery limit and dead-letter a job that was only being patient. Deferral is bounded by MAX_SUMMARY_DEFERRALS; the pre-broker implementation had no bound and would re-queue forever against a permanently-stuck transcription. Automatic summaries are only created once their transcription has settled (below), so deferral now only covers summaries requested by hand mid-transcription. A summary whose source transcription has failed or came back empty fails immediately with Source transcription failed: … instead of waiting out the budget.

Summary trigger

kair_voice_core::summary_trigger::maybe_enqueue_session_summaries runs when a final transcription job settles (Completed or Failed, from the transcription worker) and when a track fails or finalises (from the API). Per session, under a Postgres advisory lock:
  • Track sessions — waits until no participating track is still recording and every participating track’s final transcription has settled. It then creates one merged Session summary, plus a Transcript summary per track that produced text when the session has ≥ 2 recording devices (non-failed tracks with recorded chunks); a single-device session gets only the Session summary. A track whose transcription failed simply drops out, so it never blocks siblings. If nothing produced text, no summary is created.
  • Single-stream recordings (finalise_chunks) — one Session summary over the stitched transcription once it completes with text.
  • A summary that previously failed for the same source is requeued rather than duplicated, so re-running a failed transcription from the admin panel also brings its summary back.
Uploaded audio keeps its existing on-demand trigger.

Topic graph job

Runs in the rag worker after a session summary completes. Steps, in order:
  1. Merge the session’s summary paragraphs into the workspace’s topics (skipped for workspaces without an organisation).
  2. Propose cross-workspace topic links.
  3. Project the workspace into the Neo4j topic graph (rag.neo4j_v2).
  4. Run strand enrichment (best effort), then queue pattern_synthesis (best effort). Pattern synthesis takes its entities from the session summary’s paragraph entities.
Erased sessions are skipped. See Topic History for the merge rules.

Chunk vs final transcription

Sessions are recorded in chunks (short audio segments sent as the recording progresses). Each chunk gets its own transcription job immediately — this gives live feedback in the moderator view. When the session ends, all chunks are stitched into one audio file. A final transcription job runs on the full audio. This final transcript is what gets summarised — chunk transcripts are for live display only.
Whisper segments carry no speaker labels. Where a source supplies them (Recall P{id} labels), they are reconciled onto the final-pass segments.

Job states

A worker takes ownership with an atomic claim — a single conditional UPDATE that flips the row to processing and stamps claimed_by/claimed_at. Whoever wins owns the job; every other delivery is acked and discarded. That is what makes at-least-once delivery safe with three replicas: Postgres does the serialising. claimed_at is re-stamped on each delivery, so a stale stamp is the signal that nothing owns a row any more. Both the reconciler and the admin “re-run” action use it to tell an abandoned job from a genuinely long-running one. Failed jobs are visible in the admin panel. Individual failed chunks can be re-run from the admin UI. If the final transcription fails, summarisation and the topic graph job do not run. Admin re-runs reset the row (status, error, claim, retry_count) before publishing, so a fast worker never sees the old failed status and drops the message. The reset is a single conditional update that only matches failed, interrupted or completed rows, so a job still pending or processing (or a second click on the same re-run) gets 409 instead of a duplicate run.

What the broker does not cover

The API runs a reconciler every 60 s (crates/api/src/util/job_reconciler.rs) for the gaps redelivery cannot close:
  • Lost retries. An interrupted job unchanged and unclaimed for 30 min (twice the longest backoff tier) is re-published, at most once per 30 min.
  • Orphaned Pending rows. The job row is committed and then published. Publisher confirms shrink that window but cannot close it, so a crash in between leaves a row nothing will pick up. The reconciler republishes them once they have been Pending for 30 min, at most once per 30 min. Rows Pending and unchanged (by jobs.updated_at, kept current by a trigger) for more than 6 h, or created more than 24 h ago and untouched for 30 min, are aged out to Failed instead, so a queue with no consumer (worker outage, PR preview) does not collect duplicates for ever. The 30 min grace spares admin re-runs, which keep the original created_at.
  • Dead-lettered work. Nothing writes to Postgres when a message exhausts its delivery limit, so the row would sit at processing for ever. The reconciler marks it Failed and publishes on kair:job_status_changed so the admin stream updates.
Document-processing jobs are the one case the reconciler cannot rebuild: documents has no job_id column, so the document↔job link exists only in the queue message. Re-upload is the recovery path. Adding documents.job_id would close this.

Transcription backends

Two backends are supported, switchable at runtime via the admin panel: Switching backend takes effect per-job — no worker restart needed (except switching back to local on a Scaleway-booted worker, which requires a restart to load the Whisper model).
Scaleway caps upload size (transcription.scaleway.max_upload_mb, default 25). Audio over it is re-encoded as 32 kbit/s mono MP3 (~14 MB per hour) before upload; only audio still too large after that fails.

Monitoring

  • Admin panel → Jobs: shows transcription jobs (chunk groups, status, progress) plus topic_graph, document_graph and pattern_synthesis jobs
  • Failed jobs: visible in admin with a “re-run” action per chunk
  • Admin panel → Queues: ready-message depths read live from the broker. Note the summarisation figure counts summaries and evaluations, since they share a queue
  • Dead-letter queues: anything in <queue>.dlq exhausted its retries and needs a human; the matching job row is marked Failed
  • Logs: worker processes log job pickup and completion with session/job IDs

Configuration

Credentials belong in the RABBITMQ_URL environment variable, which overrides the YAML value. Both the API and every worker declare the full topology on connect and again after a reconnect, so start order does not matter.
PR preview stacks share staging’s broker as well as its Postgres. A PR stack runs no workers of its own, so jobs it publishes are only processed because staging’s workers consume the same queues — giving each PR its own vhost would isolate the queues and strand every job.