Skip to main content

PostgreSQL — the source of truth

Everything structured lives here: users, workspaces, sessions, transcripts, jobs, summaries. If you need to understand “what happened in session X”, the answer is in Postgres. Key tables: Why Postgres? Relational integrity matters here — a transcript belongs to a job, a job belongs to a session, a session belongs to a workspace. Foreign keys and transactions prevent orphaned records.

RabbitMQ — job delivery

Heavy work (transcription, summarisation, topic graph, documents) is dispatched through RabbitMQ. When a chunk is uploaded, the API commits a job row to Postgres and publishes the job to a quorum queue; a worker consumes it, does the work and acknowledges the message only when the job is done. A worker killed mid-job does not lose the job — the broker redelivers it to another replica.
Postgres remains the source of truth for job state; RabbitMQ only owns delivery. Queues, priorities, retries and dead-lettering are covered in Queue System. Why RabbitMQ? Acknowledgement-based delivery survives worker crashes, quorum queues replicate across broker nodes, and delayed retry and dead-letter handling are built in. The previous Redis list queue lost in-flight jobs when a worker died.

Redis — pub/sub & ephemeral state

Redis is no longer a job queue. It holds short-lived coordination state and fans events out to browsers.

WebSocket pub/sub & recording status

Live session state (is a recording active? who’s recording?) is stored in Redis with a TTL. The web app subscribes via the realtime WebSocket (see Realtime); the API publishes invalidations so the UI refetches without polling. Worker liveness heartbeats, device command lists and rate-limit windows live here too.
Why Redis? Low-latency reads/writes, built-in pub/sub, TTL for ephemeral state. Nothing in Redis is the only copy of anything that matters: job state is in Postgres.

Neo4j — topic graph

One instance is configured, rag.neo4j_v2 (topic graph). Topic graph (rag.neo4j_v2). After a session summary completes, the topic_graph job in the rag worker merges its paragraphs into the workspace’s topics in Postgres and projects the workspace into Neo4j: topics, sessions, entities (places, organisations, concepts; never people) and themes, plus a vector index over topic sentences. Uploaded documents are projected beside them as Document and DocumentParagraph nodes (the anonymised paragraphs of their summary, with their own vector index) by the document_graph job; they link only to entities the topics already hold. See Topic History. This powers the “ask a question across all sessions” feature — instead of searching raw text, the moderator queries the topics a workspace has discussed, how each developed across sessions, and which themes they form. The v1 config rag.neo4j and the v1 entity graph are gone (#132). Why Neo4j? Graph traversal queries (“find all topics that mention entity X across sessions Y and Z”) are natural in Cypher but painful in SQL. Postgres stays the source of truth; the topic graph can be rebuilt from it.
The topic graph is only populated after a session summary completes and the topic_graph job runs. If rag.neo4j_v2 is unset, topics stay in Postgres and nothing is projected.

S3 — audio file storage

Raw audio (chunks and stitched files) is stored in S3-compatible object storage. The API writes files here after upload; the transcription worker reads from here to run inference. In local development, the API falls back to local disk storage when no S3 credentials are configured. Why S3? Audio files are large and binary — not suited for a relational DB. S3 scales cheaply, and the transcription worker can stream files directly without going through the API.

Encryption

Every connection to Postgres, RabbitMQ, Redis, Neo4j and S3 is encrypted in transit, and the stores are reachable only over a private VPN. See Architecture.