Skip to main content

Clients

Two transcription backends are supported, selected via transcription.client in config.yml (switchable at runtime via the admin UI): See Configuration for all config keys.

Silence hallucinations (local whisper.cpp)

Over silence or room noise, whisper.cpp can emit stock subtitle phrases (“Thank you”, “Thanks for watching”) and repeat them. The transcription worker mitigates this in layers:
  1. Decode-time (transcription.local in config): whisper_no_context (default on) disables conditioning on previous text; optional Silero VAD (local.vad, model on the ai_models volume) skips non-speech audio; suppress_non_speech_tokens, logprob/entropy/no-speech thresholds, and temperature fallback match whisper.cpp defaults unless overridden.
  2. Post-decode (transcription.hallucination_filter): drops segments with very low confidence or high no_speech_prob; removes known stock phrases when the segment window is quiet or low-confidence. A segment that is only back-to-back repeats of one or more stock phrases (“Thank you. Thank you.”) counts as that phrase, matched on whole words; segments starting with a prefix such as “Untertitel” (German subtitle credits) also match. Genuine short “thank you” with strong confidence is kept.
  3. Collapse (kair_voice_core::repeat_collapse): consecutive identical segments (run ≥ 3) and intra-segment comma-clause loops are collapsed before persistence.
The same hallucination_filter block applies to Scaleway responses (confidence from avg_logprob; no no_speech_prob on the remote path).

Switching behaviour

  • local → scaleway: takes effect per-job, no restart needed.
  • scaleway → local: requires a worker restart — whisper.cpp models are only loaded when the worker boots with client: local.

Audio decoding

Audio is decoded in-process with symphonia (WAV, MP3, AAC/ADTS, FLAC, Vorbis). If symphonia rejects the file — multi-frame ADTS AAC, truncated streams, MP4/M4A containers — the worker re-decodes it through ffmpeg -err_detect ignore_err into a 16 kHz mono WAV and retries. The job fails with Failed to load audio only when both decoders fail; the message names both errors. Both worker images ship ffmpeg. S3 downloads retry transient failures (3 attempts) unless the object is missing or access is denied.

Upload size limits

The Scaleway backend uploads the normalised 16 kHz mono WAV (~1.9 MB per minute). When that exceeds transcription.scaleway.max_upload_mb (default 25), it is re-encoded as 32 kbit/s mono MP3 (~14 MB per hour) first. Only audio still over the limit after compression fails, with Remote transcription upload too large. Separately, the API caps single-file session audio uploads at 200 MiB; long sessions must use chunked upload. Config resolver: api/crates/transcription/src/settings.rs.

Scaleway

Credentials live in ai_inference.scaleway (base_url + api_key); model names live in transcription.scaleway. The worker still POSTs to {base_url}/audio/transcriptions. If Scaleway rejects verbose_json, the client retries with json (untimed single segment if no segments array is returned). Dedicated Whisper deployments should use model: whisper-large-v3 and a base_url ending in /v1 (not /audio/transcriptions).

Name redaction (NER)

Person names are redacted at transcript time, before segments are stored, so summaries, strands and the topic graph only ever see <PERSON>. The regex redactor (emails, phones, IBANs, …) runs first, then an ONNX NER model (transcription.pii.ner, see configuration) runs over the whole transcript in overlapping 256-token windows. Names it detects are also replaced wherever they recur in the same transcript, including German genitives (“Annas Vorschlag”). Locations and organisations are opt-in (kinds).
  • Fail-closed: a model that cannot load stops the worker; an inference error fails the transcription job. There is no silent skip.
  • Pasted summaries get the same two passes (crate kair-voice-pii): the rag worker redacts a paste before asking the model to split it into topics. It reads transcription.pii, always runs NER on CPU (the main image ships CPU ONNX Runtime; device only applies to the transcription worker), and fails the job rather than prompt on unredacted text. Uploaded documents are not redacted.
  • The same step runs for remote providers (staging/dev), after the provider returns text.
  • Live chunks are redacted one chunk at a time, so a name only spoken in a chunk of its own can be missed there; the final pass sees the whole recording.
  • Summaries are told that <PERSON> is a redacted name and must become a role or “a participant”. A stray <PERSON> in a summary is rewritten to [participant].
  • Run device: cuda only after checking the ONNX Runtime GPU build has kernels for your card (cuobjdump --list-elf libonnxruntime_providers_cuda.so | grep sm_61 for the GTX 1080); otherwise use device: cpu, which is usually fast enough.

GPU notes (CUDA image)

api/Dockerfile.transcription-cuda builds whisper.cpp for one architecture (--build-arg CUDAARCHS=61 for the GTX 1080) and bundles Microsoft’s ONNX Runtime GPU build for the NER model (ORT_DYLIB_PATH). Whisper tuning lives under transcription.local (use_gpu, gpu_device, flash_attn, beam_size, warm_up); nr_of_threads_for_transcription still sets the CPU threads whisper.cpp uses beside the GPU, so keep it low (1-2) on the GPU worker. Quantised ggml models (q8_0, q5_0) cut VRAM; compare word error rate on invented audio before switching.

Speaker labels

The transcription worker does not diarise. Whisper segments (local, Scaleway) carry no speaker label. Labels exist only for sources that supply them: Recall (P{id} labels reconciled onto final-pass segments) and phone/track names. Speaker claims & relabelling still operate on the labels present on segments.