Overview
When enabled, completed summarisations are scored into thesummary_evaluations table with a 0–100 health_score:
- Session summaries (generated) are scored inline by the quality gate before they are shown, and get exactly one evaluation row.
- Document summaries are scored asynchronously after completion, inside the summarisation worker off the shared
kair.summarisationqueue at priority 0, so summaries always drain first. - Transcript (per-track) summaries are not evaluated. They are excluded from the backfill and from the admin stats (daily average, count and recent list); their existing rows stay in the table.
- LLM judge (
atla/selene-mini:q4_k_mon Ollama) - Heuristics (ASR confidence, formatting)
- Embedding cosine similarity between transcript and summary
brief_similarity, 0–1): the proportion of summary sentences whose embedding similarity to goal / what-matters lines exceeds transcript-chunk similarity by the calibrated BRIEF_LEAN_MARGIN for that language triple. Background is excluded from this comparison. This is a review flag only — it does not enter health_score and does not bump WEIGHTS_VERSION.
Only the de/de/de triple is calibrated today (BRIEF_LEAN_MARGIN = 0.05). Other language pairs leave brief_similarity null until they have their own labelled set.
The faithfulness judge receives the session brief (including background) and penalises restating goal or what-matters-most as if discussed, or reporting agreement the transcript does not show. Restating a background fact such as a role is not, by itself, a failure.
Human spot check: the judge may run on the same model family as the summariser, so a high faithfulness score is not a substitute for reading a sample of summaries against their transcripts periodically.
Score weights (v1)
Weights are renormalised over non-NULL components, so a missing component doesn’t zero the score.
Transcript quality / summary quality
health.rs splits the score into two sub-scores with the same weights, renormalised over the components present (NULL when both are missing). Formatting only counts towards health_score. The sub-scores are logged, not stored; they can be recomputed from the stored *_component columns.
Quality gate
evaluation/gate.rs runs for generated Session summaries when evaluation is enabled. It scores the candidate without storing it and retries once when should_retry holds:
health_score < QUALITY_GATE_THRESHOLD(75.0), andsummary_qualityis known and below 75, andtranscript_qualityis unknown or at least 75.
retry_temperature: a configured summarisation.temperature below 0.5 goes up by 0.3, otherwise down by 0.3; an unset temperature stays unset (the provider default already samples, and some models reject custom values). The higher health_score wins; a tie or a failed retry keeps the first candidate. The summarisation job’s tokens sum both generations; the evaluation job’s tokens sum both scorings. See Session summaries for what the user sees.
Configuration
Add asummary_evaluation block to config.yml:
model and embedding_model need to be set; Ollama connection details inherit from the rag Ollama config.
Admin API
GET /api/admin/system/summary-evaluation-config — read current config (merged: file + runtime overrides)
PUT /api/admin/system/summary-evaluation-config — update runtime overrides; sending an empty string for a field clears the override and reverts to rag inheritance.
POST /api/admin/system/summary-evaluation-backfill — re-score all existing summaries that have no evaluation yet, except Transcript summaries.
Token tracking
LLM token usage (prompt + completion) from both summarisation and judge jobs lands injobs.llm_prompt_tokens / jobs.llm_completion_tokens. Transcription jobs stamp jobs.inference_backend (scaleway/<model> or local/<model stem>) for backend-usage reporting.
Metabase SQL pack: api/docs/summary_health_metabase.md.
Brief-echo review flag
The admin Summary Health page shows recent evaluations withbrief_similarity and a Review badge when the share is high. Use it to spot summaries that may be paraphrasing the facilitator brief rather than the room.
Local SQL for the column (if your DB ran an earlier migration version):
Session practices
Sessions with Product Feature or Project Clarifier selected append practice-specific instructions tobuild_summary_prompt via SummaryContext.practice_block. The summariser still stores only the model narrative; work items and project candidate sections are composed at read time from session_work_items and session_project_candidates, so edits and dismissals stay in sync.
Practice outputs do not change summary health weights. Use the dedicated practice reference fixtures and eval-session-practices when measuring extraction quality (see Session practices).
Implementation
api/crates/summarisation/src/evaluation/