> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kair.is/llms.txt
> Use this file to discover all available pages before exploring further.

# Summary Health Evaluation

> Automatic quality scoring for completed summaries — how the score is computed and how to configure it.

## Overview

When enabled, completed summarisations are scored into the `summary_evaluations` table with a 0–100 `health_score`:

* **Session summaries** (generated) are scored inline by the [quality gate](#quality-gate) before they are shown, and get exactly one evaluation row.
* **Document summaries** are scored asynchronously after completion, inside the summarisation worker off the shared `kair.summarisation` queue at priority 0, so summaries always drain first.
* **Transcript (per-track) summaries** are not evaluated. They are excluded from the backfill and from the admin stats (daily average, count and recent list); their existing rows stay in the table.

The scorer combines:

* **LLM judge** (`atla/selene-mini:q4_k_m` on Ollama)
* **Heuristics** (ASR confidence, formatting)
* **Embedding cosine** similarity between transcript and summary

Session summaries with a brief also get a **brief-echo share** (`brief_similarity`, 0–1): the proportion of summary sentences whose embedding similarity to goal / what-matters lines exceeds transcript-chunk similarity by the calibrated `BRIEF_LEAN_MARGIN` for that language triple. Background is excluded from this comparison. This is a **review flag only** — it does not enter `health_score` and does not bump `WEIGHTS_VERSION`.

Only the `de`/`de`/`de` triple is calibrated today (`BRIEF_LEAN_MARGIN = 0.05`). Other language pairs leave `brief_similarity` null until they have their own labelled set.

The faithfulness judge receives the session brief (including background) and penalises restating goal or what-matters-most as if discussed, or reporting agreement the transcript does not show. Restating a background fact such as a role is not, by itself, a failure.

**Human spot check:** the judge may run on the same model family as the summariser, so a high faithfulness score is not a substitute for reading a sample of summaries against their transcripts periodically.

## Score weights (v1)

| Component | Weight |
| - | - |
| Coverage | 0.30 |
| ASR confidence | 0.25 |
| Coherence | 0.20 |
| Faithfulness | 0.15 |
| Formatting | 0.10 |

Weights are renormalised over non-NULL components, so a missing component doesn't zero the score.

## Transcript quality / summary quality

`health.rs` splits the score into two sub-scores with the same weights, renormalised over the components present (NULL when both are missing). Formatting only counts towards `health_score`. The sub-scores are logged, not stored; they can be recomputed from the stored `*_component` columns.

| Sub-score | Components | Low means |
| - | - | - |
| `transcript_quality` | ASR confidence, coverage | Bad audio or a thin transcript |
| `summary_quality` | Coherence, faithfulness | The summariser did poorly on that input |

## Quality gate

`evaluation/gate.rs` runs for generated Session summaries when evaluation is enabled. It scores the candidate without storing it and retries once when `should_retry` holds:

* `health_score < QUALITY_GATE_THRESHOLD` (75.0), and
* `summary_quality` is known and below 75, and
* `transcript_quality` is unknown or at least 75.

Re-running the same transcript at the same settings scores the same, and the judge is the same model family as the summariser, so the retry uses `retry_temperature`: a configured `summarisation.temperature` below 0.5 goes up by 0.3, otherwise down by 0.3; an unset temperature stays unset (the provider default already samples, and some models reject custom values). The higher `health_score` wins; a tie or a failed retry keeps the first candidate. The summarisation job's tokens sum both generations; the evaluation job's tokens sum both scorings. See [Session summaries](https://docs.kair.is/platform/summaries.md#quality-gate) for what the user sees.

## Configuration

Add a `summary_evaluation` block to `config.yml`:

```yaml theme={null}
summary_evaluation:
  enabled: false           # opt-in; off by default
  model: "atla/selene-mini:q4_k_m"
  num_ctx: 4096
  transcript_char_budget: 8000   # truncate transcript to this before sending to judge
  # Optional overrides — fall back to resolved rag Ollama config if omitted
  ollama:
    base_url: "http://localhost:11434"
  embedding_model: "nomic-embed-text-v2-moe"
```

Normally only `model` and `embedding_model` need to be set; Ollama connection details inherit from the rag Ollama config.

## Admin API

`GET /api/admin/system/summary-evaluation-config` — read current config (merged: file + runtime overrides)
`PUT /api/admin/system/summary-evaluation-config` — update runtime overrides; sending an empty string for a field clears the override and reverts to rag inheritance.
`POST /api/admin/system/summary-evaluation-backfill` — re-score all existing summaries that have no evaluation yet, except Transcript summaries.

## Token tracking

LLM token usage (prompt + completion) from both summarisation and judge jobs lands in `jobs.llm_prompt_tokens` / `jobs.llm_completion_tokens`. Transcription jobs stamp `jobs.inference_backend` (`scaleway/<model>` or `local/<model stem>`) for backend-usage reporting.

Metabase SQL pack: `api/docs/summary_health_metabase.md`.

## Brief-echo review flag

The admin Summary Health page shows recent evaluations with `brief_similarity` and a **Review** badge when the share is high. Use it to spot summaries that may be paraphrasing the facilitator brief rather than the room.

Local SQL for the column (if your DB ran an earlier migration version):

```sql theme={null}
ALTER TABLE summary_evaluations
    ADD COLUMN IF NOT EXISTS brief_similarity REAL;
COMMENT ON COLUMN summary_evaluations.brief_similarity IS
    'Share of summary sentences (0-1) whose similarity to goal and what-matters lines exceeds transcript-chunk similarity by the calibrated BRIEF_LEAN_MARGIN for that language triple. Background is excluded. Null when the language pair is not calibrated. Review flag only; not part of health_score.';
```

## Session practices

Sessions with **Product Feature** or **Project Clarifier** selected append practice-specific instructions to `build_summary_prompt` via `SummaryContext.practice_block`. The summariser still stores only the model narrative; work items and project candidate sections are composed at **read time** from `session_work_items` and `session_project_candidates`, so edits and dismissals stay in sync.

Practice outputs do **not** change summary health weights. Use the dedicated practice reference fixtures and `eval-session-practices` when measuring extraction quality (see [Session practices](https://docs.kair.is/platform/session-practices)).

## Implementation

`api/crates/summarisation/src/evaluation/`


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.