> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kair.is/llms.txt
> Use this file to discover all available pages before exploring further.

# AI Playground

> Run invented scenarios through the summary pipeline, score every session against declared expectations and compare runs before and after a change.

## Overview

The AI playground measures the AI pipeline on **invented** workspaces instead of real sessions. Each scenario is a workspace (or several) with ordered sessions of transcript and a list of expectations: what a good summary must mention, traps it must not fall into, yes/no questions for a judge model, and, for the HiRAG v2 topic history, what should happen to each topic from one session to the next.

A run sends every session through a pipeline **variant**, scores the result and writes a JSON **run report**. Two reports compare check by check, so a prompt or pipeline change shows as regressions and improvements rather than impressions.

* **Before**: record a baseline run on the current pipeline.
* **During**: every report keeps a trace of each stage (rendered transcript, system prompt, raw model reply, judge answers, health scores), so you can see where a session went wrong, not just that it did.
* **After**: run again after the change and compare.

Nothing touches the database, queues or Neo4j. The crate lives in `api/crates/playground`.

## Running it

From `api/`:

```bash theme={null}
# Check the scenario files (schema, references, privacy lint)
cargo run -p kair-voice-playground -- validate

# List scenarios and print what the summariser sees for one session
cargo run -p kair-voice-playground -- list
cargo run -p kair-voice-playground -- transcript spielplatz-verlegung s2

# Record a baseline with the configured model, including judge assertions
CONFIG_FILE=../config.local.yml cargo run -p kair-voice-playground -- \
  run --backend record --assertions --label baseline

# Replay it offline (no model calls, no cost); exits 1 when any session failed
cargo run -p kair-voice-playground -- run --backend replay --assertions --label replay

# Re-score a stored report's topic decisions with the current rules (no model, no cassette)
cargo run -p kair-voice-playground -- rescore runs/<report>.json

# Compare two reports; exit 1 on a regressed or vanished check, more failed sessions, topic totals
# that disappear, more swallowed disagreements, or a wrong-merge rate above 10%
cargo run -p kair-voice-playground -- compare runs/<before>.json runs/<after>.json --fail-on-regression
```

`run` options:

| Option | Default | Effect |
| - | - | - |
| `--scenario <id>` | all | Repeatable |
| `--tag <tag>` | all | Repeatable, e.g. `disagreement`, `prompt_injection` |
| `--variant` | `summary-v2` | Pipeline under test: `summary-v2` or `hirag-v2` |
| `--backend` | `live` | `live`, `record` (cassette first, live for the rest) or `replay` (cassette only) |
| `--cassette <path>` | `cassettes/summary-v1.jsonl` | Recorded model replies, shared by all variants |
| `--samples <n>` | `1` | Runs per session; check results become pass rates |
| `--prompt-file <path>` | — | Replaces the default conversation style, like a customer prompt override |
| `--assertions` | off | Ask the judge model the scenarios' yes/no questions |
| `--health` | off | Score with the production Summary Health judge (live Ollama calls) |
| `--out <dir>` | `runs/` | Report directory (git-ignored) |

Reports are written to `api/crates/playground/runs/` and can be opened in the admin viewer at **Admin → AI Playground** (`/admin/playground`). Load one report to inspect it, or a second one to compare before and after.

## Configuration

The playground reads the same YAML file as the workers (`--config`, then `CONFIG_FILE`, then `config.yml`) and parses only the AI blocks:

```yaml theme={null}
ai_inference:          # credentials, shared with the workers
  anthropic:
    api_key: ...
summarisation:         # the model under test
  agent_client: anthropic
  model: <model id>
  temperature: 0.0
summary_evaluation:    # only for --health
  model: atla/selene-mini:q4_k_m
playground:
  judge:               # optional; defaults to the summarisation model
    agent_client: anthropic
    model: <model id>
    temperature: 0.0
```

`app_config` overrides stored in the database do not apply. Replay needs no config at all.

## Record and replay

A cassette (`cassettes/<variant>.jsonl`) stores one line per model call, keyed by a hash of the call kind, the sample index and the exact prompt. Any change to the prompt, the transcript or the style override misses the cassette, and replay reports the session as failed instead of silently calling the model. `record` reuses a reply the same model already recorded and calls the model only for the rest, so a new variant that shares a step (hirag-v2 summarises like summary-v2) pays only for its new calls. Recording with another model replaces the earlier reply. Cassettes hold model output on invented scenarios only and can be committed, so a baseline replays anywhere.

## Variants

| Variant | What runs |
| - | - |
| `summary-v2` | The production path ([Session Summaries](https://docs.kair.is/platform/summaries.md)): per-line loop collapse and gap note, `build_summary_messages`, structured paragraphs parsed (one retry), anonymised against the transcript's speaker labels and rendered to Markdown. Sessions are independent. |
| `hirag-v2` | summary-v2, then the [merge step](https://docs.kair.is/platform/topic-history.md) with the worker's prompt, parser and shortlist against an in-memory topic store; one topic decision per paragraph. Merge calls are recorded as `merge`; the `topic_store` stage shows candidates, the raw reply and every decision. |

HiRAG v2 variants (structured topic paragraphs, the merge step, the topic store) are added slice by slice; see `agent-os/specs/2026-10-17-1200-hirag-v2-topic-history/`. Checks a variant cannot answer are reported as `not_applicable`, so a variant is scored only on what it can answer.

## Checks

| Check | Source | Passes when |
| - | - | - |
| `must_mention` | `expect.must_mention` | Any listed phrase appears (case, accents and ß folded; stems match inflections) |
| `trap` | `expect.traps` | No listed phrase appears. `scope: before_closing` ignores the closing brief section; `allow_negated` ignores hits in a negated sentence; `unless_sentence_has` ignores hits whose sentence frames them ("as a joke", "a note read out") |
| `assertion` | `expect.assertions` | The judge model answers the yes/no question as expected (`--assertions`). The answer is the first word of the reply's last line (else of the reply); anything else is an error, which counts against the pass rate |
| `words` | `expect.words` | Word count is within bounds |
| `closing_section` | `expect.closing_section` | The "purpose of the session" heading is present (or absent) as expected |
| `language` | always | A stopword vote matches the session language (de, en, fr, es, nl) |
| `no_html` | always | The summary contains no HTML tags |
| `topic_present` / `topic_absent` | `expect.topics` | The topic's anchors appear in sessions that discussed it, and stay out of sessions that did not |
| `topic_outcome` | `expect.topics` | The variant's topic decisions for the topic add up to the expected outcome (new, fast\_forward, wording\_conflict, identity\_conflict, disagreement, split, absent). Topic-store variants only |
| `link` | `expect.links` | The variant's cross-workspace link proposals (production rule, organisation and opt-in) connect the two topics, or do not. `hirag-v2` only; scored once per sample |
| `erasure` | `erasures` | Erased sessions leave no trace. Not applicable yet |

### Topic decisions

A variant reports decisions with its own `strand_key` and never sees topic ids. The harness assigns each decision to the workspace topic whose anchors appear earliest in its topic sentence (the longest anchor wins at the same position, then the earlier-declared topic). Only when the sentence has no anchor does it fall back to the free text, and the report records which one matched; v1's entity descriptions name neighbouring topics, so a text match credits the topic's outcome but never defines a strand or counts as a wrong merge. A decision no topic matches is an **extra topic**. Strands keep the first topic their sentence mapped them to, so continuing a strand that belongs to another topic is a **wrong merge** (a wrong merge credits no outcome, whichever it claimed), and `new` for a topic that already had a strand from an earlier session is **fragmentation**. Several decisions about one topic reduce to one outcome: person-routed (wording, identity, split), then disagreement, then fast-forward onto the topic's own strand, then new, else absent.

Run totals then carry `topic`: a confusion matrix (expected × actual), recall per outcome (printed as `n=k` below five expectations), `disagreement_swallowed` (target zero; an increase fails `compare --fail-on-regression`), `wrong_merge` and `wrong_merge_rate` (wrong merges per continuing decision mapped through its sentence, because variants emit very different numbers of continuing decisions; a rate above 10% fails the gate, since a moderator moves a wrongly merged bullet in one click), `disagreement_disconnected`, `fragmentation`, `restated_in_store`, person load and extra topics per session. `compare` reports topic counts per sample, so runs with different `--samples` compare, and marks per-check changes as noisy when either run has a single sample.

Trap kinds name the failure: `brief_only`, `earlier_session`, `later_session` (a late upload seeing a session processed before it), `prompt_injection`, `invented`, `false_consensus`, `wrong_number`, `other_workspace`.

## Scenarios

Scenarios are YAML files in `api/crates/playground/scenarios/`; the schema is `src/scenario.rs`. A session's transcript is either inline `segments` (`at`, `speaker`, optional `track`, `text`) or an `import` of the synthetic Westerwald assembly fixture (`civic_markdown`). Both render to the merged-session format the worker uses: `[mm:ss] Track / Speaker: text`.

Rules for authors:

* **Invented content only.** No real people, organisations, emails, phone or account numbers. `validate` flags emails and long digit runs.
* Every expectation must be grounded in the transcript you wrote. Prefer word stems and give a few variants.
* Mark a topic `absent` only when the session does not discuss it at all.
* Run `validate` before committing; the crate's tests fail on an invalid scenario.

## UI edge cases

The command centre dev panel (local dev only) can inject summary edge cases into the session page: a practice appendix, v2 topic paragraphs, Markdown stress, HTML injection, empty sections, a thin session and a very long summary. They live in `ui/src/lib/stores/mockSummaryEdgeCases.ts`.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.