Overview
The AI playground measures the AI pipeline on invented workspaces instead of real sessions. Each scenario is a workspace (or several) with ordered sessions of transcript and a list of expectations: what a good summary must mention, traps it must not fall into, yes/no questions for a judge model, and, for the HiRAG v2 topic history, what should happen to each topic from one session to the next. A run sends every session through a pipeline variant, scores the result and writes a JSON run report. Two reports compare check by check, so a prompt or pipeline change shows as regressions and improvements rather than impressions.- Before: record a baseline run on the current pipeline.
- During: every report keeps a trace of each stage (rendered transcript, system prompt, raw model reply, judge answers, health scores), so you can see where a session went wrong, not just that it did.
- After: run again after the change and compare.
api/crates/playground.
Running it
Fromapi/:
run options:
Reports are written to
api/crates/playground/runs/ and can be opened in the admin viewer at Admin → AI Playground (/admin/playground). Load one report to inspect it, or a second one to compare before and after.
Configuration
The playground reads the same YAML file as the workers (--config, then CONFIG_FILE, then config.yml) and parses only the AI blocks:
app_config overrides stored in the database do not apply. Replay needs no config at all.
Record and replay
A cassette (cassettes/<variant>.jsonl) stores one line per model call, keyed by a hash of the call kind, the sample index and the exact prompt. Any change to the prompt, the transcript or the style override misses the cassette, and replay reports the session as failed instead of silently calling the model. record reuses a reply the same model already recorded and calls the model only for the rest, so a new variant that shares a step (hirag-v2 summarises like summary-v2) pays only for its new calls. Recording with another model replaces the earlier reply. Cassettes hold model output on invented scenarios only and can be committed, so a baseline replays anywhere.
Variants
HiRAG v2 variants (structured topic paragraphs, the merge step, the topic store) are added slice by slice; see
agent-os/specs/2026-10-17-1200-hirag-v2-topic-history/. Checks a variant cannot answer are reported as not_applicable, so a variant is scored only on what it can answer.
Checks
Topic decisions
A variant reports decisions with its ownstrand_key and never sees topic ids. The harness assigns each decision to the workspace topic whose anchors appear earliest in its topic sentence (the longest anchor wins at the same position, then the earlier-declared topic). Only when the sentence has no anchor does it fall back to the free text, and the report records which one matched; v1’s entity descriptions name neighbouring topics, so a text match credits the topic’s outcome but never defines a strand or counts as a wrong merge. A decision no topic matches is an extra topic. Strands keep the first topic their sentence mapped them to, so continuing a strand that belongs to another topic is a wrong merge (a wrong merge credits no outcome, whichever it claimed), and new for a topic that already had a strand from an earlier session is fragmentation. Several decisions about one topic reduce to one outcome: person-routed (wording, identity, split), then disagreement, then fast-forward onto the topic’s own strand, then new, else absent.
Run totals then carry topic: a confusion matrix (expected × actual), recall per outcome (printed as n=k below five expectations), disagreement_swallowed (target zero; an increase fails compare --fail-on-regression), wrong_merge and wrong_merge_rate (wrong merges per continuing decision mapped through its sentence, because variants emit very different numbers of continuing decisions; a rate above 10% fails the gate, since a moderator moves a wrongly merged bullet in one click), disagreement_disconnected, fragmentation, restated_in_store, person load and extra topics per session. compare reports topic counts per sample, so runs with different --samples compare, and marks per-check changes as noisy when either run has a single sample.
Trap kinds name the failure: brief_only, earlier_session, later_session (a late upload seeing a session processed before it), prompt_injection, invented, false_consensus, wrong_number, other_workspace.
Scenarios
Scenarios are YAML files inapi/crates/playground/scenarios/; the schema is src/scenario.rs. A session’s transcript is either inline segments (at, speaker, optional track, text) or an import of the synthetic Westerwald assembly fixture (civic_markdown). Both render to the merged-session format the worker uses: [mm:ss] Track / Speaker: text.
Rules for authors:
- Invented content only. No real people, organisations, emails, phone or account numbers.
validateflags emails and long digit runs. - Every expectation must be grounded in the transcript you wrote. Prefer word stems and give a few variants.
- Mark a topic
absentonly when the session does not discuss it at all. - Run
validatebefore committing; the crate’s tests fail on an invalid scenario.
UI edge cases
The command centre dev panel (local dev only) can inject summary edge cases into the session page: a practice appendix, v2 topic paragraphs, Markdown stress, HTML injection, empty sections, a thin session and a very long summary. They live inui/src/lib/stores/mockSummaryEdgeCases.ts.