What it answers
superdialog eval answers one question: does running your playbook on the
SuperDialog engine beat handing the same playbook to a raw LLM as a flat system
prompt?
It drives two modes over one dataset and scores both from the conversation
transcript only — never from engine internals — so the playbook and the
vanilla baseline face an identical rubric. Use it to justify adopting the
engine, to catch regressions, or to expose your playbook to any external
benchmark.
This is different from superdialog optimize and the persona-eval loop in the
API Reference. That
loop asks “is this playbook good enough, and how do I improve its prose?” — the
A/B harness here asks “is the playbook machinery earning its keep over a
plain prompt?” The two modes
| Mode | What runs |
|---|
playbook | The full Director + Talker checkpoint runtime loading your playbook |
vanilla | One raw LLM handed the playbook file as a single flat system prompt |
str in / str out, so a mode is just a factory that
returns a conversation endpoint. Endpoints ship for in-process playbook /
vanilla, a remote HTTP SuperDialog server, and any OpenAI-compatible model.
Headline metrics
| Metric | Kind | Reads |
|---|
task_success | LLM judge (0–1) | full transcript vs the case goal |
slot_accuracy | LLM judge (0–1) | transcript vs ground_truth_slots |
guardrail | LLM judge (hard gate) | each guardrail-probe reply |
efficiency | pure code | user turns + assistant latency p50/p95 |
token_cost | pure code | input tokens per assistant turn, with a director/talker split in playbook mode |
guardrail is a hard gate. If either mode complies with a probe attack,
that case’s composite score is zeroed and it counts toward the
guardrail_violation_rate — no matter how well it did on the other metrics.
efficiency and token_cost are pure code — no judge tokens, no added
latency — so every run includes them regardless of --metrics. The report
gains a Latency & tokens table (p50/p95, input tok/turn, director/talker
split, LLM calls/turn) and a framework score: zero unless quality is
perfect (task_success=1, slot_accuracy=1, guardrail clean), then higher
for lower latency and fewer tokens — the framework’s goal as one number.
Run it
Two phases: build the dataset once (offline, commit it), then A/B-run it.
# 1. Build <playbook>.evalcases.yaml — personas auto-generated, probes injected
superdialog eval gen-dataset --playbook spa.yaml --n-probes 8
# 2. A/B both modes → report.json (full) + report.md (headline table + drilldown)
superdialog eval run \
--playbook spa.yaml --dataset spa.evalcases.yaml \
--modes vanilla,playbook \
--agent-model openai/gpt-4.1-mini \
--judge-model openai/gpt-4.1-mini \
--metrics task_success,slot_accuracy,guardrail,efficiency \
--out ./eval-out
eval run flags:
| Flag | Default | Description |
|---|
--modes | vanilla,playbook | Which modes to compare |
--agent-model | openai/gpt-4.1-mini | Model both modes answer with |
--director-model / --talker-model | agent model | Per-role LLMs for playbook mode |
--judge-model | openai/gpt-4.1-mini | LLM that scores the transcripts |
--user-model | agent model | LLM that simulates the persona / caller |
--metrics | all four | Comma-separated headline metrics |
--repeats | 1 | Runs per case (average out variance) |
ground_truth_slots, and a list of probes.
Or run both phases in one shot with eval bench — it builds the dataset if
missing (--regen rebuilds, --personas seeds), A/Bs every --models entry
into its own report directory, and adds --max-turns to override each
persona’s turn budget:
superdialog eval bench --playbook spa.yaml \
--models openai/gpt-4o-mini --max-turns 20 --out ./eval-out
Serve it to an external benchmark
Expose the playbook as an OpenAI-compatible endpoint and grade it like any other
model:
superdialog eval serve --playbook spa.yaml --port 8000
# POST /v1/chat/completions → the playbook answers as the "model"
RAGAS is optional (and version-pinned)
The custom LLM judges produce every headline metric with no RAGAS installed.
RAGAS metrics are opt-in via the ragas extra:
pip install superdialog[ragas]
SuperDialog ships two RAGAS-based harnesses on incompatible RAGAS lines:
the A/B ragas extra (RAGAS 0.4.3) and the separate benchmark extra
(RAGAS 0.2.x, used by superdialog benchmark). They cannot co-install —
pick one extra per environment. With uv, they are declared conflicting so the
project still resolves; with pip, install only one.
Legacy session audit
The older single-session audit lives under the same command group:
superdialog eval flow --flow kyc.json --traversal session.json
eval
subcommand.
