The gap
A strong model is not a reliable workflow
Model benchmarks answer a narrower question than the one production asks. Four reasons the scores don't carry over:
Where benchmark scores stop
- Different unit Benchmarks score isolated responses on curated inputs. A workflow is a multi-step process running on messy, shifting, incomplete ones.
- Compounding steps A workflow chains retrieval, tools, checks, and gates. Small per-step error rates compound across the chain — a model that looks strong in isolation can still sit inside a workflow that fails often.
- The action is the risk The expensive failure is rarely a badly worded answer. It is a wrong action, a policy breach, or an agent exceeding its authority — none of which a response benchmark measures.
- Environment drift Data sources, tools, policies, and upstream systems change underneath a workflow even when the model does not. Yesterday's score describes yesterday's environment.
So we evaluate the unit that actually carries the risk: the governed workflow, end to end.
The surface
What to evaluate — the full surface
A production-grade evaluation covers everything that can make a workflow fail in practice, not just whether the answer reads well.
Eleven dimensions of workflow quality
- Evidence quality Sources resolve, material claims are supported, gaps and conflicts are flagged rather than smoothed over.
- Output correctness The artifact is right by the rubric agreed for that workflow — not right in general.
- Policy compliance Every step stays inside declared policy, and deviations surface as exceptions.
- Tool behavior The right tools ran, in scope, and returned what the workflow assumed they did.
- Authority adherence The agent acted only at the authority level it holds — never above it.
- Action safety Consequential actions queued behind their gates; nothing left the boundary unapproved.
- Reliability The workflow completes consistently, and fails safely when it fails.
- Latency The output arrives in time to matter for the decision it serves.
- Cost Per-run economics are known and bounded, not discovered on the invoice.
- Human effort Review and correction time is counted as a cost of the workflow, not hidden outside it.
- Business usefulness The outcome the workflow exists to improve actually improved.
Configured per deployment. Evaluation harnesses, case sets, rubrics, and reporting dashboards are configured per deployment as part of an engagement — this is not a self-serve evaluation product. Which dimensions carry the most weight depends on the workflow's risk class and what it is allowed to touch.
The lifecycle
Production is earned, then re-earned
Every material change — model, prompt, skill, tool, data route, or authority — travels the same path before it runs with real consequences, and keeps traveling it afterward.
Six stages, no shortcuts
- Offline evaluation The candidate runs against versioned case sets — representative, edge, adversarial, and no-answer cases — scored by the rubric agreed for the workflow. Nothing touches production.
- Controlled trial It then runs in shadow, advisory, or approval-required mode with full telemetry. Authority is not widened for the trial; the point is to observe behavior, not to grant scope.
- Acceptance review Results are compared to the current baseline on quality, exceptions, review effort, reliability, and economics. A human owns the verdict.
- Explicit promotion Promotion is a recorded decision, not a merge: version, authority level, owner, rollback plan, and next review date are written down together.
- Monitoring and safe degradation In production the workflow is watched continuously. When evidence, confidence, policy, provider, or tool conditions fail, it abstains, narrows scope, requests review, or stops — it never improvises broader authority.
- Re-evaluation or rollback Drift, an incident, or any material change sends the workflow back through the gate. Rollback is a standing, rehearsed option — not an emergency invented under pressure.
The mechanics
The governance layer does the measuring
Evaluation is not bolted on. The same mechanisms that govern agents day to day — described in the Trust Center and on the Architecture page — are what make workflow evaluation possible.
Existing mechanisms, put to work substantiated
- Calibrated confidence Confidence is scored against tracked outcomes with Wilson confidence intervals — so evaluation reads a measured number, and a workflow that is confidently wrong shows up as exactly that.
- Graduation state machine The state machine that grants scope as agents demonstrate reliability is the enforcement arm of the lifecycle above: promotion decisions become granted scope through it, not around it.
- Gated authority Controlled trials are a native mode, not a special build — a candidate simply runs at draft, recommend, or approval-required authority until it earns more.
- Watchdog supervision The monitoring stage rides on the watchdog supervision that already observes the running system.
- Replayable audit trail Every evaluated run lands in hash-chained, replayable logs, so a failure is a record you open and diagnose — not an anecdote you argue about.
To see what one of those records contains, read the sample decision and execution record.
The consequence
Reliability is earned — and revocable
On this platform, reliability is not a property a workflow claims. It is a property the record shows.
One mechanism, both directions
Agents earn wider scope through tracked outcomes: sustained performance against the rubric moves them up the graduation ladder, and the authority they hold reflects the reliability they have demonstrated. The same mechanism runs in reverse. Degrading outcomes, failed conditions, or a broken assumption narrow scope, pull a workflow back behind approval gates, or take it out of production entirely. Nothing about the system requires you to keep trusting a workflow that has stopped deserving it — the evidence that granted authority is the evidence that revokes it.