3.1 Baseline
V1 Baseline Agent
Config 1 - Starting Agent
Details
- BM25 keyword search
- Unconstrained tool schemas
- Prompt caching
- Evaluator-Optimizer
Pass rate: 85.0% Cost: Highest Latency: Highest
Companion Benchmark & Evidence Locker
Empirical evidence behind "Inside the AI Deal Desk": 60 benchmark runs testing deterministic guardrails, hybrid retrieval, and critic loops.
Enterprise deals often stall in commercial review: discount authority, payment terms, margin floors, and non-standard legal clauses must be checked before a quote can move. Manual deal-desk review is slow, inconsistent and hard to scale. This application automates a first-pass review so RevOps and deal desks can focus on exceptions.
Note: Approval authority stays with leadership. Rather than automating sign-off, the agent delivers a decision-ready brief covering verified margins, required approval lanes, contract risk flags, and an executive summary.
One deal-desk agent pipeline, three configurations: version 1 baseline, version 2 guardrailed, and version 3 with a critic.
3.1 Baseline
V1 Baseline Agent
Config 1 - Starting Agent
Pass rate: 85.0% Cost: Highest Latency: Highest
3.2 Guardrailed
V2 Guardrailed Agent
Config 2 - Guardrails + Caching
Pass rate: 100.0% Cost: Lowest Latency: Lowest
3.3 Evaluator-Optimizer
V3 Evaluator-Optimizer Agent
Config 3 - Model critic review
Pass rate: 100.0% Cost: Higher Latency: Higher
Pass Rate: Percentage of test scenarios in which the system correctly calculated deal economics, assigned the required approval route and identified all expected contractual and policy risks. All three criteria must be met for a scenario to pass.
Four headline metrics from the Version 2 configuration on the 20-scenario golden suite:
Verifies what changed across Config 1 (v1), Config 2 (v2), and Config 3 (v3) on the same 20 scenarios. Figures are measured API-level agent costs and latency, not full deal-desk turnaround with human queues.
Median (p50) = typical deal. 90th percentile (p90) = slower tail. Deterministic pass rate = share of cases where approval lane, margin and risk-flag recall all matched golden labels.
Measured: deterministic pass rate vs median API cost per deal.
Bubble size represents median processing time.
Measured: mean precision (correctness of raised flags) and recall (coverage of required flags) by version.
Measured: median API cost (input tokens, output tokens, cache) and median end-to-end agent latency by version.
Measured: deterministic pass rate by difficulty band. Small bands show exact counts (k/n).
Verifies how pass rates, risk-flag scores, and confidence bounds were calculated, including why 60/60 completed runs is not a 100% evaluation pass rate.
Inspect the implementation evidence behind the article: architecture, retrieval, prompt caching, critic overhead, failure taxonomy, and proposed next experiments.
Estimate only: observed evaluation agent costs × volume. Not guaranteed production spend.
Open the exact runs cited in the article (v1 taxonomy misses, margin traps, and the v3 GE-016 critic revision) without leaving the evidence locker.
| Version | Case | Account | Difficulty | Status | Lane | Margin | Flag F1 | Judge | API cost | Revised |
|---|