Companion Benchmark & Evidence Locker

Inside the AI Deal Desk

Empirical evidence behind "Inside the AI Deal Desk": 60 benchmark runs testing deterministic guardrails, hybrid retrieval, and critic loops.

ObservedMeasured Result InterpretationAnalytical Reading HypothesisUntested

1. Solution Overview

What problem does this solve?

Enterprise deals often stall in commercial review: discount authority, payment terms, margin floors, and non-standard legal clauses must be checked before a quote can move. Manual deal-desk review is slow, inconsistent and hard to scale. This application automates a first-pass review so RevOps and deal desks can focus on exceptions.

Note: Approval authority stays with leadership. Rather than automating sign-off, the agent delivers a decision-ready brief covering verified margins, required approval lanes, contract risk flags, and an executive summary.

What was built

One deal-desk agent pipeline, three configurations: version 1 baseline, version 2 guardrailed, and version 3 with a critic.

  • Code owns exact calculations: margin, penalties and approval lanes.
  • The AI Agent handles unstructured judgment: reading redlines and non-standard clauses, choosing risk flags, and drafting the executive memo.
1. Deal input
Details
Numerical:
  • Discount
  • Payment
  • Service hours
AI-required:
  • Contract redlines
Context:
  • Account identity
All versions
2. Deterministic calculations
Details
Calculates:
  • Margin
  • Approval lane
  • Working-capital penalties
Engine:
  • Python/SQLite rules
All versions
3. Retrieval + Deal Intake Agent (Multiple Versions)

3.1 Baseline

V1 Baseline Agent

Config 1 - Starting Agent

Details
Retrieval:
  • BM25 keyword search
Tools:
  • Unconstrained tool schemas
Not enabled:
  • Prompt caching
  • Evaluator-Optimizer

Pass rate: 85.0% Cost: Highest Latency: Highest

3.2 Guardrailed

V2 Guardrailed Agent

Config 2 - Guardrails + Caching

Details
Retrieval:
  • BM25 keyword + dense RRF
    (↑ pass rate)
Guardrails:
  • Must use exact risk flags
    (↑ pass rate)
  • Constrained tool inputs
    (↑ pass rate)
  • Stricter memo schema & prompt
    (↓ latency)
Enabled:
  • Prompt caching
    (↓ cost, ↓ latency)

Pass rate: 100.0% Cost: Lowest Latency: Lowest

3.3 Evaluator-Optimizer

V3 Evaluator-Optimizer Agent

Config 3 - Model critic review

Details
Additional review loop:
  • Critic reviews every memo
    (↑ cost, ↑ latency)
  • Conditional producer revision
    (extra turn only when needed)
Observed:
  • Revised 1/20 deals
    (no pass-rate lift, weaker ROI)

Pass rate: 100.0% Cost: Higher Latency: Higher

4. Human-ready output
Details
From code:
  • Lane
  • Margin
From AI:
  • Risk flags
  • Memo
All versions

Pass Rate: Percentage of test scenarios in which the system correctly calculated deal economics, assigned the required approval route and identified all expected contractual and policy risks. All three criteria must be met for a scenario to pass.

Key findings Version 2

Four headline metrics from the Version 2 configuration on the 20-scenario golden suite:

Important limitations

2. Experiment Results

Verifies what changed across Config 1 (v1), Config 2 (v2), and Config 3 (v3) on the same 20 scenarios. Figures are measured API-level agent costs and latency, not full deal-desk turnaround with human queues.

Version comparison

Median (p50) = typical deal. 90th percentile (p90) = slower tail. Deterministic pass rate = share of cases where approval lane, margin and risk-flag recall all matched golden labels.

Primary visualisations

Quality versus cost

Measured: deterministic pass rate vs median API cost per deal.
Bubble size represents median processing time.

Risk-flag precision and recall

Measured: mean precision (correctness of raised flags) and recall (coverage of required flags) by version.

Cost and processing time

Measured: median API cost (input tokens, output tokens, cache) and median end-to-end agent latency by version.

Scenario-level pass rates

Measured: deterministic pass rate by difficulty band. Small bands show exact counts (k/n).

Interpretation

3. Evaluation Methodology

Verifies how pass rates, risk-flag scores, and confidence bounds were calculated, including why 60/60 completed runs is not a 100% evaluation pass rate.

Statistical confidence & paired comparisons
AI qualitative grading (LLM-as-a-Judge)
Known evaluation limitations

4. Technical Deep Dive

Inspect the implementation evidence behind the article: architecture, retrieval, prompt caching, critic overhead, failure taxonomy, and proposed next experiments.

A. System architecture
B. Agent design & version differences
C. Retrieval & policy grounding
D. Cost, tokens & latency detail
Cost distribution
Token mix & caching

Monthly API-cost projection

Estimate only: observed evaluation agent costs × volume. Not guaranteed production spend.

E. Failure analysis

Scenario-level paired comparison

F. Recommended next experiments Proposed
5. Case Explorer & Traceability

Open the exact runs cited in the article (v1 taxonomy misses, margin traps, and the v3 GE-016 critic revision) without leaving the evidence locker.

VersionCaseAccountDifficultyStatus LaneMarginFlag F1JudgeAPI costRevised

Traceability