CASE STUDY 06 — AI EVALUATION
Voice AI Evals
A hybrid framework for post-production collection calls that separates business outcome, borrower intent, conversation quality, technical performance and compliance—then traces every semantic failure back to transcript evidence.
OVERVIEW
A score is not enough. Teams need the failure story.
Voice AI collection calls can complete a task while still mishandling a borrower, exposing sensitive information or creating a poor spoken experience. This framework models those concerns separately instead of compressing them into one opaque success value.
Each call becomes an auditable evaluation with canonical outcomes, stage-level judgments, technical signals, normalized failure types and transcript evidence. Campaign aggregation then shows which failures happen most often and which ones are associated with unresolved calls.
ARCHITECTURE
Objective telemetry and semantic judgment, together.
Latency, silence and connectivity stay deterministic. Outcomes, objection handling and harmful language use LLM judgment. Critical compliance findings apply hard score caps so task completion cannot hide unsafe behavior.
EVALUATION MODEL
Different questions get different evaluators.
Outcome and intent
Separates borrower state from final resolution and distinguishes task resolution from collection success.
Stage-level quality
Opening, verification, discovery, objections, resolution, commitment, confirmation and closing are scored independently.
Voice telemetry
Deterministic rules handle latency, silence, interruptions, connectivity and metadata failures.
Independent safety path
Privacy disclosure, coercion, sensitive information and misleading claims are evaluated separately.
Auditable failures
Semantic findings include turn IDs, customer text, agent text and a plain-language explanation.
Root-cause analytics
Aggregations connect failure frequency and severity to resolution impact across a campaign.
DASHBOARD
From campaign health to one broken turn.
- Connection, resolution, collection-success and compliance-failure rates.
- Business outcome, payment intent and final resolution distributions.
- Pass, partial, fail and not-reached status for every conversation stage.
- Top failure modes ranked by frequency, severity and impact on resolution.
- Critical-call review with transcript evidence and compliance violations.
- Individual-call drill-down alongside a 10-call sample and 10,000-call synthetic demo.
VALIDATION
Evaluation quality is evaluated, too.
A manually labelled golden dataset checks outcome, payment intent, task resolution and expected key failures. The included sample reports 8/8 on all three classification checks, while tests cover evaluator behavior and scripts validate, aggregate and generate demo campaigns.
The repository clearly labels the 10,000-call campaign as synthetic demonstration data generated from evaluated call archetypes—not production traffic.
TECH STACK
Built for explainability and extension.
FastAPI, Pydantic, pydantic-settings and Uvicorn
Python deterministic policies plus OpenAI-powered semantic and compliance judges
Pandas aggregation, Streamlit dashboard and Plotly visualizations
Golden dataset validation, pytest coverage and evidence-grounded failure schemas
Dockerfile, environment-driven configuration and scriptable batch workflows