How to Run a Live Traffic Evaluation of AI Voice Agent Platforms: The A/B Test Most Vendors Hope You Skip

Arkadas Kilic 9 min read · 2026-08-06

Every AI voice agent vendor will show you a demo. Carefully scripted, acoustically clean, with a cooperative caller who says exactly the right thing. What they will not volunteer is a live traffic split test against your real call volume, your real customers, and your real edge cases.

That test is the only one that matters.

This guide walks you through how to design and execute a production-grade A/B evaluation of AI voice agent platforms. The goal is not to find the vendor with the best pitch deck. The goal is to find the platform that performs under load, in your environment, against your compliance requirements, and with your actual customers on the line.

Why Vendor Demos Are Structurally Misleading

Vendor demos optimize for the 80th-percentile caller. They are built on:

• Clean audio from a studio microphone
• Cooperative testers who pause correctly and speak in full sentences
• A narrow set of intents the model was tuned to handle
• No background noise, no accents, no emotional escalation
• No integration with your actual CRM, policy engine, or compliance guardrails

In a regulated industry, the 20th-percentile caller is the one who creates liability. The elderly caller with a soft voice on a landline. The frustrated customer who interrupts mid-sentence. The caller asking a question that sits at the edge of what your AI is permitted to answer.

A live traffic A/B test surfaces all of that. Which is exactly why most vendors steer you toward sandbox evaluations instead.

Step 1: Define Your Evaluation Metrics Before You Touch Any Platform

The single most common mistake in platform evaluations is letting the vendor define success. Fix this before you start.

Set your metrics in three categories:

• Self-service containment rate: percentage of calls fully resolved by the AI without transfer
• First-call resolution rate: percentage of callers who do not call back within 48 hours
• Escalation accuracy: of calls that do escalate, what percentage were legitimately complex vs. the AI failing

• Average handle time for AI-handled calls vs. agent-handled calls
• Post-call survey score (if you have IVR survey infrastructure)
• Caller dropout rate: percentage of callers who hang up during the AI interaction before resolution or transfer
• Barge-in handling accuracy: how often the AI correctly processes an interruption vs. ignoring it or misrouting

• Prohibited topic containment: zero tolerance for the AI providing advice outside its defined scope
• PII handling audit: confirmation that no PII is logged in unencrypted transcript stores
• Silence and dead-air rate: percentage of interactions with more than 3 seconds of unintended silence

Document these metrics, assign owners, and lock them before any vendor conversation. This prevents scope creep and post-hoc rationalization.

Step 2: Design the Traffic Split Architecture

For a valid A/B test you need true randomization, not time-based segmentation. Routing calls by time of day introduces confounds because call intent and caller demographics shift across the day.

1. Create a single inbound phone number as the entry point

2. Use a Contact Flow with a random percentage split block: 50% to Platform A, 50% to Platform B

3. Tag each contact record with the platform assignment at the point of routing using contact attributes

4. Ensure both platforms receive identical caller context: ANI, DNIS, queue metadata, and any pre-authenticated account data

5. Route escalations from both platforms to the same agent queue so agent handling time is not a confound

For statistical significance at 95% confidence with a 5% minimum detectable effect on containment rate, you need approximately 385 calls per variant. In practice, run for a minimum of 500 calls per variant to account for variance in call mix. For most mid-size contact centers handling 2,000 to 5,000 calls per day, a 10% traffic split gives you enough volume within 3 to 5 business days.

Do not run the test on a Monday or the day after a major holiday. Call mix is not representative.

Step 3: Build the Baseline Before You Route Live Traffic

You cannot measure improvement without a baseline. Before splitting any live traffic, pull 30 days of historical data from your current environment:

• Average containment rate by intent category
• Average handle time by call type
• Transfer rate by time of day and day of week
• Agent-scored quality metrics if available
• CSAT or NPS scores tied to call records

If your current environment is a legacy IVR with no AI, your baseline is the transfer rate and handle time of that IVR. If you are replacing an existing AI platform, your baseline is that platform's performance on the same metrics you defined in Step 1.

This baseline becomes your floor. Any platform that does not beat the baseline on containment rate and CSAT within the test window is not production-ready for your environment.

Step 4: Configure Both Platforms to Production Parity

This is where most evaluations fail. Teams spend 3 weeks configuring Platform A and 3 days configuring Platform B. The result is a test of configuration effort, not platform capability.

Production parity means:

• Both platforms have access to the same knowledge base or FAQ corpus
• Both platforms are integrated with the same backend systems: CRM lookup, account balance API, policy retrieval
• Both platforms have the same fallback and escalation logic defined
• Both platforms are operating under the same compliance constraints: prohibited topics, required disclosures, escalation triggers for distressed callers
• Both platforms have been tested against your top 10 intent categories with at least 20 test calls each before live traffic begins

Document the configuration effort in hours for each platform. This is a real cost that belongs in your total cost of ownership calculation.

Step 5: Instrument Everything Before Day One

You cannot trust vendor-provided analytics for a comparative evaluation. Both platforms will report their own metrics in ways that favor their architecture.

Build your own measurement layer:

• Export call records from Amazon Connect Contact Lens or your CCaaS analytics layer for every contact in the test
• Tag records with platform assignment, intent category (from the AI or from post-call classification), outcome (contained, transferred, dropped), and handle time
• Store transcripts in a centralized S3 bucket with consistent naming and tagging, not in vendor-specific storage
• Run a compliance audit sample: pull 50 random transcripts from each platform weekly and score them against your prohibited topics list and required disclosure checklist
• Set up a real-time dashboard in Amazon QuickSight or your BI tool of choice that pulls from your own data, not the vendor portal

If a vendor tells you their analytics portal is sufficient and you do not need your own instrumentation, that is a red flag.

Step 6: Run the Test for a Minimum of Two Full Business Weeks

One week is not enough. Call mix varies by day of week. Week one often shows artificially high containment rates because callers are encountering the new experience and cooperating more than usual. Week two reflects normalized behavior.

During the test period:

• Do not make configuration changes to either platform. Changes invalidate the comparison.
• Monitor dropout rates daily. If either platform shows a dropout rate above 15% in the first 48 hours, pause that arm and investigate before continuing.
• Hold a daily 15-minute review of flagged transcripts: calls that escalated unexpectedly, calls with long silence periods, and calls where the AI provided an ambiguous or potentially non-compliant response.
• Track agent feedback. Agents handling escalations from both platforms will have direct observations about call quality and caller frustration levels that metrics alone will not capture.

Step 7: Score the Results Against Your Pre-Defined Metrics

At the end of the test period, score both platforms against every metric you defined in Step 1. Use a simple weighted scorecard:

Weights should reflect your business priorities. In a regulated industry, compliance audit pass rate often carries more weight than containment rate. A 5% improvement in containment that comes with a 2% compliance failure rate is not a win.

Also score the operational dimension:

• Configuration effort in hours
• Time to first production-ready deployment
• Quality of vendor support during the test period
• Clarity and completeness of documentation
• AWS native integration depth vs. API wrapper complexity

What Real Numbers Look Like in a Well-Run Evaluation

Based on production deployments we have run across financial services, insurance, and healthcare contact centers:

• A well-configured AI voice agent on a high-volume intent category (balance inquiry, appointment scheduling, claim status) should achieve 65% to 80% self-service containment on that intent within the first 30 days of production.
• Caller dropout rates above 12% on a single intent category signal a prompt design or latency problem that requires investigation before scaling.
• Compliance audit pass rates below 98% on prohibited topics are not acceptable in regulated industries. The threshold is not 95%. It is not 97%. It is 98% or higher, and the failures require root cause analysis, not just retraining.
• Latency matters more than most teams expect. AI response latency above 1.5 seconds on a turn-by-turn basis measurably increases caller frustration and barge-in frequency. Measure it.

The Questions Vendors Hope You Do Not Ask During the Evaluation

Ask these directly, in writing, before you start:

1. What is your platform's median response latency at 500 concurrent calls? At 2,000?

2. How are transcripts stored, for how long, in what regions, and who has access?

3. What happens when your platform cannot classify an intent? Walk me through the exact fallback path.

4. How does your platform handle a caller who says something that triggers a mandatory escalation under our compliance policy?

5. What is your data processing agreement structure and how does it map to our regulatory obligations?

6. If we need to make a configuration change at 11pm on a Sunday, what is the support path?

The answers to these questions will differentiate platforms faster than any demo.

Why AWS Native Architecture Changes the Evaluation Criteria

If your contact center runs on Amazon Connect, the evaluation criteria shift in important ways. A platform built natively on AWS is not the same as a platform that connects to Amazon Connect via an API.

Native AWS architecture means:

• Data stays in your AWS account and your VPC. It does not traverse a third-party cloud.
• IAM roles govern access, not vendor-managed credentials.
• CloudWatch gives you observability across the full stack without a separate vendor monitoring layer.
• Lambda, DynamoDB, and your existing AWS services integrate directly without middleware.
• Compliance controls you have already implemented at the AWS account level extend to the AI layer without renegotiation.

In a live traffic evaluation, test this explicitly. Ask both platforms to demonstrate a full data flow diagram showing exactly where call audio, transcripts, and extracted data reside at each step. If any step in that diagram shows data leaving your AWS environment without your explicit control, document it and factor it into your compliance scoring.

The Evaluation Most Teams Skip Because It Feels Slow

The live traffic A/B test takes 3 to 4 weeks from setup to final scorecard. Most teams skip it because they are under pressure to ship and because vendors make sandbox demos feel sufficient.

The cost of skipping it is a production deployment that underperforms, a compliance incident that was predictable, and a platform migration 12 months later that costs more than the evaluation would have.

We build AI voice agents that go into production in 4 to 6 weeks. We run these evaluations as part of the build process, not as a separate consulting engagement. The evaluation is the foundation of the deployment, not a gate before it.

If you are selecting an AI voice agent platform for a regulated contact center and you want to run this test with a team that has done it before, we should talk.