A voice-agent score is useful only when it shortens the path from a weak call to a verified improvement. This worked example follows one fictional appointment-booking conversation through VaaniEval's current scorecard: task completion, intent understanding, required-information capture, and human-like delivery.
The conversation is synthetic and deliberately small. It does not represent a customer or a benchmark. Its purpose is to show the reasoning chain a production team can reuse: define the expected outcome, inspect the evidence, identify the weakest behavior, make one targeted change, and evaluate new calls for regression.
The fictional call
Caller: I need to move my dental cleaning to Tuesday morning.
Agent: I can help with that. What time on Tuesday works for you?
Caller: Anything before eleven.
Agent: I am sorry, but I cannot access the scheduling system right now. I am sorry for the inconvenience. Please call us again later and we will help you.
Caller: So my existing appointment is still booked?
Agent: Please call us again later. Is there anything else I can help you with?
At a glance, the agent understood the request and remained polite. A generic completion dashboard might still classify the call as handled because the intent was detected and the conversation ended normally. The customer outcome tells a different story: no appointment was moved, the original booking was never confirmed, and no safe next step was created.
Step 1: define success before scoring
For this workflow, task completion should not mean that the agent discussed rescheduling. It should mean that the appointment was moved and the caller received confirmation. If the scheduling tool is unavailable, the approved fallback should preserve the existing booking, capture the caller's preferred window and contact details, and create a clear handoff for staff.
This definition matters because an evaluator cannot reliably infer a business outcome from conversational fluency alone. Teams should write the completion condition in operational language that a reviewer can verify from the transcript, tool result, or provider metadata.
Step 2: score the four current metrics
- Task completion — 20/100. The requested appointment change did not happen. The agent also failed to create an alternative path to completion. Evidence: “Please call us again later.”
- Intent understanding — 90/100. The agent correctly recognized that the caller wanted to move an existing cleaning to Tuesday before 11:00. The failure occurred after understanding, so rewriting intent detection would not address the main problem.
- Required-information capture — 35/100. The agent captured the preferred day and time window but did not confirm the caller's identity, the existing appointment, or a callback number. Those fields would be necessary for a human handoff.
- Human-like delivery — 55/100. The wording was grammatically clear, but the repeated apology and repeated “call us again later” response made the interaction feel scripted. This is the higher-is-better display score; reviewers should still use the transcript and available audio context before changing delivery behavior.
The scores alone are not the diagnosis. The rationale and conversation evidence explain why the result is weak. In this call, task completion is the weakest business behavior, while the high intent score shows where not to spend engineering time.
Step 3: map evidence to one targeted change
The primary fix belongs in the scheduling-failure branch, not in the general prompt. When the scheduling tool is unavailable, the agent should follow an explicit recovery sequence:
- State that the appointment has not been changed.
- Confirm whether the existing appointment remains active using available tool state or an approved uncertainty statement.
- Collect the minimum fields required for a staff callback.
- Repeat the requested date and time window back to the caller.
- Create the handoff through the configured workflow, or explain the exact next step if no handoff tool exists.
A revised response might say: “I could not reach the scheduling system, so I have not changed your appointment. I have Tuesday before 11:00 as your preferred window. May I confirm the best callback number so the scheduling team can follow up?”
That response does not pretend the original task succeeded. It makes the system state clear, preserves the caller's preference, and moves the conversation toward a recoverable outcome. Whether the agent can actually create the callback must be verified through the configured workflow rather than assumed from the wording.
Step 4: verify the change without moving the goalposts
After changing the failure branch, keep the original success definition and scorecard stable. Re-run a small representative set that includes a successful reschedule, a scheduling outage, an ambiguous time request, and a caller who declines a callback. Then review new production conversations as they arrive.
For each call, ask:
- Did task completion improve because the outcome improved, or only because the response sounded more confident?
- Did the agent retain correct intent understanding?
- Did it collect only the information required for the handoff?
- Did the new recovery wording reduce repetition without hiding uncertainty?
Compare evaluator judgments with human review on the calls that matter most. If reviewers consistently disagree with the scores, adjust the rubric or evidence requirements before changing thresholds. A threshold is useful only after the team agrees on what the metric means in its own workflow.
Add broader signals only when they change a decision
Unsupported claims, fallback quality, latency, interruptions, silence, and premature termination can all matter in voice-agent QA. They should be added when the team can define their evidence and connect a failure to an owner. Availability of operational signals also depends on the voice provider and the fields captured for each call.
Start with a small stable scorecard, then segment trends by agent, workflow, provider, and time period. Aggregates identify where performance changed; the individual conversation, rationale, and evidence explain what to fix.
The reusable score-to-fix loop
- Define the customer and business outcome before evaluating the call.
- Score a small set of distinct behaviors instead of relying on one opaque quality number.
- Attach a rationale and specific conversation evidence to every score.
- Use the weakest relevant behavior to choose one prompt, tool, workflow, or policy change.
- Evaluate representative and new production calls with the same definitions.
- Keep human review for uncertain, high-impact, or disputed results.