AI Voice Agent Stops, Then Resumes the Old Answer

A voice agent is answering a question. You interrupt it with a new request. It stops speaking, so the interruption seems to have worked. Then part of the old answer comes through the speaker anyway. That sequence matters: the first pause shows that playback stopped, but it does not show that the old answer lost every route back to playback.

We reproduced this failure in one voice integration. After an interruption, audio from the canceled response could still be admitted to the playback queue. We changed the application to accept audio only from the current response. Local tests then rejected late audio from the old response, both before and after cancellation was acknowledged, while allowing the next response to play. That is a tested repair for this case, not a diagnosis of every agent that repeats itself.

Why stopping the sound is only one step

A streaming voice agent can have several pieces of an answer in different places at once. Some audio is already playing. Some may be waiting in a local or provider queue. More may still be arriving from the response that produced it. Stopping the speaker handles what is audible now. Canceling the response tells the generator to stop. Neither action, on its own, proves that an old piece cannot later enter the playback queue.

This interruption is often called barge-in. The controls depend on the product and connection. OpenAI's Realtime documentation distinguishes client-managed WebSocket playback from server-managed WebRTC and SIP buffers. Amazon's Nova guidance describes stopping playback, clearing queued audio, and starting the new reply. These are examples of the responsibilities, not descriptions of the provider in our case.

A useful way to think about the failure is that each answer has an identity. Call the interrupted answer A and the next answer B. Once the caller interrupts A, an audio piece labeled A should no longer be allowed into playback. A piece labeled B should still be allowed. In our case, the missing check was at that admission point. The application stopped the sound and requested cancellation, but late A audio could still pass through.

Test the audible problem with two unrelated requests

You can establish the symptom without seeing code or private logs. In a test conversation with no customer information, ask the agent, “Give me three ways to prepare for a meeting.” Interrupt while it is still speaking: “Stop. Say the word blue.” Then listen for the full sequence, including what happens after any pause.

Write down three observations: Did the meeting advice stop? Did any meeting advice return afterward? Did the agent answer the new request? The two unrelated topics make an old answer easier to recognize. Repeat at another point in the first answer if the failure is intermittent, and note when you interrupted.

• It never pauses. Investigate whether the interruption was detected and whether playback was told to stop.
• It pauses, then returns to meeting advice. You have reproduced the symptom in this article. You have not yet proved where those words came from.
• It pauses and stays silent. The old answer may be gone, but the new request may also have been lost.
• It stops the advice and answers the new request. That is the expected behavior for this exercise. One good run does not establish reliability under other conditions.

A short recording or written sequence is useful evidence of what the listener experienced. It cannot, by itself, tell whether the old words were already queued, arrived late from A, or were newly generated by B using old conversation context.

Trace the answer that actually reached playback

If someone can inspect the integration, the question is: Which response owned the audio heard after the interruption? For one reproduction, line up the interruption event, playback stop, cancellation request and acknowledgement, incoming audio pieces, playback admission decisions, and creation of the next response. Record a timestamp, response identity, and admission decision at the playback boundary where possible. Keep customer speech and private identifiers out of a shared report.

That trace separates three different repairs:

1. A was already queued for playback. Inspect the queue or buffer clear path. A response ownership check on newly arriving audio cannot remove sound that is already queued elsewhere.
2. A arrived after interruption and was admitted. Reject audio whose response identity is no longer current. This was the path reproduced in our local case.
3. B produced the repeated words. The audio belongs to a new response, so investigate the conversation state and what the model believed the caller had heard. Blocking A cannot correct B's content.

The distinction prevents a costly wrong fix. Tuning the assistant's wording or interruption sensitivity will not repair an admission rule that accepts canceled audio. Conversely, an ownership guard will not explain every repeated phrase a new response generates.

Prove both sides of the repair

Our application retired A as the active playback owner when the interruption was committed, before sending the cancellation. It rejected later A audio at the playback boundary. Local tests delivered A audio while cancellation was pending and again after cancellation acknowledgement. Neither reached playback. They also confirmed that B could still play. Testing B matters because a broad rule that blocks all post-interruption audio would make the agent quiet but unusable.

For any proposed repair, use the same two-topic conversation through the real calling path as a separate check. The old meeting answer should stop and stay stopped, and the request for “blue” should receive a reply. Our local tests establish the behavior of the changed integration under those test conditions. They do not prove that every live call, network timing, or voice product behaves the same way.

If you need to hand the problem to someone else, share the two test requests, the point of interruption, what played in order, whether it repeats, and which records are available. You can also give this prompt to your preferred AI with anonymized details:

Help me investigate a voice agent that stops when interrupted, then may return to its old answer.

First request: [safe test request]
Interruption and new request: [words used and when]
What I heard in order: [pause, old answer, new answer, or silence]
Available evidence: [recording, event timeline, code, or only my observations]

Ask one question at a time. Separate what I observed from possible causes. Do not infer from repeated words alone that old audio played. Help me write a concise report with reproduction steps, expected and actual behavior, unanswered questions, and the next useful check. Do not ask for customer details, credentials, or private logs.

The practical standard is simple: after an interruption, the old answer must stay out of playback and the next valid answer must still get through. Support chatbots that also take phone calls need this standard too; our page on AI chatbot development for support teams shows one in use.