Evidence at a glance
The Core Conflict: Agent Voice Interference and Lost User Short Replies
In traditional voice agent-to-user interactions, Speech-to-Text (STT) models typically process only the user's speech. However, when an agent needs to utter prompts, ask questions, or provide information, its own voice can also be captured by the STT system. This leads to two primary issues: First, leakage or echo of the agent's own voice can be misidentified as user input. This is particularly problematic when processing short, rapid user replies, causing the agent to mishear or miss crucial information, such as mistaking 'a tow' for 'a tour'.
Second, audio played by the agent can feed back through speakers or laptops into the microphone, creating echo. If the STT model transcribes these echoes without distinction, the agent might mistakenly believe the user is conversing with it, leading to the agent interrupting itself or generating a confusing interaction flow. These issues significantly degrade the fluency and accuracy of voice assistants.
Soniox STT v6's New Mechanism: Agent Text Input
Soniox STT v6 introduces a new feature called "Agent Text Input" designed to directly address the aforementioned conflicts. By setting a simple flag (`"enable_agent_text_input": true`) in the request, developers can activate this functionality. Unlike processing only user speech, Agent Text Input allows developers to send the agent's upcoming or current spoken text in real-time on the same WebSocket stream as the user's audio. This enables the STT model to be aware of what the agent is saying.
The core of this feature lies in "timing": the agent's text data itself does not carry timestamps. The STT model infers its temporal significance based on its arrival position within the audio stream. For accuracy, Soniox recommends sending the agent's text 500 milliseconds to 1 second before playback, paced by Text-to-Speech (TTS) timestamps. When a user attempts to interrupt the agent (barge-in), the system should mark the already played portion as ended, discard the unplayed remainder of the agent's utterance, and prioritize the user's speech. Crucially, the agent's text content is not included in the final transcription; it's used solely for internal model processing.
Echo Filtering: The Side Benefit of Agent Text Input
A critical and powerful side benefit of Agent Text Input is its built-in "agent voice echo filtering" mechanism. Once Agent Text Input is enabled, the STT v6 model can leverage the received agent text information to more accurately distinguish agent echo mixed within the user's speech.
Specifically, the model actively identifies and filters out audio segments that match what the agent is currently speaking. This means that even if residual echoes are generated by devices (like speakers or laptops), the STT system can effectively strip them from the main user audio stream, preventing these echoes from being mistakenly transcribed as user utterances. This mechanism is not intended to replace existing Acoustic Echo Cancellation (AEC) technologies but rather to complement them, handling the residual portions that AEC cannot fully eliminate, thereby significantly reducing the negative impact of agent voice interference and echo on transcription accuracy.
Other v6 Enhancements and Implications for Practical Work
Beyond Agent Text Input and echo filtering, Soniox STT v6 brings significant improvements in other areas. It substantially enhances the accuracy of voice typing, with better recognition of numbers, currencies, units, dates, IDs, proper nouns, and specialized terms. Spoken commands like "period" or "new line" are now accurately transcribed as words, allowing applications to directly apply these formatting instructions.
Furthermore, v6 has optimized capabilities for meeting transcription and speaker diarization, whether for live streams or recorded files. It also boasts improved robustness against accents, code-switching, and in-sentence language switching. It's important to note a formatting change in v6 that requires attention: it will transcribe expressions like "fifteen dollars and twenty cents" as spoken, rather than directly converting them to "$15.20". Similarly, hyphens in codes are retained only when spoken. These changes necessitate that developers update and test their data processing and parsing logic when integrating the new version.
Performance, Cost, and Availability Considerations
Soniox STT v6 is claimed to offer "significant" accuracy improvements over its predecessor, though official v6 performance metrics have not yet been released. Its predecessor, v5, achieved a Semantic Word Error Rate (WER) of 1.34% on the Pipecat STT benchmark, performing well among similar offerings. The pricing for v6 is highly competitive: $0.12/hour for streaming and $0.10/hour for asynchronous processing, with speaker diarization and language identification bundled in.
However, v6 is exclusively available as a proprietary hosted model via Soniox's cloud API (WebSocket streaming and REST asynchronous), with no self-hosted or on-device deployment options disclosed. The Agent Text Input feature is specifically limited to the streaming model `stt-rt-v6`. In practical applications, developers must weigh Soniox's pricing advantages and ease of use against their needs for model control, data privacy, or offline capabilities. The cloud-hosted nature might be a limiting factor for scenarios demanding strict data governance or ultra-low latency requirements.