Text-to-Speech API Platforms and the 300-Millisecond Rule: What I Got Wrong Buying Voice Infrastructure

The call lasted nine seconds. First day of live traffic on a booking agent we had built for a client's service business, and the caller opened with "Hi, yeah, I'm calling about the—" and our agent came in over the top of them with its greeting. The caller hung up.

Most of the advice about text-to-speech API platforms — and nearly all of the advice about phone agents built on them — is downstream of a single number that almost nobody has read the source for. That morning, our whole team was chasing it. The timing log said our response landed in 410 milliseconds. We had been told to get under 300. So we spent four days getting under 300, and the agent kept talking over people.

The actual bug was in our endpointing. The voice activity detector was set to call end-of-turn after 500 milliseconds of silence, and the caller had trailed off mid-word to check a reference number on their phone. Speed had nothing to do with it. Being faster made it worse, because now the agent barged in sooner. That week cost us most of a sprint and a chunk of the client's goodwill, and it started something I have been unpicking ever since: the loudest rule in voice infrastructure has a source, and the source is thinner than the rule.

The verdict, up front

If you are shipping both narration and live phone agents, buy one platform that owns the realtime transport and the turn-taking logic, and keep your speech synthesis vendor swappable behind an interface you control. That is the split that has held up for me: bundle the hard realtime plumbing, unbundle the voice.

The reasoning is unglamorous. Transport, interruption handling, echo cancellation and session state are where a stitched stack bleeds engineering hours, because every failure is somebody else's fault and nobody's log has the whole call. Voice quality, by contrast, is a swap — one function signature, one A/B, done in an afternoon if you designed for it. Buying it the other way round, which is what most teams drift into, means you own the plumbing forever and you are locked to whichever voice your orchestrator happened to bundle.

Where the 300-millisecond rule came from

It is worth tracing, because the lineage explains the confusion.

The first ancestor is telecoms. ITU-T Recommendation G.114 has for decades set guidance on one-way mouth-to-ear delay for voice transmission: keep it under about 150 milliseconds for most applications, treat 150 to 400 as a range that degrades with awareness, and consider anything past 400 unacceptable for interactive use. Real document, real engineering, still load-bearing for anyone provisioning SIP trunks. Note what it measures: how long your voice takes to reach another human's ear. It says nothing whatsoever about how quickly a party should reply.

The second ancestor is conversation analysis. A study published in the late 2000s looked at turn-taking across ten languages on five continents and found that the gap between one speaker finishing and the next starting clusters remarkably tightly near a couple hundred milliseconds, with modest cultural variation around that centre. It is a genuinely lovely result and it is the number that escaped into our industry.

The third ancestor is the voice agent boom of the last few years. Demos on conference stages, latency dashboards in every pitch deck, arena-style leaderboards where models are ranked by listener preference and the rankings move week to week. Somewhere in there the two ancestors fused. Transmission budget plus human turn gap became "under 300 milliseconds end to end or your agent sounds robotic," and vendor landing pages started quoting millisecond figures the way espresso machines quote bar pressure — a real measurement, prominently displayed, weakly connected to whether the output is any good.

What the sources actually say

Here is the part that undoes the rule. The turn-taking research measures gaps between humans who are predicting the end of your sentence before you finish it. That is the whole mechanism: listeners parse syntax and prosody in flight, load a response mid-turn, and fire it when your intonation lands. The gap is short because the reply was already built.

A machine that waits for silence, transcribes, reasons, then synthesizes is doing something structurally different. It is not responding fast; it is responding late and compressing the lateness. Holding it to a number produced by anticipation is a category error wearing a benchmark's clothes. And G.114 is measuring transmission, on a clock that starts when you speak — while your latency dashboard is measuring end-of-speech to first audio byte. Different clocks, different quantities, same units, endless confusion.

I am not arguing latency is free. Two and a half seconds of dead air will lose you the call, and I have listened to enough of those to be sure. The narrower claim: below roughly a second, consistency and turn-taking behaviour beat raw speed, and every hour spent past that point is an hour not spent on the thing that is actually hanging up on you.

What actually breaks a call

Endpointing. When does the agent decide you have finished? Fixed silence timers fail on the exact utterances that matter most — phone numbers, addresses, postcodes, anything read off a screen with a pause in the middle. Platforms that expose semantic endpointing, where the model judges whether the utterance is syntactically complete before the timer fires, are worth more than 200 milliseconds of headline speed.

Barge-in. Can the caller interrupt? When they do, does the platform cancel the synthesis stream and flush the audio already buffered downstream, or does the agent keep talking into their ear for another second because a chunk was in flight? Related: does the agent hear itself on a speakerphone and interrupt its own turn? That one produces the single most unhinged transcript you will ever read.

Tail latency, not mean. A stack that answers in 500 milliseconds every single time feels calm. A stack averaging 320 with a p99 of 1.8 seconds feels broken, because humans notice variance, not averages. Measure p50, p95 and p99 or do not bother measuring.

Recovery. What plays when the language model times out? Silence is the default and silence is the worst option. A short holding line — even a breath and "one moment" — buys you two full seconds of tolerance you were otherwise trying to engineer away.

The part nobody tells you about phone audio

I came to this from sound design, so let me hand you the thing that reframed the whole budget for me.

A call that touches the public phone network is almost certainly running G.711 mu-law: 8 kHz sample rate, roughly 300 Hz to 3.4 kHz of usable bandwidth. Your gorgeous 48 kHz voice, the one with the breath and the air and the sibilance you paid a premium tier for, gets everything above 3.4 kHz thrown in the bin before it reaches the caller. The expressive top end that makes a voice sound alive in a browser demo does not exist on a phone line.

Three consequences. First, spend your fidelity budget on the channels that can carry it — video, in-app playback, anything over a wideband codec — and stop paying flagship rates for the PSTN leg. Second, audition candidate voices after downsampling to 8 kHz mu-law, not in the vendor's web player, because the ranking changes. What survives that pipe is consonant clarity, pacing and level consistency, not warmth. Third, consistent loudness matters more than any target number here; a voice that drifts in level across a sentence gets misheard in a way a slightly dull voice never does.

And a free one: over a 3.4 kHz band, "B" and "V" and "M" and "N" are genuinely hard to separate. If your agent reads back confirmation codes, phoneticise them in the prompt. That single change will do more for your call success rate than the next three latency optimisations combined.

How I'd decide

The criteria I would actually score a shortlist on, in roughly this order:

• Output quality per channel. Rate every candidate twice: once at 48 kHz for the web, once after 8 kHz mu-law encoding for the phone. Blind, with three people, on your own script.
• Turn-taking control. Are VAD sensitivity, endpoint timing, semantic endpointing and interruption behaviour exposed as parameters you can tune per use case? If they are hardcoded, you have bought someone else's opinion about your callers.
• Formats and hooks. Streaming chunk format, PCM and mu-law and Opus output, word-level timestamps (you will want them for captions, analytics and any visual sync), and some form of prosody control. Timestamps are the quiet differentiator; a lot of teams discover they need them a month after signing.
• Licensing clarity. Commercial use at the tier you will actually run, ownership of generated audio, whether your audio is retained or used for training, synthetic-voice disclosure obligations in your markets, and the consent and attestation requirements for cloned voices. This is where teams get burned worst, and it is the cheapest thing on this list to check.
• Concurrency limits, not monthly volume. Your bill is driven by minutes; your outage is driven by simultaneous sessions. Find the concurrency table for your plan before you find the price.
• Cost over twelve months at a realistic answer rate, including telephony minutes, which are usually billed by someone else.
• Failure behaviour. Documented timeouts, retries, and a clear boundary of whose problem a dropped call is.
• Who it is wrong for, stated by the vendor. The ones who can answer that question are the ones worth talking to.

Bundled or stitched: the real trade

The honest read: bundled wins on everything that costs engineering time, stitched wins on everything that costs money at scale. Which is why the split I recommended up front is not a compromise so much as an admission — bundle the layer where debugging is expensive, unbundle the layer where switching is cheap.

Reading a voice rate card

Voice pricing is quoted in at least five incompatible units: per character for synthesis, per generated second or minute, per minute all-in for agent sessions, per concurrent session for capacity, and per audio token for speech-to-speech models. Comparing a per-character rate against a per-minute rate is the most common budgeting error I see.

The conversion is straightforward. Conversational speech runs around 150 words per minute; English averages roughly five to six characters per word including the space. So a minute of speech is somewhere near 850 to 900 characters. Work out your own figure from a real script — technical readbacks with lots of digits skew it — and use it consistently.

Then build the twelve-month model with your own current rates, because published prices move and anything I quoted here would be wrong by the time you read it. Model it as: (calls per month x average completed minutes) x all-in per-minute rate, plus telephony, plus the narration character volume, plus whatever seat or platform fee sits underneath. Run it at three volumes — pilot, plan, and the optimistic case — because the optimistic case is where per-minute bundles stop being cheap and per-layer pricing pulls ahead.

Two contract traps worth naming. Credits that expire monthly turn a spiky workload into a much worse effective rate than the sticker suggests. And commercial-use rights gated behind a tier are still common — check that the tier you are piloting on grants the same rights as the tier you will scale on, because discovering otherwise after launch is a genuinely bad afternoon.

The vendors, with one honest complaint each

As of writing, and with the caveat that this market reshuffles fast.

ElevenLabs still sets the bar for expressive, characterful voices and cloning, and if narration quality is your product it is the obvious anchor. The complaint: character metering plus plan-gated concurrency means a traffic burst can hit a ceiling that has nothing to do with your monthly spend, and the cloning tiers carry conditions you must read before putting a real person's voice into an outbound dialer.

Cartesia is built around low-latency streaming and it shows in agent workloads. The complaint: a narrower voice catalogue and less expressive range for long-form narration than the premium cloning shops, so it may win your phone leg and lose your trailer.

Deepgram earns its place on transcription quality and now speaks as well, which simplifies one hop. The complaint: the voices are tuned for agents rather than performance, and you will notice if you ask them to carry emotion.

OpenAI's realtime speech-to-speech collapses the whole pipeline and the naturalness of the turn-taking is the best argument for it. The complaint: you give up per-layer control entirely — you cannot swap the voice independently, you cannot see which stage went wrong, and audio-token pricing is harder to forecast than a flat per-minute rate.

Speechify and Play compete hard on all-in per-minute agent pricing, which makes the spreadsheet easy. The complaint: bundled rates obscure which layer is degrading when a call goes bad, and leaderboard positions cited in marketing move week to week.

Azure, Google and Amazon Polly are the boring correct answer for regulated buyers: regional availability, compliance paperwork, price stability. The complaint: outside the newest neural voices, the expressive range reads as slightly dated next to the specialists.

Orchestrators — Vapi, Retell, LiveKit Agents, Pipecat — solve the transport and turn-taking problem properly, which is the expensive problem. The complaint: managed ones add a hop and an SLA between you and your caller; the self-hosted ones hand you the pain back in exchange for control.

The bake-off I'd run before signing anything

Define one internal interface — a synthesize call and a session call — and write vendor adapters behind it. That single afternoon of work is what makes every later decision reversible.

Then run 200 real calls, or a replay corpus of recorded caller audio if live traffic is too precious. Log end-of-speech to first audio at p50, p95 and p99; barge-in success rate; endpoint false-fires per call; cost per completed call rather than per minute; and a blind human rating of the 8 kHz audio. Run two candidates in parallel on a slice of traffic rather than sequentially, because week-to-week traffic differences will otherwise decide it for you.

Decide on completed-call cost and false-fire rate. Not on the millisecond figure.

Who this is for, and who should skip it

This is for teams carrying both workloads — marketing narration and product audio on one side, live phone agents on the other — at enough volume that a per-minute bundle starts to sting.

Skip the bundling advice if long-form narration is the product; buy the best voice available and ignore realtime entirely. Skip it if you are regulated into on-premise or bring-your-own-cloud, where the vendor list shortens to the hyperscalers. Skip it if you are under a few hundred calls a month, where a managed orchestrator will cost less than the engineer-weeks. And skip it if your stitched stack already meets its p95 and your team knows where the bodies are — migration is never free, and "it works" is a feature.

The number that started all this is a real measurement of a real phenomenon, borrowed for a job it was never doing. Most of what the field believes about voice infrastructure is like that: sourced, then stretched.

So, tonight's rule: if your agent can be interrupted mid-sentence and recover gracefully, ship it at 800 milliseconds — and if it cannot, no amount of speed will save it.