AssemblyAI
Also known as: AssemblyAI, Inc.
AssemblyAI supplies the listening, speaking and language model services that developers connect to their own voice agents. Its Realtime Agent transcription model detects when callers finish a turn, separates the main speaker from background speech and keeps conversation context as audio arrives. The text to speech API generates the response audio, and LLM Gateway lets developers choose the model that handles transcript reasoning. The builder controls the conversation loop, the phone connection and any actions it takes.
AssemblyAI supplies speech recognition, voice generation and language model access to developers building voice agents. The Realtime Agent model listens on a WebSocket, processes audio as it arrives and signals when a caller has finished a turn, with a median of 307 milliseconds to a final transcript on the open Pipecat benchmark.
It covers 32 languages, carries context through a conversation and accepts key terms for names or specialized vocabulary. Voice Focus separates the main speaker from background speech. Builders can also supply their own turn boundaries for conversations that use a separate detector or a control such as push to talk, and the Sync API transcribes finished utterances for teams running their own turn detection.
AssemblyAI TTS 1 streams speech over WebSocket or HTTP. It offers English, Spanish, French, German, Italian and Portuguese, with a typical time to first audio of around 90 milliseconds. The Speech Understanding API can extract structured details from transcripts, and Guardrails can redact personally identifiable information from text or audio.
LLM Gateway lets a developer pick among model families including GPT, Claude, Gemini and Qwen through one interface, with fallback routing. LiveKit and Pipecat reference projects show developers how to wire the listening and speaking services into a voice agent they deploy, and Vapi, Twilio SIP, Retell and Recall.ai sit among the voice platforms it works with. The older hosted Voice Agent API stays available to existing customers until its scheduled retirement on February 1, 2027.
Realtime Agent transcription costs $0.45 per hour with unlimited concurrent streams, and the separate realtime model for meeting notes and captions costs $0.30 per hour. Recorded audio can use AssemblyAI STT 3.7 for $0.21 per hour or Universal 2 for $0.15 per hour, and multichannel audio bills per channel. Speech generation is $15 per 1 million input characters, with the same rate for cloned voices and EU processing.
A free tier with no credit card covers up to 185 hours of recorded or 333 hours of streaming transcription, and AWS Marketplace billing is available. Customers can scope API keys by project and manage access through organization roles. AssemblyAI maintains a SOC 2 Type 2 report, signs HIPAA business associate agreements and supports configured limits on audio and transcript retention. Speech processing has US and EU options.
A self hosted option runs the Realtime Agent and Sync models inside the customer's own infrastructure on Kubernetes or AWS ECS, in any region including government clouds, so call audio never leaves that environment. It is sold through sales at the same usage rates as the cloud service, with no premium for self hosting, and AWS Marketplace agreements can count toward committed cloud spend.
The service fits software teams that want to control how an agent talks, what it can call and where the conversation goes next. More than 200,000 developers use AssemblyAI. Super builds voice agents for real estate on it and cut real time transcription costs by 30%. Siro reports a 90% reduction in customer complaints and support tickets. CallRail improved phone call transcription accuracy by up to 23% and doubled the number of customers using its conversation intelligence product after integrating AssemblyAI's speech models.
On 12,460 voice agent calls in AssemblyAI's own comparison with Universal 3.6 Pro Realtime, the new recognition model made 80% fewer number errors and missed 53% fewer entities. Developers can receive a completed transcript by webhook when a streaming session ends. The callback includes the session identifier and finalized turns, so downstream transcription records can stay linked to the original call.
Vendor details
Canonical URL
https://www.assemblyai.com/
Category
Agent infrastructure
Subcategory
Speech recognition, voice generation and model routing for voice agents
Company status
independent
Use cases & customers
Primary use cases
Target customers
Deployment options
Integrations
LiveKit and Pipecat reference projects, with Vapi, Twilio SIP, Retell and Recall.ai among the voice platforms it works with. HTTP and WebSocket APIs, Python SDKs, streaming transcript webhooks, an OpenAI compatible model gateway, a remote MCP server for transcription and transcript analysis, and AWS Marketplace billing.
In practice
A real estate team uses streaming recognition and turn detection to feed the voice agent it runs for inbound calls.
A contact center software team measures recognition on recorded calls in the Playground and uses structured extraction to populate its own analytics.
A developer routes a long meeting transcript to a model through LLM Gateway and chooses a regional endpoint for the request.
Sources & related URLs
Research sources
Agentic Index coverage score
8.0 / 14 capabilities · 57%
| Integrations & Tool Calling | Partial |
|---|---|
|
Streaming session callbacks deliver completed transcripts to a customer endpoint alongside the live transcript events. A callback carries the session identity and finalized turns, and AssemblyAI retries eligible delivery failures. LiveKit and Pipecat reference projects connect the speech services to a voice workflow. Updates in a CRM or other business system run in the customer's orchestration code. Sourceassemblyai.com/docs/streaming/webhooksread 2026-11-20 |
|
| Workflow Orchestration | Partial |
|
The speech services stream recognition results and generate audio from text over separate connections. An application can pass the transcript to LLM Gateway, which can route a request to a fallback model, and use the returned text to generate speech. Turn detection helps coordinate when audio passes through that sequence. The developer deploys and runs the application that coordinates the complete voice workflow. Sourceassemblyai.com/solutions/voice-agentsread 2026-11-20 |
|
| Knowledge Grounding & RAG | Partial |
|
Key terms give the recognition service vocabulary for a particular customer or domain. A developer can include company names, product names and other terms that are likely to come up in the audio and can supply context for the conversation the model is about to hear. Recognition uses those inputs within the stream. The speech service has no maintained document index that retrieves an organization's materials. Sourceassemblyai.com/blog/which-assemblyai-stt-3-7-modelread 2026-11-20 |
|
| Human Oversight & Guardrails | Partial |
|
Transcript controls can mask sensitive values in returned text and remove them from the associated audio. Content moderation can mark categories in recordings, and profanity filtering can mask words in the transcript. The streaming model can also accept turn boundaries from the application, which lets the customer choose when a response starts. Approval of an agent's business action belongs in the customer's voice workflow. Sourceassemblyai.com/docs/guardrails/getting-startedread 2026-11-20 |
|
| Security, Identity & Governance | Full |
|
Workspace owners and admins can invite members with assigned roles and require multifactor authentication. Each project has a separate API key, and account activity logs retain changes and sign ins. Paid workspace administrators can configure data retention and model improvement settings. AssemblyAI maintains a SOC 2 Type 2 report covering security, availability and confidentiality, runs a trust center, and signs HIPAA business associate agreements that opt the account out of model training. Sourceassemblyai.com/securityread 2026-11-20 |
|
| Observability & Auditability | Partial |
|
The dashboard reports requests and spending, and organization activity logs cover changes and sign ins. The speech stream returns partial and final turns, and the completion webhook associates the transcript with a session ID. These outputs let an application relate transcript activity to a call. The complete sequence of decisions made by a customer deployed voice agent remains in that application's logs. Sourceassemblyai.com/docs/account-managementread 2026-11-20 |
|
| Memory & State Persistence | Partial |
|
Conversation memory carries information from earlier speech through the stream, so later turns can be transcribed with context from what the caller has already said. Terms and other context can accompany the audio, and the model uses them when it resolves an ambiguous word or name. This continuity lives within the audio session. An identity and history shared across calls would reside in the customer's application. Sourceassemblyai.com/blog/which-assemblyai-stt-3-7-modelread 2026-11-20 |
|
| Deployment & Data Residency | Full |
|
Managed speech services process audio through US or EU endpoints, with speech generation at the same rate in both regions, and customers choose the endpoint for each service they call. A self hosted option runs the Realtime Agent and Sync models in the customer's own infrastructure on Kubernetes or AWS ECS, in any region including government clouds, at the same usage rates. LLM Gateway offers separate regional routing for inference, and paid workspaces can set retention for supported artifacts. Sourceassemblyai.com/pricingread 2026-11-20 |
|
| Prebuilt Agents / Templates / Packs | Not documented |
|
LiveKit and Pipecat reference projects put streaming transcription and speech generation into an example voice application. A developer runs the reference code and supplies their own workflow, tools and call transport. The deployable services handle individual speech and inference tasks. A new customer assembles the voice agent from those services rather than selecting a running packaged agent. Sourceassemblyai.com/changelogread 2026-11-20 |
|
| Triggers & Channel Coverage | Partial |
|
Audio sessions begin when an application opens a streaming connection and sends audio. The service sends partial and finalized turns as speech arrives. A completion webhook delivers the assembled transcript once the session ends. The application owns the phone events and decides when to open each session, including sessions created in response to an inbound call. Sourceassemblyai.com/docs/streaming/webhooksread 2026-11-20 |
|
| Model Flexibility & Routing | Full |
|
LLM Gateway lets customers pick models from families including GPT, Claude, Gemini and Qwen. Requests use a common interface, and the gateway can route to fallback models. The same application can select different models for a short call and a longer transcript without changing its speech processing connection. Realtime Agent and the separate meeting transcription model can also be selected independently. Sourceassemblyai.com/products/llm-gatewayread 2026-11-20 |
|
| APIs / SDKs / MCP Extensibility | Full |
|
HTTP and WebSocket APIs handle live and recorded transcription, voice generation and transcript analysis. The remote MCP server accepts AssemblyAI sign in or an API key. Its tools can submit, retrieve, summarize, analyze and delete transcripts in the customer's AssemblyAI project. The streaming API also sends session events that a customer application can use to manage its connection and read completed turns. Sourceassemblyai.com/changelogread 2026-11-20 |
|
| Testing, Debugging & Optimization | Partial |
|
The Playground can run a prompt against a sample transcript and switch inference models so the developer can inspect how their answers differ. The gateway also provides model comparisons on recordings of different lengths. An application developer can assess results on a representative transcript there, but evaluation of a deployed voice agent and its decisions runs outside AssemblyAI. Sourceassemblyai.com/products/llm-gatewayread 2026-11-20 |
|
| Browser / Computer-use | Not documented |
|
The voice services process audio streams, return transcript events and generate audio for the customer's application. A developer can connect them to a browser microphone or to the audio from a phone call. Their interfaces carry speech, text and model requests, while control of the browser or desktop belongs to the application the developer builds around those services. Sourceassemblyai.com/solutions/voice-agentsread 2026-11-20 |
|
The Agentic Index coverage score grades every vendor Full, Partial or Not documented against the same 14 buyer facing capabilities, from public evidence only. Each capability links to how all vendors in the index score on it. How this evidence is graded
Pricing
From $0.15 per recorded audio hour
Audio hours, speech input characters and model usage
Included quota
The free tier covers up to 185 hours of recorded transcription or up to 333 hours of streaming transcription, with up to 5 new streaming connections a minute.
What is public
AssemblyAI charges by audio hour for transcription. Universal 2 costs $0.15 and STT 3.7 costs $0.21 for recorded audio. Realtime costs $0.30 and Realtime Agent $0.45. TTS 1 costs $15 per 1 million input characters.
Billing mechanics
LLM Gateway charges the underlying model rates for global routing. US or EU regional routing costs 10% more. Speech generation meters input characters at the same rate for cloned voices and EU processing. Multichannel audio bills per channel, so a one hour stereo file counts as two hours. Usage is invoiced monthly in arrears.
Variable cost rationale
Audio duration, generated text, inference and optional processing features drive the bill.
Additional watchouts
Voice applications incur separate charges for live recognition, generated speech and any model usage routed through the gateway.
Sales call required
No, self serve available
Free / trial
Free tier with no credit card, covering up to 185 hours of recorded or 333 hours of streaming transcription
Lowest paid plan
Universal 2 recorded transcription
Commercial notes
Enterprise contracts can include custom rate limits and higher concurrency. Pay as you go accounts start at 100 new streaming sessions a minute and scale automatically with no concurrency fee. Self hosted deployment uses the same usage rates with no premium, and purchases through AWS Marketplace can count toward committed AWS spend.
Missing data
Enterprise rates depend on workload and contract terms.
Related vendors
- AgentOps — Agent observability and debugging platform: open source SDKs trace…
- AgentPhone — AgentPhone gives the AI agents a team builds their own US or…
- Agno — Python agent framework and AgentOS runtime (formerly Phidata) for…
- Airbyte — Airbyte moves data from operational systems into warehouses and…
- AIsa — Resource and payment gateway for AI agents, with one key to 110+…
- Albato — Albato is a no code integration platform that sells an embedded,…
Alternatives to AssemblyAI
Other products that do the work AssemblyAI does, and what each one is.
- Deepgram
- Inworld AIRealtime voice AI platform
- TeradataData layer agents run
- GentoroMCP platform
- HyperspellCompany brain for AI agents
- AgentPhone