LiveKit

Agents call tools the developer writes as Python or Node.js functions, with typed arguments, async tools that keep talking while work runs, and toolsets added or removed as a unit. MCPToolset attaches tools from any MCP server, tools can be forwarded to the frontend over RPC, and provider tools such as OpenAI web search run on the model server. The no code Agent Builder adds HTTP tools that read, create, update or delete data through outside APIs, with authorization headers drawn from stored secrets, so an agent can change records in a calendar, CRM or booking system during a call.

Sourcedocs.livekit.io/agents/logic/tools/definitionread 2026-11-09

Conversations are built from agents that hold control and hand off to one another, tasks that take temporary control and return a typed result, and task groups that run ordered steps and let a caller go back to correct an earlier answer.

Named patterns cover a supervisor agent that routes work to specialist tasks, subagent delegation that sends slow reasoning to a background model while the call continues, and handoffs that pass, summarize or reset the conversation history.

A booking flow can route lookup, rescheduling and cancellation to separate tasks, then hand payment to a billing agent with its own tools.

Sourcedocs.livekit.io/agents/logic/workflowsread 2026-11-09

Retrieval runs through the developer's own stores. A hook that fires when the caller finishes a turn can run a vector search and add the results to the context before the model replies, a search tool can query a knowledge base mid call while the agent tells the caller it is still looking, and a LlamaIndex example answers questions from a document set. Initial context such as a customer profile loads from a database before the session starts. LiveKit keeps no index, embeddings store or document library of its own, and the Agent Builder has no place to upload knowledge files.

Sourcedocs.livekit.io/agents/logic/external-dataread 2026-11-09

A warm transfer task puts the caller on hold, dials a person over SIP, briefs them with a summary of the call and then connects them, and SIP REFER forwards calls cold. Developers can mark tools as uninterruptible, let a running tool be canceled, and make the model ask again before a duplicate call such as a second booking runs. Prebuilt data collection tasks read a name, address or date of birth back to the caller for confirmation. There is no queue or console where an operator approves an agent's action before it commits, so an approval step is code the developer writes.

Sourcedocs.livekit.io/agents/prebuilt/tasksread 2026-11-09

LiveKit Cloud offers role based access that assigns roles and capabilities to project collaborators on the Scale plan, and single sign on with the customer's identity provider on Enterprise. Every connection needs a signed access token whose grants limit room access, publishing, subscribing and recording, with a lifetime the application sets.

LiveKit holds a SOC 2 Type II attestation covering security, availability and confidentiality, runs network penetration tests, signs HIPAA BAAs on Scale and Enterprise, and is certified under the Data Privacy Framework. Audit reports sit in a trust center under NDA, and ISO 27001 and PCI DSS are in progress.

Sourcelivekit.com/securityread 2026-11-09

Agent insights in LiveKit Cloud keep each session's turn by turn transcript, audio, runtime logs and traces, with spans for every user and agent turn, each pipeline step and each tool call, so a reviewer can see which tools ran, with what metadata, and how long each turn took. Tool calls and handoffs sit on the session timeline beside latency and token metrics. Session data is kept for 30 days unless downloaded, PII can be redacted before storage, traces export to any OpenTelemetry backend, and log drains send runtime logs to Datadog, CloudWatch, Sentry, New Relic or Splunk.

Sourcedocs.livekit.io/testing/observability/insightsread 2026-11-09

Within a session, the chat context holds every message, tool call and handoff, the full history across agents stays readable on the session, and session user data carries state such as a caller's order details from one agent to the next. Handoffs can pass the history on, summarize it or start fresh. Nothing carries into the next call by default. The framework suggests loading a profile from the customer's own database when a session starts and saving history to the customer's own server when it ends, and it points to Letta and Mem0, which those vendors run, for long term memory.

Sourcedocs.livekit.io/agents/logic/chat-contextread 2026-11-09

Each LiveKit Cloud agent deployment runs in a region the customer picks: US East in Virginia, EU Central in Frankfurt or South Asia in Mumbai, and one agent can run in several.

The project data region, chosen when the project is created and fixed afterward, decides where transcripts, recordings, traces and logs are stored, and LiveKit Inference keeps model requests on EU models when agents run in the EU. Region pinning on Scale holds media to one region.

Agents also run as containers on the customer's own Kubernetes cluster, the media server is open source and self hosted, and private deployments go through sales.

Sourcedocs.livekit.io/deploy/admin/regions/data-residencyread 2026-11-09

Prebuilt tasks collect and validate a caller's name, email, mailing address, date of birth, phone number, card details and keypad digits, a warm transfer task hands calls to a person, and prebuilt tools end calls and send keypad tones. These are parts a developer places inside an agent.

Starter projects for Python and Node.js scaffold one voice assistant, starter apps cover web, iOS, Android, Flutter and React Native frontends, and a recipes library gathers example applications. No finished agent for a given job, such as a receptionist or a collections agent, is offered for a customer to adopt as is.

Sourcedocs.livekit.io/agents/prebuilt/tasksread 2026-11-09

Inbound phone calls start agents without anyone at the customer's end. A caller dials a LiveKit Phone Number or a SIP trunk from Twilio, Telnyx or Plivo, a dispatch rule places the call in a room, and the named agent is dispatched to answer, with Twilio and WhatsApp connectors as further routes in. The customer's backend can also dispatch an agent through the dispatch API with job metadata, place outbound calls by creating a SIP participant, and receive webhooks on room and participant events. Web and mobile apps open sessions with an access token that names the agent to dispatch.

Sourcedocs.livekit.io/telephonyread 2026-11-09

The developer picks every model in the pipeline.

LiveKit Inference serves language models from OpenAI, Google Gemini, DeepSeek, Kimi and an open weight model LiveKit hosts, speech recognition from Deepgram, AssemblyAI, Cartesia and Speechmatics, voices from Cartesia, Rime, Inworld and others, and realtime models such as the OpenAI Realtime API and Gemini Live, all under one LiveKit key.

Plugins add Anthropic, Amazon Bedrock, Azure OpenAI, Groq, Mistral AI, OpenRouter, local models through Ollama and any OpenAI compatible endpoint, fallback adapters switch providers when one fails, and the Agent Builder offers the same choice.

Sourcedocs.livekit.io/agents/models/inferenceread 2026-11-09

The Agents framework is an open source SDK for Python and Node.js with a full API reference. Server SDKs for Python, Node.js, Go, Rust, Ruby and Kotlin call the room, agent dispatch, SIP and egress APIs, so a backend can create rooms, dispatch agents, place calls and manage trunks and phone numbers. The LiveKit CLI creates, deploys, rolls back and simulates agents and manages secrets, the Analytics API returns session data on Scale, and client SDKs cover web, Swift, Android, Flutter, React Native, Unity, C++ and ESP32. A docs MCP server gives coding agents the documentation.

Sourcedocs.livekit.io/referenceread 2026-11-09

Agent Simulations run the developer's real agent against an LLM driven simulated caller with a persona and a goal, then a judge scores the transcript against written expectations, and an added check can test the end state, such as whether a booking was written. Text mode tests prompts, tools and logic, audio mode tests the whole voice pipeline and turn timing, scenarios can be derived from a recorded production session, and scenario sets run in CI on every commit. Unit tests on pytest and Vitest assert on each turn's messages and tool calls, with an LLM judge for open ended replies.

Sourcedocs.livekit.io/testing/simulationsread 2026-11-09

Agents talk, see and act through tools, and LiveKit runs no browser, desktop or remote computer for them to operate. The Anthropic plugin passes through Anthropic's computer use tool, but the desktop it would control is left to the developer to supply and run. Vision lets an agent watch a caller's camera or screen share without controlling it, and web egress loads a page in headless Chromium only to record or stream it. Robotics features stream video and sensor data and support human teleoperation of physical hardware, which is not control of a software interface.

Sourcedocs.livekit.io/agents/logic/toolsread 2026-11-09