LiteLLM
Also known as: BerriAI LiteLLM, LiteLLM Proxy, LiteLLM AI Gateway
Open source AI gateway and Python SDK with one OpenAI compatible API to 140+ model providers, plus MCP and agent gateways, keys, budgets, routing and self hosting.
LiteLLM, built by BerriAI, is an open source AI gateway that puts a team's models, MCP servers and agents behind one OpenAI compatible API and one login. It ships in two forms that share a name: a Python SDK that runs inside application code, and a proxy server, the gateway, deployed as a central service for a team or organization. The core is MIT licensed and free to self host, and the enterprise features sit under a separate license. LiteLLM counts more than 92 million PyPI downloads a month and more than 1,700 contributors, and AT&T, NVIDIA, Netflix, Lemonade and Okta speak for it on its site, with Ramp, NASA, Zurich and Cloudera among the logos.
The gateway reaches more than 140 providers and 1,800 models, and the model behind a request is set in the customer's configuration, so teams swap models without changing application code. Around that sit virtual keys, per key, team and project budgets that stop requests at the cap, rate limits, load balancing, fallbacks, caching, spend tracking for chargeback, and request logging that routes to tools such as Langfuse, LangSmith, Arize and Prometheus.
The MCP Gateway gives agents one endpoint for every registered tool server with access set per key, team, organization or user and OAuth, the A2A Agent Gateway registers and proxies agents built on Azure AI Foundry, Vertex AI Agent Engine, Bedrock AgentCore, LangGraph and Pydantic AI, and newer endpoints add a per user and per team memory store and a document ingestion pipeline into provider vector stores. The Auto Router, sold as an add on, sends each prompt to a model tier and can be tested against live traffic with shadow evaluations before a switch. A Rust version of the gateway is now live, and Lens, in early access, analyzes agent traces.
Because it is self hosted, LiteLLM runs in the customer's own cloud account, VPC or a fully air gapped environment, and its data and keys stay there. A commercial Enterprise license, priced through sales on annual request capacity and not per token, adds SSO and SCIM (SSO is free for up to five users), JWT authentication, audit logs, secret managers, organization and team admins, a multi region control plane and support with a one hour response target for the most severe issues.
Enterprise customers also get security fixes early through a private registry, and releases now ship weekly on Tuesdays with a long term support pilot. LiteLLM reports SOC 2 Type II, with the report in its Trust Center, and claims ISO 27001. The tradeoff is ownership. The software is free, while running the database, cache and upgrades sits with the adopting team.
Vendor details
Canonical URL
https://www.litellm.ai
Category
Agent infrastructure
Subcategory
LLM gateway and routing
Funding status
Independent, built by BerriAI (legal entity Berrie AI Incorporated). MIT licensed open source core with a commercial Enterprise license sold alongside it.
Company status
independent
Use cases & customers
Primary use cases
Target customers
Deployment options
Integrations
One OpenAI compatible interface to more than 140 providers as an SDK or proxy. MCP Gateway for registered tool servers with per key and team access and OAuth; A2A Agent Gateway for Azure AI Foundry, Vertex AI Agent Engine, Bedrock AgentCore, LangGraph and Pydantic AI agents. Logs route to Langfuse, LangSmith, Arize, Prometheus and GCS or Azure Blob. Procurement through AWS Marketplace and authorized resellers.
In practice
Your platform team wants one gateway across every provider, fully in your own cloud. You deploy the MIT licensed LiteLLM proxy, issue virtual keys per squad, and enforce per team budgets that hard stop when exceeded.
You have air gapped or strict residency requirements that rule out SaaS. LiteLLM self hosts entirely in your environment, so provider keys and request logs never leave infrastructure you control.
Your CISO needs single sign on, role based access, and audit logs across LLM traffic. You add the LiteLLM Enterprise license on top of the open source proxy to unlock governance without changing the deployment model.
Sources & related URLs
Agentic Index coverage score
9.5 / 14 capabilities · 68%
| Integrations & Tool Calling | Full |
|---|---|
|
LiteLLM's MCP Gateway gives agents one fixed endpoint for every registered MCP server, with access controlled per key and team and server authentication for OAuth, OAuth passthrough, on-behalf-of, AWS SigV4 and JWT signer schemes. On the Responses and Chat Completions endpoints the proxy fetches the MCP server's tools and, when configured to, executes the returned tool calls itself. So agents take authenticated action in outside systems through the gateway and do not just hand a suggestion back. Sourcedocs.litellm.ai/docs/mcpread 2026-11-03 |
|
| Workflow Orchestration | Partial |
|
With an MCP tool's require_approval set to never, the proxy executes the model's tool calls and feeds the results back into the model before returning the answer, so the gateway runs a multi-step agent loop itself, and the A2A Agent Gateway routes calls to registered agents. There is no stated workflow definition, deterministic node or versioned flow, so sequencing, branching and retries that mix deterministic steps with agent steps are not covered. Sourcedocs.litellm.ai/docs/mcpread 2026-11-03 |
|
| Knowledge Grounding & RAG | Full |
|
The /rag/ingest endpoint is an all in one ingestion pipeline (upload, chunk, embed, write to a vector store) into OpenAI vector stores, Bedrock Knowledge Bases, Vertex AI RAG Engine, Gemini or AWS S3 Vectors, and /rag/query searches the ingested content and generates a response from it. Vector store access is permissioned per team in the gateway. New customer knowledge enters a persistent index without retraining, so the retrieval structure over the customer's knowledge is maintained, persists, scales past the context window and stays queryable. Sourcedocs.litellm.ai/docs/rag_ingestread 2026-11-03 |
|
| Human Oversight & Guardrails | Partial |
|
Guardrails run before, during or after a call (PII masking, prompt injection and secret detection, content moderation and third party guardrail providers), and scoping them per key, team or model is an Enterprise feature. The MCP Gateway adds per key and per team tool permissions and guardrails on tool results. When an MCP tool's require_approval is set to anything other than never, the proxy returns the tool calls to the client so they can be reviewed and executed manually, and that review happens in the customer's own client. LiteLLM itself has no surface where a person reviews and approves an agent action before it commits. Sourcedocs.litellm.ai/docs/mcpread 2026-11-03 |
|
| Security, Identity & Governance | Full |
|
Access control covers SSO for the Admin UI through Okta, Azure AD, Google Workspace or any OIDC or SAML provider (free for up to five users, an Enterprise license beyond that), SCIM, JWT authentication against the customer's own identity provider, role based access control across organizations, teams and user roles, IP allowlists, public and private route controls, key rotation and external secret managers. LiteLLM reports SOC 2 Type II, with the current report available through its Trust Center, and its enterprise page also claims ISO 27001. Audit logs with retention policies add to that access model, and most controls sit in the Enterprise tier. Sourcedocs.litellm.ai/docs/enterpriseread 2026-11-03 |
|
| Observability & Auditability | Full |
|
The gateway records what each caller's agent did through it, covering request and response content for model and agent calls with user, key and team attribution, latency and cost (shown in the A2A Agent Gateway's Logs tab for invoked agents), and Prometheus metrics in the open source core. Enterprise adds per-key or per-team log routing to Langfuse, LangSmith, Arize and other callbacks, audit logs of admin actions with retention policies kept apart from request logs, and log export to GCS or Azure Blob. Traffic that passes through the gateway can be inspected step by step, with audit logs kept apart from runtime traces. Sourcedocs.litellm.ai/docs/a2aread 2026-11-03 |
|
| Memory & State Persistence | Partial |
|
The proxy's Memory Management API (/v1/memory, LiteLLM v1.83.10 or later with PostgreSQL connected) keeps entries across sessions, scoped per user and team with role-based read and write rules, and supports create, read, update, list by key prefix and delete. Memory lives in the gateway's own Postgres, and one user's entries can be deleted without touching the rest. The application reads entries and places them in the prompt. The scope is stated, but no lifetime is. Sourcedocs.litellm.ai/docs/proxy/memoryread 2026-11-03 |
|
| Deployment & Data Residency | Full |
|
Deployment is self hosted. Official Docker images, a Helm chart and a Terraform module run on the customer's own Postgres and Redis, with one click deploy into AWS, GCP or Azure and fully air gapped installation. Enterprise includes air gapped deployment and a multi region control plane under one license, and the customer's data and keys never leave its own infrastructure. The customer can run it in its own cloud account, its own VPC or fully air gapped. Sourcelitellm.airead 2026-11-03 |
|
| Prebuilt Agents, Templates & Packs | Not documented |
|
LiteLLM offers no agents of its own for a customer to adopt. The AI Hub on the Enterprise tier lists the models, agents, MCP servers and skills the customer has registered, which is a directory of the customer's own assets, and the Google AI Studio managed agents LiteLLM supports live entirely on Google's side, with LiteLLM as the auth and routing layer. Nothing there is a ready made workflow, template or role specific agent, and a model or tool catalog is integration, not a prebuilt pack. Sourcedocs.litellm.ai/docs/enterpriseread 2026-11-03 |
|
| Triggers & Channel Coverage | Not documented |
|
Work reaches agents through LiteLLM when a caller sends a request, and model, MCP and A2A calls are all invoked by the client. There is no stated schedule, event, webhook or inbound queue that starts an agent run, and budget alerts notify people and do not wake an agent. A gateway that only answers callers has no way to wake an agent on its own. Sourcedocs.litellm.ai/docs/a2aread 2026-11-03 |
|
| Model Flexibility & Routing | Full |
|
The customer chooses the model. One OpenAI-compatible API reaches 140+ providers and 1,800+ models, with models set in the customer's own configuration and swapped without changing application code, the customer's internal, fine-tuned and self-hosted models behind the same key, load balancing across providers, regions and keys, lowest-cost routing, and an Auto Router that sends prompts to model tiers the customer configures. Sourcelitellm.airead 2026-11-03 |
|
| APIs, SDKs & MCP Extensibility | Full |
|
A Python SDK and a proxy with an OpenAI compatible REST API ship with LiteLLM, and the gateway's own features are callable from outside. The A2A Agent Gateway serves a proxied agent card for each registered agent, pinned to A2A 0.3 or 1.0, and agents are invoked through the A2A SDK or the OpenAI SDK. Memory Management and ingestion have their own REST endpoints (/v1/memory, /v1/rag/ingest) with curl and Python examples. Sourcedocs.litellm.ai/docs/a2aread 2026-11-03 |
|
| Testing, Debugging & Optimization | Full |
|
Shadow evaluations sample a key's, team's or user's live traffic, send each sampled request through a candidate router configuration without returning that answer to the client, and have an LLM judge compare it blind against the answer the current model served. A job runs up to 30 days and can compare several configurations on the same traffic before anything changes, and after a switch each request carries its routing decision and savings. A change is evaluated against the customer's own traffic, with a judge verdict. Sourcedocs.litellm.ai/docs/auto_router/evaluateread 2026-11-03 |
|
| Browser & Computer Use | Not documented |
|
The product is a gateway for models, MCP servers and agents, and there is no stated browser, desktop or remote computer session that LiteLLM runs for an agent. The sandboxes in the Managed Agents Platform announcement, and the swap of OpenAI's Code Interpreter for E2B or OpenSandbox, are code execution, not control of a real interface that an agent drives itself. Sourcelitellm.airead 2026-11-03 |
|
The Agentic Index coverage score grades every vendor Full, Partial or Not documented against the same 14 buyer facing capabilities, from public evidence only. Each capability links to how all vendors in the index score on it. How this evidence is graded
Product updates
LiteLLM open sourced Moyai, the cloud coding agent its own team runs, under the MIT license. A user hands it a task in Slack or the browser, each session gets its own workspace with a terminal, filesystem and browser, and the agent can split large tasks across parallel workers and opens a pull request for review, with Hermes, Claude Code, Codex, OpenCode or Deep Agents as the harness.
Bears on: Deployment / data residency
View sourceLens, new from LiteLLM, uses AI agents to review the traces passing through the LiteLLM gateway, group recurring failures and link each finding to the exact step where it happened. Investigations run on demand or on a schedule, and the traces stay in a store the customer runs beside the gateway.
Bears on: Observability / auditability
View sourceLiteLLM launched the public preview of LiteAgents, an SDK that supports Deep Agents, Pydantic AI, Claude Agent SDK, Codex and OpenCode through a common query interface. Profiles select the runtime while retaining shared tools, MCP connections and model configuration; native runtime options remain available. Optional Temporal integration adds recovery from recorded operations and checkpoints, while simple local agents require neither Temporal nor PostgreSQL. Changing runtimes starts new runs and does not migrate an active native session.
Bears on: Workflow orchestration
View sourcePricing
The open source core is free to self host, and Enterprise is an annual license quoted by sales.
license and self hosted infrastructure
Included quota
The MIT licensed open source proxy and SDK are free with no usage fees, log limits, or request quotas imposed by LiteLLM; you fund only infrastructure and pay providers directly. LiteLLM Enterprise is a self hosted commercial license adding governance features.
What is public
The open source tier is published at $0, self-hosted. Enterprise has no published price: the pricing page says it is sized to annual gateway request capacity, deployment architecture and support needs, never per token, with volume discounts.
Billing mechanics
Open source: no license fee; the proxy runs on the customer's infrastructure and providers bill the customer directly. Enterprise: a license key unlocks governance features on the same self-hosted deployment, bought direct, through AWS Marketplace or authorized resellers.
Cost watchouts
The customer runs and pays for the infrastructure (Postgres, Redis, the proxy fleet) and pays model providers directly. LiteLLM ships a new minor line roughly weekly and supports only the four most recent stable lines, so staying supported means a regular upgrade cadence. Standard Enterprise support has no guaranteed response time; 24/7 SLAs cost extra.
Variable cost rationale
The software license is fixed or zero, but total cost is dominated by self hosted infrastructure, engineering operations, and direct provider spend, all of which scale with usage.
Overage / add-ons
No usage metering on the open source core. Enterprise is sold by annual request capacity with volume discount tiers; provider and infrastructure costs are the customer's own.
Sales call required
Mixed (some tiers require a call)
Free / trial
MIT open source core free to self-host; 30-day Enterprise trial key, no credit card
Lowest paid plan
Enterprise, annual, priced through sales by gateway request capacity, deployment architecture and support needs; no published figure
Commercial notes
Independent, built by BerriAI (Y Combinator). More than 40,000 GitHub stars. Widely adopted as the default open source LLM gateway.
Key ambiguities
Enterprise figures are quoted per deployment; the Auto Router add-on's price is not published.
Support SLA / resale
Standard support included with Enterprise (weekday Slack or Teams channel, no guaranteed response time); 24/7 SLAs for an additional fee
Related vendors
- AgentOps — Agent observability and debugging platform: open source SDKs trace…
- AgentPhone — AgentPhone gives the AI agents a team builds their own US or…
- Agno — Python agent framework and AgentOS runtime (formerly Phidata) for…
- AIsa — Resource and payment gateway for AI agents, with one key to 110+…
- Albato — Albato is a no code integration platform that sells an embedded,…
- AlphaBitCore — AI control plane for regulated financial firms: one gateway enforces…
Alternatives to LiteLLM
The closest documented capability profiles to LiteLLM among agent infrastructure platforms tracked by Agentic Index, ordered by similarity on the same 14 point evidence the rankings use. No vendor pays for placement.
- Kong8.5 / 14A lighter documented profile than LiteLLM
- Arcade7.5 / 14A lighter documented profile than LiteLLM
- Cartesia10.5 / 14Adds documented Prebuilt Agents, Templates & Packs and Triggers & Channel Coverage
- Faros10.5 / 14Adds documented Prebuilt Agents, Templates & Packs and Triggers & Channel Coverage
- Inworld AI8.5 / 14Adds documented Triggers & Channel Coverage
- Portkey7.5 / 14A lighter documented profile than LiteLLMLiteLLM vs Portkey →
Similarity is computed from each vendor's Agentic Index coverage score evidence, axis by axis, not from the totals. How this evidence is graded