LiteLLM vs TrueFoundry (2026)

LiteLLM's MCP Gateway gives agents one fixed endpoint for every registered MCP server, with access controlled per key and team and server authentication for OAuth, OAuth passthrough, on-behalf-of, AWS SigV4 and JWT signer schemes. On the Responses and Chat Completions endpoints the proxy fetches the MCP server's tools and, when configured to, executes the returned tool calls itself. So agents take authenticated action in outside systems through the gateway and do not just hand a suggestion back.

Sourcedocs.litellm.ai/docs/mcpread 2026-09-22

MCP servers registered in TrueFoundry's MCP Gateway are listed in the agent builder, where the customer picks the individual tools each agent sees. The gateway holds the credentials, runs per-user OAuth flows and refreshes tokens, and in the gateway playground a model uses a connected MCP server to act on outside systems such as a calendar or a database, so agents call into external systems through a governed tool layer.

Sourcetruefoundry.com/docs/agent-platform/agent-harness/what-truefoundry-addsread 2026-09-21

With an MCP tool's require_approval set to never, the proxy executes the model's tool calls and feeds the results back into the model before returning the answer, so the gateway runs a multi-step agent loop itself, and the A2A Agent Gateway routes calls to registered agents. There is no stated workflow definition, deterministic node or versioned flow, so sequencing, branching and retries that mix deterministic steps with agent steps are not covered.

Sourcedocs.litellm.ai/docs/mcpread 2026-09-22

Planning, tool calls, context management, approvals and state all run in the agent's execution loop inside TrueFoundry's hosted agent harness, which delegates focused subtasks to parallel subagents and can chain MCP tool calls in a single sandbox script. There are no deterministic workflow nodes, branching or retry paths that a run can combine with those autonomous steps.

Sourcetruefoundry.com/docs/agent-platform/agent-harness/overviewread 2026-09-21

Work reaches agents through LiteLLM when a caller sends a request, and model, MCP and A2A calls are all invoked by the client. There is no stated schedule, event, webhook or inbound queue that starts an agent run, and budget alerts notify people and do not wake an agent. A gateway that only answers callers has no way to wake an agent on its own.

Sourcedocs.litellm.ai/docs/a2aread 2026-09-22

Agents, sessions and schedules live in the platform through TrueFoundry's hosted agent harness, shared with the organization under the same authentication and access control as the rest of TrueFoundry. A schedule sends work to an agent without a person starting it.

Sourcetruefoundry.com/docs/agent-platform/agent-harness/what-truefoundry-addsread 2026-09-21

The /rag/ingest endpoint is an all in one ingestion pipeline (upload, chunk, embed, write to a vector store) into OpenAI vector stores, Bedrock Knowledge Bases, Vertex AI RAG Engine, Gemini or AWS S3 Vectors, and /rag/query searches the ingested content and generates a response from it. Vector store access is permissioned per team in the gateway. New customer knowledge enters a persistent index without retraining, so the retrieval structure over the customer's knowledge is maintained, persists, scales past the context window and stays queryable.

Sourcedocs.litellm.ai/docs/rag_ingestread 2026-09-22

Chat, embedding and rerank model calls route through TrueFoundry's AI Gateway, so a customer can run its own retrieval through the gateway, but TrueFoundry maintains no document ingestion, index or retrieval layer over the customer's content. The Skills Registry holds versioned skills, not the customer's documents.

Sourcetruefoundry.com/docs/ai-gateway/playground-overviewread 2026-09-21

The proxy's Memory Management API (/v1/memory, LiteLLM v1.83.10 or later with PostgreSQL connected) keeps entries across sessions, scoped per user and team with role-based read and write rules, and supports create, read, update, list by key prefix and delete. Memory lives in the gateway's own Postgres, and one user's entries can be deleted without touching the rest. The application reads entries and places them in the prompt. The scope is stated, but no lifetime is.

Sourcedocs.litellm.ai/docs/proxy/memoryread 2026-09-22

Agent state and context are managed by TrueFoundry's hosted agent harness, which keeps sessions in the platform under the organization's access control and summarizes older history when context nears the model's limit. That is session memory, scoped by the platform's access control. There is no long term memory layer and no way to review, edit or delete what an agent remembers.

Sourcetruefoundry.com/docs/agent-platform/agent-harness/overviewread 2026-09-21

Guardrails run before, during or after a call and can be scoped per key and team (PII masking, prompt injection and secret detection, content moderation and third-party guardrail providers), and the MCP Gateway adds per-key and per-team tool permissions and guardrails on tool results. These are constraints the customer controls on what an agent may do.

When an MCP tool's require_approval is set to anything other than never, the proxy returns the tool calls to the client so they can be reviewed and executed manually, and that review happens in the customer's own client. LiteLLM itself has no stated surface where a person reviews and approves an agent action before it commits.

Sourcedocs.litellm.ai/docs/mcpread 2026-09-22

Tools flagged as destructive at TrueFoundry's MCP Gateway automatically pause for human approval in every agent, and the hosted agent harness's human checkpoints pause a run for tool approval or ask the user a structured question before it continues. The gate is shipped, and it holds the agent's action until a person decides.

Sourcetruefoundry.com/docs/agent-platform/agent-harness/what-truefoundry-addsread 2026-09-21

Access control covers SSO for the Admin UI through Okta, Azure AD, Google Workspace or any OIDC or SAML provider (free for up to five users, an Enterprise license beyond that), SCIM, JWT authentication against the customer's own identity provider, role-based access control across organizations, teams and user roles, IP allowlists, public and private route controls, key rotation and external secret managers.

LiteLLM reports SOC 2 Type II, with the current report available through the LiteLLM Trust Center. Audit logs with retention policies add to that access model, identity integration and attestation. Most controls sit in the Enterprise tier.

Sourcedocs.litellm.ai/docs/enterpriseread 2026-09-22

TrueFoundry SaaS maintains SOC 2 Type II, GDPR and HIPAA programs, with reports and trust documentation at trust.truefoundry.com, and customers get single sign-on over SAML or OIDC with SCIM provisioning, role-based access control across users, teams and virtual accounts, and personal and application tokens with lifetime and rotation controls.

Sourcetruefoundry.com/docs/platform/saas-securityread 2026-09-21

The gateway records what each caller's agent did through it, covering request and response content for model and agent calls with user, key and team attribution, latency and cost (shown in the A2A Agent Gateway's Logs tab for invoked agents), and Prometheus metrics in the open source core.

Enterprise adds per-key or per-team log routing to Langfuse, LangSmith, Arize and other callbacks, audit logs of admin actions with retention policies kept apart from request logs, and log export to GCS or Azure Blob. Traffic that passes through the gateway can be inspected step by step, with audit logs kept apart from runtime traces.

Sourcedocs.litellm.ai/docs/a2aread 2026-09-22

Every agent session on TrueFoundry is captured with cost, tokens, turns, tool calls and a turn-by-turn transcript, with traces per run covering model calls, tool calls, sandbox executions and subagents, and it inherits the AI Gateway's request logs, analytics, OpenTelemetry export and Prometheus and Grafana. Audit logging of platform activity is available by dashboard and API, and request logs and traces follow the retention the customer configures. That shows what the agent did step by step, with an audit trail separate from runtime traces.

Sourcetruefoundry.com/docs/agent-platform/agent-harness/what-truefoundry-addsread 2026-09-21

Deployment is self hosted. Official Docker images, a Helm chart and a Terraform module run on the customer's own Postgres and Redis, with one-click deploy into AWS, GCP or Azure and fully air-gapped installation. Enterprise includes air-gapped deployment and multi-region deployment under one license with an admin and worker split, and the customer's data and keys never leave its own infrastructure. The customer can run it in its own cloud, on premises or fully air gapped.

Sourcelitellm.airead 2026-09-22

On TrueFoundry's managed SaaS the customer chooses the region the AI Gateway is deployed in and where request logs and traces are stored, and can bring its own Amazon S3, Azure Blob or Google Cloud Storage so logs and traces stay in its own buckets. The control plane and gateway also install self-hosted on AWS, GCP, Azure, OpenShift or on-prem, and the Enterprise plan includes VPC, on-prem and air-gapped deployment. So the customer both picks the region and can run the platform in its own environment.

Sourcetruefoundry.com/docs/platform/saas-securityread 2026-09-21

LiteLLM offers no agents of its own for a customer to adopt. The AI Hub on the Enterprise tier lists the models, agents, MCP servers and skills the customer has registered, which is a directory of the customer's own assets, and the Google AI Studio managed agents LiteLLM supports live entirely on Google's side, with LiteLLM as the auth and routing layer. Nothing there is a ready made workflow, template or role specific agent, and a model or tool catalog is integration, not a prebuilt pack.

Sourcedocs.litellm.ai/docs/enterpriseread 2026-09-22

Agents draw on a versioned catalog of skills in TrueFoundry's Skills Registry, and its Agent Registry records the customer's own agents. Neither is a catalog of prebuilt agents a customer adopts, and TrueFoundry offers no agent templates or packaged agents. Saved prompts serve as the customer's own templates.

Sourcetruefoundry.com/docs/llms.txtread 2026-09-21

The customer chooses the model. One OpenAI-compatible API reaches 140+ providers and 1,800+ models, with models set in the customer's own configuration and swapped without changing application code, the customer's internal, fine-tuned and self-hosted models behind the same key, load balancing across providers, regions and keys, lowest-cost routing, and an Auto Router that sends prompts to model tiers the customer configures.

Sourcelitellm.airead 2026-09-22

Every model enabled for a user in TrueFoundry's AI Gateway appears in the agent builder's model picker, grouped by provider, and switching models is a one-click change. The gateway playground lets teams browse and select chat, embedding, rerank and image models across providers, and virtual models back one logical name with several models for routing and failover. The customer chooses the model and the provider.

Sourcetruefoundry.com/docs/agent-platform/agent-harness/what-truefoundry-addsread 2026-09-21

A Python SDK and a proxy with an OpenAI compatible REST API ship with LiteLLM, and the gateway's own features are callable from outside. The A2A Agent Gateway serves a proxied agent card for each registered agent, pinned to A2A 0.3 or 1.0, and agents are invoked through the A2A SDK or the OpenAI SDK. Memory Management and ingestion have their own REST endpoints (/v1/memory, /v1/rag/ingest) with curl and Python examples.

Sourcedocs.litellm.ai/docs/a2aread 2026-09-22

TrueFoundry offers an API reference and OpenAPI specifications for the platform and the AI Gateway, a Python SDK and a CLI, and the gateway playground generates integration code for the OpenAI SDK, the REST API and cURL. Agents built in the hosted harness are reachable through its HTTP API and TypeScript SDK, so the platform can be called from outside through a documented API and SDK.

Sourcetruefoundry.com/docs/llms.txtread 2026-09-21

Shadow evaluations sample a key's, team's or user's live traffic, send each sampled request through a candidate router configuration without returning that answer to the client, and have an LLM judge compare it blind against the answer the current model served. A job runs up to 30 days and can compare several configurations on the same traffic before anything changes, and after a switch each request carries its routing decision and savings. A change is evaluated against the customer's own traffic, with a judge verdict.

Sourcedocs.litellm.ai/docs/auto_router/evaluateread 2026-09-22

Teams can test prompts, models, guardrails and MCP tools before production in TrueFoundry's AI Gateway playground, which shows a latency breakdown per request across the gateway and the model and saves versioned prompts, and the agent builder tests an agent against a live chat. There are no dataset runs or scoring of output quality.

Sourcetruefoundry.com/docs/ai-gateway/playground-overviewread 2026-09-21

The product is a gateway for models, MCP servers and agents, and there is no stated browser, desktop or remote computer session that LiteLLM runs for an agent. The sandboxes in the Managed Agents Platform announcement, and the swap of OpenAI's Code Interpreter for E2B or OpenSandbox, are code execution, not control of a real interface that an agent drives itself.

Sourcelitellm.airead 2026-09-22

Code, file and shell execution run in an isolated sandbox that TrueFoundry's hosted agent harness provisions, and browser tools reach an agent only as MCP servers the customer registers in the gateway. TrueFoundry ships no browser, desktop or computer control of its own.

Sourcetruefoundry.com/docs/agent-platform/agent-harness/overviewread 2026-09-21