LLM Gateway

Requests carry tool definitions and returned tool calls in one OpenAI compatible format to every tools capable model in the catalog, and coding agents such as Claude Code, Cline and Codex run their own tools over the gateway. Realtime voice sessions also accept remote MCP servers with an authorization header and an approval setting, which the gateway validates and passes to the model provider. The gateway executes no tools itself, stores no tool credentials and offers no MCP gateway for outside systems, so actions happen in the customer's code.

Sourcedocs.llmgateway.io/features/realtimeread 2026-11-09

Nothing in the product runs an agent loop or sequences steps. Each request resolves to a single model call, and the Batch API works through an uploaded file of independent requests in the background with the usual routing, retries and limits. Planning, tool execution and branching between steps stay in the customer's own agent code or framework, such as the Vercel AI SDK, with the gateway answering each model call as it arrives and holding nothing about the run between calls.

Sourcedocs.llmgateway.io/features/batchread 2026-11-09

Embedding, rerank, OCR and document reading requests pass through to providers, and a search endpoint returns ranked web results with page snippets for a customer's own retrieval code. The Files API holds batch input files only. The gateway keeps no index, vector store or knowledge base over the customer's documents, so any retrieval layer is built and stored by the customer with the models the gateway routes to, outside the product.

Sourcedocs.llmgateway.io/features/searchread 2026-11-09

Guardrails on Enterprise scan every prompt for prompt injection, jailbreaks, PII, secrets, blocked terms, custom regex, restricted topics and file types, and each rule blocks the request, redacts the content or logs a warning. Owners and admins set them for the whole organization or per project, and per key, member and team budgets reject spend over a cap before a provider is called. Every control acts automatically, and no step pauses a request or tool call for a person to approve.

Sourcedocs.llmgateway.io/features/guardrailsread 2026-11-09

Enterprise organizations sign in through SAML 2.0 or OIDC with Okta, Microsoft Entra ID or Google Workspace, and SCIM provisions users and maps directory groups to roles and teams. Four roles (owner, admin, project admin and developer) scope access by project, and IAM rules per member, team and key restrict models, providers, pricing tiers and IP addresses. A SOC 2 Type II report and a penetration test under NDA are available through the trust center, which lists GDPR as in progress.

Sourcedocs.llmgateway.io/features/ssoread 2026-11-09

Every request appears in an activity log with model, provider, tokens, cost, latency, cache status and routing metadata, including the dynamic route version and node path taken, and an Agents view splits coding tool traffic into sessions with CSV export. Free plans keep metadata only, while Enterprise keeps full payloads for 30 days. Tamper evident audit logs record who changed keys, roles, IAM rules, guardrails and SSO settings, and data streams send them to a SIEM in signed, ordered batches.

Sourcedocs.llmgateway.io/features/audit-logsread 2026-11-09

Stored Responses API items let a later request chain onto an earlier response, and they sit in dedicated storage for 30 days, or not at all when a request sets store to false. A session header ties a conversation's requests together so routing stays on one provider and the activity log shows the whole thread. That is conversation state for one thread with a fixed lifetime, and the gateway offers no memory store that an agent reads and writes across sessions per user or team.

Sourcedocs.llmgateway.io/features/data-retentionread 2026-11-09

The whole stack, meaning gateway, API, dashboard and worker, self hosts under AGPLv3 on Docker, Docker Compose or Kubernetes with a Helm chart, with guides for AWS, Google Cloud and Azure managed services, and Enterprise adds Terraform modules that deploy it into the customer's own cloud account. The managed cloud runs on Google Cloud in the European Union and the United States. Per project routing overrides on Enterprise pin model traffic to a provider region with its own fallback chain.

Sourcedocs.llmgateway.io/self-hostread 2026-11-09

Twelve open source starter agents can be cloned and run against the gateway, among them a research agent that writes cited reports, a code review agent that gates CI on GitHub Actions, a SQL agent that corrects its own queries, a support triage agent and a meeting notes agent that posts to Slack or Discord. Each stands alone as a TypeScript program on the AI SDK. App templates add a chatbot, an image generator and a Slack Q&A bot, and agent skills install into coding tools.

Sourcellmgateway.io/agentsread 2026-11-09

Work reaches the gateway only when a client sends a request to its API, its Anthropic compatible endpoint, a realtime socket or the MCP server. Notifications about budgets, provider issues, model retirements and compliance changes go to people in the app or by email, and Slack and Discord alerts carry billing and guardrail events to a team's channels. No schedule, webhook, event or inbound queue starts an agent run, since the product runs no agents of its own.

Sourcedocs.llmgateway.io/features/notificationsread 2026-11-09

Customers pick among 250+ models from 40+ providers, pin a provider with a prefix, or name only a model and let routing score providers on price, uptime, throughput and latency, with failover inside the same request. Requests can also ask for the cheapest or fastest provider or use auto routing with filters. On Enterprise, dynamic routes send traffic through versioned graphs with header and body conditions, a difficulty and task classifier and percentage splits, and custom OpenAI compatible providers join the catalog.

Sourcedocs.llmgateway.io/features/routingread 2026-11-09

Applications call an OpenAI compatible REST API, described by an OpenAPI specification, with endpoints for chat, responses, embeddings, images, video, speech, batches and files, plus an Anthropic compatible messages endpoint, an AI SDK provider, a CLI and a VS Code extension. Enterprise master keys create projects, API keys, IAM rules and custom providers from the customer's backend. A hosted MCP server takes a bearer key or OAuth and offers tools for account details, usage and cost breakdowns, model listing, chat and image generation.

Sourcellmgateway.io/guides/mcpread 2026-11-09

Percentage splits in dynamic routes send a fixed share of traffic to a second model, assigned per session so a conversation stays in one arm, and route analytics compare each route's spend with sending everything to its priciest model. Prompt versions move between labels such as staging, canary and production with instant rollback, and a model arena template compares answers side by side. Nothing scores output quality, since there are no datasets, judges, feedback scores or test runs.

Sourcedocs.llmgateway.io/features/dynamic-routesread 2026-11-09

LLM Gateway is a routing layer for model calls, so it hosts no browser, desktop or remote computer session for an agent to drive. The nearest thing is a QA agent template the customer clones and deploys on its own infrastructure, where Playwright turns plain English test steps into navigation, clicks and typing; that browser belongs to the customer's copy of the code, not to the gateway. Anthropic models reached through the gateway keep their own web fetch and web search tools, which read pages without operating them.

Sourcellmgateway.io/templatesread 2026-11-09