Braintrust
AI observability and evaluation platform: traces, datasets, experiments, online scoring and human review, with hosted prompts and tools, a model gateway, the Loop agent and an MCP server.
Braintrust is an AI observability and evaluation platform for instrumenting, understanding and improving agents. It connects the pieces of an AI development loop: production traces, human feedback, test datasets, experiments, quality scores, prompt iteration, model routing and deployment.
Traces record each request's nested steps, including model calls, tool calls, retrieved context, errors, cost and latency, and teams filter, examine and debug them, with dashboards, alerts to Slack or webhooks, and Topics, which classifies production logs by task, sentiment and issue. Evals run an AI system over versioned datasets and score it with LLM-as-a-judge, autoevals, custom code or human reviewers; experiments are compared side by side, evals run in CI, and online scoring applies the same scorers to live traffic so failures become new test cases.
Braintrust also hosts functions: prompts, tools that deployed prompts can call, scorers and workflows that chain prompts, versioned and tagged by environment, plus a Gateway that routes model requests to providers on the customer's own keys. Loop, its built-in agent, investigates project data, writes scorers and test cases, iterates on prompts and runs on a schedule, pausing for approval before it changes anything in an interactive thread. SDKs cover Python, TypeScript, Go, Java, Ruby and C#, alongside a REST API, the bt CLI and an MCP server for coding agents.
Braintrust lists SOC 2 Type II compliance on every plan, supports SSO and SAML, OIDC, role-based access with custom groups and audit logs, and signs BAAs on Enterprise. It runs as SaaS, as BYOC in the customer's cloud operated by Braintrust, or self-hosted on Enterprise. Starter is free with unlimited users, Pro is $249 a month, both with usage-based overage, and Enterprise is custom.
Vendor details
Canonical URL
https://www.braintrust.dev
Category
Agent infrastructure
Company status
independent
Use cases & customers
Target customers
Deployment options
In practice
Your AI feature works in testing but quietly regresses in production and you find out from users. Braintrust scores live traffic with LLM-based judges, surfacing regressions as they happen and turning real failures into new test cases.
A prompt change looks better on one example but you can't tell if it's better overall. Braintrust runs your prompts and models against a dataset, scores them side by side, and captures an immutable experiment you can compare over time.
A bad change keeps slipping into your agent before anyone catches it. Braintrust runs your evals on every pull request and blocks the merge when a change degrades quality, gating releases on measured output quality.
Sources & related URLs
Agentic Index coverage score
10.0 / 14 capabilities · 71%
| Integrations & Tool Calling | Full |
|---|---|
|
Tools are hosted as functions, general purpose code that deployed prompts can call to perform operations or reach external data. They run serverless in an isolated environment, and any function can be used as a tool. Prompts can also connect to public MCP servers, after OAuth authentication, to reach external tools and data. SourceBraintrust, braintrust.dev/docs deploy/functions and evaluate/write-prompts (MCP servers)read 2026-09-21 |
|
| Workflow Orchestration | Partial |
|
Workflows are hosted functions that chain two or more prompts and pass outputs between them. Prompts, tools and scorers also compose into applications. Functions are versioned and can be tagged for production, staging or development environments, so each sequence can be versioned and traced. Workflows do not include branching, loops, retries or fallback paths, or deterministic steps mixed with agent steps. SourceBraintrust, braintrust.dev/docs deploy/functions, deploy/prompts and deploy/environmentsread 2026-09-21 |
|
| Knowledge Grounding & RAG | Not documented |
|
Datasets hold test examples, and traces record the context a customer's retrieval returned. Braintrust's example RAG agent is built from a vector search tool the customer writes and a prompt. Braintrust does not ingest documents or keep an index or retrieval layer over company data. SourceBraintrust, braintrust.dev/docs deploy/functions and annotate/datasetsread 2026-09-21 |
|
| Human Oversight & Guardrails | Full |
|
Loop, Braintrust's agent, does read only work without interrupting. In an interactive thread it pauses for approval before it creates or edits anything. A scheduled Loop automation has nobody to ask, so each one carries write tool permissions that set what it may change, from nothing (read and report only) to named object types. Human review scores and review queues let people rate the customer's own traces. SourceBraintrust, braintrust.dev/docs loop, loop/automations and annotate/human-reviewread 2026-09-21 |
|
| Security, Identity & Governance | Full |
|
Sign in supports SSO and SAML with Okta Workforce, Microsoft Entra ID and Google Workspace, and OIDC for custom providers. Role based access control uses built in and custom permission groups set at organization and project level. API keys are stored as one way hashes and scoped to projects, provider secrets use AES-256 encryption, and all data is encrypted at rest and in transit. Retention policies are configurable, and an audit log records administrative actions such as permission changes and API key creation. Every plan includes SOC 2 Type II compliance, Enterprise adds a BAA for HIPAA, and a trust center sits at trust.braintrust.dev. SourceBraintrust, braintrust.dev/docs security and admin/audit-logs, and braintrust.dev/pricingread 2026-09-21 |
|
| Observability & Auditability | Full |
|
Traces record each AI request's nested steps, including model calls, tool calls, retrieved context, errors, cost and latency, and teams examine, filter and debug them in the trace view. Logs can be exported, with automatic export to the customer's own S3 bucket on Enterprise. Retention is 14 days on Starter, 30 days on Pro with paid extension, and custom on Enterprise. Administrative actions go to a separate audit log. SourceBraintrust, braintrust.dev/llms.txt, braintrust.dev/docs observe, admin/data-management/export and admin/audit-logs, and braintrust.dev/pricingread 2026-09-21 |
|
| Memory & State Persistence | Not documented |
|
Loop's threads persist so a person can return to an investigation, and traces, datasets and experiments are kept as records. Agents have no session, workflow or long term memory that they read and write across runs, and there are no controls to review or scope such memory. SourceBraintrust, braintrust.dev/docs loop and deploy/functionsread 2026-09-21 |
|
| Deployment & Data Residency | Full |
|
Customers can run Braintrust three ways. As SaaS, Braintrust operates both the control plane and the data plane. As BYOC, Braintrust operates the data plane inside the customer's cloud account. Self hosted on Enterprise, the customer deploys and controls the infrastructure that stores its AI data, with the data plane in its own VPC, while Braintrust provides the managed UI and platform. BYOC and self hosting keep logs, traces, datasets and prompts in the customer's account and region. SourceBraintrust, braintrust.dev/docs admin/self-hosting, admin/deployment/byoc and securityread 2026-09-21 |
|
| Prebuilt Agents, Templates & Packs | Partial |
|
Braintrust ships one ready made agent, Loop, which investigates project data, bootstraps scorers, generates test cases, iterates on prompts and builds charts. The same agent powers Patterns, a preconfigured automation that finds recurring failures, and the trace Debugger. There is no set of ready made agents or templates to choose from. SourceBraintrust, braintrust.dev/docs loop, loop/automations and observe/patterns, and braintrust.dev/pricingread 2026-09-21 |
|
| Triggers & Channel Coverage | Full |
|
Loop automations run Braintrust's agent on a daily, weekly, interval or cron schedule, with an instruction, a model and set write permissions, and send results to Slack or a webhook. Online scoring runs scorers automatically on incoming production logs. Alerts fire on log, time window and environment conditions. SourceBraintrust, braintrust.dev/docs loop/automations, evaluate/score-online and observe/alertsread 2026-09-21 |
|
| Model Flexibility & Routing | Full |
|
The Braintrust Gateway gives one API for routing model requests to AI providers. It supports the customer's own provider keys, configured at organization level, workload identity federation for supported providers, custom providers, and Braintrust's built in models billed against monthly credits. Deployed prompts carry their own model configuration, and Loop runs on the model and provider the customer chooses. SourceBraintrust, braintrust.dev/llms.txt and braintrust.dev/docs deploy/gateway, admin/ai-providers and loopread 2026-09-21 |
|
| APIs, SDKs & MCP Extensibility | Full |
|
Developers get a REST API with an OpenAPI description, SDKs in Python, TypeScript, Go, Java, Ruby and C#, and the bt CLI for running evals, querying logs, syncing data and managing functions. An MCP server, authenticated with OAuth 2.0 and PKCE, lets coding agents reason over project data. Custom tools, scorers and workflows are pushed from code as versioned functions, and evals run in CI. SourceBraintrust, braintrust.dev/llms.txt and braintrust.dev/docs security, deploy/functions and evaluate/run-in-ciread 2026-09-21 |
|
| Testing, Debugging & Optimization | Full |
|
Evals run an AI system over versioned datasets and score outputs with LLM as a judge, autoevals, custom code or human review scorers. Experiments are compared side by side, and evals run in CI, where configurable checks gate quality before production. Online scoring applies the same scorers to production logs, so quality is scored over time and failures become new dataset examples. SourceBraintrust, braintrust.dev/docs evaluate, run-in-ci, score-online and annotate/datasetsread 2026-09-21 |
|
| Browser & Computer Use | Not documented |
|
Custom code functions run in isolated serverless environments. No agent controls a browser, a desktop or a computer. SourceBraintrust, braintrust.dev/docs security and llms-full.txtread 2026-09-21 |
|
The Agentic Index coverage score grades every vendor Full, Partial or Not documented against the same 14 buyer facing capabilities, from public evidence only. Each capability links to how all vendors in the index score on it. How this evidence is graded
Product updates
Coding agent insights, now in public preview for Braintrust SaaS and BYOC customers and for self hosted customers on data plane 2.13, traces a team's coding agent sessions from Claude Code, Codex, OpenCode, Pi, Antigravity or Grok. A dashboard breaks cost and tokens down by developer, model, tool, branch and API key, and a daily Loop automation surfaces recurring problems across sessions as findings in Patterns.
Bears on: Pricing / packaging
View sourceBraintrust announced Nitro, a new asynchronous query engine for Brainstore that separates storage reads from compute work to accelerate trace investigations. Nitro is automatically enabled for SaaS and BYOC customers, with support starting at dataplane 2.15 for deployments customers host themselves. Braintrust reports that full text searches ran more than twice as fast in its production comparison.
Bears on: Observability / auditability
View sourceBraintrust introduced support for tracing Cloudflare Agents. Users can ingest traces by exporting them via OpenTelemetry or by instrumenting their agents directly in JavaScript on the Cloudflare Workers runtime.
Bears on: Integrations
View sourcePricing
Starter is free with 1 GB of data and 10K scores. Pro is $249 a month.
Usage. A flat monthly platform fee per plan covers set amounts of processed data and scores, use beyond them is metered, and there are no per seat fees.
Included quota
Starter includes 1 GB processed data and 10K scores a month, 14 day retention and unlimited seats. Pro includes 5 GB and 50K scores, 30 day retention and Google SSO. Neither plan has per seat fees.
What is public
Braintrust publishes full self serve pricing. Starter is free with 1 GB processed data and 10K scores a month, Pro is $249 a month with 5 GB and 50K scores plus published overage rates, and Enterprise is custom, with BYOC and self hosted deployment. Qualifying early stage startups can get 6 to 12 months of free Pro.
Billing mechanics
Each plan carries a flat monthly platform fee, plus metered on demand usage beyond the included processed data and scores. Pro overage rates are lower than Starter's. Topics and built in model usage draw from a separate monthly credit with uniform overage rates.
Cost watchouts
Usage past the included amounts is billed with no stated cap, at $4/GB and $2.50 per 1K scores on Starter, $3/GB and $1.50 per 1K scores on Pro, token rates for model use past the monthly credits, and $0.50 per GB a month for retention beyond 30 days on Pro. SAML SSO, custom roles, audit logs, the BAA and self hosting are Enterprise only.
Variable cost rationale
Braintrust meters processed data, scores and model use. The platform fee is flat, and past the included amounts each is billed on demand, at $4/GB and $2.50 per 1K scores on Starter, $3/GB and $1.50 per 1K scores on Pro, and token rates for model use past the monthly credits. Cost follows trace volume rather than seats, since users are unlimited. No cap on overage is published, so the included monthly allowances and credits are the only limit. Retention past 30 days on Pro adds $0.50 per GB a month.
Additional watchouts
Overage adds up quickly. A Pro team logging 10 GB and 100K scores a month pays about $339, not $249.
Sales call required
Mixed (some tiers require a call)
Free / trial
Free Starter plan with no card, 1 GB processed data, 10K scores and $10 model credits a month. Qualifying startups get 6 to 12 months of Pro free
Lowest paid plan
Pro at $249 per month with 5 GB processed data and 50K scores
Commercial notes
Unlimited users on every plan, so cost follows processed data, scores and model use rather than seats. Qualifying startups can get 6 to 12 months of Pro free.
Key ambiguities
Enterprise pricing is not published, and model use past the included credits is billed at token rates listed on a separate detailed pricing page.
Support SLA / resale
Starter has community support and Pro priority support. Enterprise adds dedicated support with guaranteed SLAs and an uptime SLA.
Missing data
Enterprise pricing is not published.
Related vendors
- AgentOps — Agent observability and debugging platform: open source SDKs trace…
- AgentPhone — AgentPhone gives the AI agents a team builds their own US or…
- Agno — Python agent framework and AgentOS runtime (formerly Phidata) for…
- AIsa — Resource and payment gateway for AI agents, with one key to 110+…
- Albato — Albato is a no code integration platform that sells an embedded,…
- AlphaBitCore — AI control plane for regulated financial firms: one gateway enforces…
Alternatives to Braintrust
The closest documented capability profiles to Braintrust among agent infrastructure platforms tracked by Agentic Index, ordered by similarity on the same 14 point evidence the rankings use. No vendor pays for placement.
- LangWatch10.0 / 14Matches Braintrust across all 14 documented capabilities
- F5 AI Guardrails9.0 / 14A lighter documented profile than Braintrust
- Galileo9.0 / 14A lighter documented profile than BraintrustBraintrust vs Galileo →
- Opik9.0 / 14A lighter documented profile than BraintrustBraintrust vs Opik →
- W&B Weave9.0 / 14A lighter documented profile than BraintrustBraintrust vs W&B Weave →
- Bernstein10.5 / 14Adds documented Memory & State Persistence
Similarity is computed from each vendor's Agentic Index coverage score evidence, axis by axis, not from the totals. How this evidence is graded