Braintrust

Tools are hosted as functions, general-purpose code that deployed prompts can call to perform operations or access external data, running serverless in an isolated environment, and any function can be used as a tool; prompts can also connect to public MCP servers after OAuth authentication to reach external tools and data. That gives agents custom, actionable tools.

SourceBraintrust, braintrust.dev/docs deploy/functions and evaluate/write-prompts (MCP servers)read 2026-09-21

Workflows are hosted functions that chain two or more prompts and pass outputs between them, alongside prompts, tools and scorers that compose into applications, and functions are versioned and can be tagged for production, staging or development environments. That sequencing is versioned and traceable. Branching, loops, retries and fallback paths, and deterministic nodes mixed with agent steps, are not documented.

SourceBraintrust, braintrust.dev/docs deploy/functions, deploy/prompts and deploy/environmentsread 2026-09-21

Datasets hold test examples, traces record the context a customer's retrieval returned, and the documentation shows a RAG agent built from a customer-written vector search tool and a prompt. No document ingestion, index or retrieval layer that Braintrust maintains over company data is documented.

SourceBraintrust, braintrust.dev/docs deploy/functions and annotate/datasetsread 2026-09-21

Loop, Braintrust's agent, runs read-only work without interrupting and pauses for approval before it creates or edits anything in an interactive thread; a scheduled Loop automation has nobody to ask, so each one carries write tool permissions that set what it may change, from none (read and report only) to named object types. Human review scores and review queues let people rate the customer's own traces. That puts an approval step before an agent's changes and sets different autonomy levels for scheduled and supervised runs.

SourceBraintrust, braintrust.dev/docs loop, loop/automations and annotate/human-reviewread 2026-09-21

Braintrust supports SSO and SAML with Okta Workforce, Microsoft Entra ID and Google Workspace and OIDC for custom providers, role-based access control with built-in and custom permission groups set at organization and project level, API keys stored as one-way hashes and scoped to projects, AES-256 encryption for provider secrets and encryption of all data at rest and in transit, configurable retention policies, and an audit log of administrative actions such as permission changes and API key creation. The pricing page lists SOC 2 Type II compliance on every plan and a BAA for HIPAA on Enterprise, with a trust center at trust.braintrust.dev.

SourceBraintrust, braintrust.dev/docs security and admin/audit-logs, and braintrust.dev/pricingread 2026-09-21

Each AI request's nested steps, including model calls, tool calls, retrieved context, errors, cost and latency, are recorded in Braintrust traces, and teams examine, filter and debug them in the trace view; logs can be exported, including automatic export to the customer's own S3 bucket on Enterprise, retention is 14 days on Starter, 30 days on Pro with paid extension, and custom on Enterprise, and administrative actions go to a separate audit log.

SourceBraintrust, braintrust.dev/llms.txt, braintrust.dev/docs observe, admin/data-management/export and admin/audit-logs, and braintrust.dev/pricingread 2026-09-21

Loop's threads persist so a person can return to an investigation, and traces, datasets and experiments persist as records. No session, workflow or long-term memory that an agent reads and writes across runs, or controls to review and scope it, is documented.

SourceBraintrust, braintrust.dev/docs loop and deploy/functionsread 2026-09-21

Braintrust runs as SaaS, where it operates both the control plane and the data plane; as BYOC, where Braintrust operates the data plane inside the customer's cloud account; or self-hosted on Enterprise, where the customer deploys and controls the infrastructure that stores its AI data while Braintrust provides the managed UI and platform, with the data plane in the customer's own VPC. BYOC and self-hosted keep logs, traces, datasets and prompts in the customer's account and region.

SourceBraintrust, braintrust.dev/docs admin/self-hosting, admin/deployment/byoc and securityread 2026-09-21

As its ready-made agent, Braintrust ships Loop, which investigates project data, bootstraps scorers, generates test cases, iterates on prompts and builds charts, and the same agent powers Patterns, whose preconfigured automation finds recurring failures, and the trace Debugger. That is one packaged agent with preset uses rather than a set of ready-made agents or templates a buyer selects among.

SourceBraintrust, braintrust.dev/docs loop, loop/automations and observe/patterns, and braintrust.dev/pricingread 2026-09-21

Loop automations run Braintrust's agent on a daily, weekly, interval or cron schedule with an instruction, a model and set write permissions, sending results to Slack or a webhook; online scoring runs scorers automatically on incoming production logs, and alerts fire on log, time-window and environment conditions.

SourceBraintrust, braintrust.dev/docs loop/automations, evaluate/score-online and observe/alertsread 2026-09-21

The Braintrust Gateway gives one API for routing model requests to AI providers, with the customer's own provider keys configured at organization level, workload identity federation for supported providers, custom providers, and Braintrust's built-in models billed against monthly credits; deployed prompts carry their own model configuration, and Loop runs on the customer's chosen model and provider. The customer controls model choice with its own providers and keys.

SourceBraintrust, braintrust.dev/llms.txt and braintrust.dev/docs deploy/gateway, admin/ai-providers and loopread 2026-09-21

Braintrust publishes a REST API with an OpenAPI description, SDKs in Python, TypeScript, Go, Java, Ruby and C#, the bt CLI for running evals, querying logs, syncing data and managing functions, and an MCP server authenticated with OAuth 2.0 and PKCE for reasoning over project data from coding agents; custom tools, scorers and workflows are pushed from code as versioned functions, and evals run in CI.

SourceBraintrust, braintrust.dev/llms.txt and braintrust.dev/docs security, deploy/functions and evaluate/run-in-ciread 2026-09-21

Evals run an AI system over versioned datasets and score outputs with LLM-as-a-judge, autoevals, custom code or human review scorers, experiments are compared side by side, evals run in CI, and online scoring applies the same scorers to production logs so failures become new dataset examples. That tests on datasets before production, gates quality with configurable checks, scores quality over time and closes the loop after deployment.

SourceBraintrust, braintrust.dev/docs evaluate, run-in-ci, score-online and annotate/datasetsread 2026-09-21

Custom code functions run in isolated serverless environments, and the documentation covers tracing, evaluation, deployment and administration, but no browser, desktop or computer control by an agent is documented.

SourceBraintrust, braintrust.dev/docs security and llms-full.txtread 2026-09-21