Braintrust vs Opik (2026)

Tools are hosted as functions, general purpose code that deployed prompts can call to perform operations or reach external data. They run serverless in an isolated environment, and any function can be used as a tool. Prompts can also connect to public MCP servers, after OAuth authentication, to reach external tools and data.

SourceBraintrust, braintrust.dev/docs deploy/functions and evaluate/write-prompts (MCP servers)read 2026-09-21

Integrations cover more than 40 AI frameworks, model providers and gateways, including LangChain, OpenAI, Google ADK, LangGraph and CrewAI, and alerts send webhooks out. These bring trace data in and send notifications, and none connects an agent to a business system to take actions. Through Opik Connect, Ollie reads the customer's agent codebase, writes approved fixes to it and reruns the agent from a failing trace. That is the one path that takes action, and it reaches only the local codebase.

SourceComet, comet.com/site/pricing and comet.com/docs/opik/llms.txt (Ollie, integrations)read 2026-09-21

Workflows are hosted functions that chain two or more prompts and pass outputs between them. Prompts, tools and scorers also compose into applications. Functions are versioned and can be tagged for production, staging or development environments, so each sequence can be versioned and traced. Workflows do not include branching, loops, retries or fallback paths, or deterministic steps mixed with agent steps.

SourceBraintrust, braintrust.dev/docs deploy/functions, deploy/prompts and deploy/environmentsread 2026-09-21

From traces, Opik draws the customer's agent execution graphs, the Agent Playground runs the customer's agent locally while traced, and agent configuration versions prompts, model settings and tool definitions outside the codebase. Opik runs no workflows that sequence, branch or retry steps, or that combine deterministic nodes with agent steps.

SourceComet, comet.com/docs/opik/llms.txt (development overview, agent playground) and comet.com/site/pricingread 2026-09-21

Loop automations run Braintrust's agent on a daily, weekly, interval or cron schedule, with an instruction, a model and set write permissions, and send results to Slack or a webhook. Online scoring runs scorers automatically on incoming production logs. Alerts fire on log, time window and environment conditions.

SourceBraintrust, braintrust.dev/docs loop/automations, evaluate/score-online and observe/alertsread 2026-09-21

Online evaluation rules score production traces automatically as they are logged, Diagnostics scans a project's traces with Ollie and surfaces recurring agent issues with a root cause and suggested fix, and alerts send webhook notifications on events such as trace errors, new feedback scores and prompt changes. Evaluation and investigation start on incoming traces without a person asking.

SourceComet, comet.com/docs/opik/llms.txt (online evaluation rules, diagnostics, alerts) and comet.com/site/pricingread 2026-09-21

Datasets hold test examples, and traces record the context a customer's retrieval returned. Braintrust's example RAG agent is built from a vector search tool the customer writes and a prompt. Braintrust does not ingest documents or keep an index or retrieval layer over company data.

SourceBraintrust, braintrust.dev/docs deploy/functions and annotate/datasetsread 2026-09-21

RAG metrics in Opik score retrieval quality. Opik also traces the context a customer's retrieval returns and generates synthetic examples from traces for datasets. There is no document ingestion, index, retrieval layer or knowledge API that grounds an agent in company data.

SourceComet, comet.com/docs/opik/llms.txt and comet.com/site/pricingread 2026-09-21

Loop's threads persist so a person can return to an investigation, and traces, datasets and experiments are kept as records. Agents have no session, workflow or long term memory that they read and write across runs, and there are no controls to review or scope such memory.

SourceBraintrust, braintrust.dev/docs loop and deploy/functionsread 2026-09-21

The customer's traces, sessions and threads are recorded, datasets and prompts are versioned, and an offline fallback replays tracing messages after an outage. There is no session, workflow or long-term memory that an agent reads and writes.

SourceComet, comet.com/docs/opik/llms.txt (tracing concepts, offline fallback) and comet.com/site/pricingread 2026-09-21

Loop, Braintrust's agent, does read only work without interrupting. In an interactive thread it pauses for approval before it creates or edits anything. A scheduled Loop automation has nobody to ask, so each one carries write tool permissions that set what it may change, from nothing (read and report only) to named object types. Human review scores and review queues let people rate the customer's own traces.

SourceBraintrust, braintrust.dev/docs loop, loop/automations and annotate/human-reviewread 2026-09-21

Guardrails in Opik run inline with LLM calls on live traffic and can stop or alter a response before it reaches a user, with five guard types (PII, allowed and restricted topics, prompt injection and jailbreak, an LLM-as-a-judge check, and a custom classifier) grouped into named policies that can be enforced workspace-wide, and every run logged as a span. Ollie, Opik's assistant, proposes code edits that a person reviews and approves before anything changes on disk.

SourceComet, comet.com/docs/opik/guardrails/overview and comet.com/site/pricing (Ollie, Opik Connect); comet.com/docs/opik/guardrails/overview.mdread 2026-09-21

Sign in supports SSO and SAML with Okta Workforce, Microsoft Entra ID and Google Workspace, and OIDC for custom providers. Role based access control uses built in and custom permission groups set at organization and project level.

API keys are stored as one way hashes and scoped to projects, provider secrets use AES-256 encryption, and all data is encrypted at rest and in transit. Retention policies are configurable, and an audit log records administrative actions such as permission changes and API key creation. Every plan includes SOC 2 Type II compliance, Enterprise adds a BAA for HIPAA, and a trust center sits at trust.braintrust.dev.

SourceBraintrust, braintrust.dev/docs security and admin/audit-logs, and braintrust.dev/pricingread 2026-09-21

The Enterprise plan includes enterprise SSO over OAuth 2.0, SAML and LDAP with SSO enforcement, custom role-based access at project and organization level, view-only users, service accounts, and SOC 2, ISO 27001, ISO 9001, HIPAA and GDPR compliance. Opik also supports SAML, OIDC and JWT authentication and workspace roles and permissions, and Comet says Opik produces audit logs for governance teams.

SourceComet, comet.com/site/pricing, comet.com/docs/opik/llms.txt (authentication, roles and permissions) and comet.com/site/products/opikread 2026-09-21

Traces record each AI request's nested steps, including model calls, tool calls, retrieved context, errors, cost and latency, and teams examine, filter and debug them in the trace view. Logs can be exported, with automatic export to the customer's own S3 bucket on Enterprise. Retention is 14 days on Starter, 30 days on Pro with paid extension, and custom on Enterprise. Administrative actions go to a separate audit log.

SourceBraintrust, braintrust.dev/llms.txt, braintrust.dev/docs observe, admin/data-management/export and admin/audit-logs, and braintrust.dev/pricingread 2026-09-21

Every step of an application's execution is traced, from context retrieval to model responses and tool calls, with agent execution graphs, sessions for multi-turn conversations, token and cost tracking and error surfacing. Traces, spans, threads, datasets and experiments export through the SDKs, REST API, UI and command line. Span retention is 60 days on Free and Pro and custom on Enterprise, and Comet says Opik produces audit logs for governance teams.

SourceComet, comet.com/site/pricing, comet.com/docs/opik/llms.txt (observability, export data) and comet.com/site/products/opikread 2026-09-21

Customers can run Braintrust three ways. As SaaS, Braintrust operates both the control plane and the data plane. As BYOC, Braintrust operates the data plane inside the customer's cloud account. Self hosted on Enterprise, the customer deploys and controls the infrastructure that stores its AI data, with the data plane in its own VPC, while Braintrust provides the managed UI and platform. BYOC and self hosting keep logs, traces, datasets and prompts in the customer's account and region.

SourceBraintrust, braintrust.dev/docs admin/self-hosting, admin/deployment/byoc and securityread 2026-09-21

The open source build can be downloaded, installed and run on the customer's own infrastructure from the same codebase as the hosted versions. Opik Cloud stores data in the US on the Free and Pro plans, and Enterprise offers flexible deployments across cloud, on-premises and fully managed options with a custom data region.

SourceComet, comet.com/site/pricingread 2026-09-21

Braintrust ships one ready made agent, Loop, which investigates project data, bootstraps scorers, generates test cases, iterates on prompts and builds charts. The same agent powers Patterns, a preconfigured automation that finds recurring failures, and the trace Debugger. There is no set of ready made agents or templates to choose from.

SourceBraintrust, braintrust.dev/docs loop, loop/automations and observe/patterns, and braintrust.dev/pricingread 2026-09-21

Opik ships Ollie, a ready-made assistant that reads traces, searches the workspace, builds test suites, proposes fixes and, through Opik Connect, edits and reruns the customer's agent. The same assistant powers Diagnostics, which scans traces for recurring issues. It is one packaged agent with preset uses, not a set of ready-made agents or templates a customer selects among.

SourceComet, comet.com/docs/opik/llms.txt (Ollie, diagnostics) and comet.com/site/pricingread 2026-09-21

The Braintrust Gateway gives one API for routing model requests to AI providers. It supports the customer's own provider keys, configured at organization level, workload identity federation for supported providers, custom providers, and Braintrust's built in models billed against monthly credits. Deployed prompts carry their own model configuration, and Loop runs on the model and provider the customer chooses.

SourceBraintrust, braintrust.dev/llms.txt and braintrust.dev/docs deploy/gateway, admin/ai-providers and loopread 2026-09-21

Through its Gateway, Opik gives a consistent API across multiple LLM providers, workspace AI Providers settings configure connections to providers, built-in LLM-as-a-judge metrics can run on a custom model, the Agent Optimizer works with OpenAI, Anthropic, Gemini, Azure and Ollama, and agent configuration keeps model settings outside the codebase, versioned and changeable without redeploying. Model choice stays with the customer, using its own providers.

SourceComet, comet.com/docs/opik/llms.txt (Gateway, AI Providers, custom model, configuring LLM providers) and comet.com/site/pricing (agent configuration)read 2026-09-21

Developers get a REST API with an OpenAPI description, SDKs in Python, TypeScript, Go, Java, Ruby and C#, and the bt CLI for running evals, querying logs, syncing data and managing functions. An MCP server, authenticated with OAuth 2.0 and PKCE, lets coding agents reason over project data. Custom tools, scorers and workflows are pushed from code as versioned functions, and evals run in CI.

SourceBraintrust, braintrust.dev/llms.txt and braintrust.dev/docs security, deploy/functions and evaluate/run-in-ciread 2026-09-21

There is a REST API with an OpenAPI specification and a complete Python client that work against both the open source platform and Opik Cloud, along with Python and TypeScript SDKs, native OpenTelemetry ingestion for other languages, the opik command line, and an MCP server that lets an AI coding assistant instrument code, query traces and manage prompts.

SourceComet, comet.com/docs/opik/llms.txt and comet.com/site/pricingread 2026-09-21

Evals run an AI system over versioned datasets and score outputs with LLM as a judge, autoevals, custom code or human review scorers. Experiments are compared side by side, and evals run in CI, where configurable checks gate quality before production. Online scoring applies the same scorers to production logs, so quality is scored over time and failures become new dataset examples.

SourceBraintrust, braintrust.dev/docs evaluate, run-in-ci, score-online and annotate/datasetsread 2026-09-21

Test suites run pass or fail assertions at item and suite level with execution policies for how many runs must pass, experiments test an application over datasets with more than 30 built-in and custom LLM-as-a-judge, heuristic and code metrics for single steps, whole agents and full conversations, annotation queues gather expert review, online evaluation scores production traces, and the Agent Optimizer tunes prompts, tools and parameters with several algorithms.

SourceComet, comet.com/site/pricing and comet.com/docs/opik/llms.txt (test suites, online evaluation, Agent Optimizer)read 2026-09-21

Custom code functions run in isolated serverless environments. No agent controls a browser, a desktop or a computer.

SourceBraintrust, braintrust.dev/docs security and llms-full.txtread 2026-09-21

Opik covers tracing, evaluation, optimization, guardrails and administration, and there is no browser, desktop or computer control by an agent. The Agent Playground runs the customer's own agent locally with tracing.

SourceComet, comet.com/site/pricing and comet.com/docs/opik/llms.txtread 2026-09-21