AgentOps vs Braintrust (2026)

The major agent frameworks and providers are instrumented natively, including AG2, Agno, CrewAI, the OpenAI Agents SDK, LangChain and LlamaIndex, and AgentOps traces the tool calls those agents make. These integrations bring trace data in, and none connects an agent to a real system to take actions.

SourceAgentOps, docs.agentops.ai/llms.txt (integrations) and agentops.ai homepageread 2026-09-21

Tools are hosted as functions, general purpose code that deployed prompts can call to perform operations or reach external data. They run serverless in an isolated environment, and any function can be used as a tool. Prompts can also connect to public MCP servers, after OAuth authentication, to reach external tools and data.

SourceBraintrust, braintrust.dev/docs deploy/functions and evaluate/write-prompts (MCP servers)read 2026-09-21

Multi agent workflows the customer runs are visualized from their traces, with operations tied to named agents. AgentOps does not run workflows of its own, and nothing in it sequences, branches or retries steps or combines deterministic nodes with agent steps.

SourceAgentOps, docs.agentops.ai/llms.txt (tracking agents, recording operations) and agentops.ai homepageread 2026-09-21

Workflows are hosted functions that chain two or more prompts and pass outputs between them. Prompts, tools and scorers also compose into applications. Functions are versioned and can be tagged for production, staging or development environments, so each sequence can be versioned and traced. Workflows do not include branching, loops, retries or fallback paths, or deterministic steps mixed with agent steps.

SourceBraintrust, braintrust.dev/docs deploy/functions, deploy/prompts and deploy/environmentsread 2026-09-21

When the customer's instrumented code runs, traces arrive and appear in the dashboard. AgentOps names no schedule, event trigger, alert or automation that starts work on its own.

SourceAgentOps, docs.agentops.ai/llms.txt and agentops.ai homepageread 2026-09-21

Loop automations run Braintrust's agent on a daily, weekly, interval or cron schedule, with an instruction, a model and set write permissions, and send results to Slack or a webhook. Online scoring runs scorers automatically on incoming production logs. Alerts fire on log, time window and environment conditions.

SourceBraintrust, braintrust.dev/docs loop/automations, evaluate/score-online and observe/alertsread 2026-09-21

Calls a customer's agents make are traced, including retrieval steps where the customer instruments them. AgentOps names no document ingestion, index, retrieval layer or knowledge API that grounds an agent in company data.

SourceAgentOps, docs.agentops.ai/llms.txtread 2026-09-21

Datasets hold test examples, and traces record the context a customer's retrieval returned. Braintrust's example RAG agent is built from a vector search tool the customer writes and a prompt. Braintrust does not ingest documents or keep an index or retrieval layer over company data.

SourceBraintrust, braintrust.dev/docs deploy/functions and annotate/datasetsread 2026-09-21

Traces, sessions and completions are stored as records of the customer's agent runs. AgentOps names no session, workflow or long term memory that an agent reads and writes.

SourceAgentOps, docs.agentops.ai/llms.txt (traces, core concepts)read 2026-09-21

Loop's threads persist so a person can return to an investigation, and traces, datasets and experiments are kept as records. Agents have no session, workflow or long term memory that they read and write across runs, and there are no controls to review or scope such memory.

SourceBraintrust, braintrust.dev/docs loop and deploy/functionsread 2026-09-21

Logs, errors and prompt injection attacks leave a data trail, and runs can be replayed for review. AgentOps names no approval step, consent checkpoint, runtime guardrail or pause and resume control over an agent's actions.

SourceAgentOps, agentops.ai homepage and docs.agentops.ai/llms.txtread 2026-09-21

Loop, Braintrust's agent, does read only work without interrupting. In an interactive thread it pauses for approval before it creates or edits anything. A scheduled Loop automation has nobody to ask, so each one carries write tool permissions that set what it may change, from nothing (read and report only) to named object types. Human review scores and review queues let people rate the customer's own traces.

SourceBraintrust, braintrust.dev/docs loop, loop/automations and annotate/human-reviewread 2026-09-21

The Pro plan adds role based permissioning, and the Enterprise plan adds custom SSO, a custom data retention policy and on premise or self hosted deployment. Self hosting lets a customer apply its own security policies. The plans mention compliance standards, with no published report or auditor behind them.

SourceAgentOps, agentops.ai homepage pricing section and docs.agentops.ai/v2/self-hosting/overviewread 2026-09-21

Sign in supports SSO and SAML with Okta Workforce, Microsoft Entra ID and Google Workspace, and OIDC for custom providers. Role based access control uses built in and custom permission groups set at organization and project level.

API keys are stored as one way hashes and scoped to projects, provider secrets use AES-256 encryption, and all data is encrypted at rest and in transit. Retention policies are configurable, and an audit log records administrative actions such as permission changes and API key creation. Every plan includes SOC 2 Type II compliance, Enterprise adds a BAA for HIPAA, and a trust center sits at trust.braintrust.dev.

SourceBraintrust, braintrust.dev/docs security and admin/audit-logs, and braintrust.dev/pricingread 2026-09-21

Each agent run is recorded as a trace of spans for LLM calls, tool calls, operations and named agents, with the host environment captured automatically. The dashboard replays runs step by step with point in time precision, visualizes multi agent interactions and tracks tokens and cost. The Pro plan adds session and event export and unlimited log retention. A read only public API and an MCP server expose trace and span data.

SourceAgentOps, docs.agentops.ai/llms.txt (traces, spans, host environment, public API, MCP server) and agentops.ai homepageread 2026-09-21

Traces record each AI request's nested steps, including model calls, tool calls, retrieved context, errors, cost and latency, and teams examine, filter and debug them in the trace view. Logs can be exported, with automatic export to the customer's own S3 bucket on Enterprise. Retention is 14 days on Starter, 30 days on Pro with paid extension, and custom on Enterprise. Administrative actions go to a separate audit log.

SourceBraintrust, braintrust.dev/llms.txt, braintrust.dev/docs observe, admin/data-management/export and admin/audit-logs, and braintrust.dev/pricingread 2026-09-21

Customers can use the hosted service or self host the whole platform (API server, dashboard, Supabase and ClickHouse data layers and an OpenTelemetry collector) on their own servers or cloud through Docker Compose, Kubernetes or deployment to AWS, GCP and Azure, so data stays on their own infrastructure. The Enterprise plan adds on premise deployment, self hosting on AWS, GCP and Azure, and a custom data retention policy.

SourceAgentOps, docs.agentops.ai/v2/self-hosting/overview and agentops.ai homepage; docs.agentops.ai/v2/self-hosting/overview.mdread 2026-09-21

Customers can run Braintrust three ways. As SaaS, Braintrust operates both the control plane and the data plane. As BYOC, Braintrust operates the data plane inside the customer's cloud account. Self hosted on Enterprise, the customer deploys and controls the infrastructure that stores its AI data, with the data plane in its own VPC, while Braintrust provides the managed UI and platform. BYOC and self hosting keep logs, traces, datasets and prompts in the customer's account and region.

SourceBraintrust, braintrust.dev/docs admin/self-hosting, admin/deployment/byoc and securityread 2026-09-21

Examples show how to instrument agents built with various frameworks, and AgentOps offers expert help building agents as a service. It names no ready made agents, templates or packaged workflows for customers to adopt.

SourceAgentOps, agentops.ai homepage and docs.agentops.ai/llms.txt (examples)read 2026-09-21

Braintrust ships one ready made agent, Loop, which investigates project data, bootstraps scorers, generates test cases, iterates on prompts and builds charts. The same agent powers Patterns, a preconfigured automation that finds recurring failures, and the trace Debugger. There is no set of ready made agents or templates to choose from.

SourceBraintrust, braintrust.dev/docs loop, loop/automations and observe/patterns, and braintrust.dev/pricingread 2026-09-21

More than 400 LLMs and frameworks that a customer's agents call can be traced and costed, and tracing does not constrain which model the agent uses. Customers do not choose, route or key any model the product itself runs, and AgentOps names no model policy.

SourceAgentOps, agentops.ai homepage and docs.agentops.ai/llms.txt (tracking LLM calls)read 2026-09-21

The Braintrust Gateway gives one API for routing model requests to AI providers. It supports the customer's own provider keys, configured at organization level, workload identity federation for supported providers, custom providers, and Braintrust's built in models billed against monthly credits. Deployed prompts carry their own model configuration, and Loop runs on the model and provider the customer chooses.

SourceBraintrust, braintrust.dev/llms.txt and braintrust.dev/docs deploy/gateway, admin/ai-providers and loopread 2026-09-21

AgentOps publishes open source Python and TypeScript SDKs with decorators, context managers and manual trace control, a read only HTTP public API for trace and span data, and an MCP server that exposes trace and span data to MCP clients, alongside an MCP docs server.

SourceAgentOps, docs.agentops.ai/llms.txt (SDK reference, TypeScript SDK, public API, MCP server)read 2026-09-21

Developers get a REST API with an OpenAPI description, SDKs in Python, TypeScript, Go, Java, Ruby and C#, and the bt CLI for running evals, querying logs, syncing data and managing functions. An MCP server, authenticated with OAuth 2.0 and PKCE, lets coding agents reason over project data. Custom tools, scorers and workflows are pushed from code as versioned functions, and evals run in CI.

SourceBraintrust, braintrust.dev/llms.txt and braintrust.dev/docs security, deploy/functions and evaluate/run-in-ciread 2026-09-21

Teams debug agents by rewinding and replaying runs with point in time precision, and a data trail of errors and prompt injection attempts is kept. AgentOps also offers fine tuning of specialized models on saved completions as an optimization step after deployment. It names no testing against fixtures or datasets, configurable quality gates or scoring of output quality over time.

SourceAgentOps, agentops.ai homepage and docs.agentops.ai/llms.txtread 2026-09-21

Evals run an AI system over versioned datasets and score outputs with LLM as a judge, autoevals, custom code or human review scorers. Experiments are compared side by side, and evals run in CI, where configurable checks gate quality before production. Online scoring applies the same scorers to production logs, so quality is scored over time and failures become new dataset examples.

SourceBraintrust, braintrust.dev/docs evaluate, run-in-ci, score-online and annotate/datasetsread 2026-09-21

The product covers SDK instrumentation, tracing, the dashboard, a public API, an MCP server and self hosting. AgentOps names no browser, desktop or computer control by an agent.

SourceAgentOps, docs.agentops.ai/llms.txtread 2026-09-21

Custom code functions run in isolated serverless environments. No agent controls a browser, a desktop or a computer.

SourceBraintrust, braintrust.dev/docs security and llms-full.txtread 2026-09-21