Traceloop

OpenLLMetry instruments more than twenty model providers, vector databases such as Pinecone and Chroma and frameworks such as LangChain, LlamaIndex and CrewAI, exports to more than twenty-five observability backends, and Traceloop connects to Posthog, Slack and GitHub. These bring the customer's agent telemetry in and send results out. The customer's agent calls its own tools, and no integration through which an agent reads or writes a real system is described.

SourceTraceloop, traceloop.com/docs supported integrations, integrations overview, Posthog, Slack and GitHubread 2026-09-21

Applications the customer builds in its own code or frameworks are observed and evaluated here, and the Traceloop Hub runs plugin pipelines around each model request. No sequencing, branching, retries or routing of agent steps run by Traceloop is documented.

SourceTraceloop, traceloop.com/docs Hub configuration and introductionread 2026-09-21

Traceloop traces vector database calls and evaluates whether retrieved context answers the question. No document ingestion, index, retrieval layer or knowledge API that grounds an agent in company data is documented.

SourceTraceloop, traceloop.com/docs introduction and evaluatorsread 2026-09-21

Guardrails are evaluators applied through the SDK's @guardrail decorator. They run synchronously before an output returns, intercept and block or modify unsafe, off policy or low quality responses, stack in layers for input, output and context checks, and log every execution for audit.

SourceTraceloop, traceloop.com/docs guardrailsread 2026-09-21

Compliance is documented, access control is not. Traceloop states it is SOC 2 and HIPAA compliant, lists SOC 2 Compliance on the Enterprise plan and links a trust center at trust.traceloop.com. No SSO, SAML, role model or permission scheme is documented on the pricing page or in the docs. Custom data retention on Enterprise, per user controls over logging prompts and responses and an API to delete a user's trace data are privacy controls, not a model of who inside the customer's organization can do what.

SourceTraceloop, traceloop.com/pricing and docs.traceloop.com privacy APIread 2026-10-01

Every LLM and vector database call is traced as OpenTelemetry spans by the open source OpenLLMetry SDK, with workflow and task annotations, sessions for multi turn conversations, workflow and prompt versions, user feedback and multimodal logging. The traces flow to the Traceloop platform or to Datadog, Dynatrace, Grafana, Splunk, New Relic and other OpenTelemetry backends, and a spans API returns them programmatically. Retention is 24 hours on the free plan and custom on Enterprise.

SourceTraceloop, traceloop.com/docs sessions, annotations, integrations overview and get spans API, and traceloop.com/pricingread 2026-09-21

Traceloop stores traces, sessions, datasets and experiment results for observability and testing. No session, workflow or long term memory that a customer's agent reads and writes through Traceloop is documented.

SourceTraceloop, traceloop.com/docs sessions and datasetsread 2026-09-21

Three deployment routes are documented: a managed cloud, a hybrid deployment where all trace data stays on the customer's infrastructure while Traceloop runs the processing and dashboard, and a full platform self hosted on Kubernetes in AWS, GCP, Azure or any cluster, including air gapped environments. OpenLLMetry can also send data to a backend the customer already runs instead.

SourceTraceloop, traceloop.com/docs self-hosting overview, hybrid and full deployment, and traceloop.com/pricingread 2026-09-21

Traceloop ships automatic instrumentations for providers, vector databases and frameworks and a library of preconfigured evaluators. No ready made agents, templates or packaged workflows a buyer adopts for its own work are documented.

SourceTraceloop, traceloop.com/docs made by Traceloop evaluators and supported integrationsread 2026-09-21

Monitors run an evaluator in real time on every production span that matches a filter, and auto monitor setups create monitors programmatically. The Slack integration sends daily or weekly updates on a schedule, and experiments run on each pull request through GitHub. Work starts on an incoming span, a schedule or a code change without a person asking.

SourceTraceloop, traceloop.com/docs monitors, Slack and GitHub integrationsread 2026-09-21

A gateway, Traceloop Hub, takes the customer's own providers and models, including the same model on different providers such as GPT-4o on Azure and on OpenAI. Its pipelines use a model-router plugin that routes each request across the listed models and switches between them according to availability. The playground and experiments run prompts with full model configuration. The customer controls model choice and can see the fallback policy.

SourceTraceloop, traceloop.com/docs Hub configuration and getting startedread 2026-09-21

A REST API with a published OpenAPI specification covers spans, costs, metrics, evaluators, auto monitor setups, organizations and privacy requests. The Apache-2.0 OpenLLMetry SDKs for Python, TypeScript, Go and Ruby also emit standard OpenTelemetry that any compatible backend consumes. Experiments run from CI through the GitHub integration, so the API and SDKs fit into a CI/CD pipeline.

SourceTraceloop, traceloop.com/docs API reference, OpenLLMetry getting started and GitHub integrationread 2026-09-21

Experiments run over managed datasets from the SDK or in CI, posting evaluation results into GitHub pull requests. Outputs are scored with a library of built in evaluators for faithfulness, relevance, agent goal completion, tool selection and trajectory, and with custom LLM-as-a-judge evaluators trained on annotated examples. The same evaluators run continuously on production traffic as monitors.

SourceTraceloop, traceloop.com/docs experiments, GitHub integration, evaluators and monitorsread 2026-09-21

Traceloop documents tracing, evaluation, monitoring, guardrails, prompt management and a model gateway, and no browser, desktop or computer control by an agent is documented.

SourceTraceloop, traceloop.com/docs llms.txt and introductionread 2026-09-21