Freeplay vs HoneyHive (2026)

Tool schemas are versioned inside prompt templates, and Freeplay records the tool calls a customer's agent makes. It integrates with LangGraph, the Vercel AI SDK, Google ADK and OpenTelemetry to capture traces, and it posts automation alerts to Slack. The customer's own code executes every tool call. There is no connector that lets an agent read or write a real system through Freeplay.

SourceFreeplay, docs.freeplay.ai tool calls, AI framework integrations, saved searches and automationsread 2026-09-21

More than fifty libraries are instrumented automatically, including LangChain, LangGraph, AWS Strands, Google ADK and the OpenAI Agents SDK. HoneyHive accepts traces from third party agent platforms such as ServiceNow, Microsoft Copilot Studio and Salesforce Agentforce and sends alerts by email, Slack or webhook. These bring trace data in and send notifications out, and none connects an agent to a real system to take actions.

SourceHoneyHive, honeyhive.ai/llms.txt and pricing, and docs.honeyhive.ai alertsread 2026-09-21

However the customer orchestrates its agents, Freeplay observes, evaluates and tests agents built elsewhere and records multi prompt chains as traces. It does not sequence, branch, retry or route agent steps itself.

SourceFreeplay, docs.freeplay.ai agents in Freeplay, multi-prompt chain reciperead 2026-09-21

The customer's own agent workflows are drawn as graphs and trajectories from their traces. HoneyHive itself runs no workflows that sequence, branch or retry steps, or that combine deterministic nodes with agent steps.

SourceHoneyHive, docs.honeyhive.ai graph-view and trajectory-viewread 2026-09-21

Automations run in the background, monitor production traffic continuously and act when logs match a saved search. They run evaluations, route completions to a review queue, add them to a dataset or post a Slack notification. Evaluation Insights runs a weekly AI analysis of production evaluation data.

SourceFreeplay, docs.freeplay.ai saved searches and automations, evaluation insightsread 2026-09-21

Online evaluations run enabled evaluators automatically and asynchronously on incoming traces that match their filters, at a set sampling rate, and alerts check cost, latency, error and evaluator-score thresholds on a regular cycle and notify by email, Slack or webhook when one is crossed. Evaluation starts on an incoming trace or a monitoring cycle, with no person asking.

SourceHoneyHive, docs.honeyhive.ai monitoring onlineevals, alerts overview and alertsread 2026-09-21

Retrieved context that a customer's application passes to a prompt is logged, and RAG pipelines can be tested end to end. There is no document ingestion, index, retrieval layer or knowledge API that grounds an agent in company data.

SourceFreeplay, docs.freeplay.ai end-to-end test runs and llms.txtread 2026-09-21

Evaluator templates score the customer's own retrieval for context relevance and answer faithfulness, and traces record the context that retrieval returned. There is no index, retrieval layer or knowledge API that grounds an agent's behavior in company data.

SourceHoneyHive, docs.honeyhive.ai evaluator-templates and tracing conceptsread 2026-09-21

Sessions, traces, datasets and prompt versions are stored for observability and testing. In multi turn chat, the customer's application keeps the conversation history. There is no session, workflow or long term memory that an agent reads and writes through Freeplay.

SourceFreeplay, docs.freeplay.ai llms.txt and managing multi-turn chat history reciperead 2026-09-21

Sessions, events and datasets persist as observability and test records of what the customer's agent did, and a context retention evaluator scores whether that agent kept information across turns. There is no session, conversation, workflow or long term memory that an agent reads and writes.

SourceHoneyHive, docs.honeyhive.ai tracing concepts and evaluator-templatesread 2026-09-21

Review queues assign production completions and traces to named reviewers for human evaluation, and automations route guardrailed or low scoring responses into those queues so a person reviews them. Review happens after an output is produced. There is no approval step, checkpoint or pause that holds an agent's action for a person.

SourceFreeplay, docs.freeplay.ai review queues, saved searches and automationsread 2026-09-21

Annotation queues organize events for human review against human evaluators that define the fields a team rates, and those judgments become ground truth for automated evaluators. Feedback from end users can be attached to traces. People review the agent's outputs, but there is no approval step, consent checkpoint, runtime guardrail or control to pause and resume an agent's actions.

SourceHoneyHive, docs.honeyhive.ai annotation-queues, evaluators/human and setting-user-feedbackread 2026-09-21

SOC 2 Type II certification is in place, with details on Freeplay's trust center, and HIPAA Business Associate Agreements are offered. Customers get SAML single sign on through WorkOS with Okta, Microsoft Entra ID, Cisco Duo, OneLogin and JumpCloud, along with SCIM provisioning and deprovisioning. There are four account roles with role overrides for each project, mandatory multi factor authentication, revocable API keys and a configurable retention window on Enterprise.

SourceFreeplay, docs.freeplay.ai security overview, single sign-on and SCIM, user roles and access controls, data retentionread 2026-09-21

HoneyHive holds SOC 2 Type II, audited annually across all hosting options, with reports available through its Drata Trust Center, plus GDPR and HIPAA BAAs on Enterprise.

Users sign in with Google, GitHub or Microsoft SSO or SAML 2.0 with Okta, Entra ID, Google Workspace, OneLogin or Ping, with roles provisioned from SAML group claims and MFA for every account. Role based access applies at organization, workspace and project level, with custom roles on Enterprise.

Data is encrypted with AWS KMS at rest, with keys managed by the customer on dedicated and self hosted deployments, and with TLS 1.2+ in transit.

SourceHoneyHive, docs.honeyhive.ai setup/security and workspace rolesread 2026-09-21

Every LLM call is recorded as a completion tied to its prompt template version, and Freeplay groups completions into traces for agent flows and into sessions for whole interactions, records tool calls, and lets teams search and filter production logs by inputs, outputs, evaluation scores, metadata, cost and latency. It ingests OpenTelemetry traces, exports traces to CSV and exposes search endpoints for sessions, traces and completions. Logged data is kept 90 days by default, and the window is configurable on Enterprise plans.

SourceFreeplay, docs.freeplay.ai sessions, traces and completions, automations and filters, data retentionread 2026-09-21

Each agent run is recorded as a session of OpenTelemetry events, covering model calls, tool calls, retrieval and custom spans, and shown in tree, timeline, graph, trajectory and thread views. Trace data can be queried through the API, every plan can export data, and retention is 30 days on Developer and custom on Enterprise. The platform is built as an audit trail, so individual records cannot be deleted through the API or UI.

SourceHoneyHive, docs.honeyhive.ai tracing concepts, tree-view, graph-view, query-data and self-hosted data-flow, and honeyhive.ai/pricingread 2026-09-21

Besides the multi tenant cloud service, Enterprise customers can run Freeplay as Bring-Your-Own-Cloud inside their own AWS, GCP or Azure account, installed through Replicated KOTS or Helm. All prompts and responses then stay in the customer's cloud, and Freeplay never receives them. Single tenant private hosting over a site to site VPN is also offered.

SourceFreeplay, docs.freeplay.ai private deployment (BYOC) and security overviewread 2026-09-21

Multi tenant SaaS runs in AWS US-West-2. Dedicated Cloud keeps the data plane on physically isolated infrastructure in the customer's chosen AWS region, including the EU, with the control plane managed by HoneyHive. A hybrid option pairs a control plane managed by HoneyHive with a self hosted data plane, and a fully self hosted option deploys both planes in the customer's own account, with PrivateLink for private connectivity.

SourceHoneyHive, docs.honeyhive.ai setup managed, dedicated, self-hosted and security, and honeyhive.ai/pricingread 2026-09-21

Customers get evaluator types, an eval creation assistant, code recipes, and skills for coding agents that integrate with Freeplay. There are no ready made agents, templates or packaged workflows for customers to put to work.

SourceFreeplay, docs.freeplay.ai llms.txt, code recipes overview and developer resourcesread 2026-09-21

Three official agent skills, honeyhive-instrument, honeyhive-evaluate and honeyhive-improve, are reusable HoneyHive workflows that a coding agent follows. On Enterprise, organization templates let platform teams define blueprints of evaluators and monitoring charts that every new project inherits. These are packaged starting points for setting HoneyHive up, and teams can edit them. There are no prebuilt agents, packaged employees or deployable workflows for a customer's own work.

SourceHoneyHive, docs.honeyhive.ai ai-coding-agents and workspace/templatesread 2026-09-21

Supported providers include OpenAI, Anthropic, Azure OpenAI, Amazon Bedrock, SageMaker, Groq, Baseten and more, used with the customer's own keys. Admins can disable default models and add their own models and endpoints to control which models the team may use and deploy. Any model called through the SDKs is recorded, and a LiteLLM proxy works as a custom provider. A fallback provider can be configured, tested and deployed.

SourceFreeplay, docs.freeplay.ai model management, LiteLLM proxy, fallback provider guideread 2026-09-21

Workspace admins store their own provider credentials for Anthropic, OpenAI, Azure OpenAI, Amazon Bedrock, Gemini and Vertex AI, plus an OAuth gateway on self hosted deployments. HoneyHive uses them for LLM evaluators and the Playground, with provider access scoped by workspace. Custom model providers are available in prompt management on every plan, and evaluators can run through Portkey.

SourceHoneyHive, docs.honeyhive.ai workspace/provider-keys and evaluators/portkey, and honeyhive.ai/pricingread 2026-09-21

A REST API with an OpenAPI 3.1 specification covers observability, search, prompt templates, datasets, evaluation criteria, test runs and configuration. Freeplay also offers open source Python and Node SDKs under Apache-2.0 plus a Java and Kotlin SDK, an MCP server with skills and a Claude Code plugin for coding agents, and a CLI that bundles production prompts for a GitHub Actions check.

SourceFreeplay, docs.freeplay.ai developer resources overview, API introduction and llms.txtread 2026-09-21

Python and TypeScript SDKs built on OpenTelemetry make HoneyHive callable from code, alongside a REST API for events, sessions, datasets, experiments and evaluators, a Control Plane API for workspaces, data planes and alerts, and a CLI for terminal access to resources. Other languages send OpenTelemetry traces to its collector, and CI regression detection fits evaluation into a delivery pipeline. It also ships agent skills and an MCP docs search for coding agents.

SourceHoneyHive, docs.honeyhive.ai sdk-reference, API and control-plane references, cli-reference, ai-coding-agents and ci-regression-detectionread 2026-09-21

Evaluations can be model graded, code based or labeled by humans. Freeplay runs batch test runs against datasets at the component level and end to end through a customer's full agent, including tool use, retrieval and multi step workflows. Test runs start from the SDK or the API and return scored results that can be compared across versions. Freeplay also aligns LLM judges with human labels and runs online evaluations on sampled production logs.

SourceFreeplay, docs.freeplay.ai evaluations overview, component and end-to-end test runs, aligning LLM judgesread 2026-09-21

Experiments run an application over datasets and score it with versioned Python, LLM as a judge and human evaluators, and results are compared across runs. CI regression detection catches drops before a change ships. Online evaluations score production traces continuously, and production events can be curated into datasets for the next experiment.

SourceHoneyHive, docs.honeyhive.ai evaluation introduction, comparing evals, ci-regression-detection, onlineevals and dataset-curationread 2026-09-21

The product covers observability, prompt management, evaluations, test runs and human review for AI applications. No agent controls a browser, desktop or computer through it.

SourceFreeplay, docs.freeplay.ai llms.txtread 2026-09-21

The product covers tracing, monitoring, evaluation, datasets, prompts, workspace settings and deployment, and there is no browser, desktop or computer control.

SourceHoneyHive, docs.honeyhive.ai and llms-full.txtread 2026-09-21