Crawl4AI

Agents get the web as tools. The hosted API and its MCP server offer scrape, search, extract, answer, SERP, price, batch and background job tools, plus more than 90 ready made recipes for sites such as Amazon, LinkedIn and Zillow, and some recipes read a site as the user when handed that user's own login cookie for one run. These tools return content rather than change anything in another system, and there is no connector catalog for business applications such as a CRM, ticketing or messaging tool.

Sourcecrawl4ai.com/llms.txtread 2026-11-09

Crawl4AI fetches and shapes web data rather than running an agent's workflow. Batch calls take up to 50 URLs and background jobs up to 10,000, with a retry for the URLs that failed, and in the open source library deep crawling walks a site breadth first, depth first or by best score, while adaptive crawling stops once its confidence says enough has been gathered for a query. Planning, branching and hand offs between agents stay in the customer's own agent.

Sourcedocs.crawl4ai.com/core/deep-crawlingread 2026-11-09

The public web is what Crawl4AI reads, and the customer's own knowledge is not held anywhere in it. Repeated URLs are served from a shared archive at a lower price, which is a cache of public pages, and the library writes Markdown, chunks and extracted JSON that the customer loads into its own vector store. No index, embeddings layer or knowledge base over a customer's content is maintained for agents to query.

Sourcecrawl4ai.com/llms.txtread 2026-11-09

Spending limits are the controls the cloud offers. An account can set a spend cap in credits, automatic recharge only fires within a monthly dollar limit the user picks, the MCP top up tool is meant to ask the user before buying credits, and each app connected through sign in appears as its own key that can be revoked. Plan rate and concurrency limits refuse calls past them. No step holds a scrape, extraction or form submission for a person to approve.

Sourcecrawl4ai.com/llms.txtread 2026-11-09

An account signs in with Google or GitHub and works through API keys, stored only as hashes and revocable one by one, with no organization roles, team permissions or single sign on for a buyer's staff. A SOC 2 report is still in progress with an outside auditor. Crawl4AI's security page describes TLS on every connection, encrypted backups, staff access limited to operators with multi factor authentication and quarterly reviews, a named subprocessor list and a DPA on request.

Sourcecrawl4ai.com/securityread 2026-11-09

Each cloud response says how the page was fetched: the engine that served it, the proxy exit and country, and the ladder of steps tried in order with each step's result, along with its cost. Jobs report counts, cost and credited failures, the dashboard shows keys and usage, and a self hosted Docker server adds a monitoring dashboard with live WebSocket updates, while the library can capture network requests and console messages. There is no account level audit log of who called what.

Sourcecrawl4ai.com/llms.txtread 2026-11-09

Nothing in Crawl4AI remembers on an agent's behalf. The closest things are browser state: sessions keep one tab open across the steps of a crawl, and managed browser profiles keep cookies, local storage and logins so later crawls browse as the same user. The cloud's session recipes use a login cookie for one run and never store it. Crawl results go back to the caller and are not kept as context for later runs.

Sourcedocs.crawl4ai.com/advanced/identity-based-crawlingread 2026-11-09

The crawler can run entirely on the customer's side: the Apache 2.0 Python library installs anywhere Python and Chromium run, and a Docker image serves the same crawler with a REST API, MCP server and dashboard on the customer's own servers. Crawl4AI Cloud runs in three regions, the European Union (Germany), Singapore and the United States, and routes each request to the nearest one, while account, usage and billing records stay in the EU region only. Enterprise terms add region choices.

Sourcecrawl4ai.com/securityread 2026-11-09

More than 90 recipes come ready to run in the cloud, each a tested scraper for one site and one task, such as Amazon reviews, Zillow listings, LinkedIn people search, YouTube videos, GitHub repositories or Steam games. A free catalog call lists every recipe's inputs and output columns with a sentence per column, a recipe returns the same named columns every time, larger ones run as background jobs, and a health page shows each recipe's last check in every region.

Sourcecrawl4ai.com/llms.txtread 2026-11-09

Work reaches Crawl4AI only when a caller asks: an HTTP request to the hosted API, a tool call through its MCP server, a call into the Python library or a request to a self hosted Docker server. Background jobs are submitted and then polled for status rather than reporting back, and there is no scheduler, page watcher or webhook that starts a crawl or alerts an agent when a site changes.

Sourcecrawl4ai.com/llms.txtread 2026-11-09

In the open source library, LLM extraction takes a provider string the developer picks, from any model LiteLLM supports, such as OpenAI, Anthropic or a local Ollama model, along with the developer's own key. On Crawl4AI Cloud, the models behind extract, answer and page change summaries are Crawl4AI's own choice, reached through OpenRouter and the Gemini API, with no picker for the caller.

Sourcedocs.crawl4ai.com/extraction/llm-strategiesread 2026-11-09

The hosted REST API at api.crawl4ai.com covers scrape, batch and background jobs, extract, search, SERP, answer, price and recipes with bearer key auth and published request and response fields. Its remote MCP server takes a key or a sign in and lists every tool, from scrape and recipe runs to balance, spend caps and job submission. The open source Python library is the SDK, with a crwl command line tool, and the self hosted Docker server exposes its own REST API and MCP server.

Sourcecrawl4ai.com/llms.txtread 2026-11-09

Nothing in Crawl4AI tests or scores an agent's behavior. The homepage playground and the self hosted playground let a developer try a request and read the raw output, the estimate tool prices a call before it runs, and Crawl4AI checks its own recipes in every region on a health page. Comparing extraction quality across prompts or models is left to the developer.

Sourcecrawl4ai.comread 2026-11-09

The open source library drives a real Chromium through Playwright. Code or C4A-Script commands navigate, wait for elements, type, click, scroll, fill forms and submit them, sessions keep one tab alive across multistep flows, managed profiles carry a logged in identity, and an undetected browser mode handles sites that block bots. On the hosted API, a page that needs it is rendered in a real Chrome, and some scrapes can be sent straight to a browser engine.

Sourcedocs.crawl4ai.com/core/c4a-scriptread 2026-11-09