Feature
Narrated research and web crawling in ClawAI
How ClawAI researches the web: a planner that chooses to search or crawl, a live narrated work log, tiered fetching that respects robots.txt, and cited sources.
All features · Last reviewed:
A planner decides, not a keyword
In Auto research mode, a planning model reads the message and chooses one of four paths: answer from what it already knows, search the web, crawl a specific site, or crawl and then search. A link you paste is always opened. If the planning model returns something unusable, the next model is tried rather than the research silently stopping.
You can also pick the mode yourself in the composer: off, Auto, search only, search and fetch pages, or search and extract structured content.
A work log you can watch
Each step — the plan, every search, every page fetched or skipped — is streamed into a narrated log above the answer as it happens, and saved with the answer so it is still there after a refresh. The sources the answer drew on are listed with it, so you can open and check them yourself.
Tiered fetching that follows the rules
Pages are fetched cheapest first: a site’s official API where one exists, then a plain HTTP request, then escalation to a headless browser only when a page needs one, with a reader service and archive snapshots as later fallbacks. A crawl can cover up to two hundred pages of one site.
robots.txt is honoured on every fetch under the ClawAI-ResearchBot user agent, and a disallowed page is not retried another way. Sign-in walls and legal blocks stop the fetch, captchas are never solved, every redirect is checked against private network addresses, and an archived copy is always labelled as one.
Questions people ask
- Does ClawAI respect robots.txt?
- Yes, on every fetch, under the ClawAI-ResearchBot user agent. A page robots.txt disallows is skipped and not retried through another fetching method.
- Can I see what the research actually did?
- Yes. A narrated work log shows the plan, each search and each page fetched or skipped, and it is saved with the answer along with the list of sources it used.
- Is web research available on every plan?
- Research modes are plan features (RESEARCH_MODE, WEB_SEARCH, WEB_FETCH and WEB_EXTRACT) with their own allowances, which the operator sets per plan. The pricing page shows what each plan includes.
Try it rather than take our word for it
The research loop runs in its own research service with admin-editable fetch tiers, metered on its own surfaces rather than as ordinary chat tokens.