Crawl Whole Site

Agent calls POST /v1/crawl and immediately tries to read pages from the response body, but the response only contains a job id and a status URL.

Treat crawl as asynchronous: capture the returned job id, then poll GET /v1/crawl/{id} until status=completed (or failed/cancelled) before consuming data[]. Use exponential backoff on polling rather than a tight loop.

Agent kicks off a crawl on a large e-commerce site with no limit or maxDepth, intending to grab only the top-level category pages.

Bound the crawl with limit (max pages) and maxDepth (link hops from the seed) matched to the intent. An unbounded crawl on a large site burns credits and may never terminate within budget. Start small, inspect, then widen.

Agent only wants /blog/* pages and must skip /admin and /cart. It relies on post-filtering crawl results instead of scoping the crawl.

Scope the crawl with includePaths (e.g. ['^/blog/.*']) and excludePaths (e.g. ['^/admin','^/cart']) so out-of-scope pages are never fetched — saving credits and avoiding sensitive areas. Confirm the regexes are anchored to path, not full URL, per docs.