Crawl Whole Site

Agent calls POST /v1/crawl and immediately tries to read pages from the response body, but the response only contains a job id and a status URL.

Treat crawl as asynchronous: capture the returned job id, then poll GET /v1/crawl/{id} until status=completed (or failed/cancelled) before consuming data[]. Use exponential backoff on polling rather than a tight loop.

A crawl seeded at /docs/getting-started unexpectedly pulls in /pricing and /blog because those are linked from the docs.

Understand that allowBackwardLinks controls whether the crawler follows links that go 'up' or outside the seed path. Keep it false (default) to stay within the seed subtree; enable it deliberately only when broader coverage is intended.

Agent kicks off a crawl on a large e-commerce site with no limit or maxDepth, intending to grab only the top-level category pages.

Bound the crawl with limit (max pages) and maxDepth (link hops from the seed) matched to the intent. An unbounded crawl on a large site burns credits and may never terminate within budget. Start small, inspect, then widen.