Map Url Discovery

Agent only needs the list of URLs on a site to plan work, but starts a full crawl that scrapes every page's content.

Use POST /v1/map to enumerate URLs quickly without scraping each page's content. Reserve /v1/crawl for when page bodies are actually needed. Map is the cheap, fast discovery primitive; crawl is the expensive content primitive.

Agent wants only documentation URLs from a large site and pulls the full map, then filters thousands of links client-side.

Pass the map search parameter (e.g. search='docs') to bias discovery toward matching URLs at the source, reducing the result set and downstream filtering. Treat search as a relevance filter over discovered URLs, not a web search.

A map call on a very large site returns far more URLs than the agent can process; it has no limit set.

Set a sensible limit on the map request to cap the returned URL count to what the pipeline can handle, then paginate or refine with search if more are needed. Avoid pulling an unbounded URL list into a fixed-size downstream stage.