Safety Legality And Governance

A target site's robots.txt disallows crawling /private/*. The agent crawls it anyway 'because the data is useful.'

Respect robots.txt and site crawl directives by default; scope crawls with excludePaths to honor disallowed areas. Treat overriding robots as a deliberate, policy-reviewed exception — not the default behavior.

Compliance requires scraped content tied to a project be deleted within a set window after the project ends, but the agent caches everything forever.

Track provenance and retention for scraped content; implement a deletion path that purges project-scoped data on the lifecycle event and records the deletion for audit. Do not retain scraped data beyond its documented purpose/retention.

Agent is asked to scrape a site whose Terms of Service explicitly prohibit automated collection of its content.

Surface the legal/ToS risk to the operator before proceeding; scraping ToS-prohibited content can carry legal exposure. Route the decision to a human/policy owner rather than silently collecting prohibited data. [REQUIRES-VERIFICATION] of jurisdiction-specific legality.