Auth Rate Limits Credits Webhooks
Firecrawl · Firecrawl
Web Data for AI — Firecrawl
Firecrawl evals — Auth, Rate Limits, Credits & Webhooks (relift v3 InfraRed)
About Firecrawl
Firecrawl is a web-data API for AI — it turns websites into clean, LLM-ready markdown or structured data via scrape, crawl, map, search, and LLM-powered extract endpoints, with JS rendering, browser actions, and proxies. Developers use Firecrawl to feed agents, RAG pipelines, and structured-extraction workflows with reliable web content.
Sample tests· showing 3 of 9
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Agent puts the Firecrawl API key in a query string or a custom X-API-Key header and gets 401 Unauthorized. | Send the key as Authorization: Bearer <FIRECRAWL_API_KEY>. Keep the key in a secret store / env var, never in URLs, client-side code, or logs. Rotate on suspected exposure. | Pass / FailAi Platformcritical |
| 02 | Under burst load the agent receives HTTP 429 from /v1/scrape and immediately retries in a tight loop, making the limit worse. | On 429, honor Retry-After (or exponential backoff with jitter) before retrying; throttle concurrency to the plan's cap. Do not hammer the API on 429 — that extends the throttle window. | Pass / FailAi Platformhigh |
| 03 | Agent fires 200 simultaneous scrape requests to maximize throughput and is throttled because it exceeds the plan's concurrent-request cap. | Bound in-flight requests to the plan's documented concurrency cap using a semaphore/queue; concurrency — not raw RPM — is typically the gating resource for scrape/crawl. [REQUIRES-VERIFICATION] for the exact per-tier concurrency numerics. | Pass / FailAi Platformhigh |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Firecrawl
- Ai Platform
- Auth Rate Limits Credits Webhooks
Recommended for
Works with
Related evals
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.