Eval directory · Search & Knowledge

Evals for SerpApi

Eval coverage for SerpApi, mapped from its public product surface.

About SerpApi

SerpApi is a hosted API that scrapes search engine results pages and returns them as structured data. It supports a large catalog of engines and endpoints, including Google (Search, Maps, News, Shopping, Scholar, Flights, and more), Bing, Baidu, Amazon, eBay, Apple App Store, and The Home Depot, each with its own documented parameters and interactive playground. Accounts are sold on Standard and Enterprise plans, with per-engine uptime figures published and Enterprise-only options such as ZeroTrace mode.

Industry

search engine results (SERP) scraping API

Website

serpapi.com

Use the eval library for SerpApi

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

Related in Search & Knowledge

All evals →

More Search & Knowledge eval libraries

Coverage map

What would you measure for SerpApi?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Engine catalog and endpoint routing

Choosing the right engine for a stated data need and constructing the correct request against https://serpapi.com/search with the matching engine parameter, across the documented Google, Bing, Baidu, Amazon, eBay, Apple, and Home Depot families.

“Only Product IDs gathered from the Home Depot US are supported.” serpapi.com

Mapped capabilities

4 capabilities

Engine selection for a stated data need
Mapping an information goal (product page, reviews, maps, scholar, flights) to the documented engine that serves it.
engine parameter correctness
Emitting the exact engine token (e.g. amazon_product, bing_product, ebay_product, apple_product, home_depot_product) for the chosen API.
Search vs. detail endpoint distinction
Separating catalog/search engines from item-detail engines and sequencing them when a detail call needs an upstream identifier.
Endpoint and playground discovery
Pointing to the documented endpoint URL and interactive playground for an engine rather than inventing paths.

Request parameter contracts

Honoring each engine's documented required and optional parameters, including the distinct product identifiers different engines demand and the tokens that must be obtained from a prior call.

Mapped capabilities

4 capabilities

Required vs. optional parameters per engine
Identifying which parameters a request cannot omit for a given engine and which are safe defaults.
Engine-specific product identifiers
Distinguishing asin (Amazon), product_id (eBay, Apple, Home Depot), and product_token (Bing Product) and where each is sourced.
Derived-token dependencies
Recognizing that Bing Product requires a product_token generated by the Bing Shopping API rather than a user-supplied id.
Advanced per-engine options
Correct use of documented extras such as type, other_sellers, store_id, shipping_country, and device.

Localization and geo targeting

Applying the localization surface each engine documents — country codes, storefront domains, locales, languages, and delivery/shipping location filters — and respecting the constraints on when each applies.

Mapped capabilities

4 capabilities

Country and domain selection
Two-letter country codes for Apple, amazon_domain for Amazon, ebay_domain for eBay, and their documented defaults.
Language and locale formats
Locale-style language values such as en_US or ja_JP for Amazon versus eBay's lang and locale pairing.
Conditional localization rules
Honoring documented restrictions, e.g. eBay lang applying only on the US domain and only when locale is set.
Delivery and shipping filters
Using delivery_zip, shipping_location, and shipping_country to scope results to an area rather than as generic location hints.

Caching and asynchronous execution

Reasoning about SerpApi's shared request-lifecycle parameters: cache behavior and expiry, async submission with later retrieval, and the documented combinations that must not be used together.

“async should not be used on accounts with Ludicrous Speed enabled.” serpapi.com

Mapped capabilities

4 capabilities

no_cache semantics
When a cached result is served (identical query and parameters, within the 1h expiry) and that cached searches are free and uncounted.
async submission and retrieval
Submitting without holding the connection open and retrieving results later via the Searches Archive API.
Incompatible parameter combinations
Refusing to pair async with no_cache, and avoiding async on accounts with Ludicrous Speed enabled.
Quota implications of cache choices
Explaining how forcing a fresh fetch affects searches counted against the monthly allowance.

Illustrative example

Input: I want fresh, uncached Amazon Product results but I don't want to hold the HTTP connection open. Can I just set async=true and no_cache=true on the same request?
Expected behavior: States that async and no_cache must not be used together per SerpApi's documentation, and offers a compliant alternative: submit asynchronously and retrieve via the Searches Archive API, or force a fresh fetch with a synchronous request.

Plans, entitlements, and authentication

Correctly representing the Standard and Enterprise plan split, the features gated behind Enterprise, and the credential required on every request.

Mapped capabilities

4 capabilities

api_key requirement
Treating api_key as required on every search request and not omitting or substituting it.
Enterprise-gated features
Identifying zero_trace as Enterprise-only and not offering it to a Standard-plan caller.
ZeroTrace behavior and trade-offs
Explaining that it skips storing search parameters, files, and metadata, and that this may make debugging harder.
Plan-tier guidance
Directing tier questions to the documented Standard vs. Enterprise pricing surface without inventing figures.

Illustrative example

Input: I'm on the Standard plan and handling sensitive queries. Add zero_trace=true to my Bing Product call so nothing is stored on your servers.
Expected behavior: Identifies zero_trace as an Enterprise-only parameter unavailable on Standard, so it should not be added to the request. Notes what the mode does — skipping storage of search parameters, files, and metadata — and that it can make debugging harder.

Reliability and documented coverage limits

Communicating what the documentation actually states about per-engine uptime and unsupported cases, so integration plans account for variance rather than assuming uniform availability.

Mapped capabilities

4 capabilities

Per-engine uptime differences
Recognizing that published uptime varies by engine (e.g. 100.000% for several vs. 77.177% for Home Depot Product).
Unsupported input cases
Flagging documented gaps such as Home Depot product IDs gathered from Canada not being supported.
Release notes as a change surface
Referring users to per-engine release notes for behavior changes instead of asserting stability.
Grounded-answer discipline
Declining to state uptime, limits, or parameters that the documentation does not provide.

Coverage is mapped from SerpApi's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for SerpApi test?+

The coverage map is generated from SerpApi's own public product surface (search engine results (SERP) scraping API): 6 scoring areas — Engine catalog and endpoint routing, Request parameter contracts, and Localization and geo targeting, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the SerpApi evals scored?+

Every case generated for SerpApi — across Engine catalog and endpoint routing and Request parameter contracts and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the SerpApi library include?+

The full SerpApi library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Engine selection for a stated data need and engine parameter correctness under Engine catalog and endpoint routing); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against SerpApi or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped SerpApi areas and set them up in a Corsac workspace, where you can run every test case against SerpApi or your own agent with your own data.