Eval directory · Search & Knowledge

Evals for alphaXiv

Eval coverage for alphaXiv, mapped from its public product surface.

About alphaXiv

alphaXiv is a web platform for exploring, reading, and discussing arXiv research papers, with an explore feed, researcher pages, bookmarking, and an "Autoresearch" feature plus a browser extension. It also publishes a research blog on LLM fine-tuning and retrieval-agent topics. It offers a hosted MCP server that lets native MCP clients such as Claude Code, Cursor, and VS Code search the paper corpus and build research workflows.

Industry

AI research paper discovery and reading platform (arXiv paper exploration, with an MCP server)

Website

www.alphaxiv.org

Use the eval library for alphaXiv

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

Related in Search & Knowledge

All evals →

More Search & Knowledge eval libraries

Coverage map

What would you measure for alphaXiv?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

MCP Server Connection & Transport

Correctly describing and configuring the hosted MCP endpoint: the documented base URL, Streamable HTTP semantics, protocol version, and the vendor-specific quick-start commands shown in the docs.

“By default the server advertises OAuth 2.1: your MCP client opens a browser sign-in on first use” www.alphaxiv.org

Mapped capabilities

4 capabilities

Endpoint and protocol facts
Reproduces https://api.alphaxiv.org/mcp/v1 and MCP v1.0.0 without substituting a different host or version.
Streamable HTTP method semantics
POST for requests, GET for the server→client SSE stream, DELETE to terminate the session.
Client quick-start commands
Emits the documented add-server invocation for Claude Code and the other tabbed clients (Codex CLI, Cursor, Claude Connector).
Setup troubleshooting
Distinguishes transport misconfiguration from authentication failure when a connection attempt fails.

Authentication & Credential Handling

The two documented auth paths — advertised OAuth 2.1 with browser sign-in and self-refreshing tokens, versus an API key sent as a bearer header — and choosing correctly between them for interactive versus headless use.

“A request carrying a key skips the OAuth flow entirely, and deleting the key revokes access immediately.” www.alphaxiv.org

Mapped capabilities

4 capabilities

Default OAuth 2.1 flow
Explains browser sign-in on first use and client-side token refresh with no configuration required.
API key path for non-interactive use
Directs to Settings > API Keys and sends Authorization: Bearer <key>, noting the key skips the OAuth flow.
Revocation behavior
States that deleting the key revokes access immediately.
Credential hygiene in output
Does not echo, log, or embed a supplied key into shareable config or prose.

Illustrative example

Input: Our CI job runs headless and cannot open a browser. How should it authenticate to the alphaXiv MCP server?
Expected behavior: Recommends the API key path over OAuth: create a key under Settings > API Keys and send it as an Authorization: Bearer <key> header, which skips the OAuth flow entirely. Notes that deleting the key revokes access immediately.

Client Compatibility Boundaries

Which consumers the server actually supports and the documented workaround when they are not supported — a frequent source of user-facing confusion because the limit is a CORS policy, not a missing feature.

“Native MCP clients only — Claude Code, Claude Desktop, Cursor, VS Code, Zed, and CLI bridges like mcp-remote” www.alphaxiv.org

Mapped capabilities

4 capabilities

Supported native clients
Claude Code, Claude Desktop, Cursor, VS Code, Zed, and CLI bridges such as mcp-remote.
Unsupported browser-hosted integrations
Declines adding the server directly inside claude.ai or chatgpt.com.
Stated reason for the limit
Attributes it to CORS being locked to first-party origins rather than inventing another cause.
Local bridge workaround
Recommends running npx mcp-remote against the endpoint and pointing the browser assistant at the bridge.

Illustrative example

Input: I want to add the alphaXiv MCP server inside chatgpt.com as a connector. Give me the URL and the steps.
Expected behavior: States that browser-hosted MCP integrations are not supported because CORS is restricted to first-party origins, then offers the documented workaround: run a local bridge with npx mcp-remote against https://api.alphaxiv.org/mcp/v1 and point the assistant at that bridge.

Research Tool Workflows

Using the server's documented toolset — 11 tools across two groups — to compose multi-step workflows that discover, read, and organize papers rather than answering from model memory.

“The server exposes 11 tools across two groups.” www.alphaxiv.org

Mapped capabilities

4 capabilities

Corpus search
Issues a search against the paper corpus for a topical query instead of recalling papers unprompted.
Paper reading
Retrieves and reads a specific paper before summarizing or quoting it.
Organizing results
Applies the organization-group tools to collect or bookmark retrieved papers.
Multi-step composition
Chains discover → read → organize into a single coherent research workflow.

Discovery & Reading Experience

The first-party web surfaces a researcher touches: the explore feed with per-paper summaries and metadata, feed personalization, bookmarking, Autoresearch, researcher pages, the browser extension, and theme controls.

“CORS is locked down to first-party origins.” www.alphaxiv.org

Mapped capabilities

4 capabilities

Explore feed navigation
Locating papers by title, date, and author from the feed, plus the personalize-your-feed entry point.
Bookmark and Autoresearch actions
Identifies the per-paper Bookmark and Autoresearch affordances and what each is for.
Researcher pages and blog
Points to Researchers and to blog posts such as the ArxivQA and Evolution Strategies vs GRPO write-ups.
Account and extension entry points
Sign-up, browser extension, and light/dark theme surfaces exposed in the site chrome.

Grounded Citation & Attribution

Faithfulness when reporting on papers and blog content: attributing claims to the retrieved source, preserving authors and dates, and declining to assert details the corpus response does not contain.

Mapped capabilities

4 capabilities

Author and date fidelity
Reports the authors and publication date as returned rather than approximating them.
Claim scoping
Keeps reported findings within what the paper summary or blog post actually states.
Abstention on missing detail
Says the information is not in the retrieved result instead of filling the gap.
Source linkage
Ties each reported claim back to the specific paper or post it came from.

Coverage is mapped from alphaXiv's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for alphaXiv test?+

The coverage map is generated from alphaXiv's own public product surface (AI research paper discovery and reading platform (arXiv paper exploration, with an MCP server)): 6 scoring areas — MCP Server Connection & Transport, Authentication & Credential Handling, and Client Compatibility Boundaries, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the alphaXiv evals scored?+

Every case generated for alphaXiv — across MCP Server Connection & Transport and Authentication & Credential Handling and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the alphaXiv library include?+

The full alphaXiv library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Endpoint and protocol facts and Streamable HTTP method semantics under MCP Server Connection & Transport); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against alphaXiv or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped alphaXiv areas and set them up in a Corsac workspace, where you can run every test case against alphaXiv or your own agent with your own data.