Eval directory · AI Platform

Evals for LightSeek Foundation

Eval coverage for LightSeek Foundation, mapped from its public product surface.

About LightSeek Foundation

LightSeek Foundation is a 501(c)(3) nonprofit that funds and coordinates open research and open-source projects for AI systems, focusing on open infrastructure, research acceleration, and community education. Its flagship engineering work is TokenSpeed, an LLM inference stack with portable multi-silicon kernels that ships Day 0 support for large open-weight models on NVIDIA and AMD accelerators. The foundation is a member of the PyTorch Foundation and the Linux Foundation and is funded through GPU compute, cash sponsorship, and engineering support from ecosystem partners.

Industry

open-source LLM inference infrastructure / AI research nonprofit

Website

lightseek.org

Use the eval library for LightSeek Foundation

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

Related in AI Platform

All evals →

More AI Platform eval libraries

Coverage map

What would you measure for LightSeek Foundation?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

TokenSpeed Day 0 Model Enablement

Claims about bringing newly released large open-weight models up on leading accelerators at or near launch, including the specific models, silicon targets, and timelines described in the foundation's engineering blog.

“LightSeek Foundation is a 501(c)(3) nonprofit organization advancing open research and open-source innovation” lightseek.org

Mapped capabilities

4 capabilities

Model and parameter-scale facts
Correctly attributes Kimi K3 (2.8T, Moonshot AI), TML Inkling (975B MoE, Thinking Machines Lab), and Qwen3.5-397B-A17B without inflating or swapping figures.
Silicon target coverage
States NVIDIA Blackwell and AMD CDNA4 as the named Day 0 targets and does not assert support for untargeted vendors or accelerators.
Enablement timeline claims
Reports the stated one-week Kimi K3 enablement window and avoids generalizing it into a guaranteed SLA for future models.
Serving architecture features
Describes unified flat KV cache, disaggregated serving, and quantized (NVFP4/MXFP4) inference only where the source attributes them.

Illustrative example

Input: We're standardizing on AMD. Did TokenSpeed's Day 0 support for Kimi K3 cover AMD, or was it NVIDIA-only, and how fast was it?
Expected behavior: States that TokenSpeed enabled Day 0 support for the 2.8T-parameter Kimi K3 on both NVIDIA Blackwell and AMD CDNA4 within one week, and does not present that timeline as a commitment for future model launches.

Multi-Silicon Kernel Portability

The TokenSpeed-kernel subsystem as a standalone open-source project: layered APIs and registry-based dispatch that separate runtime logic from hardware-specific backends.

“TokenSpeed brings native NVFP4 and MXFP4 serving to Thinking Machines Lab’s 975B-parameter open-source MoE model Inkling” lightseek.org

Mapped capabilities

4 capabilities

Layered API and dispatch model
Explains registry-based dispatch and the runtime/backend separation as the portability mechanism.
Backend kernel families
Attributes CuteDSL, TensorRT-LLM, and Gluon kernels to the correct role as specialized backend implementations.
Standalone subsystem boundary
Distinguishes TokenSpeed-kernel as a separable subsystem from the full TokenSpeed inference engine.
Performance claim handling
Cites throughput figures such as the Qwen3.5 agentic-workload record with their stated workload and hardware context rather than as universal benchmarks.

Foundation Identity and Governance

Who LightSeek is as an organization: nonprofit status, ecosystem memberships, the projects it created, and the scope of its mission statements and public positions.

“LightSeek Foundation is a member of the PyTorch Foundation and the Linux Foundation” lightseek.org

Mapped capabilities

4 capabilities

Nonprofit and tax status
States 501(c)(3) status and IRS Tax Exempt Organization Search listing without implying tax advice or donor-specific outcomes.
Ecosystem memberships
Correctly reports membership in the PyTorch Foundation and the Linux Foundation without upgrading it to sponsorship or ownership.
Project portfolio
Identifies TokenSpeed, TorchSpec, and SMG as foundation-created projects and does not attribute unlisted projects.
Published positions and mission scope
Represents the open-weights open letter signature and mission pillars (open infrastructure, research acceleration, community and education) as stated.

Sponsorship and Partner Support

How organizations and individuals can support the foundation, the accepted forms of contribution, and the boundaries of what sponsorship confers.

“We welcome GPU compute, cash sponsorship, and engineering support.” lightseek.org

Mapped capabilities

4 capabilities

Accepted contribution types
Names GPU compute and credits, cash sponsorship, and engineering support as welcomed forms.
Sponsorship channels
Routes to GitHub Sponsors, Benevity, or the contact path rather than inventing invoicing or tier structures.
Partner collaboration framing
Describes Day 0 model support and Triton/LLVM community collaboration as partnership forms without asserting unnamed partners.
Sponsor entitlement boundaries
Declines to promise influence, roadmap control, or benefits not described on the sponsors page.

Brand and Trademark Usage

Correct use of LightSeek Foundation and TokenSpeed marks and the licensing boundary between openly licensed content and reserved trademarks.

“is licensed under the Creative Commons Attribution 4.0 International License” lightseek.org

Mapped capabilities

4 capabilities

Asset selection guidance
Matches the right asset to the surface: logo vs. wordmark, light vs. dark, TS monogram for compact placements, banner for headers.
Format guidance
Directs SVG to web and vector workflows and PNG to raster applications.
Unmodified use requirement
Requires using artwork as provided, kept legible and clearly associated with the relevant project or organization.
Trademark vs. content license
Separates CC BY 4.0 site content from logos and trademarks, which remain their owners' property with no rights granted.

Illustrative example

Input: Can we recolor the TokenSpeed logo to match our brand palette and put it on our product page under a 'Backed by LightSeek' headline?
Expected behavior: Declines the recolor, explaining that marks must be used as provided, kept legible, and clearly associated with the relevant project. Notes that logos and trademarks remain their owners' property with no license granted, so an endorsement claim is not supported, and points to the Brand Guidelines.

Privacy, Terms, and Contributor Data

How the foundation describes its collection and use of personal information, the special case of open-source contribution metadata, and the disclaimers governing site content.

“TokenSpeed-kernel is a standalone open-source subsystem for LLM inference kernels” lightseek.org

Mapped capabilities

4 capabilities

Collection categories
Distinguishes information you provide, open-source contribution data, and automatically collected technical/usage data.
Contribution metadata handling
Explains that commit metadata, usernames, emails, and DCO sign-offs are collected and maintained with the contribution.
Third-party platform boundary
Notes that interactions via GitHub or other platforms are also governed by that platform's privacy policy.
Content accuracy disclaimers
Conveys that content may contain inaccuracies, may change without notice, and is used at the recipient's own risk.

Coverage is mapped from LightSeek Foundation's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for LightSeek Foundation test?+

The coverage map is generated from LightSeek Foundation's own public product surface (open-source LLM inference infrastructure / AI research nonprofit): 6 scoring areas — TokenSpeed Day 0 Model Enablement, Multi-Silicon Kernel Portability, and Foundation Identity and Governance, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the LightSeek Foundation evals scored?+

Every case generated for LightSeek Foundation — across TokenSpeed Day 0 Model Enablement and Multi-Silicon Kernel Portability and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the LightSeek Foundation library include?+

The full LightSeek Foundation library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Model and parameter-scale facts and Silicon target coverage under TokenSpeed Day 0 Model Enablement); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against LightSeek Foundation or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped LightSeek Foundation areas and set them up in a Corsac workspace, where you can run every test case against LightSeek Foundation or your own agent with your own data.