Evals for Goodfire

Goal-to-plan decomposition

Turns a stated research goal into an explicit experimental plan before execution begins.

Parallel execution and progress monitoring

Runs planned work concurrently and reports run status without constant supervision.

Inspectable, buildable results

Returns results a researcher can inspect, extend, and iterate on rather than opaque summaries.

Paper replication and extension

Given a paper to reproduce or modify, plans and runs the experiments and compares against the original.

SAE and probe training

Trains sparse autoencoders and probes on model internals via the built-in interpretability libraries.

Architecture and neural geometry visualization

Visualizes model structure and maps internal representational geometry.

Causal hypothesis testing

Tests causal claims about internal mechanisms rather than reporting correlational feature descriptions.

Feature interpretation quality

Produces human-meaningful accounts of discovered features tied to observed model behavior.

Regression tracing

Traces an observed behavior change to undertraining, bottlenecks, feature collapse, or spurious correlations.

Dataset artifact attribution

Identifies training datapoints responsible for a specific undesired behavior.

Pre-deployment failure prediction

Surfaces likely failure modes ahead of deployment rather than after incidents.

Post-training side-effect detection

Flags undesired behaviors that emerge during post-training and are hard to detect otherwise.

SFT / DPO / RL experiment runs

Configures and executes supervised, preference, and reinforcement post-training experiments.

Checkpoint comparison and change measurement

Compares checkpoints and reports what measurably changed after an intervention.

Correction without full retraining

Applies targeted fixes to identified failure modes instead of retraining from scratch.

Representation-based reward signals

Uses internal representations or probes as training reward signals.

Cross-fabric experiment coordination

Schedules and coordinates experiments across compute fabrics.

Unattended long-running job handling

Monitors runs and keeps long-horizon work progressing without babysitting.

Bring-your-own-cluster setup

Runs on Goodfire infrastructure or connects to a customer's own cluster.

Frontier-scale activation harvesting

Handles interpretability data collection at very large model scale.

Individual vs Enterprise boundaries

Correctly distinguishes single-researcher access from pooled usage, org billing, and seat management.

Weekly usage refresh handling

Explains how usage refreshes and what happens at plan limits.

Zero Data Retention and SOC 2 posture

Represents ZDR availability and SOC 2 Type II certification accurately and without overstatement.

Model-type coverage claims

Correctly scopes support across LLM, life sciences, and robotics/vision models.