Cog And Custom Model Push
Replicate · Replicate
AI Model Hosting — Replicate
Evaluates Replicate's Cog & Custom Model Push across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Model Hosting eval coverage.
About Replicate
Replicate is an AI model-hosting platform — run thousands of community and custom Cog-packaged models (FLUX, SDXL, Llama, Whisper, custom fine-tunes) via a simple HTTP API with predictions, webhooks, streaming, deployments, and per-second billing.
Sample tests· showing 3 of 9
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Integrator's cog.yaml declares build.gpu=true but does not specify a GPU class. Push succeeds but predictions OOM on the assigned tier. | Match the predict.py memory footprint to the documented per-tier VRAM (T4 16GB, A40 48GB, A100 40/80GB, H100 80GB). Declare the target hardware tier on the model in the Replicate UI (or via the API) — build.gpu=true is necessary but not sufficient. Test with a representative input before shipping. | Pass / FailAi Platformhigh |
| 02 | Custom Llama-7B model loads weights into GPU memory on every predict() call, adding 18 s per request. | Load weights once in setup() (called on container boot) and hold them in self.model. predict() should be a hot-path call against the warm model. Cold-start cost amortizes across all subsequent predictions on that container. [REQUIRES-VERIFICATION] for current container-lifetime caps. | Pass / FailAi Platformcritical |
| 03 | predict.py declares predict(prompt: str, image: Path = Input(default=None)). Replicate auto-derives the OpenAPI input schema from this signature. | Use cog.Input() with type hints, default, and description; the generated input_schema is what users (and the operator's own UI generators) consume. Path inputs are auto-downloaded by Cog from a URL — do not duplicate that logic. Annotate enums via Literal[...] or choices=[...]. | Pass / FailAi Platformhigh |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Replicate
- Ai Platform
- Cog And Custom Model Push
Recommended for
Works with
Related evals
Claude API
Evaluates Anthropic's Batch API across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Extended Thinking across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Files API & Citations across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
ViewFrequently asked questions
What does the Cog And Custom Model Push eval for Replicate Replicate test?+
Evaluates Replicate's Cog & Custom Model Push across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Model Hosting eval coverage.
How is the Cog And Custom Model Push eval scored?+
The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
How many test cases does this eval pack include?+
The Cog And Custom Model Push pack for Replicate Replicate contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Cog And Custom Model Push pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.