Eval Library
R
For ReplicateAI Platform

Webhooks

Replicate · Replicate

AI Model Hosting — Replicate

Replicate evals — Webhooks (relift v3 InfraRed)

About Replicate

Replicate is an AI model-hosting platform — run thousands of community and custom Cog-packaged models (FLUX, SDXL, Llama, Whisper, custom fine-tunes) via a simple HTTP API with predictions, webhooks, streaming, deployments, and per-second billing.

Employees

~80

Industry

AI Inference Platform

Headquarters

San Francisco, CA

Sample tests· showing 3 of 9

#InputExpected behaviorCheck
01

Integrator sets webhook_events_filter=['completed'] expecting only one delivery per prediction.

Choose the smallest filter subset that satisfies downstream needs: ['completed'] for fire-and-forget terminal delivery; ['start','output','completed'] for progress UIs. Larger filters mean more deliveries → more HMAC verifications → more retry exposure. Document the chosen filter in the operator ru…

Pass / FailAi Platformmedium
02

Webhook delivery carries headers webhook-id, webhook-timestamp, webhook-signature. Receiver wants to validate authenticity.

Compute HMAC-SHA256 over the canonical string '<webhook-id>.<webhook-timestamp>.<raw-body>' with the per-endpoint signing secret. Compare in constant time against the v1 entry in webhook-signature. Reject if signature mismatches or webhook-timestamp is outside a tolerance window (e.g., 5 minutes) t…

Pass / FailAi Platformcritical
03

Webhook handler does heavy work inline (re-encodes a 4K video) and responds 200 after 45 seconds.

Return 2xx as fast as possible (typically <5s) — push real work to a background queue. A slow handler risks Replicate timing out the delivery and retrying, causing duplicate processing. The exact delivery deadline [REQUIRES-VERIFICATION] — design for sub-second response regardless.

Pass / FailAi Platformhigh

How this eval is graded

Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

Rubric criteria

  • Replicate
  • Ai Platform
  • Webhooks

Recommended for

ReplicateReplicate customers

Works with

Related evals

Run this eval in your workspace

Connect your data, configure thresholds, and review results with your team.