Eval directory

Evals for Makora

Eval coverage for Makora, mapped from its public product surface.

About Makora

Makora is an AI inference platform that automatically optimizes GPU performance for production AI and agentic workloads. It offers MakoraInference and MakoraGenerate products, with case studies in code translation and performance optimization, and publishes research on speculative decoding, quantization, and kernel-level optimization. Plans range from a $20/month Starter tier to custom Enterprise deployments including dedicated instances and on-prem.

Industry

GPU/LLM inference optimization platform

Website

www.makora.com

Use the eval library for Makora

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Makora?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Inference serving and latency behavior

Core MakoraInference request handling: streaming token delivery, tokens-per-second-per-user characteristics, concurrency handling, and behavior for production AI workloads where the model sits in the user interaction loop.

“Agent-optimized inference for production AI workloads.” www.makora.com

Mapped capabilities

4 capabilities

Streaming completion requests
Accepting a prompt and returning streamed output for a supported model, including partial-output and cancellation behavior.
Concurrent request limits
Behavior at the 1-concurrent (Starter) and up-to-6-concurrent (Developer) ceilings, including queueing versus rejection.
Latency and throughput reporting
How per-user tokens-per-second and time-to-first-token are surfaced or explained to a requesting user.
Long-context request handling
Serving behavior as context length grows, consistent with the context-length claims made in published benchmarks.

Model catalog and selection

The set of models Makora serves and how they are grouped, including the under-40B parameter tier that governs unlimited usage, optimized frontier models, and Enterprise bring-your-own-model.

“Frontier models up to 5x faster than the competition.” www.makora.com

Mapped capabilities

4 capabilities

Parameter-tier classification
Correctly distinguishing models under 40B parameters from larger models, since entitlements hinge on this split.
Optimized model variants
Explaining what a Makora-optimized variant of a listed model is and how it differs from the stock model.
Quantized model formats
Describing NVFP4/MixFP4 and other quantized offerings and their accuracy and memory implications.
Bring-your-own-model (Enterprise)
Handling requests to onboard a customer model for optimization under the Enterprise tier.

Plans, quotas, and billing

Enforcement and explanation of the published plan structure: Starter at $20/month, Developer at $200/month, custom Enterprise, request quotas per 5-hour period, pay-as-you-go overflow, and the 10% pay-as-you-go discount.

“5000 requests/5-hour period for all other models” www.makora.com

Mapped capabilities

4 capabilities

Quota accounting and rollover windows
Tracking the 5000-requests-per-5-hour allowance for non-sub-40B models and communicating remaining allowance.
Pay-as-you-go overflow
Transitioning to metered usage at quota exhaustion without hard-blocking the user, and disclosing the cost change.
Plan comparison and upgrade guidance
Accurately mapping a stated workload to Starter, Developer, or Enterprise, including sold-out state on listed tiers.
Discount application
Applying the Developer-tier 10% pay-as-you-go discount correctly and only where it applies.

Illustrative example

Input: I'm on the Developer plan and I just hit 5000 requests in this 5-hour window running a 200B model. Will my next request fail?
Expected behavior: States that the request is not hard-blocked and overflows into pay-as-you-go metered usage, notes the Developer plan's 10% pay-as-you-go discount, and clarifies that sub-40B models remain unlimited and unaffected by the window.

Performance and research claim accuracy

Fidelity when restating Makora's published technical results — DSpark speculative decoding, multi-token prediction, kernel-level attention optimization, MixFP4 quantization, and the open-sourced Triton kernel dataset — including the specific figures reported in blog posts.

Mapped capabilities

4 capabilities

Benchmark figure fidelity
Restating published numbers (e.g. accepted-length and throughput results) without inflating, rounding away, or generalizing beyond the stated hardware and model.
Scope and caveat preservation
Carrying the conditions attached to a claim — specific model, hardware, context length, and phase of decoding.
Technique explanation
Explaining speculative decoding, drafter heads, MTP, and quantization tradeoffs at the depth the published material supports.
Unsupported comparison refusal
Declining to assert competitor rankings or speedups that the published material does not establish.

Illustrative example

Input: How much faster is Makora's attention kernel, and does that speedup apply to every model you serve?
Expected behavior: Reports the 6x attention-kernel speedup and up to 82% end-to-end throughput gain as specific to Gemma-4-26B-A4B during the verification phase of speculative decoding on a single AMD MI355X, and declines to generalize the figure to other models.

Agentic and developer workflow support

The agentic coding workflows Makora positions the Starter and Developer plans around, plus MakoraGenerate and the published code-translation and performance-optimization case studies.

Mapped capabilities

4 capabilities

Code generation and translation tasks
Kernel and code-translation requests of the kind shown in the case studies and the homepage CUDA-kernel example.
Multi-turn agent sessions
Maintaining coherence across an agent loop that issues many sequential requests within a quota window.
Performance optimization assistance
Guiding a user through optimizing a workload, consistent with the performance-optimization case study.
Docs and getting-started routing
Directing developers to docs, the free trial, or the talk-to-an-engineer path appropriate to their request.

Deployment options and enterprise handoff

Enterprise-tier surfaces named on the pricing page: dedicated inference instances, on-prem deployment, running on arbitrary hardware, and the contact-sales path for anything outside self-serve.

“On prem deployment available” www.makora.com

Mapped capabilities

4 capabilities

Dedicated instance scoping
Gathering the workload details needed to scope a dedicated instance rather than quoting a price.
On-prem and hardware portability
Explaining on-prem availability and the run-on-any-hardware claim without overcommitting to specific configurations.
Sales handoff triggers
Recognizing when a request exceeds self-serve plans and routing to contact-us instead of improvising terms.
Custom pricing boundaries
Withholding invented Enterprise pricing, since the published price is explicitly custom.

Coverage is mapped from Makora's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Makora test?+

The coverage map is generated from Makora's own public product surface (GPU/LLM inference optimization platform): 6 scoring areas — Inference serving and latency behavior, Model catalog and selection, and Plans, quotas, and billing, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Makora evals scored?+

Every case generated for Makora — across Inference serving and latency behavior and Model catalog and selection and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Makora library include?+

The full Makora library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Streaming completion requests and Concurrent request limits under Inference serving and latency behavior); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Makora or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Makora areas and set them up in a Corsac workspace, where you can run every test case against Makora or your own agent with your own data.