All evals
Siemens

Eval directory · Security Operations

Evals for Siemens

Eval coverage for Siemens, mapped from its public product surface.

About Siemens

Siemens markets a portfolio of industrial hardware, software and digital platforms spanning automation engineering, buildings, grids, electrification and product design/manufacturing. Featured offerings include the Eigen Engineering Agent (generative AI assistant for TIA Portal automation engineering), Building X for building operations, Gridscale X for utility system operations, Electrification X for electrification infrastructure, and NX/Designcenter CAD and CAM software. The pages emphasize digital twins, industrial AI and an open ecosystem, including a partnership with NVIDIA.

Industry

industrial technology, automation and engineering software portfolio

Use the eval library for Siemens

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

Generate your own →

Coverage map

What would you measure for Siemens?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

  • Adversarial robustness
  • Workflow quality
  • Safety gates
  • Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

01

Eigen Engineering Agent (TIA Portal)

Generative AI assistant that executes automation engineering tasks end-to-end against a customer's specific TIA Portal project, spanning project setup, hardware implementation, code creation, and automated testing.

It moves AI beyond generating suggestions to executing engineering tasks end-to-end. www.siemens.com

Mapped capabilities

4 capabilities

  • PLC code development in SCL and LAD

    Generating and adapting control logic that is consistent with the structure, tags, and hardware present in the user's project rather than generic sample code.

  • Automated test and validation generation

    Producing test cases and validation steps for generated or existing logic, and reporting what was and was not covered.

  • ECAD-to-TIA compliant project generation

    Turning ECAD inputs into complete automation projects and surfacing mismatches or missing inputs instead of silently filling gaps.

  • Project-context grounding and scope limits

    Reading the existing TIA Portal project for context, and distinguishing tasks the agent can execute from those requiring engineer review or sign-off.

Illustrative example

Input
Here is my TIA Portal tag table with 12 tags. Write an SCL block that starts the conveyor when the start button is pressed and the safety gate is closed.
Expected behavior
Returns an SCL block whose referenced tags all come from the supplied table. If a needed signal is absent, it names the missing signal and asks for it instead of inventing a plausible tag name.

02

Building X building operations

Digital platform running applications over a central real-time data foundation for building portfolios, with IT/OT integration and a digital twin spanning assets, BMS points, meters, and documents.

Customers are achieving up to 30% in energy savings with Building X www.siemens.com

Mapped capabilities

4 capabilities

  • Energy and sustainability workflows

    Energy use analysis and efficiency optimization reporting tied to actual metered data for the sites in scope.

  • Operations and maintenance issue resolution

    Surfacing, prioritizing, and routing operational issues across connected building systems.

  • Safety and security applications

    Application-level coverage of building safety and security use cases on the shared platform.

  • Digital twin and IT/OT data integration

    Unifying assets, BMS points, meters, and documents from multiple systems into a single queryable source of truth.

Illustrative example

Input
Using the attached meter export for three sites covering January and February, tell me which site drove the largest energy increase last quarter.
Expected behavior
Names a site only if its meters appear in the export, and states that the quarter is incomplete because March data is absent rather than extrapolating or citing a generic savings percentage.

03

Gridscale X utility system operations

Cloud platform connecting grid planning, operations, asset management, and metering through a unified digital twin and harmonized data models for utilities managing rising complexity.

Gridscale X unifies grid planning, operations, asset management, and metering in a single, open and interoperable platform. www.siemens.com

Mapped capabilities

4 capabilities

  • Planning-to-operations continuity

    Carrying long-term planning assumptions into real-time operational decisions on a common backbone.

  • Asset management and metering on shared data models

    Reusing harmonized asset and metering data consistently across workflows rather than per-tool silos.

  • Flexibility orchestration and system-wide visibility

    Coordinating distributed assets and grid capacity decisions with visibility across the operated system.

  • Open integration of third-party and custom applications

    Interoperating with non-Siemens technologies and customer-built applications on the platform.

04

Electrification X infrastructure IoT suite

Integrated IoT suite for MV/LV distribution grids, substations, renewable and industrial sites, targeting uptime, reliability, and cybersecurity across electrification and automation assets.

one integrated IoT suite for Electrification and Automation built to increase uptime, stability, reliability, cybersecurity and profitability www.siemens.com

Mapped capabilities

4 capabilities

  • Asset condition and health monitoring

    Equipment condition, operating data, and health status across distributed electrification assets.

  • Load and microgrid management

    EV charging load control, dynamic load management, and microgrid data transparency for renewable utilization.

  • Distribution grid visibility and fault detection

    Oversight of distribution stations and street cabinets, and fault monitoring across overhead and substation networks.

  • OT security posture support

    Continuous inventory, vendor-agnostic vulnerability and baseline management, and the associated workflows.

05

NX and Designcenter design-to-manufacturing

Integrated CAD and CAM software taking product engineering through part manufacturing, covering MCAD/ECAD design, CNC and robot programming, additive, and part quality monitoring.

Mapped capabilities

4 capabilities

  • Designcenter CAD product engineering

    MCAD and ECAD design work and workflow-specific bundles across startup to multinational scale.

  • NX CAM machine and robot programming

    Programming CNC machine tools, controlling robots, and driving 3D printers from one integrated system.

  • Part quality monitoring in the CAM workflow

    Monitoring part quality as part of the manufacturing program rather than as a separate downstream step.

  • Toolpath optimization via NX CAM plug-ins

    Digital-twin-driven feed rate and spindle speed optimization surfaced through plug-ins such as SenseNC.

06

Ecosystem, openness, and getting started

Cross-portfolio surfaces that determine how a buyer enters and extends the platform: the Xcelerator seller portfolio, open interfaces and partner technology, and trial and consultation entry points.

Mapped capabilities

4 capabilities

  • Siemens Xcelerator portfolio and partner offerings

    Discovering Siemens and third-party seller products and how partner solutions extend a Siemens product.

  • Open interfaces and third-party interoperability

    Documented openness claims across platforms, including app-based and third-party provider integration.

  • Trial, consultation, and guided-tour onboarding

    Free trial start, consultation requests, and guided tours as first-run paths into a platform.

  • Industrial AI positioning and partnerships

    How AI capabilities and the NVIDIA collaboration are represented relative to what a given product ships today.

Coverage is mapped from Siemens's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Siemens test?+

The coverage map is generated from Siemens's own public product surface (industrial technology, automation and engineering software portfolio): 6 scoring areas — Eigen Engineering Agent (TIA Portal), Building X building operations, and Gridscale X utility system operations, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Siemens evals scored?+

Every case generated for Siemens — across Eigen Engineering Agent (TIA Portal) and Building X building operations and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Siemens library include?+

The full Siemens library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, PLC code development in SCL and LAD and Automated test and validation generation under Eigen Engineering Agent (TIA Portal)); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Siemens or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Siemens areas and set them up in a Corsac workspace, where you can run every test case against Siemens or your own agent with your own data.