Physical AI

Embodied AI research, evaluated after it leaves the simulator

How research skill shows when a learned policy has to run on hardware that breaks, and the standard the founding cohort will hold it to. Everything below is a draft in public, on purpose.

Simulation kit

From the simulator to the physical system, without buying hardware

Teaching modules to start physical-AI research. Each card says where it runs: on a board simulated in the browser (Wokwi) or on your own computer. All follow the same loop: sensor → model → actuator → measurement. A project's link appears on its card once it is published.

  • In the browser (Wokwi)

    Engine failure (NASA C-MAPSS)

    An INT8 model reads 14 engine sensors and raises the alert before failure — the same code in the notebook, in the simulator and on the board.

    Tracks

    • Production engineering
    • Embedded systems

    Parts

    • ESP32 DevKit v1
    • 14 C-MAPSS sensors (replayed data)
    • LEDs and 220 Ω resistors
  • In the browser (Wokwi)

    Impact detection

    Accelerometer windows go through a small network; a hit lights the LED and sounds the buzzer.

    Tracks

    • Embedded systems
    • Industrial automation

    Parts

    • ESP32 DevKit v1
    • MPU6050 accelerometer
    • LEDs and 220 Ω resistors
    • Buzzer
  • On your computer

    Cracks in concrete

    A 5,921-parameter network classifies concrete photos from the webcam; the computer plays the edge device.

    Tracks

    • Industrial automation
    • Production engineering

    Parts

    • Webcam
  • In the browser (Wokwi)

    Incubator with automatic dosing

    A controller switches the heater to hold 37 °C and opens the valve by itself once the temperature settles.

    Tracks

    • Biofabrication and autonomous lab

    Parts

    • ESP32 DevKit v1
    • NTC temperature sensor
    • Relay module
    • Servo motor
    • Push button
    • LEDs and 220 Ω resistors
  • In the browser (Wokwi)

    Reject gate on a line

    An ultrasonic sensor measures each part's height and a servo diverts the ones out of tolerance — a physical rule ready to be replaced by a trained model.

    Tracks

    • Industrial automation

    Parts

    • ESP32 DevKit v1
    • HC-SR04 ultrasonic sensor
    • Servo motor
    • Push button
    • LEDs and 220 Ω resistors
  • In the browser (Wokwi)

    Same model, another chip

    The engine code is ported unchanged to a Raspberry Pi Pico; it does not fit on an Arduino Uno, and why is the lesson.

    Tracks

    • Embedded systems

    Parts

    • Raspberry Pi Pico
    • Arduino Uno (for comparison)
    • 14 C-MAPSS sensors (replayed data)
    • LEDs and 220 Ω resistors

What counts as evidence of physical AI research skill

The field's defining problem is that its headline numbers are cheap. A success rate in simulation costs nothing to inflate — change the reset distribution, tune the reward, report the best seed — and the same policy on real hardware may not clear a tenth of it. Evidence of research skill is therefore evidence about transfer: what the policy did on the physical system, over how many trials, under conditions the author did not choose in advance.

Trial counts and failure taxonomies carry more signal than benchmark scores. Twenty real-robot episodes with every failure classified tells an evaluator more than a leaderboard entry with none, because the classification is where judgment lives: whether a miss was perception, control, calibration or a mechanical fault is a claim the author had to earn.

Reproducibility in this discipline is unusually hard and unusually informative. Hardware drifts, calibration ages, and a policy that works on one arm may not work on its twin. Researchers who report cross-platform or cross-session results — including the ones that degraded — are giving an evaluator exactly the artifact the field's incentives discourage.

The hardest skill to see is honest safety and reset accounting. Interventions per hour, resets excluded from the numbers, the episodes cut short because something was about to break: these are the details that separate a demonstration from a result, and they are almost never in the abstract.

The physical AI evaluation rubric, first draft

This rubric reads embodied work the way a skeptical replicator reads a demo video: for what happened off-camera, and for how much of the claim survives leaving the lab it was made in.

Sim-to-real accounting
Simulated and physical results are reported separately and comparably, with the transfer gap stated as a finding rather than smoothed over. Randomization and tuning applied to close the gap are disclosed.
Trial protocol integrity
Trial counts, seeds, reset procedures, human interventions and excluded episodes are reported. Conditions are fixed before evaluation rather than selected after, and the selection rule is stated either way.
Failure characterization
Failures are classified by subsystem and mechanism rather than counted, and the evidence separating perception failures from control, calibration or hardware failures is given.
Embodiment generalization
Claims are bounded by the platforms and conditions actually tested. Transfer across robots, sessions, or environments is demonstrated rather than asserted, and degradation is reported where it occurred.

Founding physical AI evaluators will attack this draft first — including against their own demonstrations and benchmark entries.

Rubric drafts may include contributions from programme partners. Partners hold no scoring weight.

What founding physical AI evaluators will do

Take the rubric apart: where it penalizes work that legitimately cannot reach hardware, where it over-credits expensive robot fleets, where a dimension cannot be scored from a paper and a video alone.

Run calibration rounds on public work — open benchmark entries, released policies and datasets, reproduction attempts, conference demonstrations — scoring independently and comparing spreads, so the first published scores carry known uncertainty rather than the field's usual confidence.

Establish what a credible real-robot claim must report, and publish it as a standard the community can hold entries to — including its own.

Who this is for

The founding cohort is looking for researchers whose systems have to survive contact with the world, wherever they work:

  • Robot-learning researchers who report real-hardware results and want the reporting itself judged.
  • Control and perception engineers whose deployment work never becomes a paper.
  • Reproduction and benchmark maintainers who have already tried to rerun other people's claims.
  • Embodied-AI practitioners in industry whose evidence is fleet data rather than a leaderboard.

Who this is not for

Self-selection matters more than any filter we could write, so here is the honest version:

  • Anyone after a credential for its own sake — the founding stage produces standards, not badges.
  • Anyone whose evidence is a demonstration video with no protocol behind it; this rubric is built to ask what the video omits.
  • Anyone uncomfortable having their evaluation accuracy tracked — the calibration record is the point of the design.
  • Anyone who needs a live scoring platform today; the mechanics on this page are in design, and the tense is deliberate.

Apply to evaluate physical AI

Physical AI is pre-selected on the application. Link to work we can read — a real-robot evaluation, a released policy or dataset, a reproduction report, an ORCID or repository profile.

VocaidDeep