Leaderboard
Safety is the share of episodes that satisfy every safety specification applicable to the task — not a success rate. Ranking by it rather than by SR is the point: the note under each SR shows where that policy would place on a success-only board, and the two orders do not agree. The full definitions are below.
SR ↑ native task success SBU ↓ successes that still violate a spec VSI ↓ severity of the worst violation
| # | Model | Safety ↑ | SR ↑ | SBU ↓ | VSI ↓ |
|---|---|---|---|---|---|
| 🥇 | RLDX-1-FT-RC365 | 53.0% | 58.4%#2 by SR | 24.1% | 0.132 |
| 🥈 | π0.5 | 51.2% | 42.3%#4 by SR | 19.8% | 0.114 |
| 🥉 | π0 | 44.0% | 32.4%#6 by SR | 12.4% | 0.198 |
| 4 | GR00T-N1.5 | 40.7% | 47.2%#3 by SR | 26.3% | 0.173 |
| 5 | PRTS | 40.2% | 42.1%#5 by SR | 19.6% | 0.268 |
| 6 | Xiaomi-Robotics-1 | 36.0% | 59.6%#1 by SR | 27.8% | 0.338 |
| # | Model | Train | Safety ↑ | SR ↑ | SBU ↓ | VSI ↓ |
|---|---|---|---|---|---|---|
| 🥇 | Cosmos-Policy-2B | SFT | 82.0% | 95.2%#2 by SR | 16.6% | 0.084 |
| 🥈 | π0.5 | SFT | 80.8% | 96.6%#1 by SR | 18.4% | 0.086 |
| 🥉 | OpenVLA-7B | SFT | 80.6% | 69.4%#5 by SR | 8.5% | 0.125 |
| 4 | π-RL-130 | RL | 80.4% | 93.0%#4 by SR | 17.4% | 0.080 |
| 5 | GR00T-N1.7 | SFT | 79.1% | 94.2%#3 by SR | 17.8% | 0.093 |
Ranked on unrounded Safety, so rows that tie on screen need not tie in the ordering. Every LIBERO row runs 200 rollouts per suite (10 tasks × 20 episodes); every RoboCasa-365 row runs 900 (18 task families × 50 episodes).
RoboCasa-365 atomic-seen covers 18 kitchen task families at 50 episodes each; LIBERO covers four tabletop suites. Policies run through their native inference wrappers with fixed seeds, so the success rate reported here is the one the policy's own harness produces — we add the safety layer on top rather than re-implementing the task.
SR is the benchmark's native success rate. Safety is the fraction of all episodes satisfying every applicable STL specification. SBU (Succ-But-Unsafe) is the fraction of episodes that both succeed and violate — the failures an SR-only evaluation cannot see. VSI is a bounded worst-violation depth, so a policy can have few unsafe successes and still score badly when the violations it does commit are severe.
Open a pull request against
SafeVLA-Bench-release
adding one JSON file under submissions/ with your policy name, the
benchmark and suite, episode count, and the per-episode verdict file produced by
scripts/compute_stl_robustness.py. We re-score from the raw trajectories
rather than taking reported numbers, so please include the per-step logs or a link to
them. Entries currently on this board were all evaluated by us.
Scores come from the same aggregate the paper reads, regenerated by
scripts/make_leaderboard.py. RoboCasa-365 verdicts were recomputed from the
per-step force traces in September 2026; earlier numbers derived from the action log alone
understated contact violations. See the
paper for the specification library and
threshold sensitivity analysis.