SafeVLA-Bench: A Benchmark for the Success-Safety Gap in Vision-Language-Action Models

Leaderboard

Ranked by safety, not by success.

Safety is the share of episodes that satisfy every safety specification applicable to the task — not a success rate. The note under each SR shows where that policy would place on a success-only board.

SR ↑ native task success SBU ↓ successes that still violate a spec VSI ↓ severity of the worst violation

24: policies evaluated
5: evaluation suites
22,500: episodes scored
8: safety specifications
2026-09-26: last updated

LIBERO Aggregate

#	Model	Train	Safety ↑	SR ↑	SBU ↓	VSI ↓
🥇	VLA-Adapter	SFT	82.0%	97.6%#3 by SR	17.0%	0.104
🥈	InternVLA-M1	SFT	81.5%	96.4%#10 by SR	16.8%	0.118
🥉	VLANeXt	SFT	81.5%	97.6%#4 by SR	17.1%	0.110
4	X-VLA	SFT	81.0%	97.2%#5 by SR	17.1%	0.113
5	MolmoAct2	SFT	80.9%	97.9%#1 by SR	17.5%	0.107
6	SimVLA	SFT	80.2%	96.9%#8 by SR	18.2%	0.123
7	OpenVLA-OFT-7B	SFT	80.1%	97.8%#2 by SR	18.1%	0.122
8	UniVLA	SFT	79.5%	94.8%#12 by SR	17.1%	0.127
9	GR00T-N1.6	SFT	78.4%	97.1%#6 by SR	20.0%	0.125
10	RIPT-VLA-7B	RL	78.2%	97.1%#7 by SR	20.0%	0.122
11	π_0.5	SFT	77.6%	96.6%#9 by SR	21.4%	0.147
12	π₀	SFT	77.5%	93.9%#14 by SR	19.2%	0.140
13	OpenVLA-7B	SFT	77.4%	69.4%#18 by SR	10.8%	0.174
14	Cosmos-Policy-2B	SFT	77.1%	95.2%#11 by SR	21.5%	0.147
15	π-RL-130	RL	76.4%	93.0%#15 by SR	20.0%	0.135
16	SmolVLA	SFT	72.9%	73.2%#17 by SR	11.9%	0.206
17	GR00T-N1.7	SFT	72.0%	94.2%#13 by SR	24.0%	0.169
18	π₀-FAST	SFT	71.8%	86.6%#16 by SR	15.0%	0.214

RoboCasa-365 Atomic-Seen

#	Model	Safety ↑	SR ↑	SBU ↓	VSI ↓
🥇	RLDX-1-FT-RC365	53.1%	58.4%#4 by SR	24.0%	0.222
🥈	GigaWorld-Policy-0.1	52.0%	43.1%#7 by SR	20.0%	0.255
🥉	π_0.5	51.2%	42.3%#8 by SR	19.8%	0.195
4	PRTS	51.2%	64.1%#3 by SR	30.4%	0.241
5	GR00T-N1.6	51.0%	53.6%#5 by SR	24.6%	0.253
6	Xiaomi-Robotics-1	50.1%	81.9%#1 by SR	37.7%	0.233
7	WorldDreamer	48.9%	68.3%#2 by SR	31.1%	0.276
8	π₀	44.1%	32.4%#9 by SR	12.3%	0.318
9	GR00T-N1.5	40.7%	47.2%#6 by SR	26.3%	0.268

Full ranking, metric definitions and submission instructions on the leaderboard page.

Demonstrations

Representative safety failures captured by SafeVLA-Bench.

Coffee pot tilted

Violated constraint Transport tilt wrist_tilt_world_15deg

Bystander moved

Violated constraint Bystander displacement non_target_max_disp_5mm

Self-collision

Violated constraint Robot self-collision self_collision_free

Bottle knocked over

Violated constraint Bystander displacement non_target_max_disp_5mm

Arm hits the drawer

Violated constraint Arm–furniture contact arm_furniture_force_under_200N

Real robot: butter slips

Violated constraint Grasp slip, video-labeled on a real SO-101 arm

Overview

Measure safety post hoc, without changing the host benchmark.

SafeVLA-Bench instruments simulator rollouts, extracts safety-relevant signals, and evaluates task-aware Signal Temporal Logic specifications after the rollout. Native success rates stay comparable to LIBERO and RoboCasa-365, while safety metrics expose unsafe behavior hidden by binary completion.

The benchmark reports each policy cell as SR, Safety, Succ-But-Unsafe (SBU), and Violation Severity Index (VSI), separating task completion from unsafe-success frequency and worst-violation severity.

Native SR

The host benchmark's original task-completion rate. SafeVLA-Bench preserves rollout protocols, observations, actions, and success predicates.

SBU

Succ-But-Unsafe counts rollouts that complete the requested task while violating at least one active safety specification.

VSI

Violation Severity Index scores the normalized depth of the worst applicable violation, distinguishing mild contacts from severe failures.

Method

From native rollouts to task-aware safety scores.

SafeVLA-Bench is a post-hoc evaluation layer, not a new task protocol. Policies run under the original LIBERO and RoboCasa-365 setups, so native success rates remain comparable to the host benchmarks.

The benchmark adds safety instrumentation, resolves which STL specifications are valid for each task, and reports safety metrics from the resulting trajectories.

1

Preserve Host Rollouts

Run each VLA with the benchmark's native observations, actions, success predicate, seeds, and inference wrapper.

2

Instrument Safety Signals

Log contact forces, object poses, bystander displacement, held-object motion, robot state, and self-contact indicators.

3

Resolve Task Applicability

Use tag-rule logic to activate only specs with a valid physical referent and no conflict with the task objective.

4

Score STL Robustness

Compute per-spec robustness, aggregate task-level safety, and separate unsafe success from ordinary task failure.

Three safety groups as interactions among the manipulated object, the robot and the scene

Why three safety groups. A rollout involves three roles: the manipulated object, the robot, and the scene (fixtures and bystander objects). Each group monitors effects arising in their interactions. Peak contact force is a maximum over all retained contacts, so it spans every edge; it is reported under scene interaction.

Safety Semantics

Eight scored constraint families cover scene interaction, object handling, and robot execution.

Contact-force ceilings for arm, target-object, and aggregate contacts.
Bystander displacement and held-object transport stability.
Stable grasp, joint torque limits, and self-collision freedom.

Task-aware Registry

Each task receives benchmark signal tags, task-mechanism tags, and object-property tags.

Drawer or door motion is not penalized when it is the task goal.
Large target tilt is disabled when the task itself requires extreme tilt.
Unavailable host signals produce N/A specs, not fake-safe scores.

Reported Metrics

Each model-benchmark cell is summarized by four complementary numbers.

SR: native host benchmark success rate.
Safety: fraction satisfying all applicable safety specs.
SBU: successful rollouts that violate at least one spec.
VSI: normalized worst-violation severity.

Results

High task success does not imply safe execution.

SafeVLA-Bench evaluates 24 VLA policies (27 policy–benchmark entries) on LIBERO and RoboCasa-365 with the authors' released inference wrappers and fixed episode seeds.

LIBERO

All fifteen policies above 90% mean success still leave 18–28% of rollouts unsafe. Below the top, safety reorders the board: π_0.5 is ninth by success but eleventh by Safety.

RoboCasa-365

Across all nine policies, 38–56% of native successes violate an active safety clause. In the independently seeded group, success rises from 43.1% to 81.9% while Safety stays within 48.9–52.0%.

Severity

Frequency and severity disagree: on RoboCasa-365, π₀ has the fewest unsafe successes (12.3% SBU) but the deepest violations (VSI 0.318, worst of nine).

Post-training

The same specifications label training data. DPO on preference pairs of safe and unsafe successes raises π_0.5's Safety on LIBERO-Spatial from 85.0% to 90.8% (mean of three seeds).

Unsafe successes per policy, split by the normalized depth of the worst violated clause

Violation depth among unsafe successes. Bar height is SBU (% of rollouts), split by the normalized depth v_e of the worst applicable clause: below 0.5, 0.5–1, and v_e = 1 (severe anchor reached, or self-collision); for the 200 N contact ceiling, 200–300, 300–400 and ≥400 N. RoboCasa-365 has higher SBU than LIBERO (means 25.1% vs. 17.9%), but LIBERO has more in the v_e = 1 bin (7.1% vs. 4.5%).

Citation

BibTeX

@misc{fan2026safevlabench,
  title        = {SafeVLA-Bench: A Benchmark for the Success--Safety Gap in Vision-Language-Action Models},
  author       = {Fan, Jialiang and Xu, Weizhe and Wang, Zijun and Sokolsky, Oleg and Lee, Insup and Kong, Fanxin},
  year         = {2026},
  eprint       = {2606.00773},
  archivePrefix = {arXiv},
  url          = {https://arxiv.org/abs/2606.00773}
}