AI-generated editorial illustration. It is explanatory and does not represent measured data or a documented experiment.

1. Core conclusion

Task-success rate remains useful, but it is too compressed to describe whether a vision-language-action policy is ready for deployment. A robot can trigger a benchmark’s final goal predicate after colliding with another object, and two policies with the same success rate can differ sharply in how they fail or whether they recover. Recent evaluation work points toward a layered scorecard: retain task completion, then add controlled perturbations, failure categories, recovery opportunities, trajectory-level safety constraints and violation severity. These additions still do not certify a robot as safe or reliable in an open human environment. They make a narrower improvement: benchmark results become more diagnostic and harder to mistake for deployment evidence.

2. The problem

Most manipulation benchmarks reduce an episode to a binary question: did the robot satisfy the task’s success condition before time ran out? That creates a clean percentage, supports repeatable comparisons and makes large experiments manageable. It also discards nearly everything that happened between the first observation and the final state. A successful rollout may contain a dropped object that was picked up again, unnecessary contact with the workspace, repeated motions, unstable transport or a collision that the goal predicate never checks.

Failure receives similarly weak treatment. If ten failed attempts are pooled into one number, the result does not reveal whether the model misunderstood the instruction, missed the grasp, oscillated between actions, lost track of the world state or could not align an object precisely. Those distinctions matter because they imply different technical remedies. More training data may help one failure; better force control, state estimation or explicit recovery behavior may help another.

Deployment also changes the distribution of conditions. Lighting shifts, cameras drift, objects begin in unfamiliar poses and instructions are paraphrased. A benchmark built around one clean arrangement can measure repeatability inside that arrangement while saying little about robustness outside it. The evaluation problem is therefore not to replace success rate with one superior number. It is to preserve completion while exposing the paths, disturbances and consequences hidden by the aggregate.

3. How it works

A richer evaluation begins by logging the complete rollout: observations, actions, object poses, contacts, robot state and task events. The evaluator can then ask several independent questions. Did the task finish? Did a controlled variation break the policy? What kind of failure occurred? When a recoverable error appeared, did the policy resume progress without human intervention? Did the trajectory obey safety constraints? If not, how severe was the worst violation?

These questions operate at different levels. Perturbation testing changes the input distribution while keeping the task recognizable. Failure taxonomy assigns interpretable labels to unsuccessful behavior. Recovery analysis marks moments when correction was possible and counts successful self-correction. Safety evaluation inspects the trajectory even when the final goal was reached. Formal rules such as Signal Temporal Logic can express conditions over time—for example, that contact force stays below a threshold or a bystander object remains within a displacement limit.

The resulting scorecard is deliberately plural. Task success measures outcome. A robustness curve measures degradation as conditions vary. Failure distributions identify dominant weaknesses. Recovery rate measures successful recoveries divided by identified recovery opportunities. Successful-but-unsafe rate measures episodes that finish while violating at least one active safety rule. A severity score distinguishes a slight threshold crossing from a much deeper violation.

AI-generated editorial illustration. It is explanatory and does not represent measured data or a documented experiment.

4. Representative approaches

RADAR treats generalization as a controlled stress test. The 2026 benchmark varies manipulated objects, robot initial states, instructions and environmental conditions, including lighting, distractors and sensor noise. It adds tasks intended to probe spatial and physical reasoning, then uses multi-view sensing and three-dimensional metrics for automated scoring. This path asks whether competence survives a changing physical setup rather than whether a policy repeats one familiar routine. The paper reports that its expected 3D intersection-over-union score fell from 0.261 to 0.068 under sensor noise, a result tied to RADAR’s models, apparatus and perturbation settings.

The SO-101 study opens the failed episode. Yi Yu and Xinchuan Qiu evaluated π0.5, SmolVLA, Wall-X and ACT on four physical manipulation tasks using the low-cost SO-101 arm. They classified dominant failures as grasp instability, repetition loop, state mismatch or precision misalignment, and separated semantic-level from execution-level problems. They also defined recovery rate around identified opportunities to self-correct. Reported recovery rates ranged from 3.23% for SmolVLA to 30.77% for π0.5, while execution instability remained a prominent source of failure. This approach is valuable because identical hardware and task protocols reveal behavioral differences that an overall score can conceal.

SafeVLA-Bench inspects successful trajectories for unsafe behavior. It runs policies under the native LIBERO and RoboCasa-365 simulator protocols, then applies task-aware safety specifications after each rollout. Its reported metrics include native success, overall safety, Successful-but-Unsafe (SBU) and Violation Severity Index (VSI). The specification library covers contact forces, bystander displacement, held-object stability, grasp stability, joint torque and self-collision, while applicability rules disable clauses that conflict with the task itself. For example, opening a drawer should not be penalized simply because the drawer moved.

Safety-CHORES and SafeVLA connect measurement to training. SafeVLA formulates safety requirements in a constrained Markov decision process and uses a purpose-built environment to elicit and reduce unsafe behavior. It differs from a post-hoc benchmark: the safety signal shapes policy optimization. Its reported improvements are author-reported results within Safety-CHORES and do not establish physical-world safety. Together, these approaches show a useful division of labor: stress-test distribution shifts, diagnose failures, measure recovery, score trajectory safety, and only then use those signals to change a policy.

5. Real-world applications

The most immediate application is model selection for a defined robot cell. A manufacturer choosing between two policies for packing, sorting or insertion needs to know more than average completion. It needs to know whether failure comes from perception or motion, how often the robot can recover, what happens under camera noise and whether successful cycles disturb neighboring objects. A multi-metric report can expose that operational profile before a limited pilot expands.

The same framework can guide engineering. Repetition loops may motivate progress detection or replanning; grasp instability may point toward better contact sensing or low-level control; state mismatch may call for observation refresh and explicit belief tracking. Safety violations can identify which variables require monitoring, runtime limits or mechanical safeguards. None of this removes the need for site-specific risk assessment, protective hardware and human supervision. A benchmark supplies evidence about a tested system under documented conditions; it does not issue a general safety certificate.

6. How it is evaluated

RADAR’s contribution is the combination of controlled real-world dynamics, spatial tasks and autonomous 3D scoring. Its system records trajectories and computes geometric outcomes rather than relying only on a human observer or a two-dimensional detector. The reported sensor-noise result demonstrates sensitivity under one benchmark protocol, not a universal failure rate for VLA models.

The SO-101 benchmark uses four real manipulation tasks—pen transfer, color sorting, multi-object packing and precision pen placement—and evaluates four policy families on the same inexpensive embodiment. Its recovery rate is the number of successful recoveries divided by the number of identified recovery opportunities. That denominator is essential: a model cannot be credited for recovery when no recoverable error occurred. The study reports π0.5 at 30.77%, Wall-X at 20.51%, ACT at 6.45% and SmolVLA at 3.23%, within its task definitions and annotation protocol.

SafeVLA-Bench preserves native success predicates and adds post-hoc safety scoring to LIBERO and RoboCasa-365. Across nine policy–benchmark entries, the paper reports that high-success LIBERO baselines still produced 13–15% unsafe episodes, while 36–56% of successful RoboCasa-365 rollouts violated an active safety clause. SBU captures frequency; VSI uses normalized Signal Temporal Logic robustness to capture the depth of the worst applicable violation. The two metrics can disagree, which is informative: rare violations may be severe, while frequent violations may be shallow.

AI-generated editorial illustration. It is explanatory and does not represent measured data or a documented experiment.

7. Current limitations

Every richer metric introduces judgment. Failure taxonomies depend on definitions and annotation consistency. A recovery opportunity must be identified before recovery rate can be computed, and borderline cases may be ambiguous. Safety thresholds depend on the robot, object, task and environment. A contact limit suitable for a simulator or industrial fixture cannot automatically be transferred to a home, hospital or human-collaborative workspace.

SafeVLA-Bench is explicitly simulator-only. Its authors note that MuJoCo under-resolves transient impacts and that force, displacement, tilt and torque thresholds are scoring proxies, not calibrated physical safety boundaries. RADAR adds physical hardware and automated 3D evaluation, but one apparatus and task family cannot represent all embodiments, sensor stacks or deployment conditions. The SO-101 study provides accessible real-robot evidence, yet four tasks on one low-cost arm remain a narrow sample.

Metrics can also be gamed. Optimizing one threshold may shift risk elsewhere, and aggregating many measures into a single leaderboard score would recreate the compression problem. Rare, high-consequence events require more trials than typical robotics studies can afford. Human proximity, deformable objects, mobile whole-body behavior, wear, network failure and long-duration autonomy remain thinly covered. Better evaluation narrows uncertainty; it does not eliminate it.

AI-generated editorial illustration. It is explanatory and does not represent measured data or a documented experiment.

8. Why it deserves attention

Robotics is moving from demonstrations toward repeated work. That transition changes the important question from “can the robot do it once?” to “how does it behave across variation, error and consequence?” Success rate cannot answer that alone. A benchmark that publishes perturbation conditions, failure distributions, recovery denominators and safety violations gives readers and engineers a clearer map of what has—and has not—been demonstrated.

The strongest future evidence would combine standardized simulated stress tests with calibrated physical measurements across multiple robots and sites. It would report confidence intervals, release failure annotations, preserve raw trajectory logs, document threshold choices and measure intervention-free runtime. Most importantly, it would keep outcome, robustness, recovery and safety separate rather than collapsing them into a reassuring headline number.

9. Sources & evidence

1. Yuhao Chen et al., “RADAR: Benchmarking Vision-Language-Action Generalization via Real-World Dynamics, Spatial-Physical Intelligence, and Autonomous Evaluation,” arXiv, 11 February 2026.
2. Yi Yu and Xinchuan Qiu, “Benchmarking Vision-Language-Action Models on SO-101: Failure and Recovery Analysis,” arXiv v2, 11 June 2026.
3. Jialiang Fan et al., “SafeVLA-Bench: A Benchmark for the Success–Safety Gap in Vision-Language-Action Models,” arXiv, 30 May 2026.
4. SafeVLA-Bench authors, “SafeVLA-Bench project page,” 2026.
5. Borong Zhang et al., “SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learning,” NeurIPS 2025 Spotlight project page.

All numerical results above are attributed to the respective papers or project pages and remain limited to their reported models, tasks, hardware or simulator conditions.

Related reading

• A Robot Needs an Action Vocabulary. No Tokenizer Preserves Everything
• Vision-Language-Action Models Are Learning to Touch—but Contact Needs a Faster Loop
• Robot World Models Are Becoming Action Models. Prediction Is Not Yet Reliable Planning

Sources — VLA evaluation beyond success rate

• RADAR: Benchmarking Vision-Language-Action Generalization via Real-World Dynamics, Spatial-Physical Intelligence, and Autonomous Evaluation

  • Authors: Yuhao Chen et al.
  • URL: https://arxiv.org/abs/2602.10980
  • Published: 2026-02-11
  • Accessed: 2026-09-01
  • Type: Primary benchmark paper
  • Supports: Controlled physical, visual, semantic and sensor perturbations; autonomous 3D evaluation; reported expected 3D IoU change from 0.261 to 0.068 under sensor noise.

• Benchmarking Vision-Language-Action Models on SO-101: Failure and Recovery Analysis

  • Authors: Yi Yu and Xinchuan Qiu
  • URL: https://arxiv.org/abs/2606.08881
  • Published: 2026-06-07; revised 2026-06-11
  • Accessed: 2026-09-01
  • Type: Primary real-robot benchmark paper
  • Supports: Four tasks and four evaluated policies; failure taxonomy; semantic/execution decomposition; recovery-rate definition and reported values.

• SafeVLA-Bench: A Benchmark for the Success–Safety Gap in Vision-Language-Action Models

  • Authors: Jialiang Fan, Weizhe Xu, Oleg Sokolsky, Insup Lee and Fanxin Kong
  • URL: https://arxiv.org/abs/2606.00773
  • Published: 2026-05-30
  • Accessed: 2026-09-01
  • Type: Primary simulator benchmark paper
  • Supports: Task-aware STL specifications; native SR, Safety, SBU and VSI; nine policy–benchmark entries; limitations of simulator contact and threshold proxies.

• SafeVLA-Bench project page

  • Institution: SafeVLA-Bench authors
  • URL: https://safevla.org/
  • Published: 2026
  • Accessed: 2026-09-01
  • Type: Official project documentation
  • Supports: Safety-signal instrumentation, eight constraint families, applicability rules, aggregate LIBERO and RoboCasa-365 results.

• SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learning

  • Authors: Borong Zhang et al.
  • URL: https://safevla.github.io/
  • Published: 2025; NeurIPS 2025 Spotlight
  • Accessed: 2026-09-01
  • Type: Primary project page and paper
  • Supports: Constrained Markov decision process framing, Safety-CHORES and safe reinforcement-learning approach; author-reported safety/performance claims.