AI-generated editorial illustration. It explains robot-policy evaluation and does not represent measured data.

1. The central question

A robot policy completes 78 percent of its trials. Is it good?

The number sounds decisive, but it is incomplete. The trials may cover one object, one camera position and one laboratory. They may exclude resets, human interventions and unsafe motions. A failed attempt near the final step may be counted identically to a robot that never moved. A policy that succeeds on memorized tasks can therefore outrank one that attempts unfamiliar tasks, recognizes uncertainty and recovers safely.

For a specialist machine, a narrow success rate may answer a useful engineering question: can this fixed system repeat this fixed operation? A generalist policy makes a broader claim. It is supposed to work across tasks, objects, layouts, instructions and sometimes robot bodies. Its evaluation must reveal where that breadth is real.

The right question is not simply “How often did it succeed?” It is: what distribution was tested, what counted as success, how reliable was the estimate, and what happened when conditions changed or execution went wrong?

2. Conceptual foundation

A policy maps observations and instructions to actions. Evaluation samples that mapping under defined conditions. The result is never a context-free property of the model; it is a measurement produced by a task definition, environment distribution, hardware setup, rollout protocol and scoring rule.

Three distinctions matter.

First, task competence is not the same as generality. A policy may execute many tasks because they share familiar objects and motions, yet fail when visual appearance, geometry or language changes. Breadth should be measured across explicit axes rather than inferred from a task count.

Second, average performance is not the same as reliability. A mean can hide catastrophic failures, large variation between tasks, or dependence on favorable starting states. Repeated trials and uncertainty estimates show whether a reported difference is stable.

Third, completion is not the whole operational outcome. In physical settings, time, collisions, human takeover, recovery, resource use and failure severity can matter as much as binary success. A policy that pauses safely and requests help may be preferable to one that occasionally completes a task but behaves unpredictably.

3. How the system works

A credible evaluation begins by defining the claim. If a team claims multi-task competence, the protocol should separate trained and held-out tasks. If it claims environmental robustness, the test should vary objects, backgrounds, lighting, camera placement and initial configurations. If it claims real-world usefulness, simulation results must be supplemented by physical trials under representative operating conditions.

Each rollout then needs a reproducible specification: the instruction, initial state, allowed time or action horizon, termination conditions, success predicate, intervention policy and reset procedure. The evaluator should record more than the final bit. Useful traces include subgoal completion, contact events, policy confidence where available, recovery attempts, human assistance, elapsed time and failure category.

Results should be aggregated in layers. Report per-task outcomes before an overall average; separate seen from unseen conditions; show repeated-trial counts and uncertainty; and disclose exclusions. The final output is best understood as a capability profile, not a single leaderboard number.

4. Major approaches

Fixed-task success

The simplest approach repeats a predefined task and reports the fraction of successful rollouts. It is easy to communicate and compare when the environment, hardware and success predicate are controlled. It remains valuable for diagnosing whether a method improves a specific skill.

Its weakness is scope. A high score can reflect adaptation to a benchmark’s object set, camera geometry or reset distribution. Without held-out conditions, the metric says little about generality.

Out-of-distribution generalization

Generalization tests deliberately alter one or more variables: novel objects, unseen layouts, new visual appearances, paraphrased instructions or held-out task compositions. The test should name the axis of novelty. “Unseen” is not sufficient if the reader cannot tell whether it means a new texture, a new object category or a genuinely new sequence of skills.

This approach reveals whether the policy transfers structure rather than memorizing correlations. It also exposes trade-offs: a model may retain performance under visual changes but fail under new geometry or task composition.

Long-horizon composite tasks

Composite tasks require several skills in sequence—navigate, open, pick, place, close—while preserving state across steps. They test planning, memory and error accumulation. A binary final score is especially lossy here. Subgoal progress and the location of failure help distinguish a planning error from a grasping error or a timeout.

Long horizons are closer to useful household work, but protocols become more sensitive to action limits, reset details and simulator behavior. Comparisons require unusually explicit reporting.

Repeated real-robot trials

Physical trials test sensing noise, latency, calibration drift, wear, contact uncertainty and environmental variation that simulation cannot fully reproduce. They should record human interventions, aborted runs, safety stops and recovery behavior. A small number of curated demonstrations is evidence of possibility, not a reliable estimate of performance.

Real trials are expensive, so fewer conditions may be tested. That makes transparent rollout counts, sampling procedures and failure logs essential.

5. Evidence and examples

LIBERO illustrates why benchmark design must match the learning claim. Its four suites contain 130 manipulation tasks and were designed to study transfer in lifelong robot learning, including procedural and declarative knowledge. The benchmark’s value is not merely the number of tasks; it is the structure used to ask what transfers as a policy learns sequentially.

RoboCasa365 expands evaluation along task and environment dimensions. Its official materials describe 365 everyday kitchen tasks across 2,500 environments, with atomic and composite tasks plus seen and unseen evaluation splits. The separation matters: an overall average can conceal a large gap between familiar atomic skills and unfamiliar task compositions.

BEHAVIOR-1K emphasizes long-horizon, human-centered activities. It defines 1,000 household activities grounded in 50 interactive scenes and includes complex object states and physical interactions. Such tasks make partial progress informative. A robot that retrieves four required items but cannot manipulate a deformable object has a different limitation from one that fails to navigate to the workspace.

DROID addresses another part of the evidence chain: real-world diversity. Its dataset contains 76,000 demonstration trajectories, about 350 hours of interaction, collected across 564 scenes and 84 tasks by 50 collectors. Training diversity can improve robustness, but dataset scale does not substitute for evaluation. A policy trained on broad real-world data should still be tested on declared held-out environments, repeated physical trials and operational failures.

Together these resources show that “generalist” is not a single benchmark property. LIBERO probes transfer over task sequences; RoboCasa365 provides controlled scale and compositional splits; BEHAVIOR-1K stresses long-horizon household activity in simulation; DROID supplies geographically and visually varied physical interaction data. None alone establishes deployment readiness.

6. Trade-offs

Standardization versus realism. Standard benchmarks make methods comparable, but controlled conditions omit many disturbances. More realistic tests capture operational difficulty, yet hardware and site differences weaken direct comparison.

Simulation scale versus physical credibility. Simulation can run many tasks and controlled counterfactuals. Physical trials reveal contact, sensing and maintenance problems. Strong evidence uses simulation for breadth and real robots for calibration and confirmation.

One leaderboard versus a multidimensional profile. A ranking is easy to read and incentivizes progress. It also encourages optimization for one aggregate. Separate scores for seen tasks, unseen composition, environment shift, long-horizon completion, recovery and safety preserve the shape of performance.

Reproducibility versus hardware variation. Identical software can behave differently across cameras, grippers, control frequencies and calibration procedures. Reproducibility therefore requires both open policy artifacts and detailed embodiment documentation. Cross-laboratory replication should be treated as new evidence, not assumed equivalence.

7. Current limitations

Robot evaluation lacks a universal failure taxonomy. Papers may report “failure” without separating perception errors, instruction misunderstanding, poor grasp selection, control instability, planning mistakes, timeouts or environmental faults. That makes it difficult to compare systems or determine what improvement is needed.

Rollout budgets are often too small to estimate rare failures. If a policy is tested ten times, one additional success changes its score by ten percentage points. Confidence intervals and per-task trial counts are therefore not statistical decoration; they define how much trust the result deserves.

Benchmarks also age. Public task suites can influence training choices, hyperparameters and model selection even when nominal test demonstrations remain hidden. A strong evaluation program refreshes held-out conditions and separates development feedback from final testing.

Finally, safety and recovery remain underreported. Binary success can reward aggressive behavior if collisions, near misses or emergency stops are not part of the score. Deployment-oriented evaluation needs severity-weighted failures and explicit human-takeover accounting.

8. Open questions

The field still needs agreement on a compact but useful failure vocabulary. It must be detailed enough to guide engineering, yet consistent enough to support comparisons across tasks and embodiments.

There is also no settled minimum for rollout counts. The answer depends on expected reliability, task variance and failure cost. Reports should justify their sample size and publish uncertainty rather than rely on a universal threshold.

Recovery deserves its own protocol. Can a policy recognize a failed grasp, re-plan after an object moves, or request assistance before creating risk? Measuring successful recovery, time to recovery and unnecessary interventions would expose capabilities hidden by terminal success.

The hardest question is how to combine metrics without recreating the same problem under a new composite score. A public profile may be more honest than a universal scalar: task success, distribution shift, long-horizon progress, safety events, intervention rate, efficiency and reproducibility shown side by side.

9. Editorial synthesis

Success rate should remain part of robot-policy evaluation, but it should lose its privileged status as the answer. The number becomes meaningful only when attached to a declared test distribution and a transparent protocol.

For research claims, the minimum credible package is per-task repeated trials, seen and held-out splits, uncertainty estimates and failure categories. For generalist claims, add controlled shifts in objects, layouts, language and task composition. For deployment claims, add physical trials, intervention records, safety events, recovery and operational constraints.

The objective is not to make every experiment enormous. It is to make the boundary of the evidence visible. A narrow but well-described result is more useful than a broad claim supported by an opaque average.

10. Key takeaways

• A success rate measures performance only within a specified task, environment, hardware and rollout protocol.
• Generality should be tested across named axes such as novel objects, layouts, instructions and task compositions.
• Long-horizon tasks require subgoal and failure-stage reporting, not only final completion.
• Real-robot evaluations should disclose trial counts, uncertainty, interventions, safety stops and recovery behavior.
• The most informative result is a multidimensional capability profile rather than one aggregate leaderboard score.

11. Further reading

• What Is a Robot Foundation Model?
• How Robots Learn from Demonstrations
• Crossing the Reality Gap: What Actually Transfers from Simulation to a Robot

12. Sources & evidence

This article relies on primary benchmark papers and official project documentation. Quantitative descriptions refer to the released benchmark or dataset scope, not to an independent claim of deployment readiness. Full source notes and claim mapping are stored in sources.md and research-ledger.md.

Sources

Primary sources

1. LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning — Bo Liu et al., NeurIPS 2023 Dataset and Benchmark Track.
   Paper: https://arxiv.org/abs/2306.03310
   Project: https://libero-project.github.io/main.html
   Used for: four benchmark suites, 130 total tasks, lifelong robot learning and knowledge-transfer framing.

2. RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robots — Soroush Nasiriany et al., ICLR 2026.
   Paper: https://robocasa.ai/assets/robocasa365_iclr26.pdf
   Project: https://robocasa.ai/
   Benchmark documentation: https://robocasa.ai/leaderboard.html
   Used for: 365 tasks, 2,500 kitchen environments, atomic/composite tasks, seen/unseen compositions and split-level evaluation.

3. BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation — Chengshu Li et al., 2024.
   Paper: https://arxiv.org/abs/2403.09227
   Project: https://behavior.stanford.edu/
   Used for: 1,000 activity definitions, 50 scenes, long-horizon household tasks, interactive physics and initial sim-to-real calibration.

4. DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset — Alexander Khazatsky et al., 2024; revised 2025.
   Paper: https://arxiv.org/abs/2403.12945
   Project: https://droid-dataset.github.io/
   Used for: 76,000 demonstration trajectories, about 350 hours of data, 564 scenes, 84 tasks, 50 collectors and the distinction between training diversity and evaluation evidence.

Source policy

• Quantitative facts are attributed to benchmark authors or official project documentation.
• The article does not treat benchmark authors’ results as independent validation of deployment readiness.
• No current leaderboard ranking is reproduced.
• Editorial recommendations about uncertainty, failure categories, interventions and multidimensional reporting are synthesis, not claims that these four projects use one shared protocol.