Overview
All datasets · 10 sources · 898 episodes
Total episodes
Overall success rate
103 of 650 scored · 248 unscored
Avg. episode duration
Collision rate
of episodes with collision telemetry
Policy failure rate
Real historical outcomes per named policy checkpoint, from independent third-party evaluations
| policy checkpoint | trials | failures | failure rate | 95% confidence interval |
|---|---|---|---|---|
| paligemma_binning_droid | 65 | 63 | 96.9% | [89.5%, 99.2%] |
| pi0_droid (closed-weight) | 101 | 92 | 91.1% | [83.9%, 95.2%] |
| paligemma_vq_droid | 75 | 66 | 88.0% | [78.7%, 93.6%] |
| pi0_fast_droid (closed-weight) | 64 | 56 | 87.5% | [77.2%, 93.5%] |
| paligemma_fast_droid | 86 | 74 | 86.0% | [77.2%, 91.8%] |
| paligemma_fast_specialist_droid | 92 | 79 | 85.9% | [77.3%, 91.6%] |
| pi05_droid (closed-weight) | 25 | 21 | 84.0% | [65.3%, 93.6%] |
| paligemma_diffusion_droid | 73 | 61 | 83.6% | [73.4%, 90.3%] |
| human-teleop | 60 | 27 | 45.0% | [33.1%, 57.5%] |
Each row is a real, named policy checkpoint's measured pass/fail record across independent evaluation trials, pooled across every task it was tested on, with a Wilson confidence interval on the true rate. This describes that exact checkpoint's own historical behavior, not a specific customer's robot or environment. How we verify this.
Policy duration risk
Duration-anomaly rollup per deployed policy version, worst first
| policy version | task | scored runs | unusual | highly unusual | status |
|---|---|---|---|---|---|
| human-teleop | Push T-block to target | 52 | 4 | 2 | Highly unusual runs |
| paligemma_binning_droid | Pick up the pineapple and place into the bowl | 1 | 1 | 0 | Some unusual runs |
| human-teleop | Push T-block to target (real UR5) | 8 | 0 | 0 | Typical |
| pi0_fast_droid (closed-weight) | Put the cylinder in the bowl | 1 | 0 | 0 | Typical |
| paligemma_fast_droid | Put the cylinder in the bowl | 1 | 0 | 0 | Typical |
| paligemma_diffusion_droid | Put the cylinder in the bowl | 1 | 0 | 0 | Typical |
| paligemma_vq_droid | Put the cylinder in the bowl | 1 | 0 | 0 | Typical |
| paligemma_binning_droid | Put the cylinder in the bowl | 1 | 0 | 0 | Typical |
| pi0_droid (closed-weight) | Put the cylinder in the bowl | 1 | 0 | 0 | Typical |
| paligemma_fast_specialist_droid | Put the cylinder in the bowl | 1 | 0 | 0 | Typical |
Rolls up the same per-episode duration-anomaly score by policy version: how often a policy's logged runs have looked statistically unlike its own task's successful runs so far. It does not predict what the policy will do on its next run. How we verify this.
Rollouts by day
Stacked outcomes across all datasets, last 14 days
Failures by category
All episodes, all versions
Datasets
Ingested via format adapters, canonical schema underneath
| dataset | format | episodes | avg. coverage | ingested |
|---|---|---|---|---|
| PushT - T-block pushing (LeRobot community) lerobot-pusht | LeRobot | 52 | 100% | Sep 27, 2025, 11:40 AM |
| ALOHA static coffee (bimanual teleop) lerobot-aloha_static_coffee | LeRobot | 50 | 80% | Sep 27, 2025, 11:40 AM |
| SO-101 pick & place (SmolVLA) lerobot-svla_so101_pickplace | LeRobot | 50 | 80% | Sep 27, 2025, 11:25 AM |
| Mobile ALOHA - cook shrimp (Stanford) lerobot-aloha_mobile_shrimp | LeRobot | 18 | 80% | Mar 10, 2026, 6:44 PM |
| Mobile ALOHA - wash pan (Stanford) lerobot-aloha_mobile_wash_pan | LeRobot | 50 | 80% | Sep 27, 2025, 11:40 AM |
| RJ45 cable insertion - ACT policy eval (Piper) BensoAI-eval_ACT_RJ45_piper_20260113_ckpt40k_ensemble_50runs | LeRobot | 50 | 80% | Jan 14, 2026, 9:14 AM |
| Sawyer laundry layout (RoboTurk, real teleop) roboturk-sawyer-laundry-layout | CSV + video | 10 | 80% | Jul 28, 2026, 5:20 AM |
| PushT real robot (UR5, Open X-Embodiment / Diffusion Policy) oxe-columbia-pusht-real | RLDS | 8 | 100% | Jul 28, 2026, 6:25 AM |
| PushT sim replay buffer (Diffusion Policy, native Zarr) dp-pusht-zarr | Zarr | 20 | 80% | Jul 28, 2026, 6:39 AM |
| RoboArena eval dump (autonomous policy rollouts, DROID) roboarena-datadump-2026-02-03 | Eval logs (YAML + video) | 590 | 100% | Sep 26, 2026, 1:07 AM |
Recent episodes
Most recently recorded across all datasets
| episode | task | policy | outcome | failure | recorded |
|---|---|---|---|---|---|
| shrimp_ep_00000 | Sauté and serve shrimp | human-teleop | Unscored | - | Mar 10, 2026, 6:44 PM |
| shrimp_ep_00001 | Sauté and serve shrimp | human-teleop | Unscored | - | Mar 10, 2026, 6:44 PM |
| shrimp_ep_00002 | Sauté and serve shrimp | human-teleop | Unscored | - | Mar 10, 2026, 6:44 PM |
| shrimp_ep_00003 | Sauté and serve shrimp | human-teleop | Unscored | - | Mar 10, 2026, 6:44 PM |
| shrimp_ep_00004 | Sauté and serve shrimp | human-teleop | Unscored | - | Mar 10, 2026, 6:44 PM |
| shrimp_ep_00005 | Sauté and serve shrimp | human-teleop | Unscored | - | Mar 10, 2026, 6:44 PM |