A robot benchmark says it will publish the failures. The study already finished is not the fleet being advertised

The launch, as reported on 4 November, is an offer: a team can send a policy and have it run on hardware it does not own. BitRobot’s Michael Cho is quoted to that effect. The names on the announcement are Hugging Face, BitRobot, the Singapore Embodied AI Centre, and Jiafei Duan. The advertised fleet is more than 100 robots, including Unitree G1 humanoids, SO-101 arms, YAM systems and Franka arms, across more than 140 scenes described as homes, pavements and public spaces rather than a single lab. The hours on offer are put at 10,000. Recordings, including failed attempts, are to be published, with Hugging Face named as the place they go. Unitree has publicly backed the use of the G1. The statement does not say Unitree paid, or handed over machines.

That offer is not the study. A preprint submitted on 2 November, under the same name, reports an initial evaluation of five vision-language-action policies and three frontier agents, on four bodies: the SO-101, a Franka, a two-armed YAM, and the G1. The open track is blinded and pairwise, with no task-specific retraining. Across 5,699 of those comparisons at 124 sites, the paper says the frontier agents rival the specialist policies, and that all of the systems struggle to finish open-world tasks reliably. After fine-tuning, each policy given the same 100 demonstrations a task, GR00T N1.7 leads on the SO-101 and pi0.5 leads on the Franka and the two-armed YAM. Rankings agree weakly across the different machines. Kendall’s W is 0.24. The open-track order does not predict the order after fine-tuning. The completed study is not the same thing as access to the 140 scenes in the launch. Duan is reported as saying open-source models were not yet fit to run the humanoid open track with no extra training, and that most still struggled after the 100 demonstrations.