New benchmark tests humanoid robots under real-world forces

As humanoid robots become increasingly capable of performing real-world tasks such as carrying heavy objects or wiping tables, a need has arisen for better methods to assess their performance under physical loads. Traditional tests often evaluate robot movement in empty spaces, failing to account for the external forces encountered during actual work, which can lead to a loss of balance and task failure.

To address this gap, a research team created ThorArena, a novel benchmark designed to evaluate humanoid robots' ability to manage realistic physical interactions. The system was developed by gathering data from human operators wearing VR headsets and motion trackers while performing everyday chores. These activities included lifting water containers, pushing and pulling chairs, and collaborative object carrying.

During these human-performed tasks, specialized 3D-printed tools with sensors measured the directional forces exerted by the operators' hands. This motion and force data was then transferred to a virtual humanoid robot within a physics simulator. The simulator replayed the recorded hand forces against the virtual robot as it attempted to replicate the human movements. The robot's performance was scored using a newly developed metric called the Force-Aware Tracking Score (FATS), which assesses tracking accuracy and balance.

Experiments using ThorArena on four different humanoid control systems demonstrated significant performance disparities that were not apparent in traditional no-force evaluations. When physical forces were removed from the simulation, all tested systems performed well. However, upon reintroducing the recorded forces, scores dropped, and performance gaps emerged, highlighting the importance of force-aware testing.