SO Arm 101 - Isaac Sim Training

This project explores the cutting edge of sim-to-real robotics transfer using the SO Arm 101 platform and NVIDIA's Isaac Sim. By combining powerful simulation tools with immersive teleoperation, we've developed a highly efficient pipeline for training robotic manipulation policies.

The work focuses on pick-and-place tasks, utilizing Pi 0 VLA for policy training and Meta Quest 3 for intuitive data collection.

The software stack leverages NVIDIA Isaac Sim's USD-based environment. We developed a custom bridge to interface the Meta Quest 3 with the simulator, allowing for real-time teleoperation and data logging.

Training uses the Pi 0 VLA architecture to learn pick-and-place behaviors from the collected demonstrations. The system includes domain randomization to improve the robustness of the learned policies for eventual hardware transfer.

Successfully collected hundreds of demonstrations via Quest 3 teleoperation, leading to a highly reliable pick-and-place policy. The SO Arm 101 demonstrated 90%+ success rates in simulation for table-to-box transfers. The bridge proved to be a powerful tool for rapid data collection and human-in-the-loop training.