Two Hands, One or Two Robots

The research question is how to preserve the goal of a human demonstration when the robot has a different number of manipulators. Two simultaneous human moves can become a sequential plan for one arm or a parallel plan for two arms. A scheduling heuristic considers dependencies, path clearance, and a penalty for recruiting the second arm.

This is an ongoing independent experiment. The current video is a development baseline using generated input; real-data evaluation is pending.

OpenCV detects the colored objects and targets in the baseline video. Four tabletop markers and supplied workspace dimensions map image coordinates into a planar task representation. A scheduler selects an arm configuration, and an inverse-kinematics controller executes reach, grasp, lift, transport, and release phases.

MuJoCo simulates free objects moved by finger contact, without grasp welds or object teleportation. The motion controller uses simulator object state. Coordination messages are templates, and the demonstrated system is not a trained VLA or learned world model.

An experimental pretrained vision-language perception extension is under development to interpret other objects. It has not yet established reliable end-to-end generalization to real cups, balls, or charger bricks.

In the initial Gemini-video scene, both objects were placed successfully with either arm configuration: 19.6 seconds of simulated execution for one arm and 10.2 seconds for two. Six additional rollouts with small position perturbations passed the baseline checks. These are development results on a narrow setup, not a generalization benchmark.

Early failures included confusing target rings with objects, hand occlusion, and contact between the two robot hands. The baseline was refined using those observations. Next, I will collect phone videos of real objects, measure their geometry, and evaluate success, contacts, and timing on new layouts. Generated and personally collected data will be reported separately.