Towards Generalization of Precise Robot Skills: Accurate Pick-and-Place of Novel Objects

Abstract: Reliable robots must understand their environment and act on it with precision. Practical robots should also be able to achieve wide generalization; i.e., a single robot should be capable of solving multiple tasks. For instance, we would like to have, but still lack, a robot that can reliably assemble most IKEA furniture instead of having one robot tailored to each piece of furniture. Towards this, in this talk, I will present an approach to robotic pick-and-place that provides robots with both high-precision and generalization skills. The proposed method only uses simulation to learn probabilistic models for grasping, planning, and localization that transfer to the actual robotic system with high accuracy. In real experiments, we show that our dual-arm robot can exert task-aware picks on new objects, use visuo-tactile sensing to localize them, and perform dexterous placings of these objects that involve in-hand regrasps and tight placing requirements with less than 1mm of tolerance. Overall, our pr