CORINNE

Collaborative robots, or cobots, are a major topic in robotics, with impressive predicted growth rates. However, conventional cobots can only perform pre-programmed actions and cannot react flexibly to unexpected events. fortiss is therefore pursuing the goal of teaching cobots to recognize gestures or hand movements in real time. This ability should then be able to be used in the application field of welding work, whereby the learned knowledge should not be re-taught but transferred.

fortiss aims to realize a complex interactive welding process together with a human operator. To this end, algorithms are being developed for the MAiRA robot from NEURA Robotics. These algorithms use a new type of neuromorphic sensor to recognize human gestures. This enables the robot to intelligently recognize complex patterns and movements and to interact naturally with the operator and react to their gestures in order to optimize the welding process. Concerning the sensor, fortiss can already draw on project experience. This is based on an event-based camera in HD resolution that delivers visual events (pixel changes) to a pre-trained neural spiking network running on a neuromorphic chip (Loihi research chip from Intel).

The following questions are to be researched and addressed in the project:

• New gestures should be learned during runtime in an unsupervised fashion in a so-called continual learning paradigm. This implies to build a system able to constantly refine its knowledge of learned gestures and to recognize and learn unknown gestures autonomously.
• The use case envisaged in this project comprises several cobots that have learned a wide variety of gestures over time. The central research question here is how the knowledge of a newly learned gesture can be transferred from one cobot to the others without them also having to be trained for it.