Industrial robots see with neuromorphic eyes
We propose to solve the complex and currently widely researched problem of robotic insertion of objects (e.g., plugs) by using low-latency event-based image processing and neuromorphic hardware to precisely position the object in a fast control loop, establishing a new state of the art in image processing-driven robot control. Spiking neural network-based robot control meets the requirements of low latency and precision of visual analysis of the insertion process, adaptability to different object shapes and insertion dynamics, and energy efficiency of AI processing. This also enables mobile robotics applications.
Our approach is based in the use of optical flow and event-based 3D reconstruction in a spiking neural network on a neuromorphic research chip. This enables real-time control of precise centering of the object, relative to the slot in preparation for a force-based insertion process. The relevance for Bavarian companies in robotics and image processing is very high, as it can give them a significant edge in AI technologies.
The innovation of our project lies, first, in the use of an event-based camera for precise vision-guided arm control. Such cameras have been extensively used in research on mobile robots, such as drones (see work by Prof. D. Scaramuzza from Univ. Zurich), and enable fast maneuvers and 3D reconstructions. The extraction of geometric and kinematic measurements of an object from event streams has not yet been demonstrated.
Second, we will develop the event-based image processing algorithms and adaptive controller as a spiking neural network on a neuromorphic chip, further reducing latency and power consumption and enabling adaptivity of the resulting solution. This approach will contribute to the growing field of neuromorphic computing in robotics.
Finally, the integration of event-based image processing and spike-based control will open up the possibility of using this fast vision sensor for other robotic tasks that require precise positioning of the robot relative to the object, such as grasping, manipulating, or placing. This will take event-based vision beyond the realm of mobile robotics, into industrial robotics and arm control.
Bavarian Ministry of Economic Affairs, Regional Development and Energy (StMWi)
Funding code DIK-2105-0036// DIK0368/01
01.10.2021 - 31.09.2024