Case study SpikingBody
Traditional motion analysis systems often suffer from high latency, inefficient energy consumption and limited adaptability. In the SpikingBody project, fortiss developed a neuromorphic AI system that combines event-based cameras with spiking neural networks (SNNs) to recognise movements in real time and with low energy consumption. The result is a groundbreaking prototype for motion recognition – initially tested in tennis – which lays the foundation for smarter human-machine interactions.
The SpikingBody project was funded by the Bavarian Research Foundation and led by fortiss. The aim was to further develop neuromorphic AI technologies for energy-efficient, real-time motion detection. The focus was on applications in robotics, industrial automation and interactive systems.
Challenge
Accurate real-time detection of human movements is essential for safe and responsive human-machine interaction. Conventional systems reach their limits in this regard due to high latency, high energy consumption and limited adaptability. In particular, traditional image-based methods require large amounts of data and significant computing power.
Solution
fortiss has developed a system modelled on the high energy efficiency and adaptability of biological neural processes. To achieve this, the team combined event-based image sensors with spiking neural networks (SNNs) running on Intel’s Loihi 2 neuromorphic chip. This interaction enables very fast yet extremely low-power data processing, as only relevant changes in the image signal are processed. The system follows a two-stage learning approach involving offline training and subsequent online adaptation, enabling new movement patterns to be detected even during operation. As a demonstrator, the system was used to recognise tennis strokes, reliably classifying and analysing movements such as serves, forehands and backhands in real time.
Result
- Development of a neuromorphic prototype for real-time gesture recognition using event-driven cameras and SNNs
- Recognition of tennis strokes such as the serve, forehand and backhand using just one sensor and with minimal latency
- High energy efficiency thanks to event-driven data processing and neuromorphic architecture (Loihi 2)
- Introduction of a two-stage learning process: offline pre-training + online adaptation without complete retraining
- Parallel estimation of body posture to enhance action classification
- Extension of the use case to include, amongst other things, artistic applications (e.g. interactive music installations) and robotic interfaces
- Creating a scalable foundation for adaptive human-machine interfaces in industry and smart environments
Outcome
The project demonstrates that neuromorphic AI, through the combination of event-based sensing and spiking neural networks, enables particularly fast and energy-efficient real-time motion detection. This creates a scalable foundation for adaptive human-machine interactions that can be flexibly adapted to new motion patterns and fields of application in industry, robotics and interactive systems.
Project partner
