Joint Action for Multimodal Embodied Social Systems

Project Description 

The goal of the JAMES project is to develop a robot that is capable of interacting with humans in public spaces. Here it is important that the robot correctly fulfils its assigned task while keeping the social needs of the humans in mind. For this, the robot needs to be able to quickly analyse various situations, specifically situations in which it interacts with several humans at once. To test this goal, the JAMES project builds a robot that works as a bartender, which can take orders from humans and handout drinks to its customers. The robotics group at fortiss is responsible for setting up and maintaining the JAMES robot system. Furthermore, fortiss does research in generation of robot arm and head movements that appear natural to humans and in keeping the intearction safe for humans.


The research buy the JAMES project partners is guided by four central principles:

  1. Social interaction should be seen as an instance of joint action.
  2. Successful task-based interaction relies on successful social interaction.
  3. Social interaction is often multi-party, dynamic, and short-horizon.
  4. Social skills should be learnt rather than preprogrammed. 

To reach the project's main goal, the creation of a robot that interacts with humans in a socially appropriate way, the JAMES project partners research new methods and algorimthms in the areas of

  • visual tracking of multiple humans with frequent occlusions
  • robust natural language processing and multimodal fusion
  • knowledgebased planning with uncertainty
  • machine learning-based social state processing and social skill execution
  • multimodal output generation and generation of natural and safe robot motion
  • data collection of human-human interactions in public spaces


University of Edinburgh
Insitute for Language, Cognition and Computation
Dr. Ronald Petrick, Prof. Alex Lascarides, Amy Isard

Universität Bielefeld
Lehrstuhl für Psycholinguistik
Prof. Jan de Ruiter, Dr. Sebastian Loth, Kerstin Huth

Foundation for Research and Technology - Hellas
Computational Vision and Robotics Laboratory,
Prof. Panos Trahanias, Dr. Maria Pateraki, Markos Sigalas

Heriot-Watt University Edinburgh
Interaction Lab
Prof. Oliver Lemon, Dr. Mary Ellen Foster, Dr. Simon Keizer, Dr. Zhouran Wang


The JAMES project (Grant No. 270435) is funded by the European Commission through the 7th Framework Programme.