Joint Action for Multimodal Embodied Social Systems

Project Results

In the JAMES project, we conducted experiments on short-term human-robot interaction. On the scientific side, we evaluated our robot bartender system in multiple user studies (see our ICMI 2012  and ICMI 2013 papers), studied human-human interaction (see our Frontiers paper), and contributed to intelligent HRI systems (see ICAPS 2013).

Our project also attracted great attention by public media, including a Scientific American short article, radio interviews on BBC and WGN Chicago, and TV documentaries on German science shows 3sat nano and Pro7 Galileo.

Project Description 

The goal of the JAMES project is to develop a robot that is capable of interacting with humans in public spaces. Here it is important that the robot correctly fulfils its assigned task while keeping the social needs of the humans in mind. For this, the robot needs to be able to quickly analyse various situations, specifically situations in which it interacts with several humans at once. To test this goal, the JAMES project builds a robot that works as a bartender, which can take orders from humans and handout drinks to its customers. The robotics group at fortiss is responsible for setting up and maintaining the JAMES robot system. Furthermore, fortiss does research in generation of robot arm and head movements that appear natural to humans and in keeping the intearction safe for humans.


The research buy the JAMES project partners is guided by four central principles:

  1. Social interaction should be seen as an instance of joint action.
  2. Successful task-based interaction relies on successful social interaction.
  3. Social interaction is often multi-party, dynamic, and short-horizon.
  4. Social skills should be learnt rather than preprogrammed. 

To reach the project's main goal, the creation of a robot that interacts with humans in a socially appropriate way, the JAMES project partners research new methods and algorimthms in the areas of

  • visual tracking of multiple humans with frequent occlusions
  • robust natural language processing and multimodal fusion
  • knowledgebased planning with uncertainty
  • machine learning-based social state processing and social skill execution
  • multimodal output generation and generation of natural and safe robot motion
  • data collection of human-human interactions in public spaces


University of Edinburgh
Insitute for Language, Cognition and Computation
Dr. Ronald Petrick, Prof. Alex Lascarides, Amy Isard

Universität Bielefeld
Lehrstuhl für Psycholinguistik
Prof. Jan de Ruiter, Dr. Sebastian Loth, Kerstin Huth

Foundation for Research and Technology - Hellas
Computational Vision and Robotics Laboratory,
Prof. Panos Trahanias, Dr. Maria Pateraki, Markos Sigalas

Heriot-Watt University Edinburgh
Interaction Lab
Prof. Oliver Lemon, Dr. Mary Ellen Foster, Dr. Simon Keizer, Dr. Zhouran Wang


The JAMES project (Grant No. 270435) is funded by the European Commission through the 7th Framework Programme.

Publications (Selection)

S Keizer, ME Foster, A Gaschler, M Giuliani, A Isard, O Lemon, Handling uncertain input in multi-user human-robot interaction, Proceedings of IEEE RO-MAN 2014. [pdf

M Giuliani, RPA Petrick, ME Foster, A Gaschler, A Isard, M Pateraki, M Sigalas, Comparing task-based and socially intelligent behaviour in a robot bartender, Proceedings of the 15th International Conference on Multimodal Interaction. [link]

A Gaschler, R Petrick, M Giuliani, M Rickert, A Knoll, KVP: A knowledge of volumes approach to robot task planning, Intelligent Robots and Systems (IROS), 2013 IEEE/RSJ International Conference on. [link]

ME Foster, A Gaschler, M Giuliani, How can I help you? comparing engagement classification strategies for a robot bartender, Proceedings of the 15th International Conference on Multimodal Interaction [link]

S Keizer, ME Foster, O Lemon, A Gaschler, M Giuliani, Training and evaluation of an MDP model for social multi-user human-robot interaction, Proceedings of the SIGDIAL 2013 Conference, 223-232. [pdf]

ME Foster, A Gaschler, M Giuliani, A Isard, M Pateraki, RPA Petrick, Two people walk into a bar: Dynamic multi-party social interaction with a robot agent, 14th ACM international conference on Multimodal interaction (ICMI 2012) [link]

S Loth, K Huth, JP De Ruiter, Automatic detection of service initiation signals used in bars, Frontiers in psychology 4. [link]

A Gaschler, K Huth, M Giuliani, I Kessler, J de Ruiter, A Knoll, Modelling state of interaction from head poses for social human-robot interaction, Proc. of the Gaze in Human-Robot Interaction Workshop, HRI. [pdf]