JAMES

Joint Action for Multimodal Embodied Social Systems

Project Results

In the JAMES project, we conducted experiments on short-term human-robot interaction. On the scientific side, we evaluated our robot bartender system in multiple user studies (see our ICMI 2012  and ICMI 2013 papers), studied human-human interaction (see our Frontiers paper), and contributed to intelligent HRI systems (see ICAPS 2013).

Our project also attracted great attention by public media, including a Scientific American short article, radio interviews on BBC and WGN Chicago, and TV documentaries on German science shows 3sat nano and Pro7 Galileo.

Project Description 

The goal of the JAMES project is to develop a robot that is capable of interacting with humans in public spaces. Here it is important that the robot correctly fulfils its assigned task while keeping the social needs of the humans in mind. For this, the robot needs to be able to quickly analyse various situations, specifically situations in which it interacts with several humans at once. To test this goal, the JAMES project builds a robot that works as a bartender, which can take orders from humans and handout drinks to its customers. The robotics group at fortiss is responsible for setting up and maintaining the JAMES robot system. Furthermore, fortiss does research in generation of robot arm and head movements that appear natural to humans and in keeping the intearction safe for humans.

Contributions

The research buy the JAMES project partners is guided by four central principles:

  1. Social interaction should be seen as an instance of joint action.
  2. Successful task-based interaction relies on successful social interaction.
  3. Social interaction is often multi-party, dynamic, and short-horizon.
  4. Social skills should be learnt rather than preprogrammed. 

To reach the project's main goal, the creation of a robot that interacts with humans in a socially appropriate way, the JAMES project partners research new methods and algorimthms in the areas of

  • visual tracking of multiple humans with frequent occlusions
  • robust natural language processing and multimodal fusion
  • knowledgebased planning with uncertainty
  • machine learning-based social state processing and social skill execution
  • multimodal output generation and generation of natural and safe robot motion
  • data collection of human-human interactions in public spaces

Partners

University of Edinburgh
Insitute for Language, Cognition and Computation
Dr. Ronald Petrick, Prof. Alex Lascarides, Amy Isard
http://www.ilcc.inf.ed.ac.uk/ 

Universität Bielefeld
Lehrstuhl für Psycholinguistik
Prof. Jan de Ruiter, Dr. Sebastian Loth, Kerstin Huth
http://www.uni-bielefeld.de/lili/personen/jruiter/

Foundation for Research and Technology - Hellas
Computational Vision and Robotics Laboratory,
Prof. Panos Trahanias, Dr. Maria Pateraki, Markos Sigalas
http://www.ics.forth.gr/cvrl/ 

Heriot-Watt University Edinburgh
Interaction Lab
Prof. Oliver Lemon, Dr. Mary Ellen Foster, Dr. Simon Keizer, Dr. Zhouran Wang
http://www.macs.hw.ac.uk/InteractionLab

Funding

The JAMES project (Grant No. 270435) is funded by the European Commission through the 7th Framework Programme.

PUBLICATIONS

Export list as BibTeX file

  • Andre Gaschler, Ingmar Kessler, Ronald P. A. Petrick and Alois Knoll. Extending the Knowledge of Volumes Approach to Robot Task Planning with Efficient Geometric Predicates. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2015. Details DOI BibTeX
  • Ronald P. A. Petrick and Andre Gaschler. Knowledge-Level Planning for Robot Task Planning and Human-Robot Interaction. In RSS Workshop on Combining AI Reasoning and Cognitive Science with Robotics, 2015. Details BibTeX
  • Andre Gaschler, Quirin Fischer and Alois Knoll. The Bounding Mesh Algorithm. TUM-I1522, Technische Universität München, Munich, Germany, 2015. Details BibTeX
  • Sören Jentzsch, Andre Gaschler, Oussama Khatib and Alois Knoll. MOPL: A Multi-Modal Path Planner for Generic Manipulation Tasks. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Hamburg, Germany, 2015. Details DOI BibTeX
  • Alina Roitberg, Alexander Perzylo, Nikhil Somani, Manuel Giuliani, Markus Rickert and Alois Knoll. Human Activity Recognition in the Context of Industrial Human-Robot Interaction. In Proceedings of the AsiaPacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 1–10, Siem Reap, Cambodia, 2014. Details DOI BibTeX
  • Andre Gaschler, Maximilian Springer, Markus Rickert and Alois Knoll. Intuitive Robot Tasks with Augmented Reality and Virtual Obstacles. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 6026–6031, Hong Kong, China, 2014. Details DOI BibTeX
  • Manuel Giuliani, Thomas Marschall and Amy Isard. Using Ellipsis Detection and Word Similarity for Transformation of Spoken Language into Grammatically Valid Sentences. In Proceedings of the 15th Annual SIGdial Meeting on Discourse and Dialogue (SIGDIAL 2014), Philadelphia, USA, 2014. Details BibTeX
  • Sebastian Loth, Manuel Giuliani and Jan P. de Ruiter. Ghost-in-the-Machine: Initial Results. In Proceedings of the 9th ACM/IEEE International Conference on Human-Robot Interaction (HRI 2014), Bielefeld, Germany, 2014. Details BibTeX
  • Ronald P. A. Petrick and Andre Gaschler. Extending Knowledge-Level Contingent Planning to Robot Task Planning. In International Conference on Automated Planning and Scheduling Workshop on Planning and Robotics (PlanRob 2014), 2014. Details BibTeX
  • Simon Keizer, Mary Ellen Foster, Andre Gaschler, Manuel Giuliani, Amy Isard and Oliver Lemon. Handling uncertain input in multi-user human-robot interaction. In Proceedings of the 23rd IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 2014. Details BibTeX
  • Andre Gaschler, Svetlana Nogina, Ronald P. A. Petrick and Alois Knoll. Planning Perception and Action for Cognitive Mobile Manipulators. In Proceedings of SPIE Volume 9025 – Intelligent Robots and Computer Vision XXXI: Algorithms and Techniques, 2014. Details BibTeX
  • Andre Gaschler, Ronald P. A. Petrick, Manuel Giuliani, Markus Rickert and Alois Knoll. KVP: A Knowledge of Volumes Approach to Robot Task Planning. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 202–208, Tokyo, Japan, 2013. Details DOI BibTeX
  • Mary Ellen Foster, Manuel Giuliani and Amy Isard. Task-Based Evaluation of Context-Sensitive Referring Expressions in Human-Robot Dialogue. Language and Cognitive Processes, ():, 2013. Details DOI BibTeX
  • Manuel Giuliani, Ronald P. A. Petrick, Mary Ellen Foster, Andre Gaschler, Amy Isard, Maria Pateraki and Markos Sigalas. Comparing Task-Based and Socially Intelligent Behaviour in a Robot Bartender. In Proceedings of the 15th ACM International Conference on Multimodal Interaction (ICMI 2013), Sydney, Australia, 2013. Details DOI BibTeX
  • Mary Ellen Foster, Andre Gaschler and Manuel Giuliani. How Can I Help You? Comparing Engagement Classification Strategies for a Robot Bartender. In Proceedings of the 15th ACM International Conference on Multimodal Interaction (ICMI 2013), Sydney, Australia, 2013. Details DOI BibTeX
  • Andre Gaschler, Ronald P. A. Petrick, Torsten Kröger, Oussama Khatib and Alois Knoll. Robot Task and Motion Planning with Sets of Convex Polyhedra. In Robotics: Science and Systems (RSS) Workshop on Combined Robot Motion Planning and AI Planning for Practical Applications, 2013. Details BibTeX
  • Andre Gaschler, Ronald P. A. Petrick, Torsten Kröger, Alois Knoll and Oussama Khatib. Robot Task Planning with Contingencies for Run-time Sensing. In Proceedings of the Workshop on Combining Task and Motion Planning, IEEE International Conference on Robotics and Automation (ICRA), 2013. Details BibTeX
  • Simon Keizer, Mary Ellen Foster, Oliver Lemon, Andre Gaschler and Manuel Giuliani. Training and evaluation of an MDP model for social multi-user human-robot interaction. In Proceedings of the 14th Annual SIGdial Meeting on Discourse and Dialogue, 2013. Details BibTeX
  • Manuel Giuliani and Alois Knoll. Using Embodied Multimodal Fusion to Perform Supportive and Instructive Robot Roles in Human-Robot Interaction. International Journal of Social Robotics, 5(3):345–356, 2013. Details DOI BibTeX
  • Andre Gaschler, Kerstin Huth, Manuel Giuliani, Ingmar Kessler, Jan de Ruiter and Alois Knoll. Modelling State of Interaction from Head Poses for Social Human-Robot Interaction. In Proceedings of the Gaze in Human-Robot Interaction Workshop held at the 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI 2012), Boston, MA, 2012. Details BibTeX
  • Andre Gaschler, Sören Jentzsch, Manuel Giuliani, Kerstin Huth, Jan de Ruiter and Alois Knoll. Social Behavior Recognition Using Body Posture and Head Pose for Human-Robot Interaction. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2012. Details DOI BibTeX
  • Mary Ellen Foster, Andre Gaschler, Manuel Giuliani, Amy Isard, Maria Pateraki and Ronald P. A. Petrick. Two People Walk Into a Bar: Dynamic Multi-Party Social Interaction with a Robot Agent. In Proceedings of the 14th ACM International Conference on Multimodal Interaction (ICMI 2012), 2012. Details BibTeX