Joint Action for Multimodal Embodied Social Systems
In the JAMES project, we conducted experiments on short-term human-robot interaction. On the scientific side, we evaluated our robot bartender system in multiple user studies (see our ICMI 2012 and ICMI 2013 papers), studied human-human interaction (see our Frontiers paper), and contributed to intelligent HRI systems (see ICAPS 2013).
Our project also attracted great attention by public media, including a Scientific American short article, radio interviews on BBC and WGN Chicago, and TV documentaries on German science shows 3sat nano and Pro7 Galileo.
The goal of the JAMES project is to develop a robot that is capable of interacting with humans in public spaces. Here it is important that the robot correctly fulfils its assigned task while keeping the social needs of the humans in mind. For this, the robot needs to be able to quickly analyse various situations, specifically situations in which it interacts with several humans at once. To test this goal, the JAMES project builds a robot that works as a bartender, which can take orders from humans and handout drinks to its customers. The robotics group at fortiss is responsible for setting up and maintaining the JAMES robot system. Furthermore, fortiss does research in generation of robot arm and head movements that appear natural to humans and in keeping the intearction safe for humans.
The research buy the JAMES project partners is guided by four central principles:
- Social interaction should be seen as an instance of joint action.
- Successful task-based interaction relies on successful social interaction.
- Social interaction is often multi-party, dynamic, and short-horizon.
- Social skills should be learnt rather than preprogrammed.
To reach the project's main goal, the creation of a robot that interacts with humans in a socially appropriate way, the JAMES project partners research new methods and algorimthms in the areas of
- visual tracking of multiple humans with frequent occlusions
- robust natural language processing and multimodal fusion
- knowledgebased planning with uncertainty
- machine learning-based social state processing and social skill execution
- multimodal output generation and generation of natural and safe robot motion
- data collection of human-human interactions in public spaces
University of Edinburgh
Insitute for Language, Cognition and Computation
Dr. Ronald Petrick, Prof. Alex Lascarides, Amy Isard
Lehrstuhl für Psycholinguistik
Prof. Jan de Ruiter, Dr. Sebastian Loth, Kerstin Huth
Foundation for Research and Technology - Hellas
Computational Vision and Robotics Laboratory,
Prof. Panos Trahanias, Dr. Maria Pateraki, Markos Sigalas
Heriot-Watt University Edinburgh
Prof. Oliver Lemon, Dr. Mary Ellen Foster, Dr. Simon Keizer, Dr. Zhouran Wang
The JAMES project (Grant No. 270435) is funded by the European Commission through the 7th Framework Programme.
- Extending the Knowledge of Volumes Approach to Robot Task Planning with Efficient Geometric Predicates. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2015. .
- Knowledge-Level Planning for Robot Task Planning and Human-Robot Interaction. In RSS Workshop on Combining AI Reasoning and Cognitive Science with Robotics, 2015. .
- The Bounding Mesh Algorithm. TUM-I1522, Technische Universität München, Munich, Germany, 2015. .
- MOPL: A Multi-Modal Path Planner for Generic Manipulation Tasks. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Hamburg, Germany, 2015. .
- Human Activity Recognition in the Context of Industrial Human-Robot Interaction. In Proceedings of the AsiaPacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 1–10, Siem Reap, Cambodia, 2014. .
- Intuitive Robot Tasks with Augmented Reality and Virtual Obstacles. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 6026–6031, Hong Kong, China, 2014. .
- Using Ellipsis Detection and Word Similarity for Transformation of Spoken Language into Grammatically Valid Sentences. In Proceedings of the 15th Annual SIGdial Meeting on Discourse and Dialogue (SIGDIAL 2014), Philadelphia, USA, 2014. .
- Ghost-in-the-Machine: Initial Results. In Proceedings of the 9th ACM/IEEE International Conference on Human-Robot Interaction (HRI 2014), Bielefeld, Germany, 2014. .
- Extending Knowledge-Level Contingent Planning to Robot Task Planning. In International Conference on Automated Planning and Scheduling Workshop on Planning and Robotics (PlanRob 2014), 2014. .
- Handling uncertain input in multi-user human-robot interaction. In Proceedings of the 23rd IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 2014. .
- Planning Perception and Action for Cognitive Mobile Manipulators. In Proceedings of SPIE Volume 9025 – Intelligent Robots and Computer Vision XXXI: Algorithms and Techniques, 2014. .
- KVP: A Knowledge of Volumes Approach to Robot Task Planning. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 202–208, Tokyo, Japan, 2013. .
- Task-Based Evaluation of Context-Sensitive Referring Expressions in Human-Robot Dialogue. Language and Cognitive Processes, ():, 2013. .
- Comparing Task-Based and Socially Intelligent Behaviour in a Robot Bartender. In Proceedings of the 15th ACM International Conference on Multimodal Interaction (ICMI 2013), Sydney, Australia, 2013. .
- How Can I Help You? Comparing Engagement Classification Strategies for a Robot Bartender. In Proceedings of the 15th ACM International Conference on Multimodal Interaction (ICMI 2013), Sydney, Australia, 2013. .
- Robot Task and Motion Planning with Sets of Convex Polyhedra. In Robotics: Science and Systems (RSS) Workshop on Combined Robot Motion Planning and AI Planning for Practical Applications, 2013. .
- Robot Task Planning with Contingencies for Run-time Sensing. In Proceedings of the Workshop on Combining Task and Motion Planning, IEEE International Conference on Robotics and Automation (ICRA), 2013. .
- Training and evaluation of an MDP model for social multi-user human-robot interaction. In Proceedings of the 14th Annual SIGdial Meeting on Discourse and Dialogue, 2013. .
- Using Embodied Multimodal Fusion to Perform Supportive and Instructive Robot Roles in Human-Robot Interaction. International Journal of Social Robotics, 5(3):345–356, 2013. .
- Modelling State of Interaction from Head Poses for Social Human-Robot Interaction. In Proceedings of the Gaze in Human-Robot Interaction Workshop held at the 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI 2012), Boston, MA, 2012. .
- Social Behavior Recognition Using Body Posture and Head Pose for Human-Robot Interaction. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2012. .
- Two People Walk Into a Bar: Dynamic Multi-Party Social Interaction with a Robot Agent. In Proceedings of the 14th ACM International Conference on Multimodal Interaction (ICMI 2012), 2012. .