Gaze-Based Intention Recognition for Human-Agent Collaboration: Towards Nonverbal Communication in Human-AI Interaction

Abstract

Human-agent collaboration has repeatedly been proposed over the decades as a way forward to leverage the strengths of artificial intelligence. As it has become common for humans to work and play alongside intelligent agents, it is increasingly imperative to improve the capacity of agents to interact with their human counterparts socially, naturally and effectively. However, current agents are still limited in their capacity to recognise nonverbal signals and cues, which in turn, limits their capabilities for natural interaction. This thesis addresses this limitation by investigating how artificial agents might support humans in real-time collaboration, given the increased capacity to recognise human intentions afforded by processing their gaze data in real-time. We hypothesise that a socially interactive agent with an increased capacity to recognise intentions can drastically improve its interactive capability, such as by adapting its recommendations to their anticipated intentions as well as to the intentions of others.

Using a scenario-based based design approach, we designed five studies to inform and evaluate the different capabilities of a collaborative gaze-enabled intention-aware artificial agent.

In Studies 1 and 2, we first evaluate the capacity of human subjects to perform intention recognition using gaze visualisation and its corresponding effects in a competitive gameplay setting. The findings showed that humans players could improve their capacity to infer their intentions of their opponent when shown a live visualisation of their opponent's gaze throughout the game. However, this capacity can be hampered when the opponent was aware that their gaze was being watched. The findings further indicate that humans have a limited capacity in performing gaze-based intention recognition, suggesting that the task may be more suitable for an artificial agent that is trained to process the rich multimodal information available in our setting.

In Study 3, we present the implementation details and evaluation of a gaze-enabled intention-aware artificial agent, developed as part of this thesis, that incorporates gaze into its intention recognition process. The evaluation, which uses the data from the previous two studies, demonstrates that incorporating gaze into the agent's planning process not only increases the agent's capacity to recognise intentions but also that it performs better overall than human subjects. In Studies 4 and 5, we operationalise the artificial agent by first giving the agent both the ability to communicate intentions of their opponent to its human collaborator and to explain its reasoning process if required. Subsequently, we evaluated the experience of the players playing the game with and without the assistance of the agent, which ultimately provided insights into how we can further improve the interaction between the human and an intention-aware artificial agent.

The findings in this thesis resulted in three contributions towards the understanding of how artificial agents can support human-agent collaboration, given the ability increased capacity to recognise intentions with eye-tracking. The findings from Studies 1, 2 and 3 extend the relationship between gaze awareness and intention, by demonstrating that gaze when tracked over time, can lead to the detection of distal intentions (i.e. long-term intentions that often require several steps to be fulfilled). Following, Studies 3, 4 and 5 contribute to the design of a collaborative gaze-enabled intention-aware artificial agent, and the demonstration of increased situation awareness through gaze awareness for human-agent collaboration. Overall, the thesis highlights the importance of incorporating nonverbal communication in human-AI interaction.