Back to Projects

Communication Through Gestures, Expression and Shared Perception

Personal assistants such as Siri have changed the way people interact with computers by introducing virtual assistants that collaborate with humans through natural speech-based interfaces. However, relying on speech alone as the medium of communication can be a limitation; non-verbal aspects of communication also play a vital role in natural human discourse. Thus, it is necessary to identify the use of gesture and other non-verbal aspects in order to apply them towards the development of computer systems. The goal of this project is to understand how systems can model non-verbal communication as a means to mimic human-human communication.

Headshot of Jaime Ruiz wearing a Hololens

Jaime Ruiz

Associate Professor
Headshot of Isaac Wang

Isaac Wang

Alumni Ph.D. Student

Communication through Gestures, Expression and Shared Perception

Diana’s World: A Situated Multimodal Interactive Agent

Nikhil Krishnaswamy, Pradyumna Narayana, Rahul Bangar, Kyeongmin Rim, Dhruva Patil, David McNeely-White, Jaime Ruiz, Bruce Draper, Ross Beveridge, and James Pustejovsky. 2020. Proceedings of the AAAI Conference on Artificial Intelligence 34, 09: 13618–13619.

User-Aware Shared Perception for Embodied Agents

David White, Ross Beveridge, Bruce Draper, Francisco Ortega, Rahul Bangar, Dhruva Patil, James Pustejovsky, Nikhil Krishnaswamy, Kyeongmin Rim, Jaime Ruiz, and Isaac Wang. 2019. In 2019 IEEE International Conference on Humanized Computing and Communication (HCC), Laguna Hills, CA, USA, 2019, pp. 46-51.

Exploring Virtual Agents for Augmented Reality

Isaac Wang, Jesse Smith, and Jaime Ruiz. 2019. In 2019 CHI Conference on Human Factors in Computing Systems Proceedings (CHI 2019), May 4–9, 2019, Glasgow, Scotland, UK. ACM, New York, NY, USA. Paper 281, 10 pages.

Cooperating with Avatars Through Gesture, Language and Action

Pradyumna Narayana, Nikhil Krishnaswamy, Isaac Wang, Rahul Bangar, Dhruva Patil, Gururaj Mulay, Kyeongmin Rim, Ross Beveridge, Jaime Ruiz, James Pustejovsky, and Bruce Draper. 2018.  In Intelligent Systems and Applications (Advances in Intelligent Systems and Computing), 272–293.

EASEL: Easy Automatic Segmentation Event Labeler

Isaac Wang, Pradyumna Narayana, Jesse Smith, Bruce Draper, Ross Beveridge, and Jaime Ruiz. 2018. In Proceedings of the 23rd International Conference on Intelligent User Interfaces (IUI ’18). ACM, New York, NY, USA. DOI:

Exploring the Use of Gesture in Collaborative Tasks

Isaac Wang, Pradyumna Narayana, Dhruva Patil, Gururaj Mulay, Rahul Bangar, Bruce Draper, Ross Beveridge, and Jaime Ruiz. 2017. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA ’17). Association for Computing Machinery, New York, NY, USA, 2990–2997.