Exploring the Use of Gesture in Collaborative Tasks
Isaac Wang, Pradyumna Narayana, Dhruva Patil, Gururaj Mulay, Rahul Bangar, Bruce Draper, Ross Beveridge, and Jaime Ruiz. 2017. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '17). ACM, New York, NY, USA, 2990-2997. DOI: https://doi.org/10.1145/3027063.3053239
Communication Through Gestures, Expression and Shared Perception
Personal assistants such as Siri have changed the way people interact with computers by introducing virtual assistants that collaborate with humans through natural speech-based interfaces. However, relying on speech alone as the medium of communication can be a limitation; non-verbal aspects of communication also play a vital role in natural human discourse. Thus, it is necessary to identify the use of gesture and other non-verbal aspects in order to apply them towards the development of computer systems. The goal of this project is to understand how systems can model non-verbal communication as a means to mimic human-human communication.
EGGNOG: A Continuous, Multi-modal Data Set of Naturally Occurring Gestures with Ground Truth Labels
Isaac Wang, Mohtadi Ben Fraj, Pradyumna Narayana, Dhruva Patil, Gururaj Mulay, Rahul Bangar, J. Ross Beveridge, Bruce A. Draper, and Jaime Ruiz. 2017. In 2017 12th IEEE International Conference on Automatic Face Gesture Recognition (FG 2017), 414–421. DOI: https://doi.org/10.1109/FG.2017.145
Communicating and Acting: Understanding Gesture in Simulation Semantics
Nikhil Krishnaswamy, Pradyumna Narayana, Isaac Wang, Kyeongmin Rim, Rahul Bangar, Dhruva Patil, Gururaj Mulay, Ross Beveridge, Jaime Ruiz, Bruce Draper, and James Pustejovsky. 2017. In 12th International Conference on Computational Semantics (IWCS 2017)--Short papers.
Cooperating with Avatars Through Gesture, Language and Action
Pradyumna Narayana, Nikhil Krishnaswamy, Isaac Wang, Rahul Bangar, Dhruva Patil, Gururaj Mulay, Kyeongmin Rim, Ross Beveridge, Jaime Ruiz, James Pustejovsky, and Bruce Draper. 2018. In Intelligent Systems and Applications (Advances in Intelligent Systems and Computing), 272–293.
Exploring Virtual Agents for Augmented Reality
Isaac Wang, Jesse Smith, and Jaime Ruiz. 2019. In 2019 CHI Conference on Human Factors in Computing Systems Proceedings (CHI 2019), May 4–9, 2019, Glasgow, Scotland, UK. ACM, New York, NY, USA. Paper 281, 10 pages. https://doi.org/10.1145/3290605.3300511
User-Aware Shared Perception for Embodied Agents
David White, Ross Beveridge, Bruce Draper, Francisco Ortega, Rahul Bangar, Dhruva Patil, James Pustejovsky, Nikhil Krishnaswamy, Kyeongmin Rim, Jaime Ruiz, and Isaac Wang. 2019. User-Aware Shared Perception for Embodied Agents. In 2019 IEEE International Conference on Humanized Computing and Communication (HCC), Laguna Hills, CA, USA, 2019, pp. 46-51.
Diana's World: A Situated Multimodal Interactive Agent
Nikhil Krishnaswamy, Pradyumna Narayana, Rahul Bangar, Kyeongmin Rim, Dhruva Patil, David McNeely-White, Jaime Ruiz, Bruce Draper, Ross Beveridge, and James Pustejovsky. 2020. Proceedings of the AAAI Conference on Artificial Intelligence 34, 09: 13618–13619. https://doi.org/10.1609/aaai.v34i09.7096
Examining the Use of Nonverbal Communication in Virtual Agents
Isaac Wang and Jaime Ruiz. 2021. International Journal of Human–Computer Interaction (Mar. 2021), 1–26. DOI: https://doi.org/10.1080/10447318.2021.1898851
It’s a Joint Effort: Understanding Speech and Gesture in Collaborative Tasks
Isaac Wang, Pradyumna Narayana, Dhruva Patil, Rahul Bangar, Bruce Draper, Ross Beveridge, and Jaime Ruiz. 2021. In Human-Computer Interaction. Interaction Techniques and Novel Applications (Lecture Notes in Computer Science), 159–178. https://doi.org/10.1007/978-3-030-78465-2_13