Computers are evolving from computational tools to collaborative agents through the emergence of natural, speech-driven interfaces. However, relying on speech alone is a limitation; gesture and other non-verbal aspects of communication also play a vital role in natural human discourse. To understand the use of gesture in human communication, we conducted a study to explore how people use gesture and speech to communicate when solving collaborative tasks. We asked 30 pairs of people to build structures out of blocks, limiting their communication to either Gesture Only, Speech Only, or Gesture and Speech. We found differences in how gesture and speech were used to communicate across the three conditions and found that pairs in the Gesture and Speech condition completed tasks faster than those in Speech Only. From our results, we draw conclusions about how our work impacts the design of collaborative systems and virtual agents that support gesture.

As well as: Pradyumna Narayana, Dhruva Patil, Rahul Bangar, Bruce Draper, Ross Beveridge

Isaac Wang, Pradyumna Narayana, Dhruva Patil, Rahul Bangar, Bruce Draper, Ross Beveridge, and Jaime Ruiz. 2021. It’s a Joint Effort: Understanding Speech and Gesture in Collaborative Tasks. In Human-Computer Interaction. Interaction Techniques and Novel Applications (Lecture Notes in Computer Science), 159–178. https://doi.org/10.1007/978-3-030-78465-2_13

@inproceedings{10.1007/978-3-030-78465-2_13,
	title        = {It's a Joint Effort: Understanding Speech and Gesture in Collaborative Tasks},
	author       = {Wang, Isaac and Narayana, Pradyumna and Patil, Dhruva and Bangar, Rahul and Draper, Bruce and Beveridge, Ross and Ruiz, Jaime},
	year         = 2021,
	booktitle    = {Human-Computer Interaction. Interaction Techniques and Novel Applications},
	publisher    = {Springer International Publishing},
	address      = {Cham},
	pages        = {159--178},
	isbn         = {978-3-030-78465-2},
	editor       = {Kurosu, Masaaki}
}