CAREER: Next Generation Multimodal InterfacesPI: Jaime Ruiz

Ever smaller computational devices coupled with advances in input affordances are revolutionizing the way users interact with technology. New, more natural user interfaces exploit touch, speech, gestures, handwriting, and vision in an effort to reduce the barriers imposed by interfaces, so that computing technology acts more like a dynamic partner and less like a tool. Nevertheless, human-computer interaction continues for the most part to mimic the traditional point-and-click paradigm associated with desktop computers. This research will take human-computer communication to the next level by leveraging human-human nonverbal communication such as gaze, body posture, and facial expressions. Project outcomes will have strong societal impact by moving us closer to truly smart environments and lowering the bar for those individuals who find using current technology difficult. The educational activities encompassed by this work will expose students to new ways of interacting with technology, in order to encourage them to pursue careers in computer science.

To further the understanding of nonverbal input and then use this understanding to discover new natural multimodal interactions, the research will include a number of thrusts. (i) Data collection and analysis: Collect human-human interactions in different scenarios and analyze nonverbal aspects of communication to determine common characteristics that can inform the design of more natural gestural interfaces. (ii) Recognizing and understanding intent: Create new recognition and input fusion methods that are not only able to correctly recognize the input (e.g., movement as a gesture) but also to understand the intended meaning. (iii) Develop and evaluate: Define new interactions that incorporate multimodal nonverbal communication and characterize the temporal and cognitive costs of these interactions.

The outcomes of this work will include: (1) an understanding of how nonverbal communication can be leveraged in the design of multimodal interfaces; (2) a generalized formal taxonomy for gestural interaction; (3) novel methods for fusing multimodal input in order to infer user intent; (4) new multimodal interaction techniques that leverage characteristics of natural human communication; and (5) experimentally validated mathematical models that allow designers and researchers to predict the temporal and cognitive costs associated with any proposed interaction techniques.