In this paper, we introduce an architecture for multimodal communication between humans and computers engaged in a shared task. We describe a representative dialogue between an artificial agent and a human that will be demonstrated live during the presentation. This assumes a multimodal environment and semantics for facilitating communication and interaction with a computational agent. To this end, we have created an embodied 3D simulation environment enabling both the generation and interpretation of multiple modalities, including: language, gesture, and the visualization of objects moving and agents performing actions. Objects are encoded with rich semantic typing and action affordances, while actions themselves are encoded as multimodal expressions (programs), allowing for contextually salient inferences and decisions in the environment.

As well as: Nikhil Krishnaswamy, Pradyumna Narayana, Kyeongmin Rim, Rahul Bangar, Dhruva Patil, Gururaj Mulay, Ross Beveridge,, Bruce Draper, James Pustejovsky

Nikhil Krishnaswamy, Pradyumna Narayana, Isaac Wang, Kyeongmin Rim, Rahul Bangar, Dhruva Patil, Gururaj Mulay, Ross Beveridge, Jaime Ruiz, Bruce Draper, and James Pustejovsky. 2017. Communicating and Acting: Understanding Gesture in Simulation Semantics. In 12th International Conference on Computational Semantics (IWCS 2017)--Short papers.

@inproceedings{krishnaswamy2017communicating,
  title={Communicating and Acting: Understanding Gesture in Simulation Semantics},
  author={Krishnaswamy, Nikhil and Narayana, Pradyumna and Wang, Isaac and Rim, Kyeongmin and Bangar, Rahul and Patil, Dhruva and Mulay, Gururaj and Beveridge, Ross and Ruiz, Jaime and Draper, Bruce and others},
  booktitle={IWCS 2017—12th International Conference on Computational Semantics—Short papers},
  year={2017}
}