MuCHEx: A Multimodal Conversational Debugging Tool for Interactive Visual Exploration of Hierarchical Object ClassificationReza Shahriari, Yichi Yang, Danish Nisar Ahmed Tamboli, Michael Perez, Yuheng Zha, Jinyu Hou, Mingkai Deng, Eric D. Ragan, Jaime Ruiz, Daisy Zhe Wang, Zhiting Hu, Eric Xing

Object recognition is a fundamental challenge in computer vision, particularly for fine-grained object classification, where classes differ in minor features. Improved fine-grained object classification requires a teaching system with numerous classes and instances of data. As the number of hierarchical levels and instances grows, debugging these models becomes increasingly complex. Moreover, different types of debugging tasks require varying approaches, explanations, and levels of detail. We present MuCHEx, a multimodal conversational system that blends natural language and visual interaction for interactive debugging of hierarchical object classification. Natural language allows users to flexibly express high-level questions or debugging goals without needing to navigate complex interfaces, while adaptive explanations surface only the most relevant visual or textual details based on the user’s current task. This multimodal approach combines the expressiveness of language with the precision of direct manipulation, enabling context-aware exploration during model debugging.

Authors

  • Michael Perez
  • Headshot of Jaime Ruiz wearing a HololensJaime Ruiz

    • As well as: Reza Shahriari, Yichi Yang, Danish Nisar Ahmed Tamboli, Yuheng Zha, Jinyu Hou, Mingkai Deng, Eric D. Ragan, Daisy Zhe Wang, Zhiting Hu, Eric Xing

Citation

Reza Shahriari, Yichi Yang, Danish Nisar Ahmed Tamboli, Michael Perez, Yuheng Zha, Jinyu Hou, Mingkai Deng, Eric D. Ragan, Jaime Ruiz, Daisy Zhe Wang, Zhiting Hu, and Eric Xing. 2025. MuCHEx: A Multimodal Conversational Debugging Tool for Interactive Visual Exploration of Hierarchical Object Classification. IEEE Computer Graphics and Applications, 1–13. https://doi.org/10.1109/MCG.2025.3598204

Bibtex

@ARTICLE{11122913,
author={Shahriari, Reza and Yang, Yichi and Tamboli, Danish Nisar Ahmed and Perez, Michael and Zha, Yuheng and Hou, Jinyu and Deng, Mingkai and Ragan, Eric D. and Ruiz, Jaime and Wang, Daisy Zhe and Hu, Zhiting and Xing, Eric},
journal={ IEEE Computer Graphics and Applications },
title={{ MuCHEx: A Multimodal Conversational Debugging Tool for Interactive Visual Exploration of Hierarchical Object Classification }},
year={5555},
volume={},
number={01},
ISSN={1558-1756},
pages={1-13},
abstract={ Object recognition is a fundamental challenge in computer vision, particularly for fine-grained object classification, where classes differ in minor features. Improved fine-grained object classification requires a teaching system with numerous classes and instances of data. As the number of hierarchical levels and instances grows, debugging these models becomes increasingly complex. Moreover, different types of debugging tasks require varying approaches, explanations, and levels of detail. We present MuCHEx, a multimodal conversational system that blends natural language and visual interaction for interactive debugging of hierarchical object classification. Natural language allows users to flexibly express high-level questions or debugging goals without needing to navigate complex interfaces, while adaptive explanations surface only the most relevant visual or textual details based on the user's current task. This multimodal approach combines the expressiveness of language with the precision of direct manipulation, enabling context-aware exploration during model debugging. },
keywords={Visualization;Debugging;Natural languages;Image segmentation;Dogs;Adaptation models;Training;Predictive models;Navigation;Analytical models},
doi={10.1109/MCG.2025.3598204},
url = {https://doi.ieeecomputersociety.org/10.1109/MCG.2025.3598204},
publisher={IEEE Computer Society},
address={Los Alamitos, CA, USA},
month=aug}

Downloads

Related Funding