1 code implementation • ACL 2022 • Jing Gu, Eliana Stefani, Qi Wu, Jesse Thomason, Xin Eric Wang
A long-term goal of AI research is to build intelligent agents that can communicate with humans in natural language, perceive the environment, and perform real-world tasks.
1 code implementation • 10 Nov 2021 • Yizhou Zhao, Kaixiang Lin, Zhiwei Jia, Qiaozi Gao, Govind Thattai, Jesse Thomason, Gaurav S. Sukhatme
However, current simulators for Embodied AI (EAI) challenges only provide simulated indoor scenes with a limited number of layouts.
1 code implementation • 1 Oct 2021 • Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan-Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, Dilek Hakkani-Tur
Robots operating in human spaces must be able to engage in natural language interaction with people, both understanding and executing instructions, and using conversation to resolve ambiguity and recover from mistakes.
1 code implementation • 10 Aug 2021 • Alessandro Suglia, Qiaozi Gao, Jesse Thomason, Govind Thattai, Gaurav Sukhatme
Language-guided robots performing home and office tasks must navigate in and interact with the world.
1 code implementation • 26 Jul 2021 • Jesse Thomason, Mohit Shridhar, Yonatan Bisk, Chris Paxton, Luke Zettlemoyer
We introduce several CLIP-based models for distinguishing objects and demonstrate that while recent advances in jointly modeling vision and language are useful for robotic language understanding, it is still the case that these image-based models are weaker at understanding the 3D nature of objects -- properties which play a key role in manipulation.
1 code implementation • Findings of the Association for Computational Linguistics 2020 • Homero Roman Roman, Yonatan Bisk, Jesse Thomason, Asli Celikyilmaz, Jianfeng Gao
In this paper, we go beyond instruction following and introduce a two-agent task where one agent navigates and asks questions that a second, guiding agent answers.
no code implementations • 23 Oct 2020 • Shurjo Banerjee, Jesse Thomason, Jason J. Corso
In each trial, the pair first cooperates to localize the robot on a global map visible to the Commander, then the Driver follows Commander instructions to move the robot to a sequence of target objects.
1 code implementation • 2 May 2020 • Homero Roman Roman, Yonatan Bisk, Jesse Thomason, Asli Celikyilmaz, Jianfeng Gao
In this paper, we go beyond instruction following and introduce a two-agent task where one agent navigates and asks questions that a second, guiding agent answers.
2 code implementations • EMNLP 2020 • Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, Joseph Turian
Language understanding research is held back by a failure to relate language to the physical world it describes and to the social interactions it facilitates.
5 code implementations • CVPR 2020 • Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, Dieter Fox
We present ALFRED (Action Learning From Realistic Environments and Directives), a benchmark for learning a mapping from natural language instructions and egocentric vision to sequences of actions for household tasks.
2 code implementations • 10 Jul 2019 • Jesse Thomason, Michael Murray, Maya Cakmak, Luke Zettlemoyer
To train agents that search an environment for a goal location, we define the Navigation from Dialog History task.
no code implementations • NAACL 2019 • Jesse Thomason, Daniel Gordon, Yonatan Bisk
We demonstrate the surprising strength of unimodal baselines in multimodal domains, and make concrete recommendations for best practices in future research.
1 code implementation • 2 Apr 2019 • Rosario Scalise, Jesse Thomason, Yonatan Bisk, Siddhartha Srinivasa
We collect over 13 hours of egocentric manipulation data for training a model to reason about whether a robot successfully placed unseen objects in or on one another.
1 code implementation • 29 Mar 2019 • Collin Burns, Jesse Thomason, Wesley Tansey
In science and medicine, model interpretations may be reported as discoveries of natural phenomena or used to guide patient treatments.
no code implementations • 20 Mar 2019 • Chris Paxton, Yonatan Bisk, Jesse Thomason, Arunkumar Byravan, Dieter Fox
High-level human instructions often correspond to behaviors with multiple implicit steps.
1 code implementation • 1 Mar 2019 • Jesse Thomason, Aishwarya Padmakumar, Jivko Sinapov, Nick Walker, Yuqian Jiang, Harel Yedidsion, Justin Hart, Peter Stone, Raymond J. Mooney
Natural language understanding for robotics can require substantial domain- and platform-specific engineering.
no code implementations • 1 Nov 2018 • Jesse Thomason, Daniel Gordon, Yonatan Bisk
We demonstrate the surprising strength of unimodal baselines in multimodal domains, and make concrete recommendations for best practices in future research.
no code implementations • IJCNLP 2017 • Rodolfo Corona, Jesse Thomason, Raymond Mooney
Speech is a natural channel for human-computer interaction in robotics and consumer applications.
Automatic Speech Recognition
Natural Language Understanding
+2
no code implementations • 6 Aug 2017 • Wesley Tansey, Jesse Thomason, James G. Scott
We consider the problem of estimating a regression function in the common situation where the number of features is small, where interpretability of the model is a high priority, and where simple linear or additive models fail to provide adequate performance.
no code implementations • WS 2017 • Jesse Thomason, Jivko Sinapov, Raymond Mooney
Multi-modal grounded language learning connects language predicates to physical properties of objects in the world.
no code implementations • EACL 2017 • Aishwarya Padmakumar, Jesse Thomason, Raymond J. Mooney
Natural language understanding and dialog management are two integral components of interactive dialog systems.