MARRS: Multimodal Reference Resolution System

no code implementations • 3 Nov 2023 • Halim Cagri Ates, Shruti Bhargava, Site Li, Jiarui Lu, Siddhardha Maddula, Joel Ruben Antony Moniz, Anil Kumar Nalamalapu, Roman Hoang Nguyen, Melis Ozyildirim, Alkesh Patel, Dhivya Piraviperumal, Vincent Renkens, Ankit Samal, Thy Tran, Bo-Hsiang Tseng, Hong Yu, Yuan Zhang, Rong Zou

Successfully handling context is essential for any dialog understanding task.

Natural Language Understanding

Referring to Screen Texts with Voice Assistants

no code implementations • 10 Jun 2023 • Shruti Bhargava, Anand Dhoot, Ing-Marie Jonsson, Hoang Long Nguyen, Alkesh Patel, Hong Yu, Vincent Renkens

We collect a dataset and propose a lightweight general-purpose model for this novel experience.

Navigate Visual Grounding

MMIU: Dataset for Visual Intent Understanding in Multimodal Assistants

no code implementations • 13 Oct 2021 • Alkesh Patel, Joel Ruben Antony Moniz, Roman Nguyen, Nick Tzou, Hadas Kotek, Vincent Renkens

In multimodal assistant, where vision is also one of the input modalities, the identification of user intent becomes a challenging task as visual input can influence the outcome.

intent-classification Intent Classification +4

Generating Natural Questions from Images for Multimodal Assistants

no code implementations • 17 Nov 2020 • Alkesh Patel, Akanksha Bindal, Hadas Kotek, Christopher Klein, Jason Williams

We evaluate our approach using standard evaluation metrics such as BLEU, METEOR, ROUGE, and CIDEr to show the relevance of generated questions with human-provided questions.

Common Sense Reasoning Natural Questions +4

Noise Robust Named Entity Understanding for Voice Assistants

no code implementations • NAACL 2021 • Deepak Muralidharan, Joel Ruben Antony Moniz, Sida Gao, Xiao Yang, Justine Kao, Stephen Pulman, Atish Kothari, Ray Shen, Yinying Pan, Vivek Kaul, Mubarak Seyed Ibrahim, Gang Xiang, Nan Dun, Yidan Zhou, Andy O, Yuan Zhang, Pooja Chitkara, Xuan Wang, Alkesh Patel, Kushal Tayal, Roger Zheng, Peter Grasch, Jason D. Williams, Lin Li

Named Entity Recognition (NER) and Entity Linking (EL) play an essential role in voice assistant interaction, but are challenging due to the special difficulties associated with spoken user queries.

domain classification Entity Linking +5