1 code implementation • ACL (dialdoc) 2021 • Yan Xu, Etsuko Ishii, Genta Indra Winata, Zhaojiang Lin, Andrea Madotto, Zihan Liu, Peng Xu, Pascale Fung
Information-seeking dialogue systems, including knowledge identification and response generation, aim to respond to users with fluent, coherent, and informative responses based on users’ needs, which.
no code implementations • 25 Nov 2024 • Wang Bill Zhu, Deqing Fu, Kai Sun, Yi Lu, Zhaojiang Lin, Seungwhan Moon, Kanika Narang, Mustafa Canim, Yue Liu, Anuj Kumar, Xin Luna Dong
We hypothesize that a user's visual history with images reflecting their daily life, offers valuable insights into their interests and preferences, and can be leveraged for personalization.
no code implementations • 7 Mar 2024 • JieLin Qiu, Andrea Madotto, Zhaojiang Lin, Paul A. Crook, Yifan Ethan Xu, Xin Luna Dong, Christos Faloutsos, Lei LI, Babak Damavandi, Seungwhan Moon
We have developed the \textbf{SnapNTell Dataset}, distinct from traditional VQA datasets: (1) It encompasses a wide range of categorized entities, each represented by images and explicitly named in the answers; (2) It features QA pairs that require extensive knowledge for accurate responses.
1 code implementation • 16 Feb 2024 • Zekun Li, Zhiyu Zoey Chen, Mike Ross, Patrick Huber, Seungwhan Moon, Zhaojiang Lin, Xin Luna Dong, Adithya Sagar, Xifeng Yan, Paul A. Crook
We also show that by fine-tuning on a small collection of diverse task-oriented dialogues, we can equip modestly sized models, specifically a 13B parameter LLaMA2-Chat model, with function-calling capabilities and DST performance comparable to ChatGPT while maintaining their chat capabilities.
1 code implementation • 27 Sep 2023 • Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Tushar Nagarajan, Matt Smith, Shashank Jain, Chun-Fu Yeh, Prakash Murugesan, Peyman Heidari, Yue Liu, Kavya Srinet, Babak Damavandi, Anuj Kumar
We present Any-Modality Augmented Language Model (AnyMAL), a unified model that reasons over diverse input modality signals (i. e. text, image, video, audio, IMU motion sensor), and generates textual responses.
Ranked #9 on
Video Question Answering
on STAR Benchmark
1 code implementation • 23 May 2023 • Hyundong Cho, Andrea Madotto, Zhaojiang Lin, Khyathi Raghavi Chandu, Satwik Kottur, Jing Xu, Jonathan May, Chinnadhurai Sankar
Dialogue systems are frequently updated to accommodate new services, but naively updating them by continually training with data for new services in diminishing performance on previously learnt services.
no code implementations • 15 Nov 2022 • Derek Xu, Shuyan Dong, Changhan Wang, Suyoun Kim, Zhaojiang Lin, Akshat Shrivastava, Shang-Wen Li, Liang-Hsuan Tseng, Alexei Baevski, Guan-Ting Lin, Hung-Yi Lee, Yizhou Sun, Wei Wang
Recent studies find existing self-supervised speech encoders contain primarily acoustic rather than semantic information.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+11
2 code implementations • 26 Oct 2022 • Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Alireza Dirafzoon, Aparajita Saraf, Amy Bearman, Babak Damavandi
We present IMU2CLIP, a novel pre-training approach to align Inertial Measurement Unit (IMU) motion sensor recordings with video and text, by projecting them into the joint representation space of Contrastive Language-Image Pre-training (CLIP).
no code implementations • 14 Oct 2022 • Yejin Bang, Tiezheng Yu, Andrea Madotto, Zhaojiang Lin, Mona Diab, Pascale Fung
Therefore, we introduce a framework for value-aligned classification that performs prediction based on explicitly written human values in the command.
1 code implementation • CVPR 2022 • Yingruo Fan, Zhaojiang Lin, Jun Saito, Wenping Wang, Taku Komura
Speech-driven 3D facial animation is challenging due to the complex geometry of human faces and the limited availability of 3D audio-visual data.
Ranked #1 on
3D Face Animation
on VOCASET
no code implementations • 4 Dec 2021 • Yingruo Fan, Zhaojiang Lin, Jun Saito, Wenping Wang, Taku Komura
The existing datasets are collected to cover as many different phonemes as possible instead of sentences, thus limiting the capability of the audio-based model to learn more diverse contexts.
2 code implementations • 15 Oct 2021 • Andrea Madotto, Zhaojiang Lin, Genta Indra Winata, Pascale Fung
A simple yet unexplored solution is prompt-based few-shot learning (Brown et al. 2020) which does not require gradient-based fine-tuning but instead uses a few examples in the LM context as the only source of learning.
1 code implementation • EMNLP (MRL) 2021 • Genta Indra Winata, Andrea Madotto, Zhaojiang Lin, Rosanne Liu, Jason Yosinski, Pascale Fung
General-purpose language models have demonstrated impressive capabilities, performing on par with state-of-the-art approaches on a range of downstream natural language processing (NLP) tasks and benchmarks when inferring instructions from very few examples.
1 code implementation • EMNLP 2021 • Zhaojiang Lin, Bing Liu, Andrea Madotto, Seungwhan Moon, Paul Crook, Zhenpeng Zhou, Zhiguang Wang, Zhou Yu, Eunjoon Cho, Rajen Subba, Pascale Fung
Zero-shot transfer learning for dialogue state tracking (DST) enables us to handle a variety of task-oriented dialogue domains without the expense of collecting in-domain data.
1 code implementation • 7 Jun 2021 • Etsuko Ishii, Yan Xu, Genta Indra Winata, Zhaojiang Lin, Andrea Madotto, Zihan Liu, Peng Xu, Pascale Fung
Information-seeking dialogue systems, including knowledge identification and response generation, aim to respond to users with fluent, coherent, and informative responses based on users' needs, which.
1 code implementation • 5 Jun 2021 • Zhaojiang Lin, Andrea Madotto, Genta Indra Winata, Peng Xu, Feijun Jiang, Yuxiang Hu, Chen Shi, Pascale Fung
However, existing datasets for end-to-end ToD modeling are limited to a single language, hindering the development of robust end-to-end ToD systems for multilingual countries and regions.
1 code implementation • NAACL 2021 • Zhaojiang Lin, Bing Liu, Seungwhan Moon, Paul Crook, Zhenpeng Zhou, Zhiguang Wang, Zhou Yu, Andrea Madotto, Eunjoon Cho, Rajen Subba
Zero-shot cross-domain dialogue state tracking (DST) enables us to handle unseen domains without the expense of collecting in-domain data.
2 code implementations • 10 May 2021 • Zhaojiang Lin, Bing Liu, Seungwhan Moon, Paul Crook, Zhenpeng Zhou, Zhiguang Wang, Zhou Yu, Andrea Madotto, Eunjoon Cho, Rajen Subba
Zero-shot cross-domain dialogue state tracking (DST) enables us to handle task-oriented dialogue in unseen domains without the expense of collecting in-domain data.
no code implementations • NAACL (CALCS) 2021 • Genta Indra Winata, Samuel Cahyawijaya, Zihan Liu, Zhaojiang Lin, Andrea Madotto, Pascale Fung
Multilingual language models have shown decent performance in multilingual and cross-lingual natural language understanding tasks.
1 code implementation • EMNLP 2021 • Andrea Madotto, Zhaojiang Lin, Zhenpeng Zhou, Seungwhan Moon, Paul Crook, Bing Liu, Zhou Yu, Eunjoon Cho, Zhiguang Wang
Continual learning in task-oriented dialogue systems can allow us to add new domains and functionalities through time without incurring the high cost of a whole system retraining.
1 code implementation • Findings of the Association for Computational Linguistics 2020 • Andrea Madotto, Etsuko Ishii, Zhaojiang Lin, Sumanth Dathathri, Pascale Fung
These large conversational models provide little control over the generated responses, and this control is further limited in the absence of annotated conversational datasets for attribute specific generation that can be used for fine-tuning the model.
1 code implementation • EMNLP 2020 • Zihan Liu, Genta Indra Winata, Peng Xu, Zhaojiang Lin, Pascale Fung
Despite the promising results of current cross-lingual models for spoken language understanding systems, they still suffer from imperfect cross-lingual representation alignments between the source and target languages, which makes the performance sub-optimal.
1 code implementation • Findings of the Association for Computational Linguistics 2020 • Andrea Madotto, Samuel Cahyawijaya, Genta Indra Winata, Yan Xu, Zihan Liu, Zhaojiang Lin, Pascale Fung
In this paper, we propose a method to embed the KB, of any size, directly into the model parameters.
1 code implementation • EMNLP 2020 • Zhaojiang Lin, Andrea Madotto, Genta Indra Winata, Pascale Fung
In this paper, we propose Minimalist Transfer Learning (MinTL) to simplify the system design process of task-oriented dialogue systems and alleviate the over-dependency on annotated data.
Ranked #15 on
Multi-domain Dialogue State Tracking
on MULTIWOZ 2.1
Dialogue State Tracking
Multi-domain Dialogue State Tracking
+3
1 code implementation • 28 Aug 2020 • Andrea Madotto, Zhaojiang Lin, Yejin Bang, Pascale Fung
The dialogue skills can be triggered automatically via a dialogue manager, or manually, thus allowing high-level control of the generated responses.
no code implementations • 21 Aug 2020 • Peng Xu, Zihan Liu, Genta Indra Winata, Zhaojiang Lin, Pascale Fung
Most emotion recognition methods tackle the emotion understanding task by considering individual emotion independently while ignoring their fuzziness nature and the interconnections among them.
Ranked #3 on
Emotion Classification
on SemEval 2018 Task 1E-c
no code implementations • 14 Aug 2020 • Andrea Madotto, Zihan Liu, Zhaojiang Lin, Pascale Fung
In this paper, we evaluate the priming few-shot ability of language models in the NLU, DST, DP and NLG tasks.
1 code implementation • ACL 2020 • Genta Indra Winata, Samuel Cahyawijaya, Zhaojiang Lin, Zihan Liu, Peng Xu, Pascale Fung
An increasing number of people in the world today speak a mixed-language as a result of being multilingual.
1 code implementation • Findings of the Association for Computational Linguistics 2020 • Zhaojiang Lin, Andrea Madotto, Pascale Fung
Fine-tuning pre-trained generative language models to down-stream language generation tasks has shown promising results.
2 code implementations • 28 Mar 2020 • Zhaojiang Lin, Genta Indra Winata, Peng Xu, Zihan Liu, Pascale Fung
Despite the great promise of Transformers in many sequence modeling tasks (e. g., machine translation), their deterministic nature hinders them from generalizing to high entropy tasks such as dialogue response generation.
1 code implementation • EMNLP (NLP4ConvAI) 2021 • Zhaojiang Lin, Zihan Liu, Genta Indra Winata, Samuel Cahyawijaya, Andrea Madotto, Yejin Bang, Etsuko Ishii, Pascale Fung
Experimental results show that the multilingual trained models outperform the translation-pipeline and that they are on par with the monolingual models, with the advantage of having a single model across multiple languages.
1 code implementation • 4 Mar 2020 • Genta Indra Winata, Samuel Cahyawijaya, Zihan Liu, Zhaojiang Lin, Andrea Madotto, Peng Xu, Pascale Fung
The great variability and complex characteristics of accents creates a major challenge for training a robust and accent-agnostic automatic speech recognition (ASR) system.
Audio and Speech Processing Sound
no code implementations • 30 Jan 2020 • Zihan Liu, Genta Indra Winata, Samuel Cahyawijaya, Andrea Madotto, Zhaojiang Lin, Pascale Fung
To verify this hypothesis, we investigate whether making models insensitive to the word order of the source language can improve the adaptation performance in target languages.
no code implementations • 7 Jan 2020 • Andrea Madotto, Zhaojiang Lin, Chien-Sheng Wu, Jamin Shin, Pascale Fung
Dialogue systems require a great deal of different but complementary expertise to assist, inform, and entertain humans.
1 code implementation • 21 Nov 2019 • Zihan Liu, Genta Indra Winata, Zhaojiang Lin, Peng Xu, Pascale Fung
Recently, data-driven task-oriented dialogue systems have achieved promising performance in English.
no code implementations • 30 Oct 2019 • Genta Indra Winata, Samuel Cahyawijaya, Zhaojiang Lin, Zihan Liu, Pascale Fung
Highly performing deep neural networks come at the cost of computational complexity that limits their practicality for deployment on portable devices.
1 code implementation • IJCNLP 2019 • Genta Indra Winata, Zhaojiang Lin, Jamin Shin, Zihan Liu, Pascale Fung
In countries that speak multiple main languages, mixing up different languages within a conversation is commonly called code-switching.
5 code implementations • IJCNLP 2019 • Zhaojiang Lin, Andrea Madotto, Jamin Shin, Peng Xu, Pascale Fung
Previous research on empathetic dialogue systems has mostly focused on generating responses given certain emotions.
1 code implementation • LREC 2020 • Chien-Sheng Wu, Andrea Madotto, Zhaojiang Lin, Peng Xu, Pascale Fung
User attributes provide rich and useful information for user understanding, yet structured and easy-to-use attributes are often sparsely populated.
no code implementations • WS 2019 • Genta Indra Winata, Zhaojiang Lin, Pascale Fung
In this paper, we propose Multilingual Meta-Embeddings (MME), an effective method to learn multilingual representations by leveraging monolingual pre-trained embeddings.
2 code implementations • 28 Jul 2019 • Zhaojiang Lin, Peng Xu, Genta Indra Winata, Farhad Bin Siddique, Zihan Liu, Jamin Shin, Pascale Fung
In this paper, we present an end-to-end empathetic conversation agent CAiRE.
no code implementations • 10 Jun 2019 • Genta Indra Winata, Andrea Madotto, Zhaojiang Lin, Jamin Shin, Yan Xu, Peng Xu, Pascale Fung
Detecting emotion from dialogue is a challenge that has not yet been extensively surveyed.
no code implementations • SEMEVAL 2019 • Genta Indra Winata, Andrea Madotto, Zhaojiang Lin, Jamin Shin, Yan Xu, Peng Xu, Pascale Fung
Detecting emotion from dialogue is a challenge that has not yet been extensively surveyed.
1 code implementation • ACL 2019 • Zhaojiang Lin, Andrea Madotto, Chien-Sheng Wu, Pascale Fung
Existing personalized dialogue models use human designed persona descriptions to improve dialogue consistency.
no code implementations • 29 Oct 2018 • Zhaojiang Lin, Genta Indra Winata, Pascale Fung
Existing models on open-domain comment generation are difficult to train, and they produce repetitive and uninteresting responses.