Search Results for author:
Found 71 papers, 34 papers with code
Modeling Entity Knowledge for Fact Verification
Fact verification is a challenging task of identifying the truthfulness of given claims based on the retrieval of relevant evidence texts.
CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations
CoVoMix is capable of first converting dialogue text into multiple streams of discrete tokens, with each token stream representing semantic information for individual talkers.
Making Flow-Matching-Based Zero-Shot Text-to-Speech Laugh as You Like
In this work, we propose ELaTE, a zero-shot TTS that can generate natural laughing speech of any speaker based on a short audio prompt with precise control of laughter timing and expression.
Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks
We introduce Florence-2, a novel vision foundation model with a unified, prompt-based representation for a variety of computer vision and vision-language tasks.
Diffusion Conditional Expectation Model for Efficient and Robust Target Speech Extraction
Additionally, we introduce Regenerate-DCEM (R-DCEM) that can regenerate and optimize speech quality based on pre-processed speech from a discriminative model.
Adapting Large Language Model with Speech for Fully Formatted End-to-End Speech Recognition
Most end-to-end (E2E) speech recognition models are composed of encoder and decoder blocks that perform acoustic and language modeling functions.
Adapting Multi-Lingual ASR Models for Handling Multiple Talkers
State-of-the-art large-scale universal speech models (USMs) show a decent automatic speech recognition (ASR) performance across multiple domains and languages.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text Translation
Joint speech-language training is challenging due to the large demand for training data and GPU consumption, as well as the modality gap between speech and language.
i-Code Studio: A Configurable and Composable Framework for Integrative AI
Artificial General Intelligence (AGI) requires comprehensive understanding and generation capabilities for a variety of tasks spanning different modalities and functionalities.
InheritSumm: A General, Versatile and Compact Summarizer by Distilling from GPT
While large models such as GPT-3 demonstrate exceptional performance in zeroshot and fewshot summarization tasks, their extensive serving and fine-tuning costs hinder their utilization in various applications.
LMGQS: A Large-scale Dataset for Query-focused Summarization
We hypothesize that there is a hidden query for each summary sentence in a generic summarization annotation, and we utilize a large-scale pretrained language model to recover it.
i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data
The convergence of text, visual, and audio data is a key step towards human-like artificial intelligence, however the current Vision-Language-Speech landscape is dominated by encoder-only models which lack generative abilities.
Any-to-Any Generation via Composable Diffusion
We present Composable Diffusion (CoDi), a novel generative model capable of generating any combination of output modalities, such as language, image, video, or audio, from any combination of input modalities.
Ranked #7 on
Audio Generation
on AudioCaps
Automatic Prompt Optimization with "Gradient Descent" and Beam Search
Large Language Models (LLMs) have shown impressive performance as general purpose agents, but their abilities remain highly dependent on prompts which are hand written with onerous trial-and-error effort.
Code-Switching Text Generation and Injection in Mandarin-English ASR
Code-switching speech refers to a means of expression by mixing two or more languages within a single utterance.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action
We propose MM-REACT, a system paradigm that integrates ChatGPT with a pool of vision experts to achieve multimodal reasoning and action.
Ranked #22 on
Visual Question Answering
on MM-Vet
Target Sound Extraction with Variable Cross-modality Clues
Automatic target sound extraction (TSE) is a machine learning approach to mimic the human auditory perception capability of attending to a sound source of interest from a mixture of sources.
Unifying Vision, Text, and Layout for Universal Document Processing
UDOP leverages the spatial correlation between textual content and document image to model image, text, and layout modalities with one uniform representation.
Ranked #5 on
Visual Question Answering (VQA)
on InfographicVQA
(using extra training data)
ReCo: Region-Controlled Text-to-Image Generation
Human evaluation on PaintSkill shows that ReCo is +19. 28% and +17. 21% more accurate in generating images with correct object count and spatial relationship than the T2I model.
Ranked #2 on
Conditional Text-to-Image Synthesis
on COCO-MIG
UniSumm and SummZoo: Unified Model and Diverse Benchmark for Few-Shot Summarization
The high annotation costs and diverse demands of various summarization tasks motivate the development of few-shot summarization.
MACSum: Controllable Summarization with Mixed Attributes
We propose two simple and effective parameter-efficient approaches for the new task of mixed controllable summarization based on hard prompt tuning and soft prefix tuning.
Task Compass: Scaling Multi-task Pre-training with Task Prefix
Leveraging task-aware annotated data as supervised signals to assist with self-supervised learning on large-scale unlabeled data has become a new trend in pre-training language models.
Ranked #1 on
Sentence Completion
on HellaSwag
Generate rather than Retrieve: Large Language Models are Strong Context Generators
We call our method generate-then-read (GenRead), which first prompts a large language model to generate contextutal documents based on a given question, and then reads the generated documents to produce the final answer.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
Z-Code++ creates new state of the art on 9 out of 13 text summarization tasks across 5 languages.
Visual Clues: Bridging Vision and Language Foundations for Image Paragraph Captioning
Thanks to the strong zero-shot capability of foundation models, we start by constructing a rich semantic representation of the image (e. g., image tags, object attributes / locations, captions) as a structured textual prompt, called visual clues, using a vision foundation model.
Automatic Rule Induction for Interpretable Semi-Supervised Learning
Semi-supervised learning has shown promise in allowing NLP models to generalize from small amounts of labeled data.
i-Code: An Integrative and Composable Multimodal Learning Framework
Human intelligence is multimodal; we integrate visual, linguistic, and acoustic signals to maintain a holistic worldview.
Impossible Triangle: What's Next for Pre-trained Language Models?
Recent development of large-scale pre-trained language models (PLM) have significantly improved the capability of models in various NLP tasks, in terms of performance after task-specific fine-tuning and zero-shot / few-shot learning.
Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Data
Surprisingly, we found that REtrieving from the traINing datA (REINA) only can lead to significant gains on multiple NLG and NLU tasks.
AdaPrompt: Adaptive Model Training for Prompt-based NLP
However, for prompt learning, there are still two salient gaps between NLP tasks and pretraining.
Unsupervised Multi-Granularity Summarization
In this paper, we propose the first unsupervised multi-granularity summarization framework, GranuSum.
CLIP-Event: Connecting Text and Images with Event Structures
Vision-language (V+L) pretraining models have achieved great success in supporting multimedia applications by understanding the alignments between images and text.
Sequence-level self-learning with multiple hypotheses
For untranscribed speech data, the hypothesis from an ASR system must be used as a label.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
MLP Architectures for Vision-and-Language Modeling: An Empirical Study
Based on this, we ask an even bolder question: can we have an all-MLP architecture for VL modeling, where both VL fusion and the vision encoder are replaced with MLPs?
Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention
In particular, we focus on the task of Commonsense Reasoning, demonstrating that the proposed external attention mechanism can augment existing transformer models and significantly improve the model's reasoning capabilities.
Ranked #1 on
Common Sense Reasoning
on CommonsenseQA
(using extra training data)
Florence: A New Foundation Model for Computer Vision
Computer vision foundation models, which are trained on diverse, large-scale dataset and can be adapted to a wide range of downstream tasks, are critical for this mission to solve real-world computer vision applications.
Ranked #1 on
Action Recognition In Videos
on Kinetics-600
An Empirical Study of Training End-to-End Vision-and-Language Transformers
Vision-and-language (VL) pre-training has proven to be highly effective on various VL downstream tasks.
Ranked #20 on
Cross-Modal Retrieval
on COCO 2014
(using extra training data)
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
Self-supervised learning (SSL) achieves great success in speech recognition, while limited exploration has been attempted for other speech processing tasks.
Leveraging Knowledge in Multilingual Commonsense Reasoning
Then we utilize a diverse of 4 English knowledge sources to provide more comprehensive coverage of knowledge in different formats.
End-to-End Segmentation-based News Summarization
In this paper, we bring a new way of digesting news content by introducing the task of segmenting a news article into multiple sections and generating the corresponding summary to each section.
Dict-BERT: Enhancing Language Model Pre-training with Dictionary
In addition to training with the masked language modeling objective, we propose two novel self-supervised pre-training tasks on word and sentence-level alignment between input text sequence and rare word definitions to enhance language modeling representation with dictionary.
KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answering
The recent proposed Fusion-in-Decoder (FiD), which is built on top of the pretrained generative model T5, achieves the state-of-the-art performance in the reading module.
DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarization
For a dialogue, it corrupts a window of text with dialogue-inspired noise, and guides the model to reconstruct this window based on the content of the remaining conversation.
Does Knowledge Help General NLU? An Empirical Study
It is often observed in knowledge-centric tasks (e. g., common sense question and answering, relation classification) that the integration of external knowledge such as entity representation into language models can help provide useful information to boost the performance.
Want To Reduce Labeling Cost? GPT-3 Can Help
Data annotation is a time-consuming and labor-intensive process for many NLP tasks.
A Joint and Domain-Adaptive Approach to Spoken Language Understanding
In this paper, we attempt to bridge these two lines of research and propose a joint and domain adaptive approach to SLU.
MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarization
MediaSum, a large-scale media interview dataset consisting of 463. 6K transcripts with abstractive summaries.
Generating Human Readable Transcript for Automatic Speech Recognition with Pre-trained Language Model
Many downstream tasks and human readers rely on the output of the ASR system; therefore, errors introduced by the speaker and ASR system alike will be propagated to the next task in the pipeline.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Improving Zero-shot Neural Machine Translation on Language-specific Encoders-Decoders
However, the performance of using multiple encoders and decoders on zero-shot translation still lags behind universal NMT.
Speech-language Pre-training for End-to-end Spoken Language Understanding
End-to-end (E2E) spoken language understanding (SLU) can infer semantics directly from speech signal without cascading an automatic speech recognizer (ASR) with a natural language understanding (NLU) module.
Ranked #3 on
Spoken Language Understanding
on Fluent Speech Commands
(using extra training data)
UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data
In this paper, we propose a unified pre-training approach called UniSpeech to learn speech representations with both unlabeled and labeled data, in which supervised phonetic CTC learning and phonetically-aware contrastive self-supervised learning are conducted in a multi-task learning manner.
Fusing Context Into Knowledge Graph for Commonsense Question Answering
However, although a KG contains rich structural information, it lacks the context to provide a more precise understanding of the concepts.
Ranked #4 on
Common Sense Reasoning
on CommonsenseQA
(using extra training data)
LSTM-LM with Long-Term History for First-Pass Decoding in Conversational Speech Recognition
The context information is captured by the hidden states of LSTM-LMs across utterance and can be used to guide the first-pass search effectively.
Mixed-Lingual Pre-training for Cross-lingual Summarization
Cross-lingual Summarization (CLS) aims at producing a summary in the target language for an article in the source language.
SPLAT: Speech-Language Joint Pre-Training for Spoken Language Understanding
Besides conducting a self-supervised masked language modeling task on the two individual modules using unpaired speech and text, SPLAT aligns representations from the two modules in a shared latent space using a small amount of paired speech and text.
JAKET: Joint Pre-training of Knowledge Graph and Language Understanding
Knowledge graphs (KGs) contain rich information about world knowledge, entities and relations.
Mind The Facts: Knowledge-Boosted Coherent Abstractive Text Summarization
In this work, we propose a novel architecture that extends Transformer encoder-decoder architecture in order to improve on these shortcomings.
Meta Dialogue Policy Learning
Dialog policy determines the next-step actions for agents and hence is central to a dialogue system.
Data Augmentation for Spoken Language Understanding via Pretrained Language Models
The training of spoken language understanding (SLU) models often faces the problem of data scarcity.
Improving Readability for Automatic Speech Recognition Transcription
In this work, we propose a novel NLP task called ASR post-processing for readability (APR) that aims to transform the noisy ASR output into a readable text for humans and downstream tasks while maintaining the semantic meaning of the speaker.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
A Hierarchical Network for Abstractive Meeting Summarization with Cross-Domain Pretraining
With the abundance of automatic meeting transcripts, meeting summarization is of great interest to both participants and other parties.
Enhancing Factual Consistency of Abstractive Summarization
Automatic abstractive summaries are found to often distort or fabricate facts in the article.
Few-shot Natural Language Generation for Task-Oriented Dialog
It is pre-trained on a large set of annotated NLG corpus to acquire the controllable generation ability, and fine-tuned with only a few domain-specific labels to adapt to new domains.
Ranked #4 on
Data-to-Text Generation
on MULTIWOZ 2.1
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
Text summarization aims to extract essential information from a piece of text and transform the text into a concise version.
Leveraging Lead Bias for Zero-shot Abstractive News Summarization
A typical journalistic convention in news articles is to deliver the most salient information in the beginning, also known as the lead bias.
Multi-task Learning for Natural Language Generation in Task-Oriented Dialogue
In this paper, we propose a novel multi-task learning framework, NLG-LM, for natural language generation.
SIM: A Slot-Independent Neural Model for Dialogue State Tracking
In this paper, we put forward a slot-independent neural model (SIM) to track dialogue states while keeping the model complexity invariant to the number of dialogue slots.
Make Lead Bias in Your Favor: A Simple and Effective Method for News Summarization
For example, the pretrained model without finetuning outperforms pointer-generator network on CNN/DailyMail dataset.
SDNet: Contextualized Attention-based Deep Network for Conversational Question Answering
Conversational question answering (CQA) is a novel QA task that requires understanding of dialogue context.
Ranked #3 on
Question Answering
on CoQA
(Overall metric)
Conversational Question Answering
Machine Reading Comprehension
+1
