Search Results for author:
Found 56 papers, 34 papers with code
Sparse Modular Activation for Efficient Sequence Modeling
Linear State Space Models (SSMs) have demonstrated strong performance in a variety of sequence modeling tasks due to their efficient encoding of the recurrent structure.
Ranked #2 on
Long-range modeling
on LRA
In-Context Demonstration Selection with Cross Entropy Difference
Our method is based on the observation that the effectiveness of in-context demonstrations negatively correlates with the perplexity of the test example by a language model that was finetuned on that demonstration.
PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documents
PEARL outperforms zero-shot and chain-of-thought prompting on this dataset, and ablation experiments show that each stage of PEARL is critical to its performance.
InheritSumm: A General, Versatile and Compact Summarizer by Distilling from GPT
While large models such as GPT-3 demonstrate exceptional performance in zeroshot and fewshot summarization tasks, their extensive serving and fine-tuning costs hinder their utilization in various applications.
LMGQS: A Large-scale Dataset for Query-focused Summarization
We hypothesize that there is a hidden query for each summary sentence in a generic summarization annotation, and we utilize a large-scale pretrained language model to recover it.
Small Models are Valuable Plug-ins for Large Language Models
Large language models (LLMs) such as GPT-3 and GPT-4 are powerful but their weights are often publicly unavailable and their immense sizes make the models difficult to be tuned with common hardware.
G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
In this work, we present G-Eval, a framework of using large language models with chain-of-thoughts (CoT) and a form-filling paradigm, to assess the quality of NLG outputs.
APOLLO: A Simple Approach for Adaptive Pretraining of Language Models for Logical Reasoning
Logical reasoning of text is an important ability that requires understanding the information present in the text, their interconnections, and then reasoning through them to infer new conclusions.
Empowering Language Models with Knowledge Graph Reasoning for Question Answering
Answering open-domain questions requires world knowledge about in-context entities.
Retrieval Augmentation for Commonsense Reasoning: A Unified Approach
However, applying such methods to commonsense reasoning tasks faces two unique challenges, i. e., the lack of a general large-scale corpus for retrieval and a corresponding effective commonsense retriever.
Prompting GPT-3 To Be Reliable
While reliability is a broad and vaguely defined term, we decompose reliability into four main facets that correspond to the existing framework of ML safety and are well-recognized to be important: generalizability, social biases, calibration, and factuality.
Task Compass: Scaling Multi-task Pre-training with Task Prefix
Leveraging task-aware annotated data as supervised signals to assist with self-supervised learning on large-scale unlabeled data has become a new trend in pre-training language models.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
We call our method generate-then-read (GenRead), which first prompts a large language model to generate contextutal documents based on a given question, and then reads the generated documents to produce the final answer.
Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners
The goal of this work is to build flexible video-language models that can generalize to various video-to-text tasks from few examples, such as domain-specific captioning, question answering, and future event prediction.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Data
Surprisingly, we found that REtrieving from the traINing datA (REINA) only can lead to significant gains on multiple NLG and NLU tasks.
AdaPrompt: Adaptive Model Training for Prompt-based NLP
However, for prompt learning, there are still two salient gaps between NLP tasks and pretraining.
CLIP-Event: Connecting Text and Images with Event Structures
Vision-language (V+L) pretraining models have achieved great success in supporting multimedia applications by understanding the alignments between images and text.
MLP Architectures for Vision-and-Language Modeling: An Empirical Study
Based on this, we ask an even bolder question: can we have an all-MLP architecture for VL modeling, where both VL fusion and the vision encoder are replaced with MLPs?
Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention
In particular, we focus on the task of Commonsense Reasoning, demonstrating that the proposed external attention mechanism can augment existing transformer models and significantly improve the model's reasoning capabilities.
Ranked #1 on
Common Sense Reasoning
on CommonsenseQA
(using extra training data)
Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models
In this paper, we present Adversarial GLUE (AdvGLUE), a new multi-task benchmark to quantitatively and thoroughly explore and evaluate the vulnerabilities of modern large-scale language models under various types of adversarial attacks.
Ranked #1 on
Adversarial Robustness
on AdvGLUE
An Empirical Study of Training End-to-End Vision-and-Language Transformers
Vision-and-language (VL) pre-training has proven to be highly effective on various VL downstream tasks.
Ranked #18 on
Cross-Modal Retrieval
on COCO 2014
Leveraging Knowledge in Multilingual Commonsense Reasoning
Then we utilize a diverse of 4 English knowledge sources to provide more comprehensive coverage of knowledge in different formats.
Dict-BERT: Enhancing Language Model Pre-training with Dictionary
In addition to training with the masked language modeling objective, we propose two novel self-supervised pre-training tasks on word and sentence-level alignment between input text sequence and rare word definitions to enhance language modeling representation with dictionary.
KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answering
The recent proposed Fusion-in-Decoder (FiD), which is built on top of the pretrained generative model T5, achieves the state-of-the-art performance in the reading module.
NOAHQA: Numerical Reasoning with Interpretable Graph Question Answering Dataset
While diverse question answering (QA) datasets have been proposed and contributed significantly to the development of deep learning models for QA tasks, the existing datasets fall short in two aspects.
Want To Reduce Labeling Cost? GPT-3 Can Help
Data annotation is a time-consuming and labor-intensive process for many NLP tasks.
On Orthogonality Constraints for Transformers
Orthogonality constraints encourage matrices to be orthogonal for numerical stability.
Playing Lottery Tickets with Vision and Language
However, we can find "relaxed" winning tickets at 50%-70% sparsity that maintain 99% of the full accuracy.
UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training
Vision-and-language pre-training has achieved impressive success in learning multimodal representations between vision and language.
The Elastic Lottery Ticket Hypothesis
Based on these results, we articulate the Elastic Lottery Ticket Hypothesis (E-LTH): by mindfully replicating (or dropping) and re-ordering layers for one network, its corresponding winning ticket could be stretched (or squeezed) into a subnetwork for another deeper (or shallower) network from the same family, whose performance is nearly the same competitive as the latter's winning ticket directly found by IMP.
LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval
Multimodal pre-training has propelled great advancement in vision-and-language research.
Adversarial Masking: Towards Understanding Robustness Trade-off for Generalization
By incorporating different feature maps after the masking, we can distill better features to help model generalization.
Cluster-Former: Clustering-based Sparse Transformer for Question Answering
Transformer has become ubiquitous in the deep learning field.
EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets
Heavily overparameterized language models such as BERT, XLNet and T5 have achieved impressive success in many NLP tasks.
Counterfactual Variable Control for Robust and Interpretable Question Answering
We then conduct two novel CVC inference methods (on trained models) to capture the effect of comprehensive reasoning as the final prediction.
Cross-Thought for Sentence Encoder Pre-training
In this paper, we propose Cross-Thought, a novel approach to pre-training sequence encoder, which is instrumental in building reusable sequence embeddings for large-scale NLP tasks such as question answering.
Multi-Fact Correction in Abstractive Text Summarization
Pre-trained neural abstractive summarization systems have dominated extractive strategies on news summarization performance, at least in terms of ROUGE.
InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective
Large-scale language models such as BERT have achieved state-of-the-art performance across a wide range of NLP tasks.
Ranked #1 on
Natural Language Inference
on ANLI test
(using extra training data)
Contrastive Distillation on Intermediate Representations for Language Model Compression
Existing language model compression methods mostly use a simple L2 loss to distill knowledge in the intermediate representations of a large BERT model to a smaller one.
Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency Encoding
Transformer has become ubiquitous in the deep learning field.
Ranked #1 on
Open-Domain Question Answering
on SearchQA
Accelerating Real-Time Question Answering via Question Generation
Although deep neural networks have achieved tremendous success for question answering (QA), they are still suffering from heavy computational and energy cost for real product deployment.
FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding
During inference, the model makes predictions based on the text input in the target language and its translation in the source language.
Ranked #18 on
Zero-Shot Cross-Lingual Transfer
on XTREME
Multi-level Head-wise Match and Aggregation in Transformer for Textual Sequence Matching
Transformer has been successfully applied to many natural language processing tasks.
T3: Tree-Autoencoder Constrained Adversarial Text Generation for Targeted Attack
In particular, we propose a tree-based autoencoder to embed the discrete text data into a continuous representation space, upon which we optimize the adversarial perturbation.
Compositional De-Attention Networks
Attentional models are distinctly characterized by their ability to learn relative importance, i. e., assigning a different weight to input values.
Hierarchical Graph Network for Multi-hop Question Answering
In this paper, we present Hierarchical Graph Network (HGN) for multi-hop question answering.
Ranked #35 on
Question Answering
on HotpotQA
What does BERT Learn from Multiple-Choice Reading Comprehension Datasets?
Based on our experiments on the 5 key MCRC datasets - RACE, MCTest, MCScript, MCScript2. 0, DREAM - we observe that 1) fine-tuned BERT mainly learns how keywords lead to correct prediction, instead of learning semantic understanding and reasoning; and 2) BERT does not need correct syntactic information to solve the task; 3) there exists artifacts in these datasets such that they can be solved even without the full context.
Lightweight and Efficient Neural Natural Language Processing with Quaternion Networks
Many state-of-the-art neural models for NLP are heavily parameterized and thus memory inefficient.
Simple and Effective Curriculum Pointer-Generator Networks for Reading Comprehension over Long Narratives
This paper tackles the problem of reading comprehension over long narratives where documents easily span over thousands of tokens.
Unsupervised Deep Structured Semantic Models for Commonsense Reasoning
Commonsense reasoning is fundamental to natural language understanding.
Ranked #3 on
Natural Language Understanding
on PDP60
A Co-Matching Model for Multi-choice Reading Comprehension
Multi-choice reading comprehension is a challenging task, which involves the matching between a passage and a question-answer pair.
Evidence Aggregation for Answer Re-Ranking in Open-Domain Question Answering
We propose two methods, namely, strength-based re-ranking and coverage-based re-ranking, to make use of the aggregated evidence from different passages to better determine the answer.
Ranked #1 on
Open-Domain Question Answering
on Quasar
R$^3$: Reinforced Reader-Ranker for Open-Domain Question Answering
Second, we propose a novel method that jointly trains the Ranker along with an answer-generation Reader model, based on reinforcement learning.
Ranked #4 on
Open-Domain Question Answering
on Quasar
A Compare-Aggregate Model for Matching Text Sequences
We particularly focus on the different comparison functions we can use to match two vectors.
Machine Comprehension Using Match-LSTM and Answer Pointer
We propose two ways of using Pointer Net for our task.
Ranked #47 on
Question Answering
on SQuAD1.1 dev
Learning Natural Language Inference with LSTM
On the SNLI corpus, our model achieves an accuracy of 86. 1%, outperforming the state of the art.
Ranked #61 on
Natural Language Inference
on SNLI
