Search Results for author:
Found 88 papers, 54 papers with code
Pseudo-Masked Language Models for Unified Language Model Pre-Training
We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model (PMLM).
Decentralized Multi-agent Reinforcement Learning based State-of-Charge Balancing Strategy for Distributed Energy Storage System
Specifically, the first-order average consensus algorithm is utilized to expand the observations of the DESS state in a fully-decentralized way, and the initial actions (i. e., output power) are decided by the agents (i. e., energy storage units) according to these observations.
Testing for Peer Effects without Specifying the Network Structure
This paper proposes an Anderson-Rubin (AR) test for the presence of peer effects in panel data without the need to specify the network structure.
Augmenting Language Models with Long-Term Memory
Such a decoupled memory design can easily cache and update long-term past contexts for memory retrieval without suffering from memory staleness.
Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding
Scientific literature understanding tasks have gained significant attention due to their potential to accelerate scientific discovery.
Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers
This paper explores the effectiveness of model-generated signals in improving zero-shot generalization of text-to-text Transformers such as T5.
Chain-of-Skills: A Configurable Model for Open-domain Question Answering
Our approach outperforms recent self-supervised retrievers in zero-shot evaluations and achieves state-of-the-art fine-tuned retrieval performance on NQ, HotpotQA and OTT-QA.
Ranked #5 on
Question Answering
on HotpotQA
Bridging Discrete and Backpropagation: Straight-Through and Beyond
To address this issue, we propose a novel approach for approximating the gradient of parameters involved in generating discrete latent variables.
Pre-training Transformers for Knowledge Graph Completion
Learning transferable representation of knowledge graphs (KGs) is challenging due to the heterogeneous, multi-relational nature of graph structures.
Deep Vision in Analysis and Recognition of Radar Data: Achievements, Advancements and Challenges
Radars are widely used to obtain echo information for effective prediction, such as precipitation nowcasting.
Language Models as Inductive Reasoners
To this end, we propose a new task, which is to induce natural language rules from natural language facts, and create a dataset termed DEER containing 1. 2k rule-fact pairs for the task, where rules and facts are written in natural language.
Joint Beamforming and PD Orientation Design for Mobile Visible Light Communications
To address this challenge, we propose an alternating optimization algorithm to obtain the transmit beamforming and the PD orientation.
Efficient Long Sequence Modeling via State Space Augmented Transformer
Specifically, we augment a SSM into the bottom layer of SPADE, and we employ efficient local attention methods for the other layers.
AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning
Standard fine-tuning of large pre-trained language models (PLMs) for downstream tasks requires updating hundreds of millions to billions of parameters, and storing a large copy of the PLM weights for every task resulting in increased cost for storing, sharing and serving the models.
Open-domain Question Answering via Chain of Reasoning over Heterogeneous Knowledge
We propose a novel open-domain question answering (ODQA) framework for answering single/multi-hop questions across heterogeneous knowledge sources.
AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translation
Furthermore, existing MoE works do not consider computational constraints (e. g., FLOPs, latency) to guide their design.
Task-Aware Specialization for Efficient and Robust Dense Retrieval for Open-Domain Question Answering
Given its effectiveness on knowledge-intensive natural language processing tasks, dense retrieval models have become increasingly popular.
PTSEFormer: Progressive Temporal-Spatial Enhanced TransFormer Towards Video Object Detection
The temporal information is introduced by the temporal feature aggregation model (TFAM), by conducting an attention mechanism between the context frames and the target frame (i. e., the frame to be detected).
Ranked #3 on
Video Object Detection
on ImageNet VID
Deep Generative Modeling on Limited Data with Regularization by Nontransferable Pre-trained Models
Deep generative models (DGMs) are data-eager because learning a complex model on limited data suffers from a large variance and easily overfits.
AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning
Standard fine-tuning of large pre-trained language models (PLMs) for downstream tasks requires updating hundreds of millions to billions of parameters, and storing a large copy of the PLM weights for every task resulting in increased cost for storing, sharing and serving the models.
Visually-Augmented Language Modeling
With the visually-augmented context, VaLM uses a visual knowledge fusion layer to enable multimodal grounded language modeling by attending to both text context and visual knowledge in images.
METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals
We present an efficient method of pretraining large-scale autoencoding language models using training signals generated by an auxiliary model.
Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
Hyperparameter (HP) tuning in deep learning is an expensive process, prohibitively so for neural networks (NNs) with billions of parameters.
A Survey of Knowledge-Intensive NLP with Pre-Trained Language Models
With the increasing of model capacity brought by pre-trained language models, there emerges boosting needs for more knowledgeable natural language processing (NLP) models with advanced functionalities including providing and making flexible use of encyclopedic and commonsense knowledge.
No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Models
Analysis shows that the proposed schedule indeed reduces the redundancy and improves generalization performance.
AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models
Our framework AutoDistil addresses above challenges with the following steps: (a) Incorporates inductive bias and heuristics to partition Transformer search space into K compact sub-spaces (K=3 for typical student sizes of base, small and tiny); (b) Trains one SuperLM for each sub-space using task-agnostic objective (e. g., self-attention distillation) with weight-sharing of students; (c) Lightweight search for the optimal student without re-training.
Fine-Tuning Large Neural Language Models for Biomedical Natural Language Processing
Overall, domainspecific vocabulary and pretraining facilitate more robust models for fine-tuning.
Knowledge-Rich Self-Supervision for Biomedical Entity Linking
Zero-shot entity linking has emerged as a promising direction for generalizing to new entities, but it still requires example gold entity mentions during training and canonical descriptions for all entities, both of which are rarely available outside of Wikipedia.
Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention
In particular, we focus on the task of Commonsense Reasoning, demonstrating that the proposed external attention mechanism can augment existing transformer models and significantly improve the model's reasoning capabilities.
Ranked #1 on
Common Sense Reasoning
on CommonsenseQA
(using extra training data)
Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
Hyperparameter (HP) tuning in deep learning is an expensive process, prohibitively so for neural networks (NNs) with billions of parameters. We show that, in the recently discovered Maximal Update Parametrization ($\mu$P), many optimal HPs remain stable even as model size changes.
CLUES: Few-Shot Learning Evaluation in Natural Language Understanding
We demonstrate that while recent models reach human performance when they have access to large amounts of labeled data, there is a huge gap in performance in the few-shot setting for most tasks.
Open Domain Question Answering with A Unified Knowledge Interface
The retriever-reader framework is popular for open-domain question answering (ODQA) due to its ability to use explicit knowledge.
LiST: Lite Prompted Self-training Makes Parameter-Efficient Few-shot Learners
The first is the use of self-training to leverage large amounts of unlabeled data for prompt-based FN in few-shot settings.
Taming Sparsely Activated Transformer with Stochastic Experts
While most on-going research focuses on improving SAMs models by exploring methods of routing inputs to experts, our analysis reveals that such research might not lead to the solution we expect, i. e., the commonly-used routing methods based on gating mechanisms do not work better than randomly routing inputs to experts.
Learning to Persuade
Then we update the sender model to obtain an approximately optimal scheme using the receiver model.
ARCH: Efficient Adversarial Regularized Training with Caching
Adversarial regularization can improve model generalization in many natural language processing tasks.
Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization
The Lottery Ticket Hypothesis suggests that an over-parametrized network consists of ``lottery tickets'', and training a certain collection of them (i. e., a subnetwork) can match the performance of the full model.
Targeted Adversarial Training for Natural Language Understanding
We present a simple yet effective Targeted Adversarial Training (TAT) algorithm to improve adversarial training for natural language understanding.
Adversarial Regularization as Stackelberg Game: An Unrolled Optimization Approach
Adversarial regularization has been shown to improve the generalization performance of deep learning models in various natural language processing tasks.
Unveiling personnel movement in a larger indoor area with a non-overlapping multi-camera system
Surveillance cameras are widely applied for indoor occupancy measurement and human movement perception, which benefit for building energy management and social security.
Token-wise Curriculum Learning for Neural Machine Translation
Existing curriculum learning approaches to Neural Machine Translation (NMT) require sampling sufficient amounts of "easy" samples from training data at the early training stage.
Tracking Air Pollution in China: Near Real-Time PM2.5 Retrievals from Multiple Data Sources
Accordingly, a full-coverage high-resolution air pollutant dataset with timely updates and historical long-term records is essential to support both research and environmental management.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
To date, most of recent work under the retrieval-reader framework for open-domain QA focuses on either extractive or generative reader exclusively.
Ranked #1 on
Open-Domain Question Answering
on TriviaQA
Rider: Reader-Guided Passage Reranking for Open-Domain Question Answering
Current open-domain question answering systems often follow a Retriever-Reader architecture, where the retriever first retrieves relevant passages and the reader then reads the retrieved passages to form an answer.
NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned
We review the EfficientQA competition from NeurIPS 2020.
Posterior Differential Regularization with f-divergence for Improving Model Robustness
Theoretically, we provide a connection of two recent methods, Jacobian Regularization and Virtual Adversarial Training, under this framework.
A Tale of Two Linkings: Dynamically Gating between Schema Linking and Structural Linking for Text-to-SQL Parsing
In Text-to-SQL semantic parsing, selecting the correct entities (tables and columns) for the generated SQL query is both crucial and challenging; the parser is required to connect the natural language (NL) question and the SQL query to the structured knowledge in the database.
Generation-Augmented Retrieval for Open-domain Question Answering
We demonstrate that the generated contexts substantially enrich the semantics of the queries and GAR with sparse representations (BM25) achieves comparable or better performance than state-of-the-art dense retrieval methods such as DPR.
Ranked #9 on
Passage Retrieval
on Natural Questions
HittER: Hierarchical Transformers for Knowledge Graph Embeddings
Our proposed model consists of two different Transformer blocks: the bottom block extracts features of each entity-relation pair in the local neighborhood of the source entity and the top block aggregates the relational information from outputs of the bottom block.
Ranked #1 on
Link Prediction
on FB15k-237
(Hit@10 metric)
Very Deep Transformers for Neural Machine Translation
We explore the application of very deep Transformer models for Neural Machine Translation (NMT).
Ranked #1 on
Machine Translation
on WMT2014 English-French
(using extra training data)
Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
In this paper, we challenge this assumption by showing that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains over continual pretraining of general-domain language models.
Ranked #2 on
Participant Intervention Comparison Outcome Extraction
on EBM-NLP
(using extra training data)
DeBERTa: Decoding-enhanced BERT with Disentangled Attention
Recent progress in pre-trained neural language models has significantly improved the performance of many natural language processing (NLP) tasks.
Ranked #1 on
Natural Language Inference
on MRPC Dev
Adversarial Training for Commonsense Inference
We propose an AdversariaL training algorithm for commonsense InferenCE (ALICE).
Adversarial Training for Large Neural Language Models
In natural language processing (NLP), pre-training large neural language models such as BERT have demonstrated impressive gain in generalization for a variety of tasks, with further improvement from adversarial fine-tuning.
Ranked #4 on
Natural Language Inference
on ANLI test
(using extra training data)
Understanding the Difficulty of Training Transformers
Transformers have proved effective in many NLP tasks.
Ranked #5 on
Machine Translation
on WMT2014 English-French
UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model (PMLM).
Ranked #4 on
Question Generation
on SQuAD1.1
(using extra training data)
The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding
We present MT-DNN, an open-source natural language understanding (NLU) toolkit that makes it easy for researchers and developers to train customized deep learning models.
MLFcGAN: Multi-level Feature Fusion based Conditional GAN for Underwater Image Color Correction
Color correction for underwater images has received increasing interests, due to its critical role in facilitating available mature vision algorithms for underwater scenarios.
Multi-Modality Cascaded Fusion Technology for Autonomous Driving
Multi-modality fusion is the guarantee of the stability of autonomous driving systems.
RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers
The generalization challenge lies in (a) encoding the database relations in an accessible way for the semantic parser, and (b) modeling alignment between database columns and their mentions in a given query.
Ranked #9 on
Semantic Parsing
on spider
SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization
However, due to limited data resources from downstream tasks and the extremely large capacity of pre-trained models, aggressive fine-tuning often causes the adapted model to overfit the data of downstream tasks and forget the knowledge of the pre-trained model.
Ranked #1 on
Natural Language Inference
on QNLI
Adversarial Domain Adaptation for Machine Reading Comprehension
In this paper, we focus on unsupervised domain adaptation for Machine Reading Comprehension (MRC), where the source domain has a large amount of labeled data, while only unlabeled passages are available in the target domain.
The Learning of Fuzzy Cognitive Maps With Noisy Data: A Rapid and Robust Learning Method With Maximum Entropy
Index Terms-Fuzzy cognitive maps (FCMs), maximum entropy, noisy data, rapid and robust learning.
On the Variance of the Adaptive Learning Rate and Beyond
The learning rate warmup heuristic achieves remarkable success in stabilizing training, accelerating convergence and improving generalization for adaptive stochastic optimization algorithms like RMSprop and Adam.
A Hybrid Neural Network Model for Commonsense Reasoning
An HNN consists of two component models, a masked language model and a semantic similarity model, which share a BERT-based contextual encoder but use different model-specific input and output layers.
Ranked #1 on
Natural Language Understanding
on WNLI
DoubleTransfer at MEDIQA 2019: Multi-Source Transfer Learning for Natural Language Understanding in the Medical Domain
We use a multi-source transfer learning approach to transfer the knowledge from MT-DNN and SciBERT to natural language understanding tasks in the medical domain.
Conversing by Reading: Contentful Neural Conversation with On-demand Machine Reading
Although neural conversation models are effective in learning how to produce fluent responses, their primary challenge lies in knowing what to say to make the conversation contentful and non-vacuous.
Unified Language Model Pre-training for Natural Language Understanding and Generation
This paper presents a new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks.
Ranked #2 on
Generative Question Answering
on CoQA
(using extra training data)
Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
This paper explores the use of knowledge distillation to improve a Multi-Task Deep Neural Network (MT-DNN) (Liu et al., 2019) for learning text representations across multiple natural language understanding tasks.
Ranked #1 on
Semantic Textual Similarity
on SentEval
A Hybrid Retrieval-Generation Neural Conversation Model
In this paper, we propose a hybrid neural conversation model that combines the merits of both response retrieval and generation methods.
Unsupervised Deep Structured Semantic Models for Commonsense Reasoning
Commonsense reasoning is fundamental to natural language understanding.
Ranked #3 on
Natural Language Understanding
on PDP60
Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing
Variational autoencoders (VAEs) with an auto-regressive decoder have been applied for many natural language processing (NLP) tasks.
Multi-Task Deep Neural Networks for Natural Language Understanding
In this paper, we present a Multi-Task Deep Neural Network (MT-DNN) for learning representations across multiple natural language understanding (NLU) tasks.
Ranked #2 on
Natural Language Inference
on SciTail
ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
We present a large-scale dataset, ReCoRD, for machine reading comprehension requiring commonsense reasoning.
Stochastic Answer Networks for SQuAD 2.0
This paper presents an extension of the Stochastic Answer Network (SAN), one of the state-of-the-art machine reading comprehension models, to be able to judge whether a question is unanswerable or not.
Multi-task Learning with Sample Re-weighting for Machine Reading Comprehension
We propose a multi-task learning framework to learn a joint Machine Reading Comprehension (MRC) model that can be applied to a wide range of MRC tasks in different domains.
Navigating with Graph Representations for Fast and Scalable Decoding of Neural Language Models
Neural language models (NLMs) have recently gained a renewed interest by achieving state-of-the-art performance across many natural language processing (NLP) tasks.
Stochastic Answer Networks for Natural Language Inference
We propose a stochastic answer network (SAN) to explore multi-step inference strategies in Natural Language Inference.
Ranked #32 on
Natural Language Inference
on SNLI
Stochastic Answer Networks for Machine Reading Comprehension
We propose a simple yet robust stochastic answer network (SAN) that simulates multi-step reasoning in machine reading comprehension.
Ranked #24 on
Question Answering
on SQuAD1.1 dev
Language-Based Image Editing with Recurrent Attentive Models
First, we introduce a synthetic dataset, called CoSaL, to evaluate the end-to-end performance of our LBIE system.
Dynamic Fusion Networks for Machine Reading Comprehension
This paper presents a novel neural model - Dynamic Fusion Network (DFN), for machine reading comprehension (MRC).
An Empirical Analysis of Multiple-Turn Reasoning Strategies in Reading Comprehension Tasks
Using a state-of-the-art RC model, we empirically investigate the performance of single-turn and multiple-turn reasoning on the SQuAD and MS MARCO datasets.
Lexical Simplification with the Deep Structured Similarity Model
We explore the application of a Deep Structured Similarity Model (DSSM) to ranking in lexical simplification.
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
The size of the dataset and the fact that the questions are derived from real user search queries distinguishes MS MARCO from other well-known publicly available datasets for machine reading comprehension and question-answering.
