Search Results for author:
Found 36 papers, 21 papers with code
Modular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inference
Modular Transformers train modularized layers that have the same function of two or more consecutive layers in the original model via module replacing and knowledge distillation.
Commonsense Knowledge Transfer for Pre-trained Language Models
In this work, we introduce commonsense knowledge transfer, a framework to transfer the commonsense knowledge stored in a neural commonsense knowledge model to a general-purpose pre-trained language model.
SmartTrim: Adaptive Tokens and Parameters Pruning for Efficient Vision-Language Models
Experimental results demonstrate that SmartTrim significantly reduces the computation overhead (2-3 times) of various VLMs with comparable performance (only a 1-2% degradation) on various vision-language tasks.
RecurrentGPT: Interactive Generation of (Arbitrarily) Long Text
In addition to producing AI-generated content (AIGC), we also demonstrate the possibility of using RecurrentGPT as an interactive fiction that directly interacts with consumers.
To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis
First, we explore the consequences of repeating pre-training data, revealing that the model is susceptible to overfitting, leading to multi-epoch degradation.
Interactive Natural Language Processing
Interactive Natural Language Processing (iNLP) has emerged as a novel paradigm within the field of NLP, aimed at addressing limitations in existing frameworks while aligning with the ultimate goals of artificial intelligence.
Efficient Prompting via Dynamic In-Context Learning
To achieve this, we train a meta controller that predicts the number of in-context examples suitable for the generalist model to make a good prediction based on the performance-efficiency trade-off for a specific input.
Controlled Text Generation with Natural Language Instructions
Large language models generate fluent texts and can follow natural language instructions to solve a wide range of tasks without task-specific training.
Poor Man's Quality Estimation: Predicting Reference-Based MT Metrics Without the Reference
Machine translation quality estimation (QE) predicts human judgements of a translation hypothesis without seeing the reference.
X$^2$-VLM: All-In-One Pre-trained Model For Vision-Language Tasks
Moreover, we show that the modular design of X$^2$-VLM results in high transferability for X$^2$-VLM to be utilized in any language or domain.
Ranked #1 on
Cross-Modal Retrieval
on Flickr30k
Efficiently Tuned Parameters are Task Embeddings
Thus, we propose to exploit these efficiently tuned parameters as off-the-shelf task embeddings for the efficient selection of source datasets for intermediate-task transfer.
EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning
Pre-trained vision-language models (VLMs) have achieved impressive results in a range of vision-language tasks.
Write and Paint: Generative Vision-Language Models are Unified Modal Learners
In this work, we disclose the potential of symmetric generative vision-language pre-training in learning to write and paint concurrently, and propose a new unified modal model, named DaVinci, trained with prefix language modeling and prefix image modeling, a simple generative self-supervised objective on image-text pairs.
Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training
To this end, the cross-view language modeling framework considers both multi-modal data (i. e., image-caption pairs) and multi-lingual data (i. e., parallel sentence pairs) as two different views of the same object, and trains the model to align the two views by maximizing the mutual information between them with conditional masked language modeling and contrastive learning.
VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Models
We release the VLUE benchmark to promote research on building vision-language models that generalize well to more diverse images and concepts unseen during pre-training, and are practical in terms of efficiency-performance trade-off.
Contextual Representation Learning beyond Masked Language Modeling
How do masked language models (MLMs) such as BERT learn contextual representations?
Learning to Predict Persona Information forDialogue Personalization without Explicit Persona Description
Personalizing dialogue agents is important for dialogue systems to generate more specific, consistent, and engaging responses.
A Survey on Green Deep Learning
In recent years, larger and deeper models are springing up and continuously pushing state-of-the-art (SOTA) results across various fields like natural language processing (NLP) and computer vision (CV).
Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression
Recent studies on compression of pretrained language models (e. g., BERT) usually use preserved accuracy as the metric for evaluation.
BERT Learns to Teach: Knowledge Distillation with Meta Learning
We present Knowledge Distillation with Meta Learning (MetaDistil), a simple yet effective alternative to traditional knowledge distillation (KD) methods where the teacher model is fixed during training.
Learning from Perturbations: Diverse and Informative Dialogue Generation with Inverse Adversarial Training
By giving higher rewards for responses whose output probability reduces more significantly when dialogue history is perturbed, the model is encouraged to generate more diverse and consistent responses.
Blow the Dog Whistle: A Chinese Dataset for Cant Understanding with Common Sense and World Knowledge
Cant is important for understanding advertising, comedies and dog-whistle politics.
Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting
In this paper, we generalize text infilling (e. g., masked language models) by proposing Sequence Span Rewriting (SSR) as a self-supervised sequence-to-sequence (seq2seq) pre-training objective.
Pre-training Text-to-Text Transformers to Write and Reason with Concepts
To augment PTLMs with common sense, we propose generative and contrastive objectives as intermediate self-supervised pre-training tasks between general pre-training and downstream task-specific fine-tuning.
Pre-training Text-to-Text Transformers for Concept-centric Common Sense
Pre-trained language models (PTLM) have achieved impressive results in a range of natural language understanding (NLU) and generation (NLG) tasks.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
In this paper, we develop a general framework for interpretable natural language understanding that requires only a small set of human annotated explanations for training.
Connecting the Dots Between Fact Verification and Fake News Detection
Fact verification models have enjoyed a fast advancement in the last two years with the development of pre-trained language models like BERT and the release of large scale datasets such as FEVER.
BERT Loses Patience: Fast and Robust Inference with Early Exit
In this paper, we propose Patience-based Early Exit, a straightforward yet effective inference method that can be used as a plug-and-play technique to simultaneously improve the efficiency and robustness of a pretrained language model (PLM).
Scheduled DropHead: A Regularization Method for Transformer Models
In this paper, we introduce DropHead, a structured dropout method specifically designed for regularizing the multi-head attention mechanism, which is a key component of transformer, a state-of-the-art model for various NLP tasks.
Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models
While able to be trained in a fully self-supervised fashion, our model can be further fine-tuned with a little amount of human preference annotation to better imitate human judgment.
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing
Our approach first divides the original BERT into several modules and builds their compact substitutes.
Pseudo-Bidirectional Decoding for Local Sequence Transduction
PBD copies the corresponding representation of source tokens to the decoder as pseudo future context to enable the decoder to attends to its bi-directional context.
Self-Adversarial Learning with Comparative Discrimination for Text Generation
Conventional Generative Adversarial Networks (GANs) for text generation tend to have issues of reward sparsity and mode collapse that affect the quality and diversity of generated samples.
CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning
In this paper, we present a constrained text generation task, CommonGen associated with a benchmark dataset, to explicitly test machines for the ability of generative commonsense reasoning.
Ranked #1 on
Text Generation
on CommonGen
Improving Grammatical Error Correction with Machine Translation Pairs
The poor translation model resembles the ESL (English as a second language) learner and tends to generate translations of low quality in terms of fluency and grammatical correctness, while the good translation model generally generates fluent and grammatically correct translations.
BERT-based Lexical Substitution
Our approach first applies dropout to the target word{'}s embedding for partially masking the word, allowing BERT to take balanced consideration of the target word{'}s semantics and contexts for proposing substitute candidates, and then validates the candidates based on their substitution{'}s influence on the global contextualized representation of the sentence.
