Search Results for author:
Found 62 papers, 44 papers with code
Chain of Draft: Thinking Faster by Writing Less
Large Language Models (LLMs) have demonstrated remarkable performance in solving complex reasoning tasks through mechanisms like Chain-of-Thought (CoT) prompting, which emphasizes verbose, step-by-step reasoning.
Switchable Decision: Dynamic Neural Generation Networks
Auto-regressive generation models achieve competitive performance across many different NLP tasks such as summarization, question answering, and classifications.
Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective
In this paper, we seek to empirically investigate knowledge transfer from larger to smaller models through a parametric perspective.
LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models
Quantization is an indispensable technique for serving Large Language Models (LLMs) and has recently found its way into LoRA fine-tuning.
Learning Stackable and Skippable LEGO Bricks for Efficient, Reconfigurable, and Variable-Resolution Diffusion Modeling
Diffusion models excel at generating photo-realistic images but come with significant computational costs in both training and sampling.
Ranked #13 on
Image Generation
on CelebA 64x64
DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models
Despite their impressive capabilities, large language models (LLMs) are prone to hallucinations, i. e., generating content that deviates from facts seen during pretraining.
Deep Reinforcement Learning from Hierarchical Preference Design
Researchers typically utilize feedback signals from the environment to handcraft a reward function, but this process is not always effective due to the varying scale and intricate dependencies of the feedback signals.
Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection
In this work, we establish a benchmark to evaluate the robustness of instruction-following LLMs against prompt injection attacks.
Summaries, Highlights, and Action items: Design, implementation and evaluation of an LLM-powered meeting recap system
We operationalize these representations into high-fidelity prototypes using dialogue summarization.
LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation
Pruning enhances the diversity of low-rank approximations, and low-rank approximation prevents pruning from losing too many expressive neurons.
Interactive Editing for Text Summarization
Summarizing lengthy documents is a common and essential task in our daily lives.
Query Rewriting for Retrieval-Augmented Large Language Models
Furthermore, to better align the query to the frozen modules, we propose a trainable scheme for our pipeline.
PROM: A Phrase-level Copying Mechanism with Pre-training for Abstractive Summarization
Based on the remarkable achievements of pre-trained language models in abstractive summarization, the copying mechanism has proved helpful by improving the factuality, stability, and overall performance.
Summarization with Precise Length Control
Many applications of text generation such as summarization benefit from accurately controlling the text length.
Personalized Abstractive Summarization by Tri-agent Generation Pipeline
The inference-only large language model (ChatGPT) serves as both the generator and editor, with a smaller model acting as the instructor to guide output generation.
In-Context Learning Unlocked for Diffusion Models
We present Prompt Diffusion, a framework for enabling in-context learning in diffusion-based generative models.
POUF: Prompt-oriented unsupervised fine-tuning for large pre-trained models
Through prompting, large-scale pre-trained models have become more expressive and powerful, gaining significant attention in recent years.
Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Models
Patch Diffusion meanwhile improves the performance of diffusion models trained on relatively small datasets, $e. g.$, as few as 5, 000 images to train from scratch.
Instruction Tuning with GPT-4
Prior work has shown that finetuning large language models (LLMs) using machine-generated instruction-following data enables such models to achieve remarkable zero-shot capabilities on new tasks, and no human-written instructions are needed.
AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
Therefore, many fine-tuning methods are proposed to learn incremental updates of pre-trained weights in a parameter efficient way, e. g., low-rank increments.
Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback
Large language models (LLMs), such as ChatGPT, are able to generate human-like, fluent responses for many downstream tasks, e. g., task-oriented dialog and question answering.
Guiding Large Language Models via Directional Stimulus Prompting
Our experiments demonstrate that the framework consistently improves LLMs' (e. g., ChatGPT, Codex, InstructGPT) performance on these supervised tasks using minimal labeled data.
A Prototype-Oriented Clustering for Domain Shift with Source Privacy
Finally, it refines the target model on the target domain data without guidance from the source model.
Attend to the Right Context: A Plug-and-Play Module for Content-Controllable Summarization
Content-Controllable Summarization generates summaries focused on the given controlling signals.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
In this work, we propose DIONYSUS (dynamic input optimization in pre-training for dialogue summarization), a pre-trained encoder-decoder model for summarizing dialogues in any new domain.
Momentum Calibration for Text Generation
The input and output of most text generation tasks can be transformed to two sequences of tokens and they can be modeled using sequence-to-sequence learning modeling tools such as Transformers.
Ranked #3 on
Text Summarization
on SAMSum
HyperTuning: Toward Adapting Large Language Models without Back-propagation
Fine-tuning large language models for different tasks can be costly and inefficient, and even methods that reduce the number of tuned parameters still require full gradient-based optimization.
Less is More: Task-aware Layer-wise Distillation for Language Model Compression
As such, TED reduces the knowledge gap between the two models and helps the student to fit better on the target task.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
Z-Code++ creates new state of the art on 9 out of 13 text summarization tasks across 5 languages.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
The information in tables can be an important complement to text, making table-based question answering (QA) systems of great value.
Ranked #14 on
Semantic Parsing
on WikiTableQuestions
PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance
Large Transformer-based models have exhibited superior performance in various natural language processing and computer vision tasks.
GODEL: Large-Scale Pre-Training for Goal-Directed Dialog
We introduce GODEL (Grounded Open Dialogue Language Model), a large pre-trained language model for dialog.
Diffusion-GAN: Training GANs with Diffusion
Both the observed and generated data are diffused by the same adaptive diffusion process.
Ranked #1 on
Image Generation
on LSUN Bedroom 256 x 256
ALLSH: Active Learning Guided by Local Sensitivity and Hardness
Active learning, which effectively collects informative unlabeled data for annotation, reduces the demand for labeled data.
MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation
We propose MoEBERT, which uses a Mixture-of-Experts structure to increase model capacity and inference speed.
CAMERO: Consistency Regularized Ensemble of Perturbed Language Models with Weight Sharing
To retain ensemble benefits while maintaining a low memory cost, we propose a consistency-regularized ensemble learning approach based on perturbed models, named CAMERO.
Truncated Diffusion Probabilistic Models and Diffusion-based Adversarial Auto-Encoders
Employing a forward diffusion chain to gradually map the data to a noise distribution, diffusion-based generative models learn how to generate the data by inferring a reverse diffusion chain.
Ranked #2 on
Text-to-Image Generation
on CUB
Mixing and Shifting: Exploiting Global and Local Dependencies in Vision MLPs
In this paper, to exploit both global and local dependencies without self-attention, we present Mix-Shift-MLP (MS-MLP) which makes the size of the local receptive field used for mixing increase with respect to the amount of spatial shifting.
No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Models
Analysis shows that the proposed schedule indeed reduces the redundancy and improves generalization performance.
Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention
In particular, we focus on the task of Commonsense Reasoning, demonstrating that the proposed external attention mechanism can augment existing transformer models and significantly improve the model's reasoning capabilities.
Ranked #2 on
Common Sense Reasoning
on CommonsenseQA
(using extra training data)
DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
We thus propose a new gradient-disentangled embedding sharing method that avoids the tug-of-war dynamics, improving both training efficiency and the quality of the pre-trained model.
Ranked #1 on
Natural Language Inference
on MRPC
Crossformer: Transformer with Alternated Cross-Layer Guidance
Crossformer with states sharing not only provides the desired cross-layer guidance and regularization but also reduces the memory requirement.
ARCH: Efficient Adversarial Regularized Training with Caching
Adversarial regularization can improve model generalization in many natural language processing tasks.
Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization
The Lottery Ticket Hypothesis suggests that an over-parametrized network consists of ``lottery tickets'', and training a certain collection of them (i. e., a subnetwork) can match the performance of the full model.
Adversarial Regularization as Stackelberg Game: An Unrolled Optimization Approach
Adversarial regularization has been shown to improve the generalization performance of deep learning models in various natural language processing tasks.
Token-wise Curriculum Learning for Neural Machine Translation
Existing curriculum learning approaches to Neural Machine Translation (NMT) require sampling sufficient amounts of "easy" samples from training data at the early training stage.
Greedy-Step Off-Policy Reinforcement Learning
Most of the policy evaluation algorithms are based on the theories of Bellman Expectation and Optimality Equation, which derive two popular approaches - Policy Iteration (PI) and Value Iteration (VI).
NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned
We review the EfficientQA competition from NeurIPS 2020.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
To date, most of recent work under the retrieval-reader framework for open-domain QA focuses on either extractive or generative reader exclusively.
Ranked #1 on
Open-Domain Question Answering
on TriviaQA
Rider: Reader-Guided Passage Reranking for Open-Domain Question Answering
Current open-domain question answering systems often follow a Retriever-Reader architecture, where the retriever first retrieves relevant passages and the reader then reads the retrieved passages to form an answer.
Generation-Augmented Retrieval for Open-domain Question Answering
We demonstrate that the generated contexts substantially enrich the semantics of the queries and GAR with sparse representations (BM25) achieves comparable or better performance than state-of-the-art dense retrieval methods such as DPR.
Ranked #9 on
Passage Retrieval
on Natural Questions
DeBERTa: Decoding-enhanced BERT with Disentangled Attention
Recent progress in pre-trained neural language models has significantly improved the performance of many natural language processing (NLP) tasks.
Ranked #1 on
Common Sense Reasoning
on SWAG
Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning
In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre-training, to inject language models with structured knowledge via learning from raw text.
Adversarial Training for Large Neural Language Models
In natural language processing (NLP), pre-training large neural language models such as BERT have demonstrated impressive gain in generalization for a variety of tasks, with further improvement from adversarial fine-tuning.
Ranked #6 on
Natural Language Inference
on ANLI test
(using extra training data)
The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding
We present MT-DNN, an open-source natural language understanding (NLU) toolkit that makes it easy for researchers and developers to train customized deep learning models.
SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization
However, due to limited data resources from downstream tasks and the extremely large capacity of pre-trained models, aggressive fine-tuning often causes the adapted model to overfit the data of downstream tasks and forget the knowledge of the pre-trained model.
Ranked #1 on
Natural Language Inference
on QNLI
X-SQL: reinforce schema representation with context
In this work, we present X-SQL, a new network architecture for the problem of parsing natural language to SQL query.
On the Variance of the Adaptive Learning Rate and Beyond
The learning rate warmup heuristic achieves remarkable success in stabilizing training, accelerating convergence and improving generalization for adaptive stochastic optimization algorithms like RMSprop and Adam.
A Hybrid Neural Network Model for Commonsense Reasoning
An HNN consists of two component models, a masked language model and a semantic similarity model, which share a BERT-based contextual encoder but use different model-specific input and output layers.
Ranked #1 on
Natural Language Understanding
on PDP60
Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
This paper explores the use of knowledge distillation to improve a Multi-Task Deep Neural Network (MT-DNN) (Liu et al., 2019) for learning text representations across multiple natural language understanding tasks.
Ranked #1 on
Semantic Textual Similarity
on SentEval
Multi-Task Deep Neural Networks for Natural Language Understanding
In this paper, we present a Multi-Task Deep Neural Network (MT-DNN) for learning representations across multiple natural language understanding (NLU) tasks.
Ranked #2 on
Natural Language Inference
on SciTail
