Search Results for author:
Found 117 papers, 74 papers with code
What Makes Good In-Context Examples for GPT-3?
In this work, we investigate whether there are more effective strategies for judiciously selecting in-context examples (relative to random sampling) that better leverage GPT-3’s in-context learning capabilities. Inspired by the recent success of leveraging a retrieval module to augment neural networks, we propose to retrieve examples that are semantically-similar to a test query sample to formulate its corresponding prompt.
Finding the Dominant Winning Ticket in Pre-Trained Language Models
Empirically, we show that (a) the dominant winning ticket can achieve performance that is comparable with that of the full-parameter model, (b) the dominant winning ticket is transferable across different tasks, (c) and the dominant winning ticket has a natural structure within each parameter matrix.
Few-Shot Named Entity Recognition: An Empirical Baseline Study
This paper presents an empirical study to efficiently build named entity recognition (NER) systems when a small amount of in-domain labeled data is available.
Rho-1: Not All Tokens Are What You Need
After fine-tuning, Rho-1-1B and 7B achieved state-of-the-art results of 40. 6% and 51. 8% on MATH dataset, respectively - matching DeepSeekMath with only 3% of the pretraining tokens.
A Note on LoRA
LoRA (Low-Rank Adaptation) has emerged as a preferred method for efficiently adapting Large Language Models (LLMs) with remarkable simplicity and efficacy.
Exploring the Mystery of Influential Data for Mathematical Reasoning
Additionally, we showcase the use of QaDS in creating efficient fine-tuning mixtures with various selection ratios, and analyze the quality of a wide range of open-source datasets, which can perform as a reference for future works on mathematical reasoning tasks.
Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoning
Large language models (LLMs) have shown great potential in complex reasoning tasks, yet their performance is often hampered by the scarcity of high-quality and reasoning-focused training datasets.
Ranked #49 on
Math Word Problem Solving
on MATH
(using extra training data)
Multi-LoRA Composition for Image Generation
Low-Rank Adaptation (LoRA) is extensively utilized in text-to-image models for the accurate rendition of specific elements like distinct characters or unique styles in generated images.
SciAgent: Tool-augmented Language Models for Scientific Reasoning
To make this task more practical and solvable for LLMs, we introduce a new task setting named tool-augmented scientific reasoning.
Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts
In the field of large language models (LLMs), aligning models with the diverse preferences of users is a critical challenge.
Supervised Knowledge Makes Large Language Models Better In-context Learners
Large Language Models (LLMs) exhibit emerging in-context learning abilities through prompt engineering.
Competition-Level Problems are Effective LLM Evaluators
Large language models (LLMs) have demonstrated impressive reasoning capabilities, yet there is ongoing debate about these abilities and the potential data contamination problem recently.
Language Models can be Logical Solvers
Despite their impressive performance, any parsing errors will inevitably result in the failure of the execution of the external logical solver and no answer to the logical questions.
Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective
In this paper, we seek to empirically investigate knowledge transfer from larger to smaller models through a parametric perspective.
LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models
Quantization is an indispensable technique for serving Large Language Models (LLMs) and has recently found its way into LoRA fine-tuning.
Sparse Backpropagation for MoE Training
One defining characteristic of Mixture-of-Expert (MoE) models is their capacity for conducting sparse computation via expert routing, leading to remarkable scalability.
Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency
We propose the Multi-Perspective Self-Consistency (MPSC) framework incorporating both inter- and intra-consistency across outputs from multiple perspectives.
ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving
Large language models have made significant progress in various language tasks, yet they still struggle with complex mathematics.
Ranked #10 on
Math Word Problem Solving
on MATH
(using extra training data)
Deep Reinforcement Learning with Hierarchical Reward Modeling
Researchers typically utilize feedback signals from the environment to handcraft a reward function, but this process is not always effective due to the varying scale and intricate dependencies of the feedback signals.
LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation
Pruning enhances the diversity of low-rank approximations, and low-rank approximation prevents pruning from losing too many expressive neurons.
GRILL: Grounded Vision-language Pre-training via Aligning Text and Image Regions
Generalization to unseen tasks is an important ability for few-shot learners to achieve better zero-/few-shot performance on diverse tasks.
Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy
In this paper, we show that strong performance can be achieved by a method we call Iter-RetGen, which synergizes retrieval and generation in an iterative manner.
Skill-Based Few-Shot Selection for In-Context Learning
Few-shot selection -- selecting appropriate examples for each test instance separately -- is important for in-context learning.
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
Unlike these models, humans typically utilize external tools to cross-check and refine their initial content, like using a search engine for fact-checking, or a code interpreter for debugging.
AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation
Diffusion models have gained significant attention in the realm of image generation due to their exceptional performance.
Code Execution with Pre-trained Language Models
Code execution is a fundamental aspect of programming language semantics that reflects the exact behavior of the code.
In-Context Learning Unlocked for Diffusion Models
We present Prompt Diffusion, a framework for enabling in-context learning in diffusion-based generative models.
Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Models
Patch Diffusion meanwhile improves the performance of diffusion models trained on relatively small datasets, $e. g.$, as few as 5, 000 images to train from scratch.
AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
Impressively, GPT-4 surpasses average human performance on SAT, LSAT, and math competitions, attaining a 95% accuracy rate on the SAT Math test and a 92. 5% accuracy on the English test of the Chinese national college entrance exam.
AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators
Many natural language processing (NLP) tasks rely on labeled data to train machine learning models with high performance.
RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation
The task of repository-level code completion is to continue writing the unfinished code based on a broader context of the repository.
AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
Therefore, many fine-tuning methods are proposed to learn incremental updates of pre-trained weights in a parameter efficient way, e. g., low-rank increments.
Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback
Large language models (LLMs), such as ChatGPT, are able to generate human-like, fluent responses for many downstream tasks, e. g., task-oriented dialog and question answering.
Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Models
However, the quality of the prompts depends on the demonstrations given to the models, and creating many of them by hand is costly.
Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise
In this paper, we introduce a novel dIffusion language modEl pre-training framework for text generation, which we call GENIE.
Generation-Augmented Query Expansion For Code Retrieval
Pre-trained language models have achieved promising success in code retrieval tasks, where a natural language documentation query is given to find the most relevant existing code snippet.
HyperTuning: Toward Adapting Large Language Models without Back-propagation
Fine-tuning large language models for different tasks can be costly and inefficient, and even methods that reduce the number of tuned parameters still require full gradient-based optimization.
GENIUS: Sketch-based Language Model Pre-training via Extreme and Selective Masking for Text Generation and Augmentation
We introduce GENIUS: a conditional text generation model using sketches as input, which can fill in the missing contexts for a given sketch (key information consisting of textual spans, phrases, or words, concatenated by mask tokens).
SimANS: Simple Ambiguous Negatives Sampling for Dense Text Retrieval
Thus, we propose a simple ambiguous negatives sampling method, SimANS, which incorporates a new sampling probability distribution to sample more ambiguous negatives.
Soft-Labeled Contrastive Pre-training for Function-level Code Representation
In this paper, we present \textbf{SCodeR}, a \textbf{S}oft-labeled contrastive pre-training framework with two positive sample construction methods to learn functional-level \textbf{Code} \textbf{R}epresentation.
Less is More: Task-aware Layer-wise Distillation for Language Model Compression
As such, TED reduces the knowledge gap between the two models and helps the student to fit better on the target task.
CodeT: Code Generation with Generated Tests
A natural way to evaluate the quality and correctness of a code solution is to run it against a set of test cases, but the manual creation of such test cases is often costly and time-consuming.
Ranked #1 on
Code Generation
on APPS
(Introductory Pass@1 metric)
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
The information in tables can be an important complement to text, making table-based question answering (QA) systems of great value.
Ranked #7 on
Semantic Parsing
on WikiTableQuestions
Joint Generator-Ranker Learning for Natural Language Generation
Generate-then-rank is a widely used mechanism for text generation, where a generator produces multiple text candidates and a ranker chooses the best one among the text candidates.
PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance
Large Transformer-based models have exhibited superior performance in various natural language processing and computer vision tasks.
CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generation
Usually, expensive text-code paired data is essential for training a code generation model.
Ranked #121 on
Code Generation
on HumanEval
Making Large Language Models Better Reasoners with Step-Aware Verifier
Few-shot learning is a challenging task that requires language models to generalize from limited examples.
Ranked #49 on
Arithmetic Reasoning
on GSM8K
Diffusion-GAN: Training GANs with Diffusion
Both the observed and generated data are diffused by the same adaptive diffusion process.
Ranked #1 on
Image Generation
on LSUN Bedroom 256 x 256
A Self-Paced Mixed Distillation Method for Non-Autoregressive Generation
To further illustrate the commercial value of our approach, we conduct experiments on three generation tasks in real-world advertisements applications.
ALLSH: Active Learning Guided by Local Sensitivity and Hardness
Active learning, which effectively collects informative unlabeled data for annotation, reduces the demand for labeled data.
DialogVED: A Pre-trained Latent Variable Encoder-Decoder Model for Dialog Response Generation
Dialog response generation in open domain is an important research topic where the main challenge is to generate relevant and diverse responses.
MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation
We propose MoEBERT, which uses a Mixture-of-Experts structure to increase model capacity and inference speed.
CAMERO: Consistency Regularized Ensemble of Perturbed Language Models with Weight Sharing
To retain ensemble benefits while maintaining a low memory cost, we propose a consistency-regularized ensemble learning approach based on perturbed models, named CAMERO.
Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
Hyperparameter (HP) tuning in deep learning is an expensive process, prohibitively so for neural networks (NNs) with billions of parameters.
Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models
This motivates us to propose input-tuning, which fine-tunes both the continuous prompts and the input representations, leading to a more effective way to adapt unfamiliar inputs to frozen PLMs.
Controllable Natural Language Generation with Contrastive Prefixes
We propose a novel supervised method and also an unsupervised method to train the prefixes for single-aspect control while the combination of these two methods can achieve multi-aspect control.
Truncated Diffusion Probabilistic Models and Diffusion-based Adversarial Auto-Encoders
Employing a forward diffusion chain to gradually map the data to a noise distribution, diffusion-based generative models learn how to generate the data by inferring a reverse diffusion chain.
Ranked #1 on
Text-to-Image Generation
on CUB
Mixing and Shifting: Exploiting Global and Local Dependencies in Vision MLPs
In this paper, to exploit both global and local dependencies without self-attention, we present Mix-Shift-MLP (MS-MLP) which makes the size of the local receptive field used for mixing increase with respect to the amount of spatial shifting.
No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Models
Analysis shows that the proposed schedule indeed reduces the redundancy and improves generalization performance.
Reasoning Like Program Executors
Reasoning over natural language is a long-standing goal for the research community.
Ranked #2 on
Question Answering
on DROP Test
(using extra training data)
CodeRetriever: Unimodal and Bimodal Contrastive Learning for Code Search
For bimodal contrastive learning, we leverage the documentation and in-line comments of code to build code-text pairs.
Contextual Bandit Applications in Customer Support Bot
We do, however, have access to partial feedback provided by the user (clicks, surveys, and other events) which can be leveraged to improve the user experience.
Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
Hyperparameter (HP) tuning in deep learning is an expensive process, prohibitively so for neural networks (NNs) with billions of parameters. We show that, in the recently discovered Maximal Update Parametrization ($\mu$P), many optimal HPs remain stable even as model size changes.
DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
We thus propose a new gradient-disentangled embedding sharing method that avoids the tug-of-war dynamics, improving both training efficiency and the quality of the pre-trained model.
Ranked #1 on
Natural Language Inference
on MRPC
Natural Language Inference
Natural Language Understanding
+2
DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models
To address these pain points, we propose a framework for resource- and parameter-efficient fine-tuning by leveraging the sparsity prior in both weight updates and the final model weights.
A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models
Large pre-trained vision-language (VL) models can learn a new task with a handful of examples and generalize to a new task without fine-tuning.
Ranked #4 on
Image Captioning
on Flickr30k Captions test
(SPICE metric)
Adversarial Retriever-Ranker for dense text retrieval
To address these challenges, we present Adversarial Retriever-Ranker (AR2), which consists of a dual-encoder retriever plus a cross-encoder ranker.
Crossformer: Transformer with Alternated Cross-Layer Guidance
Crossformer with states sharing not only provides the desired cross-layer guidance and regularization but also reduces the memory requirement.
XLM-K: Improving Cross-Lingual Language Model Pre-training with Multilingual Knowledge
The results on MLQA and NER exhibit the superiority of XLM-K in knowledge related tasks.
ARCH: Efficient Adversarial Regularized Training with Caching
Adversarial regularization can improve model generalization in many natural language processing tasks.
HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalizability
Fine-tuning large pre-trained models with task-specific data has achieved great success in NLP.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
TAPEX addresses the data scarcity challenge via guiding the language model to mimic a SQL executor on the diverse, large-scale and high-quality synthetic corpus.
Ranked #1 on
Semantic Parsing
on WikiSQL
(Denotation accuracy (test) metric)
LoRA: Low-Rank Adaptation of Large Language Models
We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks.
HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalization
Fine-tuning large pre-trained models with task-specific data has achieved great success in NLP.
Memory-Efficient Differentiable Transformer Architecture Search
To this end, we propose a multi-split reversible network and combine it with DARTS.
Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization
The Lottery Ticket Hypothesis suggests that an over-parametrized network consists of ``lottery tickets'', and training a certain collection of them (i. e., a subnetwork) can match the performance of the full model.
Poolingformer: Long Document Modeling with Pooling Attention
We first evaluate Poolingformer on two long sequence QA tasks: the monolingual NQ and the multilingual TyDi QA.
A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation
Large pretrained generative models like GPT-3 often suffer from hallucinating non-existent or incorrect content, which undermines their potential merits in real applications.
Adversarial Regularization as Stackelberg Game: An Unrolled Optimization Approach
Adversarial regularization has been shown to improve the generalization performance of deep learning models in various natural language processing tasks.
Finetuning Pretrained Transformers into RNNs
Specifically, we propose a swap-then-finetune procedure: in an off-the-shelf pretrained transformer, we replace the softmax attention with its linear-complexity recurrent alternative and then finetune.
Ranked #2 on
Machine Translation
on WMT2017 Chinese-English
Token-wise Curriculum Learning for Neural Machine Translation
Existing curriculum learning approaches to Neural Machine Translation (NMT) require sampling sufficient amounts of "easy" samples from training data at the early training stage.
What Makes Good In-Context Examples for GPT-$3$?
Inspired by the recent success of leveraging a retrieval module to augment large-scale neural network models, we propose to retrieve examples that are semantically-similar to a test sample to formulate its corresponding prompt.
NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned
We review the EfficientQA competition from NeurIPS 2020.
UnitedQA: A Hybrid Approach for Open Domain Question Answering
To date, most of recent work under the retrieval-reader framework for open-domain QA focuses on either extractive or generative reader exclusively.
Ranked #1 on
Open-Domain Question Answering
on TriviaQA
Rider: Reader-Guided Passage Reranking for Open-Domain Question Answering
Current open-domain question answering systems often follow a Retriever-Reader architecture, where the retriever first retrieves relevant passages and the reader then reads the retrieved passages to form an answer.
BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretraining
In this paper, we propose BANG, a new pretraining model to Bridge the gap between Autoregressive (AR) and Non-autoregressive (NAR) Generation.
Few-Shot Named Entity Recognition: A Comprehensive Study
This paper presents a comprehensive study to efficiently build named entity recognition (NER) systems when a small number of in-domain labeled data is available.
GLGE: A New General Language Generation Evaluation Benchmark
Multi-task benchmarks such as GLUE and SuperGLUE have driven great progress of pretraining and transfer learning in Natural Language Processing (NLP).
MixKD: Towards Efficient Distillation of Large-scale Language Models
Large-scale language models have recently demonstrated impressive empirical performance.
CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding
To verify the effectiveness of the proposed framework, we apply CoDA to Transformer-based models on a wide range of natural language understanding tasks.
Improving Self-supervised Pre-training via a Fully-Explored Masked Language Model
We prove, from a theoretical perspective, that the gradients derived from this new masking schema have a smaller variance and can lead to more efficient self-supervised training.
Ranked #1 on
Sentence Classification
on ACL-ARC
A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation
Adversarial training has been shown effective at endowing the learned representations with stronger generalization ability.
Ranked #8 on
Machine Translation
on IWSLT2014 German-English
Generation-Augmented Retrieval for Open-domain Question Answering
We demonstrate that the generated contexts substantially enrich the semantics of the queries and GAR with sparse representations (BM25) achieves comparable or better performance than state-of-the-art dense retrieval methods such as DPR.
Ranked #9 on
Passage Retrieval
on Natural Questions
Example-Based Named Entity Recognition
We present a novel approach to named entity recognition (NER) in the presence of scarce data that we call example-based NER.
DeBERTa: Decoding-enhanced BERT with Disentangled Attention
Recent progress in pre-trained neural language models has significantly improved the performance of many natural language processing (NLP) tasks.
Ranked #1 on
Common Sense Reasoning
on SWAG
Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning
In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre-training, to inject language models with structured knowledge via learning from raw text.
Adversarial Training for Large Neural Language Models
In natural language processing (NLP), pre-training large neural language models such as BERT have demonstrated impressive gain in generalization for a variety of tasks, with further improvement from adversarial fine-tuning.
Ranked #6 on
Natural Language Inference
on ANLI test
(using extra training data)
Understanding the Difficulty of Training Transformers
Transformers have proved effective in many NLP tasks.
Ranked #5 on
Machine Translation
on WMT2014 English-French
The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding
We present MT-DNN, an open-source natural language understanding (NLU) toolkit that makes it easy for researchers and developers to train customized deep learning models.
Conditional Self-Attention for Query-based Summarization
Thereby, the contextual dependencies modeled by CSA will be highly relevant to the query.
SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization
However, due to limited data resources from downstream tasks and the extremely large capacity of pre-trained models, aggressive fine-tuning often causes the adapted model to overfit the data of downstream tasks and forget the knowledge of the pre-trained model.
Ranked #1 on
Natural Language Inference
on QNLI
X-SQL: reinforce schema representation with context
In this work, we present X-SQL, a new network architecture for the problem of parsing natural language to SQL query.
On the Variance of the Adaptive Learning Rate and Beyond
The learning rate warmup heuristic achieves remarkable success in stabilizing training, accelerating convergence and improving generalization for adaptive stochastic optimization algorithms like RMSprop and Adam.
A Hybrid Neural Network Model for Commonsense Reasoning
An HNN consists of two component models, a masked language model and a semantic similarity model, which share a BERT-based contextual encoder but use different model-specific input and output layers.
Ranked #1 on
Natural Language Understanding
on PDP60
Lessons from Contextual Bandit Learning in a Customer Support Bot
In this work, we describe practical lessons we have learned from successfully using contextual bandits (CBs) to improve key business metrics of the Microsoft Virtual Agent for customer support.
Zero-training Sentence Embedding via Orthogonal Basis
We model the semantic meaning of a word in a sentence based on two aspects.
Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
This paper explores the use of knowledge distillation to improve a Multi-Task Deep Neural Network (MT-DNN) (Liu et al., 2019) for learning text representations across multiple natural language understanding tasks.
Ranked #1 on
Semantic Textual Similarity
on SentEval
Multi-Task Deep Neural Networks for Natural Language Understanding
In this paper, we present a Multi-Task Deep Neural Network (MT-DNN) for learning representations across multiple natural language understanding (NLU) tasks.
Ranked #2 on
Natural Language Inference
on SciTail
Parameter-free Sentence Embedding via Orthogonal Basis
Inspired by the Gram-Schmidt Process in geometric theory, we build an orthogonal basis of the subspace spanned by a word and its surrounding context in a sentence.
IncSQL: Training Incremental Text-to-SQL Parsers with Non-Deterministic Oracles
We present a sequence-to-action parsing approach for the natural language to SQL task that incrementally fills the slots of a SQL query with feasible actions from a pre-defined inventory.
Learning to Attend On Essential Terms: An Enhanced Retriever-Reader Model for Open-domain Question Answering
In this paper we propose a retriever-reader model that learns to attend on essential terms during the question answering process.
Multiple-choice
Multiple Choice Question Answering (MCQA)
+2
FusionNet: Fusing via Fully-Aware Attention with Application to Machine Comprehension
This paper introduces a new neural structure called FusionNet, which extends existing attention approaches from three perspectives.
Ranked #26 on
Question Answering
on SQuAD1.1 dev
DSCOVR: Randomized Primal-Dual Block Coordinate Algorithms for Asynchronous Distributed Optimization
Machine learning with big data often involves large optimization models.
ReasoNet: Learning to Stop Reading in Machine Comprehension
Teaching a computer to read and answer general questions pertaining to a document is a challenging yet unsolved problem.
Ranked #7 on
Question Answering
on CNN / Daily Mail
Large-scale L-BFGS using MapReduce
L-BFGS has been applied as an effective parameter estimation method for various machine learning algorithms since 1980s.
