Search Results for author:
Found 22 papers, 10 papers with code
Sequence-level Large Language Model Training with Contrastive Preference Optimization
The next token prediction loss is the dominant self-supervised training objective for large language models and has achieved promising results in a variety of downstream tasks.
Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruning
Sparse Mixture of Expert (SMoE) models have emerged as a scalable alternative to dense models in language modeling.
DEM: Distribution Edited Model for Training with Mixed Data Distributions
Training with mixed data distributions is a common and important part of creating multi-task and instruction-following models.
Pre-training Differentially Private Models with Limited Public Data
The superior performance of large foundation models relies on the use of massive amounts of high-quality data, which often contain sensitive, private and copyrighted material that requires formal protection.
Extreme Miscalibration and the Illusion of Adversarial Robustness
Deep learning-based Natural Language Processing (NLP) models are vulnerable to adversarial attacks, where small perturbations can cause a model to misclassify.
Zero redundancy distributed learning with differential privacy
Deep learning using large models have achieved great success in a wide range of domains.
On the accuracy and efficiency of group-wise clipping in differentially private optimization
Recent advances have substantially improved the accuracy, memory cost, and training speed of differentially private (DP) deep learning, especially on large vision and language models with millions to billions of parameters.
Efficient Long-Range Transformers: You Need to Attend More, but Not Necessarily at Every Layer
These models leverage the attention mechanism to capture long- and short-range dependencies in the sequence.
Coupling public and private gradient provably helps optimization
The success of large neural networks is crucially determined by the availability of data.
HYTREL: Hypergraph-enhanced Tabular Data Representation Learning
Language models pretrained on large collections of tabular data have demonstrated their effectiveness in several downstream tasks.
Large Language Models of Code Fail at Completing Code with Potential Bugs
We find that the presence of potential bugs significantly degrades the generation performance of the high-performing Code-LLMs.
Better Context Makes Better Code Language Models: A Case Study on Function Call Argument Completion
To better solve this task, we query a program analyzer for information relevant to a given function call, and consider ways to provide the analyzer results to different code completion models during inference and training.
Parameter and Data Efficient Continual Pre-training for Robustness to Dialectal Variance in Arabic
While prior research studies have tried to adapt these multilingual models for dialectal variants of Arabic, it still remains a challenging problem owing to the lack of sufficient monolingual dialectal data and parallel translation data of such dialectal variants.
Differentially Private Optimization on Large Model at Small Cost
Our implementation achieves state-of-the-art (SOTA) accuracy with very small extra cost: on GPT2 and at almost the same memory cost (<1% overhead), BK has 1. 03X the time complexity of the standard training (0. 83X training speed in practice), and 0. 61X the time complexity of the most efficient DP implementation (1. 36X training speed in practice).
Differentially Private Bias-Term Fine-tuning of Foundation Models
We study the problem of differentially private (DP) fine-tuning of large pre-trained models -- a recent privacy-preserving approach suitable for solving downstream tasks with sensitive data.
Exploring the Role of Task Transferability in Large-Scale Multi-Task Learning
Recent work has found that multi-task training with a large number of diverse tasks can uniformly improve downstream performance on unseen target tasks.
Meta-learning via Language Model In-context Tuning
The goal of meta-learning is to learn to adapt to a new task with only a few labeled examples.
Accelerated Large Batch Optimization of BERT Pretraining in 54 minutes
Using the proposed LANS method and the learning rate scheme, we scaled up the mini-batch sizes to 96K and 33K in phases 1 and 2 of BERT pretraining, respectively.
Unlearn Dataset Bias in Natural Language Inference by Fitting the Residual
We first learn a biased model that only uses features that are known to relate to dataset bias.
GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processing
We present GluonCV and GluonNLP, the deep learning toolkits for computer vision and natural language processing based on Apache MXNet (incubating).
Dynamic Mini-batch SGD for Elastic Distributed Training: Learning in the Limbo of Resources
One difficulty we observe is that the noise in the stochastic momentum estimation is accumulated over time and will have delayed effects when the batch size changes.
Question Type Guided Attention in Visual Question Answering
Visual Question Answering (VQA) requires integration of feature maps with drastically different structures and focus of the correct regions.
