Search Results for author:
Found 12 papers, 5 papers with code
Sequence-level Large Language Model Training with Contrastive Preference Optimization
The next token prediction loss is the dominant self-supervised training objective for large language models and has achieved promising results in a variety of downstream tasks.
DEM: Distribution Edited Model for Training with Mixed Data Distributions
Training with mixed data distributions is a common and important part of creating multi-task and instruction-following models.
Extreme Miscalibration and the Illusion of Adversarial Robustness
Deep learning-based Natural Language Processing (NLP) models are vulnerable to adversarial attacks, where small perturbations can cause a model to misclassify.
Continual Learning with Transformers for Image Classification
This phenomenon is known as catastrophic forgetting and it is often difficult to prevent due to practical constraints, such as the amount of data that can be stored or the limited computation sources that can be used.
Memory Efficient Continual Learning with Transformers
Moreover, applications increasingly rely on large pre-trained neural networks, such as pre-trained Transformers, since the resources or data might not be available in sufficiently large quantities to practitioners to train the model from scratch.
Synthetic Petri Dish: A Novel Surrogate Model for Rapid Architecture Search
Neural Architecture Search (NAS) explores a large space of architectural motifs -- a compute-intensive process that often involves ground-truth evaluation of each motif by instantiating it within a large network, and training and evaluating the network with thousands of domain-specific data samples.
Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions
Creating open-ended algorithms, which generate their own never-ending stream of novel and appropriately challenging learning opportunities, could help to automate and accelerate progress in machine learning.
Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity
We show that neuromodulated plasticity improves the performance of neural networks on both reinforcement learning and supervised learning tasks.
Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data
This paper introduces GTNs, discusses their potential, and showcases that they can substantially accelerate learning.
First-Order Preconditioning via Hypergradient Descent
Standard gradient descent methods are susceptible to a range of issues that can impede training, such as high correlations and different scaling in parameter space. These difficulties can be addressed by second-order approaches that apply a pre-conditioning matrix to the gradient to improve convergence.
From Nodes to Networks: Evolving Recurrent Neural Networks
Gated recurrent networks such as those composed of Long Short-Term Memory (LSTM) nodes have recently been used to improve state of the art in many sequential processing tasks such as speech recognition and machine translation.
Evolving Deep Neural Networks
The success of deep learning depends on finding an architecture to fit the task.
