Search Results for author:
Found 23 papers, 8 papers with code
Simple and Scalable Strategies to Continually Pre-train Large Language Models
In this work, we show that a simple and scalable combination of learning rate (LR) re-warming, LR re-decaying, and replay of previous data is sufficient to match the performance of fully re-training from scratch on all available data, as measured by the final loss and the average score on several language model (LM) evaluation benchmarks.
Continual Learning Under Language Shift
The recent increase in data and model scale for language model pre-training has led to huge training costs.
Continual Pre-Training of Large Language Models: How to (re)warm your model?
We study the warmup phase of models pre-trained on the Pile (upstream data, 300B tokens) as we continue to pre-train on SlimPajama (downstream data, 297B tokens), following a linear warmup and cosine decay schedule.
Beyond Supervised Continual Learning: a Review
Continual Learning (CL, sometimes also termed incremental learning) is a flavor of machine learning where the usual assumption of stationary data distribution is relaxed or omitted.
Challenging Common Assumptions about Catastrophic Forgetting
In this paper, we study the progressive knowledge accumulation (KA) in DNNs trained with gradient-based algorithms in long sequences of tasks with data re-occurrence.
Continual Feature Selection: Spurious Features in Continual Learning
Our experiments highlight that continual learning algorithms face two related problems: (1) spurious features and (2) local spurious features.
Sequoia: A Software Framework to Unify Continual Learning Research
We propose a taxonomy of settings, where each setting is described as a set of assumptions.
Continual Learning in Deep Networks: an Analysis of the Last Layer
Our analysis and results shed light on the dynamics of the output layer in continual learning scenarios and suggest a way of selecting the best type of output layer for a given scenario.
Understanding Continual Learning Settings with Data Distribution Drift Analysis
In this paper, we aim to identify and categorize different types of context drifts and potential assumptions about them, to better characterize various continual-learning scenarios.
Continuum: Simple Management of Complex Continual Learning Scenarios
Those drifts might cause interferences in the trained model and knowledge learned on previous states of the data distribution might be forgotten.
Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processes
The replay processes make possible a compromise between optimizing the current learning objective and the past ones enabling learning without forgetting in sequences of tasks settings.
Regularization Shortcomings for Continual Learning
In most machine learning algorithms, training data is assumed to be independent and identically distributed (iid).
DisCoRL: Continual Reinforcement Learning via Policy Distillation
In multi-task reinforcement learning there are two main challenges: at training time, the ability to learn different policies with a single model; at test time, inferring which of those policies applying without an external signal.
Continual Learning for Robotics: Definition, Framework, Learning Strategies, Opportunities and Challenges
An important challenge for machine learning is not necessarily finding solutions that work in the real world but rather finding stable algorithms that can learn in real world.
Continual Reinforcement Learning deployed in Real-life using Policy Distillation and Sim2Real Transfer
We focus on the problem of teaching a robot to solve tasks presented sequentially, i. e., in a continual learning scenario.
Decoupling feature extraction from policy learning: assessing benefits of state representation learning in goal based robotics
Scaling end-to-end reinforcement learning to control real robots from vision presents a series of challenges, in particular in terms of sample efficiency.
Generative Models from the perspective of Continual Learning
We experiment with sequential tasks on three commonly used benchmarks for Continual Learning (MNIST, Fashion MNIST and CIFAR10).
Marginal Replay vs Conditional Replay for Continual Learning
We present a new replay-based method of continual classification learning that we term "conditional replay" which generates samples and labels together by sampling from a distribution conditioned on the class.
S-RL Toolbox: Environments, Datasets and Evaluation Metrics for State Representation Learning
State representation learning aims at learning compact representations from raw observations in robotics and control applications.
Training Discriminative Models to Evaluate Generative Ones
By comparing results with different generated datasets we are able to classify and compare generative models.
State Representation Learning for Control: An Overview
State representation learning (SRL) focuses on a particular kind of representation learning where learned features are in low dimension, evolve through time, and are influenced by actions of an agent.
Evaluation of generative networks through their data augmentation capacity
This mixture of real and generated data is thus used to train a classifier which is afterwards tested on a given labeled test dataset.
Unsupervised state representation learning with robotic priors: a robustness benchmark
We reproduce this simplification process using a neural network to build a low dimensional state representation of the world from images acquired by a robot.
