Search Results for author:
Found 39 papers, 24 papers with code
Continual Learning with Dynamic Sparse Training: Exploring Algorithms for Effective Model Updates
Therefore, we perform a comprehensive study in which we investigate various DST components to find the best topology per task on well-known CIFAR100 and miniImageNet benchmarks in a task-incremental CL setup since our primary focus is to evaluate the performance of various DST criteria, rather than the process of mask selection.
Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Training
The key components of this framework are the pruning and growing criteria, which are repeatedly applied during the training process to adjust the network's sparse connectivity.
Adaptive Sparsity Level during Training for Efficient Time Series Forecasting with Transformers
Efficiency in DNNs can be achieved through sparse connectivity and reducing the model size.
Supervised Feature Selection with Neuron Evolution in Sparse Neural Networks
Feature selection that selects an informative subset of variables from data not only enhances the model interpretability and performance but also alleviates the resource demands.
Automatic Noise Filtering with Dynamic Sparse Training in Deep Reinforcement Learning
Tomorrow's robots will need to distinguish useful information from noise when performing different tasks.
Dynamic Sparse Network for Time Series Classification: Learning What to "see''
The receptive field (RF), which determines the region of time series to be ``seen'' and used, is critical to improve the performance for time series classification (TSC).
You Can Have Better Graph Neural Networks by Not Training Weights at All: Finding Untrained GNNs Tickets
Recent works have impressively demonstrated that there exists a subnetwork in randomly initialized convolutional neural networks (CNNs) that can match the performance of the fully trained dense networks at initialization, without any optimization of the weights of the network (i. e., untrained networks).
Where to Pay Attention in Sparse Training for Feature Selection?
Our proposed approach outperforms the state-of-the-art methods in terms of selecting informative features while reducing training iterations and computational costs substantially.
Memory-free Online Change-point Detection: A Novel Neural Network Approach
We show that ALACPD, on average, ranks first among state-of-the-art CPD algorithms in terms of quality of the time series segmentation, and it is on par with the best performer in terms of the accuracy of the estimated change-points.
Superposing Many Tickets into One: A Performance Booster for Sparse Neural Network Training
Recent works on sparse neural network training (sparse training) have shown that a compelling trade-off between performance and efficiency can be achieved by training intrinsically sparse neural networks from scratch.
The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training
In this paper, we focus on sparse training and highlight a perhaps counter-intuitive finding, that random pruning at initialization can be quite powerful for the sparse training of modern neural networks.
Avoiding Forgetting and Allowing Forward Transfer in Continual Learning via Sparse Networks
To address this challenge, we propose a new CL method, named AFAF, that aims to Avoid Forgetting and Allow Forward transfer in class-IL using fix-capacity models.
Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity
Our framework, FreeTickets, is defined as the ensemble of these relatively cheap sparse subnetworks.
Sparse Training via Boosting Pruning Plasticity with Neuroregeneration
Works on lottery ticket hypothesis (LTH) and single-shot network pruning (SNIP) have raised a lot of attention currently on post-training pruning (iterative magnitude pruning), and before-training pruning (pruning at initialization).
Ranked #3 on
Sparse Learning
on ImageNet
Dynamic Sparse Training for Deep Reinforcement Learning
In this paper, we introduce for the first time a dynamic sparse training approach for deep reinforcement learning to accelerate the training process.
Sparse Training Theory for Scalable and Efficient Agents
A fundamental task for artificial intelligence is learning.
Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training
By starting from a random sparse network and continuously exploring sparse connectivities during training, we can perform an Over-Parameterization in the space-time manifold, closing the gap in the expressibility between sparse training and dense training.
Ranked #4 on
Sparse Learning
on ImageNet
Truly Sparse Neural Networks at Scale
Recently, sparse training methods have started to be established as a de facto approach for training and inference efficiency in artificial neural networks.
Self-Attention Meta-Learner for Continual Learning
In this paper, we propose a new method, named Self-Attention Meta-Learner (SAM), which learns a prior knowledge for continual learning that permits learning a sequence of tasks, while avoiding catastrophic forgetting.
Selfish Sparse RNN Training
Sparse neural networks have been widely applied to reduce the computational demands of training and deploying over-parameterized deep neural networks.
Learning Invariant Representation for Continual Learning
Finally, we analyze the role of the shared invariant representation in mitigating the forgetting problem especially when the number of replayed samples for each previous task is small.
Quick and Robust Feature Selection: the Strength of Energy-efficient Sparse Training for Autoencoders
This method, named QuickSelection, introduces the strength of the neuron in sparse neural networks as a criterion to measure the feature importance.
SpaceNet: Make Free Space For Continual Learning
Regularization-based methods maintain a fixed model capacity; however, previous studies showed the huge performance degradation of these methods when the task identity is not available during inference (e. g. class incremental learning scenario).
Topological Insights into Sparse Neural Networks
However, comparing different sparse topologies and determining how sparse topologies evolve during training, especially for the situation in which the sparse structure optimization is involved, remain as challenging open questions.
Novelty Producing Synaptic Plasticity
A learning process with the plasticity property often requires reinforcement signals to guide the process.
On improving deep learning generalization with adaptive sparse connectivity
Large neural networks are very successful in various tasks.
Evolving Plasticity for Autonomous Learning under Changing Environmental Conditions
Hebbian learning is a biologically plausible mechanism for modeling the plasticity property in artificial neural networks (ANNs), based on the local interactions of neurons.
Learning with Delayed Synaptic Plasticity
Inspired by biology, plasticity can be modeled in artificial neural networks by using Hebbian learning rules, i. e. rules that update synapses based on the neuron activations and reinforcement signals.
A Brain-inspired Algorithm for Training Highly Sparse Neural Networks
Concretely, by exploiting the cosine similarity metric to measure the importance of the connections, our proposed method, Cosine similarity-based and Random Topology Exploration (CTRE), evolves the topology of sparse neural networks by adding the most important connections to the network without calculating dense gradient in the backward.
Intrinsically Sparse Long Short-Term Memory Networks
However, LSTMs are prone to be memory-bandwidth limited in realistic applications and need an unbearable period of training and inference time as the model size is ever-increasing.
Sparse evolutionary Deep Learning with over one million artificial neurons on commodity hardware
Despite the success of ANNs, it is challenging to train and deploy modern ANNs on commodity hardware due to the ever-increasing model size and the unprecedented growth in the data volumes.
Limited Evaluation Cooperative Co-evolutionary Differential Evolution for Large-scale Neuroevolution
Many real-world control and classification tasks involve a large number of features.
One-Shot Learning using Mixture of Variational Autoencoders: a Generalization Learning approach
In an attempt to solve this problem, the one-shot learning paradigm, which makes use of just one labeled sample per class and prior knowledge, becomes increasingly important.
On-line Building Energy Optimization using Deep Reinforcement Learning
Unprecedented high volumes of data are becoming available with the growth of the advanced metering infrastructure.
Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science
Through the success of deep learning in various domains, artificial neural networks are currently among the most used artificial intelligence methods.
Online Contrastive Divergence with Generative Replay: Experience Replay without Storing Data
Conceived in the early 1990s, Experience Replay (ER) has been shown to be a successful mechanism to allow online learning algorithms to reuse past experiences.
Predictive No-Reference Assessment of Video Quality
Among the various means to evaluate the quality of video streams, No-Reference (NR) methods have low computation and may be executed on thin clients.
A topological insight into restricted Boltzmann machines
Thirdly, we show that, for a fixed number of weights, our proposed sparse models (which by design have a higher number of hidden neurons) achieve better generative capabilities than standard fully connected RBMs and GRBMs (which by design have a smaller number of hidden neurons), at no additional computational costs.
Estimating 3D Trajectories from 2D Projections via Disjunctive Factored Four-Way Conditional Restricted Boltzmann Machines
Estimation, recognition, and near-future prediction of 3D trajectories based on their two dimensional projections available from one camera source is an exceptionally difficult problem due to uncertainty in the trajectories and environment, high dimensionality of the specific trajectory states, lack of enough labeled data and so on.
