Search Results for author: Decebal Constantin Mocanu

Found 39 papers, 24 papers with code

Continual Learning with Dynamic Sparse Training: Exploring Algorithms for Effective Model Updates

no code implementations28 Aug 2023 Murat Onur Yildirim, Elif Ceren Gok Yildirim, Ghada Sokar, Decebal Constantin Mocanu, Joaquin Vanschoren

Therefore, we perform a comprehensive study in which we investigate various DST components to find the best topology per task on well-known CIFAR100 and miniImageNet benchmarks in a task-incremental CL setup since our primary focus is to evaluate the performance of various DST criteria, rather than the process of mask selection.

Continual Learning

Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Training

1 code implementation NeurIPS 2023 Aleksandra I. Nowak, Bram Grooten, Decebal Constantin Mocanu, Jacek Tabor

The key components of this framework are the pruning and growing criteria, which are repeatedly applied during the training process to adjust the network's sparse connectivity.

Supervised Feature Selection with Neuron Evolution in Sparse Neural Networks

1 code implementation10 Mar 2023 Zahra Atashgahi, Xuhao Zhang, Neil Kichler, Shiwei Liu, Lu Yin, Mykola Pechenizkiy, Raymond Veldhuis, Decebal Constantin Mocanu

Feature selection that selects an informative subset of variables from data not only enhances the model interpretability and performance but also alleviates the resource demands.

feature selection

Dynamic Sparse Network for Time Series Classification: Learning What to "see''

1 code implementation19 Dec 2022 Qiao Xiao, Boqian Wu, Yu Zhang, Shiwei Liu, Mykola Pechenizkiy, Elena Mocanu, Decebal Constantin Mocanu

The receptive field (RF), which determines the region of time series to be ``seen'' and used, is critical to improve the performance for time series classification (TSC).

Time Series Time Series Analysis +1

You Can Have Better Graph Neural Networks by Not Training Weights at All: Finding Untrained GNNs Tickets

1 code implementation28 Nov 2022 Tianjin Huang, Tianlong Chen, Meng Fang, Vlado Menkovski, Jiaxu Zhao, Lu Yin, Yulong Pei, Decebal Constantin Mocanu, Zhangyang Wang, Mykola Pechenizkiy, Shiwei Liu

Recent works have impressively demonstrated that there exists a subnetwork in randomly initialized convolutional neural networks (CNNs) that can match the performance of the fully trained dense networks at initialization, without any optimization of the weights of the network (i. e., untrained networks).

Out-of-Distribution Detection

Where to Pay Attention in Sparse Training for Feature Selection?

1 code implementation26 Nov 2022 Ghada Sokar, Zahra Atashgahi, Mykola Pechenizkiy, Decebal Constantin Mocanu

Our proposed approach outperforms the state-of-the-art methods in terms of selecting informative features while reducing training iterations and computational costs substantially.

feature selection

Memory-free Online Change-point Detection: A Novel Neural Network Approach

1 code implementation8 Jul 2022 Zahra Atashgahi, Decebal Constantin Mocanu, Raymond Veldhuis, Mykola Pechenizkiy

We show that ALACPD, on average, ranks first among state-of-the-art CPD algorithms in terms of quality of the time series segmentation, and it is on par with the best performer in terms of the accuracy of the estimated change-points.

Change Point Detection Time Series +1

Superposing Many Tickets into One: A Performance Booster for Sparse Neural Network Training

no code implementations30 May 2022 Lu Yin, Vlado Menkovski, Meng Fang, Tianjin Huang, Yulong Pei, Mykola Pechenizkiy, Decebal Constantin Mocanu, Shiwei Liu

Recent works on sparse neural network training (sparse training) have shown that a compelling trade-off between performance and efficiency can be achieved by training intrinsically sparse neural networks from scratch.

The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training

1 code implementation ICLR 2022 Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, Mykola Pechenizkiy

In this paper, we focus on sparse training and highlight a perhaps counter-intuitive finding, that random pruning at initialization can be quite powerful for the sparse training of modern neural networks.

Adversarial Robustness Out-of-Distribution Detection

Avoiding Forgetting and Allowing Forward Transfer in Continual Learning via Sparse Networks

1 code implementation11 Oct 2021 Ghada Sokar, Decebal Constantin Mocanu, Mykola Pechenizkiy

To address this challenge, we propose a new CL method, named AFAF, that aims to Avoid Forgetting and Allow Forward transfer in class-IL using fix-capacity models.

Class Incremental Learning Incremental Learning +2

Sparse Training via Boosting Pruning Plasticity with Neuroregeneration

2 code implementations NeurIPS 2021 Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, Decebal Constantin Mocanu

Works on lottery ticket hypothesis (LTH) and single-shot network pruning (SNIP) have raised a lot of attention currently on post-training pruning (iterative magnitude pruning), and before-training pruning (pruning at initialization).

Network Pruning Sparse Learning

Dynamic Sparse Training for Deep Reinforcement Learning

1 code implementation8 Jun 2021 Ghada Sokar, Elena Mocanu, Decebal Constantin Mocanu, Mykola Pechenizkiy, Peter Stone

In this paper, we introduce for the first time a dynamic sparse training approach for deep reinforcement learning to accelerate the training process.

Continuous Control Decision Making +3

Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training

3 code implementations4 Feb 2021 Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, Mykola Pechenizkiy

By starting from a random sparse network and continuously exploring sparse connectivities during training, we can perform an Over-Parameterization in the space-time manifold, closing the gap in the expressibility between sparse training and dense training.

Image Classification Sparse Learning

Truly Sparse Neural Networks at Scale

1 code implementation2 Feb 2021 Selima Curci, Decebal Constantin Mocanu, Mykola Pechenizkiyi

Recently, sparse training methods have started to be established as a de facto approach for training and inference efficiency in artificial neural networks.

Self-Attention Meta-Learner for Continual Learning

1 code implementation28 Jan 2021 Ghada Sokar, Decebal Constantin Mocanu, Mykola Pechenizkiy

In this paper, we propose a new method, named Self-Attention Meta-Learner (SAM), which learns a prior knowledge for continual learning that permits learning a sequence of tasks, while avoiding catastrophic forgetting.

Continual Learning Split-CIFAR-10 +1

Selfish Sparse RNN Training

1 code implementation22 Jan 2021 Shiwei Liu, Decebal Constantin Mocanu, Yulong Pei, Mykola Pechenizkiy

Sparse neural networks have been widely applied to reduce the computational demands of training and deploying over-parameterized deep neural networks.

Learning Invariant Representation for Continual Learning

1 code implementation15 Jan 2021 Ghada Sokar, Decebal Constantin Mocanu, Mykola Pechenizkiy

Finally, we analyze the role of the shared invariant representation in mitigating the forgetting problem especially when the number of replayed samples for each previous task is small.

Class Incremental Learning Incremental Learning +2

SpaceNet: Make Free Space For Continual Learning

1 code implementation15 Jul 2020 Ghada Sokar, Decebal Constantin Mocanu, Mykola Pechenizkiy

Regularization-based methods maintain a fixed model capacity; however, previous studies showed the huge performance degradation of these methods when the task identity is not available during inference (e. g. class incremental learning scenario).

Class Incremental Learning Incremental Learning +1

Topological Insights into Sparse Neural Networks

3 code implementations24 Jun 2020 Shiwei Liu, Tim Van der Lee, Anil Yaman, Zahra Atashgahi, Davide Ferraro, Ghada Sokar, Mykola Pechenizkiy, Decebal Constantin Mocanu

However, comparing different sparse topologies and determining how sparse topologies evolve during training, especially for the situation in which the sparse structure optimization is involved, remain as challenging open questions.

Novelty Producing Synaptic Plasticity

no code implementations10 Feb 2020 Anil Yaman, Giovanni Iacca, Decebal Constantin Mocanu, George Fletcher, Mykola Pechenizkiy

A learning process with the plasticity property often requires reinforcement signals to guide the process.

Evolving Plasticity for Autonomous Learning under Changing Environmental Conditions

no code implementations2 Apr 2019 Anil Yaman, Giovanni Iacca, Decebal Constantin Mocanu, Matt Coler, George Fletcher, Mykola Pechenizkiy

Hebbian learning is a biologically plausible mechanism for modeling the plasticity property in artificial neural networks (ANNs), based on the local interactions of neurons.

Learning with Delayed Synaptic Plasticity

no code implementations22 Mar 2019 Anil Yaman, Giovanni Iacca, Decebal Constantin Mocanu, George Fletcher, Mykola Pechenizkiy

Inspired by biology, plasticity can be modeled in artificial neural networks by using Hebbian learning rules, i. e. rules that update synapses based on the neuron activations and reinforcement signals.

A Brain-inspired Algorithm for Training Highly Sparse Neural Networks

2 code implementations17 Mar 2019 Zahra Atashgahi, Joost Pieterse, Shiwei Liu, Decebal Constantin Mocanu, Raymond Veldhuis, Mykola Pechenizkiy

Concretely, by exploiting the cosine similarity metric to measure the importance of the connections, our proposed method, Cosine similarity-based and Random Topology Exploration (CTRE), evolves the topology of sparse neural networks by adding the most important connections to the network without calculating dense gradient in the backward.

Learning Theory

Intrinsically Sparse Long Short-Term Memory Networks

no code implementations26 Jan 2019 Shiwei Liu, Decebal Constantin Mocanu, Mykola Pechenizkiy

However, LSTMs are prone to be memory-bandwidth limited in realistic applications and need an unbearable period of training and inference time as the model size is ever-increasing.

Model Compression Sentiment Analysis

Sparse evolutionary Deep Learning with over one million artificial neurons on commodity hardware

4 code implementations26 Jan 2019 Shiwei Liu, Decebal Constantin Mocanu, Amarsagar Reddy Ramapuram Matavalam, Yulong Pei, Mykola Pechenizkiy

Despite the success of ANNs, it is challenging to train and deploy modern ANNs on commodity hardware due to the ever-increasing model size and the unprecedented growth in the data volumes.

One-Shot Learning using Mixture of Variational Autoencoders: a Generalization Learning approach

no code implementations18 Apr 2018 Decebal Constantin Mocanu, Elena Mocanu

In an attempt to solve this problem, the one-shot learning paradigm, which makes use of just one labeled sample per class and prior knowledge, becomes increasingly important.

Classification General Classification +1

Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science

2 code implementations15 Jul 2017 Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H. Nguyen, Madeleine Gibescu, Antonio Liotta

Through the success of deep learning in various domains, artificial neural networks are currently among the most used artificial intelligence methods.

Online Contrastive Divergence with Generative Replay: Experience Replay without Storing Data

no code implementations18 Oct 2016 Decebal Constantin Mocanu, Maria Torres Vega, Eric Eaton, Peter Stone, Antonio Liotta

Conceived in the early 1990s, Experience Replay (ER) has been shown to be a successful mechanism to allow online learning algorithms to reuse past experiences.

reinforcement-learning Reinforcement Learning (RL)

Predictive No-Reference Assessment of Video Quality

no code implementations25 Apr 2016 Maria Torres Vega, Decebal Constantin Mocanu, Antonio Liotta

Among the various means to evaluate the quality of video streams, No-Reference (NR) methods have low computation and may be executed on thin clients.

BIG-bench Machine Learning

A topological insight into restricted Boltzmann machines

no code implementations20 Apr 2016 Decebal Constantin Mocanu, Elena Mocanu, Phuong H. Nguyen, Madeleine Gibescu, Antonio Liotta

Thirdly, we show that, for a fixed number of weights, our proposed sparse models (which by design have a higher number of hidden neurons) achieve better generative capabilities than standard fully connected RBMs and GRBMs (which by design have a smaller number of hidden neurons), at no additional computational costs.

Estimating 3D Trajectories from 2D Projections via Disjunctive Factored Four-Way Conditional Restricted Boltzmann Machines

no code implementations20 Apr 2016 Decebal Constantin Mocanu, Haitham Bou Ammar, Luis Puig, Eric Eaton, Antonio Liotta

Estimation, recognition, and near-future prediction of 3D trajectories based on their two dimensional projections available from one camera source is an exceptionally difficult problem due to uncertainty in the trajectories and environment, high dimensionality of the specific trajectory states, lack of enough labeled data and so on.

Future prediction Time Series +1

Cannot find the paper you are looking for? You can Submit a new open access paper.