Search Results for author: Mykola Pechenizkiy

Found 74 papers, 37 papers with code

Are Large Kernels Better Teachers than Transformers for ConvNets?

1 code implementation30 May 2023 Tianjin Huang, Lu Yin, Zhenyu Zhang, Li Shen, Meng Fang, Mykola Pechenizkiy, Zhangyang Wang, Shiwei Liu

We hereby carry out a first-of-its-kind study unveiling that modern large-kernel ConvNets, a compelling competitor to Vision Transformers, are remarkably more effective teachers for small-kernel ConvNets, due to more similar architectures.

Knowledge Distillation

Dynamic Sparsity Is Channel-Level Sparsity Learner

no code implementations30 May 2023 Lu Yin, Gen Li, Meng Fang, Li Shen, Tianjin Huang, Zhangyang Wang, Vlado Menkovski, Xiaolong Ma, Mykola Pechenizkiy, Shiwei Liu

Dynamic sparse training (DST), as a leading sparse training approach, can train deep neural networks at high sparsity from scratch to match the performance of their dense counterparts.

GRD: A Generative Approach for Interpretable Reward Redistribution in Reinforcement Learning

no code implementations28 May 2023 Yudi Zhang, Yali Du, Biwei Huang, Ziyan Wang, Jun Wang, Meng Fang, Mykola Pechenizkiy

While the majority of current approaches construct the reward redistribution in an uninterpretable manner, we propose to explicitly model the contributions of state and action from a causal perspective, resulting in an interpretable return decomposition.


NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklist

7 code implementations15 May 2023 Iftitahu Ni'mah, Meng Fang, Vlado Menkovski, Mykola Pechenizkiy

Our proposed framework provides access: (i) for verifying whether automatic metrics are faithful to human preference, regardless of their correlation level to human; and (ii) for inspecting the strengths and limitations of NLG systems via pairwise evaluation.

Controllable Language Modelling Dialogue Generation +2

Algorithmic Unfairness through the Lens of EU Non-Discrimination Law: Or Why the Law is not a Decision Tree

no code implementations5 May 2023 Hilde Weerts, Raphaële Xenidis, Fabien Tarissan, Henrik Palmer Olsen, Mykola Pechenizkiy

In this paper, we aim to illustrate to what extent European Union (EU) non-discrimination law coincides with notions of algorithmic fairness proposed in computer science literature and where they differ.


Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoML

no code implementations15 Mar 2023 Hilde Weerts, Florian Pfisterer, Matthias Feurer, Katharina Eggensperger, Edward Bergman, Noor Awad, Joaquin Vanschoren, Mykola Pechenizkiy, Bernd Bischl, Frank Hutter

The field of automated machine learning (AutoML) introduces techniques that automate parts of the development of machine learning (ML) systems, accelerating the process and reducing barriers for novices.

AutoML Fairness

Supervised Feature Selection with Neuron Evolution in Sparse Neural Networks

1 code implementation10 Mar 2023 Zahra Atashgahi, Xuhao Zhang, Neil Kichler, Shiwei Liu, Lu Yin, Mykola Pechenizkiy, Raymond Veldhuis, Decebal Constantin Mocanu

Feature selection that selects an informative subset of variables from data not only enhances the model interpretability and performance but also alleviates the resource demands.

feature selection

Dynamic Sparse Network for Time Series Classification: Learning What to "see''

1 code implementation19 Dec 2022 Qiao Xiao, Boqian Wu, Yu Zhang, Shiwei Liu, Mykola Pechenizkiy, Elena Mocanu, Decebal Constantin Mocanu

The receptive field (RF), which determines the region of time series to be ``seen'' and used, is critical to improve the performance for time series classification (TSC).

Time Series Analysis Time Series Classification

You Can Have Better Graph Neural Networks by Not Training Weights at All: Finding Untrained GNNs Tickets

1 code implementation28 Nov 2022 Tianjin Huang, Tianlong Chen, Meng Fang, Vlado Menkovski, Jiaxu Zhao, Lu Yin, Yulong Pei, Decebal Constantin Mocanu, Zhangyang Wang, Mykola Pechenizkiy, Shiwei Liu

Recent works have impressively demonstrated that there exists a subnetwork in randomly initialized convolutional neural networks (CNNs) that can match the performance of the fully trained dense networks at initialization, without any optimization of the weights of the network (i. e., untrained networks).

Out-of-Distribution Detection

Where to Pay Attention in Sparse Training for Feature Selection?

1 code implementation26 Nov 2022 Ghada Sokar, Zahra Atashgahi, Mykola Pechenizkiy, Decebal Constantin Mocanu

Our proposed approach outperforms the state-of-the-art methods in terms of selecting informative features while reducing training iterations and computational costs substantially.

feature selection

FAL-CUR: Fair Active Learning using Uncertainty and Representativeness on Fair Clustering

no code implementations21 Sep 2022 Ricky Fajri, Akrati Saxena, Yulong Pei, Mykola Pechenizkiy

This paper presents a novel active learning strategy called Fair Active Learning using fair Clustering, Uncertainty, and Representativeness (FAL-CUR) that provides a high accuracy while maintaining fairness during the sample acquisition phase.

Active Learning Fairness

Memory-free Online Change-point Detection: A Novel Neural Network Approach

1 code implementation8 Jul 2022 Zahra Atashgahi, Decebal Constantin Mocanu, Raymond Veldhuis, Mykola Pechenizkiy

We show that ALACPD, on average, ranks first among state-of-the-art CPD algorithms in terms of quality of the time series segmentation, and it is on par with the best performer in terms of the accuracy of the estimated change-points.

Change Point Detection Time Series Analysis

Superposing Many Tickets into One: A Performance Booster for Sparse Neural Network Training

no code implementations30 May 2022 Lu Yin, Vlado Menkovski, Meng Fang, Tianjin Huang, Yulong Pei, Mykola Pechenizkiy, Decebal Constantin Mocanu, Shiwei Liu

Recent works on sparse neural network training (sparse training) have shown that a compelling trade-off between performance and efficiency can be achieved by training intrinsically sparse neural networks from scratch.

Phrase-level Textual Adversarial Attack with Label Preservation

1 code implementation Findings (NAACL) 2022 Yibin Lei, Yu Cao, Dianqi Li, Tianyi Zhou, Meng Fang, Mykola Pechenizkiy

Generating high-quality textual adversarial examples is critical for investigating the pitfalls of natural language processing (NLP) models and further promoting their robustness.

Adversarial Attack

Survey on Fair Reinforcement Learning: Theory and Practice

no code implementations20 May 2022 Pratik Gajane, Akrati Saxena, Maryam Tavakol, George Fletcher, Mykola Pechenizkiy

In this article, we provide an extensive overview of fairness approaches that have been implemented via a reinforcement learning (RL) framework.

Decision Making Fairness +3

Does the End Justify the Means? On the Moral Justification of Fairness-Aware Machine Learning

no code implementations17 Feb 2022 Hilde Weerts, Lambèr Royakkers, Mykola Pechenizkiy

In this paper, we present a framework for moral reasoning about the justification of fairness metrics and explore the moral implications of the use of fair-ml algorithms that optimize for them.

Ethics Fairness

The Impact of Batch Learning in Stochastic Linear Bandits

1 code implementation14 Feb 2022 Danil Provodin, Pratik Gajane, Mykola Pechenizkiy, Maurits Kaptein

Our main theoretical results show that the impact of batch learning is a multiplicative factor of batch size relative to the regret of online behavior.

The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training

1 code implementation ICLR 2022 Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, Mykola Pechenizkiy

In this paper, we focus on sparse training and highlight a perhaps counter-intuitive finding, that random pruning at initialization can be quite powerful for the sparse training of modern neural networks.

Adversarial Robustness Out-of-Distribution Detection

Semantic-Based Few-Shot Learning by Interactive Psychometric Testing

no code implementations16 Dec 2021 Lu Yin, Vlado Menkovski, Yulong Pei, Mykola Pechenizkiy

In this work, we advance the few-shot learning towards this more challenging scenario, the semantic-based few-shot learning, and propose a method to address the paradigm by capturing the inner semantic relationships using interactive psychometric learning.

Few-Shot Learning

Avoiding Forgetting and Allowing Forward Transfer in Continual Learning via Sparse Networks

1 code implementation11 Oct 2021 Ghada Sokar, Decebal Constantin Mocanu, Mykola Pechenizkiy

To address this challenge, we propose a new CL method, named AFAF, that aims to Avoid Forgetting and Allow Forward transfer in class-IL using fix-capacity models.

class-incremental learning Class Incremental Learning +3

Calibrated Adversarial Training

1 code implementation1 Oct 2021 Tianjin Huang, Vlado Menkovski, Yulong Pei, Mykola Pechenizkiy

In this paper, we present the Calibrated Adversarial Training, a method that reduces the adverse effects of semantic perturbations in adversarial training.

Beyond Discriminant Patterns: On the Robustness of Decision Rule Ensembles

no code implementations21 Sep 2021 Xin Du, Subramanian Ramamoorthy, Wouter Duivesteijn, Jin Tian, Mykola Pechenizkiy

Specifically, we propose to leverage causal knowledge by regarding the distributional shifts in subpopulations and deployment environments as the results of interventions on the underlying system.

Hierarchical Semantic Segmentation using Psychometric Learning

no code implementations7 Jul 2021 Lu Yin, Vlado Menkovski, Shiwei Liu, Mykola Pechenizkiy

One of the major challenges in the supervised learning approaches is expressing and collecting the rich knowledge that experts have with respect to the meaning present in the image data.

Image Segmentation Metric Learning +1

Sparse Training via Boosting Pruning Plasticity with Neuroregeneration

2 code implementations NeurIPS 2021 Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, Decebal Constantin Mocanu

Works on lottery ticket hypothesis (LTH) and single-shot network pruning (SNIP) have raised a lot of attention currently on post-training pruning (iterative magnitude pruning), and before-training pruning (pruning at initialization).

Network Pruning Sparse Learning

Dynamic Sparse Training for Deep Reinforcement Learning

1 code implementation8 Jun 2021 Ghada Sokar, Elena Mocanu, Decebal Constantin Mocanu, Mykola Pechenizkiy, Peter Stone

In this paper, we introduce for the first time a dynamic sparse training approach for deep reinforcement learning to accelerate the training process.

Continuous Control Decision Making +3

Direction-Aggregated Attack for Transferable Adversarial Examples

1 code implementation19 Apr 2021 Tianjin Huang, Vlado Menkovski, Yulong Pei, Yuhao Wang, Mykola Pechenizkiy

Deep neural networks are vulnerable to adversarial examples that are crafted by imposing imperceptible changes to the inputs.

Hop-Count Based Self-Supervised Anomaly Detection on Attributed Networks

1 code implementation16 Apr 2021 Tianjin Huang, Yulong Pei, Vlado Menkovski, Mykola Pechenizkiy

Although various approaches have been proposed to solve this problem, two major limitations exist: (1) unsupervised approaches usually work much less efficiently due to the lack of supervisory signal, and (2) existing anomaly detection methods only use local contextual information to detect anomalous nodes, e. g., one- or two-hop information, but ignore the global contextual information.

Self-Supervised Anomaly Detection supervised anomaly detection

Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training

3 code implementations4 Feb 2021 Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, Mykola Pechenizkiy

By starting from a random sparse network and continuously exploring sparse connectivities during training, we can perform an Over-Parameterization in the space-time manifold, closing the gap in the expressibility between sparse training and dense training.

Image Classification Sparse Learning

Self-Attention Meta-Learner for Continual Learning

1 code implementation28 Jan 2021 Ghada Sokar, Decebal Constantin Mocanu, Mykola Pechenizkiy

In this paper, we propose a new method, named Self-Attention Meta-Learner (SAM), which learns a prior knowledge for continual learning that permits learning a sequence of tasks, while avoiding catastrophic forgetting.

Continual Learning Split-CIFAR-10 +1

Selfish Sparse RNN Training

1 code implementation22 Jan 2021 Shiwei Liu, Decebal Constantin Mocanu, Yulong Pei, Mykola Pechenizkiy

Sparse neural networks have been widely applied to reduce the computational demands of training and deploying over-parameterized deep neural networks.

Learning Invariant Representation for Continual Learning

1 code implementation15 Jan 2021 Ghada Sokar, Decebal Constantin Mocanu, Mykola Pechenizkiy

Finally, we analyze the role of the shared invariant representation in mitigating the forgetting problem especially when the number of replayed samples for each previous task is small.

class-incremental learning Class Incremental Learning +3

ViDi: Descriptive Visual Data Clustering as Radiologist Assistant in COVID-19 Streamline Diagnostic

no code implementations30 Nov 2020 Sahithya Ravi, Samaneh Khoshrou, Mykola Pechenizkiy

In the light of the COVID-19 pandemic, deep learning methods have been widely investigated in detecting COVID-19 from chest X-rays.

Decision Making Medical Diagnosis

Bridging the Performance Gap between FGSM and PGD Adversarial Training

1 code implementation7 Nov 2020 Tianjin Huang, Vlado Menkovski, Yulong Pei, Mykola Pechenizkiy

In addition, it achieves comparable performance of adversarial robustness on MNIST dataset under white-box attack, and it achieves better performance than adv. PGD under white-box attack and effectively defends the transferable adversarial attack on CIFAR-10 dataset.

Adversarial Attack Adversarial Robustness

ResGCN: Attention-based Deep Residual Modeling for Anomaly Detection on Attributed Networks

1 code implementation30 Sep 2020 Yulong Pei, Tianjin Huang, Werner van Ipenburg, Mykola Pechenizkiy

Effectively detecting anomalous nodes in attributed networks is crucial for the success of many real-world applications such as fraud and intrusion detection.

Anomaly Detection Intrusion Detection

Feedback Loop and Bias Amplification in Recommender Systems

no code implementations25 Jul 2020 Masoud Mansoury, Himan Abdollahpouri, Mykola Pechenizkiy, Bamshad Mobasher, Robin Burke

Recommendation algorithms are known to suffer from popularity bias; a few popular items are recommended frequently while the majority of other items are ignored.

Recommendation Systems

SpaceNet: Make Free Space For Continual Learning

1 code implementation15 Jul 2020 Ghada Sokar, Decebal Constantin Mocanu, Mykola Pechenizkiy

Regularization-based methods maintain a fixed model capacity; however, previous studies showed the huge performance degradation of these methods when the task identity is not available during inference (e. g. class incremental learning scenario).

class-incremental learning Class Incremental Learning +2

Topological Insights into Sparse Neural Networks

3 code implementations24 Jun 2020 Shiwei Liu, Tim Van der Lee, Anil Yaman, Zahra Atashgahi, Davide Ferraro, Ghada Sokar, Mykola Pechenizkiy, Decebal Constantin Mocanu

However, comparing different sparse topologies and determining how sparse topologies evolve during training, especially for the situation in which the sparse structure optimization is involved, remain as challenging open questions.

FairMatch: A Graph-based Approach for Improving Aggregate Diversity in Recommender Systems

no code implementations3 May 2020 Masoud Mansoury, Himan Abdollahpouri, Mykola Pechenizkiy, Bamshad Mobasher, Robin Burke

That leads to low coverage of items in recommendation lists across users (i. e. low aggregate diversity) and unfair distribution of recommended items.

Fairness Recommendation Systems

Knowledge Elicitation using Deep Metric Learning and Psychometric Testing

no code implementations14 Apr 2020 Lu Yin, Vlado Menkovski, Mykola Pechenizkiy

The main reason for such a reductionist approach is the difficulty in eliciting the domain knowledge from the experts.

Metric Learning

Investigating Potential Factors Associated with Gender Discrimination in Collaborative Recommender Systems

no code implementations18 Feb 2020 Masoud Mansoury, Himan Abdollahpouri, Jessie Smith, Arman Dehpanah, Mykola Pechenizkiy, Bamshad Mobasher

The proliferation of personalized recommendation technologies has raised concerns about discrepancies in their recommendation performance across different genders, age groups, and racial or ethnic populations.

Fairness Recommendation Systems

Novelty Producing Synaptic Plasticity

no code implementations10 Feb 2020 Anil Yaman, Giovanni Iacca, Decebal Constantin Mocanu, George Fletcher, Mykola Pechenizkiy

A learning process with the plasticity property often requires reinforcement signals to guide the process.

Causal Discovery from Incomplete Data: A Deep Learning Approach

no code implementations15 Jan 2020 Yuhao Wang, Vlado Menkovski, Hao Wang, Xin Du, Mykola Pechenizkiy

As systems are getting more autonomous with the development of artificial intelligence, it is important to discover the causal knowledge from observational sensory inputs.

Causal Discovery Imputation

The Relationship between the Consistency of Users' Ratings and Recommendation Calibration

no code implementations3 Nov 2019 Masoud Mansoury, Himan Abdollahpouri, Joris Rombouts, Mykola Pechenizkiy

In this paper, we aim to explore the relationship between the consistency of users' ratings behavior and the degree of calibrated recommendations they receive.

Fairness Recommendation Systems

BSDAR: Beam Search Decoding with Attention Reward in Neural Keyphrase Generation

no code implementations17 Sep 2019 Iftitahu Ni'mah, Vlado Menkovski, Mykola Pechenizkiy

This study mainly investigates two decoding problems in neural keyphrase generation: sequence length bias and beam diversity.

Keyphrase Generation

A Human-Grounded Evaluation of SHAP for Alert Processing

no code implementations7 Jul 2019 Hilde J. P. Weerts, Werner van Ipenburg, Mykola Pechenizkiy

In this paper we present the results of a human-grounded evaluation of SHAP, an explanation method that has been well-received in the XAI and related communities.

BIG-bench Machine Learning Decision Making

Case-Based Reasoning for Assisting Domain Experts in Processing Fraud Alerts of Black-Box Machine Learning Models

no code implementations7 Jul 2019 Hilde J. P. Weerts, Werner van Ipenburg, Mykola Pechenizkiy

In many contexts, it can be useful for domain experts to understand to what extent predictions made by a machine learning model can be trusted.

BIG-bench Machine Learning

Adversarial Balancing-based Representation Learning for Causal Effect Inference with Observational Data

1 code implementation30 Apr 2019 Xin Du, Lei Sun, Wouter Duivesteijn, Alexander Nikolaev, Mykola Pechenizkiy

The challenges for this problem are two-fold: on the one hand, we have to derive a causal estimator to estimate the causal quantity from observational data, where there exists confounding bias; on the other hand, we have to deal with the identification of CATE when the distribution of covariates in treatment and control groups are imbalanced.

Causal Inference Representation Learning +2

Evolving Plasticity for Autonomous Learning under Changing Environmental Conditions

no code implementations2 Apr 2019 Anil Yaman, Giovanni Iacca, Decebal Constantin Mocanu, Matt Coler, George Fletcher, Mykola Pechenizkiy

Hebbian learning is a biologically plausible mechanism for modeling the plasticity property in artificial neural networks (ANNs), based on the local interactions of neurons.

Learning with Delayed Synaptic Plasticity

no code implementations22 Mar 2019 Anil Yaman, Giovanni Iacca, Decebal Constantin Mocanu, George Fletcher, Mykola Pechenizkiy

Inspired by biology, plasticity can be modeled in artificial neural networks by using Hebbian learning rules, i. e. rules that update synapses based on the neuron activations and reinforcement signals.

A Brain-inspired Algorithm for Training Highly Sparse Neural Networks

2 code implementations17 Mar 2019 Zahra Atashgahi, Joost Pieterse, Shiwei Liu, Decebal Constantin Mocanu, Raymond Veldhuis, Mykola Pechenizkiy

Concretely, by exploiting the cosine similarity metric to measure the importance of the connections, our proposed method, Cosine similarity-based and Random Topology Exploration (CTRE), evolves the topology of sparse neural networks by adding the most important connections to the network without calculating dense gradient in the backward.

Learning Theory

Sparse evolutionary Deep Learning with over one million artificial neurons on commodity hardware

4 code implementations26 Jan 2019 Shiwei Liu, Decebal Constantin Mocanu, Amarsagar Reddy Ramapuram Matavalam, Yulong Pei, Mykola Pechenizkiy

Despite the success of ANNs, it is challenging to train and deploy modern ANNs on commodity hardware due to the ever-increasing model size and the unprecedented growth in the data volumes.

Intrinsically Sparse Long Short-Term Memory Networks

no code implementations26 Jan 2019 Shiwei Liu, Decebal Constantin Mocanu, Mykola Pechenizkiy

However, LSTMs are prone to be memory-bandwidth limited in realistic applications and need an unbearable period of training and inference time as the model size is ever-increasing.

Model Compression Sentiment Analysis

Looking Deeper into Deep Learning Model: Attribution-based Explanations of TextCNN

no code implementations8 Nov 2018 Wenting Xiong, Iftitahu Ni'mah, Juan M. G. Huesca, Werner van Ipenburg, Jan Veldsink, Mykola Pechenizkiy

Layer-wise Relevance Propagation (LRP) and saliency maps have been recently used to explain the predictions of Deep Learning models, specifically in the domain of text classification.

text-classification Text Classification

Controversy Rules - Discovering Regions Where Classifiers (Dis-)Agree Exceptionally

no code implementations22 Aug 2018 Oren Zeev-Ben-Mordehai, Wouter Duivesteijn, Mykola Pechenizkiy

Finding regions for which there is higher controversy among different classifiers is insightful with regards to the domain and our models.

General Classification General Knowledge

struc2gauss: Structural Role Preserving Network Embedding via Gaussian Embedding

no code implementations25 May 2018 Yulong Pei, Xin Du, Jianpeng Zhang, George Fletcher, Mykola Pechenizkiy

Almost all previous methods represent a node into a point in space and focus on local structural information, i. e., neighborhood information.

Network Embedding

On Formalizing Fairness in Prediction with Machine Learning

no code implementations9 Oct 2017 Pratik Gajane, Mykola Pechenizkiy

Machine learning algorithms for prediction are increasingly being used in critical decisions affecting human lives.

BIG-bench Machine Learning Fairness

Rule-based Emotion Detection on Social Media: Putting Tweets on Plutchik's Wheel

no code implementations15 Dec 2014 Erik Tromp, Mykola Pechenizkiy

We study sentiment analysis beyond the typical granularity of polarity and instead use Plutchik's wheel of emotions model.

Sentiment Analysis

Predictive User Modeling with Actionable Attributes

no code implementations23 Dec 2013 Indre Zliobaite, Mykola Pechenizkiy

We study how to learn to choose the value of an actionable attribute in order to maximize the probability of a desired outcome in predictive modeling.


Cannot find the paper you are looking for? You can Submit a new open access paper.