Search Results for author:
Found 81 papers, 54 papers with code
HALO: Hardware-Aware Learning to Optimize
There has been an explosive demand for bringing machine learning (ML) powered intelligence into numerous Internet-of-Things (IoT) devices.
APP: Anytime Progressive Pruning
With the latest advances in deep learning, there has been a lot of focus on the online learning paradigm due to its relevance in practical settings.
Symbolic Learning to Optimize: Towards Interpretability and Scalability
Recent studies on Learning to Optimize (L2O) suggest a promising path to automating and accelerating the optimization procedure for complicated tasks.
Optimizer Amalgamation
Selecting an appropriate optimizer for a given problem is of major interest for researchers and practitioners.
The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancy
However, a "head-to-toe assessment" regarding the extent of redundancy in ViTs, and how much we could gain by thoroughly mitigating such, has been absent for this field.
Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice
The first technique, termed AttnScale, decomposes a self-attention block into low-pass and high-pass components, then rescales and combines these two filters to produce an all-pass self-attention matrix.
Don't Be So Dense: Sparse-to-Sparse GAN Training Without Sacrificing Performance
Even more unconventionally, our proposed method enables directly training sparse unbalanced GANs with an extremely sparse generator from scratch.
Sparsity Winning Twice: Better Robust Generalization from More Efficient Training
We introduce two alternatives for sparse adversarial training: (i) static sparsity, by leveraging recent results from the lottery ticket hypothesis to identify critical sparse subnetworks arising from the early training; (ii) dynamic sparsity, by allowing the sparse subnetwork to adaptively adjust its connectivity pattern (while sticking to the same sparsity ratio) throughout training.
Coarsening the Granularity: Towards Structurally Sparse Lottery Tickets
The lottery ticket hypothesis (LTH) has shown that dense models contain highly sparse subnetworks (i. e., winning tickets) that can be trained in isolation to match full accuracy.
The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training
In this paper, we focus on sparse training and highlight a perhaps counter-intuitive finding, that random pruning at initialization can be quite powerful for the sparse training of modern neural networks.
VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformer
To the best of our knowledge, this is the first time quantization has been incorporated into ViT acceleration on FPGAs with the help of a fully automatic framework to guide the quantization strategy on the software side and the accelerator implementations on the hardware side given the target frame rate.
Bringing Your Own View: Graph Contrastive Learning without Prefabricated Data Augmentations
Accordingly, we have extended the prefabricated discrete prior in the augmentation set, to a learnable continuous prior in the parameter space of graph generators, assuming that graph priors per se, similar to the concept of image manifolds, can be learned by data generation.
Improving Contrastive Learning on Imbalanced Seed Data via Open-World Sampling
Contrastive learning approaches have achieved great success in learning visual representations with few labels of the target classes.
DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models
To address these pain points, we propose a framework for resource- and parameter-efficient fine-tuning by leveraging the sparsity prior in both weight updates and the final model weights.
You are caught stealing my winning lottery ticket! Making a lottery ticket claim its ownership
The lottery ticket hypothesis (LTH) emerges as a promising framework to leverage a special sparse subnetwork (i. e., winning ticket) instead of a full model for both training and inference, that can lower both costs without sacrificing the performance.
CAP: Co-Adversarial Perturbation on Weights and Features for Improving Generalization of Graph Neural Networks
In this paper, we investigate GNNs from the lens of weight and feature loss landscapes, i. e., the loss changes with respect to model weights and node features, respectively.
Towards Lifelong Learning of Multilingual Text-To-Speech Synthesis
This work presents a lifelong learning approach to train a multilingual Text-To-Speech (TTS) system, where each language was seen as an individual task and was learned sequentially and continually.
Universality of Winning Tickets: A Renormalization Group Perspective
We make use of renormalization group theory, a powerful tool from theoretical physics, to address this need.
Scaling the Depth of Vision Transformers via the Fourier Domain Analysis
The first technique, termed AttnScale, decomposes a self-attention block into low-pass and high-pass components, then rescales and combines these two filters to produce an all-pass self-attention matrix.
Generalizable Learning to Optimize into Wide Valleys
Learning to optimize (L2O) has gained increasing popularity in various optimization tasks, since classical optimizers usually require laborious, problem-specific design and hyperparameter tuning.
Stingy Teacher: Sparse Logits Suffice to Fail Knowledge Distillation
However, it is unclear why the distorted distribution of the logits is catastrophic to the student model.
AutoCoG: A Unified Data-Modal Co-Search Framework for Graph Neural Networks
Despite the preliminary success, we argue that for GNNs, NAS has to be customized further, due to the topological complicacy of GNN input data (graph) as well as the notorious training instability.
Sparse Unbalanced GAN Training with In-Time Over-Parameterization
Perhaps most importantly, we find instead of inheriting parameters from expensive pre-trained GANs, directly training sparse GANs from scratch can be a much more efficient solution.
Bayesian Modeling and Uncertainty Quantification for Learning to Optimize: What, Why, and How
Optimizing an objective function with uncertainty awareness is well-known to improve the accuracy and confidence of optimization solutions.
Inductive Lottery Ticket Learning for Graph Neural Networks
In this work, we propose a simple and effective learning paradigm, Inductive Co-Pruning of GNNs (ICPG), to endow graph lottery tickets with inductive pruning capacity.
Learning Pruning-Friendly Networks via Frank-Wolfe: One-Shot, Any-Sparsity, And No Retraining
Conventional methods often require (iterative) pruning followed by re-training, which not only incurs large overhead beyond the original DNN training but also can be sensitive to retraining hyperparameters.
Lottery Tickets can have Structural Sparsity
The lottery ticket hypothesis (LTH) has shown that dense models contain highly sparse subnetworks (i. e., $\textit{winning tickets}$) that can be trained in isolation to match full accuracy.
Universality of Deep Neural Network Lottery Tickets: A Renormalization Group Perspective
Foundational work on the Lottery Ticket Hypothesis has suggested an exciting corollary: winning tickets found in the context of one task can be transferred to similar tasks, possibly even across different architectures.
Audio Lottery: Speech Recognition Made Ultra-Lightweight, Noise-Robust, and Transferable
In this paper, we investigate the tantalizing possibility of using lottery ticket hypothesis to discover lightweight speech recognition models, that are (1) robust to various noise existing in speech; (2) transferable to fit the open-world personalization; and 3) compatible with structured sparsity.
Bag of Tricks for Training Deeper Graph Neural Networks: A Comprehensive Benchmark Study
In view of those, we present the first fair and reproducible benchmark dedicated to assessing the "tricks" of training deep GNNs.
DANCE: DAta-Network Co-optimization for Efficient Segmentation Model Training and Inference
Semantic segmentation for scene understanding is nowadays widely demanded, raising significant challenges for the algorithm efficiency, especially its applications on resource-limited platforms.
Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?
Based on our analysis, we summarize a guideline for parameter settings in regards of specific architecture characteristics, which we hope to catalyze the research progress on the topic of lottery ticket hypothesis.
Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity
Our framework, FreeTickets, is defined as the ensemble of these relatively cheap sparse subnetworks.
Scalable Perception-Action-Communication Loops with Convolutional and Graph Neural Networks
Our framework is implemented by a cascade of a convolutional and a graph neural network (CNN / GNN), addressing agent-level visual perception and feature learning, as well as swarm-level communication, local information aggregation and agent action inference, respectively.
Sparse Training via Boosting Pruning Plasticity with Neuroregeneration
Works on lottery ticket hypothesis (LTH) and single-shot network pruning (SNIP) have raised a lot of attention currently on post-training pruning (iterative magnitude pruning), and before-training pruning (pruning at initialization).
Ranked #3 on
Sparse Learning
on ImageNet
Sparse and Imperceptible Adversarial Attack via a Homotopy Algorithm
Compared to state-of-the-art methods, our homotopy attack leads to significantly fewer perturbations, e. g., reducing 42. 91% on CIFAR-10 and 75. 03% on ImageNet (average case, targeted attack), at similar maximal perturbation magnitudes, when still achieving 100% attack success rates.
Graph Contrastive Learning Automated
Unfortunately, unlike its counterpart on image data, the effectiveness of GraphCL hinges on ad-hoc data augmentations, which have to be manually picked per dataset, by either rules of thumb or trial-and-errors, owing to the diverse nature of graph data.
Chasing Sparsity in Vision Transformers: An End-to-End Exploration
For example, our sparsified DeiT-Small at (5%, 50%) sparsity for (data, architecture), improves 0. 28% top-1 accuracy, and meanwhile enjoys 49. 32% FLOPs and 4. 40% running time savings.
Self-Damaging Contrastive Learning
Hence, the key innovation in SDCLR is to create a dynamic self-competitor model to contrast with the target model, which is a pruned version of the latter.
Efficient Lottery Ticket Finding: Less Data is More
We observe that a high-quality winning ticket can be found with training and pruning the dense network on the very compact PrAC set, which can substantially save training iterations for the ticket finding process.
GANs Can Play Lottery Tickets Too
In this work, we for the first time study the existence of such trainable matching subnetworks in deep GANs.
Improving Contrastive Learning on Imbalanced Data via Open-World Sampling
Contrastive learning approaches have achieved great success in learning visual representations with few labels of the target classes.
Undistillable: Making A Nasty Teacher That CANNOT teach students
Knowledge Distillation (KD) is a widely used technique to transfer knowledge from pre-trained teacher models to (usually more lightweight) student models.
Troubleshooting Blind Image Quality Models in the Wild
Recently, the group maximum differentiation competition (gMAD) has been used to improve blind image quality assessment (BIQA) models, with the help of full-reference metrics.
Playing Lottery Tickets with Vision and Language
However, we can find "relaxed" winning tickets at 50%-70% sparsity that maintain 99% of the full accuracy.
Learning Transferable 3D Adversarial Cloaks for Deep Trained Detectors
This paper presents a novel patch-based adversarial attack pipeline that trains adversarial patches on 3D human meshes.
"BNN - BN = ?": Training Binary Neural Networks without Batch Normalization
However, the BN layer is costly to calculate and is typically implemented with non-binary parameters, leaving a hurdle for the efficient implementation of BNN training.
Ranked #139 on
Image Classification
on CIFAR-100
Learning to Optimize: A Primer and A Benchmark
It automates the design of an optimization method based on its performance on a set of training problems.
Adversarial Feature Augmentation and Normalization for Visual Recognition
Recent advances in computer vision take advantage of adversarial data augmentation to ameliorate the generalization ability of classification models.
Data-Efficient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspective
Training generative adversarial networks (GANs) with limited real image data generally results in deteriorated performance and collapsed models.
Sandwich Batch Normalization: A Drop-In Replacement for Feature Distribution Heterogeneity
We present Sandwich Batch Normalization (SaBN), a frustratingly easy improvement of Batch Normalization (BN) with only a few lines of code changes.
A Unified Lottery Ticket Hypothesis for Graph Neural Networks
With graphs rapidly growing in size and deeper graph neural networks (GNNs) emerging, the training and inference of GNNs become increasingly expensive.
Spending Your Winning Lottery Better After Drawing It
In this paper, we demonstrate that it is unnecessary for spare retraining to strictly inherit those properties from the dense network.
Efficiently Troubleshooting Image Segmentation Models with Human-In-The-Loop
Image segmentation lays the foundation for many high-stakes vision applications such as autonomous driving and medical image analysis.
Bayesian Learning to Optimize: Quantifying the Optimizer Uncertainty
Optimizing an objective function with uncertainty awareness is well-known to improve the accuracy and confidence of optimization solutions.
Robust Overfitting may be mitigated by properly learned smoothening
A recent study (Rice et al., 2020) revealed overfitting to be a dominant phenomenon in adversarially robust training of deep networks, and that appropriate early-stopping of adversarial training (AT) could match the performance gains of most recent algorithmic improvements.
Learning A Minimax Optimizer: A Pilot Study
We first present Twin L2O, the first dedicated minimax L2O framework consisting of two LSTMs for updating min and max variables, respectively.
ALFA: Adversarial Feature Augmentation for Enhanced Image Recognition
Adversarial training is an effective method to combat adversarial attacks in order to create robust neural networks.
Long Live the Lottery: The Existence of Winning Tickets in Lifelong Learning
In view of those, we introduce two pruning options, e. g., top-down and bottom-up, for finding lifelong tickets.
The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
We extend the scope of LTH and question whether matching subnetworks still exist in pre-trained computer vision models, that enjoy the same downstream transfer performance.
Robust Pre-Training by Adversarial Contrastive Learning
Recent work has shown that, when integrated with adversarial training, self-supervised pre-training can lead to state-of-the-art robustness In this work, we improve robustness-aware self-supervised pre-training by learning representations that are consistent under both data augmentations and adversarial perturbations.
Graph Contrastive Learning with Augmentations
In this paper, we propose a graph contrastive learning (GraphCL) framework for learning unsupervised representations of graph data.
Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness and Accuracy for Free
The trained model could be adjusted among different standard and robust accuracies "for free" at testing time.
Training Stronger Baselines for Learning to Optimize
Learning to optimize (L2O) has gained increasing attention since classical optimizers require laborious problem-specific design and hyperparameter tuning.
PCAL: A Privacy-preserving Intelligent Credit Risk Modeling Framework Based on Adversarial Learning
While machine learning is increasingly used in this field, the resulting large-scale collection of user private information has reinvigorated the privacy debate, considering dozens of data breach incidents every year caused by unauthorized hackers, and (potentially even more) information misuse/abuse by authorized parties.
The Lottery Ticket Hypothesis for Pre-trained BERT Networks
For a range of downstream tasks, we indeed find matching subnetworks at 40% to 90% sparsity.
Can 3D Adversarial Logos Cloak Humans?
Contrary to the traditional adversarial patch, this new form of attack is mapped into the 3D object world and back-propagates to the 2D image domain through differentiable rendering.
Self-PU: Self Boosted and Calibrated Positive-Unlabeled Training
Many real-world applications have to tackle the Positive-Unlabeled (PU) learning problem, i. e., learning binary classifiers from a large amount of unlabeled data and a few labeled positive examples.
When Does Self-Supervision Help Graph Convolutional Networks?
We first elaborate three mechanisms to incorporate self-supervision into GCNs, analyze the limitations of pretraining & finetuning and self-training, and proceed to focus on multi-task learning.
Focus Longer to See Better:Recursively Refined Attention for Fine-Grained Image Classification
In this paper, we tried to focus on these marginal differences to extract more representative features.
AutoSpeech: Neural Architecture Search for Speaker Recognition
Speaker recognition systems based on Convolutional Neural Networks (CNNs) are often built with off-the-shelf backbones such as VGG-Net or ResNet.
Ranked #1 on
Speaker Identification
on VoxCeleb1
Dataset and Enhanced Model for Eligibility Criteria-to-SQL Semantic Parsing
Compared to existing datasets, the queries in the dataset here are derived from the eligibility criteria of clinical trials and include \textit{Order-sensitive, Counting-based, and Boolean-type} cases which are not seen before.
L^2-GCN: Layer-Wise and Learned Efficient Training of Graph Convolutional Networks
Graph convolution networks (GCN) are increasingly popular in many applications, yet remain notoriously hard to train over large graph datasets.
Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning
We conduct extensive experiments to demonstrate that the proposed framework achieves large performance margins (eg, 3. 83% on robust accuracy and 1. 3% on standard accuracy, on the CIFAR-10 dataset), compared with the conventional end-to-end adversarial training baseline.
I Am Going MAD: Maximum Discrepancy Competition for Comparing Classifiers Adaptively
On the other hand, the trained classifiers have traditionally been evaluated on small and fixed sets of test images, which are deemed to be extremely sparsely distributed in the space of all natural images.
Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference
Deep networks were recently suggested to face the odds between accuracy (on clean natural images) and robustness (on adversarially perturbed images) (Tsipras et al., 2019).
VGAI: End-to-End Learning of Vision-Based Decentralized Controllers for Robot Swarms
More specifically, we consider that each robot has access to a visual perception of the immediate surroundings, and communication capabilities to transmit and receive messages from other neighboring robots.
Calibrated Domain-Invariant Learning for Highly Generalizable Large Scale Re-Identification
Many real-world applications, such as city-scale traffic monitoring and control, requires large-scale re-identification.
Learning to Optimize in Swarms
Learning to optimize has emerged as a powerful framework for various optimization and machine learning tasks.
ABD-Net: Attentive but Diverse Person Re-Identification
Attention mechanism has been shown to be effective for person re-identification (Re-ID).
Ranked #17 on
Person Re-Identification
on MSMT17
