Search Results for author:
Found 316 papers, 198 papers with code
HALO: Hardware-Aware Learning to Optimize
There has been an explosive demand for bringing machine learning (ML) powered intelligence into numerous Internet-of-Things (IoT) devices.
Eliminating the Invariance on the Loss Landscape of Linear Autoencoders
For this new loss, we characterize the full structure of the loss landscape in the following sense: we establish analytical expression for the set of all critical points, show that it is a subset of critical points of MSE, and that all local minima are still global.
Safe and Robust Watermark Injection with a Single OoD Image
Training a high-performance deep neural network requires large amounts of data and computational resources.
Enhancing NeRF akin to Enhancing LLMs: Generalizable NeRF Transformer with Mixture-of-View-Experts
Cross-scene generalizable NeRF models, which can directly synthesize novel views of unseen scenes, have become a new spotlight of the NeRF field.
Robust Mixture-of-Expert Training for Convolutional Neural Networks
Since the lack of robustness has become one of the main hurdles for CNNs, in this paper we ask: How to adversarially robustify a CNN-based MoE model?
How Does Pruning Impact Long-Tailed Multi-Label Medical Image Classifiers?
This work represents a first step toward understanding the impact of pruning on model behavior in deep long-tailed, multi-label medical image classification.
INR-Arch: A Dataflow Architecture and Compiler for Arbitrary-Order Gradient Computations in Implicit Neural Representation Processing
In this work, we introduce INR-Arch, a framework that transforms the computation graph of an nth-order gradient into a hardware-optimized dataflow architecture.
Doubly Robust Instance-Reweighted Adversarial Training
Experiments on standard classification datasets demonstrate that our proposed approach outperforms related state-of-the-art baseline methods in terms of average robust performance, and at the same time improves the robustness against attacks on the weakest data points.
Reference-based Painterly Inpainting via Diffusion: Crossing the Wild Reference Domain Gap
We propose Reference-based Painterly Inpainting, a novel task that crosses the wild reference domain gap and implants novel objects into artworks.
Physics-Driven Turbulence Image Restoration with Stochastic Refinement
Although fast and physics-grounded simulation tools have been introduced to help the deep-learning models adapt to real-world turbulence conditions recently, the training of such models only relies on the synthetic data and ground truth pairs.
Polynomial Width is Sufficient for Set Representation with High-dimensional Features
To investigate the minimal value of $L$ that achieves sufficient expressive power, we present two set-element embedding layers: (a) linear + power activation (LP) and (b) linear + exponential activations (LE).
Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunities
Recently, zero-shot (or training-free) Neural Architecture Search (NAS) approaches have been proposed to liberate the NAS from training requirements.
FarSight: A Physics-Driven Whole-Body Biometric System at Large Distance and Altitude
Whole-body biometric recognition is an important area of research due to its vast applications in law enforcement, border security, and surveillance.
H$_2$O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
Based on these insights, we propose Heavy Hitter Oracle (H$_2$O), a KV cache eviction policy that dynamically retains a balance of recent and H$_2$ tokens.
Graph Ladling: Shockingly Simple Parallel GNN Training without Intermediate Communication
By dividing giant graph data, we build multiple independently and parallelly trained weaker GNNs (soup ingredient) without any intermediate communication, and combine their strength using a greedy interpolation soup procedure to achieve state-of-the-art performance.
Instant Soup: Cheap Pruning Ensembles in A Single Pass Can Draw Lottery Tickets from Large Models
Motivated by the recent observations of model soups, which suggest that fine-tuned weights of multiple models can be merged to a better minima, we propose Instant Soup Pruning (ISP) to generate lottery ticket quality subnetworks, using a fraction of the original IMP cost by replacing the expensive intermediate pruning stages of IMP with computationally efficient weak mask generation and aggregation routine.
Learning to Estimate 6DoF Pose from Limited Data: A Few-Shot, Generalizable Approach using RGB Images
The accurate estimation of six degrees-of-freedom (6DoF) object poses is essential for many applications in robotics and augmented reality.
The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter
Large pre-trained transformers are show-stealer in modern-day deep learning, and it becomes crucial to comprehend the parsimonious patterns that exist within them as they grow in scale.
Dynamic Sparsity Is Channel-Level Sparsity Learner
Dynamic sparse training (DST), as a leading sparse training approach, can train deep neural networks at high sparsity from scratch to match the performance of their dense counterparts.
Are Large Kernels Better Teachers than Transformers for ConvNets?
We hereby carry out a first-of-its-kind study unveiling that modern large-kernel ConvNets, a compelling competitor to Vision Transformers, are remarkably more effective teachers for small-kernel ConvNets, due to more similar architectures.
Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Experts
Computer vision researchers are embracing two promising paradigms: Vision Transformers (ViTs) and Multi-task Learning (MTL), which both show great performance but are computation-intensive, given the quadratic complexity of self-attention in ViT and the need to activate an entire large MTL model for one task.
Towards Constituting Mathematical Structures for Learning to Optimize
Learning to Optimize (L2O), a technique that utilizes machine learning to learn an optimization algorithm automatically from data, has gained arising attention in recent years.
Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models
Text-to-image (T2I) research has grown explosively in the past year, owing to the large-scale pre-trained diffusion models and many emerging personalization and editing approaches.
POPE: 6-DoF Promptable Pose Estimation of Any Object, in Any Scene, with One Reference
To mitigate this issue, we propose a general paradigm for object pose estimation, called Promptable Object Pose Estimation (POPE).
MMG-Ego4D: Multi-Modal Generalization in Egocentric Action Recognition
In this paper, we study a novel problem in egocentric action recognition, which we term as "Multimodal Generalization" (MMG).
In-Context Learning Unlocked for Diffusion Models
To achieve this, we propose a vision-language prompt that can model a wide range of vision-language tasks and a diffusion model that takes it as input.
Outline, Then Details: Syntactically Guided Coarse-To-Fine Code Generation
For a complicated algorithm, its implementation by a human programmer usually starts with outlining a rough control flow followed by iterative enrichments, eventually yielding carefully generated syntactic structures and variables in a hierarchy.
Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Models
Patch Diffusion meanwhile improves the performance of diffusion models trained on relatively small datasets, $e. g.$, as few as 5, 000 images to train from scratch.
AutoShot: A Short Video Dataset and State-of-the-Art Shot Boundary Detection
For video content creation and understanding, the shot boundary detection (SBD) is one of the most essential components in various scenarios.
Ranked #1 on
Camera shot boundary detection
on ClipShots
Graph Mixture of Experts: Learning on Large-Scale Graphs with Explicit Diversity Modeling
Additionally, GMoE includes information aggregation experts with varying aggregation hop sizes, where the experts with larger hop sizes are specialized in capturing information over longer ranges.
Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models
The unlearning problem of deep learning models, once primarily an academic concern, has become a prevalent issue in the industry.
PAIR-Diffusion: Object-Level Image Editing with Structure-and-Appearance Paired Diffusion Models
Nevertheless, most of them lack fine-grained control over the properties of the different objects present in the image, i. e. object-level image editing.
Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators
Recent text-to-video generation approaches rely on computationally heavy training and require large-scale video datasets.
Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!
In pursuit of a more general evaluation and unveiling the true potential of sparse algorithms, we introduce "Sparsity May Cry" Benchmark (SMC-Bench), a collection of carefully-curated 4 diverse tasks with 10 datasets, that accounts for capturing a wide range of domain-specific and sophisticated knowledge.
Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers
Despite their remarkable achievement, gigantic transformers encounter significant drawbacks, including exorbitant computational and memory footprints during training, as well as severe collapse evidenced by a high degree of parameter redundancy.
Learning to Grow Pretrained Models for Efficient Transformer Training
Scaling transformers has led to significant breakthroughs in many domains, leading to a paradigm in which larger versions of existing models are trained and released on a periodic basis.
M-L2O: Towards Generalizable Learning-to-Optimize by Test-Time Fast Self-Adaptation
This data-driven procedure yields L2O that can efficiently solve problems similar to those seen in training, that is, drawn from the same ``task distribution".
You Only Transfer What You Share: Intersection-Induced Graph Transfer Learning for Link Prediction
We identify this setting as Graph Intersection-induced Transfer Learning (GITL), which is motivated by practical applications in e-commerce or academic co-authorship predictions.
Layer Grafted Pre-training: Bridging Contrastive Learning And Masked Image Modeling For Label-Efficient Representations
This motivates us to shift the paradigm from combining loss at the end, to choosing the proper learning method per network layer.
Robust Weight Signatures: Gaining Robustness as Easy as Patching Weights?
Given a robust model trained to be resilient to one or multiple types of distribution shifts (e. g., natural image corruptions), how is that "robustness" encoded in the model weights, and how easily can it be disentangled and/or "zero-shot" transferred to some other models?
Learning to Generalize Provably in Learning to Optimize
While the optimizer generalization has been recently studied, the optimizee generalization (or learning to generalize) has not been rigorously studied in the L2O context, which is the aim of this paper.
Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researchers
In response, we summarize ten Q\&As of SNNs from many key aspects, including dense vs. sparse, unstructured sparse vs. structured sparse, pruning vs. sparse training, dense-to-sparse training vs. sparse-to-sparse training, static sparsity vs. dynamic sparsity, before-training/during-training vs. post-training sparsity, and many more.
Turning the Curse of Heterogeneity in Federated Learning into a Blessing for Out-of-Distribution Detection
We propose to take advantage of such heterogeneity and turn the curse into a blessing that facilitates OoD detection in FL.
Evaluate underdiagnosis and overdiagnosis bias of deep learning model on primary open-angle glaucoma diagnosis in under-served patient populations
In the United States, primary open-angle glaucoma (POAG) is the leading cause of blindness, especially among African American and Hispanic individuals.
Specialist Diffusion: Plug-and-Play Sample-Efficient Fine-Tuning of Text-to-Image Diffusion Models To Learn Any Unseen Style
Diffusion models have demonstrated impressive capability of text-conditioned image synthesis, and broader application horizons are emerging by personalizing those pretrained diffusion models toward generating some specialized target object or style.
NeuralLift-360: Lifting an In-the-Wild 2D Photo to a 3D Object With 360deg Views
In this work, we study the challenging task of lifting a single image to a 3D object and, for the first time, demonstrate the ability to generate a plausible 3D object with 360deg views that corresponds well with the given reference image.
Pruning Before Training May Improve Generalization, Provably
It is shown that as long as the pruning fraction is below a certain threshold, gradient descent can drive the training loss toward zero and the network exhibits good generalization performance.
MMG-Ego4D: Multimodal Generalization in Egocentric Action Recognition
In this paper, we study a novel problem in egocentric action recognition, which we term as "Multimodal Generalization" (MMG).
Convergence and Generalization of Wide Neural Networks with Large Bias
This work studies training one-hidden-layer overparameterized ReLU networks via gradient descent in the neural tangent kernel (NTK) regime, where the networks' biases are initialized to some constant rather than zero.
Symbolic Visual Reinforcement Learning: A Scalable Framework with Object-Level Abstraction and Differentiable Expression Search
Learning efficient and interpretable policies has been a challenging task in reinforcement learning (RL), particularly in the visual RL setting with complex scenes.
Attend Who is Weak: Pruning-assisted Medical Image Localization under Sophisticated and Implicit Imbalances
However, DNNs are notoriously fragile to the class imbalance in image classification.
StegaNeRF: Embedding Invisible Information within Neural Radiance Fields
Recent advances in neural rendering imply a future of widespread visual data distributions through sharing NeRF model weights.
NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with 360° Views
In this work, we study the challenging task of lifting a single image to a 3D object and, for the first time, demonstrate the ability to generate a plausible 3D object with 360{\deg} views that correspond well with the given reference image.
You Can Have Better Graph Neural Networks by Not Training Weights at All: Finding Untrained GNNs Tickets
Recent works have impressively demonstrated that there exists a subnetwork in randomly initialized convolutional neural networks (CNNs) that can match the performance of the fully trained dense networks at initialization, without any optimization of the weights of the network (i. e., untrained networks).
Search Behavior Prediction: A Hypergraph Perspective
With these hyperedges, we augment the original bipartite graph into a new \textit{hypergraph}.
Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training
Vision transformers (ViTs) have recently obtained success in many applications, but their intensive computation and heavy memory usage at both training and inference time limit their generalization.
AligNeRF: High-Fidelity Neural Radiance Fields via Alignment-Aware Training
Neural Radiance Fields (NeRFs) are a powerful representation for modeling a 3D scene as a continuous function.
Versatile Diffusion: Text, Images and Variations All in One Diffusion Model
In this work, we expand the existing single-flow diffusion pipeline into a multi-task multimodal network, dubbed Versatile Diffusion (VD), that handles multiple flows of text-to-image, image-to-text, and variations in one unified model.
StyleNAT: Giving Each Head a New Perspective
Image generation has been a long sought-after but challenging task, and performing the generation task in an efficient manner is similarly difficult.
Ranked #1 on
Image Generation
on FFHQ 256 x 256
QuanGCN: Noise-Adaptive Training for Robust Quantum Graph Convolutional Networks
Quantum neural networks (QNNs), an interdisciplinary field of quantum computing and machine learning, have attracted tremendous research interests due to the specific quantum advantages.
Scaling Multimodal Pre-Training via Cross-Modality Gradient Harmonization
For example, even in the commonly adopted instructional videos, a speaker can sometimes refer to something that is not visually present in the current frame; and the semantic misalignment would only be more unpredictable for the raw videos from the internet.
Sparse Winning Tickets are Data-Efficient Image Recognizers
Improving the performance of deep networks in data-limited regimes has warranted much attention.
M$^3$ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design
However, when deploying MTL onto those real-world systems that are often resource-constrained or latency-sensitive, two prominent challenges arise: (i) during training, simultaneously optimizing all tasks is often difficult due to gradient conflicts across tasks; (ii) at inference, current MTL regimes have to activate nearly the entire model even to just execute a single task.
Symbolic Distillation for Learned TCP Congestion Control
At the core of our proposal is a novel symbolic branching algorithm that enables the rule to be aware of the context in terms of various network conditions, eventually converting the NN policy into a symbolic tree.
Signal Processing for Implicit Neural Representations
We answer this question by proposing an implicit neural signal processing network, dubbed INSP-Net, via differential operators on INR.
Data-Model-Circuit Tri-Design for Ultra-Light Video Intelligence on Edge Devices
In this paper, we propose a data-model-hardware tri-design framework for high-throughput, low-cost, and high-accuracy multi-object tracking (MOT) on High-Definition (HD) video stream.
RoS-KD: A Robust Stochastic Knowledge Distillation Approach for Noisy Medical Imaging
Our extensive experiments on popular medical imaging classification tasks (cardiopulmonary disease and lesion classification) using real-world datasets, show the performance benefit of RoS-KD, its ability to distill knowledge from many popular large networks (ResNet-50, DenseNet-121, MobileNet-V2) in a comparatively small network, and its robustness to adversarial attacks (PGD, FSGM).
Old can be Gold: Better Gradient Flow can Make Vanilla-GCNs Great Again
In this paper, firstly, we provide a new perspective of gradient flow to understand the substandard performance of deep GCNs and hypothesize that by facilitating healthy gradient flow, we can significantly improve their trainability, as well as achieve state-of-the-art (SOTA) level performance from vanilla-GCNs.
A Comprehensive Study on Large-Scale Graph Training: Benchmarking and Rethinking
Large-scale graph training is a notoriously challenging problem for graph neural networks (GNNs).
Ranked #2 on
Node Property Prediction
on ogbn-products
Trap and Replace: Defending Backdoor Attacks by Trapping Them into an Easy-to-Replace Subnetwork
As a result, both the stem and the classification head in the final network are hardly affected by backdoor training samples.
Augmentations in Hypergraph Contrastive Learning: Fabricated and Generative
This paper targets at improving the generalizability of hypergraph neural networks in the low-label regime, through applying the contrastive learning approach from images/graphs (we refer to it as HyperGCL).
DynImp: Dynamic Imputation for Wearable Sensing Data Through Sensory and Temporal Relatedness
We argue that traditional methods have rarely made use of both times-series dynamics of the data as well as the relatedness of the features from different sensors.
NeRF-SOS: Any-View Self-supervised Object Segmentation on Complex Scenes
Our framework, called NeRF with Self-supervised Object Segmentation NeRF-SOS, couples object segmentation and neural radiance field to segment objects in any view within a scene.
Can We Solve 3D Vision Tasks Starting from A 2D Vision Transformer?
Vision Transformers (ViTs) have proven to be effective, in solving 2D image understanding tasks by training over large-scale image datasets; and meanwhile as a somehow separate track, in modeling the 3D visual world too such as voxels or point clouds.
Long-Tailed Classification of Thorax Diseases on Chest X-Ray: A New Benchmark Study
Imaging exams, such as chest radiography, will yield a small set of common findings and a much larger set of uncommon findings.
Ranked #1 on
Long-tail Learning
on MIMIC-CXR-LT
Auto-ViT-Acc: An FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantization
Compared with state-of-the-art ViT quantization work (algorithmic approach only without hardware acceleration), our quantization achieves 0. 47% to 1. 36% higher Top-1 accuracy under the same bit-width.
Is Attention All That NeRF Needs?
While prior works on NeRFs optimize a scene representation by inverting a handcrafted rendering equation, GNT achieves neural representation and rendering that generalizes across scenes using transformers at two stages.
Self-supervised contrastive learning of echocardiogram videos enables label-efficient cardiac disease diagnosis
Advances in self-supervised learning (SSL) have shown that self-supervised pretraining on medical imaging data can provide a strong initialization for downstream supervised classification and segmentation.
Density-Aware Personalized Training for Risk Prediction in Imbalanced Medical Data
Medical events of interest, such as mortality, often happen at a low rate in electronic medical records, as most admitted patients survive.
Single Frame Atmospheric Turbulence Mitigation: A Benchmark Study and A New Physics-Inspired Transformer Model
Image restoration algorithms for atmospheric turbulence are known to be much more challenging to design than traditional ones such as blur or noise because the distortion caused by the turbulence is an entanglement of spatially varying blur, geometric distortion, and sensor noise.
Equivariant Hypergraph Diffusion Neural Operators
Hypergraph neural networks (HNNs) using neural networks to encode hypergraphs provide a promising way to model higher-order relations in data and further solve relevant prediction tasks built upon such higher-order relations.
Radiomics-Guided Global-Local Transformer for Weakly Supervised Pathology Localization in Chest X-Rays
Using the learned self-attention of its image branch, RGT extracts a bounding box for which to compute radiomic features, which are further processed by the radiomics branch; learned image and radiomic features are then fused and mutually interact via cross-attention layers.
Neural Implicit Dictionary via Mixture-of-Expert Training
In this paper, we present a generic INR framework that achieves both data and training efficiency by learning a Neural Implicit Dictionary (NID) from a data collection and representing INR as a functional combination of basis sampled from the dictionary.
More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity
Transformers have quickly shined in the computer vision world since the emergence of Vision Transformers (ViTs).
Partial and Asymmetric Contrastive Learning for Out-of-Distribution Detection in Long-Tailed Recognition
However, in real-world applications, it is common for the training sets to have long-tailed distributions.
How Robust is Your Fairness? Evaluating and Sustaining Fairness under Unseen Distribution Shifts
Increasing concerns have been raised on deep learning fairness in recent years.
Aug-NeRF: Training Stronger Neural Radiance Fields with Triple-Level Physically-Grounded Augmentations
Inspired by that, we propose Augmented NeRF (Aug-NeRF), which for the first time brings the power of robust data augmentations into regularizing the NeRF training.
Removing Batch Normalization Boosts Adversarial Training
In addition, NoFrost achieves a $23. 56\%$ adversarial robustness against PGD attack, which improves the $13. 57\%$ robustness in BN-based AT.
Training Your Sparse Neural Network Better with Any Mask
Pruning large neural networks to create high-quality, independently trainable sparse masks, which can maintain similar performance to their dense counterparts, is very desirable due to the reduced space and time complexity.
Linearity Grafting: Relaxed Neuron Pruning Helps Certifiable Robustness
Certifiable robustness is a highly desirable property for adopting deep neural networks (DNNs) in safety-critical scenarios, but often demands tedious computations to establish.
Queried Unlabeled Data Improves and Robustifies Class-Incremental Learning
Inspired by the recent success of learning robust models with unlabeled data, we explore a new robustness-aware CIL setting, where the learned adversarial robustness has to resist forgetting and be transferred as new tasks come in continually.
Can pruning improve certified robustness of neural networks?
Given the fact that neural networks are often over-parameterized, one effective way to reduce such computational overhead is neural network pruning, by removing redundant parameters from trained neural networks.
A Multi-purpose Realistic Haze Benchmark with Quantifiable Haze Levels and Ground Truth
The full dataset presented in this paper, including the ground truth object classification bounding boxes and haze density measurements, is provided for the community to evaluate their algorithms at: https://a2i2-archangel. vision.
Data-Efficient Double-Win Lottery Tickets from Robust Pre-training
For example, on downstream CIFAR-10/100 datasets, we identify double-win matching subnetworks with the standard, fast adversarial, and adversarial pre-training from ImageNet, at 89. 26%/73. 79%, 89. 26%/79. 03%, and 91. 41%/83. 22% sparsity, respectively.
DiSparse: Disentangled Sparsification for Multitask Model Compression
We analyzed the pruning masks generated with DiSparse and observed strikingly similar sparse network architecture identified by each task even before the training starts.
VideoINR: Learning Video Implicit Neural Representation for Continuous Space-Time Super-Resolution
The learned implicit neural representation can be decoded to videos of arbitrary spatial resolution and frame rate.
Space-time Video Super-resolution
Video Frame Interpolation
+1
E^2VTS: Energy-Efficient Video Text Spotting from Unmanned Aerial Vehicles
Unmanned Aerial Vehicles (UAVs) based video text spotting has been extensively used in civil and military domains.
Quarantine: Sparsity Can Uncover the Trojan Attack Trigger for Free
Trojan attacks threaten deep neural networks (DNNs) by poisoning them to behave normally on most samples, yet to produce manipulated results for inputs attached with a particular trigger.
Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysis
Advanced deep neural networks (DNNs), designed by either human or AutoML algorithms, are growing increasingly complex.
Grasping the Arrow of Time from the Singularity: Decoding Micromotion in Low-dimensional Latent Spaces from StyleGAN
To study the motion features in the latent space of StyleGAN, in this paper, we hypothesize and demonstrate that a series of meaningful, natural, and versatile small, local movements (referred to as "micromotion", such as expression, head movement, and aging effect) can be represented in low-rank spaces extracted from the latent space of a conventionally pre-trained StyleGAN-v2 model for face generation, with the guidance of proper "anchors" in the form of either short text or video clips.
E^2TAD: An Energy-Efficient Tracking-based Action Detector
Video action detection (spatio-temporal action localization) is usually the starting point for human-centric intelligent analysis of videos nowadays.
Unified Implicit Neural Stylization
Representing visual signals by implicit representation (e. g., a coordinate based deep network) has prevailed among many vision tasks.
APP: Anytime Progressive Pruning
With the latest advances in deep learning, there has been a lot of focus on the online learning paradigm due to its relevance in practical settings.
SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image
Despite the rapid development of Neural Radiance Field (NeRF), the necessity of dense covers largely prohibits its wider applications.
VFDS: Variational Foresight Dynamic Selection in Bayesian Neural Networks for Efficient Human Activity Recognition
At its core is an implicit variational distribution on binary gates that are dependent on previous observations, which will select the next subset of features to observe.
On the Neural Tangent Kernel Analysis of Randomly Pruned Neural Networks
It is shown that given a pruning probability, for fully-connected neural networks with the weights randomly pruned at the initialization, as the width of each layer grows to infinity sequentially, the NTK of the pruned neural network converges to the limiting NTK of the original network with some extra scaling.
Efficient Split-Mix Federated Learning for On-Demand and In-Situ Customization
In this paper, we propose a novel Split-Mix FL strategy for heterogeneous participants that, once training is done, provides in-situ customization of model sizes and robustness.
Symbolic Learning to Optimize: Towards Interpretability and Scalability
Recent studies on Learning to Optimize (L2O) suggest a promising path to automating and accelerating the optimization procedure for complicated tasks.
The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancy
However, a "head-to-toe assessment" regarding the extent of redundancy in ViTs, and how much we could gain by thoroughly mitigating such, has been absent for this field.
Optimizer Amalgamation
Selecting an appropriate optimizer for a given problem is of major interest for researchers and practitioners.
Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice
The first technique, termed AttnScale, decomposes a self-attention block into low-pass and high-pass components, then rescales and combines these two filters to produce an all-pass self-attention matrix.
Auto-scaling Vision Transformers without Training
The motivation comes from two pain spots: 1) the lack of efficient and principled methods for designing and scaling ViTs; 2) the tremendous computational cost of training ViT that is much heavier than its convolution counterpart.
Sparsity Winning Twice: Better Robust Generalization from More Efficient Training
We introduce two alternatives for sparse adversarial training: (i) static sparsity, by leveraging recent results from the lottery ticket hypothesis to identify critical sparse subnetworks arising from the early training; (ii) dynamic sparsity, by allowing the sparse subnetwork to adaptively adjust its connectivity pattern (while sticking to the same sparsity ratio) throughout training.
Coarsening the Granularity: Towards Structurally Sparse Lottery Tickets
The lottery ticket hypothesis (LTH) has shown that dense models contain highly sparse subnetworks (i. e., winning tickets) that can be trained in isolation to match full accuracy.
The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training
In this paper, we focus on sparse training and highlight a perhaps counter-intuitive finding, that random pruning at initialization can be quite powerful for the sparse training of modern neural networks.
VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformer
To the best of our knowledge, this is the first time quantization has been incorporated into ViT acceleration on FPGAs with the help of a fully automatic framework to guide the quantization strategy on the software side and the accelerator implementations on the hardware side given the target frame rate.
Bringing Your Own View: Graph Contrastive Learning without Prefabricated Data Augmentations
Accordingly, we have extended the prefabricated discrete prior in the augmentation set, to a learnable continuous prior in the parameter space of graph generators, assuming that graph priors per se, similar to the concept of image manifolds, can be learned by data generation.
Fast and High-Quality Image Denoising via Malleable Convolutions
These spatially-varying kernels are produced by an efficient predictor network running on a downsampled input, making them much more efficient to compute than per-pixel kernels produced by a full-resolution image, and also enlarging the network's receptive field compared with static kernels.
EI-CLIP: Entity-Aware Interventional Contrastive Learning for E-Commerce Cross-Modal Retrieval
recommendation, and marketing services.
CADTransformer: Panoptic Symbol Spotting Transformer for CAD Drawings
CADTransformer tokenizes directly from the set of graphical primitives in CAD drawings, and correspondingly optimizes line-grained semantic and instance symbol spotting altogether by a pair of prediction heads.
Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better
Federated learning (FL) enables distribution of machine learning workloads from the cloud to resource-limited edge devices.
A Simple Single-Scale Vision Transformer for Object Localization and Instance Segmentation
In this paper, we comprehensively study three architecture design choices on ViT -- spatial reduction, doubled channels, and multiscale features -- and demonstrate that a vanilla ViT architecture can fulfill this goal without handcrafting multiscale features, maintaining the original ViT design philosophy.
Auto-X3D: Ultra-Efficient Video Understanding via Finer-Grained Neural Architecture Search
Efficient video architecture is the key to deploying video recognition systems on devices with limited computing resources.
Cold Brew: Distilling Graph Node Representations with Incomplete or Missing Neighborhoods
We propose Cold Brew, a teacher-student distillation approach to address the SCS and noisy-neighbor challenges for GNNs.
Improving Contrastive Learning on Imbalanced Seed Data via Open-World Sampling
Contrastive learning approaches have achieved great success in learning visual representations with few labels of the target classes.
You are caught stealing my winning lottery ticket! Making a lottery ticket claim its ownership
The lottery ticket hypothesis (LTH) emerges as a promising framework to leverage a special sparse subnetwork (i. e., winning ticket) instead of a full model for both training and inference, that can lower both costs without sacrificing the performance.
DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models
To address these pain points, we propose a framework for resource- and parameter-efficient fine-tuning by leveraging the sparsity prior in both weight updates and the final model weights.
Hyperparameter Tuning is All You Need for LISTA
Learned Iterative Shrinkage-Thresholding Algorithm (LISTA) introduces the concept of unrolling an iterative algorithm and training it like a neural network.
Delayed Propagation Transformer: A Universal Computation Engine towards Practical Control in Cyber-Physical Systems
This paper presents the Delayed Propagation Transformer (DePT), a new transformer-based model that specializes in the global modeling of CPS while taking into account the immutable constraints from the physical world.
AugMax: Adversarial Composition of Random Augmentations for Robust Training
Diversity and hardness are two complementary dimensions of data augmentation to achieve robustness.
Towards Lifelong Learning of Multilingual Text-To-Speech Synthesis
This work presents a lifelong learning approach to train a multilingual Text-To-Speech (TTS) system, where each language was seen as an individual task and was learned sequentially and continually.
Universality of Winning Tickets: A Renormalization Group Perspective
Foundational work on the Lottery Ticket Hypothesis has suggested an exciting corollary: winning tickets found in the context of one task can be transferred to similar tasks, possibly even across different architectures.
CheXT: Knowledge-Guided Cross-Attention Transformer for Abnormality Classification and Localization in Chest X-rays
During training, the image branch leverages its learned attention to estimate pathology localization, which is then utilized to extract radiomic features from images in the radiomics branch.
Generalizable Learning to Optimize into Wide Valleys
Learning to optimize (L2O) has gained increasing popularity in various optimization tasks, since classical optimizers usually require laborious, problem-specific design and hyperparameter tuning.
Bayesian Modeling and Uncertainty Quantification for Learning to Optimize: What, Why, and How
Optimizing an objective function with uncertainty awareness is well-known to improve the accuracy and confidence of optimization solutions.
Audio Lottery: Speech Recognition Made Ultra-Lightweight, Noise-Robust, and Transferable
In this paper, we investigate the tantalizing possibility of using lottery ticket hypothesis to discover lightweight speech recognition models, that are (1) robust to various noise existing in speech; (2) transferable to fit the open-world personalization; and 3) compatible with structured sparsity.
Universality of Deep Neural Network Lottery Tickets: A Renormalization Group Perspective
Foundational work on the Lottery Ticket Hypothesis has suggested an exciting corollary: winning tickets found in the context of one task can be transferred to similar tasks, possibly even across different architectures.
Lottery Tickets can have Structural Sparsity
The lottery ticket hypothesis (LTH) has shown that dense models contain highly sparse subnetworks (i. e., $\textit{winning tickets}$) that can be trained in isolation to match full accuracy.
Learning Pruning-Friendly Networks via Frank-Wolfe: One-Shot, Any-Sparsity, And No Retraining
Conventional methods often require (iterative) pruning followed by re-training, which not only incurs large overhead beyond the original DNN training but also can be sensitive to retraining hyperparameters.
Lottery Image Prior
Drawing inspirations from the recently prosperous research on lottery ticket hypothesis (LTH), we conjecture and study a novel “lottery image prior” (LIP), stated as: given an (untrained or trained) DNN-based image prior, it will have a sparse subnetwork that can be training in isolation, to match the original DNN’s performance when being applied as a prior to various image inverse problems.
Peek-a-Boo: What (More) is Disguised in a Randomly Weighted Neural Network, and How to Find It Efficiently
In this work, we define an extended class of subnetworks in randomly initialized NNs called disguised subnetworks, which are not only "hidden" in the random networks but also "disguised" -- hence can only be "unmasked" with certain transformations on weights.
Reasoning With Hierarchical Symbols: Reclaiming Symbolic Policies For Visual Reinforcement Learning
Deep vision models are nowadays widely integrated into visual reinforcement learning (RL) to parameterize the policy networks.
AutoCoG: A Unified Data-Modal Co-Search Framework for Graph Neural Networks
Despite the preliminary success, we argue that for GNNs, NAS has to be customized further, due to the topological complicacy of GNN input data (graph) as well as the notorious training instability.
Scaling the Depth of Vision Transformers via the Fourier Domain Analysis
The first technique, termed AttnScale, decomposes a self-attention block into low-pass and high-pass components, then rescales and combines these two filters to produce an all-pass self-attention matrix.
Stingy Teacher: Sparse Logits Suffice to Fail Knowledge Distillation
However, it is unclear why the distorted distribution of the logits is catastrophic to the student model.
Equalized Robustness: Towards Sustainable Fairness Under Distributional Shifts
In this paper, we first propose a new fairness goal, termed Equalized Robustness (ER), to impose fair model robustness against unseen distribution shifts across majority and minority groups.
Skeleton-Graph: Long-Term 3D Motion Prediction From 2D Observations Using Deep Spatio-Temporal Graph CNNs
We propose Skeleton-Graph, a deep spatio-temporal graph CNN model that predicts the future 3D skeleton poses in a single pass from the 2D ones.
Ranked #1 on
Trajectory Prediction
on PROX
GDP: Stabilized Neural Network Pruning via Gates with Differentiable Polarization
During the training process, the polarization effect will drive a subset of gates to smoothly decrease to exact zero, while other gates gradually stay away from zero by a large margin.
Font Completion and Manipulation by Cycling Between Multi-Modality Representations
The novel graph constructor maps a glyph's latent code to its graph representation that matches expert knowledge, which is trained to help the translation task.
Searching for Two-Stream Models in Multivariate Space for Video Recognition
We design a multivariate search space, including 6 search variables to capture a wide variety of choices in designing two-stream models.
Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics
This work targets designing a principled and unified training-free framework for Neural Architecture Search (NAS), with high performance, low cost, and in-depth interpretation.
Bag of Tricks for Training Deeper Graph Neural Networks: A Comprehensive Benchmark Study
In view of those, we present the first fair and reproducible benchmark dedicated to assessing the "tricks" of training deep GNNs.
SSH: A Self-Supervised Framework for Image Harmonization
Image harmonization aims to improve the quality of image compositing by matching the "appearance" (\eg, color tone, brightness and contrast) between foreground and background images.
Federated Adversarial Debiasing for Fair and Transferable Representations
While adversarial learning is commonly used in centralized learning for mitigating bias, there are significant barriers when extending it to the federated framework.
CERL: A Unified Optimization Framework for Light Enhancement with Realistic Noise
We present \underline{C}oordinated \underline{E}nhancement for \underline{R}eal-world \underline{L}ow-light Noisy Images (CERL), that seamlessly integrates light enhancement and noise suppression parts into a unified and physics-grounded optimization framework.
Black-Box Diagnosis and Calibration on GAN Intra-Mode Collapse: A Pilot Study
Existing diversity tests of samples from GANs are usually conducted qualitatively on a small scale, and/or depends on the access to original training data as well as the trained model parameters.
DANCE: DAta-Network Co-optimization for Efficient Segmentation Model Training and Inference
Semantic segmentation for scene understanding is nowadays widely demanded, raising significant challenges for the algorithm efficiency, especially its applications on resource-limited platforms.
Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?
Based on our analysis, we summarize a guideline for parameter settings in regards of specific architecture characteristics, which we hope to catalyze the research progress on the topic of lottery ticket hypothesis.
Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity
Our framework, FreeTickets, is defined as the ensemble of these relatively cheap sparse subnetworks.
Scalable Perception-Action-Communication Loops with Convolutional and Graph Neural Networks
Our framework is implemented by a cascade of a convolutional and a graph neura