Search Results for author:
Found 383 papers, 236 papers with code
HALO: Hardware-Aware Learning to Optimize
There has been an explosive demand for bringing machine learning (ML) powered intelligence into numerous Internet-of-Things (IoT) devices.
Eliminating the Invariance on the Loss Landscape of Linear Autoencoders
For this new loss, we characterize the full structure of the loss landscape in the following sense: we establish analytical expression for the set of all critical points, show that it is a subset of critical points of MSE, and that all local minima are still global.
A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs
Vision-language models (VLMs) have shown remarkable success across various multi-modal tasks, yet large VLMs encounter significant efficiency challenges due to processing numerous visual tokens.
Oscillation Inversion: Understand the structure of Large Flow Model through the Lens of Inversion Method
We explore the oscillatory behavior observed in inversion methods applied to large-scale text-to-image diffusion models, with a focus on the "Flux" model.
Know Where You're Uncertain When Planning with Multimodal Foundation Models: A Formal Framework
To quantify each type of uncertainty, we propose methods tailored to the unique properties of perception and decision-making: we use conformal prediction to calibrate perception uncertainty and introduce Formal-Methods-Driven Prediction (FMDP) to quantify decision uncertainty, leveraging formal verification techniques for theoretical guarantees.
Chasing Better Deep Image Priors between Over- and Under-parameterization
Besides, we also extend LIP to compressive sensing image reconstruction, where a pre-trained GAN generator is used as the prior (in contrast to untrained DIP or deep decoder), and confirm its validity in this setting too.
Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design
The proliferation of large language models (LLMs) has led to the adoption of Mixture-of-Experts (MoE) architectures that dynamically leverage specialized subnetworks for improved efficiency and performance.
Large Spatial Model: End-to-end Unposed Images to Semantic 3D
To tackle the scarcity of labeled 3D semantic data and enable natural language-driven scene manipulation, we incorporate a pre-trained 2D language-based segmentation model into a 3D-consistent semantic feature field.
AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models
Existing LLM pruning strategies typically assign uniform pruning ratios across layers, limiting overall pruning ability; and recent work on layerwise pruning of LLMs is often based on heuristics that can easily lead to suboptimal performance.
Cavia: Camera-controllable Multi-view Video Diffusion with View-Integrated Attention
Recent studies have attempted to incorporate camera control into the generation process, but their results are often limited to simple trajectories or lack the ability to generate consistent videos from multiple distinct camera paths for the same scene.
Training Dynamics of Transformers to Recognize Word Co-occurrence via Gradient Flow Analysis
In Phase 2, the attention matrices and the MLP evolve jointly to enlarge the classification margin and reduce the loss to a near minimum value.
Model-GLUE: Democratized LLM Scaling for A Large Model Zoo in the Wild
As Large Language Models (LLMs) excel across tasks and specialized domains, scaling LLMs based on existing models has garnered significant attention, which faces the challenge of decreasing performance when combining disparate models.
On The Planning Abilities of OpenAI's o1 Models: Feasibility, Optimality, and Generalizability
Recent advancements in Large Language Models (LLMs) have showcased their ability to perform complex reasoning tasks, but their effectiveness in planning remains underexplored.
LLM-PBE: Assessing Data Privacy in Large Language Models
Large Language Models (LLMs) have become integral to numerous domains, significantly advancing applications in data management, mining, and analysis.
All Against Some: Efficient Integration of Large Language Models for Message Passing in Graph Neural Networks
In this paper, we investigate how LLMs can be leveraged in a computationally efficient fashion to benefit rich graph-structured data, a modality relatively unexplored in LLM literature.
From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from Low-Rank Gradients
Modern Large Language Models (LLMs) are composed of matrices with billions of elements, making their storage and processing quite demanding in terms of computational resources and memory usage.
Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients
To address these limitations, we introduce Q-Galore, a novel approach that substantially reduces memory usage by combining quantization and low-rank projection, surpassing the benefits of GaLore.
Expressive Gaussian Human Avatars from Monocular RGB Video
Nuanced expressiveness, particularly through fine-grained hand and facial expressions, is pivotal for enhancing the realism and vitality of digital human representations.
4K4DGen: Panoramic 4D Generation at 4K Resolution
Subsequently, we propose \textbf{Dynamic Panoramic Lifting} to elevate the panoramic video into a 4D immersive environment while preserving spatial and temporal consistency.
Learning Traffic Crashes as Language: Datasets, Benchmarks, and What-if Causal Analyses
Leveraging this rich dataset, we further formulate the crash event feature learning as a novel text reasoning problem and further fine-tune various large language models (LLMs) to predict detailed accident outcomes, such as crash types, severity and number of injuries, based on contextual and environmental factors.
Flextron: Many-in-One Flexible Large Language Model
Training modern LLMs is extremely resource intensive, and customizing them for various deployment scenarios characterized by limited compute and memory resources through repeated training is impractical.
LoCoCo: Dropping In Convolutions for Long Context Compression
This paper tackles the memory hurdle of processing long context sequences in Large Language Models (LLMs), by presenting a novel approach, Dropping In Convolutions for Long Context Compression (LoCoCo).
Zero-Painter: Training-Free Layout Control for Text-to-Image Synthesis
We present Zero-Painter, a novel training-free framework for layout-conditional text-to-image synthesis that facilitates the creation of detailed and controlled imagery from textual prompts.
CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation
Recently video diffusion models have emerged as expressive generative tools for high-quality video content creation readily available to general users.
Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models
Building on this foundation, we propose a strategy to migrate the temporal consistency in video diffusion models to the spatial-temporal consistency required for 4D generation.
Harnessing the power of longitudinal medical imaging for eye disease prognosis using Transformer-based sequence modeling
Deep learning has enabled breakthroughs in automated diagnosis from medical imaging, with many successful applications in ophthalmology.
OpenBias: Open-set Bias Detection in Text-to-Image Generative Models
In this paper, we tackle the challenge of open-set bias detection in text-to-image generative models presenting OpenBias, a new pipeline that identifies and quantifies the severity of biases agnostically, without access to any precompiled set.
DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic Gaussian Splatting
This point cloud serves as the initial state for the centroids of 3D Gaussians.
MM3DGS SLAM: Multi-modal 3D Gaussian Splatting for SLAM Using Vision, Depth, and Inertial Measurements
We show for the first time that using 3D Gaussians for map representation with unposed camera images and inertial measurements can enable accurate SLAM.
InstantSplat: Sparse-view SfM-free Gaussian Splatting in Seconds
In this study, we introduce a novel and efficient framework to enhance robust NVS from sparse-view images.
Lift3D: Zero-Shot Lifting of Any 2D Vision Model to 3D
In recent years, there has been an explosion of 2D vision models for numerous tasks such as semantic segmentation, style transfer or scene editing, enabled by large-scale 2D image datasets.
Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study
Specifically, we investigate the impact of the number of data samples, the total number of experts, the sparsity in expert selection, the complexity of the routing mechanism, and the complexity of individual experts.
Comp4D: LLM-Guided Compositional 4D Scene Generation
Recent advancements in diffusion models for 2D and 3D content creation have sparked a surge of interest in generating 4D content.
StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text
To overcome these limitations, we introduce StreamingT2V, an autoregressive approach for long video generation of 80, 240, 600, 1200 or more frames with smooth transitions.
Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression
While state-of-the-art (SoTA) compression methods boast impressive advancements in preserving benign task performance, the potential risks of compression in terms of safety and trustworthiness have been largely neglected.
Shake to Leak: Fine-tuning Diffusion Models Can Amplify the Generative Privacy Risk
While diffusion models have recently demonstrated remarkable progress in generating realistic images, privacy risks also arise: published models or APIs could generate training images and thus leak privacy-sensitive training information.
GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
Our approach reduces memory usage by up to 65. 5% in optimizer states while maintaining both efficiency and performance for pre-training on LLaMA 1B and 7B architectures with C4 dataset with up to 19. 7B tokens, and on fine-tuning RoBERTa on GLUE tasks.
Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding
To address this problem, this paper introduces Multi-scale Positional Encoding (Ms-PoE) which is a simple yet effective plug-and-play approach to enhance the capacity of LLMs to handle the relevant information located in the middle of the context, without fine-tuning or introducing any additional overhead.
Principled Architecture-aware Scaling of Hyperparameters
However, most designs or optimization methods are agnostic to the choice of network structures, and thus largely ignore the impact of neural architectures on hyperparameters.
Take the Bull by the Horns: Hard Sample-Reweighted Continual Training Improves LLM Generalization
Our study starts from an empirical strategy for the light continual training of LLMs using their original pre-training data sets, with a specific focus on selective retention of samples that incur moderately high losses.
Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark
In the evolving landscape of natural language processing (NLP), fine-tuning pre-trained Large Language Models (LLMs) with first-order (FO) optimizers like SGD and Adam has become standard.
Social Reward: Evaluating and Enhancing Generative AI through Million-User Feedback from an Online Creative Community
These findings highlight the relevance and effectiveness of Social Reward in assessing community appreciation for AI-generated artworks, establishing a closer alignment with users' creative goals: creating popular visual art.
Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference
Many computational factors limit broader deployment of large language models.
LLaGA: Large Language and Graph Assistant
Graph Neural Networks (GNNs) have empowered the advance in graph-structured data analysis.
QuantumSEA: In-Time Sparse Exploration for Noise Adaptive Quantum Circuits
To address these two pain points, we propose QuantumSEA, an in-time sparse exploration for noise-adaptive quantum circuits, aiming to achieve two key objectives: (1) implicit circuits capacity during training - by dynamically exploring the circuit's sparse connectivity and sticking a fixed small number of quantum gates throughout the training which satisfies the coherence time and enjoy light noises, enabling feasible executions on real quantum devices; (2) noise robustness - by jointly optimizing the topology and parameters of quantum circuits under real device noise models.
AGG: Amortized Generative 3D Gaussians for Single Image to 3D
To overcome these challenges, we introduce an Amortized Generative 3D Gaussian framework (AGG) that instantly produces 3D Gaussians from a single image, eliminating the need for per-instance optimization.
VASE: Object-Centric Appearance and Shape Manipulation of Real Videos
Recently, several works tackled the video editing task fostered by the success of large-scale text-to-image generative models.
PAIR Diffusion: A Comprehensive Multimodal Object-Level Image Editor
We propose PAIR Diffusion a generic framework that enables a diffusion model to control the structure and appearance properties of each object in the image.
Taming Mode Collapse in Score Distillation for Text-to-3D Generation
In this paper, we reveal that the existing score distillation-based text-to-3D generation frameworks degenerate to maximal likelihood seeking on each view independently and thus suffer from the mode collapse problem, manifesting as the Janus artifact in practice.
SteinDreamer: Variance Reduction for Text-to-3D Score Distillation via Stein Identity
In this paper, we reveal that the gradient estimation in score distillation is inherent to high variance.
4DGen: Grounded 4D Content Generation with Spatial-temporal Consistency
Our pipeline facilitates controllable 4D generation, enabling users to specify the motion via monocular video or adopt image-to-video generations, thus offering superior control over content creation.
HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models
Recent progress in text-guided image inpainting, based on the unprecedented success of text-to-image diffusion models, has led to exceptionally realistic and visually plausible results.
The Counterattack of CNNs in Self-Supervised Learning: Larger Kernel Size might be All You Need
Therefore, it remains unclear whether the self-attention operation is crucial for the recent advances in SSL - or CNNs can deliver the same excellence with more advanced designs, too?
Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields
In this work, we go one step further: in addition to radiance field rendering, we enable 3D Gaussian splatting on arbitrary-dimension semantic features via 2D foundation model distillation.
Meta ControlNet: Enhancing Task Adaptation via Meta Learning
However, vanilla ControlNet generally requires extensive training of around 5000 steps to achieve a desirable control for a single task.
Rethinking PGD Attack: Is Sign Function Necessary?
Based on that, we further propose a new raw gradient descent (RGD) algorithm that eliminates the use of sign.
FSGS: Real-Time Few-shot View Synthesis using Gaussian Splatting
Novel view synthesis from limited observations remains an important and persistent task.
LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPS
Recent advances in real-time neural rendering using point-based techniques have enabled broader adoption of 3D representations.
Ranked #3 on
Novel View Synthesis
on Mip-NeRF 360
DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineer
To ensure that the prompts do not leak private information, we introduce the first private prompt generation mechanism, by a differentially-private (DP) ensemble of in-context learning with private demonstrations.
Fine-Tuning Language Models Using Formal Methods Feedback
Although pre-trained language models encode generic knowledge beneficial for planning and control, they may fail to generate appropriate control policies for domain-specific tasks.
Towards long-tailed, multi-label disease classification from chest X-ray: Overview of the CXR-LT challenge
Many real-world image recognition problems, such as diagnostic medical imaging exams, are "long-tailed" $\unicode{x2013}$ there are a few common findings followed by many more relatively rare conditions.
Multi-Concept T2I-Zero: Tweaking Only The Text Embeddings and Nothing Else
To achieve this goal, we identify the limitations in the text embeddings used for the pre-trained text-to-image diffusion models.
Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality
Dataset distillation aims to minimize the time and memory needed for training deep networks on large datasets, by creating a small set of synthetic images that has a similar generalization performance to that of the full dataset.
Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity
Large Language Models (LLMs), renowned for their remarkable performance across diverse domains, present a challenge when it comes to practical deployment due to their colossal model size.
Pose-Free Generalizable Rendering Transformer
To address this challenge, we introduce PF-GRT, a new Pose-Free framework for Generalizable Rendering Transformer, eliminating the need for pre-computed camera poses and instead leveraging feature-matching learned directly from data.
Efficient-3DiM: Learning a Generalizable Single-image Novel-view Synthesizer in One Day
Although the fidelity and generalizability are greatly improved, training such a powerful diffusion model requires a vast volume of training data and model parameters, resulting in a notoriously long time and high computational costs.
Do Compressed LLMs Forget Knowledge? An Experimental Study with Practical Implications
We start by proposing two conjectures on the nature of the damage: one is certain knowledge being forgotten (or erased) after LLM compression, hence necessitating the compressed model to (re)learn from data with additional parameters; the other presumes that knowledge is internally displaced and hence one requires merely "inference re-direction" with input-side augmentation such as prompting, to recover the knowledge-related performance.
Compressing LLMs: The Truth is Rarely Pure and Never Simple
Recently, several works have shown significant success in training-free and data-free compression (pruning and quantization) of LLMs that achieve 50 - 60% sparsity and reduce the bit width to 3 or 4 bits per weight, with negligible degradation of perplexity over the uncompressed baseline.
Pruning Small Pre-Trained Weights Irreversibly and Monotonically Impairs "Difficult" Downstream Tasks in LLMs
Contrary to this belief, this paper presents a counter-argument: small-magnitude weights of pre-trained model weights encode vital knowledge essential for tackling difficult downstream tasks - manifested as the monotonic relationship between the performance drop of downstream tasks across the difficulty spectrum, as we prune more pre-trained weights by magnitude.
Safe and Robust Watermark Injection with a Single OoD Image
Training a high-performance deep neural network requires large amounts of data and computational resources.
Enhancing NeRF akin to Enhancing LLMs: Generalizable NeRF Transformer with Mixture-of-View-Experts
Cross-scene generalizable NeRF models, which can directly synthesize novel views of unseen scenes, have become a new spotlight of the NeRF field.
Robust Mixture-of-Expert Training for Convolutional Neural Networks
Since the lack of robustness has become one of the main hurdles for CNNs, in this paper we ask: How to adversarially robustify a CNN-based MoE model?
How Does Pruning Impact Long-Tailed Multi-Label Medical Image Classifiers?
This work represents a first step toward understanding the impact of pruning on model behavior in deep long-tailed, multi-label medical image classification.
INR-Arch: A Dataflow Architecture and Compiler for Arbitrary-Order Gradient Computations in Implicit Neural Representation Processing
In this work, we introduce INR-Arch, a framework that transforms the computation graph of an nth-order gradient into a hardware-optimized dataflow architecture.
Doubly Robust Instance-Reweighted Adversarial Training
Experiments on standard classification datasets demonstrate that our proposed approach outperforms related state-of-the-art baseline methods in terms of average robust performance, and at the same time improves the robustness against attacks on the weakest data points.
Physics-Driven Turbulence Image Restoration with Stochastic Refinement
Although fast and physics-grounded simulation tools have been introduced to help the deep-learning models adapt to real-world turbulence conditions recently, the training of such models only relies on the synthetic data and ground truth pairs.
Reference-based Painterly Inpainting via Diffusion: Crossing the Wild Reference Domain Gap
We propose Reference-based Painterly Inpainting, a novel task that crosses the wild reference domain gap and implants novel objects into artworks.
Polynomial Width is Sufficient for Set Representation with High-dimensional Features
To investigate the minimal value of $L$ that achieves sufficient expressive power, we present two set-element embedding layers: (a) linear + power activation (LP) and (b) linear + exponential activations (LE).
Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunities
Recently, zero-shot (or training-free) Neural Architecture Search (NAS) approaches have been proposed to liberate NAS from the expensive training process.
FarSight: A Physics-Driven Whole-Body Biometric System at Large Distance and Altitude
Whole-body biometric recognition is an important area of research due to its vast applications in law enforcement, border security, and surveillance.
H$_2$O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
Based on these insights, we propose Heavy Hitter Oracle (H$_2$O), a KV cache eviction policy that dynamically retains a balance of recent and H$_2$ tokens.
Graph Ladling: Shockingly Simple Parallel GNN Training without Intermediate Communication
By dividing giant graph data, we build multiple independently and parallelly trained weaker GNNs (soup ingredient) without any intermediate communication, and combine their strength using a greedy interpolation soup procedure to achieve state-of-the-art performance.
Instant Soup: Cheap Pruning Ensembles in A Single Pass Can Draw Lottery Tickets from Large Models
Motivated by the recent observations of model soups, which suggest that fine-tuned weights of multiple models can be merged to a better minima, we propose Instant Soup Pruning (ISP) to generate lottery ticket quality subnetworks, using a fraction of the original IMP cost by replacing the expensive intermediate pruning stages of IMP with computationally efficient weak mask generation and aggregation routine.
Learning to Estimate 6DoF Pose from Limited Data: A Few-Shot, Generalizable Approach using RGB Images
The accurate estimation of six degrees-of-freedom (6DoF) object poses is essential for many applications in robotics and augmented reality.
The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter
Large pre-trained transformers are show-stealer in modern-day deep learning, and it becomes crucial to comprehend the parsimonious patterns that exist within them as they grow in scale.
Are Large Kernels Better Teachers than Transformers for ConvNets?
We hereby carry out a first-of-its-kind study unveiling that modern large-kernel ConvNets, a compelling competitor to Vision Transformers, are remarkably more effective teachers for small-kernel ConvNets, due to more similar architectures.
Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Experts
Computer vision researchers are embracing two promising paradigms: Vision Transformers (ViTs) and Multi-task Learning (MTL), which both show great performance but are computation-intensive, given the quadratic complexity of self-attention in ViT and the need to activate an entire large MTL model for one task.
Towards Constituting Mathematical Structures for Learning to Optimize
Learning to Optimize (L2O), a technique that utilizes machine learning to learn an optimization algorithm automatically from data, has gained arising attention in recent years.
Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models
Text-to-image (T2I) research has grown explosively in the past year, owing to the large-scale pre-trained diffusion models and many emerging personalization and editing approaches.
POPE: 6-DoF Promptable Pose Estimation of Any Object, in Any Scene, with One Reference
To mitigate this issue, we propose a general paradigm for object pose estimation, called Promptable Object Pose Estimation (POPE).
MMG-Ego4D: Multi-Modal Generalization in Egocentric Action Recognition
In this paper, we study a novel problem in egocentric action recognition, which we term as "Multimodal Generalization" (MMG).
In-Context Learning Unlocked for Diffusion Models
We present Prompt Diffusion, a framework for enabling in-context learning in diffusion-based generative models.
Outline, Then Details: Syntactically Guided Coarse-To-Fine Code Generation
For a complicated algorithm, its implementation by a human programmer usually starts with outlining a rough control flow followed by iterative enrichments, eventually yielding carefully generated syntactic structures and variables in a hierarchy.
Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Models
Patch Diffusion meanwhile improves the performance of diffusion models trained on relatively small datasets, $e. g.$, as few as 5, 000 images to train from scratch.
AutoShot: A Short Video Dataset and State-of-the-Art Shot Boundary Detection
For video content creation and understanding, the shot boundary detection (SBD) is one of the most essential components in various scenarios.
Ranked #1 on
Camera shot boundary detection
on ClipShots
PAIR-Diffusion: A Comprehensive Multimodal Object-Level Image Editor
We propose PAIR Diffusion, a generic framework that can enable a diffusion model to control the structure and appearance properties of each object in the image.
Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models
The unlearning problem of deep learning models, once primarily an academic concern, has become a prevalent issue in the industry.
Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators
Recent text-to-video generation approaches rely on computationally heavy training and require large-scale video datasets.
Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!
In pursuit of a more general evaluation and unveiling the true potential of sparse algorithms, we introduce "Sparsity May Cry" Benchmark (SMC-Bench), a collection of carefully-curated 4 diverse tasks with 10 datasets, that accounts for capturing a wide range of domain-specific and sophisticated knowledge.
Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers
Despite their remarkable achievement, gigantic transformers encounter significant drawbacks, including exorbitant computational and memory footprints during training, as well as severe collapse evidenced by a high degree of parameter redundancy.
Learning to Grow Pretrained Models for Efficient Transformer Training
Scaling transformers has led to significant breakthroughs in many domains, leading to a paradigm in which larger versions of existing models are trained and released on a periodic basis.
M-L2O: Towards Generalizable Learning-to-Optimize by Test-Time Fast Self-Adaptation
This data-driven procedure yields L2O that can efficiently solve problems similar to those seen in training, that is, drawn from the same ``task distribution".
Layer Grafted Pre-training: Bridging Contrastive Learning And Masked Image Modeling For Label-Efficient Representations
This motivates us to shift the paradigm from combining loss at the end, to choosing the proper learning method per network layer.
You Only Transfer What You Share: Intersection-Induced Graph Transfer Learning for Link Prediction
We identify this setting as Graph Intersection-induced Transfer Learning (GITL), which is motivated by practical applications in e-commerce or academic co-authorship predictions.
Robust Weight Signatures: Gaining Robustness as Easy as Patching Weights?
Given a robust model trained to be resilient to one or multiple types of distribution shifts (e. g., natural image corruptions), how is that "robustness" encoded in the model weights, and how easily can it be disentangled and/or "zero-shot" transferred to some other models?
Learning to Generalize Provably in Learning to Optimize
While the optimizer generalization has been recently studied, the optimizee generalization (or learning to generalize) has not been rigorously studied in the L2O context, which is the aim of this paper.
Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researchers
In response, we summarize ten Q\&As of SNNs from many key aspects, including dense vs. sparse, unstructured sparse vs. structured sparse, pruning vs. sparse training, dense-to-sparse training vs. sparse-to-sparse training, static sparsity vs. dynamic sparsity, before-training/during-training vs. post-training sparsity, and many more.
Turning the Curse of Heterogeneity in Federated Learning into a Blessing for Out-of-Distribution Detection
We propose to take advantage of such heterogeneity and turn the curse into a blessing that facilitates OoD detection in FL.
Evaluate underdiagnosis and overdiagnosis bias of deep learning model on primary open-angle glaucoma diagnosis in under-served patient populations
In the United States, primary open-angle glaucoma (POAG) is the leading cause of blindness, especially among African American and Hispanic individuals.
Neural Networks with Sparse Activation Induced by Large Bias: Tighter Analysis with Bias-Generalized NTK
To our knowledge, this is the first sparsity-dependent generalization result via Rademacher complexity.
Specialist Diffusion: Plug-and-Play Sample-Efficient Fine-Tuning of Text-to-Image Diffusion Models To Learn Any Unseen Style
Diffusion models have demonstrated impressive capability of text-conditioned image synthesis, and broader application horizons are emerging by personalizing those pretrained diffusion models toward generating some specialized target object or style.
NeuralLift-360: Lifting an In-the-Wild 2D Photo to a 3D Object With 360deg Views
In this work, we study the challenging task of lifting a single image to a 3D object and, for the first time, demonstrate the ability to generate a plausible 3D object with 360deg views that corresponds well with the given reference image.
AdaMV-MoE: Adaptive Multi-Task Vision Mixture-of-Experts
Instead of compressing multiple tasks' knowledge into a single model, MoE separates the parameter space and only utilizes the relevant model pieces given task type and its input, which provides stabilized MTL training and ultra-efficient inference.
Vision HGNN: An Image is More than a Graph of Nodes
In this paper, we enhance ViG by transcending conventional "pairwise" linkages and harnessing the power of the hypergraph to encapsulate image information.
MMG-Ego4D: Multimodal Generalization in Egocentric Action Recognition
In this paper, we study a novel problem in egocentric action recognition, which we term as "Multimodal Generalization" (MMG).
Pruning Before Training May Improve Generalization, Provably
It is shown that as long as the pruning fraction is below a certain threshold, gradient descent can drive the training loss toward zero and the network exhibits good generalization performance.
Symbolic Visual Reinforcement Learning: A Scalable Framework with Object-Level Abstraction and Differentiable Expression Search
Learning efficient and interpretable policies has been a challenging task in reinforcement learning (RL), particularly in the visual RL setting with complex scenes.
Attend Who is Weak: Pruning-assisted Medical Image Localization under Sophisticated and Implicit Imbalances
However, DNNs are notoriously fragile to the class imbalance in image classification.
StegaNeRF: Embedding Invisible Information within Neural Radiance Fields
Recent advances in neural rendering imply a future of widespread visual data distributions through sharing NeRF model weights.
NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with 360° Views
In this work, we study the challenging task of lifting a single image to a 3D object and, for the first time, demonstrate the ability to generate a plausible 3D object with 360{\deg} views that correspond well with the given reference image.
You Can Have Better Graph Neural Networks by Not Training Weights at All: Finding Untrained GNNs Tickets
Recent works have impressively demonstrated that there exists a subnetwork in randomly initialized convolutional neural networks (CNNs) that can match the performance of the fully trained dense networks at initialization, without any optimization of the weights of the network (i. e., untrained networks).
Search Behavior Prediction: A Hypergraph Perspective
With these hyperedges, we augment the original bipartite graph into a new \textit{hypergraph}.
Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training
Vision transformers (ViTs) have recently obtained success in many applications, but their intensive computation and heavy memory usage at both training and inference time limit their generalization.
AligNeRF: High-Fidelity Neural Radiance Fields via Alignment-Aware Training
Neural Radiance Fields (NeRFs) are a powerful representation for modeling a 3D scene as a continuous function.
Versatile Diffusion: Text, Images and Variations All in One Diffusion Model
In this work, we expand the existing single-flow diffusion pipeline into a multi-task multimodal network, dubbed Versatile Diffusion (VD), that handles multiple flows of text-to-image, image-to-text, and variations in one unified model.
StyleNAT: Giving Each Head a New Perspective
Image generation has been a long sought-after but challenging task, and performing the generation task in an efficient manner is similarly difficult.
Ranked #2 on
Image Generation
on FFHQ 256 x 256
QuanGCN: Noise-Adaptive Training for Robust Quantum Graph Convolutional Networks
Quantum neural networks (QNNs), an interdisciplinary field of quantum computing and machine learning, have attracted tremendous research interests due to the specific quantum advantages.
Scaling Multimodal Pre-Training via Cross-Modality Gradient Harmonization
For example, even in the commonly adopted instructional videos, a speaker can sometimes refer to something that is not visually present in the current frame; and the semantic misalignment would only be more unpredictable for the raw videos from the internet.
Sparse Winning Tickets are Data-Efficient Image Recognizers
Improving the performance of deep networks in data-limited regimes has warranted much attention.
M$^3$ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design
However, when deploying MTL onto those real-world systems that are often resource-constrained or latency-sensitive, two prominent challenges arise: (i) during training, simultaneously optimizing all tasks is often difficult due to gradient conflicts across tasks; (ii) at inference, current MTL regimes have to activate nearly the entire model even to just execute a single task.
Symbolic Distillation for Learned TCP Congestion Control
At the core of our proposal is a novel symbolic branching algorithm that enables the rule to be aware of the context in terms of various network conditions, eventually converting the NN policy into a symbolic tree.
Signal Processing for Implicit Neural Representations
We answer this question by proposing an implicit neural signal processing network, dubbed INSP-Net, via differential operators on INR.
Data-Model-Circuit Tri-Design for Ultra-Light Video Intelligence on Edge Devices
In this paper, we propose a data-model-hardware tri-design framework for high-throughput, low-cost, and high-accuracy multi-object tracking (MOT) on High-Definition (HD) video stream.
RoS-KD: A Robust Stochastic Knowledge Distillation Approach for Noisy Medical Imaging
Our extensive experiments on popular medical imaging classification tasks (cardiopulmonary disease and lesion classification) using real-world datasets, show the performance benefit of RoS-KD, its ability to distill knowledge from many popular large networks (ResNet-50, DenseNet-121, MobileNet-V2) in a comparatively small network, and its robustness to adversarial attacks (PGD, FSGM).
Old can be Gold: Better Gradient Flow can Make Vanilla-GCNs Great Again
In this paper, firstly, we provide a new perspective of gradient flow to understand the substandard performance of deep GCNs and hypothesize that by facilitating healthy gradient flow, we can significantly improve their trainability, as well as achieve state-of-the-art (SOTA) level performance from vanilla-GCNs.
A Comprehensive Study on Large-Scale Graph Training: Benchmarking and Rethinking
Large-scale graph training is a notoriously challenging problem for graph neural networks (GNNs).
Ranked #2 on
Node Property Prediction
on ogbn-products
Trap and Replace: Defending Backdoor Attacks by Trapping Them into an Easy-to-Replace Subnetwork
As a result, both the stem and the classification head in the final network are hardly affected by backdoor training samples.
Augmentations in Hypergraph Contrastive Learning: Fabricated and Generative
This paper targets at improving the generalizability of hypergraph neural networks in the low-label regime, through applying the contrastive learning approach from images/graphs (we refer to it as HyperGCL).
DynImp: Dynamic Imputation for Wearable Sensing Data Through Sensory and Temporal Relatedness
We argue that traditional methods have rarely made use of both times-series dynamics of the data as well as the relatedness of the features from different sensors.
NeRF-SOS: Any-View Self-supervised Object Segmentation on Complex Scenes
Our framework, called NeRF with Self-supervised Object Segmentation NeRF-SOS, couples object segmentation and neural radiance field to segment objects in any view within a scene.
Can We Solve 3D Vision Tasks Starting from A 2D Vision Transformer?
Vision Transformers (ViTs) have proven to be effective, in solving 2D image understanding tasks by training over large-scale image datasets; and meanwhile as a somehow separate track, in modeling the 3D visual world too such as voxels or point clouds.
Long-Tailed Classification of Thorax Diseases on Chest X-Ray: A New Benchmark Study
Imaging exams, such as chest radiography, will yield a small set of common findings and a much larger set of uncommon findings.
Ranked #1 on
Long-tail Learning
on MIMIC-CXR-LT