Search Results for author:
Found 54 papers, 23 papers with code
Captions Speak Louder than Images (CASLIE): Generalizing Foundation Models for E-commerce from High-quality Multimodal Instruction Data
Leveraging multimodal data to drive breakthroughs in e-commerce applications through Multimodal Foundation Models (MFMs) is gaining increasing attention from the research community.
Robust Low-rank Tensor Train Recovery
We first establish the $\ell_1/\ell_2$-restricted isometry property (RIP) for Gaussian measurement operators, demonstrating that the information in the TT format tensor can be preserved using a number of measurements that grows linearly with $N$.
On Layer-wise Representation Similarity: Application for Multi-Exit Models with a Single Classifier
Our experimental results on common transformers reveal that representations across layers are positively correlated, albeit the similarity decreases when layers are far apart.
Computational and Statistical Guarantees for Tensor-on-Tensor Regression with Tensor Train Decomposition
However, the exponential growth in tensor complexity poses challenges for storage and computation in ToT regression.
A Global Geometric Analysis of Maximal Coding Rate Reduction
The maximal coding rate reduction (MCR$^2$) objective for learning structured and compact deep representations is drawing increasing attention, especially after its recent usage in the derivation of fully explainable and highly effective deep network architectures.
AdaContour: Adaptive Contour Descriptor with Hierarchical Representation
Existing angle-based contour descriptors suffer from lossy representation for non-starconvex shapes.
The Distributional Reward Critic Architecture for Perturbed-Reward Reinforcement Learning
We study reinforcement learning in the presence of an unknown reward perturbation.
Guaranteed Nonconvex Factorization Approach for Tensor Train Recovery
We first delve into the TT factorization problem and establish the local linear convergence of RGD.
OTOv3: Automatic Architecture-Agnostic Neural Network Training and Compression from Structured Pruning to Erasing Operators
Compressing a predefined deep neural network (DNN) into a compact sub-network with competitive performance is crucial in the efficient machine learning realm.
The Efficiency Spectrum of Large Language Models: An Algorithmic Survey
The rapid growth of Large Language Models (LLMs) has been a driving force in transforming various domains, reshaping the artificial general intelligence landscape.
DREAM: Diffusion Rectification and Estimation-Adaptive Models
We present DREAM, a novel training framework representing Diffusion Rectification and Estimation Adaptive Models, requiring minimal code changes (just three lines) yet significantly enhancing the alignment of training with sampling in diffusion models.
Convergence Analysis for Learning Orthonormal Deep Linear Neural Networks
Enforcing orthonormal or isometric property for the weight matrices has been shown to enhance the training of deep neural networks by mitigating gradient exploding/vanishing and increasing the robustness of the learned networks.
Understanding Deep Representation Learning via Layerwise Feature Compression and Discrimination
To the best of our knowledge, this is the first quantitative characterization of feature evolution in hierarchical representations of deep linear networks.
Generalized Neural Collapse for a Large Number of Classes
However, most of the existing empirical and theoretical studies in neural collapse focus on the case that the number of classes is small relative to the dimension of the feature space.
The Law of Parsimony in Gradient Descent for Learning Deep Linear Networks
Second, it allows us to better understand deep representation learning by elucidating the linear progressive separation and concentration of representations from shallow to deep layers.
OTOV2: Automatic, Generic, User-Friendly
We propose the second generation of Only-Train-Once (OTOv2), which first automatically trains and compresses a general DNN only once from scratch to produce a more compact model with competitive performance without fine-tuning.
A Provable Splitting Approach for Symmetric Nonnegative Matrix Factorization
The symmetric Nonnegative Matrix Factorization (NMF), a special but important class of the general NMF, has found numerous applications in data analysis such as various clustering tasks.
Understanding and Improving Transfer Learning of Deep Models via Neural Collapse
In particular, we discovered a systematic pattern that emerges when linear probing pre-trained models on downstream training data: the more feature collapse of pre-trained models on downstream training data, the higher the transfer accuracy.
Revisiting Sparse Convolutional Model for Visual Recognition
We show that such models have equally strong empirical performance on CIFAR-10, CIFAR-100, and ImageNet datasets when compared to conventional neural networks.
Are All Losses Created Equal: A Neural Collapse Perspective
We extend such results and show through global solution and landscape analyses that a broad family of loss functions including commonly used label smoothing (LS) and focal loss (FL) exhibits Neural Collapse.
A Validation Approach to Over-parameterized Matrix and Image Recovery
This paper studies the problem of recovering a low-rank matrix from several noisy random linear measurements.
Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold
When training overparameterized deep networks for classification tasks, it has been widely observed that the learned features exhibit a so-called "neural collapse" phenomenon.
Sparsity-guided Network Design for Frame Interpolation
As a result, we achieve a considerable performance gain with a quarter of the size of the original AdaCoF.
Error Analysis of Tensor-Train Cross Approximation
Tensor train decomposition is widely used in machine learning and quantum physics due to its concise representation of high-dimensional tensors, overcoming the curse of dimensionality.
On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features
When training deep neural networks for classification tasks, an intriguing empirical phenomenon has been widely observed in the last-layer classifiers and features, where (i) the class means and the last-layer classifiers all collapse to the vertices of a Simplex Equiangular Tight Frame (ETF) up to scaling, and (ii) cross-example within-class variability of last-layer activations collapses to zero.
Robust Training under Label Noise by Over-parameterization
In this work, we propose a principled approach for robust training of over-parameterized deep networks in classification tasks where a proportion of training labels are corrupted.
Ranked #1 on
Learning with noisy labels
on CIFAR-10N-Random3
Rank Overspecified Robust Matrix Recovery: Subgradient Method and Exact Recovery
We study the robust recovery of a low-rank matrix from sparsely and grossly corrupted Gaussian measurements, with no prior knowledge on the intrinsic rank.
Only Train Once: A One-Shot Neural Network Training And Pruning Framework
Structured pruning is a commonly used technique in deploying deep neural networks (DNNs) onto resource-constrained devices.
A Geometric Analysis of Neural Collapse with Unconstrained Features
In contrast to existing landscape analysis for deep neural networks which is often disconnected from practice, our analysis of the simplified model not only does it explain what kind of features are learned in the last layer, but it also shows why they can be efficiently optimized in the simplified settings, matching the empirical observations in practical deep network architectures.
CDFI: Compression-Driven Network Design for Frame Interpolation
DNN-based frame interpolation--that generates the intermediate frames given two consecutive frames--typically relies on heavy model architectures with a huge number of features, preventing them from being deployed on systems with limited resources, e. g., mobile devices.
Ranked #1 on
Video Frame Interpolation
on Middlebury
(LPIPS metric)
Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training
Furthermore, we show that our ConvNorm can reduce the layerwise spectral norm of the weight matrices and hence improve the Lipschitzness of the network, leading to easier training and improved robustness for deep ConvNets.
A Half-Space Stochastic Projected Gradient Method for Group Sparsity Regularization
Optimizing with group sparsity is significant in enhancing model interpretability in machining learning applications, e. g., feature selection, compressed sensing and model compression.
Robust Recovery via Implicit Bias of Discrepant Learning Rates for Double Over-parameterization
This paper shows that with a double over-parameterization for both the low-rank matrix and sparse corruption, gradient descent with discrepant learning rates provably recovers the underlying matrix even without prior knowledge on neither rank of the matrix nor sparsity of the corruption.
Recovery and Generalization in Over-Realized Dictionary Learning
We thoroughly demonstrate this observation in practice and provide an analysis of this phenomenon by tying recovery measures to generalization bounds.
Geometric Analysis of Nonconvex Optimization Landscapes for Overcomplete Learning
Learning overcomplete representations finds many applications in machine learning and data analytics.
Orthant Based Proximal Stochastic Gradient Method for $\ell_1$-Regularized Optimization
Sparsity-inducing regularization problems are ubiquitous in machine learning applications, ranging from feature selection to model compression.
Finding the Sparsest Vectors in a Subspace: Theory, Algorithms, and Applications
The problem of finding the sparsest vector (direction) in a low dimensional subspace can be considered as a homogeneous variant of the sparse recovery problem, which finds applications in robust subspace recovery, dictionary learning, sparse blind deconvolution, and many other problems in signal processing and machine learning.
Analysis of the Optimization Landscapes for Overcomplete Representation Learning
In this work, we show these problems can be formulated as $\ell^4$-norm optimization problems with spherical constraint, and study the geometric properties of their nonconvex optimization landscapes.
Viewpoint-Aware Loss with Angular Regularization for Person Re-Identification
Instead of one subspace for each viewpoint, our method projects the feature from different viewpoints into a unified hypersphere and effectively models the feature distribution on both the identity-level and the viewpoint-level.
Ranked #7 on
Person Re-Identification
on DukeMTMC-reID
(using extra training data)
A Linearly Convergent Method for Non-Smooth Non-Convex Optimization on the Grassmannian with Applications to Robust Subspace and Dictionary Learning
Minimizing a non-smooth function over the Grassmannian appears in many applications in machine learning.
Distributed Low-rank Matrix Factorization With Exact Consensus
Low-rank matrix factorization is a problem of broad importance, owing to the ubiquity of low-rank models in machine learning contexts.
Weakly Convex Optimization over Stiefel Manifold Using Riemannian Subgradient-Type Methods
To the best of our knowledge, these are the first convergence guarantees for using Riemannian subgradient-type methods to optimize a class of nonconvex nonsmooth functions over the Stiefel manifold.
A Nonconvex Approach for Exact and Efficient Multichannel Sparse Blind Deconvolution
We study the multi-channel sparse blind deconvolution (MCS-BD) problem, whose task is to simultaneously recover a kernel $\mathbf a$ and multiple sparse inputs $\{\mathbf x_i\}_{i=1}^p$ from their circulant convolution $\mathbf y_i = \mathbf a \circledast \mathbf x_i $ ($i=1,\cdots, p$).
Provable Bregman-divergence based Methods for Nonconvex and Non-Lipschitz Problems
Therefore, this work not only develops guaranteed optimization methods for non-Lipschitz smooth problems but also solves an open problem of showing the second-order convergence guarantees for these alternating minimization methods.
Dual Principal Component Pursuit: Probability Analysis and Efficient Algorithms
However, its geometric analysis is based on quantities that are difficult to interpret and are not amenable to statistical analysis.
Dual Principal Component Pursuit: Improved Analysis and Efficient Algorithms
However, its geometric analysis is based on quantities that are difficult to interpret and are not amenable to statistical analysis.
Dropping Symmetry for Fast Symmetric Nonnegative Matrix Factorization
Symmetric nonnegative matrix factorization (NMF), a special but important class of the general NMF, is demonstrated to be useful for data analysis and in particular for various clustering tasks.
Global Optimality in Distributed Low-rank Matrix Factorization
We study the convergence of a variant of distributed gradient descent (DGD) on a distributed low-rank matrix approximation problem wherein some optimization variables are used for consensus (as in classical DGD) and some optimization variables appear only locally at a single node in the network.
Nonconvex Robust Low-rank Matrix Recovery
In this paper we study the problem of recovering a low-rank matrix from a number of random linear measurements that are corrupted by outliers taking arbitrary values.
Information Theory Information Theory
The Global Optimization Geometry of Shallow Linear Neural Networks
We examine the squared error loss landscape of shallow linear neural networks.
Optimized Structured Sparse Sensing Matrices for Compressive Sensing
We consider designing a robust structured sparse sensing matrix consisting of a sparse matrix with a few non-zero entries per row and a dense base matrix for capturing signals efficiently We design the robust structured sparse sensing matrix through minimizing the distance between the Gram matrix of the equivalent dictionary and the target Gram of matrix holding small mutual coherence.
Geometry of Factored Nuclear Norm Regularization
In spite of the nonconvexity of the factored formulation, we prove that when the convex loss function $f(X)$ is $(2r, 4r)$-restricted well-conditioned, each critical point of the factored problem either corresponds to the optimal solution $X^\star$ of the original convex optimization or is a strict saddle point where the Hessian matrix has a strictly negative eigenvalue.
Online Learning Sensing Matrix and Sparsifying Dictionary Simultaneously for Compressive Sensing
The simulation results on natural images demonstrate the effectiveness of the suggested online algorithm compared with the existing methods.
An Efficient Method for Robust Projection Matrix Design
Without requiring of training data, we can efficiently design the robust projection matrix and apply it for most of CS systems, like a CS system for image processing with a conventional wavelet dictionary in which the SRE matrix is generally not available.
