Search Results for author: Zhouchen Lin

Found 146 papers, 50 papers with code

Optimizing Multi-Class Spatio-Spectral Filters via Bayes Error Estimation for EEG Classification

no code implementations NeurIPS 2009 Wenming Zheng, Zhouchen Lin

The method of common spatio-spectral patterns (CSSPs) is an extension of common spatial patterns (CSPs) by utilizing the technique of delay embedding to alleviate the adverse effects of noises and artifacts on the electroencephalogram (EEG) classification.

Classification EEG +2

The Augmented Lagrange Multiplier Method for Exact Recovery of Corrupted Low-Rank Matrices

no code implementations26 Sep 2010 Zhouchen Lin, Minming Chen, Yi Ma

This paper proposes scalable and fast algorithms for solving the Robust PCA problem, namely recovering a low-rank matrix with an unknown fraction of its entries being arbitrarily corrupted.

Optimization and Control Numerical Analysis Systems and Control

A Block Lanczos with Warm Start Technique for Accelerating Nuclear Norm Minimization Algorithms

no code implementations2 Dec 2010 Zhouchen Lin, Siming Wei

Recent years have witnessed the popularity of using rank minimization as a regularizer for various signal processing and machine learning problems.

Matrix Completion

Solving Principal Component Pursuit in Linear Time via $l_1$ Filtering

no code implementations26 Aug 2011 Risheng Liu, Zhouchen Lin, Siming Wei, Zhixun Su

In this paper, we propose a novel algorithm, called $l_1$ filtering, for \emph{exactly} solving PCP with an $O(r^2(m+n))$ complexity, where $m\times n$ is the size of data matrix and $r$ is the rank of the matrix to recover, which is supposed to be much smaller than $m$ and $n$.

Linearized Alternating Direction Method with Adaptive Penalty for Low-Rank Representation

no code implementations NeurIPS 2011 Zhouchen Lin, Risheng Liu, Zhixun Su

It suffers from $O(n^3)$ computation complexity due to the matrix-matrix multiplications and matrix inversions, even if partial SVD is used.

Optimization and Control

A Counterexample for the Validity of Using Nuclear Norm as a Convex Surrogate of Rank

no code implementations23 Apr 2013 Hongyang Zhang, Zhouchen Lin, Chao Zhang

For several rank minimization problems, such a replacement has been theoretically proven to be valid, i. e., the solution to nuclear norm minimization problem is also the solution to rank minimization problem.

valid

Linearized Alternating Direction Method with Parallel Splitting and Adaptive Penalty for Separable Convex Programs in Machine Learning

no code implementations18 Oct 2013 Zhouchen Lin, Risheng Liu, Huan Li

However, the traditional alternating direction method (ADM) and its linearized version (LADM, obtained by linearizing the quadratic penalty term) are for the two-block case and cannot be naively generalized to solve the multi-block case.

Distributed Computing

Smoothed Low Rank and Sparse Matrix Recovery by Iteratively Reweighted Least Squares Minimization

no code implementations29 Jan 2014 Canyi Lu, Zhouchen Lin, Shuicheng Yan

Our convergence proof of IRLS is more general than previous one which depends on the special properties of the Schatten-$p$ norm and $\ell_{2, q}$-norm.

Proximal Iteratively Reweighted Algorithm with Multiple Splitting for Nonconvex Sparsity Optimization

no code implementations28 Apr 2014 Canyi Lu, Yunchao Wei, Zhouchen Lin, Shuicheng Yan

This paper proposes the Proximal Iteratively REweighted (PIRE) algorithm for solving a general problem, which involves a large body of nonconvex sparse and structured sparse related problems.

Generalized Nonconvex Nonsmooth Low-Rank Minimization

no code implementations CVPR 2014 Canyi Lu, Jinhui Tang, Shuicheng Yan, Zhouchen Lin

We observe that all the existing nonconvex penalty functions are concave and monotonically increasing on $[0,\infty)$.

Robust Subspace Segmentation with Block-diagonal Prior

no code implementations CVPR 2014 Jiashi Feng, Zhouchen Lin, Huan Xu, Shuicheng Yan

Most current state-of-the-art subspace segmentation methods (such as SSC and LRR) resort to alternative structural priors (such as sparseness and low-rankness) to construct the affinity matrix.

Clustering Face Clustering +3

Adaptive Partial Differential Equation Learning for Visual Saliency Detection

no code implementations CVPR 2014 Risheng Liu, Junjie Cao, Zhouchen Lin, Shiguang Shan

Then by optimizing a discrete submodular function constrained with this LESD and a uniform matroid, the saliency seeds (i. e., boundary conditions) can be learnt for this image, thus achieving an optimal PDE system to model the evolution of visual saliency.

Saliency Detection

Robust Estimation of 3D Human Poses from a Single Image

no code implementations CVPR 2014 Chunyu Wang, Yizhou Wang, Zhouchen Lin, Alan L. Yuille, Wen Gao

We address the challenges in three ways: (i) We represent a 3D pose as a linear combination of a sparse set of bases learned from 3D human skeletons.

3D Human Pose Estimation 3D Pose Estimation +2

Constructing a Non-Negative Low Rank and Sparse Graph with Data-Adaptive Features

no code implementations3 Sep 2014 Liansheng Zhuang, Shenghua Gao, Jinhui Tang, Jingjing Wang, Zhouchen Lin, Yi Ma

This paper aims at constructing a good graph for discovering intrinsic data structures in a semi-supervised learning setting.

graph construction

Relations among Some Low Rank Subspace Recovery Models

no code implementations6 Dec 2014 Hongyang Zhang, Zhouchen Lin, Chao Zhang, Junbin Gao

More specifically, we discover that once a solution to one of the models is obtained, we can obtain the solutions to other models in closed-form formulations.

Generalized Singular Value Thresholding

no code implementations6 Dec 2014 Canyi Lu, Changbo Zhu, Chunyan Xu, Shuicheng Yan, Zhouchen Lin

This work studies the Generalized Singular Value Thresholding (GSVT) operator ${\text{Prox}}_{g}^{{\sigma}}(\cdot)$, \begin{equation*} {\text{Prox}}_{g}^{{\sigma}}(B)=\arg\min\limits_{X}\sum_{i=1}^{m}g(\sigma_{i}(X)) + \frac{1}{2}||X-B||_{F}^{2}, \end{equation*} associated with a nonconvex function $g$ defined on the singular values of $X$.

Correlation Adaptive Subspace Segmentation by Trace Lasso

no code implementations18 Jan 2015 Canyi Lu, Jiashi Feng, Zhouchen Lin, Shuicheng Yan

In this work, we argue that both sparsity and the grouping effect are important for subspace segmentation.

Clustering Segmentation

Correntropy Induced L2 Graph for Robust Subspace Clustering

no code implementations18 Jan 2015 Canyi Lu, Jinhui Tang, Min Lin, Liang Lin, Shuicheng Yan, Zhouchen Lin

In this paper, we study the robust subspace clustering problem, which aims to cluster the given possibly noisy data points into their underlying subspaces.

Clustering graph construction

Subspace Clustering by Mixture of Gaussian Regression

no code implementations CVPR 2015 Baohua Li, Ying Zhang, Zhouchen Lin, Huchuan Lu

Therefore, we propose Mixture of Gaussian Regression (MoG Regression) for subspace clustering by modeling noise as a Mixture of Gaussians (MoG).

Clustering regression

Image Tag Completion and Refinement by Subspace Clustering and Matrix Completion

no code implementations10 Jun 2015 Yuqing Hou, Zhouchen Lin

Tag-based image retrieval (TBIR) has drawn much attention in recent years due to the explosive amount of digital images and crowdsourcing tags.

Clustering Image Retrieval +3

Completing Low-Rank Matrices with Corrupted Samples from Few Coefficients in General Basis

no code implementations25 Jun 2015 Hongyang Zhang, Zhouchen Lin, Chao Zhang

As an application, we also find that the solutions to extended robust Low-Rank Representation and to our extended robust MC are mutually expressible, so both our theory and algorithm can be applied to the subspace clustering problem with missing values under certain conditions.

Clustering Matrix Completion

Optimized Projections for Compressed Sensing via Direct Mutual Coherence Minimization

no code implementations13 Aug 2015 Canyi Lu, Huan Li, Zhouchen Lin

To the best of our knowledge, this is the first work which directly minimizes the mutual coherence of the projected dictionary with a convergence guarantee.

Nonconvex Nonsmooth Low-Rank Minimization via Iteratively Reweighted Nuclear Norm

no code implementations23 Oct 2015 Canyi Lu, Jinhui Tang, Shuicheng Yan, Zhouchen Lin

The nuclear norm is widely used as a convex surrogate of the rank function in compressive sensing for low rank matrix recovery with its applications in image recovery and signal processing.

Compressive Sensing

Fast Proximal Linearized Alternating Direction Method of Multiplier with Parallel Splitting

no code implementations14 Nov 2015 Canyi Lu, Huan Li, Zhouchen Lin, Shuicheng Yan

The Augmented Lagragian Method (ALM) and Alternating Direction Method of Multiplier (ADMM) have been powerful optimization methods for general convex programming subject to linear constraint.

Convex Sparse Spectral Clustering: Single-view to Multi-view

no code implementations21 Nov 2015 Canyi Lu, Shuicheng Yan, Zhouchen Lin

Spectral Clustering (SC) is one of the most widely used methods for data clustering.

Clustering

Relaxed Majorization-Minimization for Non-smooth and Non-convex Optimization

no code implementations25 Nov 2015 Chen Xu, Zhouchen Lin, Zhenyu Zhao, Hongbin Zha

We propose a new majorization-minimization (MM) method for non-smooth and non-convex programs, which is general enough to include the existing MM methods.

Learning Semi-Supervised Representation Towards a Unified Optimization Framework for Semi-Supervised Learning

no code implementations ICCV 2015 Chun-Guang Li, Zhouchen Lin, Honggang Zhang, Jun Guo

State of the art approaches for Semi-Supervised Learning (SSL) usually follow a two-stage framework -- constructing an affinity matrix from the data and then propagating the partial labels on this affinity matrix to infer those unknown labels.

Accelerated Proximal Gradient Methods for Nonconvex Programming

no code implementations NeurIPS 2015 Huan Li, Zhouchen Lin

However, it is still unknown whether the usual APG can ensure the convergence to a critical point in nonconvex programming.

Tensor Sparse and Low-Rank based Submodule Clustering Method for Multi-way Data

no code implementations2 Jan 2016 Xinglin Piao, Yongli Hu, Junbin Gao, Yanfeng Sun, Zhouchen Lin, Bao-Cai Yin

A new submodule clustering method via sparse and low-rank representation for multi-way data is proposed in this paper.

Clustering

Robust Kernel Estimation With Outliers Handling for Image Deblurring

no code implementations CVPR 2016 Jinshan Pan, Zhouchen Lin, Zhixun Su, Ming-Hsuan Yang

Estimating blur kernels from real world images is a challenging problem as the linear image formation assumption does not hold when significant outliers, such as saturated pixels and non-Gaussian noise, are present.

Deblurring Image Deblurring +1

Graph Construction with Label Information for Semi-Supervised Learning

no code implementations8 Jul 2016 Liansheng Zhuang, Zihan Zhou, Jingwen Yin, Shenghua Gao, Zhouchen Lin, Yi Ma, Nenghai Yu

In the literature, most existing graph-based semi-supervised learning (SSL) methods only use the label information of observed samples in the label propagation stage, while ignoring such valuable information when learning the graph.

graph construction Graph Learning

Globally Variance-Constrained Sparse Representation and Its Application in Image Set Coding

no code implementations17 Aug 2016 Xiang Zhang, Jiarui Sun, Siwei Ma, Zhouchen Lin, Jian Zhang, Shiqi Wang, Wen Gao

Therefore, introducing an accurate rate-constraint in sparse coding and dictionary learning becomes meaningful, which has not been fully exploited in the context of sparse representation.

Data Compression Dictionary Learning

A Unified Convex Surrogate for the Schatten-$p$ Norm

no code implementations25 Nov 2016 Chen Xu, Zhouchen Lin, Hongbin Zha

In this paper, we show that for any $p$, $p_1$, and $p_2 >0$ satisfying $1/p=1/p_1+1/p_2$, there is an equivalence between the Schatten-$p$ norm of one matrix and the Schatten-$p_1$ and the Schatten-$p_2$ norms of its two factor matrices.

Matrix Completion

Tensor Robust Principal Component Analysis: Exact Recovery of Corrupted Low-Rank Tensors via Convex Optimization

no code implementations CVPR 2016 Canyi Lu, Jiashi Feng, Yudong Chen, Wei Liu, Zhouchen Lin, Shuicheng Yan

In this work, we prove that under certain suitable assumptions, we can recover both the low-rank and the sparse components exactly by simply solving a convex program whose objective is a weighted combination of the tensor nuclear norm and the $\ell_1$-norm, i. e., $\min_{{\mathcal{L}},\ {\mathcal{E}}} \ \|{{\mathcal{L}}}\|_*+\lambda\|{{\mathcal{E}}}\|_1, \ \text{s. t.}

Image Denoising

Accelerating Asynchronous Algorithms for Convex Optimization by Momentum Compensation

no code implementations27 Feb 2018 Cong Fang, Yameng Huang, Zhouchen Lin

$O(1/\epsilon)$) convergence rate for non-strongly convex functions, and $O(\sqrt{\kappa}\log(1/\epsilon))$ (v. s.

Convolutional Neural Networks with Alternately Updated Clique

3 code implementations CVPR 2018 Yibo Yang, Zhisheng Zhong, Tiancheng Shen, Zhouchen Lin

In contrast to prior networks, there are both forward and backward connections between any two layers in the same block.

Tensor Robust Principal Component Analysis with A New Tensor Nuclear Norm

1 code implementation10 Apr 2018 Canyi Lu, Jiashi Feng, Yudong Chen, Wei Liu, Zhouchen Lin, Shuicheng Yan

Equipped with the new tensor nuclear norm, we then solve the TRPCA problem by solving a convex program and provide the theoretical guarantee for the exact recovery.

Subspace Clustering by Block Diagonal Representation

no code implementations23 May 2018 Canyi Lu, Jiashi Feng, Zhouchen Lin, Tao Mei, Shuicheng Yan

Second, we observe that many existing methods approximate the block diagonal representation matrix by using different structure priors, e. g., sparsity and low-rankness, which are indirect.

Clustering

Exact Low Tubal Rank Tensor Recovery from Gaussian Measurements

1 code implementation7 Jun 2018 Canyi Lu, Jiashi Feng, Zhouchen Lin, Shuicheng Yan

Specifically, we show that by solving a TNN minimization problem, the underlying tensor of size $n_1\times n_2\times n_3$ with tubal rank $r$ can be exactly recovered when the given number of Gaussian measurements is $O(r(n_1+n_2-r)n_3)$.

SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimator

no code implementations NeurIPS 2018 Cong Fang, Chris Junchi Li, Zhouchen Lin, Tong Zhang

For stochastic first-order method, combining SPIDER with normalized gradient descent, we propose two new algorithms, namely SPIDER-SFO and SPIDER-SFO\textsuperscript{+}, that solve non-convex stochastic optimization problems using stochastic gradients only.

Stochastic Optimization

Essential Tensor Learning for Multi-view Spectral Clustering

no code implementations10 Jul 2018 Jianlong Wu, Zhouchen Lin, Hongbin Zha

In this paper, we focus on the Markov chain based spectral clustering method and propose a novel essential tensor learning method to explore the high order correlations for multi-view representation.

Clustering

Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining

no code implementations ECCV 2018 Xia Li, Jianlong Wu, Zhouchen Lin, Hong Liu, Hongbin Zha

In heavy rain, rain streaks have various directions and shapes, which can be regarded as the accumulation of multiple rain streak layers.

Single Image Deraining

On the Convergence of Learning-based Iterative Methods for Nonconvex Inverse Problems

no code implementations16 Aug 2018 Risheng Liu, Shichao Cheng, Yi He, Xin Fan, Zhouchen Lin, Zhongxuan Luo

Moreover, there is a lack of rigorous analysis about the convergence behaviors of these reimplemented iterations, and thus the significance of such methods is a little bit vague.

Scheduling

Joint Sub-bands Learning with Clique Structures for Wavelet Domain Super-Resolution

no code implementations NeurIPS 2018 Zhisheng Zhong, Tiancheng Shen, Yibo Yang, Zhouchen Lin, Chao Zhang

To solve these problems, we propose the Super-Resolution CliqueNet (SRCliqueNet) to reconstruct the high resolution (HR) image with better textural details in the wavelet domain.

Image Super-Resolution

Optimization Algorithm Inspired Deep Neural Network Structure Design

no code implementations3 Oct 2018 Huan Li, Yibo Yang, Dongmin Chen, Zhouchen Lin

In this paper, we propose the hypothesis that the neural network structure design can be inspired by optimization algorithms and a faster optimization algorithm may lead to a better neural network structure.

Bilinear Factor Matrix Norm Minimization for Robust PCA: Algorithms and Applications

no code implementations11 Oct 2018 Fanhua Shang, James Cheng, Yuanyuan Liu, Zhi-Quan Luo, Zhouchen Lin

The heavy-tailed distributions of corrupted outliers and singular values of all channels in low-level vision have proven effective priors for many applications such as background modeling, photometric stereo and image alignment.

Moving Object Detection object-detection

Lifted Proximal Operator Machines

no code implementations5 Nov 2018 Jia Li, Cong Fang, Zhouchen Lin

LPOM is block multi-convex in all layer-wise weights and activations.

SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path-Integrated Differential Estimator

no code implementations NeurIPS 2018 Cong Fang, Chris Junchi Li, Zhouchen Lin, Tong Zhang

Specially, we prove that the SPIDER-SFO algorithm achieves a gradient computation cost of $\mathcal{O}\left( \min( n^{1/2} \epsilon^{-2}, \epsilon^{-3} ) \right)$ to find an $\epsilon$-approximate first-order stationary point.

Stochastic Optimization

Sharp Analysis for Nonconvex SGD Escaping from Saddle Points

no code implementations1 Feb 2019 Cong Fang, Zhouchen Lin, Tong Zhang

In this paper, we give a sharp analysis for Stochastic Gradient Descent (SGD) and prove that SGD is able to efficiently escape from saddle points and find an $(\epsilon, O(\epsilon^{0. 5}))$-approximate second-order stationary point in $\tilde{O}(\epsilon^{-3. 5})$ stochastic gradient computations for generic nonconvex optimization problems, when the objective function satisfies gradient-Lipschitz, Hessian-Lipschitz, and dispersive noise assumptions.

Stochastic Optimization

Towards Understanding Adversarial Examples Systematically: Exploring Data Size, Task and Model Factors

no code implementations28 Feb 2019 Ke Sun, Zhanxing Zhu, Zhouchen Lin

In this paper, we present a systematic study on adversarial examples from three aspects: the amount of training data, task-dependent and model-specific factors.

Multi-Stage Self-Supervised Learning for Graph Convolutional Networks on Graphs with Few Labels

1 code implementation28 Feb 2019 Ke Sun, Zhouchen Lin, Zhanxing Zhu

In this paper, we propose a novel training algorithm for Graph Convolutional Network, called Multi-Stage Self-Supervised(M3S) Training Algorithm, combined with self-supervised learning approach, focusing on improving the generalization performance of GCNs on graphs with few labeled nodes.

Graph Embedding Graph Learning +1

Enhancing the Robustness of Deep Neural Networks by Boundary Conditional GAN

no code implementations28 Feb 2019 Ke Sun, Zhanxing Zhu, Zhouchen Lin

In this work, we propose a novel defense mechanism called Boundary Conditional GAN to enhance the robustness of deep neural networks against adversarial examples.

Data Augmentation

Virtual Adversarial Training on Graph Convolutional Networks in Node Classification

no code implementations28 Feb 2019 Ke Sun, Zhouchen Lin, Hantao Guo, Zhanxing Zhu

The effectiveness of Graph Convolutional Networks (GCNs) has been demonstrated in a wide range of graph-based machine learning tasks.

Classification General Classification +1

Self-Supervised Convolutional Subspace Clustering Network

no code implementations CVPR 2019 Junjian Zhang, Chun-Guang Li, Chong You, Xianbiao Qi, Honggang Zhang, Jun Guo, Zhouchen Lin

However, the applicability of subspace clustering has been limited because practical visual data in raw form do not necessarily lie in such linear subspaces.

Clustering Image Clustering

Differentiable Linearized ADMM

1 code implementation15 May 2019 Xingyu Xie, Jianlong Wu, Zhisheng Zhong, Guangcan Liu, Zhouchen Lin

Recently, a number of learning-based optimization methods that combine data-driven architectures with the classical optimization algorithms have been proposed and explored, showing superior empirical performance in solving various ill-posed inverse problems, but there is still a scarcity of rigorous analysis about the convergence behaviors of learning-based optimization.

ADA-Tucker: Compressing Deep Neural Networks via Adaptive Dimension Adjustment Tucker Decomposition

no code implementations18 Jun 2019 Zhisheng Zhong, Fangyin Wei, Zhouchen Lin, Chao Zhang

Furthermore, we propose that weight tensors in networks with proper order and balanced dimension are easier to be compressed.

AdaGCN: Adaboosting Graph Convolutional Networks into Deep Models

1 code implementation ICLR 2021 Ke Sun, Zhanxing Zhu, Zhouchen Lin

The design of deep graph models still remains to be investigated and the crucial part is how to explore and exploit the knowledge from different hops of neighbors in an efficient way.

Node Classification

Tensor Q-Rank: New Data Dependent Definition of Tensor Rank

no code implementations26 Oct 2019 Hao Kong, Canyi Lu, Zhouchen Lin

Recently, the \textit{Tensor Nuclear Norm~(TNN)} regularization based on t-SVD has been widely used in various low tubal-rank tensor recovery tasks.

SOGNet: Scene Overlap Graph Network for Panoptic Segmentation

1 code implementation18 Nov 2019 Yibo Yang, Hongyang Li, Xia Li, Qijie Zhao, Jianlong Wu, Zhouchen Lin

In order to overcome the lack of supervision, we introduce a differentiable module to resolve the overlap between any pair of instances.

Instance Segmentation Panoptic Segmentation +1

Dynamical System Inspired Adaptive Time Stepping Controller for Residual Network Families

no code implementations23 Nov 2019 Yibo Yang, Jianlong Wu, Hongyang Li, Xia Li, Tiancheng Shen, Zhouchen Lin

We establish a stability condition for ResNets with step sizes and weight parameters, and point out the effects of step sizes on the stability and performance.

Histogram Transform Ensembles for Large-scale Regression

no code implementations8 Dec 2019 Hanyuan Hang, Zhouchen Lin, Xiaoyu Liu, Hongwei Wen

Instead, we apply kernel histogram transforms (KHT) equipped with smoother regressors such as support vector machines (SVMs), and it turns out that both single and ensemble KHT enjoy almost optimal convergence rates.

regression

Revisiting EXTRA for Smooth Distributed Optimization

no code implementations24 Feb 2020 Huan Li, Zhouchen Lin

EXTRA is a popular method for dencentralized distributed optimization and has broad applications.

Distributed Optimization

Spatial Pyramid Based Graph Reasoning for Semantic Segmentation

no code implementations CVPR 2020 Xia Li, Yibo Yang, Qijie Zhao, Tiancheng Shen, Zhouchen Lin, Hong Liu

The convolution operation suffers from a limited receptive filed, while global modeling is fundamental to dense prediction tasks, such as semantic segmentation.

Segmentation Semantic Segmentation

Invertible Image Rescaling

10 code implementations ECCV 2020 Mingqing Xiao, Shuxin Zheng, Chang Liu, Yaolong Wang, Di He, Guolin Ke, Jiang Bian, Zhouchen Lin, Tie-Yan Liu

High-resolution digital images are usually downscaled to fit various display screens or save the cost of storage and bandwidth, meanwhile the post-upscaling is adpoted to recover the original resolutions or the details in the zoom-in images.

Image Super-Resolution

Maximum-and-Concatenation Networks

1 code implementation ICML 2020 Xingyu Xie, Hao Kong, Jianlong Wu, Wayne Zhang, Guangcan Liu, Zhouchen Lin

While successful in many fields, deep neural networks (DNNs) still suffer from some open problems such as bad local minima and unsatisfactory generalization performance.

Improving Semantic Segmentation via Decoupled Body and Edge Supervision

2 code implementations ECCV 2020 Xiangtai Li, Xia Li, Li Zhang, Guangliang Cheng, Jianping Shi, Zhouchen Lin, Shaohua Tan, Yunhai Tong

Our insight is that appealing performance of semantic segmentation requires \textit{explicitly} modeling the object \textit{body} and \textit{edge}, which correspond to the high and low frequency of the image.

Object Segmentation +1

PDO-eConvs: Partial Differential Operator Based Equivariant Convolutions

3 code implementations ICML 2020 Zhengyang Shen, Lingshen He, Zhouchen Lin, Jinwen Ma

In implementation, we discretize the system using the numerical schemes of PDOs, deriving approximately equivariant convolutions (PDO-eConvs).

Image Classification Rotated MNIST

Variance Reduced EXTRA and DIGing and Their Optimal Acceleration for Strongly Convex Decentralized Optimization

no code implementations9 Sep 2020 Huan Li, Zhouchen Lin, Yongchun Fang

Our stochastic gradient computation complexities are the same as the ones of single-machine VR methods, such as SAG, SAGA, and SVRG, and our communication complexities keep the same as those of EXTRA and DIGing, respectively.

Towards Efficient Scene Understanding via Squeeze Reasoning

1 code implementation6 Nov 2020 Xiangtai Li, Xia Li, Ansheng You, Li Zhang, Guangliang Cheng, Kuiyuan Yang, Yunhai Tong, Zhouchen Lin

Instead of propagating information on the spatial map, we first learn to squeeze the input feature into a channel-wise global vector and perform reasoning within the single vector where the computation cost can be significantly reduced.

Instance Segmentation object-detection +4

Fast and Differentiable Matrix Inverse and Its Extension to SVD

no code implementations1 Jan 2021 Xingyu Xie, Hao Kong, Jianlong Wu, Guangcan Liu, Zhouchen Lin

First of all, to perform matrix inverse, we provide a differentiable yet efficient way, named LD-Minv, which is a learnable deep neural network (DNN) with each layer being an $L$-th order matrix polynomial.

Efficient Sampling for Generative Adversarial Networks with Coupling Markov Chains

no code implementations1 Jan 2021 Yifei Wang, Yisen Wang, Jiansheng Yang, Zhouchen Lin

Recently, sampling methods have been successfully applied to enhance the sample quality of Generative Adversarial Networks (GANs).

OT-LLP: Optimal Transport for Learning from Label Proportions

no code implementations1 Jan 2021 Jiabin Liu, Hanyuan Hang, Bo wang, Xin Shen, Zhouchen Lin

Learning from label proportions (LLP), where the training data are arranged in form of groups with only label proportions provided instead of the exact labels, is an important weakly supervised learning paradigm in machine learning.

Weakly-supervised Learning

EnTranNAS: Towards Closing the Gap between the Architectures in Search and Evaluation

no code implementations1 Jan 2021 Yibo Yang, Shan You, Hongyang Li, Fei Wang, Chen Qian, Zhouchen Lin

The Engine-cell is differentiable for architecture search, while the Transit-cell only transits the current sub-graph by architecture derivation.

Neural Architecture Search

Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search

no code implementations CVPR 2021 Yibo Yang, Shan You, Hongyang Li, Fei Wang, Chen Qian, Zhouchen Lin

Our method enables differentiable sparsification, and keeps the derived architecture equivalent to that of Engine-cell, which further improves the consistency between search and evaluation.

Neural Architecture Search

Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyond

1 code implementation27 Jan 2021 Risheng Liu, Jiaxin Gao, Jin Zhang, Deyu Meng, Zhouchen Lin

Bi-Level Optimization (BLO) is originated from the area of economic game theory and then introduced into the optimization community.

Meta-Learning Neural Architecture Search

PointFlow: Flowing Semantics Through Points for Aerial Image Segmentation

1 code implementation CVPR 2021 Xiangtai Li, Hao He, Xia Li, Duo Li, Guangliang Cheng, Jianping Shi, Lubin Weng, Yunhai Tong, Zhouchen Lin

Experimental results on three different aerial segmentation datasets suggest that the proposed method is more effective and efficient than state-of-the-art general semantic segmentation methods.

Image Segmentation Segmentation +1

Graph Contrastive Clustering

1 code implementation ICCV 2021 Huasong Zhong, Jianlong Wu, Chong Chen, Jianqiang Huang, Minghua Deng, Liqiang Nie, Zhouchen Lin, Xian-Sheng Hua

On the other hand, a novel graph-based contrastive learning strategy is proposed to learn more compact clustering assignments.

Clustering Contrastive Learning

Accelerated Gradient Tracking over Time-varying Graphs for Decentralized Optimization

no code implementations6 Apr 2021 Huan Li, Zhouchen Lin

We prove the $O((\frac{\gamma}{1-\sigma_{\gamma}})^2\sqrt{\frac{L}{\epsilon}})$ and $O((\frac{\gamma}{1-\sigma_{\gamma}})^{1. 5}\sqrt{\frac{L}{\mu}}\log\frac{1}{\epsilon})$ complexities for the practical single loop accelerated gradient tracking over time-varying graphs when the problems are nonstrongly convex and strongly convex, respectively, where $\gamma$ and $\sigma_{\gamma}$ are two common constants charactering the network connectivity, $\epsilon$ is the desired precision, and $L$ and $\mu$ are the smoothness and strong convexity constants, respectively.

Federated Learning

PDO-eS2CNNs: Partial Differential Operator Based Equivariant Spherical CNNs

no code implementations8 Apr 2021 Zhengyang Shen, Tiancheng Shen, Zhouchen Lin, Jinwen Ma

Spherical signals exist in many applications, e. g., planetary data, LiDAR scans and digitalization of 3D objects, calling for models that can process spherical data effectively.

Translation

BoundarySqueeze: Image Segmentation as Boundary Squeezing

1 code implementation25 May 2021 Hao He, Xiangtai Li, Yibo Yang, Guangliang Cheng, Yunhai Tong, Lubin Weng, Zhouchen Lin, Shiming Xiang

This module is used to squeeze the object boundary from both inner and outer directions, which contributes to precise mask representation.

Image Segmentation Instance Segmentation +2

Optimization Induced Equilibrium Networks

no code implementations27 May 2021 Xingyu Xie, Qiuhao Wang, Zenan Ling, Xia Li, Yisen Wang, Guangcan Liu, Zhouchen Lin

In this paper, we investigate an emerging question: can an implicit equilibrium model's equilibrium point be regarded as the solution of an optimization problem?

Gradient Boosted Binary Histogram Ensemble for Large-scale Regression

no code implementations3 Jun 2021 Hanyuan Hang, Tao Huang, Yuchao Cai, Hanfang Yang, Zhouchen Lin

In this paper, we propose a gradient boosting algorithm for large-scale regression problems called \textit{Gradient Boosted Binary Histogram Ensemble} (GBBHE) based on binary histogram partition and ensemble learning.

Computational Efficiency Ensemble Learning +1

GBHT: Gradient Boosting Histogram Transform for Density Estimation

no code implementations10 Jun 2021 Jingyi Cui, Hanyuan Hang, Yisen Wang, Zhouchen Lin

In this paper, we propose a density estimation algorithm called \textit{Gradient Boosting Histogram Transform} (GBHT), where we adopt the \textit{Negative Log Likelihood} as the loss function to make the boosting procedure available for the unsupervised tasks.

Anomaly Detection Density Estimation +1

Leveraged Weighted Loss for Partial Label Learning

1 code implementation10 Jun 2021 Hongwei Wen, Jingyi Cui, Hanyuan Hang, Jiabin Liu, Yisen Wang, Zhouchen Lin

As an important branch of weakly supervised learning, partial label learning deals with data where each instance is assigned with a set of candidate labels, whereas only one of them is true.

Partial Label Learning Weakly-supervised Learning

Demystifying Adversarial Training via A Unified Probabilistic Framework

no code implementations ICML Workshop AML 2021 Yifei Wang, Yisen Wang, Jiansheng Yang, Zhouchen Lin

Based on these, we propose principled adversarial sampling algorithms in both supervised and unsupervised scenarios.

Reparameterized Sampling for Generative Adversarial Networks

1 code implementation1 Jul 2021 Yifei Wang, Yisen Wang, Jiansheng Yang, Zhouchen Lin

Recently, sampling methods have been successfully applied to enhance the sample quality of Generative Adversarial Networks (GANs).

Under-bagging Nearest Neighbors for Imbalanced Classification

no code implementations1 Sep 2021 Hanyuan Hang, Yuchao Cai, Hanfang Yang, Zhouchen Lin

In this paper, we propose an ensemble learning algorithm called \textit{under-bagging $k$-nearest neighbors} (\textit{under-bagging $k$-NN}) for imbalanced classification problems.

Classification Ensemble Learning +2

Is Attention Better Than Matrix Decomposition?

2 code implementations ICLR 2021 Zhengyang Geng, Meng-Hao Guo, Hongxu Chen, Xia Li, Ke Wei, Zhouchen Lin

As an essential ingredient of modern deep learning, attention mechanism, especially self-attention, plays a vital role in the global correlation discovery.

Conditional Image Generation Semantic Segmentation

Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium State

1 code implementation NeurIPS 2021 Mingqing Xiao, Qingyan Meng, Zongpeng Zhang, Yisen Wang, Zhouchen Lin

In this work, we consider feedback spiking neural networks, which are more brain-like, and propose a novel training method that does not rely on the exact reverse of the forward computation.

Optimization inspired Multi-Branch Equilibrium Models

no code implementations ICLR 2022 Mingjie Li, Yisen Wang, Xingyu Xie, Zhouchen Lin

Works have shown the strong connections between some implicit models and optimization problems.

Chaos is a Ladder: A New Understanding of Contrastive Learning

no code implementations ICLR 2022 Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, Zhouchen Lin

Our work suggests an alternative understanding of contrastive learning: the role of aligning positive samples is more like a surrogate task than an ultimate goal, and it is the overlapping augmented views (i. e., the chaos) that create a ladder for contrastive learning to gradually learn class-separated representations.

Contrastive Learning Self-Supervised Learning

Residual Relaxation for Multi-view Representation Learning

no code implementations NeurIPS 2021 Yifei Wang, Zhengyang Geng, Feng Jiang, Chuming Li, Yisen Wang, Jiansheng Yang, Zhouchen Lin

Multi-view methods learn representations by aligning multiple views of the same image and their performance largely depends on the choice of data augmentation.

Data Augmentation Representation Learning

Efficient Equivariant Network

1 code implementation NeurIPS 2021 Lingshen He, Yuxuan Chen, Zhengyang Shen, Yiming Dong, Yisen Wang, Zhouchen Lin

Group equivariant CNNs (G-CNNs) that incorporate more equivariance can significantly improve the performance of conventional CNNs.

Gauge Equivariant Transformer

no code implementations NeurIPS 2021 Lingshen He, Yiming Dong, Yisen Wang, DaCheng Tao, Zhouchen Lin

Attention mechanism has shown great performance and efficiency in a lot of deep learning models, in which relative position encoding plays a crucial role.

Position

Restarted Nonconvex Accelerated Gradient Descent: No More Polylogarithmic Factor in the $O(ε^{-7/4})$ Complexity

1 code implementation27 Jan 2022 Huan Li, Zhouchen Lin

They do not invoke negative curvature exploitation or minimization of regularized surrogate functions as the subroutines.

A Unified Contrastive Energy-based Model for Understanding the Generative Ability of Adversarial Training

no code implementations ICLR 2022 Yifei Wang, Yisen Wang, Jiansheng Yang, Zhouchen Lin

On the other hand, our unified framework can be extended to the unsupervised scenario, which interprets unsupervised contrastive learning as an important sampling of CEM.

Contrastive Learning

Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlap

1 code implementation25 Mar 2022 Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, Zhouchen Lin

Our theory suggests an alternative understanding of contrastive learning: the role of aligning positive samples is more like a surrogate task than an ultimate goal, and the overlapped augmented views (i. e., the chaos) create a ladder for contrastive learning to gradually learn class-separated representations.

Contrastive Learning Model Selection +1

Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation

1 code implementation CVPR 2022 Qingyan Meng, Mingqing Xiao, Shen Yan, Yisen Wang, Zhouchen Lin, Zhi-Quan Luo

In this paper, we propose the Differentiation on Spike Representation (DSR) method, which could achieve high performance that is competitive to ANNs yet with low latency.

SymNMF-Net for The Symmetric NMF Problem

no code implementations26 May 2022 Mingjie Li, Hao Kong, Zhouchen Lin

Furthermore, we analyze the constraints of the inversion layer to ensure the output stability of the network to a certain extent.

Clustering

Global Convergence of Over-parameterized Deep Equilibrium Models

no code implementations27 May 2022 Zenan Ling, Xingyu Xie, Qiuhao Wang, Zongpeng Zhang, Zhouchen Lin

A deep equilibrium model (DEQ) is implicitly defined through an equilibrium point of an infinite-depth weight-tied model with an input-injection.

Two-Dimensional Weisfeiler-Lehman Graph Neural Networks for Link Prediction

1 code implementation20 Jun 2022 Yang Hu, Xiyuan Wang, Zhouchen Lin, Pan Li, Muhan Zhang

As pointed out by previous works, this two-step procedure results in low discriminating power, as 1-WL-GNNs by nature learn node-level representations instead of link-level.

Link Prediction Vocal Bursts Valence Prediction

Optimization-Induced Graph Implicit Nonlinear Diffusion

1 code implementation29 Jun 2022 Qi Chen, Yifei Wang, Yisen Wang, Jiansheng Yang, Zhouchen Lin

Moreover, we show that the optimization-induced variants of our models can boost the performance and improve training stability and efficiency as well.

PDO-s3DCNNs: Partial Differential Operator Based Steerable 3D CNNs

1 code implementation7 Aug 2022 Zhengyang Shen, Tao Hong, Qi She, Jinwen Ma, Zhouchen Lin

Steerable models can provide very general and flexible equivariance by formulating equivariance requirements in the language of representation theory and feature fields, which has been recognized to be effective for many vision tasks.

Retrieval

Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models

4 code implementations13 Aug 2022 Xingyu Xie, Pan Zhou, Huan Li, Zhouchen Lin, Shuicheng Yan

Adan first reformulates the vanilla Nesterov acceleration to develop a new Nesterov momentum estimation (NME) method, which avoids the extra overhead of computing gradient at the extrapolation point.

Rethinking Knowledge Graph Evaluation Under the Open-World Assumption

1 code implementation19 Sep 2022 Haotong Yang, Zhouchen Lin, Muhan Zhang

However, evaluation of knowledge graph completion (KGC) models often ignores the incompleteness -- facts in the test set are ranked against all unknown triplets which may contain a large number of missing facts not included in the KG yet.

Fairness

Online Training Through Time for Spiking Neural Networks

1 code implementation9 Oct 2022 Mingqing Xiao, Qingyan Meng, Zongpeng Zhang, Di He, Zhouchen Lin

With OTTT, it is the first time that two mainstream supervised SNN training methods, BPTT with SG and spike representation-based training, are connected, and meanwhile in a biologically plausible form.

Event data classification Gesture Recognition +1

Invertible Rescaling Network and Its Extensions

1 code implementation9 Oct 2022 Mingqing Xiao, Shuxin Zheng, Chang Liu, Zhouchen Lin, Tie-Yan Liu

To be specific, we develop invertible models to generate valid degraded images and meanwhile transform the distribution of lost contents to the fixed distribution of a latent variable during the forward degradation.

Colorization Image Compression

Towards Theoretically Inspired Neural Initialization Optimization

1 code implementation12 Oct 2022 Yibo Yang, Hong Wang, Haobo Yuan, Zhouchen Lin

With NIO, we improve the classification performance of a variety of neural architectures on CIFAR-10, CIFAR-100, and ImageNet.

PanopticPartFormer++: A Unified and Decoupled View for Panoptic Part Segmentation

1 code implementation3 Jan 2023 Xiangtai Li, Shilin Xu, Yibo Yang, Haobo Yuan, Guangliang Cheng, Yunhai Tong, Zhouchen Lin, Ming-Hsuan Yang, DaCheng Tao

Third, inspired by Mask2Former, based on our meta-architecture, we propose Panoptic-PartFormer++ and design a new part-whole cross-attention scheme to boost part segmentation qualities further.

Panoptic Segmentation Segmentation

SPIDE: A Purely Spike-based Method for Training Feedback Spiking Neural Networks

1 code implementation1 Feb 2023 Mingqing Xiao, Qingyan Meng, Zongpeng Zhang, Yisen Wang, Zhouchen Lin

In this paper, we study spike-based implicit differentiation on the equilibrium state (SPIDE) that extends the recently proposed training method, implicit differentiation on the equilibrium state (IDE), for supervised learning with purely spike-based computation, which demonstrates the potential for energy-efficient training of SNNs.

Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class-Incremental Learning

1 code implementation ICLR 2023 Yibo Yang, Haobo Yuan, Xiangtai Li, Zhouchen Lin, Philip Torr, DaCheng Tao

In this paper, we deal with this misalignment dilemma in FSCIL inspired by the recently discovered phenomenon named neural collapse, which reveals that the last-layer features of the same class will collapse into a vertex, and the vertices of all classes are aligned with the classifier prototypes, which are formed as a simplex equiangular tight frame (ETF).

Few-Shot Class-Incremental Learning Incremental Learning

Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learning

1 code implementation6 Feb 2023 Yibo Yang, Haobo Yuan, Xiangtai Li, Zhouchen Lin, Philip Torr, DaCheng Tao

In this paper, we deal with this misalignment dilemma in FSCIL inspired by the recently discovered phenomenon named neural collapse, which reveals that the last-layer features of the same class will collapse into a vertex, and the vertices of all classes are aligned with the classifier prototypes, which are formed as a simplex equiangular tight frame (ETF).

Few-Shot Class-Incremental Learning Incremental Learning

Towards Memory- and Time-Efficient Backpropagation for Training Spiking Neural Networks

1 code implementation ICCV 2023 Qingyan Meng, Mingqing Xiao, Shen Yan, Yisen Wang, Zhouchen Lin, Zhi-Quan Luo

In particular, our method achieves state-of-the-art accuracy on ImageNet, while the memory cost and training time are reduced by more than 70% and 50%, respectively, compared with BPTT.

PAPAL: A Provable PArticle-based Primal-Dual ALgorithm for Mixed Nash Equilibrium

no code implementations2 Mar 2023 Shihong Ding, Hanze Dong, Cong Fang, Zhouchen Lin, Tong Zhang

To circumvent this difficulty, we examine the problem of identifying a mixed Nash equilibrium, where strategies are randomized and characterized by probability distributions over continuous domains. To this end, we propose PArticle-based Primal-dual ALgorithm (PAPAL) tailored for a weakly entropy-regularized min-max optimization over probability distributions.

A Message Passing Perspective on Learning Dynamics of Contrastive Learning

1 code implementation8 Mar 2023 Yifei Wang, Qi Zhang, Tianqi Du, Jiansheng Yang, Zhouchen Lin, Yisen Wang

In recent years, contrastive learning achieves impressive results on self-supervised visual representation learning, but there still lacks a rigorous understanding of its learning dynamics.

Contrastive Learning Graph Attention +1

LION: Implicit Vision Prompt Tuning

no code implementations17 Mar 2023 Haixin Wang, Jianlong Chang, Xiao Luo, Jinan Sun, Zhouchen Lin, Qi Tian

Despite recent competitive performance across a range of vision tasks, vision Transformers still have an issue of heavy computational costs.

Transfer Learning

Visual Tuning

no code implementations10 May 2023 Bruce X. B. Yu, Jianlong Chang, Haixin Wang, Lingbo Liu, Shijie Wang, Zhiyu Wang, Junfan Lin, Lingxi Xie, Haojie Li, Zhouchen Lin, Qi Tian, Chang Wen Chen

With the surprising development of pre-trained visual foundation models, visual tuning jumped out of the standard modus operandi that fine-tunes the whole pre-trained model or just the fully connected layer.

Policy Representation via Diffusion Probability Model for Reinforcement Learning

1 code implementation22 May 2023 Long Yang, Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, Zhouchen Lin

Popular reinforcement learning (RL) algorithms tend to produce a unimodal policy distribution, which weakens the expressiveness of complicated policy and decays the ability of exploration.

Continuous Control reinforcement-learning +1

Code Prompting: a Neural Symbolic Method for Complex Reasoning in Large Language Models

no code implementations29 May 2023 Yi Hu, Haotong Yang, Zhouchen Lin, Muhan Zhang

We also consider the ensemble of code prompting and CoT prompting to combine the strengths of both.

Arithmetic Reasoning

Neural Collapse Terminus: A Unified Solution for Class Incremental Learning and Its Variants

2 code implementations3 Aug 2023 Yibo Yang, Haobo Yuan, Xiangtai Li, Jianlong Wu, Lefei Zhang, Zhouchen Lin, Philip Torr, DaCheng Tao, Bernard Ghanem

Beyond the normal case, long-tail class incremental learning and few-shot class incremental learning are also proposed to consider the data imbalance and data scarcity, respectively, which are common in real-world implementations and further exacerbate the well-known problem of catastrophic forgetting.

Few-Shot Class-Incremental Learning Incremental Learning

ShadowNet for Data-Centric Quantum System Learning

no code implementations22 Aug 2023 Yuxuan Du, Yibo Yang, Tongliang Liu, Zhouchen Lin, Bernard Ghanem, DaCheng Tao

Understanding the dynamics of large quantum systems is hindered by the curse of dimensionality.

Quantum State Tomography

CORE: Common Random Reconstruction for Distributed Optimization with Provable Low Communication Complexity

no code implementations23 Sep 2023 Pengyun Yue, Hanzhen Zhao, Cong Fang, Di He, LiWei Wang, Zhouchen Lin, Song-Chun Zhu

With distributed machine learning being a prominent technique for large-scale machine learning tasks, communication complexity has become a major bottleneck for speeding up training and scaling up machine numbers.

Distributed Optimization

Explaining the Complex Task Reasoning of Large Language Models with Template-Content Structure

no code implementations9 Oct 2023 Haotong Yang, Fanxu Meng, Zhouchen Lin, Muhan Zhang

Our framework offers an explanatory tool for the complex reasoning abilities of large language models from the perspective of modeling autoregressive generation tasks.

Answer Generation Language Modelling

On the $O(\frac{\sqrt{d}}{T^{1/4}})$ Convergence Rate of RMSProp and Its Momentum Extension Measured by $\ell_1$ Norm: Better Dependence on the Dimension

no code implementations1 Feb 2024 Huan Li, Zhouchen Lin

Although adaptive gradient methods have been extensively used in deep learning, their convergence rates have not been thoroughly studied, particularly with respect to their dependence on the dimension.

Hebbian Learning based Orthogonal Projection for Continual Learning of Spiking Neural Networks

1 code implementation19 Feb 2024 Mingqing Xiao, Qingyan Meng, Zongpeng Zhang, Di He, Zhouchen Lin

Neuromorphic computing with spiking neural networks is promising for energy-efficient artificial intelligence (AI) applications.

Continual Learning

DIGIC: Domain Generalizable Imitation Learning by Causal Discovery

no code implementations29 Feb 2024 Yang Chen, Yitao Liang, Zhouchen Lin

Causality has been combined with machine learning to produce robust representations for domain generalization.

Causal Discovery Domain Generalization +1

Implicit Euler Skip Connections: Enhancing Adversarial Robustness via Numerical Stability

no code implementations ICML 2020 Mingjie Li, Lingshen He, Zhouchen Lin

By viewing ResNet as an explicit Euler discretization of an ordinary differential equation (ODE), for the first time, we find that the adversarial robustness of ResNet is connected to the numerical stability of the corresponding dynamic system.

Adversarial Attack Adversarial Robustness

Boosted Histogram Transform for Regression

no code implementations ICML 2020 Yuchao Cai, Hanyuan Hang, Hanfang Yang, Zhouchen Lin

In this paper, we propose a boosting algorithm for regression problems called \textit{boosted histogram transform for regression} (BHTR) based on histogram transforms composed of random rotations, stretchings, and translations.

regression

Cannot find the paper you are looking for? You can Submit a new open access paper.