Search Results for author: Ziyang Wu

Found 12 papers, 9 papers with code

Emergence of Segmentation with Minimalistic White-Box Transformers

1 code implementation30 Aug 2023 Yaodong Yu, Tianzhe Chu, Shengbang Tong, Ziyang Wu, Druv Pai, Sam Buchanan, Yi Ma

Transformer-like models for vision tasks have recently proven effective for a wide range of downstream applications such as segmentation and detection.

Self-Supervised Learning

White-Box Transformers via Sparse Rate Reduction

1 code implementation1 Jun 2023 Yaodong Yu, Sam Buchanan, Druv Pai, Tianzhe Chu, Ziyang Wu, Shengbang Tong, Benjamin D. Haeffele, Yi Ma

Particularly, we show that the standard transformer block can be derived from alternating optimization on complementary parts of this objective: the multi-head self-attention operator can be viewed as a gradient descent step to compress the token sets by minimizing their lossy coding rate, and the subsequent multi-layer perceptron can be viewed as attempting to sparsify the representation of the tokens.

Representation Learning

Efficient Maximal Coding Rate Reduction by Variational Forms

no code implementations CVPR 2022 Christina Baek, Ziyang Wu, Kwan Ho Ryan Chan, Tianjiao Ding, Yi Ma, Benjamin D. Haeffele

The principle of Maximal Coding Rate Reduction (MCR$^2$) has recently been proposed as a training objective for learning discriminative low-dimensional structures intrinsic to high-dimensional data to allow for more robust training than standard approaches, such as cross-entropy minimization.

Image Classification

Incremental Learning of Structured Memory via Closed-Loop Transcription

1 code implementation11 Feb 2022 Shengbang Tong, Xili Dai, Ziyang Wu, Mingyang Li, Brent Yi, Yi Ma

Our method is simpler than existing approaches for incremental learning, and more efficient in terms of model size, storage, and computation: it requires only a single, fixed-capacity autoencoding network with a feature space that is used for both discriminative and generative purposes.

Incremental Learning

Closed-Loop Data Transcription to an LDR via Minimaxing Rate Reduction

1 code implementation12 Nov 2021 Xili Dai, Shengbang Tong, Mingyang Li, Ziyang Wu, Michael Psenka, Kwan Ho Ryan Chan, Pengyuan Zhai, Yaodong Yu, Xiaojun Yuan, Heung Yeung Shum, Yi Ma

In particular, we propose to learn a closed-loop transcription between a multi-class multi-dimensional data distribution and a linear discriminative representation (LDR) in the feature space that consists of multiple independent multi-dimensional linear subspaces.

Can We Characterize Tasks Without Labels or Features?

1 code implementation CVPR 2021 Bram Wallace, Ziyang Wu, Bharath Hariharan

The problem of expert model selection deals with choosing the appropriate pretrained network ("expert") to transfer to a target task.

Model Selection

Incremental Learning via Rate Reduction

no code implementations CVPR 2021 Ziyang Wu, Christina Baek, Chong You, Yi Ma

Current deep learning architectures suffer from catastrophic forgetting, a failure to retain knowledge of previously learned classes when incrementally trained on new classes.

Incremental Learning

Efficient AutoML Pipeline Search with Matrix and Tensor Factorization

1 code implementation7 Jun 2020 Chengrun Yang, Jicong Fan, Ziyang Wu, Madeleine Udell

Data scientists seeking a good supervised learning model on a new dataset have many choices to make: they must preprocess the data, select features, possibly reduce the dimension, select an estimation algorithm, and choose hyperparameters for each of these pipeline components.


PARN: Position-Aware Relation Networks for Few-Shot Learning

1 code implementation ICCV 2019 Ziyang Wu, Yuwei Li, Lihua Guo, Kui Jia

However, due to the inherent local connectivity of CNN, the CNN-based relation network (RN) can be sensitive to the spatial position relationship of semantic objects in two compared images.

Few-Shot Learning Relational Reasoning +1

Cannot find the paper you are looking for? You can Submit a new open access paper.