Search Results for author:
Found 4 papers, 4 papers with code
White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is?
This leads to a family of white-box transformer-like deep network architectures, named CRATE, which are mathematically fully interpretable.
Emergence of Segmentation with Minimalistic White-Box Transformers
Transformer-like models for vision tasks have recently proven effective for a wide range of downstream applications such as segmentation and detection.
Image Clustering via the Principle of Rate Reduction in the Age of Pretrained Models
In this paper, we propose a novel image clustering pipeline that leverages the powerful feature representation of large pre-trained models such as CLIP and cluster images effectively and efficiently at scale.
White-Box Transformers via Sparse Rate Reduction
Particularly, we show that the standard transformer block can be derived from alternating optimization on complementary parts of this objective: the multi-head self-attention operator can be viewed as a gradient descent step to compress the token sets by minimizing their lossy coding rate, and the subsequent multi-layer perceptron can be viewed as attempting to sparsify the representation of the tokens.
