Search Results for author:
Found 9 papers, 6 papers with code
Masked Completion via Structured Diffusion with White-Box Transformers
We do this by exploiting a fundamental connection between diffusion, compression, and (masked) completion, deriving a deep transformer-like masked autoencoder architecture, called CRATE-MAE, in which the role of each layer is mathematically fully interpretable: they transform the data distribution to and from a structured representation.
White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is?
This leads to a family of white-box transformer-like deep network architectures, named CRATE, which are mathematically fully interpretable.
What's in a Prior? Learned Proximal Networks for Inverse Problems
Proximal operators are ubiquitous in inverse problems, commonly appearing as part of algorithmic strategies to regularize problems that are otherwise ill-posed.
Emergence of Segmentation with Minimalistic White-Box Transformers
Transformer-like models for vision tasks have recently proven effective for a wide range of downstream applications such as segmentation and detection.
Canonical Factors for Hybrid Neural Fields
Factored feature volumes offer a simple way to build more compact, efficient, and intepretable neural fields, but also introduce biases that are not necessarily beneficial for real-world data.
White-Box Transformers via Sparse Rate Reduction
Particularly, we show that the standard transformer block can be derived from alternating optimization on complementary parts of this objective: the multi-head self-attention operator can be viewed as a gradient descent step to compress the token sets by minimizing their lossy coding rate, and the subsequent multi-layer perceptron can be viewed as attempting to sparsify the representation of the tokens.
Resource-Efficient Invariant Networks: Exponential Gains by Unrolled Optimization
Achieving invariance to nuisance transformations is a fundamental challenge in the construction of robust and reliable vision systems.
Deep Networks Provably Classify Data on Curves
Data with low-dimensional nonlinear structure are ubiquitous in engineering and scientific problems.
Deep Networks and the Multiple Manifold Problem
Our analysis demonstrates concrete benefits of depth and width in the context of a practically-motivated model problem: the depth acts as a fitting resource, with larger depths corresponding to smoother networks that can more readily separate the class manifolds, and the width acts as a statistical resource, enabling concentration of the randomly-initialized network and its gradients.
