Search Results for author:
Found 4 papers, 2 papers with code
Nonparametric Variational Regularisation of Pretrained Transformers
We extend the NVIB framework to replace all types of attention functions in Transformers, and show that existing pretrained Transformers can be reinterpreted as Nonparametric Variational (NV) models using a proposed identity initialisation.
Learning to Abstract with Nonparametric Variational Information Bottleneck
Finally, we show that NVIB compression results in a model which is more robust to adversarial perturbations.
A Variational AutoEncoder for Transformers with Nonparametric Variational Information Bottleneck
We propose a VAE for Transformers by developing a variational information bottleneck regulariser for Transformer embeddings.
HyperMixer: An MLP-based Low Cost Alternative to Transformers
We find that existing architectures such as MLPMixer, which achieves token mixing through a static MLP applied to each feature independently, are too detached from the inductive biases required for natural language understanding.
