Search Results for author:
Found 6 papers, 4 papers with code
Out-of-distribution generalization via composition: a lens through induction heads in Transformers
We empirically examined the training dynamics of Transformers on a synthetic example and conducted extensive experiments on a variety of pretrained LLMs, focusing on a type of components known as induction heads.
Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalability
In this study, we delve into the ICL capabilities of LLMs on composite tasks, with only simple tasks as in-context examples.
Why Larger Language Models Do In-context Learning Differently?
This sheds light on where transformers pay attention to and how that affects ICL.
Conv-Basis: A New Paradigm for Efficient Attention Inference and Gradient Computation in Transformers
We then design a fast algorithm to approximate the attention matrix via a sum of such $k$ convolution matrices.
Towards Few-Shot Adaptation of Foundation Models via Multitask Finetuning
An emerging solution with recent success in vision and NLP involves finetuning a foundation model on a selection of relevant tasks, before its adaptation to a target task with limited labeled samples.
Spatial Transcriptomics Dimensionality Reduction using Wavelet Bases
We illustrate the performance of our methods by spatial structure recovery and gene expression reconstruction in simulation.
