Search Results for author:
Found 4 papers, 3 papers with code
Look Before You Leap: A Universal Emergent Decomposition of Retrieval Tasks in Language Models
We find that LMs internally decompose retrieval tasks in a modular way: middle layers at the last token position process the request, while late layers retrieve the correct entity from the context.
Interpreting Neural Networks through the Polytope Lens
Previous mechanistic descriptions have used individual neurons or their linear combinations to understand the representations a network has learned.
Scatterbrain: Unifying Sparse and Low-rank Attention Approximation
Recent advances in efficient Transformers have exploited either the sparsity or low-rank properties of attention matrices to reduce the computational and memory bottlenecks of modeling long sequences.
Scatterbrain: Unifying Sparse and Low-rank Attention
Recent advances in efficient Transformers have exploited either the sparsity or low-rank properties of attention matrices to reduce the computational and memory bottlenecks of modeling long sequences.
