Search Results for author:
Found 4 papers, 1 papers with code
Simulating Hard Attention Using Soft Attention
Second, we demonstrate how temperature scaling allows softmax transformers to simulate a large subclass of average-hard attention transformers, those that have what we call the uniform-tieless property.
A Formal Framework for Understanding Length Generalization in Transformers
A major challenge for transformers is generalizing to sequences longer than those observed during training.
Counting Like Transformers: Compiling Temporal Counting Logic Into Softmax Transformers
Deriving formal bounds on the expressivity of transformers, as well as studying transformers that are constructed to implement known algorithms, are both effective methods for better understanding the computational power of transformers.
Masked Hard-Attention Transformers Recognize Exactly the Star-Free Languages
The expressive power of transformers over inputs of unbounded size can be studied through their ability to recognize classes of formal languages.
