Search Results for author:
Found 8 papers, 1 papers with code
Concise One-Layer Transformers Can Do Function Evaluation (Sometimes)
While transformers have proven enormously successful in a range of tasks, their fundamental properties as models of computation are not well understood.
Simulating Hard Attention Using Soft Attention
Second, we demonstrate how temperature scaling allows softmax transformers to simulate a large subclass of average-hard attention transformers, those that have what we call the uniform-tieless property.
Transformers as Transducers
B-RASP[pos] enables calculations on positions (such as copying the first half of a string) and contains all first-order regular functions.
What Formal Languages Can Transformers Express? A Survey
As transformers have gained prominence in natural language processing, some researchers have investigated theoretically what problems they can and cannot solve, by treating problems as formal languages.
Masked Hard-Attention Transformers Recognize Exactly the Star-Free Languages
The expressive power of transformers over inputs of unbounded size can be studied through their ability to recognize classes of formal languages.
Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexity
This paper analyzes three formal models of Transformer encoders that differ in the form of their self-attention mechanism: unique hard attention (UHAT); generalized unique hard attention (GUHAT), which generalizes UHAT; and averaging hard attention (AHAT).
Regular omega-Languages with an Informative Right Congruence
The right congruence of a regular omega-language is not informative enough; many regular omega-languages have a trivial right congruence, and in general it is not always possible to define an omega-automaton recognizing a given language that is isomorphic to the rightcon automaton.
Context-Free Transductions with Neural Stacks
This paper analyzes the behavior of stack-augmented recurrent neural network (RNN) models.
