Search Results for author:
Found 9 papers, 4 papers with code
Transformers as Recognizers of Formal Languages: A Survey on Expressivity
As transformers have gained prominence in natural language processing, some researchers have investigated theoretically what problems they can and cannot solve, by treating problems as formal languages.
Discovering Knowledge-Critical Subnetworks in Pretrained Language Models
In this work, we investigate whether pretrained language models contain various knowledge-critical subnetworks: particular sparse computational subgraphs responsible for encoding specific knowledge the model has memorized.
RECKONING: Reasoning through Dynamic Knowledge Encoding
In the outer loop, the model learns to use the updated weights to reproduce and answer reasoning questions about the memorized knowledge.
Thinking Like Transformers
In this paper we aim to change that, proposing a computational model for the transformer-encoder in the form of a programming language.
Synthesizing Context-free Grammars from Recurrent Neural Networks (Extended Version)
We present an algorithm for extracting a subclass of the context free grammars (CFGs) from a trained recurrent neural network (RNN).
A Formal Hierarchy of RNN Architectures
While formally extending these findings to unsaturated RNNs is left to future work, we hypothesize that the practical learnable capacity of unsaturated RNNs obeys a similar hierarchy.
Learning Deterministic Weighted Automata with Queries and Counterexamples
We present an algorithm for extraction of a probabilistic deterministic finite automaton (PDFA) from a given black-box language model, such as a recurrent neural network (RNN).
On the Practical Computational Power of Finite Precision RNNs for Language Recognition
While Recurrent Neural Networks (RNNs) are famously known to be Turing complete, this relies on infinite precision in the states and unbounded computation time.
Extracting Automata from Recurrent Neural Networks Using Queries and Counterexamples
We do this using Angluin's L* algorithm as a learner and the trained RNN as an oracle.
