Theoretical Conditions and Empirical Failure of Bracket Counting on Long Sequences with Linear Recurrent Networks

no code implementations • 7 Apr 2023 • Nadine El-Naggar, Pranava Madhyastha, Tillman Weyde

We conduct a theoretical analysis of linear RNNs and identify conditions for the models to exhibit exact counting behaviour.

Exploring the Long-Term Generalization of Counting Behavior in RNNs

no code implementations • 29 Nov 2022 • Nadine El-Naggar, Pranava Madhyastha, Tillman Weyde

Despite this and some positive empirical results for LSTMs on Dyck-1 languages, our experimental results show that LSTMs fail to learn correct counting behavior for sequences that are significantly longer than in the training data.