Search Results for author: Imanol Schlag

Found 9 papers, 7 papers with code

Improving Baselines in the Wild

1 code implementation31 Dec 2021 Kazuki Irie, Imanol Schlag, Róbert Csordás, Jürgen Schmidhuber

We share our experience with the recently released WILDS benchmark, a collection of ten datasets dedicated to developing models and training strategies which are robust to domain shifts.

Going Beyond Linear Transformers with Recurrent Fast Weight Programmers

3 code implementations NeurIPS 2021 Kazuki Irie, Imanol Schlag, Róbert Csordás, Jürgen Schmidhuber

Transformers with linearised attention (''linear Transformers'') have demonstrated the practical scalability and effectiveness of outer product-based Fast Weight Programmers (FWPs) from the '90s.

Atari Games

Linear Transformers Are Secretly Fast Weight Programmers

6 code implementations22 Feb 2021 Imanol Schlag, Kazuki Irie, Jürgen Schmidhuber

We show the formal equivalence of linearised self-attention mechanisms and fast weight controllers from the early '90s, where a ``slow" neural net learns by gradient descent to program the ``fast weights" of another net through sequences of elementary programming instructions which are additive outer products of self-invented activation patterns (today called keys and values).

Language Modelling Machine Translation +1

Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving

2 code implementations15 Oct 2019 Imanol Schlag, Paul Smolensky, Roland Fernandez, Nebojsa Jojic, Jürgen Schmidhuber, Jianfeng Gao

We incorporate Tensor-Product Representations within the Transformer in order to better support the explicit representation of relation structure.

Question Answering

Learning to Reason with Third Order Tensor Products

1 code implementation NeurIPS 2018 Imanol Schlag, Jürgen Schmidhuber

We combine Recurrent Neural Networks with Tensor Product Representations to learn combinatorial representations of sequential data.

Learning to Reason with Third-Order Tensor Products

1 code implementation29 Nov 2018 Imanol Schlag, Jürgen Schmidhuber

We combine Recurrent Neural Networks with Tensor Product Representations to learn combinatorial representations of sequential data.

GATED FAST WEIGHTS FOR ASSOCIATIVE RETRIEVAL

no code implementations ICLR 2018 Imanol Schlag, Jürgen Schmidhuber

We improve previous end-to-end differentiable neural networks (NNs) with fast weight memories.

Cannot find the paper you are looking for? You can Submit a new open access paper.