Search Results for author: Ekin Akyürek

Found 14 papers, 10 papers with code

In-Context Language Learning: Architectures and Algorithms

1 code implementation23 Jan 2024 Ekin Akyürek, Bailin Wang, Yoon Kim, Jacob Andreas

Finally, we show that hard-wiring these heads into neural models improves performance not just on ICLL, but natural language modeling -- improving the perplexity of 340M-parameter models by up to 1. 14 points (6. 7%) on the SlimPajama dataset.

In-Context Learning Language Modelling

Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatability

no code implementations16 Jan 2024 Afra Feyza Akyürek, Ekin Akyürek, Leshem Choshen, Derry Wijaya, Jacob Andreas

Given a collection of seed documents, DCT prompts LMs to generate additional text implied by these documents, reason globally about the correctness of this generated text, and finally fine-tune on text inferred to be correct.

Fact Verification Text Generation

What learning algorithm is in-context learning? Investigations with linear models

no code implementations28 Nov 2022 Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, Denny Zhou

We investigate the hypothesis that transformer-based in-context learners implement standard learning algorithms implicitly, by encoding smaller models in their activations, and updating these implicit models as new examples appear in the context.

In-Context Learning regression

Towards Tracing Factual Knowledge in Language Models Back to the Training Data

2 code implementations23 May 2022 Ekin Akyürek, Tolga Bolukbasi, Frederick Liu, Binbin Xiong, Ian Tenney, Jacob Andreas, Kelvin Guu

In this paper, we propose the problem of fact tracing: identifying which training examples taught an LM to generate a particular factual assertion.

Information Retrieval Retrieval

Pre-Trained Language Models for Interactive Decision-Making

1 code implementation3 Feb 2022 Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, Jacob Andreas, Igor Mordatch, Antonio Torralba, Yuke Zhu

Together, these results suggest that language modeling induces representations that are useful for modeling not just language, but also goals and plans; these representations can aid learning and generalization even outside of language processing.

Imitation Learning Language Modelling

Compositionality as Lexical Symmetry

1 code implementation30 Jan 2022 Ekin Akyürek, Jacob Andreas

In tasks like semantic parsing, instruction following, and question answering, standard deep networks fail to generalize compositionally from small datasets.

Data Augmentation Inductive Bias +5

Subspace Regularizers for Few-Shot Class Incremental Learning

1 code implementation ICLR 2022 Afra Feyza Akyürek, Ekin Akyürek, Derry Tanti Wijaya, Jacob Andreas

The key to this approach is a new family of subspace regularization schemes that encourage weight vectors for new classes to lie close to the subspace spanned by the weights of existing classes.

Few-Shot Class-Incremental Learning Image Classification +2

Language Model Pre-training Improves Generalization in Policy Learning

no code implementations29 Sep 2021 Shuang Li, Xavier Puig, Yilun Du, Ekin Akyürek, Antonio Torralba, Jacob Andreas, Igor Mordatch

Additional experiments explore the role of language-based encodings in these results; we find that it is possible to train a simple adapter layer that maps from observations and action histories to LM embeddings, and thus that language modeling provides an effective initializer even for tasks with no language as input or output.

Imitation Learning Language Modelling

Lexicon Learning for Few-Shot Neural Sequence Modeling

1 code implementation7 Jun 2021 Ekin Akyürek, Jacob Andreas

Sequence-to-sequence transduction is the core problem in language processing applications as diverse as semantic parsing, machine translation, and instruction following.

Instruction Following Machine Translation +3

Morphological analysis using a sequence decoder

2 code implementations TACL 2019 Ekin Akyürek, Erenay Dayanik, Deniz Yuret

Our Morse implementation and the TrMor2018 dataset are available online to support future research\footnote{See \url{https://github. com/ai-ku/Morse. jl} for a Morse implementation in Julia/Knet \cite{knet2016mlsys} and \url{https://github. com/ai-ku/TrMor2018} for the new Turkish dataset.

Decoder LEMMA +4

Cannot find the paper you are looking for? You can Submit a new open access paper.