no code implementations • 6 Sep 2024 • Jackson Petty, Sjoerd van Steenkiste, Tal Linzen
Large language models are increasingly trained on corpora containing both natural language and non-linguistic data like source code.
1 code implementation • 12 Apr 2024 • William Merrill, Jackson Petty, Ashish Sabharwal
Our analysis reveals that the expressive power of SSMs is limited very similarly to transformers: SSMs cannot express computation outside the complexity class $\mathsf{TC}^0$.
2 code implementations • 20 Nov 2023 • David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, Samuel R. Bowman
We present GPQA, a challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry.
1 code implementation • 15 Nov 2023 • Julian Michael, Salsabila Mahdi, David Rein, Jackson Petty, Julien Dirani, Vishakh Padmakumar, Samuel R. Bowman
Comparing debate to a baseline we call consultancy, where a single expert argues for only one answer which is correct half of the time, we find that debate performs significantly better, with 84% judge accuracy compared to consultancy's 74%.
1 code implementation • 13 Nov 2023 • Aaron Mueller, Albert Webson, Jackson Petty, Tal Linzen
In-context learning (ICL) is now a common method for teaching large language models (LLMs) new tasks: given labeled examples in the input context, the LLM learns to perform the task without weight updates.
1 code implementation • 8 Nov 2023 • Michael Wilson, Jackson Petty, Robert Frank
We find that LLMs perform well in generalizing the distribution of a novel noun argument between related contexts that were seen during pre-training (e. g., the active object and passive subject of the verb spray), succeeding by making use of the semantically-organized structure of the embedding space for word embeddings.
no code implementations • 30 Oct 2023 • Jackson Petty, Sjoerd van Steenkiste, Ishita Dasgupta, Fei Sha, Dan Garrette, Tal Linzen
Because model latency is approximately linear in the number of layers, these results lead us to the recommendation that, with a given total parameter budget, transformers can be made shallower than is typical without sacrificing performance.
1 code implementation • 20 Dec 2022 • Najoung Kim, Phu Mon Htut, Samuel R. Bowman, Jackson Petty
Naturally occurring information-seeking questions often contain questionable assumptions -- assumptions that are false or unverifiable.
no code implementations • 8 Feb 2022 • Jackson Petty, Michael Wilson, Robert Frank
How is knowledge of position-role mappings in natural language learned?
1 code implementation • 24 Sep 2021 • Jackson Petty, Robert Frank
Natural language exhibits patterns of hierarchically governed dependencies, in which relations between words are sensitive to syntactic structure rather than linear ordering.
1 code implementation • COLING (CRAC) 2020 • Robert Frank, Jackson Petty
Reflexive anaphora present a challenge for semantic interpretation: their meaning varies depending on context in a way that appears to require abstract variables.