Search Results for author: Nathan Scales

Found 8 papers, 3 papers with code

Large Language Models Can Be Easily Distracted by Irrelevant Context

no code implementations31 Jan 2023 Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, Denny Zhou

We use this benchmark to measure the distractibility of cutting-edge prompting techniques for large language models, and find that the model performance is dramatically decreased when irrelevant information is included.

Arithmetic Reasoning Language Modelling

Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them

1 code implementation17 Oct 2022 Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, Jason Wei

BIG-Bench (Srivastava et al., 2022) is a diverse evaluation suite that focuses on tasks believed to be beyond the capabilities of current language models.

Language Modelling

Least-to-Most Prompting Enables Complex Reasoning in Large Language Models

no code implementations21 May 2022 Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, Ed Chi

Although chain-of-thought prompting has shown impressive results on many natural language reasoning tasks, it often performs poorly on tasks which need to solve problems harder than the demonstration examples.

*-CFQ: Analyzing the Scalability of Machine Learning on a Compositional Task

no code implementations15 Dec 2020 Dmitry Tsarkov, Tibor Tihon, Nathan Scales, Nikola Momchev, Danila Sinopalnikov, Nathanael Schärli

We present *-CFQ ("star-CFQ"): a suite of large-scale datasets of varying scope based on the CFQ semantic parsing benchmark, designed for principled investigation of the scalability of machine learning systems in a realistic compositional task setting.

BIG-bench Machine Learning Semantic Parsing

Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures

1 code implementation17 Jul 2020 Daniel Furrer, Marc van Zee, Nathan Scales, Nathanael Schärli

While mainstream machine learning methods are known to have limited ability to compositionally generalize, new architectures and techniques continue to be proposed to address this limitation.

Language Modelling Semantic Parsing

Measuring Compositional Generalization: A Comprehensive Method on Realistic Data

2 code implementations ICLR 2020 Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, Olivier Bousquet

We present a large and realistic natural language question answering dataset that is constructed according to this method, and we use it to analyze the compositional generalization ability of three machine learning architectures.

BIG-bench Machine Learning Question Answering

Cannot find the paper you are looking for? You can Submit a new open access paper.