Search Results for author: Nathanael Schärli

Found 8 papers, 5 papers with code

Teaching Large Language Models to Self-Debug

no code implementations11 Apr 2023 Xinyun Chen, Maxwell Lin, Nathanael Schärli, Denny Zhou

In particular, we demonstrate that Self-Debugging can teach the large language model to perform rubber duck debugging; i. e., without any human feedback on the code correctness or error messages, the model is able to identify its mistakes by investigating the execution results and explaining the generated code in natural language.

Code Generation Language Modelling +3

Large Language Models Can Be Easily Distracted by Irrelevant Context

1 code implementation31 Jan 2023 Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, Denny Zhou

We use this benchmark to measure the distractibility of cutting-edge prompting techniques for large language models, and find that the model performance is dramatically decreased when irrelevant information is included.

Arithmetic Reasoning Language Modelling +1

Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them

2 code implementations17 Oct 2022 Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, Jason Wei

BIG-Bench (Srivastava et al., 2022) is a diverse evaluation suite that focuses on tasks believed to be beyond the capabilities of current language models.

Language Modelling

*-CFQ: Analyzing the Scalability of Machine Learning on a Compositional Task

no code implementations15 Dec 2020 Dmitry Tsarkov, Tibor Tihon, Nathan Scales, Nikola Momchev, Danila Sinopalnikov, Nathanael Schärli

We present *-CFQ ("star-CFQ"): a suite of large-scale datasets of varying scope based on the CFQ semantic parsing benchmark, designed for principled investigation of the scalability of machine learning systems in a realistic compositional task setting.

BIG-bench Machine Learning Semantic Parsing

Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures

1 code implementation17 Jul 2020 Daniel Furrer, Marc van Zee, Nathan Scales, Nathanael Schärli

While mainstream machine learning methods are known to have limited ability to compositionally generalize, new architectures and techniques continue to be proposed to address this limitation.

Language Modelling Semantic Parsing

Measuring Compositional Generalization: A Comprehensive Method on Realistic Data

3 code implementations ICLR 2020 Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, Olivier Bousquet

We present a large and realistic natural language question answering dataset that is constructed according to this method, and we use it to analyze the compositional generalization ability of three machine learning architectures.

BIG-bench Machine Learning Question Answering +1

Cannot find the paper you are looking for? You can Submit a new open access paper.