no code implementations • 29 Nov 2023 • Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, Denny Zhou
Self-consistency with chain-of-thought prompting (CoT) has demonstrated remarkable performance gains on various challenging tasks, by utilizing multiple reasoning paths sampled from large language models (LLMs).
no code implementations • NeurIPS 2023 • Du Phan, Matthew D. Hoffman, David Dohan, Sholto Douglas, Tuan Anh Le, Aaron Parisi, Pavel Sountsov, Charles Sutton, Sharad Vikram, Rif A. Saurous
Large language models (LLMs) solve problems more accurately and interpretably when instructed to work out the answer step by step using a ``chain-of-thought'' (CoT) prompt.
no code implementations • 26 Jul 2023 • Kensen Shi, Joey Hong, Manzil Zaheer, Pengcheng Yin, Charles Sutton
When writing programs, people have the ability to tackle a new complex task by decomposing it into smaller and more familiar subtasks.
no code implementations • 11 Jun 2023 • Lazar Valkov, Akash Srivastava, Swarat Chaudhuri, Charles Sutton
To address this challenge, we develop a modular CL framework, called PICLE, that accelerates search by using a probabilistic model to cheaply compute the fitness of each composition.
no code implementations • 19 Dec 2022 • Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Alex Polozov, Charles Sutton
To measure the performance of AI pair programmers that automatically synthesize programs for those tasks given natural language (NL) intents from users, we build ARCADE, a benchmark of 1082 code generation problems using the pandas data analysis framework in data science notebooks.
1 code implementation • 15 Aug 2022 • David Bieber, Kensen Shi, Petros Maniatis, Charles Sutton, Vincent Hellendoorn, Daniel Johnson, Daniel Tarlow
Graph representations of programs are commonly a central element of machine learning for code research.
1 code implementation • 21 Jul 2022 • David Dohan, Winnie Xu, Aitor Lewkowycz, Jacob Austin, David Bieber, Raphael Gontijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A. Saurous, Jascha Sohl-Dickstein, Kevin Murphy, Charles Sutton
Prompted models have demonstrated impressive few-shot learning abilities.
1 code implementation • 17 Jul 2022 • Simao Eduardo, Kai Xu, Alfredo Nazabal, Charles Sutton
Seeing as a systematic outlier is a combination of patterns of a clean instance and systematic error patterns, our main insight is that inliers can be modelled by a smaller representation (subspace) in a model than outliers.
no code implementations • 7 Apr 2022 • Kensen Shi, Joey Hong, Manzil Zaheer, Pengcheng Yin, Charles Sutton
We first characterize several different axes along which program synthesis methods would be desired to generalize, e. g., length generalization, or the ability to combine known subroutines in new ways that do not occur in the training data.
5 code implementations • Google Research 2022 • Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, Noah Fiedel
To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM.
Ranked #1 on Coreference Resolution on Winograd Schema Challenge
1 code implementation • ICLR 2022 • Kensen Shi, Hanjun Dai, Kevin Ellis, Charles Sutton
Many approaches to program synthesis perform a search within an enormous space of programs to find one that satisfies a given specification.
no code implementations • NeurIPS 2021 • Shobha Vasudevan, Wenjie (Joe) Jiang, David Bieber, Rishabh Singh, hamid shojaei, C. Richard Ho, Charles Sutton
We evaluate Design2Vec on three real-world hardware designs, including an industrial chip used in commercial data centers.
no code implementations • NeurIPS 2021 • Kai Xu, Akash Srivastava, Dan Gutfreund, Felix Sosa, Tomer Ullman, Josh Tenenbaum, Charles Sutton
In this paper, we propose a Bayesian-symbolic framework (BSP) for physical reasoning and learning that is close to human-level sample-efficiency and accuracy.
no code implementations • 30 Nov 2021 • Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, Augustus Odena
Large pre-trained language models perform remarkably well on tasks that can be done "in one pass", such as generating realistic text or synthesizing computer programs.
no code implementations • NeurIPS Workshop AIPLANS 2021 • Eirene V. Pandi, Earl T. Barr, Andrew D. Gordon, Charles Sutton
We are the first to present a detailed algorithm for NTI that is validated with theorems and proofs.
1 code implementation • 16 Aug 2021 • Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, Charles Sutton
Our largest models, even without finetuning on a code dataset, can synthesize solutions to 59. 6 percent of the problems from MBPP using few-shot learning with a well-designed prompt.
1 code implementation • 26 Jun 2021 • Xinyun Chen, Petros Maniatis, Rishabh Singh, Charles Sutton, Hanjun Dai, Max Lin, Denny Zhou
In this work, we present the first approach for synthesizing spreadsheet formulas from tabular context, which includes both headers and semi-structured tabular data.
no code implementations • 1 Jan 2021 • Kai Xu, Akash Srivastava, Dan Gutfreund, Felix Sosa, Tomer Ullman, Joshua B. Tenenbaum, Charles Sutton
As such, learning the laws is then reduced to symbolic regression and Bayesian inference methods are used to obtain the distribution of unobserved properties.
no code implementations • 1 Dec 2020 • Joey Hong, David Dohan, Rishabh Singh, Charles Sutton, Manzil Zaheer
The latent codes are learned using a self-supervised learning principle, in which first a discrete autoencoder is trained on the output sequences, and then the resulting latent codes are used as intermediate targets for the end-to-end sequence prediction task.
no code implementations • NeurIPS 2020 • Hanjun Dai, Rishabh Singh, Bo Dai, Charles Sutton, Dale Schuurmans
In this paper we propose ALOE, a new algorithm for learning conditional and unconditional EBMs for discrete structured data, where parameter gradients are estimated using a learned sampler that mimics local search.
1 code implementation • NeurIPS 2020 • David Bieber, Charles Sutton, Hugo Larochelle, Daniel Tarlow
More practically, we evaluate these models on the task of learning to execute partial programs, as might arise if using the model as a heuristic function in program synthesis.
1 code implementation • 22 Oct 2020 • Maria I. Gorinova, Andrew D. Gordon, Charles Sutton, Matthijs Vákár
The resulting program can be seen as a hybrid inference algorithm on the original program, where continuous parameters can be drawn using efficient gradient-based inference methods, while the discrete parameters are inferred using variable elimination.
no code implementations • ICLR 2021 • Augustus Odena, Kensen Shi, David Bieber, Rishabh Singh, Charles Sutton, Hanjun Dai
Program synthesis is challenging largely because of the difficulty of search in a large space of programs.
no code implementations • 19 Jun 2020 • Matej Balog, Rishabh Singh, Petros Maniatis, Charles Sutton
We present a new program synthesis approach that combines an encoder-decoder based synthesis architecture with a differentiable program fixer.
1 code implementation • ICLR 2020 • Vincent J. Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, David Bieber
By studying a popular, non-trivial program repair task, variable-misuse identification, we explore the relative merits of traditional and hybrid model families for code representation.
no code implementations • 28 Apr 2020 • Rafael - Michael Karampatsis, Charles Sutton
Continuous embeddings of tokens in computer programs have been used to support a variety of software development tools, including readability, code search, and program repair.
1 code implementation • 1 Apr 2020 • Irene Vlassi Pandi, Earl T. Barr, Andrew D. Gordon, Charles Sutton
OptTyper combines a continuous interpretation of logical constraints derived by classical static analysis of TypeScript code, with natural constraints obtained from a deep learning model, which learns naming conventions for types from a large codebase.
2 code implementations • 17 Mar 2020 • Rafael-Michael Karampatsis, Hlib Babii, Romain Robbes, Charles Sutton, Andrea Janes
Statistical language modeling techniques have successfully been applied to large source code corpora, yielding a variety of new software development tools, such as tools for code suggestion, improving readability, and API migration.
Software Engineering
no code implementations • 27 Feb 2020 • Daniel A. Abolafia, Rishabh Singh, Manzil Zaheer, Charles Sutton
Main consists of a neural controller that interacts with a variable-length input tape and learns to compose modules together with their corresponding argument choices.
1 code implementation • ICML 2020 • Kensen Shi, David Bieber, Charles Sutton
Sampling is a fundamental technique, and sampling without replacement is often desirable when duplicate samples are not beneficial.
no code implementations • ICLR 2020 • Augustus Odena, Charles Sutton
We introduce the notion of property signatures, a representation for programs and program specifications meant for consumption by machine learning algorithms.
no code implementations • 4 Nov 2019 • Daniel Tarlow, Subhodeep Moitra, Andrew Rice, Zimin Chen, Pierre-Antoine Manzagol, Charles Sutton, Edward Aftandilian
A diff specifies how to modify the code's abstract syntax tree, represented in the neural network as a sequence of tokens and of pointers to code locations.
1 code implementation • 15 Jul 2019 • Simão Eduardo, Alfredo Nazábal, Christopher K. I. Williams, Charles Sutton
We show experimentally that not only RVAE performs better than several state-of-the-art methods in cell outlier detection and repair for tabular data, but also that is robust against the initial hyper-parameter selection.
2 code implementations • 30 May 2019 • Rafael-Michael Karampatsis, Charles Sutton
One way to achieve acceptable performance is to focus on classes of simple bugs, such as bugs with single statement fixes, or that match a small set of bug templates.
Software Engineering Programming Languages
1 code implementation • 30 May 2019 • Jiaoyan Chen, Ernesto Jimenez-Ruiz, Ian Horrocks, Charles Sutton
The usefulness of tabular data such as web tables critically depends on understanding their semantics.
Ranked #1 on Column Type Annotation on T2Dv2
1 code implementation • 13 Mar 2019 • Rafael-Michael Karampatsis, Charles Sutton
We present a new open-vocabulary neural language model for code that is not limited to a fixed vocabulary of identifier names.
2 code implementations • Data Mining and Knowledge Discovery 2019 • Gerrit J. J. van den Burg, Alfredo Nazabal, Charles Sutton
Existing dialect detection approaches are few and non-robust.
Databases E.5; H.2.8
1 code implementation • 4 Nov 2018 • Jiaoyan Chen, Ernesto Jimenez-Ruiz, Ian Horrocks, Charles Sutton
Automatically annotating column types with knowledge base (KB) concepts is a critical task to gain a basic understanding of web tables.
Ranked #1 on Column Type Annotation on T2Dv2 (F1 (%) metric)
1 code implementation • 2 Nov 2018 • Maria I. Gorinova, Andrew D. Gordon, Charles Sutton
Stan is a probabilistic programming language that has been increasingly used for real-world scalable projects.
1 code implementation • 12 Jun 2018 • Annie Louis, Santanu Kumar Dash, Earl T. Barr, Charles Sutton
To address this problem, we introduce the notion of comment entailment from code, high entailment indicating that a comment's natural language semantics can be inferred directly from the code.
no code implementations • NAACL 2018 • Annie Louis, Charles Sutton
An essential aspect to understanding narratives is to grasp the interaction between characters in a story and the actions they take.
no code implementations • ICLR 2020 • Akash Srivastava, Kai Xu, Michael U. Gutmann, Charles Sutton
In this work, we take their insight of using kernels as fixed adversaries further and present a novel method for training deep generative models that does not involve saddlepoint optimization.
no code implementations • 21 Apr 2018 • Akash Srivastava, Charles Sutton
The Pachinko Allocation Machine (PAM) is a deep topic model that allows representing rich correlation structures among topics by a directed acyclic graph over topics.
2 code implementations • NeurIPS 2018 • Lazar Valkov, Dipak Chaudhari, Akash Srivastava, Charles Sutton, Swarat Chaudhuri
We present a neurosymbolic framework for the lifelong learning of algorithmic tasks that mix perception and procedural reasoning.
no code implementations • 11 Mar 2018 • Kai Xu, Dae Hoon Park, Chang Yi, Charles Sutton
Interpreting black box classifiers, such as deep networks, allows an analyst to validate a classifier before it is deployed in a high-stakes setting.
3 code implementations • ASONAM 2019 • Benedek Rozemberczki, Ryan Davies, Rik Sarkar, Charles Sutton
In this paper we propose GEMSEC a graph embedding algorithm which learns a clustering of the nodes simultaneously with computing their features.
Ranked #1 on Community Detection on Facebook Athletes
Social and Information Networks
no code implementations • 18 Sep 2017 • Miltiadis Allamanis, Earl T. Barr, Premkumar Devanbu, Charles Sutton
We contrast programming languages against natural languages and discuss how these similarities and differences drive the design of probabilistic models.
1 code implementation • NeurIPS 2017 • Akash Srivastava, Lazar Valkov, Chris Russell, Michael U. Gutmann, Charles Sutton
Deep generative models provide powerful tools for distributions over complicated manifolds, such as those of natural images.
6 code implementations • 4 Mar 2017 • Akash Srivastava, Charles Sutton
A promising approach to address this problem is autoencoding variational Bayes (AEVB), but it has proven diffi- cult to apply to topic models in practice.
Ranked #6 on Topic Models on 20NewsGroups
8 code implementations • 29 Dec 2016 • Chaoyun Zhang, Mingjun Zhong, Zongzuo Wang, Nigel Goddard, Charles Sutton
Interestingly, we systematically show that the convolutional neural networks can inherently learn the signatures of the target appliances, which are automatically added into the model to reduce the identifiability problem.
no code implementations • 8 Nov 2016 • Miltiadis Allamanis, Earl T. Barr, René Just, Charles Sutton
The results demonstrate that the location selection heuristics produce mutants more closely coupled to real faults for a given budget of mutation operator applications.
Software Engineering
1 code implementation • ICML 2017 • Miltiadis Allamanis, Pankajan Chanthirasegaran, Pushmeet Kohli, Charles Sutton
Combining abstract, symbolic reasoning with continuous neural reasoning is a grand challenge of representation learning.
no code implementations • 19 Jun 2016 • Akash Srivastava, James Zou, Ryan P. Adams, Charles Sutton
A good clustering can help a data analyst to explore and understand a data set, but what constitutes a good clustering may depend on domain-specific and application-specific criteria.
no code implementations • 22 Feb 2016 • Akash Srivastava, James Zou, Charles Sutton
A good clustering can help a data analyst to explore and understand a data set, but what constitutes a good clustering may depend on domain-specific and application-specific criteria.
1 code implementation • 16 Feb 2016 • Jaroslav Fowkes, Charles Sutton
Recent sequential pattern mining methods have used the minimum description length (MDL) principle to define an encoding scheme which describes an algorithm for mining the most compressing patterns in a database.
5 code implementations • 9 Feb 2016 • Miltiadis Allamanis, Hao Peng, Charles Sutton
Attention mechanisms in neural networks have proved useful for problems in which the input and output do not have fixed dimension.
no code implementations • 19 Nov 2015 • Krzysztof J. Geras, Abdel-rahman Mohamed, Rich Caruana, Gregor Urban, Shengjie Wang, Ozlem Aslan, Matthai Philipose, Matthew Richardson, Charles Sutton
We consider whether deep convolutional networks (CNNs) can represent decision functions with similar accuracy as recurrent networks such as LSTMs.
Automatic Speech Recognition Automatic Speech Recognition (ASR) +3
1 code implementation • NeurIPS 2015 • Mingjun Zhong, Nigel Goddard, Charles Sutton
In many statistical problems, a more coarse-grained model may be suitable for population-level behaviour, whereas a more detailed model is appropriate for accurate modelling of individual behaviour.
1 code implementation • 14 Oct 2015 • Jaroslav Fowkes, Charles Sutton
Mining itemsets that are the most interesting under a statistical model of the underlying data is a commonly used and well-studied technique for exploratory data analysis, with the most recent interestingness models exhibiting state of the art performance.
no code implementations • NeurIPS 2014 • Mingjun Zhong, Nigel Goddard, Charles Sutton
Blind source separation problems are difficult because they are inherently unidentifiable, yet the entire goal is to identify meaningful sources.
no code implementations • NeurIPS 2014 • Yichuan Zhang, Charles Sutton
Sampling from hierarchical Bayesian models is often difficult for MCMC methods, because of the strong correlations between the model parameters and the hyperparameters.
1 code implementation • 12 Jun 2014 • Krzysztof J. Geras, Charles Sutton
We present a representation learning method that learns features at multiple different levels of scale.
no code implementations • 17 Nov 2010 • Charles Sutton, Andrew McCallum
This tutorial describes conditional random fields, a popular probabilistic method for structured prediction.