Search Results for author: Charles Sutton

Found 63 papers, 32 papers with code

Universal Self-Consistency for Large Language Model Generation

no code implementations29 Nov 2023 Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, Denny Zhou

Self-consistency with chain-of-thought prompting (CoT) has demonstrated remarkable performance gains on various challenging tasks, by utilizing multiple reasoning paths sampled from large language models (LLMs).

Code Generation Language Modelling +3

Training Chain-of-Thought via Latent-Variable Inference

no code implementations NeurIPS 2023 Du Phan, Matthew D. Hoffman, David Dohan, Sholto Douglas, Tuan Anh Le, Aaron Parisi, Pavel Sountsov, Charles Sutton, Sharad Vikram, Rif A. Saurous

Large language models (LLMs) solve problems more accurately and interpretably when instructed to work out the answer step by step using a ``chain-of-thought'' (CoT) prompt.

GSM8K

ExeDec: Execution Decomposition for Compositional Generalization in Neural Program Synthesis

no code implementations26 Jul 2023 Kensen Shi, Joey Hong, Manzil Zaheer, Pengcheng Yin, Charles Sutton

When writing programs, people have the ability to tackle a new complex task by decomposing it into smaller and more familiar subtasks.

Program Synthesis

A Probabilistic Framework for Modular Continual Learning

no code implementations11 Jun 2023 Lazar Valkov, Akash Srivastava, Swarat Chaudhuri, Charles Sutton

To address this challenge, we develop a modular CL framework, called PICLE, that accelerates search by using a probabilistic model to cheaply compute the fitness of each composition.

Continual Learning

Natural Language to Code Generation in Interactive Data Science Notebooks

no code implementations19 Dec 2022 Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Alex Polozov, Charles Sutton

To measure the performance of AI pair programmers that automatically synthesize programs for those tasks given natural language (NL) intents from users, we build ARCADE, a benchmark of 1082 code generation problems using the pandas data analysis framework in data science notebooks.

Code Generation Language Modelling

A Library for Representing Python Programs as Graphs for Machine Learning

1 code implementation15 Aug 2022 David Bieber, Kensen Shi, Petros Maniatis, Charles Sutton, Vincent Hellendoorn, Daniel Johnson, Daniel Tarlow

Graph representations of programs are commonly a central element of machine learning for code research.

Repairing Systematic Outliers by Learning Clean Subspaces in VAEs

1 code implementation17 Jul 2022 Simao Eduardo, Kai Xu, Alfredo Nazabal, Charles Sutton

Seeing as a systematic outlier is a combination of patterns of a clean instance and systematic error patterns, our main insight is that inliers can be modelled by a smaller representation (subspace) in a model than outliers.

Outlier Detection

Compositional Generalization and Decomposition in Neural Program Synthesis

no code implementations7 Apr 2022 Kensen Shi, Joey Hong, Manzil Zaheer, Pengcheng Yin, Charles Sutton

We first characterize several different axes along which program synthesis methods would be desired to generalize, e. g., length generalization, or the ability to combine known subroutines in new ways that do not occur in the training data.

Program Synthesis

CrossBeam: Learning to Search in Bottom-Up Program Synthesis

1 code implementation ICLR 2022 Kensen Shi, Hanjun Dai, Kevin Ellis, Charles Sutton

Many approaches to program synthesis perform a search within an enormous space of programs to find one that satisfies a given specification.

Program Synthesis Structured Prediction

A Bayesian-Symbolic Approach to Reasoning and Learning in Intuitive Physics

no code implementations NeurIPS 2021 Kai Xu, Akash Srivastava, Dan Gutfreund, Felix Sosa, Tomer Ullman, Josh Tenenbaum, Charles Sutton

In this paper, we propose a Bayesian-symbolic framework (BSP) for physical reasoning and learning that is close to human-level sample-efficiency and accuracy.

Bayesian Inference Bilevel Optimization +3

Show Your Work: Scratchpads for Intermediate Computation with Language Models

no code implementations30 Nov 2021 Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, Augustus Odena

Large pre-trained language models perform remarkably well on tasks that can be done "in one pass", such as generating realistic text or synthesizing computer programs.

Program Synthesis with Large Language Models

1 code implementation16 Aug 2021 Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, Charles Sutton

Our largest models, even without finetuning on a code dataset, can synthesize solutions to 59. 6 percent of the problems from MBPP using few-shot learning with a well-designed prompt.

Few-Shot Learning Program Synthesis

SpreadsheetCoder: Formula Prediction from Semi-structured Context

1 code implementation26 Jun 2021 Xinyun Chen, Petros Maniatis, Rishabh Singh, Charles Sutton, Hanjun Dai, Max Lin, Denny Zhou

In this work, we present the first approach for synthesizing spreadsheet formulas from tabular context, which includes both headers and semi-structured tabular data.

Program Synthesis

A Bayesian-Symbolic Approach to Learning and Reasoning for Intuitive Physics

no code implementations1 Jan 2021 Kai Xu, Akash Srivastava, Dan Gutfreund, Felix Sosa, Tomer Ullman, Joshua B. Tenenbaum, Charles Sutton

As such, learning the laws is then reduced to symbolic regression and Bayesian inference methods are used to obtain the distribution of unobserved properties.

Bayesian Inference Common Sense Reasoning +2

Latent Programmer: Discrete Latent Codes for Program Synthesis

no code implementations1 Dec 2020 Joey Hong, David Dohan, Rishabh Singh, Charles Sutton, Manzil Zaheer

The latent codes are learned using a self-supervised learning principle, in which first a discrete autoencoder is trained on the output sequences, and then the resulting latent codes are used as intermediate targets for the end-to-end sequence prediction task.

Document Summarization Program Synthesis +1

Learning Discrete Energy-based Models via Auxiliary-variable Local Exploration

no code implementations NeurIPS 2020 Hanjun Dai, Rishabh Singh, Bo Dai, Charles Sutton, Dale Schuurmans

In this paper we propose ALOE, a new algorithm for learning conditional and unconditional EBMs for discrete structured data, where parameter gradients are estimated using a learned sampler that mimics local search.

Language Modelling

Learning to Execute Programs with Instruction Pointer Attention Graph Neural Networks

1 code implementation NeurIPS 2020 David Bieber, Charles Sutton, Hugo Larochelle, Daniel Tarlow

More practically, we evaluate these models on the task of learning to execute partial programs, as might arise if using the model as a heuristic function in program synthesis.

Code Completion Learning to Execute +2

Conditional independence by typing

1 code implementation22 Oct 2020 Maria I. Gorinova, Andrew D. Gordon, Charles Sutton, Matthijs Vákár

The resulting program can be seen as a hybrid inference algorithm on the original program, where continuous parameters can be drawn using efficient gradient-based inference methods, while the discrete parameters are inferred using variable elimination.

Probabilistic Programming

Neural Program Synthesis with a Differentiable Fixer

no code implementations19 Jun 2020 Matej Balog, Rishabh Singh, Petros Maniatis, Charles Sutton

We present a new program synthesis approach that combines an encoder-decoder based synthesis architecture with a differentiable program fixer.

Program Synthesis

Global Relational Models of Source Code

1 code implementation ICLR 2020 Vincent J. Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, David Bieber

By studying a popular, non-trivial program repair task, variable-misuse identification, we explore the relative merits of traditional and hybrid model families for code representation.

Inductive Bias Variable misuse

SCELMo: Source Code Embeddings from Language Models

no code implementations28 Apr 2020 Rafael - Michael Karampatsis, Charles Sutton

Continuous embeddings of tokens in computer programs have been used to support a variety of software development tools, including readability, code search, and program repair.

Code Search Program Repair

OptTyper: Probabilistic Type Inference by Optimising Logical and Natural Constraints

1 code implementation1 Apr 2020 Irene Vlassi Pandi, Earl T. Barr, Andrew D. Gordon, Charles Sutton

OptTyper combines a continuous interpretation of logical constraints derived by classical static analysis of TypeScript code, with natural constraints obtained from a deep learning model, which learns naming conventions for types from a large codebase.

Type prediction Vocal Bursts Type Prediction

Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code

2 code implementations17 Mar 2020 Rafael-Michael Karampatsis, Hlib Babii, Romain Robbes, Charles Sutton, Andrea Janes

Statistical language modeling techniques have successfully been applied to large source code corpora, yielding a variety of new software development tools, such as tools for code suggestion, improving readability, and API migration.

Software Engineering

Towards Modular Algorithm Induction

no code implementations27 Feb 2020 Daniel A. Abolafia, Rishabh Singh, Manzil Zaheer, Charles Sutton

Main consists of a neural controller that interacts with a variable-length input tape and learns to compose modules together with their corresponding argument choices.

Reinforcement Learning (RL)

Incremental Sampling Without Replacement for Sequence Models

1 code implementation ICML 2020 Kensen Shi, David Bieber, Charles Sutton

Sampling is a fundamental technique, and sampling without replacement is often desirable when duplicate samples are not beneficial.

Combinatorial Optimization Program Synthesis

Learning to Represent Programs with Property Signatures

no code implementations ICLR 2020 Augustus Odena, Charles Sutton

We introduce the notion of property signatures, a representation for programs and program specifications meant for consumption by machine learning algorithms.

Vocal Bursts Type Prediction

Learning to Fix Build Errors with Graph2Diff Neural Networks

no code implementations4 Nov 2019 Daniel Tarlow, Subhodeep Moitra, Andrew Rice, Zimin Chen, Pierre-Antoine Manzagol, Charles Sutton, Edward Aftandilian

A diff specifies how to modify the code's abstract syntax tree, represented in the neural network as a sequence of tokens and of pointers to code locations.

Program Repair

Robust Variational Autoencoders for Outlier Detection and Repair of Mixed-Type Data

1 code implementation15 Jul 2019 Simão Eduardo, Alfredo Nazábal, Christopher K. I. Williams, Charles Sutton

We show experimentally that not only RVAE performs better than several state-of-the-art methods in cell outlier detection and repair for tabular data, but also that is robust against the initial hyper-parameter selection.

Imputation Outlier Detection

How Often Do Single-Statement Bugs Occur? The ManySStuBs4J Dataset

2 code implementations30 May 2019 Rafael-Michael Karampatsis, Charles Sutton

One way to achieve acceptable performance is to focus on classes of simple bugs, such as bugs with single statement fixes, or that match a small set of bug templates.

Software Engineering Programming Languages

Learning Semantic Annotations for Tabular Data

1 code implementation30 May 2019 Jiaoyan Chen, Ernesto Jimenez-Ruiz, Ian Horrocks, Charles Sutton

The usefulness of tabular data such as web tables critically depends on understanding their semantics.

Column Type Annotation Type prediction

Maybe Deep Neural Networks are the Best Choice for Modeling Source Code

1 code implementation13 Mar 2019 Rafael-Michael Karampatsis, Charles Sutton

We present a new open-vocabulary neural language model for code that is not limited to a fixed vocabulary of identifier names.

Language Modelling Machine Translation

ColNet: Embedding the Semantics of Web Tables for Column Type Prediction

1 code implementation4 Nov 2018 Jiaoyan Chen, Ernesto Jimenez-Ruiz, Ian Horrocks, Charles Sutton

Automatically annotating column types with knowledge base (KB) concepts is a critical task to gain a basic understanding of web tables.

 Ranked #1 on Column Type Annotation on T2Dv2 (F1 (%) metric)

Column Type Annotation Type prediction +1

Probabilistic Programming with Densities in SlicStan: Efficient, Flexible and Deterministic

1 code implementation2 Nov 2018 Maria I. Gorinova, Andrew D. Gordon, Charles Sutton

Stan is a probabilistic programming language that has been increasingly used for real-world scalable projects.

Probabilistic Programming

Deep Learning to Detect Redundant Method Comments

1 code implementation12 Jun 2018 Annie Louis, Santanu Kumar Dash, Earl T. Barr, Charles Sutton

To address this problem, we introduce the notion of comment entailment from code, high entailment indicating that a comment's natural language semantics can be inferred directly from the code.

Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts

no code implementations NAACL 2018 Annie Louis, Charles Sutton

An essential aspect to understanding narratives is to grasp the interaction between characters in a story and the actions they take.

Topic Models

Generative Ratio Matching Networks

no code implementations ICLR 2020 Akash Srivastava, Kai Xu, Michael U. Gutmann, Charles Sutton

In this work, we take their insight of using kernels as fixed adversaries further and present a novel method for training deep generative models that does not involve saddlepoint optimization.

Variational Inference In Pachinko Allocation Machines

no code implementations21 Apr 2018 Akash Srivastava, Charles Sutton

The Pachinko Allocation Machine (PAM) is a deep topic model that allows representing rich correlation structures among topics by a directed acyclic graph over topics.

Variational Inference

HOUDINI: Lifelong Learning as Program Synthesis

2 code implementations NeurIPS 2018 Lazar Valkov, Dipak Chaudhari, Akash Srivastava, Charles Sutton, Swarat Chaudhuri

We present a neurosymbolic framework for the lifelong learning of algorithmic tasks that mix perception and procedural reasoning.

Program Synthesis Transfer Learning

Interpreting Deep Classifier by Visual Distillation of Dark Knowledge

no code implementations11 Mar 2018 Kai Xu, Dae Hoon Park, Chang Yi, Charles Sutton

Interpreting black box classifiers, such as deep networks, allows an analyst to validate a classifier before it is deployed in a high-stakes setting.

Dimensionality Reduction Model Compression

GEMSEC: Graph Embedding with Self Clustering

3 code implementations ASONAM 2019 Benedek Rozemberczki, Ryan Davies, Rik Sarkar, Charles Sutton

In this paper we propose GEMSEC a graph embedding algorithm which learns a clustering of the nodes simultaneously with computing their features.

Social and Information Networks

A Survey of Machine Learning for Big Code and Naturalness

no code implementations18 Sep 2017 Miltiadis Allamanis, Earl T. Barr, Premkumar Devanbu, Charles Sutton

We contrast programming languages against natural languages and discuss how these similarities and differences drive the design of probabilistic models.

BIG-bench Machine Learning Navigate

VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning

1 code implementation NeurIPS 2017 Akash Srivastava, Lazar Valkov, Chris Russell, Michael U. Gutmann, Charles Sutton

Deep generative models provide powerful tools for distributions over complicated manifolds, such as those of natural images.

Autoencoding Variational Inference For Topic Models

6 code implementations4 Mar 2017 Akash Srivastava, Charles Sutton

A promising approach to address this problem is autoencoding variational Bayes (AEVB), but it has proven diffi- cult to apply to topic models in practice.

Topic Models Variational Inference

Sequence-to-point learning with neural networks for nonintrusive load monitoring

8 code implementations29 Dec 2016 Chaoyun Zhang, Mingjun Zhong, Zongzuo Wang, Nigel Goddard, Charles Sutton

Interestingly, we systematically show that the convolutional neural networks can inherently learn the signatures of the target appliances, which are automatically added into the model to reduce the identifiability problem.

blind source separation

Tailored Mutants Fit Bugs Better

no code implementations8 Nov 2016 Miltiadis Allamanis, Earl T. Barr, René Just, Charles Sutton

The results demonstrate that the location selection heuristics produce mutants more closely coupled to real faults for a given budget of mutation operator applications.

Software Engineering

Clustering with a Reject Option: Interactive Clustering as Bayesian Prior Elicitation

no code implementations19 Jun 2016 Akash Srivastava, James Zou, Ryan P. Adams, Charles Sutton

A good clustering can help a data analyst to explore and understand a data set, but what constitutes a good clustering may depend on domain-specific and application-specific criteria.

Clustering

Clustering with a Reject Option: Interactive Clustering as Bayesian Prior Elicitation

no code implementations22 Feb 2016 Akash Srivastava, James Zou, Charles Sutton

A good clustering can help a data analyst to explore and understand a data set, but what constitutes a good clustering may depend on domain-specific and application-specific criteria.

Clustering Computational Efficiency

A Subsequence Interleaving Model for Sequential Pattern Mining

1 code implementation16 Feb 2016 Jaroslav Fowkes, Charles Sutton

Recent sequential pattern mining methods have used the minimum description length (MDL) principle to define an encoding scheme which describes an algorithm for mining the most compressing patterns in a database.

Sequential Pattern Mining

A Convolutional Attention Network for Extreme Summarization of Source Code

5 code implementations9 Feb 2016 Miltiadis Allamanis, Hao Peng, Charles Sutton

Attention mechanisms in neural networks have proved useful for problems in which the input and output do not have fixed dimension.

Descriptive Extreme Summarization +1

Blending LSTMs into CNNs

no code implementations19 Nov 2015 Krzysztof J. Geras, Abdel-rahman Mohamed, Rich Caruana, Gregor Urban, Shengjie Wang, Ozlem Aslan, Matthai Philipose, Matthew Richardson, Charles Sutton

We consider whether deep convolutional networks (CNNs) can represent decision functions with similar accuracy as recurrent networks such as LSTMs.

Automatic Speech Recognition Automatic Speech Recognition (ASR) +3

Latent Bayesian melding for integrating individual and population models

1 code implementation NeurIPS 2015 Mingjun Zhong, Nigel Goddard, Charles Sutton

In many statistical problems, a more coarse-grained model may be suitable for population-level behaviour, whereas a more detailed model is appropriate for accurate modelling of individual behaviour.

blind source separation

A Bayesian Network Model for Interesting Itemsets

1 code implementation14 Oct 2015 Jaroslav Fowkes, Charles Sutton

Mining itemsets that are the most interesting under a statistical model of the underlying data is a commonly used and well-studied technique for exploratory data analysis, with the most recent interestingness models exhibiting state of the art performance.

Signal Aggregate Constraints in Additive Factorial HMMs, with Application to Energy Disaggregation

no code implementations NeurIPS 2014 Mingjun Zhong, Nigel Goddard, Charles Sutton

Blind source separation problems are difficult because they are inherently unidentifiable, yet the entire goal is to identify meaningful sources.

blind source separation

Semi-Separable Hamiltonian Monte Carlo for Inference in Bayesian Hierarchical Models

no code implementations NeurIPS 2014 Yichuan Zhang, Charles Sutton

Sampling from hierarchical Bayesian models is often difficult for MCMC methods, because of the strong correlations between the model parameters and the hyperparameters.

Scheduled denoising autoencoders

1 code implementation12 Jun 2014 Krzysztof J. Geras, Charles Sutton

We present a representation learning method that learns features at multiple different levels of scale.

Denoising Representation Learning

An Introduction to Conditional Random Fields

no code implementations17 Nov 2010 Charles Sutton, Andrew McCallum

This tutorial describes conditional random fields, a popular probabilistic method for structured prediction.

General Classification Structured Prediction

Cannot find the paper you are looking for? You can Submit a new open access paper.