Search Results for author:
Found 32 papers, 25 papers with code
How Attentive are Graph Attention Networks?
Because GATs use a static attention mechanism, there are simple graph problems that GAT cannot express: in a controlled problem, we show that static attention hinders GAT from even fitting the training data.
Ranked #25 on
Node Property Prediction
on ogbn-arxiv
A Systematic Evaluation of Large Language Models of Code
We aim to fill in some of these blanks through a systematic evaluation of the largest existing models: Codex, GPT-J, GPT-Neo, GPT-NeoX-20B, and CodeParrot, across various programming languages.
CAT-LM: Training Language Models on Aligned Code And Tests
We also drastically increase the maximum sequence length of inputs to 8, 192 tokens, 4x more than typical code generation models, to ensure that the code context is available to the model when generating test code.
Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
Lingvo is a Tensorflow framework offering a complete solution for collaborative deep learning research, with a particular focus towards sequence-to-sequence models.
Self-Refine: Iterative Refinement with Self-Feedback
Motivated by how humans refine their written text, we introduce Self-Refine, an approach for improving initial outputs from LLMs through iterative feedback and refinement.
PAL: Program-aided Language Models
Much of this success can be attributed to prompting methods such as "chain-of-thought'', which employ LLMs for both understanding the problem description by decomposing it into steps, as well as solving each step of the problem.
Ranked #17 on
Arithmetic Reasoning
on GSM8K
code2vec: Learning Distributed Representations of Code
We demonstrate the effectiveness of our approach by using it to predict a method's name from the vector representation of its body.
Adversarial Examples for Models of Code
Our evaluations demonstrate that DAMP has up to 89% success rate in changing a prediction to the adversary's choice (a targeted attack) and a success rate of up to 94% in changing a given prediction to any incorrect prediction (a non-targeted attack).
Unlimiformer: Long-Range Transformers with Unlimited Length Input
This kNN index can be kept on either the GPU or CPU memory and queried in sub-linear time; this way, we can index practically unlimited input sequences, while every attention head in every decoder layer retrieves its top-k keys, instead of attending to every key.
code2seq: Generating Sequences from Structured Representations of Code
The ability to generate natural language sequences from source code snippets has a variety of applications such as code summarization, documentation, and retrieval.
WebArena: A Realistic Web Environment for Building Autonomous Agents
Building upon our environment, we release a set of benchmark tasks focusing on evaluating the functional correctness of task completions.
A General Path-Based Representation for Predicting Program Properties
A major challenge when learning from programs is $\textit{how to represent programs in a way that facilitates effective learning}$.
Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval
Retrieval-based language models (R-LM) model the probability of natural language text by combining a standard language model (LM) with examples retrieved from an external datastore at test time.
DocPrompting: Generating Code by Retrieving the Docs
Publicly available source-code libraries are continuously growing and changing.
Neural Reverse Engineering of Stripped Binaries using Augmented Control Flow Graphs
This is a challenging problem because of the low amount of syntactic information available in stripped executables, and the diverse assembly code patterns arising from compiler optimizations.
CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code
We release five language-specific pretrained models to use with our publicly available code.
On the Bottleneck of Graph Neural Networks and its Practical Implications
Since the proposal of the graph neural network (GNN) by Gori et al. (2005) and Scarselli et al. (2008), one of the major problems in training GNNs was their struggle to propagate information between distant nodes in the graph.
Structural Language Models of Code
We introduce a new approach to any-code completion that leverages the strict syntax of programming languages to model a code snippet as a tree - structural language modeling (SLM).
Language Models of Code are Few-Shot Commonsense Learners
In all these natural language tasks, we show that using our approach, a code generation LM (CODEX) outperforms natural-LMs that are fine-tuned on the target task (e. g., T5) and other strong LMs such as GPT-3 in the few-shot setting.
Learning Performance-Improving Code Edits
Next, we propose a broad range of adaptation strategies for code optimization; for prompting, these include retrieval-based few-shot prompting and chain-of-thought, and for finetuning, these include performance-conditioned generation and synthetic data augmentation based on self-play.
Why do Nearest Neighbor Language Models Work?
Language models (LMs) compute the probability of a text by sequentially computing a representation of an already-seen context and using this representation to predict the next word.
On the Expressivity Role of LayerNorm in Transformers' Attention
Layer Normalization (LayerNorm) is an inherent component in all Transformer-based models.
A Structural Model for Contextual Code Changes
We conduct a thorough evaluation, comparing our approach to a variety of representation and modeling approaches that are driven by multiple strong models such as LSTMs, Transformers, and neural CRFs.
Ranked #1 on
EditCompletion
on C# EditCompletion
Single-Node Attacks for Fooling Graph Neural Networks
Graph neural networks (GNNs) have shown broad applicability in a variety of domains.
Oversquashing in GNNs through the lens of information contraction and graph expansion
We compare the spectral expansion properties of our algorithm with that of an existing curvature-based non-local rewiring strategy.
Contextual Speech Recognition with Difficult Negative Training Examples
Improving the representation of contextual information is key to unlocking the potential of end-to-end (E2E) automatic speech recognition (ASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Structural Language Models for Any-Code Generation
We introduce a new approach to AnyGen that leverages the strict syntax of programming languages to model a code snippet as tree structural language modeling (SLM).
GPT-Calls: Enhancing Call Segmentation and Tagging by Generating Synthetic Conversations via Large Language Models
The online phase is applied to every call separately and scores the similarity between the transcripted conversation and the topic anchors found in the offline phase.
Universal Self-Consistency for Large Language Model Generation
Self-consistency with chain-of-thought prompting (CoT) has demonstrated remarkable performance gains on various challenging tasks, by utilizing multiple reasoning paths sampled from large language models (LLMs).
In-Context Principle Learning from Mistakes
We evaluate LEAP on a wide range of benchmarks, including multi-hop question answering (Hotpot QA), textual QA (DROP), Big-Bench Hard reasoning, and math problems (GSM8K and MATH); in all these benchmarks, LEAP improves the strongest available LLMs such as GPT-3. 5-turbo, GPT-4, GPT-4 turbo and Claude-2. 1.
Transformers Can Achieve Length Generalization But Not Robustly
We show that the success of length generalization is intricately linked to the data format and the type of position encoding.
Coarse-Graining and Self-Dissimilarity of Complex Networks
Can complex engineered and biological networks be coarse-grained into smaller and more understandable versions in which each node represents an entire pattern in the original network?
