Search Results for author:
Found 19 papers, 14 papers with code
On the Expressivity Role of LayerNorm in Transformers' Attention
Layer Normalization (LayerNorm) is an inherent component in all Transformer-based models.
Diffusing Graph Attention
The dominant paradigm for machine learning on graphs uses Message Passing Graph Neural Networks (MP-GNNs), in which node representations are updated by aggregating information in their local neighborhood.
Ranked #2 on
Link Prediction
on PCQM-Contact
Thinking Like Transformers
In this paper we aim to change that, proposing a computational model for the transformer-encoder in the form of a programming language.
How Attentive are Graph Attention Networks?
Because GATs use a static attention mechanism, there are simple graph problems that GAT cannot express: in a controlled problem, we show that static attention hinders GAT from even fitting the training data.
Ranked #2 on
Graph Regression
on Lipophilicity
On the Bottleneck of Graph Neural Networks and its Practical Implications
Since the proposal of the graph neural network (GNN) by Gori et al. (2005) and Scarselli et al. (2008), one of the major problems in training GNNs was their struggle to propagate information between distant nodes in the graph.
A Structural Model for Contextual Code Changes
We conduct a thorough evaluation, comparing our approach to a variety of representation and modeling approaches that are driven by multiple strong models such as LSTMs, Transformers, and neural CRFs.
Ranked #1 on
EditCompletion
on C# EditCompletion
A Formal Hierarchy of RNN Architectures
While formally extending these findings to unsaturated RNNs is left to future work, we hypothesize that the practical learnable capacity of unsaturated RNNs obeys a similar hierarchy.
Learning Deterministic Weighted Automata with Queries and Counterexamples
We present an algorithm for extraction of a probabilistic deterministic finite automaton (PDFA) from a given black-box language model, such as a recurrent neural network (RNN).
Adversarial Examples for Models of Code
Our evaluations demonstrate that DAMP has up to 89% success rate in changing a prediction to the adversary's choice (a targeted attack) and a success rate of up to 94% in changing a given prediction to any incorrect prediction (a non-targeted attack).
Structural Language Models of Code
We introduce a new approach to any-code completion that leverages the strict syntax of programming languages to model a code snippet as a tree - structural language modeling (SLM).
Structural Language Models for Any-Code Generation
We introduce a new approach to AnyGen that leverages the strict syntax of programming languages to model a code snippet as tree structural language modeling (SLM).
Towards Neural Decompilation
We address the problem of automatic decompilation, converting a program in low-level representation back to a higher-level human-readable programming language.
Neural Reverse Engineering of Stripped Binaries using Augmented Control Flow Graphs
This is a challenging problem because of the low amount of syntactic information available in stripped executables, and the diverse assembly code patterns arising from compiler optimizations.
code2seq: Generating Sequences from Structured Representations of Code
The ability to generate natural language sequences from source code snippets has a variety of applications such as code summarization, documentation, and retrieval.
On the Practical Computational Power of Finite Precision RNNs for Language Recognition
While Recurrent Neural Networks (RNNs) are famously known to be Turing complete, this relies on infinite precision in the states and unbounded computation time.
A General Path-Based Representation for Predicting Program Properties
A major challenge when learning from programs is $\textit{how to represent programs in a way that facilitates effective learning}$.
code2vec: Learning Distributed Representations of Code
We demonstrate the effectiveness of our approach by using it to predict a method's name from the vector representation of its body.
Extracting Automata from Recurrent Neural Networks Using Queries and Counterexamples
We do this using Angluin's L* algorithm as a learner and the trained RNN as an oracle.
Learning Disjunctions of Predicates
Our algorithm asks at most $|F| \cdot OPT(F_\vee)$ membership queries where $OPT(F_\vee)$ is the minimum worst case number of membership queries for learning $F_\vee$.
