Search Results for author:
Found 86 papers, 33 papers with code
Functional Interpolation for Relative Positions Improves Long Context Transformers
Preventing the performance decay of Transformers on inputs longer than those used for training has been an important challenge in extending the context length of these models.
ExeDec: Execution Decomposition for Compositional Generalization in Neural Program Synthesis
When writing programs, people have the ability to tackle a new complex task by decomposing it into smaller and more familiar subtasks.
Machine Reading Comprehension using Case-based Reasoning
Given a test question, CBR-MRC first retrieves a set of similar cases from a nonparametric memory and then predicts an answer by selecting the span in the test context that is most similar to the contextualized representations of answers in the retrieved cases.
Can Public Large Language Models Help Private Cross-device Federated Learning?
We study (differentially) private federated learning (FL) of language models.
Efficient k-NN Search with Cross-Encoders using Adaptive Multi-Round CUR Decomposition
While ANNCUR's one-time selection of anchors tends to approximate the cross-encoder distances on average, doing so forfeits the capacity to accurately estimate distances to items near the query, leading to regret in the crucial end-task: recall of top-k items.
Improving Dual-Encoder Training through Dynamic Indexes for Negative Mining
First, we introduce an algorithm that uses a tree structure to approximate the softmax with provable bounds and that dynamically maintains the tree.
EmbedDistill: A Geometric Knowledge Distillation for Information Retrieval
Large neural models (such as Transformers) achieve state-of-the-art performance for information retrieval (IR).
Multi-Task Off-Policy Learning from Bandit Feedback
We prove per-task bounds on the suboptimality of the learned policies, which show a clear improvement over not using the hierarchical model.
Differentially Private Adaptive Optimization with Delayed Preconditioners
Privacy noise may negate the benefits of using adaptive optimizers in differentially private model training.
Large Language Models with Controllable Working Memory
By contrast, when the context is irrelevant to the task, the model should ignore it and fall back on its internal knowledge.
Learning to Navigate Wikipedia by Taking Random Walks
A fundamental ability of an intelligent web-based agent is seeking out and acquiring new information.
Efficient Nearest Neighbor Search for Cross-Encoder Models using Matrix Factorization
When the similarity is measured by dot-product between dual-encoder vectors or $\ell_2$-distance, there already exist many scalable and efficient search methods.
Longtonotes: OntoNotes with Longer Coreference Chains
Ontonotes has served as the most important benchmark for coreference resolution.
Generalization Properties of Retrieval-based Models
The second class of retrieval-based approaches we explore learns a global model using kernel methods to directly map an input instance and retrieved examples to a prediction, without explicitly solving a local learning task.
A Fourier Approach to Mixture Learning
Regev and Vijayaraghavan (2017) showed that with $\Delta = \Omega(\sqrt{\log k})$ separation, the means can be learned using $\mathrm{poly}(k, d)$ samples, whereas super-polynomially many samples are required if $\Delta = o(\sqrt{\log k})$ and $d = \Omega(\log k)$.
Teacher Guided Training: An Efficient Framework for Knowledge Transfer
In this paper, we propose the teacher-guided training (TGT) framework for training a high-quality compact model that leverages the knowledge acquired by pretrained generative models, while obviating the need to go through a large volume of data.
Questions Are All You Need to Train a Dense Passage Retriever
We introduce ART, a new corpus-level autoencoding approach for training dense retrieval models that does not require any labeled training data.
StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models
Knowledge and language understanding of models evaluated through question answering (QA) has been usually studied on static snapshots of knowledge, like Wikipedia.
Compositional Generalization and Decomposition in Neural Program Synthesis
We first characterize several different axes along which program synthesis methods would be desired to generalize, e. g., length generalization, or the ability to combine known subroutines in new ways that do not occur in the training data.
Knowledge Base Question Answering by Case-based Reasoning over Subgraphs
Question answering (QA) over knowledge bases (KBs) is challenging because of the diverse, essentially unbounded, types of reasoning patterns needed.
Private Adaptive Optimization with Side Information
Adaptive optimization methods have become the default solvers for many machine learning tasks.
Deep Hierarchy in Bandits
We use this exact posterior to analyze the Bayes regret of HierTS in Gaussian bandits.
Robust Training of Neural Networks Using Scale Invariant Architectures
In contrast to SGD, adaptive gradient methods like Adam allow robust training of modern deep networks, especially large language models.
A Context-Integrated Transformer-Based Neural Network for Auction Design
One of the central problems in auction design is developing an incentive-compatible mechanism that maximizes the auctioneer's expected revenue.
Hierarchical Bayesian Bandits
We provide a unified view of all these problems, as learning to act in a hierarchical Bayesian bandit.
When in Doubt, Summon the Titans: Efficient Inference with Large Models
In a nutshell, we use the large teacher models to guide the lightweight student models to only make correct predictions on a subset of "easy" examples; for the "hard" examples, we fall-back to the teacher.
When in Doubt, Summon the Titans: A Framework for Efficient Inference with Large Models
In a nutshell, we use the large teacher models to guide the lightweight student models to only make correct predictions on a subset of "easy" examples; for the "hard" examples, we fall-back to the teacher.
A Field Guide to Federated Optimization
Federated learning and analytics are a distributed approach for collaboratively learning models (or statistics) from decentralized data, motivated by and designed for privacy protection.
No Regrets for Learning the Prior in Bandits
We propose ${\tt AdaTS}$, a Thompson sampling algorithm that adapts sequentially to bandit tasks that it interacts with.
Thompson Sampling with a Mixture Prior
We study Thompson sampling (TS) in online decision making, where the uncertain environment is sampled from a mixture distribution.
Case-based Reasoning for Natural Language Queries over Knowledge Bases
It is often challenging to solve a complex problem from scratch, but much easier if we can access other similar problems with their solutions -- a paradigm known as case-based reasoning (CBR).
Knowledge Base Question Answering
Natural Language Queries
+1
Exact and Approximate Hierarchical Clustering Using A*
In those cases, hierarchical clustering can be seen as a combinatorial optimization problem.
Model-Agnostic Graph Regularization for Few-Shot Learning
In this paper, we present a comprehensive empirical study on graph embedded few-shot learning.
Non-Stationary Latent Bandits
The key idea is to frame this problem as a latent bandit, where the prototypical models of user behavior are learned offline and the latent state of the user is inferred online from its interactions with the models.
Modifying Memories in Transformer Models
In this paper, we propose a new task of \emph{explicitly modifying specific factual knowledge in Transformer models while ensuring the model performance does not degrade on the unmodified facts}.
Latent Programmer: Discrete Latent Codes for Program Synthesis
The latent codes are learned using a self-supervised learning principle, in which first a discrete autoencoder is trained on the output sequences, and then the resulting latent codes are used as intermediate targets for the end-to-end sequence prediction task.
Differentiable Meta-Learning of Bandit Policies
Exploration policies in Bayesian bandits maximize the average reward over problem instances drawn from some distribution P. In this work, we learn such policies for an unknown distribution P using samples from P. Our approach is a form of meta-learning and exploits properties of P without making strong assumptions about its form.
PLLay: Efficient Topological Layer based on Persistent Landscapes
We propose PLLay, a novel topological layer for general deep learning models based on persistence landscapes, in which we can efficiently exploit the underlying topological features of the input data structure.
Federated Composite Optimization
We first show that straightforward extensions of primal algorithms such as FedAvg are not well-suited for FCO since they suffer from the "curse of primal averaging," resulting in poor convergence.
Differentiable Open-Ended Commonsense Reasoning
As a step towards making commonsense reasoning research more realistic, we propose to study open-ended commonsense reasoning (OpenCSR) -- the task of answering a commonsense question without any pre-defined choices -- using as a resource only a corpus of commonsense facts written in natural language.
Scalable Hierarchical Agglomerative Clustering
The applicability of agglomerative clustering, for inferring both hierarchical and flat clustering, is limited by its scalability.
Probabilistic Case-based Reasoning for Open-World Knowledge Graph Completion
A case-based reasoning (CBR) system solves a new problem by retrieving `cases' that are similar to the given problem.
Ranked #1 on
Link Prediction
on NELL-995
Unsupervised Abstractive Dialogue Summarization for Tete-a-Tetes
High-quality dialogue-summary paired data is expensive to produce and domain-sensitive, making abstractive dialogue summarization a challenging task.
Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function
In this paper, we study bidirectional LSTM network for the task of text classification using both supervised and semi-supervised approaches.
Ranked #3 on
Text Classification
on AG News
Big Bird: Transformers for Longer Sequences
To remedy this, we propose, BigBird, a sparse attention mechanism that reduces this quadratic dependency to linear.
Ranked #1 on
Text Classification
on Hyperpartisan News Detection
A Simple Approach to Case-Based Reasoning in Knowledge Bases
We present a surprisingly simple yet accurate approach to reasoning in knowledge graphs (KGs) that requires \emph{no training}, and is reminiscent of case-based reasoning in classical artificial intelligence (AI).
Latent Bandits Revisited
A latent bandit problem is one in which the learning agent knows the arm reward distributions conditioned on an unknown discrete latent state.
Non-Stationary Off-Policy Optimization
This approach is practical and analyzable, and we provide guarantees on both the quality of off-policy optimization and the regret during online deployment.
Meta-Learning Bandit Policies by Gradient Ascent
Most bandit policies are designed to either minimize regret in any problem instance, making very few assumptions about the underlying environment, or in a Bayesian sense, assuming a prior distribution over environment parameters.
Robust Large-Margin Learning in Hyperbolic Space
In this paper, we present, to our knowledge, the first theoretical guarantees for learning a classifier in hyperbolic rather than Euclidean space.
Anchor & Transform: Learning Sparse Embeddings for Large Vocabularies
In this paper, we design a simple and efficient embedding algorithm that learns a small set of anchor embeddings and a sparse transformation matrix.
Adaptive Federated Optimization
Federated learning is a distributed machine learning paradigm in which a large number of clients coordinate with a central server to learn a model without sharing their own training data.
Towards Modular Algorithm Induction
Main consists of a neural controller that interacts with a variable-length input tape and learns to compose modules together with their corresponding argument choices.
Differentiable Reasoning over a Virtual Knowledge Base
In particular, we describe a neural module, DrKIT, that traverses textual data like a KB, softly following paths of relations between mentions of entities in the corpus.
Differentiable Bandit Exploration
In this work, we learn such policies for an unknown distribution $\mathcal{P}$ using samples from $\mathcal{P}$.
PLLay: Efficient Topological Layer based on Persistence Landscapes
We propose PLLay, a novel topological layer for general deep learning models based on persistence landscapes, in which we can efficiently exploit the underlying topological features of the input data structure.
FedDANE: A Federated Newton-Type Method
Federated learning aims to jointly learn statistical models over massively distributed remote devices.
Chains-of-Reasoning at TextGraphs 2019 Shared Task: Reasoning over Chains of Facts for Explainable Multi-hop Inference
This paper describes our submission to the shared task on {``}Multi-hop Inference Explanation Regeneration{''} in TextGraphs workshop at EMNLP 2019 (Jansen and Ustalov, 2019).
Anchor & Transform: Learning Sparse Representations of Discrete Objects
Learning continuous representations of discrete objects such as text, users, and items lies at the heart of many applications including text and user modeling.
Multi-step Entity-centric Information Retrieval for Multi-Hop Question Answering
Multi-hop question answering (QA) requires an information retrieval (IR) system that can find \emph{multiple} supporting evidence needed to answer the question, making the retrieval process very challenging.
Developing Creative AI to Generate Sculptural Objects
We explore the intersection of human and machine creativity by generating sculptural objects through machine learning.
The Myths of Our Time: Fake News
While the purpose of most fake news is misinformation and political propaganda, our team sees it as a new type of myth that is created by people in the age of internet identities and artificial intelligence.
Randomized Exploration in Generalized Linear Bandits
The first, GLM-TSL, samples a generalized linear model (GLM) from the Laplace approximation to the posterior distribution.
Multi-step Retriever-Reader Interaction for Scalable Open-domain Question Answering
This paper introduces a new framework for open-domain question answering in which the retriever and the reader iteratively interact with each other.
Exchangeable Generative Models with Flow Scans
In this work, we develop a new approach to generative density estimation for exchangeable, non-i. i. d.
Federated Optimization in Heterogeneous Networks
Theoretically, we provide convergence guarantees for our framework when learning over data from non-identical distributions (statistical heterogeneity), and while adhering to device-level systems constraints by allowing each participating device to perform a variable amount of work (systems heterogeneity).
Adaptive Methods for Nonconvex Optimization
In this work, we provide a new analysis of such methods applied to nonconvex stochastic optimization problems, characterizing the effect of increasing minibatch size.
Hallucinating Point Cloud into 3D Sculptural Object
Our approach extends DeepDream from images to 3D point clouds.
Point Cloud GAN
In this paper, we first show a straightforward extension of existing GAN algorithm is not applicable to point clouds, because the constraint required for discriminators is undefined for set data.
Towards Gradient Free and Projection Free Stochastic Optimization
This paper focuses on the problem of \emph{constrained} \emph{stochastic} optimization.
Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text
In this paper we look at a more practical setting, namely QA over the combination of a KB and entity-linked text, which is appropriate when an incomplete KB is available with a large text corpus.
Graph Representation Learning
Open-Domain Question Answering
Nonparametric Density Estimation under Adversarial Losses
We study minimax convergence rates of nonparametric density estimation under a large class of loss functions called "adversarial losses", which, besides classical $\mathcal{L}^p$ losses, includes maximum mean discrepancy (MMD), Wasserstein distance, and total variation distance.
Transformation Autoregressive Networks
Further, through a comprehensive study over both real world and synthetic data, we show for that jointly leveraging transformations of variables and autoregressive conditional models, results in a considerable improvement in performance.
Investigating the Working of Text Classifiers
Text classification is one of the most widely studied tasks in natural language processing.
Compressed Video Action Recognition
), we propose to train a deep network directly on the compressed video.
Ranked #46 on
Action Classification
on Charades
(using extra training data)
State Space LSTM Models with Particle MCMC Inference
Long Short-Term Memory (LSTM) is one of the most powerful sequence models.
Go for a Walk and Arrive at the Answer: Reasoning Over Paths in Knowledge Bases using Reinforcement Learning
Knowledge bases (KB), both automatically and manually constructed, are often incomplete --- many valid facts can be inferred from the KB by synthesizing existing information.
A Generic Approach for Escaping Saddle points
A central challenge to using first-order methods for optimizing nonconvex problems is the presence of saddle points.
Latent LSTM Allocation: Joint clustering and non-linear dynamic modeling of sequence data
Recurrent neural networks, such as long-short term memory (LSTM) networks, are powerful tools for modeling sequential data like user browsing history (Tan et al., 2016; Korpusik et al., 2016) or natural language text (Mikolov et al., 2010).
Canopy --- Fast Sampling with Cover Trees
In this work, we propose Canopy, a sampler based on Cover Trees that is exact, has guaranteed runtime logarithmic in the number of atoms, and is provably polynomial in the inherent dimensionality of the underlying parameter space.
Question Answering on Knowledge Bases and Text using Universal Schema and Memory Networks
Existing question answering methods infer answers either from a knowledge base or from raw text.
Spectral Methods for Nonparametric Models
Nonparametric models are versatile, albeit computationally expensive, tool for modeling mixture models.
Deep Sets
Our main theorem characterizes the permutation invariant functions and provides a family of functions to which any permutation invariant objective function must belong.
