Search Results for author:
Found 85 papers, 27 papers with code
Pre-training Mention Representations in Coreference Models
It is thus desirable to develop coreference resolution models that can make use of unlabeled data.
EgoPet: Egomotion and Interaction Data from an Animal's Perspective
Animals perceive the world to plan their actions and interact with other agents to accomplish complex tasks, demonstrating capabilities that are still unmatched by AI systems.
Finding Visual Task Vectors
In this work, we analyze the activations of MAE-VQGAN, a recent Visual Prompting model, and find task vectors, activations that encode task-specific information.
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
In this report, we present the latest model of the Gemini family, Gemini 1. 5 Pro, a highly compute-efficient multimodal mixture-of-experts model capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio.
Ranked #19 on
Code Generation
on HumanEval
Implicit Bias of Policy Gradient in Linear Quadratic Control: Extrapolation to Unseen Initial States
This paper theoretically studies the implicit bias of policy gradient in terms of extrapolation to unseen initial states.
Gemini: A Family of Highly Capable Multimodal Models
This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding.
Ranked #1 on
Multi-task Language Understanding
on MMLU
(using extra training data)
In-Context Learning Creates Task Vectors
In-context learning (ICL) in Large Language Models (LLMs) has emerged as a powerful new learning paradigm.
Graph Neural Networks Use Graphs When They Shouldn't
We analyze the implicit bias of gradient-descent learning of GNNs and prove that when the ground truth function does not use the graphs, GNNs are not guaranteed to learn a solution that ignores the graph, even with infinite data.
Stochastic positional embeddings improve masked image modeling
Masked Image Modeling (MIM) is a promising self-supervised learning approach that enables learning from unlabeled images.
Evaluating the Ripple Effects of Knowledge Editing in Language Models
This has led to the development of various editing methods that allow updating facts encoded by the model.
Covering Uncommon Ground: Gap-Focused Question Generation for Answer Assessment
Human communication often involves information gaps between the interlocutors.
LM vs LM: Detecting Factual Errors via Cross Examination
A prominent weakness of modern language models (LMs) is their tendency to generate factually incorrect text, which hinders their usability.
Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphs
For the visual side, we incorporate a special "SG Component" in the image transformer trained to predict SG information, while for the textual side, we utilize SGs to generate fine-grained captions that highlight different compositional aspects of the scene.
Ranked #24 on
Visual Reasoning
on Winoground
Dissecting Recall of Factual Associations in Auto-Regressive Language Models
Given a subject-relation query, we study how the model aggregates information about the subject and relation to predict the correct attribute.
Crawling the Internal Knowledge-Base of Language Models
Here, we propose to address this goal by extracting a knowledge-graph of facts from a given language model.
What Are You Token About? Dense Retrieval as Distributions Over the Vocabulary
We leverage this insight and propose a simple way to enrich query and passage representations with lexical information at inference time, and show that this significantly improves performance compared to the original model in zero-shot settings, and specifically on the BEIR benchmark.
PromptonomyViT: Multi-Task Prompt Learning Improves Video Transformers using Synthetic Scene Data
In this work, we propose an approach to leverage synthetic scene data for improving video understanding.
Text-Only Training for Image Captioning using Noise-Injected CLIP
We consider the task of image-captioning using only the CLIP model and additional text data at training time, and no additional captioned images.
Ranked #1 on
Image Captioning
on MSCOCO
Language Modelling
Semi Supervised Learning for Image Captioning
Learning Low Dimensional State Spaces with Overparameterized Recurrent Neural Nets
Overparameterization in deep learning typically refers to settings where a trained neural network (NN) has representational capacity to fit the training data in many ways, some of which generalize well, while others do not.
Visual Prompting via Image Inpainting
How does one adapt a pre-trained visual model to novel downstream tasks without task-specific finetuning or any model modification?
Ranked #5 on
Personalized Segmentation
on PerSeg
TREE-G: Decision Trees Contesting Graph Neural Networks
When dealing with tabular data, models based on decision trees are a popular choice due to their high accuracy on these data types, their ease of application, and explainability properties.
Ranked #1 on
Graph Classification
on HIV dataset
(Accuracy metric)
Structured Video Tokens @ Ego4D PNR Temporal Localization Challenge 2022
First, as both images and videos contain structured information, we enrich a transformer model with a set of \emph{object tokens} that can be used across images and videos.
Point- of-no-return (PNR) temporal localization
Temporal Localization
Bringing Image Scene Structure to Video via Frame-Clip Consistency of Object Tokens
We explore a particular instantiation of scene structure, namely a \emph{Hand-Object Graph}, consisting of hands and objects with their locations as nodes, and physical relations of contact/no-contact as edges.
Active Learning with Label Comparisons
We show that there is a natural class where this approach is sub-optimal, and that there is a more comparison-efficient active learning scheme.
On the Implicit Bias of Gradient Descent for Temporal Extrapolation
When using recurrent neural networks (RNNs) it is common practice to apply trained models to sequences longer than those seen in training.
Learning to Retrieve Passages without Supervision
Dense retrievers for open-domain question answering (ODQA) have been shown to achieve impressive performance by training on large datasets of question-passage pairs.
On the Optimization Landscape of Maximum Mean Discrepancy
Our analysis shows that that the MMD optimization landscape is benign in these cases, and therefore gradient based methods will globally minimize the MMD objective.
Object-Region Video Transformers
In this work, we present Object-Region Video Transformers (ORViT), an \emph{object-centric} approach that extends video transformer layers with a block that directly incorporates object representations.
Ranked #6 on
Action Recognition
on Diving-48
Provable Learning of Convolutional Neural Networks with Data Driven Features
Convolutional networks (CNN) are computationally hard to learn.
A Theoretical Analysis of Fine-tuning with Linear Teachers
We further present results on shallow ReLU models, and analyze the dependence of sample complexity there on source and target tasks.
DETReg: Unsupervised Pretraining with Region Priors for Object Detection
Recent self-supervised pretraining methods for object detection largely focus on pretraining the backbone of the object detector, neglecting key parts of detection architecture.
Ranked #1 on
Few-Shot Object Detection
on COCO 2017
Explaining in Style: Training a GAN to explain a classifier in StyleSpace
A natural source for such attributes is the StyleSpace of StyleGAN, which is known to generate semantically meaningful dimensions in the image.
BERTese: Learning to Speak to BERT
In this work, we propose a method for automatically rewriting queries into "BERTese", a paraphrase query that is directly optimized towards better knowledge extraction.
On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descent
Recent work has highlighted the role of initialization scale in determining the structure of the solutions that gradient methods converge to.
Towards Understanding Learning in Neural Networks with Linear Teachers
Can a neural network minimizing cross-entropy learn linearly separable data?
Few-Shot Question Answering by Pretraining Span Selection
Given a passage with multiple sets of recurring spans, we mask in each set all recurring spans but one, and ask the model to select the correct span in the passage for each masked span.
On the Inductive Bias of a CNN for Distributions with Orthogonal Patterns
In this work, we consider a simplified image classification task where images contain orthogonal patches and are learned with a 3-layer overparameterized convolutional network and stochastic gradient descent (SGD).
On Learning Read-once DNFs With Neural Networks
Here we focus on this setting where the functions are learned by a convex neural network and gradient descent.
Regularizing Towards Permutation Invariance in Recurrent Models
We show that RNNs can be regularized towards permutation invariance, and that this can result in compact models, as compared to non-recurrent architectures.
Maximin Optimization for Binary Regression
Interestingly, we prove that this approach is optimal in linear regression with low noise conditions as well as robust regression with a small number of outliers.
Learning Object Detection from Captions via Textual Scene Attributes
In this work, we argue that captions contain much richer information about the image, including attributes of objects and their relations.
Holdout SGD: Byzantine Tolerant Federated Learning
HoldOut SGD first randomly selects a set of workers that use their private data in order to propose gradient updates.
Compositional Video Synthesis with Action Graphs
Our generative model for this task (AG2Vid) disentangles motion and appearance features, and by incorporating a scheduling mechanism for actions facilitates a timely and coordinated video generation.
Optimal Strategies Against Generative Attacks
Are there optimal strategies for the attacker or the authenticator?
Learning Object Permanence from Video
The fourth subtask, where a target object is carried by a containing object, is particularly challenging because it requires a system to reason about a moving location of an invisible object.
Ranked #3 on
Video Object Tracking
on CATER
An Optimization and Generalization Analysis for Max-Pooling Networks
Max-Pooling operations are a core component of deep learning architectures.
Learning Canonical Representations for Scene Graph to Image Generation
Generating realistic images of complex visual scenes becomes challenging when one wishes to control the structure of the generated images.
Ranked #3 on
Layout-to-Image Generation
on Visual Genome 256x256
A Simple and Effective Model for Answering Multi-span Questions
Models for reading comprehension (RC) commonly restrict their output space to the set of all single contiguous spans from the input, in order to alleviate the learning problem and avoid the need for a model that generates text explicitly.
Ranked #7 on
Question Answering
on DROP Test
Learning RNNs with Commutative State Transitions
Many machine learning tasks involve analysis of set valued inputs, and thus the learned functions are expected to be permutation invariant.
Coreference Resolution with Entity Equalization
A key challenge in coreference resolution is to capture properties of entity clusters, and use those in the resolution process.
Ranked #11 on
Coreference Resolution
on CoNLL 2012
(using extra training data)
Label Propagation Networks
Graph networks have recently attracted considerable interest, and in particular in the context of semi-supervised learning.
Over-parameterization Improves Generalization in the XOR Detection Problem
Empirical evidence suggests that neural networks with ReLU activations generalize better with over-parameterization.
Differentiable Scene Graphs
In many domains, it is preferable to train systems jointly in an end-to-end manner, but SGs are not commonly used as intermediate components in visual reasoning systems because being discrete and sparse, scene-graph representations are non-differentiable and difficult to optimize.
Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing
We introduce a novel method for multilingual transfer that utilizes deep contextual embeddings, pretrained in an unsupervised fashion.
Cross-lingual zero-shot dependency parsing
Few-Shot Learning
+1
Spatio-Temporal Action Graph Networks
Events defined by the interaction of objects in a scene are often of critical importance; yet important events may have insufficient labeled examples to train a conventional deep model to generalize to future object appearance.
Why do Larger Models Generalize Better? A Theoretical Perspective via the XOR Problem
Empirical evidence suggests that neural networks with ReLU activations generalize better with over-parameterization.
Explaining Queries over Web Tables to Non-Experts
Designing a reliable natural language (NL) interface for querying tables has been a longtime goal of researchers in both the data management and natural language processing (NLP) communities.
Learning to Optimize Combinatorial Functions
Submodular functions have become a ubiquitous tool in machine learning.
Weakly Supervised Semantic Parsing with Abstract Examples
Training semantic parsers from weak supervision (denotations) rather than strong supervision (programs) complicates training in two ways.
Learning Rules-First Classifiers
Complex classifiers may exhibit "embarassing" failures in cases where humans can easily provide a justified classification.
Mapping Images to Scene Graphs with Permutation-Invariant Structured Prediction
Machine understanding of complex images is a key goal of artificial intelligence.
Predict and Constrain: Modeling Cardinality in Deep Structured Prediction
Here we focus on capturing cardinality constraints in such models.
Weakly-supervised Semantic Parsing with Abstract Examples
Training semantic parsers from weak supervision (denotations) rather than strong supervision (programs) complicates training in two ways.
SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Data
Neural networks exhibit good generalization behavior in the over-parameterized regime, where the number of network parameters exceeds the number of observations.
Learning Infinite Layer Networks without the Kernel Trick
Infinite Layer Networks (ILN) have been proposed as an architecture that mimics neural networks while enjoying some of the advantages of kernel methods.
Semi-Supervised Learning with Competitive Infection Models
The goal in semi-supervised learning is to effectively combine labeled and unlabeled data.
Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputs
Deep learning models are often successfully trained using gradient descent, despite the worst case hardness of the underlying non-convex optimization problem.
Learning to generalize to new compositions in image understanding
Recurrent neural networks have recently been used for learning to describe images using natural language.
Learning Infinite-Layer Networks: Without the Kernel Trick
Infinite--Layer Networks (ILN) have recently been proposed as an architecture that mimics neural networks while enjoying some of the advantages of kernel methods.
Tight Error Bounds for Structured Prediction
We show that the prospects for achieving low expected Hamming error depend on the structure of the graph $G$ in interesting ways.
Exploring Compositional Architectures and Word Vector Representations for Prepositional Phrase Attachment
In this paper, we show that word vector representations can yield significant PP attachment performance gains.
Tighter Linear Program Relaxations for High Order Graphical Models
In this work, we study the LP relaxations that result from enforcing additional consistency constraints between the HOP and the rest of the model.
Learning Max-Margin Tree Predictors
We demonstrate the effectiveness of our approach on several domains and show that, despite the relative simplicity of the structure, prediction accuracy is competitive with a fully connected model that is computationally costly at prediction time.
Convergence Rate Analysis of MAP Coordinate Minimization Algorithms
We also provide a simple dual to primal mapping that yields feasible primal solutions with a guaranteed rate of convergence.
More data means less inference: A pseudo-max approach to structured learning
The problem of learning to predict structured labels is of key importance in many applications.
Clusters and Coarse Partitions in LP Relaxations
We propose a new class of consistency constraints for Linear Programming (LP) relaxations for finding the most probable (MAP) configuration in graphical models.
Fixing Max-Product: Convergent Message Passing Algorithms for MAP LP-Relaxations
We present a novel message passing algorithm for approximating the MAP problem in graphical models.
