Search Results for author:
Found 43 papers, 20 papers with code
Synthesize Policies for Transfer and Adaptation across Tasks and Environments
The ability to transfer in reinforcement learning is key towards building an agent of general artificial intelligence.
MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions
Image retrieval, i. e., finding desired images given a reference image, inherently encompasses rich, multi-faceted search intents that are difficult to capture solely using image-based measures.
Instruct-Imagen: Image Generation with Multi-modal Instruction
We introduce *multi-modal instruction* for image generation, a task representation articulating a range of generation intents with precision.
Gemini: A Family of Highly Capable Multimodal Models
This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding.
Ranked #1 on
Multi-task Language Understanding
on MMLU
(using extra training data)
UniIR: Training and Benchmarking Universal Multimodal Information Retrievers
Existing information retrieval (IR) models often assume a homogeneous format, limiting their applicability to diverse user needs, such as searching for images with text descriptions, searching for a news article with a headline image, or finding a similar photo with a query image.
From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces
Much of the previous work towards digital agents for graphical user interfaces (GUIs) has relied on text-based representations (derived from HTML or other structured data sources), which are not always readily available.
PaLI-X: On Scaling up a Multilingual Vision and Language Model
We present the training recipe and results of scaling up PaLI-X, a multilingual vision and language model, both in terms of size of the components and the breadth of its training task mixture.
Ranked #1 on
Fine-Grained Image Recognition
on OVEN
Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?
In this study, we introduce InfoSeek, a visual question answering dataset tailored for information-seeking questions that cannot be answered with only common sense knowledge.
Ranked #3 on
Visual Question Answering (VQA)
on InfoSeek
Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entities
Large-scale multi-modal pre-training models such as CLIP and PaLI exhibit strong generalization on various visual domains and tasks.
Ranked #2 on
Fine-Grained Image Recognition
on OVEN
Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding
Visually-situated language is ubiquitous -- sources range from textbooks with diagrams to web pages with images and tables, to mobile apps with buttons and forms.
Ranked #17 on
Visual Question Answering (VQA)
on InfographicVQA
MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text
While language Models store a massive amount of world knowledge implicitly in their parameters, even very large models often fail to encode information about rare entities and events, while incurring huge computational costs.
Re-Imagen: Retrieval-Augmented Text-to-Image Generator
To further evaluate the capabilities of the model, we introduce EntityDrawBench, a new benchmark that evaluates image generation for diverse entities, from frequent to rare, across multiple object categories including dogs, foods, landmarks, birds, and characters.
Ranked #3 on
Text-to-Image Generation
on MS COCO
PreSTU: Pre-Training for Scene-Text Understanding
The ability to recognize and reason about text embedded in visual inputs is often lacking in vision-and-language (V&L) models, perhaps because V&L pre-training methods have often failed to include such an ability in their training objective.
Visually Grounded Concept Composition
We investigate ways to compose complex concepts in texts from primitive ones while grounding them in images.
Systematic Generalization on gSCAN: What is Nearly Solved and What is Next?
We analyze the grounded SCAN (gSCAN) benchmark, which was recently proposed to study systematic generalization for grounded language understanding.
On Model Calibration for Long-Tailed Object Detection and Instance Segmentation
We propose NorCal, Normalized Calibration for long-tailed object detection and instance segmentation, a simple and straightforward recipe that reweighs the predicted scores of each class by its training sample size.
MosaicOS: A Simple and Effective Use of Object-Centric Images for Long-Tailed Object Detection
Many objects do not appear frequently enough in complex scenes (e. g., certain handbags in living rooms) for training an accurate object detector, but are often found frequently by themselves (e. g., in product images).
A Hierarchical Multi-Modal Encoder for Moment Localization in Video Corpus
Identifying a short segment in a long video that semantically matches a text query is a challenging task that has important application potentials in language-based video search, browsing, and navigation.
Learning the Best Pooling Strategy for Visual Semantic Embedding
Visual Semantic Embedding (VSE) is a dominant approach for vision-language retrieval, which aims at learning a deep embedding space such that visual data are embedded close to their semantic text labels or descriptions.
Learning to Represent Image and Text with Denotation Graph
Recent progresses have leveraged the ideas of pre-training (from language modeling) and attention layers in Transformers to learn representation from datasets containing images aligned with linguistic expressions that describe the images.
Drinking from a Firehose: Continual Learning with Web-scale Natural Language
Collectively, the POLL problem setting, the Firehose datasets, and the ConGraD algorithm enable a complete benchmark for reproducible research on web-scale continual learning.
BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Steps
To this end, we propose BabyWalk, a new VLN agent that is learned to navigate by decomposing long instructions into shorter ones (BabySteps) and completing them sequentially.
Visual Storytelling via Predicting Anchor Word Embeddings in the Stories
To narrate a sequence of images, we use the predicted anchor word embeddings and the image features as the joint input to a seq2seq model.
Ranked #15 on
Visual Storytelling
on VIST
Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulation
Model-agnostic meta-learners aim to acquire meta-learned parameters from similar tasks to adapt to novel tasks from the same distribution with few gradient updates.
Learning Adaptive Classifiers Synthesis for Generalized Few-Shot Learning
In this paper, we investigate the problem of generalized few-shot learning (GFSL) -- a model during the deployment is required to learn about tail categories with few shots and simultaneously classify the head classes.
Synthesized Policies for Transfer and Adaptation across Tasks and Environments
The ability to transfer in reinforcement learning is key towards building an agent of general artificial intelligence.
Evaluating Text-to-Image Matching using Binary Image Selection (BISON)
Providing systems the ability to relate linguistic and visual content is one of the hallmarks of computer vision.
Toward Multimodal Model-Agnostic Meta-Learning
One important limitation of such frameworks is that they seek a common initialization shared across the entire task distribution, substantially limiting the diversity of the task distributions that they are able to learn from.
Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions
Many few-shot learning methods address this challenge by learning an instance embedding function from seen classes and apply the function to instances from unseen classes with limited labels.
Engaging Image Captioning Via Personality
While such tasks are useful to verify that a machine understands the content of an image, they are not engaging to humans as captions.
Cross-Modal and Hierarchical Modeling of Video and Text
Similarly, a paragraph may contain sentences with different topics, which collectively conveys a coherent message or story.
Model-Agnostic Meta-Learning for Multimodal Task Distributions
In this paper, we augment MAML with the capability to identify tasks sampled from a multimodal task distribution and adapt quickly through gradient updates.
Multi-Task Learning for Sequence Tagging: An Empirical Study
We study three general multi-task learning (MTL) approaches on 11 sequence tagging tasks.
Cross-Dataset Adaptation for Visual Question Answering
Analogous to domain adaptation for visual recognition, this setting is appealing when the target dataset does not have a sufficient amount of labeled data to learn an "in-domain" model.
Learning Answer Embeddings for Visual Question Answering
These properties make the approach particularly appealing for transfer learning for open-ended Visual QA, where the source dataset on which the model is learned has limited overlapping with the target dataset in the space of answers.
Structured Label Inference for Visual Understanding
In this paper, we exploit this rich structure for performing graph-based inference in label space for a number of tasks: multi-label image and video classification and action detection in untrimmed videos.
Compressed Video Action Recognition
), we propose to train a deep network directly on the compressed video.
Ranked #46 on
Action Classification
on Charades
(using extra training data)
Being Negative but Constructively: Lessons Learnt from Creating Better Visual Question Answering Datasets
We apply the procedures to re-construct decoy answers for two popular Visual QA datasets as well as to create a new Visual QA dataset from the Visual Genome project, resulting in the largest dataset for this task.
LabelBank: Revisiting Global Perspectives for Semantic Segmentation
We advocate that holistic inference of image concepts provides valuable information for detailed pixel labeling.
FastMask: Segment Multi-scale Object Candidates in One Shot
Objects appear to scale differently in natural images.
Recalling Holistic Information for Semantic Segmentation
We advocate that high-recall holistic inference of image concepts provides valuable information for detailed pixel labeling.
Learning Structured Inference Neural Networks with Label Relations
Images of scenes have various objects as well as abundant attributes, and diverse levels of visual categorization are possible.
Structure Inference Machines: Recurrent Neural Networks for Analyzing Relations in Group Activity Recognition
As a concrete example, group activity recognition involves the interactions and relative spatial relations of a set of people in a scene.
Ranked #5 on
Group Activity Recognition
on Collective Activity
