Search Results for author:
Found 44 papers, 23 papers with code
EgoCast: Forecasting Egocentric Human Pose in the Wild
Accurately estimating and forecasting human body pose is important for enhancing the user's sense of immersion in Augmented Reality.
Evaluating Numerical Reasoning in Text-to-Image Models
Text-to-image generative models are capable of producing high-quality images that often faithfully depict concepts described using natural language.
Greedy Growing Enables High-Resolution Pixel-Based Diffusion Models
Building on this core model, we propose a greedy algorithm that grows the architecture into high-resolution end-to-end models, while preserving the integrity of the pre-trained representation, stabilizing training, and reducing the need for large high-resolution datasets.
DOCCI: Descriptions of Connected and Contrasting Images
Vision-language datasets are vital for both text-to-image (T2I) and image-to-text (I2T) research.
Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and Human Ratings
Human-rated prompt sets are generally small and the reliability of the ratings -- and thereby the prompt set used to compare models -- is not evaluated.
Gemini: A Family of Highly Capable Multimodal Models
This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding.
Ranked #1 on
Visual Question Answering (VQA)
on InfographicVQA
Rich Human Feedback for Text-to-Image Generation
We show that the predicted rich human feedback can be leveraged to improve image generation, for example, by selecting high-quality training data to finetune and improve the generative models, or by creating masks with predicted heatmaps to inpaint the problematic regions.
Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation
With extensive experimentation and human evaluation on a range of model configurations (LLM, VQA, and T2I), we empirically demonstrate that DSG addresses the challenges noted above.
EgoCOL: Egocentric Camera pose estimation for Open-world 3D object Localization @Ego4D challenge 2023
We present EgoCOL, an egocentric camera pose estimation method for open-world 3D object localization.
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
In this work, we introduce Vid2Seq, a multi-modal single-stage dense event captioning model pretrained on narrated videos which are readily-available at scale.
Ranked #1 on
Dense Video Captioning
on ActivityNet Captions
(using extra training data)
Connecting Vision and Language with Video Localized Narratives
We propose Video Localized Narratives, a new form of multimodal video annotations connecting vision and language.
Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting
Through extensive human evaluation on EditBench, we find that object-masking during training leads to across-the-board improvements in text-image alignment -- such that Imagen Editor is preferred over DALL-E 2 and Stable Diffusion -- and, as a cohort, these models are better at object-rendering than text-rendering, and handle material/color/size attributes better than count/shape attributes.
Two-Level Temporal Relation Model for Online Video Instance Segmentation
In Video Instance Segmentation (VIS), current approaches either focus on the quality of the results, by taking the whole video as input and processing it offline; or on speed, by handling it frame by frame at the cost of competitive performance.
Video Swin Transformers for Egocentric Video Understanding @ Ego4D Challenges 2022
We implemented Video Swin Transformer as a base architecture for the tasks of Point-of-No-Return temporal localization and Object State Change Classification.
Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset
Research in massively multilingual image captioning has been severely hampered by a lack of high-quality evaluation datasets.
PanGEA: The Panoramic Graph Environment Annotation Toolkit
PanGEA, the Panoramic Graph Environment Annotation toolkit, is a lightweight toolkit for collecting speech and text annotations in photo-realistic 3D environments.
Telling the What while Pointing to the Where: Multimodal Queries for Image Retrieval
Most existing image retrieval systems use text queries as a way for the user to express what they are looking for.
Panoptic Narrative Grounding
This paper proposes Panoptic Narrative Grounding, a spatially fine and general formulation of the natural language visual grounding problem.
Connecting Vision and Language with Localized Narratives
We ask annotators to describe an image with their voice while simultaneously hovering their mouse over the region they are describing.
Ranked #2 on
Image Captioning
on Localized Narratives
Natural Vocabulary Emerges from Free-Form Annotations
This vocabulary represents the natural distribution of objects well and is learned directly from data, instead of being an educated guess done before collecting any labels.
The 2019 DAVIS Challenge on VOS: Unsupervised Multi-Object Segmentation
We present the 2019 DAVIS Challenge on Video Object Segmentation, the third edition of the DAVIS Challenge series, a public competition designed for the task of Video Object Segmentation (VOS).
The Liver Tumor Segmentation Benchmark (LiTS)
In this work, we report the set-up and results of the Liver Tumor Segmentation Benchmark (LiTS), which was organized in conjunction with the IEEE International Symposium on Biomedical Imaging (ISBI) 2017 and the International Conferences on Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2017 and 2018.
The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale
We present Open Images V4, a dataset of 9. 2M images with unified annotations for image classification, object detection and visual relationship detection.
Iterative Deep Learning for Road Topology Extraction
This paper tackles the task of estimating the topology of road networks from aerial images.
Blazingly Fast Video Object Segmentation with Pixel-Wise Metric Learning
This paper tackles the problem of video object segmentation, given some user annotation which indicates the object of interest.
Ranked #66 on
Semi-Supervised Video Object Segmentation
on DAVIS 2016
The 2018 DAVIS Challenge on Video Object Segmentation
Motivated by the analysis of the results of the 2017 edition, the main track of the competition will be the same than in the previous edition (segmentation given the full mask of the objects in the first frame -- semi-supervised scenario).
Iterative Deep Learning for Network Topology Extraction
This paper tackles the task of estimating the topology of filamentary networks such as retinal vessels and road networks.
Detection-aided liver lesion segmentation using deep learning
A fully automatic technique for segmenting the liver and localizing its unhealthy tissues is a convenient tool in order to diagnose hepatic diseases and assess the response to the according treatments.
Deep Extreme Cut: From Extreme Points to Object Segmentation
This paper explores the use of extreme points in an object (left-most, right-most, top, bottom pixels) as input to obtain precise object segmentation for images and videos.
Video Object Segmentation Without Temporal Information
Video Object Segmentation, and video processing in general, has been historically dominated by methods that rely on the temporal consistency and redundancy in consecutive video frames.
Ranked #47 on
Semi-Supervised Video Object Segmentation
on DAVIS 2016
Semantically-Guided Video Object Segmentation
This paper tackles the problem of semi-supervised video object segmentation, that is, segmenting an object in a sequence given its mask in the first frame.
The 2017 DAVIS Challenge on Video Object Segmentation
The DAVIS Challenge follows up on the recent publication of DAVIS (Densely-Annotated VIdeo Segmentation), which has fostered the development of several novel state-of-the-art video object segmentation techniques.
Convolutional Oriented Boundaries: From Image Segmentation to High-Level Tasks
We present Convolutional Oriented Boundaries (COB), which produces multiscale oriented contours and region hierarchies starting from generic image classification Convolutional Neural Networks (CNNs).
One-Shot Video Object Segmentation
This paper tackles the task of semi-supervised video object segmentation, i. e., the separation of an object from the background in a video, given the mask of the first frame.
Deep Retinal Image Understanding
This paper presents Deep Retinal Image Understanding (DRIU), a unified framework of retinal image analysis that provides both retinal vessel and optic disc segmentation.
Convolutional Oriented Boundaries
We present Convolutional Oriented Boundaries (COB), which produces multiscale oriented contours and region hierarchies starting from generic image classification Convolutional Neural Networks (CNNs).
A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation
The dataset, named DAVIS (Densely Annotated VIdeo Segmentation), consists of fifty high quality, Full HD video sequences, spanning multiple occurrences of common video object segmentation challenges such as occlusions, motion-blur and appearance changes.
Scale-Aware Alignment of Hierarchical Image Segmentation
To demonstrate the power of our method, we perform comprehensive experiments, which show that our method, as a post-processing step, can significantly improve the quality of the hierarchical segmentation representations, and ease the usage of hierarchical image segmentation to high-level vision tasks such as object segmentation.
Boosting Object Proposals: From Pascal to COCO
Computer vision in general, and object proposals in particular, are nowadays strongly influenced by the databases on which researchers evaluate the performance of their algorithms.
Oracle MCG: A first peek into COCO Detection Challenges
The recently presented COCO detection challenge will most probably be the reference benchmark in object detection in the next years.
Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation
We propose a unified approach for bottom-up hierarchical image segmentation and object proposal generation for recognition, called Multiscale Combinatorial Grouping (MCG).
Multiscale Combinatorial Grouping
We propose a unified approach for bottom-up hierarchical image segmentation and object candidate generation for recognition, called Multiscale Combinatorial Grouping (MCG).
Measures and Meta-Measures for the Supervised Evaluation of Image Segmentation
First, it surveys and structures the measures used to compare the segmentation results with a ground truth database; and proposes a new measure: the precision-recall for objects and parts.
