Search Results for author:
Found 33 papers, 21 papers with code
MultiSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos
To address these challenges and provide a comprehensive dataset for this new direction, we have meticulously curated the MultiSum dataset.
Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation
In this paper, we propose LayoutBench, a diagnostic benchmark for layout-guided image generation that examines four categories of spatial control skills: number, position, size, and shape.
Ranked #1 on
Layout-to-Image Generation
on LayoutBench
Equivariant Similarity for Vision-Language Foundation Models
Unlike the existing image-text similarity objective which only categorizes matched pairs as similar and unmatched pairs as dissimilar, equivariance also requires similarity to vary faithfully according to the semantic changes.
NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation
In this paper, we propose NUWA-XL, a novel Diffusion over Diffusion architecture for eXtremely Long video generation.
MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action
We propose MM-REACT, a system paradigm that integrates ChatGPT with a pool of vision experts to achieve multimodal reasoning and action.
Revisiting Transformer for Point Cloud-based 3D Scene Graph Generation
In this paper, we propose the semantic graph Transformer (SGT) for 3D scene graph generation.
Learning 3D Photography Videos via Self-supervised Diffusion on Single Images
3D photography renders a static image into a video with appealing 3D visual effects.
Ranked #1 on
Image Outpainting
on MSCOCO
GRiT: A Generative Region-to-text Transformer for Object Understanding
Specifically, GRiT consists of a visual encoder to extract image features, a foreground object extractor to localize objects, and a text decoder to generate open-set object descriptions.
Ranked #1 on
Dense Captioning
on Visual Genome
ReCo: Region-Controlled Text-to-Image Generation
Human evaluation on PaintSkill shows that ReCo is +19. 28% and +17. 21% more accurate in generating images with correct object count and spatial relationship than the T2I model.
PromptCap: Prompt-Guided Task-Aware Image Captioning
PromptCap outperforms generic captions by a large margin and achieves state-of-the-art accuracy on knowledge-based VQA tasks (60. 4% on OK-VQA and 59. 6% on A-OKVQA).
Ranked #2 on
Visual Question Answering (VQA)
on A-OKVQA
Prompting GPT-3 To Be Reliable
While reliability is a broad and vaguely defined term, we decompose reliability into four main facets that correspond to the existing framework of ML safety and are well-recognized to be important: generalizability, social biases, calibration, and factuality.
TransVG++: End-to-End Visual Grounding with Language Conditioned Vision Transformer
For another, we devise Language Conditioned Vision Transformer that removes external fusion modules and reuses the uni-modal ViT for vision-language fusion at the intermediate layers.
GIT: A Generative Image-to-text Transformer for Vision and Language
In this paper, we design and train a Generative Image-to-text Transformer, GIT, to unify vision-language tasks such as image/video captioning and question answering.
Ranked #1 on
Image Captioning
on nocaps-XD out-of-domain
Cross-modal Contrastive Distillation for Instructional Activity Anticipation
In this study, we aim to predict the plausible future action steps given an observation of the past and study the task of instructional activity anticipation.
Scaling Up Vision-Language Pre-training for Image Captioning
In this paper, we present LEMON, a LargE-scale iMage captiONer, and provide the first empirical study on the scaling behavior of VLP for image captioning.
Ranked #3 on
Image Captioning
on nocaps-XD entire
(using extra training data)
UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling
On grounded captioning, UniTAB presents a simpler solution with a single output head, and significantly outperforms state of the art in both grounding and captioning evaluations.
UFO: A UniFied TransfOrmer for Vision-Language Representation Learning
In this paper, we propose a single UniFied transfOrmer (UFO), which is capable of processing either unimodal inputs (e. g., image or language) or multimodal inputs (e. g., the concatenation of the image and the question), for vision-language (VL) representation learning.
An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA
To address this challenge, we propose PICa, a simple yet effective method that Prompts GPT3 via the use of Image Captions, for knowledge-based VQA.
Ranked #10 on
Visual Question Answering (VQA)
on OK-VQA
SAT: 2D Semantics Assisted Training for 3D Visual Grounding
3D visual grounding aims at grounding a natural language description about a 3D scene, usually represented in the form of 3D point clouds, to the targeted object region.
TransVG: End-to-End Visual Grounding with Transformers
In this paper, we present a neat yet effective transformer-based framework for visual grounding, namely TransVG, to address the task of grounding a language query to the corresponding region onto an image.
Ranked #13 on
Referring Expression Comprehension
on RefCOCO
TAP: Text-Aware Pre-training for Text-VQA and Text-Caption
Due to this aligned representation learning, even pre-trained on the same downstream task dataset, TAP already boosts the absolute accuracy on the TextVQA dataset by +5. 4%, compared with a non-TAP baseline.
Pose-based Body Language Recognition for Emotion and Psychiatric Symptom Interpretation
We then evaluate the framework on a proposed URMC dataset, which consists of conversations between a standardized patient and a behavioral health professional, along with expert annotations of body language, emotions, and potential psychiatric symptoms.
Dynamic Context-guided Capsule Network for Multimodal Machine Translation
Particularly, we represent the input image with global and regional visual features, we introduce two parallel DCCNs to model multimodal context vectors with visual features at different granularities.
Ranked #3 on
Multimodal Machine Translation
on Multi30K
Improving One-stage Visual Grounding by Recursive Sub-query Construction
We improve one-stage visual grounding by addressing current limitations on grounding long and complex queries.
A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation
Multi-modal neural machine translation (NMT) aims to translate source sentences into a target language paired with images.
Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation
Our core innovation is the learning of a region-phrase score function, based on which an image-sentence score function is further constructed.
Grounding-Tracking-Integration
In this paper, we study Tracking by Language that localizes the target box sequence in a video based on a language query.
A Fast and Accurate One-Stage Approach to Visual Grounding
We propose a simple, fast, and accurate one-stage approach to visual grounding, inspired by the following insight.
Weakly Supervised Body Part Segmentation with Pose based Part Priors
The core idea is first converting the sparse weak labels such as keypoints to the initial estimate of body part masks, and then iteratively refine the part mask predictions.
Human-Centered Emotion Recognition in Animated GIFs
The framework consists of a facial attention module and a hierarchical segment temporal module.
Attentive Relational Networks for Mapping Images to Scene Graphs
Scene graph generation refers to the task of automatically mapping an image into a semantic structural graph, which requires correctly labeling each extracted object and their interaction relationships.
Action Recognition with Spatio-Temporal Visual Attention on Skeleton Image Sequences
The attention mechanism is important for skeleton based action recognition because there exist spatio-temporal key stages while the joint predictions can be inaccurate.
End-to-end Multi-Modal Multi-Task Vehicle Control for Self-Driving Cars with Visual Perception
In this work, we propose a multi-task learning framework to predict the steering angle and speed control simultaneously in an end-to-end manner.
