Search Results for author:
Found 69 papers, 39 papers with code
NICE: Neural Image Commenting with Empathy
Based on these annotations, we define two different tasks for the NICE dataset.
Exploring Probabilistic Models for Semi-supervised Learning
This thesis studies advanced probabilistic models, including both their theoretical foundations and practical applications, for different semi-supervised learning (SSL) tasks.
Entity6K: A Large Open-Domain Evaluation Dataset for Real-World Entity Recognition
Open-domain real-world entity recognition is essential yet challenging, involving identifying various entities in diverse environments.
Bring Metric Functions into Diffusion Models
The first module, similar to a standard DDPM, learns to predict the added noise and is unaffected by the metric function.
COSMO: COntrastive Streamlined MultimOdal Model with Interleaved Pre-Training
\ModelName, our unified framework, merges unimodal and multimodal elements, enhancing model performance for tasks involving textual and visual data while notably reducing learnable parameters.
InfoVisDial: An Informative Visual Dialogue Dataset by Bridging Large Multimodal and Language Models
In this paper, we build a visual dialogue dataset, named InfoVisDial, which provides rich informative answers in each round even with external knowledge related to the visual content.
How Does It Function? Characterizing Long-term Trends in Production Serverless Workloads
The first trace is derived from Huawei's internal workloads and contains detailed per-second statistics for 200 functions running across multiple Huawei cloud data centers.
Interfacing Foundation Models' Embeddings
The proposed interface is adaptive to new tasks, and new models.
Segment and Caption Anything
We propose a method to efficiently equip the Segment Anything Model (SAM) with the ability to generate regional captions.
MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learning
We present MM-Narrator, a novel system leveraging GPT-4 with multimodal in-context learning for the generation of audio descriptions (AD).
GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation
We first benchmark MM-Navigator on our collected iOS screen dataset.
MM-VID: Advancing Video Understanding with GPT-4V(ision)
We present MM-VID, an integrated system that harnesses the capabilities of GPT-4V, combined with specialized tools in vision, audio, and speech, to facilitate advanced video understanding.
DEsignBench: Exploring and Benchmarking DALL-E 3 for Imagining Visual Design
For DEsignBench benchmarking, we perform human evaluations on generated images in DEsignBench gallery, against the criteria of image-text alignment, visual aesthetic, and design creativity.
Idea2Img: Iterative Self-Refinement with GPT-4V(ision) for Automatic Image Design and Generation
We introduce ``Idea to Image,'' a system that enables multimodal iterative self-refinement with GPT-4V(ision) for automatic image design and generation.
OpenLEAF: Open-Domain Interleaved Image-Text Generation and Evaluation
We hope our proposed framework, benchmark, and LMM evaluation could help establish the intriguing interleaved image-text generation task.
The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)
We hope that this preliminary exploration will inspire future research on the next-generation multimodal task formulation, new ways to exploit and enhance LMMs to solve real-world problems, and gaining better understanding of multimodal foundation models.
NP-SemiSeg: When Neural Processes meet Semi-Supervised Semantic Segmentation
This is useful in a wide range of real-world applications where collecting pixel-wise labels is not feasible in time or cost.
MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
Problems include: (1) How to systematically structure and evaluate the complicated multimodal tasks; (2) How to design evaluation metrics that work well across question and answer types; and (3) How to give model insights beyond a simple performance ranking.
Spatial-Frequency U-Net for Denoising Diffusion Probabilistic Models
In this paper, we study the denoising diffusion probabilistic model (DDPM) in wavelet space, instead of pixel space, for visual synthesis.
Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning
To efficiently measure the hallucination generated by LMMs, we propose GPT4-Assisted Visual Instruction Evaluation (GAVIE), a stable approach to evaluate visual instruction tuning like human experts.
Ranked #4 on
Visual Question Answering (VQA)
on HallusionBench
MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos
To address these challenges and provide a comprehensive dataset for this new direction, we have meticulously curated the \textbf{MMSum} dataset.
Segment Everything Everywhere All at Once
In SEEM, we propose a novel decoding mechanism that enables diverse prompting for all types of segmentation tasks, aiming at a universal segmentation interface that behaves like large language models (LLMs).
NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation
In this paper, we propose NUWA-XL, a novel Diffusion over Diffusion architecture for eXtremely Long video generation.
MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action
We propose MM-REACT, a system paradigm that integrates ChatGPT with a pool of vision experts to achieve multimodal reasoning and action.
Ranked #22 on
Visual Question Answering
on MM-Vet
Learning 3D Photography Videos via Self-supervised Diffusion on Single Images
3D photography renders a static image into a video with appealing 3D visual effects.
Ranked #1 on
Image Outpainting
on MSCOCO
NP-Match: Towards a New Probabilistic Model for Semi-Supervised Learning
In this work, we adjust neural processes (NPs) to the semi-supervised image classification task, resulting in a new method named NP-Match.
Generalized Decoding for Pixel, Image, and Language
We present X-Decoder, a generalized decoding model that can predict pixel-level segmentation and language tokens seamlessly.
Ranked #4 on
Instance Segmentation
on ADE20K val
(using extra training data)
GRiT: A Generative Region-to-text Transformer for Object Understanding
Specifically, GRiT consists of a visual encoder to extract image features, a foreground object extractor to localize objects, and a text decoder to generate open-set object descriptions.
Ranked #2 on
Dense Captioning
on Visual Genome
ReCo: Region-Controlled Text-to-Image Generation
Human evaluation on PaintSkill shows that ReCo is +19. 28% and +17. 21% more accurate in generating images with correct object count and spatial relationship than the T2I model.
Ranked #2 on
Conditional Text-to-Image Synthesis
on COCO-MIG
Exploring Discrete Diffusion Models for Image Captioning
The image captioning task is typically realized by an auto-regressive method that decodes the text tokens one by one.
Prompting GPT-3 To Be Reliable
While reliability is a broad and vaguely defined term, we decompose reliability into four main facets that correspond to the existing framework of ML safety and are well-recognized to be important: generalizability, social biases, calibration, and factuality.
NUWA-Infinity: Autoregressive over Autoregressive Generation for Infinite Visual Synthesis
In this paper, we present NUWA-Infinity, a generative model for infinite visual synthesis, which is defined as the task of generating arbitrarily-sized high-resolution images or long-duration videos.
Ranked #1 on
Image Outpainting
on LHQC
NP-Match: When Neural Processes meet Semi-Supervised Learning
Semi-supervised learning (SSL) has been widely explored in recent years, and it is an effective way of leveraging unlabeled data to reduce the reliance on labeled data.
Rethinking Bayesian Deep Learning Methods for Semi-Supervised Volumetric Medical Image Segmentation
Secondly, in fact, they are only partially based on Bayesian deep learning, as their overall architectures are not designed under the Bayesian framework.
Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone
Vision-language (VL) pre-training has recently received considerable attention.
Ranked #1 on
Phrase Grounding
on Flickr30k Entities Dev
Detection Hub: Unifying Object Detection Datasets via Query Adaptation on Language Embedding
Detection Hub further achieves SoTA performance on UODB benchmark with wide variety of datasets.
GIT: A Generative Image-to-text Transformer for Vision and Language
In this paper, we design and train a Generative Image-to-text Transformer, GIT, to unify vision-language tasks such as image/video captioning and question answering.
Ranked #1 on
Image Captioning
on nocaps-XD near-domain
The Overlooked Classifier in Human-Object Interaction Recognition
Human-Object Interaction (HOI) recognition is challenging due to two factors: (1) significant imbalance across classes and (2) requiring multiple labels per image.
The Overlooked Classifier in Human-Object Interaction Recognition
Human-Object Interaction (HOI) recognition is challenging due to two factors: (1) significant imbalance across classes and (2) requiring multiple labels per image.
Ranked #1 on
Human-Object Interaction Detection
on HICO
Injecting Semantic Concepts into End-to-End Image Captioning
In this paper, we are concerned with a better-performing detector-free image captioning model, and propose a pure vision transformer-based image captioning model, dubbed as ViTCAP, in which grid representations are used without extracting the regional features.
Scaling Up Vision-Language Pre-training for Image Captioning
In this paper, we present LEMON, a LargE-scale iMage captiONer, and provide the first empirical study on the scaling behavior of VLP for image captioning.
Ranked #3 on
Image Captioning
on nocaps-XD entire
(using extra training data)
UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling
On grounded captioning, UniTAB presents a simpler solution with a single output head, and significantly outperforms state of the art in both grounding and captioning evaluations.
Florence: A New Foundation Model for Computer Vision
Computer vision foundation models, which are trained on diverse, large-scale dataset and can be adapted to a wide range of downstream tasks, are critical for this mission to solve real-world computer vision applications.
Ranked #1 on
Action Recognition In Videos
on Kinetics-600
UFO: A UniFied TransfOrmer for Vision-Language Representation Learning
In this paper, we propose a single UniFied transfOrmer (UFO), which is capable of processing either unimodal inputs (e. g., image or language) or multimodal inputs (e. g., the concatenation of the image and the question), for vision-language (VL) representation learning.
An Empirical Study of Training End-to-End Vision-and-Language Transformers
Vision-and-language (VL) pre-training has proven to be highly effective on various VL downstream tasks.
Ranked #20 on
Cross-Modal Retrieval
on COCO 2014
(using extra training data)
Edge Prior Augmented Networks for Motion Deblurring on Naturally Blurry Images
Motion deblurring has witnessed rapid development in recent years, and most of the recent methods address it by using deep learning techniques, with the help of different kinds of prior knowledge.
An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA
To address this challenge, we propose PICa, a simple yet effective method that Prompts GPT3 via the use of Image Captions, for knowledge-based VQA.
Ranked #20 on
Visual Question Answering (VQA)
on OK-VQA
(using extra training data)
Is Object Detection Necessary for Human-Object Interaction Recognition?
This paper revisits human-object interaction (HOI) recognition at image level without using supervisions of object location and human pose.
RSG: A Simple but Effective Module for Learning Imbalanced Datasets
Imbalanced datasets widely exist in practice and area great challenge for training deep neural models with agood generalization on infrequent classes.
Ranked #17 on
Long-tail Learning
on Places-LT
End-to-End Semi-Supervised Object Detection with Soft Teacher
This paper presents an end-to-end semi-supervised object detection approach, in contrast to previous more complex multi-stage methods.
Ranked #6 on
Semi-Supervised Object Detection
on COCO 100% labeled data
(using extra training data)
Convolutional Neural Networks with Gated Recurrent Connections
The critical element of RCNN is the recurrent convolutional layer (RCL), which incorporates recurrent connections between neurons in the standard convolutional layer.
Compressing Visual-linguistic Model via Knowledge Distillation
In this paper, we study knowledge distillation (KD) to effectively compress a transformer-based large VL model into a small VL model.
DAP: Detection-Aware Pre-training with Weak Supervision
This paper presents a detection-aware pre-training (DAP) approach, which leverages only weakly-labeled classification-style datasets (e. g., ImageNet) for pre-training, but is specifically tailored to benefit object detection tasks.
Adversarial Feature Augmentation and Normalization for Visual Recognition
Recent advances in computer vision take advantage of adversarial data augmentation to ameliorate the generalization ability of classification models.
On graphs with exactly one anti-adjacency eigenvalue and beyond
This matrix can be interpreted as the opposite of the adjacency matrix, which is instead constructed from the distance matrix of a graph by keeping in each row and each column only the distances equal to 1.
Combinatorics 05C50
LLA: Loss-aware Label Assignment for Dense Pedestrian Detection
A joint loss is then defined as the weighted summation of cls and reg losses as the assigning indicator.
SEED: Self-supervised Distillation For Visual Representation
This paper is concerned with self-supervised learning for small models.
Orthogonal Subspace Decomposition: A New Perspective of Learning Discriminative Features for Face Clustering
Learning discriminative node features is the key to further improve the performance of graph-based face clustering.
The Hoffman program of graphs: old and new
The Hoffman program with respect to any real or complex square matrix $M$ associated to a graph $G$ stems from A. J. Hoffman's pioneering work on the limit points for the spectral radius of adjacency matrices of graphs less than $\sqrt{2+\sqrt{5}}$.
Combinatorics 05C50
MiniVLM: A Smaller and Faster Vision-Language Model
We design a Two-stage Efficient feature Extractor (TEE), inspired by the one-stage EfficientDet network, to significantly reduce the time cost of visual feature extraction by $95\%$, compared to a baseline model.
TAP: Text-Aware Pre-training for Text-VQA and Text-Caption
Due to this aligned representation learning, even pre-trained on the same downstream task dataset, TAP already boosts the absolute accuracy on the TextVQA dataset by +5. 4%, compared with a non-TAP baseline.
End-to-End Object Detection with Fully Convolutional Network
Mainstream object detectors based on the fully convolutional network has achieved impressive performance.
Hashing-based Non-Maximum Suppression for Crowded Object Detection
Considering the intersection-over-union (IoU) as the metric, we propose a simple yet effective hashing algorithm, named IoUHash, which guarantees that the boxes within the same cell are close enough by a lower IoU bound.
Learning to Count Objects with Few Exemplar Annotations
In this paper, we study the problem of object counting with incomplete annotations.
SFace: An Efficient Network for Face Detection in Large Scale Variations
A new dataset called 4K-Face is also introduced to evaluate the performance of face detection with extreme large scale variations.
Gated Recurrent Convolution Neural Network for OCR
Its critical component, Gated Recurrent Convolution Layer (GRCL), is constructed by adding a gate to the Recurrent Convolution Layer (RCL), the critical component of RCNN.
Face Attention Network: An Effective Face Detector for the Occluded Faces
The performance of face detection has been largely improved with the development of convolutional neural network.
Ranked #1 on
Occluded Face Detection
on MAFA
Group $K$-Means
We study how to learn multiple dictionaries from a dataset, and approximate any data point by the sum of the codewords each chosen from the corresponding dictionary.
Optimized Cartesian $K$-Means
In OCKM, multiple sub codewords are used to encode the subvector of a data point in a subspace.
