Search Results for author:
Found 84 papers, 48 papers with code
Stay-Positive: A Case for Ignoring Real Image Features in Fake Image Detection
Additionally, unlike detectors that associate artifacts with real images, those that focus purely on fake artifacts are better at detecting inpainted real images.
LASER: Lip Landmark Assisted Speaker Detection for Robustness
Active Speaker Detection (ASD) aims to identify speaking individuals in complex visual scenes.
Active Speaker Detection
Audio-Visual Active Speaker Detection
Building a Mind Palace: Structuring Environment-Grounded Semantic Graphs for Effective Long Video Analysis with LLMs
Long-form video understanding with Large Vision Language Models is challenged by the need to analyze temporally dispersed yet spatially concentrated key moments within limited context windows.
On the Effectiveness of Dataset Alignment for Fake Image Detection
In this work, we argue that in addition to these algorithmic choices, we also require a well aligned dataset of real/fake images to train a robust detector.
TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models
TemporalBench consists of ~10K video question-answer pairs, derived from ~2K high-quality human annotations detailing the temporal dynamics in video clips.
Vinoground: Scrutinizing LMMs over Dense Temporal Reasoning with Short Videos
There has been growing sentiment recently that modern large multimodal models (LMMs) have addressed most of the key challenges related to short video comprehension.
Removing Distributional Discrepancies in Captions Improves Image-Text Alignment
In this paper, we introduce a model designed to improve the prediction of image-text alignment, targeting the challenge of compositional understanding in current visual-language models.
Interpolating Video-LLMs: Toward Longer-sequence LMMs in a Training-free Manner
In this paper, we first identify the primary challenges in interpolating Video-LLMs: (1) the video encoder and modality alignment projector are fixed, preventing the integration of additional frames into Video-LLMs, and (2) the LLM backbone is limited in its content length capabilities, which complicates the processing of an increased number of video tokens.
Cross-Modal Self-Supervised Learning with Effective Contrastive Units for LiDAR Point Clouds
3D perception in LiDAR point clouds is crucial for a self-driving vehicle to properly act in 3D environment.
CHARTOM: A Visual Theory-of-Mind Benchmark for Multimodal Large Language Models
We benchmark leading LLMs as of late 2024 - including GPT, Claude, Gemini, Qwen, Llama, and Llava - on the CHARTOM dataset and found that our benchmark was challenging to all of them, suggesting room for future large language models to improve.
VGBench: Evaluating Large Language Models on Vector Graphics Understanding and Generation
In the realm of vision models, the primary mode of representation is using pixels to rasterize the visual world.
LLaRA: Supercharging Robot Learning Data for Vision-Language Policy
In this work, we introduce LLaRA: Large Language and Robotics Assistant, a framework that formulates robot action policy as visuo-textual conversations and enables an efficient transfer of a pretrained VLM into a powerful VLA, motivated by the success of visual instruction tuning in Computer Vision.
MATE: Meet At The Embedding -- Connecting Images with Long Texts
While advancements in Vision Language Models (VLMs) have significantly improved the alignment of visual and textual data, these models primarily focus on aligning images with short descriptive captions.
Yo'LLaVA: Your Personalized Language and Vision Assistant
In this paper, we introduce the novel task of personalizing LMMs, so that they can have conversations about a specific subject.
Matryoshka Multimodal Models
Large Multimodal Models (LMMs) such as LLaVA have shown strong performance in visual-linguistic reasoning.
LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models
In response, we propose PruMerge, a novel adaptive visual token reduction strategy that significantly reduces the number of visual tokens without compromising the performance of LMMs.
LLM Inference Unveiled: Survey and Roofline Model Insights
Our survey stands out from traditional literature reviews by not only summarizing the current state of research but also by introducing a framework based on roofline model for systematic analysis of LLM inference techniques.
Cohere3D: Exploiting Temporal Coherence for Unsupervised Representation Learning of Vision-based Autonomous Driving
Observations from different angles enable the recovery of 3D object states from 2D image inputs if we can identify the same instance in different input frames.
CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examples
We first spotlight the near-chance performance of multimodal models like CLIP and LLaVA in physically grounded compositional reasoning.
Edit One for All: Interactive Batch Image Editing
With increased human control, it is now possible to edit an image in a plethora of ways; from specifying in text what we want to change, to straight up dragging the contents of the image in an interactive point-based manner.
Interfacing Foundation Models' Embeddings
To further unleash the power of foundation models, we present FIND, a generalized interface for aligning foundation models' embeddings with unified image and dataset-level understanding spanning modality and granularity.
Diversify, Don't Fine-Tune: Scaling Up Visual Recognition Training with Synthetic Images
We present a new framework leveraging off-the-shelf generative models to generate synthetic training images, addressing multiple challenges: class name ambiguity, lack of diversity in naive prompts, and domain shifts.
ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts
Furthermore, we present ViP-Bench, a comprehensive benchmark to assess the capability of models in understanding visual prompts across multiple dimensions, enabling future research in this domain.
Testing learning-enabled cyber-physical systems with Large-Language Models: A Formal Approach
The integration of machine learning (ML) into cyber-physical systems (CPS) offers significant benefits, including enhanced efficiency, predictive capabilities, real-time responsiveness, and the enabling of autonomous operations.
Improved Baselines with Visual Instruction Tuning
Large multimodal models (LMM) have recently shown encouraging progress with visual instruction tuning.
Factual Inconsistency Detection in Chart Captioning
Image Classification
+4
A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidance
Domain generalization studies the problem of training a model with samples from several domains (or distributions) and then testing the model with samples from a new, unseen domain.
Ranked #21 on
Domain Generalization
on PACS
Investigating the Catastrophic Forgetting in Multimodal Large Language Models
However, catastrophic forgetting, a notorious phenomenon where the fine-tuned model fails to retain similar performance compared to the pre-trained model, still remains an inherent problem in multimodal LLMs (MLLM).
Visual Instruction Inversion: Image Editing via Visual Prompting
Given pairs of example that represent the "before" and "after" images of an edit, our goal is to learn a text-based editing direction that can be used to perform the same edit on new images.
Benchmarking and Analyzing Generative Data for Visual Recognition
Advancements in large pre-trained generative models have expanded their potential as effective data generators in visual recognition.
Generate Anything Anywhere in Any Scene
Text-to-image diffusion models have attracted considerable interest due to their wide applicability across diverse fields.
Leveraging Large Language Models for Scalable Vector Graphics-Driven Image Understanding
To study what the LLM can do with this XML-based textual description of images, we test the LLM on three broad computer vision tasks: (i) visual reasoning and question answering, (ii) image classification under distribution shift, few-shot learning, and (iii) generating new images using visual prompting.
Visual Instruction Tuning
Instruction tuning large language models (LLMs) using machine-generated instruction-following data has improved zero-shot capabilities on new tasks, but the idea is less explored in the multimodal field.
Ranked #4 on
Spatial Reasoning
on EmbSpatial-Bench
InPL: Pseudo-labeling the Inliers First for Imbalanced Semi-supervised Learning
Without relying on model confidence, we propose to measure whether an unlabeled sample is likely to be ``in-distribution''; i. e., close to the current training data.
Towards Universal Fake Image Detectors that Generalize Across Generative Models
In this work, we first show that the existing paradigm, which consists of training a deep network for real-vs-fake classification, fails to detect fake images from newer breeds of generative models when trained to detect GAN fake images.
Learning Customized Visual Models with Retrieval-Augmented Knowledge
Image-text contrastive learning models such as CLIP have demonstrated strong task transfer ability.
Ranked #1 on
Semi-Supervised Image Classification
on ImageNet - 1% labeled data
(using extra training data)
GLIGEN: Open-Set Grounded Text-to-Image Generation
Large-scale text-to-image diffusion models have made amazing advances.
Generalized Decoding for Pixel, Image, and Language
We present X-Decoder, a generalized decoding model that can predict pixel-level segmentation and language tokens seamlessly.
Ranked #4 on
Instance Segmentation
on ADE20K val
(using extra training data)
Expeditious Saliency-guided Mix-up through Random Gradient Thresholding
Mix-up training approaches have proven to be effective in improving the generalization ability of Deep Neural Networks.
Ranked #1 on
Classifier calibration
on CIFAR-100
Contrastive Learning for Diverse Disentangled Foreground Generation
We introduce a new method for diverse foreground generation with explicit control over various factors.
EnergyMatch: Energy-based Pseudo-Labeling for Semi-Supervised Learning
However, it has been shown that softmax-based confidence scores in deep networks can be arbitrarily high for samples far from the training data, and thus, the pseudo-labels for even high-confidence unlabeled samples may still be unreliable.
ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models
In general, these language-augmented visual models demonstrate strong transferability to a variety of datasets and tasks.
Ranked #1 on
Object Detection
on ELEVATER
The Two Dimensions of Worst-case Training and the Integrated Effect for Out-of-domain Generalization
Training with an emphasis on "hard-to-learn" components of the data has been proven as an effective method to improve the generalization of machine learning models, especially in the settings where robustness (e. g., generalization across distributions) is valued.
End-to-End Instance Edge Detection
We demonstrate highly competitive instance edge detection performance compared to state-of-the-art baselines, and also show that the proposed task and loss are complementary to instance segmentation and object detection.
GIRAFFE HD: A High-Resolution 3D-aware Generative Model
3D-aware generative models have shown that the introduction of 3D information can lead to more controllable image generation.
Masked Discrimination for Self-Supervised Learning on Point Clouds
Masked autoencoding has achieved great success for self-supervised learning in the image and language domains.
Ranked #16 on
Few-Shot 3D Point Cloud Classification
on ModelNet40 5-way (10-shot)
(using extra training data)
The Two Dimensions of Worst-Case Training and Their Integrated Effect for Out-of-Domain Generalization
Training with an emphasis on "hard-to-learn" components of the data has been proven as an effective method to improve the generalization of machine learning models, especially in the settings where robustness (e. g., generalization across distributions) is valued.
Toward Learning Human-aligned Cross-domain Robust Models by Countering Misaligned Features
Machine learning has demonstrated remarkable prediction accuracy over i. i. d data, but the accuracy often drops when tested with data from another distribution.
Collaging Class-specific GANs for Semantic Image Synthesis
We propose a new approach for high resolution semantic image synthesis.
Equine Pain Behavior Classification via Self-Supervised Disentangled Pose Representation
Consequently, a pragmatic equine pain classification system would use video of the unobserved horse and weak labels.
Few-shot Image Generation via Cross-domain Correspondence
Training generative models, such as GANs, on a target domain containing limited examples (e. g., 10) can easily result in overfitting.
Ranked #3 on
10-shot image generation
on Babies
Progressive Temporal Feature Alignment Network for Video Inpainting
To achieve this goal, it is necessary to find correspondences from neighbouring frames to faithfully hallucinate the unknown content.
Generating Furry Cars: Disentangling Object Shape & Appearance across Multiple Domains
We consider the novel task of learning disentangled representations of object shape and appearance across multiple domains (e. g., dogs and cars).
Generating Furry Cars: Disentangling Object Shape and Appearance across Multiple Domains
We consider the novel task of learning disentangled representations of object shape and appearance across multiple domains (e. g., dogs and cars).
YolactEdge: Real-time Instance Segmentation on the Edge
We propose YolactEdge, the first competitive instance segmentation approach that runs on small edge devices at real-time speeds.
Delving Deeper into Anti-aliasing in ConvNets
Aliasing refers to the phenomenon that high frequency signals degenerate into completely different ones after sampling.
Instance-aware, Context-focused, and Memory-efficient Weakly Supervised Object Detection
Weakly supervised learning has emerged as a compelling tool for object detection by reducing the need for strong supervision during training.
Ranked #1 on
Weakly Supervised Object Detection
on COCO test-dev
Action Graphs: Weakly-supervised Action Localization with Graph Convolution Networks
We present a method for weakly-supervised action localization based on graph convolutions.
Audiovisual SlowFast Networks for Video Recognition
We present Audiovisual SlowFast Networks, an architecture for integrated audiovisual perception.
Don't Judge an Object by Its Context: Learning to Overcome Contextual Bias
Our key idea is to decorrelate feature representations of a category from its co-occurring context.
YOLACT++: Better Real-time Instance Segmentation
Then we produce instance masks by linearly combining the prototypes with the mask coefficients.
Ranked #14 on
Real-time Instance Segmentation
on MSCOCO
Password-conditioned Anonymization and Deanonymization with Face Identity Transformers
Cameras are prevalent in our daily lives, and enable many useful systems built upon computer vision technologies such as smart cameras and home robots for service applications.
MixNMatch: Multifactor Disentanglement and Encoding for Conditional Image Generation
We present MixNMatch, a conditional generative model that learns to disentangle and encode background, object pose, shape, and texture from real images with minimal supervision, for mix-and-match image generation.
Elastic-InfoGAN: Unsupervised Disentangled Representation Learning in Class-Imbalanced Data
We propose a novel unsupervised generative model that learns to disentangle object identity from other low-level aspects in class-imbalanced data.
YOLACT: Real-time Instance Segmentation
Then we produce instance masks by linearly combining the prototypes with the mask coefficients.
Ranked #20 on
Real-time Instance Segmentation
on MSCOCO
FineGAN: Unsupervised Hierarchical Disentanglement for Fine-Grained Object Generation and Discovery
We propose FineGAN, a novel unsupervised GAN framework, which disentangles the background, object shape, and object appearance to hierarchically generate images of fine-grained object categories.
Ranked #1 on
Image Clustering
on Stanford Dogs
Hide-and-Seek: A Data Augmentation Technique for Weakly-Supervised Localization and Beyond
Our approach only needs to modify the input image and can work with any network to improve its performance.
A Visual Attention Grounding Neural Model for Multimodal Machine Translation
The model leverages a visual attention grounding mechanism that links the visual semantics with the corresponding textual semantics.
Ranked #12 on
Multimodal Machine Translation
on Multi30K
DOCK: Detecting Objects by transferring Common-sense Knowledge
We present a scalable approach for Detecting Objects by transferring Common-sense Knowledge (DOCK) from source to target categories.
Learning to Anonymize Faces for Privacy Preserving Action Detection
The end result is a video anonymizer that performs pixel-level modifications to anonymize each person's face, with minimal effect on action detection performance.
Video Object Detection with an Aligned Spatial-Temporal Memory
We introduce Spatial-Temporal Memory Networks for video object detection.
Cross-Domain Self-supervised Multi-task Feature Learning using Synthetic Imagery
In human learning, it is common to use multiple sources of information jointly.
Who Will Share My Image? Predicting the Content Diffusion Path in Online Social Networks
Content popularity prediction has been extensively studied due to its importance and interest for both users and hosts of social media sites like Facebook, Instagram, Twitter, and Pinterest.
Weakly-supervised Visual Grounding of Phrases with Linguistic Structures
We propose a weakly-supervised approach that takes image-sentence pairs as input and learns to visually ground (i. e., localize) arbitrary linguistic phrases, in the form of spatial attention masks.
Identifying First-person Camera Wearers in Third-person Videos
We consider scenarios in which we wish to perform joint scene understanding, object tracking, activity recognition, and other tasks in environments in which multiple people are wearing body-worn cameras while a third-person static camera also captures the scene.
Interspecies Knowledge Transfer for Facial Keypoint Detection
Instead of directly finetuning a network trained to detect keypoints on human faces to animal faces (which is sub-optimal since human and animal faces can look quite different), we propose to first adapt the animal images to the pre-trained human detection network by correcting for the differences in animal and human face shape.
Hide-and-Seek: Forcing a Network to be Meticulous for Weakly-supervised Object and Action Localization
We propose `Hide-and-Seek', a weakly-supervised framework that aims to improve object localization in images and action localization in videos.
Ranked #24 on
Weakly Supervised Action Localization
on THUMOS 2014
End-to-End Localization and Ranking for Relative Attributes
We propose an end-to-end deep convolutional network to simultaneously localize and rank relative visual attributes, given only weakly-supervised pairwise image comparisons.
Track and Segment: An Iterative Unsupervised Approach for Video Object Proposals
We present an unsupervised approach that generates a diverse, ranked set of bounding box and segmentation video object proposals---spatio-temporal tubes that localize the foreground objects---in an unannotated video.
Track and Transfer: Watching Videos to Simulate Strong Human Supervision for Weakly-Supervised Object Detection
The status quo approach to training object detectors requires expensive bounding box annotations.
Discovering the Spatial Extent of Relative Attributes
We present a weakly-supervised approach that discovers the spatial extent of relative attributes, given only pairs of ordered images.
FlowWeb: Joint Image Set Alignment by Weaving Consistent, Pixel-Wise Correspondences
Given a set of poorly aligned images of the same visual concept without any annotations, we propose an algorithm to jointly bring them into pixel-wise correspondence by estimating a FlowWeb representation of the image set.
Predicting Important Objects for Egocentric Video Summarization
Our results on two egocentric video datasets show the method's promise relative to existing techniques for saliency and summarization.
Weakly-supervised Discovery of Visual Pattern Configurations
The increasing prominence of weakly labeled data nurtures a growing demand for object detection methods that can cope with minimal supervision.
