Search Results for author:
Found 57 papers, 29 papers with code
MM-GEN: Enhancing Task Performance Through Targeted Multimodal Data Curation
Vision-language models (VLMs) are highly effective but often underperform on specialized tasks; for example, Llava-1. 5 struggles with chart and diagram understanding due to scarce task-specific training data.
RiTTA: Modeling Event Relations in Text-to-Audio Generation
In this work, we systematically study audio event relation modeling in TTA generation models.
On Occlusions in Video Action Detection: Benchmark Datasets And Training Recipes
This paper explores the impact of occlusions in video action detection.
Unearthing Skill-Level Insights for Understanding Trade-Offs of Foundation Models
With models getting stronger, evaluations have grown more complex, testing multiple skills in one benchmark and even in the same instance at once.
Eureka: Evaluating and Understanding Large Foundation Models
Second, we introduce Eureka-Bench as an extensible collection of benchmarks testing capabilities that (i) are still challenging for state-of-the-art models and (ii) represent fundamental but overlooked language and multimodal capabilities.
GeoMeter: Probing Depth and Height Perception of Large Visual-Language Models
We benchmark 17 state-of-the-art VLMs using these datasets and find that they consistently struggle with both depth and height perception.
Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models
Large language models (LLMs) and vision-language models (VLMs) have demonstrated remarkable performance across a wide range of tasks and domains.
Exposing the Achilles' Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical Reasoning
Large Language Models (LLMs) have been applied to Math Word Problems (MWPs) with transformative impacts, revolutionizing how these complex problems are approached and solved in various domains including educational settings.
Navigating Hallucinations for Reasoning of Unintentional Activities
In this work we present a novel task of understanding unintentional human activities in videos.
DreamDistribution: Prompt Distribution Learning for Text-to-Image Diffusion Models
We introduce a solution that allows a pretrained T2I diffusion model to learn a set of soft prompts, enabling the generation of novel images by sampling prompts from the learned distribution.
PEEKABOO: Interactive Video Generation via Masked-Diffusion
In this work, we introduce the first solution to equip diffusion-based video generation models with spatio-temporal control.
DAMEX: Dataset-aware Mixture-of-Experts for visual understanding of mixture-of-datasets
Construction of a universal detector poses a crucial question: How can we most effectively train a model on a large mixture of datasets?
Efficiently Robustify Pre-trained Models
A recent trend in deep learning algorithms has been towards training large scale models, having high parameter count and trained on big dataset.
Beyond Generation: Harnessing Text to Image Models for Object Detection and Segmentation
A foreground-background segmentation algorithm is then used to generate foreground object masks.
Robustness Analysis on Foundational Segmentation Models
In this work, we perform a robustness analysis of Visual Foundation Models (VFMs) for segmentation tasks and focus on robustness against real-world distribution shift inspired perturbations.
A Large-Scale Analysis on Self-Supervised Video Representation Learning
In this work, we first provide a benchmark that enables a comparison of existing approaches on the same ground.
Ranked #3 on
Self-Supervised Action Recognition
on UCF101
Controllable Text-to-Image Generation with GPT-4
Control-GPT works by querying GPT-4 to write TikZ code, and the generated sketches are used as references alongside the text instructions for diffusion models (e. g., ControlNet) to generate photo-realistic images.
PLEX: Making the Most of the Available Data for Robotic Manipulation Pretraining
A rich representation is key to general robotic manipulation, but existing approaches to representation learning require large amounts of multimodal demonstrations.
Exploring the Sim2Real Gap Using Digital Twins
An emerging alternative is to use synthetic data, but if the synthetic data is not similar enough to the real data, the performance is typically below that of training with real data.
A Large-Scale Robustness Analysis of Video Action Recognition Models
In this work, we perform a large-scale robustness analysis of these existing models for video action recognition.
Benchmarking Spatial Relationships in Text-to-Image Generation
We investigate the ability of T2I models to generate correct spatial relationships among objects and present VISOR, an evaluation metric that captures how accurately the spatial relationship described in text is generated in the image.
EM-Paste: EM-guided Cut-Paste with DALL-E Augmentation for Image-level Weakly Supervised Instance Segmentation
Finally, the third component creates a large-scale pseudo-labeled instance segmentation training dataset by compositing the foreground object masks onto the original and generated background images.
Instance-Aware Image Completion
However, existing image completion methods tend to fill in the missing region with the surrounding texture instead of hallucinating a visual instance that is suitable in accordance with the context of the scene.
Learning to Simulate Realistic LiDARs
Simulating realistic sensors is a challenging part in data generation for autonomous systems, often involving carefully handcrafted sensor design, scene properties, and physics modeling.
Neural-Sim: Learning to Generate Training Data with NeRF
However, existing approaches either require human experts to manually tune each scene property or use automatic methods that provide little to no control; this requires rendering large amounts of random data variations, which is slow and is often suboptimal for the target domain.
TASKOGRAPHY: Evaluating robot task planning over large 3D scene graphs
3D scene graphs (3DSGs) are an emerging description; unifying symbolic, topological, and metric scene representations.
Scaling Novel Object Detection with Weakly Supervised Detection Transformers
A critical object detection task is finetuning an existing model to detect novel objects, but the standard workflow requires bounding box annotations which are time-consuming and expensive to collect.
Robustness Analysis of Video-Language Models Against Visual and Language Perturbations
Joint visual and language modeling on large-scale datasets has recently shown good progress in multi-modal tasks when compared to single modal learning.
Large-scale Robustness Analysis of Video Action Recognition Models
In this work, we perform a large-scale robustness analysis of these existing models for video action recognition.
DALL-E for Detection: Language-driven Compositional Image Synthesis for Object Detection
For foreground object mask generation, we use a simple textual template with object class name as input to DALL-E to generate a diverse set of foreground images.
Missingness Bias in Model Debugging
Missingness, or the absence of features from an input, is a concept fundamental to many model debugging tools.
Image Retrieval from Contextual Descriptions
In particular, models are tasked with retrieving the correct image from a set of 10 minimally contrastive candidates based on a contextual description.
Ranked #1 on
Image Retrieval
on ImageCoDe
Inferring Articulated Rigid Body Dynamics from RGBD Video
Being able to reproduce physical phenomena ranging from light interaction to contact mechanics, simulators are becoming increasingly useful in more and more application domains where real-world interaction or labeled data are difficult to obtain.
One Network Doesn't Rule Them All: Moving Beyond Handcrafted Architectures in Self-Supervised Learning
We conduct a large-scale study with over 100 variants of ResNet and MobileNet architectures and evaluate them across 11 downstream scenarios in the SSL setting.
Robust Contrastive Learning against Noisy Views
Contrastive learning relies on an assumption that positive pairs contain related views, e. g., patches of an image or co-occurring multimodal signals of a video, that share certain underlying information about an instance.
Learning to Align Sequential Actions in the Wild
To this end, we propose an approach to enforce temporal priors on the optimal transport matrix, which leverages temporal consistency, while allowing for variations in the order of actions.
CausalCity: Complex Simulations with Agency for Causal Discovery and Reasoning
The ability to perform causal and counterfactual reasoning are central properties of human intelligence.
3DB: A Framework for Debugging Computer Vision Models
We introduce 3DB: an extendable, unified framework for testing and debugging vision models using photorealistic simulation.
RANP: Resource Aware Neuron Pruning at Initialization for 3D CNNs
In this work, we introduce a Resource Aware Neuron Pruning (RANP) algorithm that prunes 3D CNNs at initialization to high sparsity levels.
Prediction of Object Geometry from Acoustic Scattering Using Convolutional Neural Networks
The present work proposes a method to infer object geometry from scattering features by training convolutional neural networks.
RANP: Resource Aware Neuron Pruning at Initialization for 3D CNNs
Specifically, the core idea is to obtain an importance score for each neuron based on their sensitivity to the loss function.
AutoSimulate: (Quickly) Learning Synthetic Data Generation
Simulation is increasingly being used for generating large labelled datasets in many machine learning problems.
Depth Completion Using a View-constrained Deep Prior
Given color images and noisy and incomplete target depth maps, we optimize a randomly-initialized CNN model to reconstruct a depth map restored by virtue of using the CNN network structure as a prior combined with a view-constrained photo-consistency loss.
Fast acoustic scattering using convolutional neural networks
Diffracted scattering and occlusion are important acoustic effects in interactive auralization and noise control applications, typically requiring expensive numerical simulation.
Learning Visuomotor Policies for Aerial Navigation Using Cross-Modal Representations
We analyze the rich latent spaces learned with our proposed representations, and show that the use of our cross-modal architecture significantly improves control policy performance as compared to end-to-end learning or purely unsupervised feature extractors.
Live Reconstruction of Large-Scale Dynamic Outdoor Worlds
For each time step, our dynamic map maintains a relative pose of each volume with respect to the stationary background.
Privacy-Preserving Action Recognition using Coded Aperture Videos
We propose a lens-free coded aperture camera system for human action recognition that is privacy-preserving.
Photorealistic Image Synthesis for Object Instance Detection
We present an approach to synthesize highly photorealistic images of 3D object models, which we use to train a convolutional neural network for detecting the objects in real images.
Playing for Data: Ground Truth from Computer Games
Recent progress in computer vision has been driven by high-capacity models trained on large datasets.
Dense Monocular Depth Estimation in Complex Dynamic Scenes
We present an approach to dense depth estimation from a single monocular camera that is moving through a dynamic scene.
Feature Space Optimization for Semantic Video Segmentation
We present an approach to long-range spatio-temporal regularization in semantic video segmentation.
SemanticPaint: A Framework for the Interactive Segmentation of 3D Scenes
We present an open-source, real-time implementation of SemanticPaint, a system for geometric reconstruction, object-class segmentation and learning of 3D scenes.
Conditional Random Fields as Recurrent Neural Networks
Pixel-level labelling tasks, such as semantic segmentation, play a central role in image understanding.
Ranked #36 on
Semantic Segmentation
on PASCAL VOC 2012 test
Dense Semantic Image Segmentation with Objects and Attributes
The concepts of objects and attributes are both important for describing images precisely, since verbal descriptions often contain both adjectives and nouns (e. g. "I see a shiny red chair').
A Tiered Move-making Algorithm for General Non-submodular Pairwise Energies
The algorithm converges to a local minimum for any general pairwise potential, and we give a theoretical analysis of the properties of the algorithm, characterizing the situations in which we can expect good performance.
Higher Order Priors for Joint Intrinsic Image, Objects, and Attributes Estimation
Many methods have been proposed to recover the intrinsic scene properties such as shape, reflectance and illumination from a single image.
ImageSpirit: Verbal Guided Image Parsing
This allows us to formulate the image parsing problem as one of jointly estimating per-pixel object and attribute labels from a set of training images.
