Search Results for author:
Found 100 papers, 46 papers with code
MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding
However, existing LLM-based large multimodal models (e. g., Video-LLaMA, VideoChat) can only take in a limited number of frames for short video understanding.
Ranked #1 on
Video Classification
on COIN
Measuring Style Similarity in Diffusion Models
We also propose a method to extract style descriptors that can be used to attribute style of a generated image to the images used in the training dataset of a text-to-image model.
What is Point Supervision Worth in Video Instance Segmentation?
Video instance segmentation (VIS) is a challenging vision task that aims to detect, segment, and track objects in videos.
LiFT: A Surprisingly Simple Lightweight Feature Transform for Dense ViT Descriptors
We present a simple self-supervised method to enhance the performance of ViT features for dense downstream tasks.
Customize-A-Video: One-Shot Motion Customization of Text-to-Video Diffusion Models
With the emergence of text-to-video (T2V) diffusion models, its temporal counterpart, motion customization, has not yet been well investigated.
Explaining the Implicit Neural Canvas: Connecting Pixels to Neurons by Tracing their Contributions
We call the aggregate of these contribution maps the Implicit Neural Canvas and we use this concept to demonstrate that the INRs which we study learn to ''see'' the frames they represent in surprising ways.
Video Dynamics Prior: An Internal Learning Approach for Robust Video Enhancements
In this paper, we present a novel robust framework for low-level vision tasks, including denoising, object removal, frame interpolation, and super-resolution, that does not require any external training data corpus.
EAGLES: Efficient Accelerated 3D Gaussians with Lightweight EncodingS
We validate the effectiveness of our approach on a variety of datasets and scenes preserving the visual quality while consuming 10-20x less memory and faster training/inference speed.
Gen2Det: Generate to Detect
In the long-tailed detection setting on LVIS, Gen2Det improves the performance on rare categories by a large margin while also significantly improving the performance on other categories, e. g. we see an improvement of 2. 13 Box AP and 1. 84 Mask AP over just training on real data on LVIS with Mask R-CNN.
Multimodality-guided Image Style Transfer using Cross-modal GAN Inversion
Compared to IIST, such approaches provide more flexibility with text-specified styles, which are useful in scenarios where the style is hard to define with reference images.
A Video is Worth 10,000 Words: Training and Benchmarking with Diverse Captions for Better Long Video Retrieval
To provide a more thorough evaluation of the capabilities of long video retrieval systems, we propose a pipeline that leverages state-of-the-art large language models to carefully generate a diverse set of synthetic captions for long videos.
Do text-free diffusion models learn discriminative visual representations?
We find that the intermediate feature maps of the U-Net are diverse, discriminative feature representations.
Multi-entity Video Transformers for Fine-Grained Video Representation Learning
In this work, we advance the state-of-the-art for this area by re-examining the design of transformer architectures for video representation learning.
SHACIRA: Scalable HAsh-grid Compression for Implicit Neural Representations
Implicit Neural Representations (INR) or neural fields have emerged as a popular framework to encode multimedia signals such as images and radiance fields while retaining high-quality.
Chop & Learn: Recognizing and Generating Object-State Compositions
Recognizing and generating object-state compositions has been a challenging task, especially when generalizing to unseen compositions.
Diff2Lip: Audio Conditioned Diffusion Models for Lip-Synchronization
We show results on both reconstruction (same audio-video inputs) as well as cross (different audio-video inputs) settings on Voxceleb2 and LRW datasets.
Diffusion Models Beat GANs on Image Classification
We explore optimal methods for extracting and using these embeddings for classification tasks, demonstrating promising results on the ImageNet classification task.
SimpSON: Simplifying Photo Cleanup with Single-Click Distracting Object Segmentation Network
In photo editing, it is common practice to remove visual distractions to improve the overall image quality and highlight the primary subject.
MOST: Multiple Object localization with Self-supervised Transformers for object discovery
In this work, we present Multiple Object localization with Self-supervised Transformers (MOST) that uses features of transformers trained using self-supervised learning to localize multiple objects in real world images.
HNeRV: A Hybrid Neural Representation for Videos
Such embedding largely limits the regression capacity and internal generalization for video interpolation.
Ranked #3 on
Video Reconstruction
on UVG
ASIC: Aligning Sparse in-the-wild Image Collections
We present a self-supervised technique that directly optimizes on a sparse collection of images of a particular object/object category to obtain consistent dense correspondences across the collection.
FlexNeRF: Photorealistic Free-viewpoint Rendering of Moving Humans from Sparse Views
We present FlexNeRF, a method for photorealistic freeviewpoint rendering of humans in motion from monocular videos.
Towards Scalable Neural Representation for Diverse Videos
Implicit neural representations (INR) have gained increasing attention in representing 3D scenes and images, and have been recently applied to encode videos (e. g., NeRV, E-NeRV).
Align and Attend: Multimodal Summarization with Dual Contrastive Losses
The goal of multimodal summarization is to extract the most important information from different modalities to form output summaries.
Ranked #3 on
Supervised Video Summarization
on SumMe
Extractive Text Summarization
Supervised Video Summarization
COVID-VTS: Fact Extraction and Verification on Short Video Platforms
We introduce a new benchmark, COVID-VTS, for fact-checking multi-modal information involving short-duration videos with COVID19- focused information from both the real world and machine generation.
BT^2: Backward-compatible Training with Basis Transformation
In this way, the new representation can be directly compared with the old representation, in principle avoiding the need for any backfilling.
NIRVANA: Neural Implicit Representations of Videos with Adaptive Networks and Autoregressive Patch-wise Modeling
This design shares computation within each group, in the spatial and temporal dimensions, resulting in reduced encoding time of the video.
Teaching Matters: Investigating the Role of Supervision in Vision Transformers
We compare ViTs trained through different methods of supervision, and show that they learn a diverse range of behaviors in terms of their attention, representations, and downstream performance.
CNeRV: Content-adaptive Neural Representation for Visual Data
We match the performance of NeRV, a state-of-the-art implicit neural representation, on the reconstruction task for frames seen during training while far surpassing for frames that are skipped during training (unseen images).
$BT^2$: Backward-compatible Training with Basis Transformation
In this way, the new representation can be directly compared with the old representation, in principle avoiding the need for any backfilling.
Learning Semantic Correspondence with Sparse Annotations
In this paper, we aim to address the challenge of label sparsity in semantic correspondence by enriching supervision signals from sparse keypoint annotations.
Beyond Supervised vs. Unsupervised: Representative Benchmarking and Analysis of Image Representation Learning
In this paper, we compare methods using performance-based benchmarks such as linear evaluation, nearest neighbor classification, and clustering for several different datasets, demonstrating the lack of a clear front-runner within the current state-of-the-art.
Disentangling Visual Embeddings for Attributes and Objects
We use visual decomposed features to hallucinate embeddings that are representative for the seen and novel compositions to better regularize the learning of our model.
Neural Space-filling Curves
We present Neural Space-filling Curves (SFCs), a data-driven approach to infer a context-based scan order for a set of images.
LilNetX: Lightweight Networks with EXtreme Model Compression and Structured Sparsification
We introduce LilNetX, an end-to-end trainable technique for neural networks that enables learning models with specified accuracy-rate-computation trade-off.
ASM-Loc: Action-aware Segment Modeling for Weakly-Supervised Temporal Action Localization
Without the boundary information of action segments, existing methods mostly rely on multiple instance learning (MIL), where the predictions of unlabeled instances (i. e., video snippets) are supervised by classifying labeled bags (i. e., untrimmed videos).
ObjectFormer for Image Manipulation Detection and Localization
Recent advances in image editing techniques have posed serious challenges to the trustworthiness of multimedia data, which drives the research of image tampering detection.
One Network Doesn't Rule Them All: Moving Beyond Handcrafted Architectures in Self-Supervised Learning
We conduct a large-scale study with over 100 variants of ResNet and MobileNet architectures and evaluate them across 11 downstream scenarios in the SSL setting.
Leveraging Bitstream Metadata for Fast, Accurate, Generalized Compressed Video Quality Enhancement
Video compression is a central feature of the modern internet powering technologies from social media to video conferencing.
SparseDet: Improving Sparsely Annotated Object Detection with Pseudo-positive Mining
On average, we improve by $2. 6$, $3. 9$ and $9. 6$ mAP over previous state-of-the-art methods on three splits of increasing sparsity on COCO.
Dual-Key Multimodal Backdoors for Visual Question Answering
This is challenging for the attacker as the detector can distort or ignore the visual trigger entirely, which leads to models where backdoors are over-reliant on the language trigger.
Burn After Reading: Online Adaptation for Cross-domain Streaming Data
In the context of online privacy, many methods propose complex privacy and security preserving measures to protect sensitive data.
PatchGame: Learning to Signal Mid-level Patches in Referential Games
We study a referential game (a type of signaling game) where two agents communicate with each other via a discrete bottleneck to achieve a common goal.
NeRV: Neural Representations for Videos
In contrast, with NeRV, we can use any neural network compression method as a proxy for video compression, and achieve comparable performance to traditional frame-based video compression approaches (H. 264, HEVC \etc).
Ranked #6 on
Video Reconstruction
on UVG
A Frequency Perspective of Adversarial Robustness
Our analysis shows that adversarial examples are neither in high-frequency nor in low-frequency components, but are simply dataset dependent.
HR-RCNN: Hierarchical Relational Reasoning for Object Detection
Incorporating relational reasoning in neural networks for object recognition remains an open problem.
Diverse Video Generation using a Gaussian Process Trigger
Our approach, Diverse Video Generator, uses a Gaussian Process (GP) to learn priors on future states given the past and maintains a probability distribution over possible futures given a particular sample.
Ranked #1 on
Video Prediction
on KTH
(Diversity metric)
Hierarchical Video Prediction Using Relational Layouts for Human-Object Interactions
Our work builds on hierarchical video prediction models, which disentangle the video generation process into two stages: predicting a high-level representation, such as pose sequence, and then learning a pose-to-pixels translation model for pixel generation.
Learning Graphs for Knowledge Transfer With Limited Labels
The standard paradigm is to utilize relationships in the input graph to transfer information using GCNs from training to testing nodes in the graph; for example, the semi-supervised, zero-shot, and few-shot learning setups.
Learning to Predict Visual Attributes in the Wild
In this paper, we introduce a large-scale in-the-wild visual attribute prediction dataset consisting of over 927K attribute annotations for over 260K object instances.
Rethinking Pseudo Labels for Semi-Supervised Object Detection
In this paper, we introduce certainty-aware pseudo labels tailored for object detection, which can effectively estimate the classification and localization quality of derived pseudo labels.
Ranked #8 on
Semi-Supervised Object Detection
on COCO 100% labeled data
(using extra training data)
Towards Discovery and Attribution of Open-world GAN Generated Images
Through extensive experiments, we show that our algorithm discovers unseen GANs with high accuracy and also generalizes to GANs trained on unseen real datasets.
The Pursuit of Knowledge: Discovering and Localizing Novel Categories using Dual Memory
We tackle object category discovery, which is the problem of discovering and localizing novel objects in a large unlabeled dataset.
Learned Spatial Representations for Few-shot Talking-Head Synthesis
In contrast, we propose to factorize the representation of a subject into its spatial and style components.
StEP: Style-based Encoder Pre-training for Multi-modal Image Synthesis
Specifically, we pre-train a generic style encoder using a novel proxy task to learn an embedding of images, from arbitrary domains, into a low-dimensional style latent space.
Knowledge Evolution in Neural Networks
We evaluate KE using relatively small datasets (e. g., CUB-200) and randomly initialized deep networks.
SVMax: A Feature Embedding Regularizer
The SVMax regularizer supports both supervised and unsupervised learning.
Deep Video Inpainting Detection
This paper studies video inpainting detection, which localizes an inpainted region in a video both spatially and temporally.
Multimodal Attention for Layout Synthesis in Diverse Domains
We address the problem of scene layout generation for diverse domains such as images, mobile applications, documents and 3D objects.
Learning What Not to Model: Gaussian Process Regression with Negative Constraints
But, what if for an input point '$\bar{\mathbf{x}}$', we want to constrain the GP to avoid a target regression value '$\bar{y}(\bar{\mathbf{x}})$' (a negative datapair)?
2D or not 2D? Adaptive 3D Convolution Selection for Efficient Video Recognition
Then, only frames and convolutions that are selected by the selection network are used in the 3D model to generate predictions.
Ranked #11 on
Action Recognition
on ActivityNet
GTA: Global Temporal Attention for Video Action Understanding
To this end, we introduce Global Temporal Attention (GTA), which performs global temporal attention on top of spatial attention in a decoupled manner.
The Lottery Ticket Hypothesis for Object Recognition
The recently proposed Lottery Ticket Hypothesis (LTH) states that deep neural networks trained on large datasets contain smaller subnetworks that achieve on par performance as the dense networks.
Analyzing and Mitigating JPEG Compression Defects in Deep Learning
We show that there is a significant penalty on common performance metrics for high compression.
Learning Visual Representations for Transfer Learning by Suppressing Texture
Recent literature has shown that features obtained from supervised training of CNNs may over-emphasize texture rather than encoding high-level information.
Ranked #19 on
Object Detection
on PASCAL VOC 2007
Pose And Joint-Aware Action Recognition
Unlike other modalities, constellation of joints and their motion generate models with succinct human motion information for activity recognition.
Ranked #1 on
Action Recognition
on Mimetics
Improved Modeling of 3D Shapes with Multi-view Depth Maps
Our simple encoder-decoder framework, comprised of a novel identity encoder and class-conditional viewpoint generator, generates 3D consistent depth maps.
All About Knowledge Graphs for Actions
Current action recognition systems require large amounts of training data for recognizing an action.
Ranked #17 on
Zero-Shot Action Recognition
on Kinetics
Deep Co-Training with Task Decomposition for Semi-Supervised Domain Adaptation
To integrate the strengths of the two classifiers, we apply the well-established co-training framework, in which the two classifiers exchange their high confident predictions to iteratively "teach each other" so that both classifiers can excel in the target domain.
Semi-supervised Domain Adaptation
Unsupervised Domain Adaptation
End-to-end Learning of Compressible Features
We propose a learned method that jointly optimizes for compressibility along with the task objective for learning the features.
A Generic Visualization Approach for Convolutional Neural Networks
Compared to classification networks, attention visualization for retrieval networks is hardly studied.
Curriculum Manager for Source Selection in Multi-Source Domain Adaptation
In this paper, we proposed an adversarial agent that learns a dynamic curriculum for source samples, called Curriculum Manager for Source Selection (CMSS).
Multi-Source Unsupervised Domain Adaptation
Unsupervised Domain Adaptation
Group Ensemble: Learning an Ensemble of ConvNets in a single ConvNet
Owing to group convolution and the shared-base, GENet can fully leverage the advantage of explicit ensemble learning while retaining the same computation as a single ConvNet.
LayoutTransformer: Layout Generation and Completion with Self-attention
Generating a new layout or extending an existing layout requires understanding the relationships between these primitives.
Quantization Guided JPEG Artifact Correction
The JPEG image compression algorithm is the most popular method of image compression because of its ability for large compression ratios.
Spatial Priming for Detecting Human-Object Interactions
The proposed method consists of a layout module which primes a visual module to predict the type of interaction between a human and an object.
PatchVAE: Learning Local Latent Codes for Recognition
Unsupervised representation learning holds the promise of exploiting large amounts of unlabeled data to learn general representations.
Hand-Priming in Object Localization for Assistive Egocentric Vision
Egocentric vision holds great promises for increasing access to visual information and improving the quality of life for people with visual impairments, with object recognition being one of the daily challenges for this population.
Depth Completion Using a View-constrained Deep Prior
Given color images and noisy and incomplete target depth maps, we optimize a randomly-initialized CNN model to reconstruct a depth map restored by virtue of using the CNN network structure as a prior combined with a view-constrained photo-consistency loss.
Style-based Encoder Pre-training for Multi-modal Image Synthesis
Then we train a generator to transform an input image along with a style-code to the output domain.
Scalable Model Compression by Entropy Penalized Reparameterization
We describe a simple and general neural network weight compression approach, in which the network parameters (weights and biases) are represented in a "latent" space, amounting to a reparameterization.
Render4Completion: Synthesizing Multi-View Depth Maps for 3D Shape Completion
Different from image-to-image translation network that completes each view separately, our novel network, multi-view completion net (MVCN), leverages information from all views of a 3D shape to help the completion of each single view.
EvalNorm: Estimating Batch Normalization Statistics for Evaluation
Batch normalization (BN) has been very effective for deep learning and is widely used.
Referring to Objects in Videos using Spatio-Temporal Identifying Descriptions
This paper presents a new task, the grounding of spatio-temporal identifying descriptions in videos.
Detecting Human-Object Interactions via Functional Generalization
We present an approach for detecting human-object interactions (HOIs) in images, based on the idea that humans interact with functionally similar objects in a similar manner.
Unsupervised Data Uncertainty Learning in Visual Retrieval Systems
We introduce an unsupervised formulation to estimate heteroscedastic uncertainty in retrieval systems.
Boosting Standard Classification Architectures Through a Ranking Regularizer
We employ triplet loss as a feature embedding regularizer to boost classification performance.
Generate, Segment and Refine: Towards Generic Manipulation Segmentation
The advent of image sharing platforms and the easy availability of advanced photo editing software have resulted in a large quantities of manipulated images being shared on the internet.
Actor-Centric Relation Network
A visualization of the learned relation features confirms that our approach is able to attend to the relevant relations for each action.
Ranked #15 on
Action Recognition
on AVA v2.1
Tracking Emerges by Colorizing Videos
We use large amounts of unlabeled video to learn models for visual tracking without manual human supervision.
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
What will happen if we increase the dataset size by 10x or 100x?
Ranked #2 on
Semantic Segmentation
on PASCAL VOC 2007
A-Fast-RCNN: Hard Positive Generation via Adversary for Object Detection
We propose to learn an adversarial network that generates examples with occlusions and deformations.
Ranked #20 on
Object Detection
on PASCAL VOC 2007
(using extra training data)
Beyond Skip Connections: Top-Down Modulation for Object Detection
But most of these fine details are lost in the early convolutional layers.
Ranked #203 on
Object Detection
on COCO test-dev
Cross-stitch Networks for Multi-task Learning
In this paper, we propose a principled approach to learn shared representations in ConvNets using multi-task learning.
Ranked #106 on
Semantic Segmentation
on NYU Depth v2
Training Region-based Object Detectors with Online Hard Example Mining
Our motivation is the same as it has always been -- detection datasets contain an overwhelming number of easy examples and a small number of hard examples.
Ranked #6 on
Face Verification
on Trillion Pairs Dataset
Watch and Learn: Semi-Supervised Learning for Object Detectors From Video
We present a semi-supervised approach that localizes multiple unknown object instances in long videos.
Watch and Learn: Semi-Supervised Learning of Object Detectors from Videos
We present a semi-supervised approach that localizes multiple unknown object instances in long videos.
Mid-level Elements for Object Detection
Building on the success of recent discriminative mid-level elements, we propose a surprisingly simple approach for object detection which performs comparable to the current state-of-the-art approaches on PASCAL VOC comp-3 detection challenge (no external data).
Enriching Visual Knowledge Bases via Object Discovery and Segmentation
In this paper, we propose to enrich these knowledge bases by automatically discovering objects and their segmentations from noisy Internet images.
