Search Results for author:
Found 84 papers, 36 papers with code
Monocular 3D Object Detection via Feature Domain Adaptation
Monocular 3D object detection is a challenging task due to unreliable depth, resulting in a distinct performance gap between monocular and LiDAR-based approaches.
AVicuna: Audio-Visual LLM with Interleaver and Context-Boundary Alignment for Temporal Referential Dialogue
In everyday communication, humans frequently use speech and gestures to refer to specific areas or objects, a process known as Referential Dialogue (RD).
Adaptive Super Resolution For One-Shot Talking-Head Generation
In this work, we propose an adaptive high-quality talking-head video generation method, which synthesizes high-resolution video without additional pre-trained modules.
Forward Learning for Gradient-based Black-box Saliency Map Generation
Gradient-based saliency maps are widely used to explain deep neural network decisions.
Discover and Mitigate Multiple Biased Subgroups in Image Classifiers
Discovering biased subgroups is the key to understanding models' failure modes and further improving models' robustness.
Approximated Likelihood Ratio: A Forward-Only and Parallel Framework for Boosting Neural Network Training
In this paper, we introduce an approximation technique for the likelihood ratio (LR) method to alleviate computational and memory demands in gradient estimation.
OSCaR: Object State Captioning and State Change Representation
To address these challenges, in this paper, we introduce the Object State Captioning and State Change Representation (OSCaR) dataset and benchmark.
Efficiently Leveraging Linguistic Priors for Scene Text Spotting
This paper proposes a method that leverages linguistic knowledge from a large text corpus to replace the traditional one-hot encoding used in auto-regressive scene text spotting and recognition models.
Emo-Avatar: Efficient Monocular Video Style Avatar through Texture Rendering
To address the above problems, we propose the Efficient Monotonic Video Style Avatar (Emo-Avatar) through deferred neural rendering that enhances StyleGAN's capacity for producing dynamic, drivable portrait videos.
Tri$^{2}$-plane: Volumetric Avatar Reconstruction with Feature Pyramid
Recent years have witnessed considerable achievements in facial avatar reconstruction with neural volume rendering.
Bag of Tricks to Boost Adversarial Transferability
In this work, we find that several tiny changes in the existing adversarial attacks can significantly affect the attack performance, \eg, the number of iterations and step size.
Learning Audio Concepts from Counterfactual Natural Language
Conventional audio classification relied on predefined classes, lacking the ability to learn from free-form text.
Video Understanding with Large Language Models: A Survey
With the burgeoning growth of online video platforms and the escalating volume of video content, the demand for proficient video understanding tools has intensified markedly.
Scalable CP Decomposition for Tensor Learning using GPU Tensor Cores
However, the application of tensor decomposition is largely hindered by the exponential increment of the computational complexity and storage consumption with the size of tensors.
MISAR: A Multimodal Instructional System with Augmented Reality
Augmented reality (AR) requires the seamless integration of visual, auditory, and linguistic channels for optimized human-computer interaction.
Separating Invisible Sounds Toward Universal Audiovisual Scene-Aware Sound Separation
The audio-visual sound separation field assumes visible sources in videos, but this excludes invisible sounds beyond the camera's view.
Emotional Listener Portrait: Neural Listener Head Generation with Emotion
Listener head generation centers on generating non-verbal behaviors (e. g., smile) of a listener in reference to the information delivered by a speaker.
Neural Acoustic Context Field: Rendering Realistic Room Impulse Response With Neural Fields
Room impulse response (RIR), which measures the sound propagation within an environment, is critical for synthesizing high-fidelity audio for a given environment.
DAVIS: High-Quality Audio-Visual Separation with Generative Diffusion Models
We propose DAVIS, a Diffusion model-based Audio-VIusal Separation framework that solves the audio-visual sound source separation task through a generative manner.
Unveiling Cross Modality Bias in Visual Question Answering: A Causal View with Possible Worlds VQA
In this paper, we first model a confounding effect that causes language and vision bias simultaneously, then propose a counterfactual inference to remove the influence of this effect.
One Forward is Enough for Neural Network Training via Likelihood Ratio Method
While backpropagation (BP) is the mainstream approach for gradient computation in neural network training, its heavy reliance on the chain rule of differentiation constrains the designing flexibility of network architecture and training pipelines.
Egocentric Audio-Visual Object Localization
In this paper, we explore the challenging egocentric audio-visual object localization task and observe that 1) egomotion commonly exists in first-person recordings, even within a short duration; 2) The out-of-view sound components can be created while wearers shift their attention.
Improving Adversarial Transferability with Scheduled Step Size and Dual Example
Motivated by this finding, we argue that the information of adversarial perturbations near the benign sample, especially the direction, benefits more on the transferability.
A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Others
Key to advancing the reliability of vision systems is understanding whether existing methods can overcome multiple shortcuts or struggle in a Whac-A-Mole game, i. e., where mitigating one shortcut amplifies reliance on others.
Ranked #1 on
Out-of-Distribution Generalization
on ImageNet-W
Discover and Mitigate Unknown Biases with Debiasing Alternate Networks
By training in an alternate manner, the discoverer tries to find multiple unknown biases of the classifier without any annotations of biases, and the classifier aims at unlearning the biases identified by the discoverer.
Ranked #1 on
Out-of-Distribution Generalization
on ImageNet-W
Transformer-empowered Multi-scale Contextual Matching and Aggregation for Multi-contrast MRI Super-resolution
However, existing methods still have two shortcomings: (1) they neglect that the multi-contrast features at different scales contain different anatomical details and hence lack effective mechanisms to match and fuse these features for better reconstruction; and (2) they are still deficient in capturing long-range dependencies, which are essential for the regions with complicated anatomical structures.
Learning to Answer Questions in Dynamic Audio-Visual Scenarios
In this paper, we focus on the Audio-Visual Question Answering (AVQA) task, which aims to answer questions regarding different visual objects, sounds, and their associations in videos.
Ranked #5 on
Audio-visual Question Answering
on MUSIC-AVQA
Cross-modal Contrastive Distillation for Instructional Activity Anticipation
In this study, we aim to predict the plausible future action steps given an observation of the past and study the task of instructional activity anticipation.
SpaceEdit: Learning a Unified Editing Space for Open-Domain Image Color Editing
Recently, large pretrained models (e. g., BERT, StyleGAN, CLIP) show great knowledge transfer and generalization capability on various downstream tasks within their domains.
Anomaly Crossing: New Horizons for Video Anomaly Detection as Cross-domain Few-shot Learning
Video anomaly detection aims to identify abnormal events that occurred in videos.
SpaceEdit: Learning a Unified Editing Space for Open-Domain Image Editing
Recently, large pretrained models (e. g., BERT, StyleGAN, CLIP) have shown great knowledge transfer and generalization capability on various downstream tasks within their domains.
Space-Time Memory Network for Sounding Object Localization in Videos
Leveraging temporal synchronization and association within sight and sound is an essential step towards robust localization of sounding objects.
Procedure Planning in Instructional Videos via Contextual Modeling and Model-based Policy Learning
In this work, we leverage instructional videos to study humans' decision-making processes, focusing on learning a model to plan goal-directed actions in real-life videos.
rQdia: Regularizing Q-Value Distributions With Image Augmentation
rQdia (pronounced “Arcadia”) regularizes Q-value distributions with augmented images in pixel-based deep reinforcement learning.
Learning to Generate Scene Graph from Natural Language Supervision
To bridge the gap between images and texts, we leverage an off-the-shelf object detector to identify and localize object instances, match labels of detected regions to concepts parsed from captions, and thus create "pseudo" labels for learning scene graph.
Learning by Planning: Language-Guided Global Image Editing
Recently, language-guided global image editing draws increasing attention with growing application potentials.
Discover the Unknown Biased Attribute of an Image Classifier
To help human experts better find the AI algorithms' biases, we study a new problem in this work -- for a classifier that predicts a target attribute of the input image, discover its unknown biased attribute.
Zooming SlowMo: An Efficient One-Stage Framework for Space-Time Video Super-Resolution
A na\"ive method is to decompose it into two sub-tasks: video frame interpolation (VFI) and video super-resolution (VSR).
Space-time Video Super-resolution
Video Frame Interpolation
+1
Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separation
There are rich synchronized audio and visual events in our daily life.
Can audio-visual integration strengthen robustness under multimodal attacks?
In this paper, we propose to make a systematic study on machines multisensory perception under attacks.
High-fidelity Face Tracking for AR/VR via Deep Lighting Adaptation
This paper addresses previous limitations by learning a deep learning lighting model, that in combination with a high-quality 3D face tracking algorithm, provides a method for subtle and robust facial motion transfer from a regular video to a 3D photo-realistic avatar.
A Simple Baseline for Weakly-Supervised Scene Graph Generation
We investigate the weakly-supervised scene graph generation, which is a challenging task since no correspondence of label and object is provided.
A Benchmark and Baseline for Language-Driven Image Editing
To solve this new task, we first present a new language-driven image editing dataset that supports both local and global editing with editing operation and mask annotations.
Cubic Spline Smoothing Compensation for Irregularly Sampled Sequences
The marriage of recurrent neural networks and neural ordinary differential networks (ODE-RNN) is effective in modeling irregularly-observed sequences.
Actor-Action Video Classification CSC 249/449 Spring 2020 Challenge Report
This technical report summarizes submissions and compiles from Actor-Action video classification challenge held as a final project in CSC 249/449 Machine Vision course (Spring 2020) at University of Rochester
Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing
In this paper, we introduce a new problem, named audio-visual video parsing, which aims to parse a video into temporal event segments and label them as either audible, visible, or both.
Talking-head Generation with Rhythmic Head Motion
When people deliver a speech, they naturally move heads, and this rhythmic head motion conveys prosodic information.
Graph Neural Network Based Coarse-Grained Mapping Prediction
The selection of coarse-grained (CG) mapping operators is a critical step for CG molecular dynamics (MD) simulation.
Explaining Local, Global, And Higher-Order Interactions In Deep Learning
We present a simple yet highly generalizable method for explaining interacting parts within a neural network's reasoning process.
What comprises a good talking-head video generation?: A Survey and Benchmark
In this work, we present a carefully-designed benchmark for evaluating talking-head video generation with standardized dataset pre-processing strategies.
Learning a Weakly-Supervised Video Actor-Action Segmentation Model with a Wise Selection
Instead of blindly trusting quality-inconsistent PAs, WS^2 employs a learning-based selection to select effective PAs and a novel region integrity criterion as a stopping condition for weakly-supervised training.
Deep Grouping Model for Unified Perceptual Parsing
The perceptual-based grouping process produces a hierarchical and compositional image representation that helps both human and machine vision systems recognize heterogeneous visual concepts.
Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution
Rather than synthesizing missing LR video frames as VFI networks do, we firstly temporally interpolate LR frame features in missing LR video frames capturing local temporal contexts by the proposed feature temporal interpolation network.
Ranked #4 on
Video Frame Interpolation
on Vid4 - 4x upscaling
Space-time Video Super-resolution
Video Frame Interpolation
+1
Assembling Semantically-Disentangled Representations for Predictive-Generative Models via Adaptation from Synthetic Domain
Deep neural networks can form high-level hierarchical representations of input data.
TailorGAN: Making User-Defined Fashion Designs
To overcome those limitations, we propose a novel self-supervised model to synthesize garment images with disentangled attributes (e. g., collar and sleeves) without paired data.
Learning from Interventions using Hierarchical Policies for Safe Learning
The recently proposed Learning from Interventions (LfI) overcomes this limitation by using an expert overseer.
Improve CAM with Auto-adapted Segmentation and Co-supervised Augmentation
In this paper, we propose a confidence segmentation (ConfSeg) module that builds confidence score for each pixel in CAM without introducing additional hyper-parameters.
Unsupervised Pose Flow Learning for Pose Guided Synthesis
Pose guided synthesis aims to generate a new image in an arbitrary target pose while preserving the appearance details from the source image.
Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Loss
We devise a cascade GAN approach to generate talking face video, which is robust to different face shapes, view angles, facial characteristics, and noisy audio conditions.
Dynamic Graph Modules for Modeling Object-Object Interactions in Activity Recognition
Video action recognition, a critical problem in video understanding, has been gaining increasing attention.
An Attempt towards Interpretable Audio-Visual Video Captioning
To achieve this, we propose a multimodal convolutional neural network-based audio-visual video captioning framework and introduce a modality-aware module for exploring modality selection during sentence generation.
TDAN: Temporally Deformable Alignment Network for Video Super-Resolution
Video super-resolution (VSR) aims to restore a photo-realistic high-resolution (HR) video frame from both its corresponding low-resolution (LR) frame (reference frame) and multiple neighboring frames (supporting frames).
GAN-EM: GAN based EM learning framework
To overcome such limitation, we propose a GAN based EM learning framework that can maximize the likelihood of images and estimate the latent variables with only the constraint of L-Lipschitz continuity.
How to Make a BLT Sandwich? Learning to Reason towards Understanding Web Instructional Videos
Furthermore, we study multiple modalities including description and transcripts for the purpose of boosting video understanding.
Navigation by Imitation in a Pedestrian-Rich Environment
Deep neural networks trained on demonstrations of human actions give robot the ability to perform self-driving on the road.
Lip Movements Generation at a Glance
In this paper, we consider a task of such: given an arbitrary audio speech and one lip image of arbitrary target identity, generate synthesized lip movements of the target identity saying the speech.
Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment
In this work, we address the task of weakly-supervised human action segmentation in long, untrimmed videos.
Generating Talking Face Landmarks from Speech
In this paper, we present a system that can generate landmark points of a talking face from an acoustic speech in real time.
Audio-Visual Event Localization in Unconstrained Videos
In this paper, we introduce a novel problem of audio-visual event localization in unconstrained videos.
MRI Tumor Segmentation with Densely Connected 3D CNN
The major difficulty of our segmentation model comes with the fact that the location, structure, and shape of gliomas vary significantly among different patients.
Video Action Segmentation with Hybrid Temporal Networks
Action segmentation as a milestone towards building automatic systems to understand untrimmed videos has received considerable attention in the recent years.
Weakly Supervised Actor-Action Segmentation via Robust Multi-Task Ranking
However, current methods for detailed understanding of actor and action have significant limitations: they require large amounts of finely labeled data, and they fail to capture any internal relationship among actors and actions.
TricorNet: A Hybrid Temporal Convolutional and Recurrent Network for Video Action Segmentation
Action segmentation as a milestone towards building automatic systems to understand untrimmed videos has received considerable attention in the recent years.
Ranked #4 on
Action Segmentation
on JIGSAWS
Action Understanding with Multiple Classes of Actors
Despite the rapid progress, existing works on action understanding focus strictly on one type of action agent, which we call actor---a human adult, ignoring the diversity of actions performed by other actors.
Deep Cross-Modal Audio-Visual Generation
Being the first to explore this new problem, we compose two new datasets with pairs of images and sounds of musical performances of different instruments.
Towards Automatic Learning of Procedures from Web Instructional Videos
To answer this question, we introduce the problem of procedure segmentation--to segment a video procedure into category-independent procedure segments.
Watch What You Just Said: Image Captioning with Text-Conditional Attention
Attention mechanisms have attracted considerable interest in image captioning due to its powerful performance.
LIBSVX: A Supervoxel Library and Benchmark for Early Video Processing
Supervoxel segmentation has strong potential to be incorporated into early video analysis as superpixel segmentation has in image analysis.
Actor-Action Semantic Segmentation with Grouping Process Models
Actor-action semantic segmentation made an important step toward advanced video understanding problems: what action is happening; who is performing the action; and where is the action in space-time.
Can Humans Fly? Action Understanding With Multiple Classes of Actors
There is no work we know of on simultaneously inferring actors and actions in the video, not to mention a dataset to experiment with.
A Study of Actor and Action Semantic Retention in Video Supervoxel Segmentation
In this paper, we conduct a systematic study of how well the actor and action semantics are retained in video supervoxel segmentation.
A Thousand Frames in Just a Few Words: Lingual Description of Videos through Latent Topics and Sparse Object Stitching
The problem of describing images through natural language has gained importance in the computer vision community.
