Search Results for author:
Found 101 papers, 27 papers with code
Where and when to look? Spatial-temporal attention for action recognition in videos
Our model is efficient, as it proposes a separable spatio-temporal mechanism for video attention, while being able to identify important parts of the video both spatially and temporally.
Action Recognition In Videos
Temporal Action Localization
+1
Response Wide Shut: Surprising Observations in Basic Vision Language Model Capabilities
Vision-Language Models (VLMs) have emerged as general purpose tools for addressing a variety of complex computer vision problems.
On Pre-training of Multimodal Language Models Customized for Chart Understanding
Experimental results show that CHOPINLLM exhibits strong performance in understanding both annotated and unannotated charts across a wide range of types.
Representing Animatable Avatar via Factorized Neural Fields
For reconstructing high-fidelity human 3D models from monocular videos, it is crucial to maintain consistent large-scale body shapes along with finely matched subtle wrinkles.
Visual Prompting for Generalized Few-shot Segmentation: A Multi-scale Approach
These learned visual prompts are used to prompt a multiscale transformer decoder to facilitate accurate dense predictions.
Preventing Catastrophic Forgetting through Memory Networks in Continuous Detection
Modern pre-trained architectures struggle to retain previous information while undergoing continuous fine-tuning on new tasks.
Visual Concept-driven Image Generation with Text-to-Image Diffusion Model
We illustrate that such joint alternating refinement leads to the learning of better tokens for concepts and, as a by-product, latent masks.
Multi-modal News Understanding with Professionally Labelled Videos (ReutersViLNews)
Towards a solution for designing this ability in algorithms, we present a large-scale analysis on an in-house dataset collected by the Reuters News Agency, called Reuters Video-Language News (ReutersViLNews) dataset which focuses on high-level video-language understanding with an emphasis on long-form news.
Joint Generative Modeling of Scene Graphs and Images via Diffusion Models
While previous works have explored image generation conditioned on scene graphs or layouts, our task is distinctive and important as it involves generating scene graphs themselves unconditionally from noise, enabling efficient and interpretable control for image generation.
Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models
Further, we leverage the findings that different timesteps of the diffusion process cater to different levels of detail in an image.
TAM-VT: Transformation-Aware Multi-scale Video Transformer for Segmentation and Tracking
In this work we propose a novel, clip-based DETR-style encoder-decoder architecture, which focuses on systematically analyzing and addressing aforementioned challenges.
TIBET: Identifying and Evaluating Biases in Text-to-Image Generative Models
Text-to-Image (TTI) generative models have shown great progress in the past few years in terms of their ability to generate complex and high-quality imagery.
Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models
From image-text pairs, large-scale vision-language models (VLMs) learn to implicitly associate image regions with words, which prove effective for tasks like visual question answering.
Uncertainty Guided Adaptive Warping for Robust and Efficient Stereo Matching
A novel Uncertainty Guided Adaptive Correlation (UGAC) module is introduced to robustly adapt the same model for different scenarios.
INVE: Interactive Neural Video Editing
We present Interactive Neural Video Editing (INVE), a real-time video editing solution, which can assist the video editing process by consistently propagating sparse frame edits to the entire video clip.
Implicit and Explicit Commonsense for Multi-sentence Video Captioning
We show that our commonsense knowledge enhanced approach produces significant improvements on this task (up to 57% in METEOR and 8. 5% in CIDEr), as well as the state-of-the-art result on more traditional video captioning in the ActivityNet Captions dataset [29].
MINOTAUR: Multi-task Video Grounding From Multimodal Queries
Video understanding tasks take many forms, from action detection to visual query localization and spatio-temporal grounding of sentences.
Frustratingly Simple but Effective Zero-shot Detection and Segmentation: Analysis and a Strong Baseline
Methods for object detection and segmentation often require abundant instance-level annotations for training, which are time-consuming and expensive to collect.
Self-Supervised Relation Alignment for Scene Graph Generation
In the auxiliary branch, relational input features are partially masked prior to message passing and predicate prediction.
Vocabulary-informed Zero-shot and Open-set Learning
Despite significant progress in object categorization, in recent years, a number of important challenges remain; mainly, the ability to learn from limited labeled data and to recognize object classes within large, potentially open, set of labels.
Omnimatte3D: Associating Objects and Their Effects in Unconstrained Monocular Video
Instead, our method applies recent progress in monocular camera pose and depth estimation to create a full, RGBD video layer for the background, along with a video layer for each foreground object.
DINN360: Deformable Invertible Neural Network for Latitude-Aware 360deg Image Rescaling
To alleviate this issue, we propose the first attempt at 360deg image rescaling, which refers to downscaling a 360deg image to a visually valid low-resolution (LR) counterpart and then upscaling to a high-resolution (HR) 360deg image given the LR variant.
Framework-agnostic Semantically-aware Global Reasoning for Segmentation
Recent advances in pixel-level tasks (e. g. segmentation) illustrate the benefit of of long-range interactions between aggregated region-based representations that can enhance local features.
GraphPNAS: Learning Distribution of Good Neural Architectures via Deep Graph Generative Models
Moreover, our graph generator leads to a learnable probabilistic search method that is more flexible and efficient than the commonly used RNN generator and random search methods.
Make-A-Story: Visual Memory Conditioned Consistent Story Generation
Our experiments for story generation on the MUGEN, the PororoSV and the FlintstonesSV dataset show that our method not only outperforms prior state-of-the-art in generating frames with high visual quality, which are consistent with the story, but also models appropriate correspondences between the characters and the background.
VLC-BERT: Visual Question Answering with Contextualized Commonsense Knowledge
In contrast to previous methods which inject knowledge from static knowledge bases, we investigate the incorporation of contextualized knowledge using Commonsense Transformer (COMET), an existing knowledge model trained on human-curated knowledge bases.
Ranked #8 on
Visual Question Answering (VQA)
on A-OKVQA
(DA VQA Score metric)
Real-Time Monitoring of User Stress, Heart Rate and Heart Rate Variability on Mobile Devices
The user's pulse wave is then used to determine stress (according to the Baevsky Stress Index), heart rate, and heart rate variability.
Iterative Scene Graph Generation
In this work, we propose a novel framework for scene graph generation that addresses this limitation, as well as introduces dynamic conditioning on the image, using message passing in a Markov Random Field.
Generalizable Patch-Based Neural Rendering
Neural rendering has received tremendous attention since the advent of Neural Radiance Fields (NeRF), and has pushed the state-of-the-art on novel-view synthesis considerably.
Self-supervision through Random Segments with Autoregressive Coding (RandSAC)
We illustrate that randomized serialization of the segments significantly improves the performance and results in distribution over spatially-long (across-segments) and -short (within-segment) predictions which are effective for feature learning.
Beyond Simple Meta-Learning: Multi-Purpose Models for Multi-Domain, Active and Continual Few-Shot Learning
The first method, Simple CNAPS, employs a hierarchically regularized Mahalanobis-distance based classifier combined with a state of the art neural adaptive feature extractor to achieve strong performance on Meta-Dataset, mini-ImageNet and tiered-ImageNet benchmarks.
Light Field Neural Rendering
Classical light field rendering for novel view synthesis can accurately reproduce view-dependent effects such as reflection, refraction, and translucency, but requires a dense view sampling of the scene.
TriBERT: Human-centric Audio-visual Representation Learning
In this work, we introduce TriBERT -- a transformer-based architecture, inspired by ViLBERT, which enables contextual feature learning across three modalities: vision, pose, and audio, with the use of flexible co-attention.
Layered Controllable Video Generation
In the first stage, with the rich set of losses and dynamic foreground size prior, we learn how to separate the frame into foreground and background layers and, conditioned on these layers, how to generate the next frame using VQ-VAE generator.
TriBERT: Full-body Human-centric Audio-visual Representation Learning for Visual Sound Separation
In this work, we introduce TriBERT -- a transformer-based architecture, inspired by ViLBERT, which enables contextual feature learning across three modalities: vision, pose, and audio, with the use of flexible co-attention.
Saliency-Guided Image Translation
In this paper, we propose a novel task for saliency-guided image translation, with the goal of image-to-image translation conditioned on the user specified saliency map.
Generative Adversarial Network
Image-to-Image Translation
+1
Referring Transformer: A One-step Approach to Multi-task Visual Grounding
As an important step towards visual reasoning, visual grounding (e. g., phrase localization, referring expression comprehension/segmentation) has been widely explored Previous approaches to referring expression comprehension (REC) or segmentation (RES) either suffer from limited performance, due to a two-stage setup, or require the designing of complex task-specific one-stage architectures.
Segmentation-grounded Scene Graph Generation
Our framework is agnostic to the underlying scene graph generation method and address the lack of segmentation annotations in target scene graph datasets (e. g., Visual Genome) through transfer and multi-task learning from, and with, an auxiliary dataset (e. g., MS COCO).
Weakly-supervised Audio-visual Sound Source Detection and Separation
Learning how to localize and separate individual object sounds in the audio channel of the video is a difficult task.
Energy-Based Learning for Scene Graph Generation
The proposed formulation allows for efficiently incorporating the structure of scene graphs in the output space.
Ranked #4 on
Scene Graph Generation
on Visual Genome
An Improved Attention for Visual Question Answering
We also propose multimodal fusion module to combine both visual and textual information.
Person-in-Context Synthesiswith Compositional Structural Space
To handle the stark difference in input structures, we proposed two separate neural branches to attentively composite the respective (context/person) inputs into shared ``compositional structural space'', which encodes shape, location and appearance information for both context and person structures in a disentangled manner.
Attribute-guided image generation from layout
Also, the generated images from our model have higher resolution, object classification accuracy and consistency, as compared to the previous state-of-the-art.
Unsupervised Video Decomposition using Spatio-temporal Iterative Inference
Unsupervised multi-object scene decomposition is a fast-emerging problem in representation learning.
Discriminative Feature Alignment: Improving Transferability of Unsupervised Domain Adaptation by Gaussian-guided Latent Alignment
To solve this problem, we introduce a Gaussian-guided latent alignment approach to align the latent feature distributions of the two domains under the guidance of the prior distribution.
Ranked #1 on
Domain Adaptation
on SYNSIG-to-GTSRB
UniT: Unified Knowledge Transfer for Any-shot Object Detection and Segmentation
Weakly-supervised approaches draw on image-level labels to build detectors/segmentors, while zero/few-shot methods assume abundant instance-level data for a set of base classes, and none to a few examples for novel classes.
Variational Hyper RNN for Sequence Modeling
In this work, we propose a novel probabilistic sequence model that excels at capturing high variability in time series data, both across sequences and within an individual sequence.
Front2Back: Single View 3D Shape Reconstruction via Front to Back Prediction
Reconstruction of a 3D shape from a single 2D image is a classical computer vision problem, whose difficulty stems from the inherent ambiguity of recovering occluded or only partially observed surfaces.
Improved Few-Shot Visual Classification
Few-shot learning is a fundamental task in computer vision that carries the promise of alleviating the need for exhaustively labeled data.
Ranked #2 on
Few-Shot Image Classification
on Mini-Imagenet 10-way (5-shot)
(using extra training data)
Generating Videos of Zero-Shot Compositions of Actions and Objects
Human activity videos involve rich, varied interactions between people and objects.
OptiBox: Breaking the Limits of Proposals for Visual Grounding
The problem of language grounding has attracted much attention in recent years due to its pivotal role in more general image-lingual high level reasoning tasks (e. g., image captioning, VQA).
DwNet: Dense warp-based network for pose-guided human video generation
In this paper, we focus on human motion transfer - generation of a video depicting a particular subject, observed in a single image, performing a series of motions exemplified by an auxiliary (driving) video.
Watch, Listen and Tell: Multi-modal Weakly Supervised Dense Event Captioning
Multi-modal learning, particularly among imaging and linguistic modalities, has made amazing strides in many high-level fundamental visual understanding problems, ranging from language grounding to dense event captioning.
LayoutVAE: Stochastic Scene Layout Generation From a Label Set
Recently there is an increasing interest in scene generation within the research community.
AttentionRNN: A Structured Spatial Attention Mechanism
Visual attention mechanisms have proven to be integrally important constituent components of many modern deep neural architectures.
A Variational Auto-Encoder Model for Stochastic Point Processes
We propose a novel probabilistic generative model for action sequences.
Neural Sequential Phrase Grounding (SeqGROUND)
We propose an end-to-end approach for phrase grounding in images.
Walking on Thin Air: Environment-Free Physics-based Markerless Motion Capture
To our knowledge, this is the first approach to take physics into account without explicit {\em a priori} knowledge of the environment or body dimensions.
Traversing the Continuous Spectrum of Image Retrieval with Deep Dynamic Models
While the proposed approaches in the literature can be roughly categorized into two main groups: category- and instance-based retrieval, in this work we show that the retrieval task is much richer and more complex.
Image Generation from Layout
The representation of each object is disentangled into a specified/certain part (category) and an unspecified/uncertain part (appearance).
Ranked #2 on
Layout-to-Image Generation
on Visual Genome 64x64
Learning the Compositional Spaces for Generalized Zero-shot Learning
This paper studies the problem of Generalized Zero-shot Learning (G-ZSL), whose goal is to classify instances belonging to both seen and unseen classes at the test time.
Interpretable Spatio-temporal Attention for Video Action Recognition
Inspired by the observation that humans are able to process videos efficiently by only paying attention where and when it is needed, we propose an interpretable and easy plug-in spatial-temporal attention mechanism for video action recognition.
Multi-level Semantic Feature Augmentation for One-shot Learning
In semantic space, we search for related concepts, which are then projected back into the image feature spaces by the decoder portion of the TriNet.
Multilevel Language and Vision Integration for Text-to-Clip Retrieval
To capture the inherent structures present in both text and video, we introduce a multilevel model that integrates vision and language features earlier and more tightly than prior work.
Modular Generative Adversarial Networks
Existing methods for multi-domain image-to-image translation (or generation) attempt to directly map an input image (or a random vector) to an image in one of the output domains.
Probabilistic Video Generation using Holistic Attribute Control
Videos express highly structured spatio-temporal patterns of visual data.
Joint Event Detection and Description in Continuous Video Streams
In order to explicitly model temporal relationships between visual events and their captions in a single video, we also propose a two-level hierarchical captioning module that keeps track of context.
A Neural Multi-sequence Alignment TeCHnique (NeuMATCH)
The alignment of heterogeneous sequential data (video to text) is an important and challenging problem.
Non-parametric Structured Output Networks
End-to-end training methods for models with structured graphical dependencies on top of neural predictions have recently emerged as a principled way of combining these two paradigms.
Recent Advances in Zero-shot Recognition
With the recent renaissance of deep convolution neural networks, encouraging breakthroughs have been achieved on the supervised recognition tasks, where each class has sufficient training data and fully annotated training data.
Visual Reference Resolution using Attention Memory for Visual Dialog
From this memory, the model retrieves the previous attention, taking into account recency, which is most relevant for the current question, in order to resolve potentially ambiguous references.
Ranked #13 on
Visual Dialog
on VisDial v0.9 val
(R@1 metric)
Action Classification and Highlighting in Videos
Inspired by recent advances in neural machine translation, that jointly align and translate using encoder-decoder networks equipped with attention, we propose an attentionbased LSTM model for human activity recognition.
Weakly-supervised Visual Grounding of Phrases with Linguistic Structures
We propose a weakly-supervised approach that takes image-sentence pairs as input and learns to visually ground (i. e., localize) arbitrary linguistic phrases, in the form of spatial attention masks.
Weakly-Supervised Spatial Context Networks
In particular, we propose spatial context networks that learn to predict a representation of one image patch from another image patch, within the same image, conditioned on their real-valued relative spatial offset.
Semi-Latent GAN: Learning to generate and modify facial images from attributes
Generating and manipulating human facial images using high-level attributal controls are important and interesting problems.
Learning to Generate Posters of Scientific Papers by Probabilistic Graphical Models
In order to bridge the gap between panel attributes and the composition within each panel, we also propose a recursive page splitting algorithm to generate the panel layout for a poster.
Learning Language-Visual Embedding for Movie Understanding with Natural-Language
We evaluate our models on large scale LSMDC16 movie dataset for two tasks: 1) Standard Ranking for video annotation and retrieval 2) Our proposed movie multiple-choice test.
Ranked #39 on
Video Retrieval
on MSR-VTT
Harnessing Object and Scene Semantics for Large-Scale Video Understanding
Large-scale action recognition and video categorization are important problems in computer vision.
Learning Activity Progression in LSTMs for Activity Detection and Early Detection
In this work we improve training of temporal deep models to better learn activity progression for activity detection and early detection.
Semi-supervised Vocabulary-informed Learning
Despite significant progress in object categorization, in recent years, a number of important challenges remain, mainly, ability to learn from limited labeled data and ability to recognize object classes within large, potentially open, set of labels.
Learning to Generate Posters of Scientific Papers
Then, given inferred layout and attributes, composition of graphical elements within each panel is synthesized.
Do Less and Achieve More: Training CNNs for Action Recognition Utilizing Action Images from the Web
In contrast, collecting action images from the Web is much easier and training on images requires much less computation.
Ranked #14 on
Action Recognition
on ActivityNet
(using extra training data)
Storyline Representation of Egocentric Videos With an Applications to Story-Based Search
To address this, we propose a storyline representation that expresses an egocentric video as a set of jointly inferred, through MRF inference, story elements comprising of actors, locations, supporting objects and events, depicted on a timeline.
Robust Classification by Pre-conditioned LASSO and Transductive Diffusion Component Analysis
Collecting datasets in this way, however, requires robust and efficient ways for detecting and excluding outliers that are common and prevalent.
Heterogeneous Knowledge Transfer in Video Emotion Recognition, Attribution and Summarization
Emotion is a key element in user-generated videos.
Ranked #5 on
Video Emotion Recognition
on Ekman6
Learning from Synthetic Data Using a Stacked Multichannel Autoencoder
In this paper, we investigate and formalize a general framework-Stacked Multichannel Autoencoder (SMCAE) that enables bridging the synthetic gap and learning from synthetic data more efficiently.
Joint Photo Stream and Blog Post Summarization and Exploration
We alternate between solving the two coupled latent SVM problems, by first fixing the summarization and solving for the alignment from blog images to photo streams and vice versa.
Space-Time Tree Ensemble for Action Recognition
Using the action vocabulary we then utilize tree mining with subsequent tree clustering and ranking to select a compact set of highly discriminative tree patterns.
Expanding Object Detector's Horizon: Incremental Learning Framework for Object Detection in Videos
By incrementally detecting object instances in video and adding confident detections into the model, we are able to dynamically adjust the complexity of the detector over time by instantiating new prototypes to span all domains the model has seen.
Ranking and Retrieval of Image Sequences From Multiple Paragraph Queries
While most previous work has dealt with the relations between a natural language sentence and an image or a video, our work extends to the relations between paragraphs and image sequences.
Learning Classifiers from Synthetic Data Using a Multichannel Autoencoder
Experimental results on two datasets validate the efficiency of our MCAE model and our methodology of generating synthetic data.
Hierarchical Maximum-Margin Clustering
We present a hierarchical maximum-margin clustering method for unsupervised data analysis.
A Unified Semantic Embedding: Relating Taxonomies and Attributes
We propose a method that learns a discriminative yet semantic space for object categorization, where we also embed auxiliary semantic entities such as supercategories and attributes.
Action is in the Eye of the Beholder: Eye-gaze Driven Model for Spatio-Temporal Action Localization
We propose a new weakly-supervised structured learning approach for recognition and spatio-temporal localization of actions in video.
Poselet Key-Framing: A Model for Human Activity Recognition
We show classification performance that is competitive with the state of the art on the benchmark UT-Interaction dataset and illustrate that our model outperforms prior methods in an on-line streaming setting.
Ranked #3 on
Human Interaction Recognition
on UT
High-Dimensional Feature Selection by Feature-Wise Kernelized Lasso
We first show that, with particular choices of kernel functions, non-redundant features with strong statistical dependence on output values can be found in terms of kernel-based independence measures.
Facial Expression Transfer with Input-Output Temporal Restricted Boltzmann Machines
We present a type of Temporal Restricted Boltzmann Machine that defines a probability distribution over an output sequence conditional on an input sequence.
