Search Results for author:
Found 48 papers, 24 papers with code
Learn to Recover Visible Color for Video Surveillance in a Day
In silicon sensors, the interference between visible and near-infrared (NIR) signals is a crucial problem.
Vinci: A Real-time Embodied Smart Assistant based on Egocentric Vision-Language Model
We introduce Vinci, a real-time embodied smart assistant built upon an egocentric vision-language model.
CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understanding
However, because of the inherent limitation of MCQ-based evaluation and the increasing reasoning ability of MLLMs, models can give the current answer purely by combining short video understanding with elimination, without genuinely understanding the video content.
Imitation Learning with Limited Actions via Diffusion Planners and Deep Koopman Controllers
Recent advances in diffusion-based robot policies have demonstrated significant potential in imitating multi-modal behaviors.
Pre-Training for 3D Hand Pose Estimation with Contrastive Learning on Large-Scale Hand Images in the Wild
Our experiments demonstrate that our method outperforms conventional contrastive learning approaches that produce positive pairs sorely from a single image with data augmentation.
ActionVOS: Actions as Prompts for Video Object Segmentation
Experimental results on VISOR dataset reveal that ActionVOS significantly reduces the mis-segmentation of inactive objects, confirming that actions help the ActionVOS model understand objects' involvement.
Masked Video and Body-worn IMU Autoencoder for Egocentric Action Recognition
Compared with visual signals, Inertial Measurement Units (IMUs) placed on human limbs can capture accurate motion signals while being robust to lighting variation and occlusion.
EgoVideo: Exploring Egocentric Foundation Model and Downstream Adaptation
In this report, we present our solutions to the EgoVis Challenges in CVPR 2024, including five tracks in the Ego4D challenge and three tracks in the EPIC-Kitchens challenge.
Ranked #1 on
Long Term Action Anticipation
on Ego4D
(using extra training data)
TextCenGen: Attention-Guided Text-Centric Background Adaptation for Text-to-Image Generation
These results demonstrate the efficacy of TextCenGen in creating more harmonious and integrated text-image compositions.
EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World
Along with the videos we record high-quality gaze data and provide detailed multimodal annotations, formulating a playground for modeling the human ability to bridge asynchronous procedural actions from different viewpoints.
Ranked #1 on
Action Anticipation
on EgoExoLearn
(using extra training data)
InternVideo2: Scaling Foundation Models for Multimodal Video Understanding
We introduce InternVideo2, a new family of video foundation models (ViFM) that achieve the state-of-the-art results in video recognition, video-text tasks, and video-centric dialogue.
Ranked #1 on
Action Classification
on MIT
Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understanding
We categorize Mamba into four roles for modeling videos, deriving a Video Mamba Suite composed of 14 models/modules, and evaluating them on 12 video understanding tasks.
Ranked #2 on
Temporal Action Localization
on FineAction
FineBio: A Fine-Grained Video Dataset of Biological Experiments with Hierarchical Annotation
The dataset consists of multi-view videos of 32 participants performing mock biological experiments with a total duration of 14. 5 hours.
Retrieval-Augmented Egocentric Video Captioning
In this paper, (1) we develop EgoInstructor, a retrieval-augmented multimodal captioning model that automatically retrieves semantically relevant third-person instructional videos to enhance the video captioning of egocentric videos.
MoVQA: A Benchmark of Versatile Question-Answering for Long-Form Movie Understanding
While several long-form VideoQA datasets have been introduced, the length of both videos used to curate questions and sub-clips of clues leveraged to answer those questions have not yet reached the criteria for genuine long-form video understanding.
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge.
Pretraining Language Models with Text-Attributed Heterogeneous Graphs
In many real-world scenarios (e. g., academic networks, social platforms), different types of entities are not only associated with texts but also connected by various relationships, which can be abstracted as Text-Attributed Heterogeneous Graphs (TAHGs).
Proposal-based Temporal Action Localization with Point-level Supervision
Point-level supervised temporal action localization (PTAL) aims at recognizing and localizing actions in untrimmed videos where only a single point (frame) within every action instance is annotated in training data.
Memory-and-Anticipation Transformer for Online Action Understanding
Based on this idea, we present Memory-and-Anticipation Transformer (MAT), a memory-anticipation-based approach, to address the online action detection and anticipation tasks.
Ranked #1 on
Action Detection
on THUMOS' 14
VideoLLM: Modeling Video Sequence with Large Language Models
Building upon this insight, we propose a novel framework called VideoLLM that leverages the sequence reasoning capabilities of pre-trained LLMs from natural language processing (NLP) for video sequence understanding.
Structural Multiplane Image: Bridging Neural View Synthesis and 3D Reconstruction
The Multiplane Image (MPI), containing a set of fronto-parallel RGBA layers, is an effective and efficient representation for view synthesis from sparse inputs.
Fine-grained Affordance Annotation for Egocentric Hand-Object Interaction Videos
Object affordance is an important concept in hand-object interaction, providing information on action possibilities based on human motor capacity and objects' physical property thus benefiting tasks such as action anticipation and robot imitation learning.
Weakly Supervised Temporal Sentence Grounding With Uncertainty-Guided Self-Training
The task of weakly supervised temporal sentence grounding aims at finding the corresponding temporal moments of a language description in the video, given video-language correspondence only at video-level.
InternVideo-Ego4D: A Pack of Champion Solutions to Ego4D Challenges
In this report, we present our champion solutions to five tracks at Ego4D challenge.
Ranked #1 on
State Change Object Detection
on Ego4D
Compound Prototype Matching for Few-shot Action Recognition
Each global prototype is encouraged to summarize a specific aspect from the entire video, for example, the start/evolution of the action.
Precise Affordance Annotation for Egocentric Action Video Datasets
Object affordance is an important concept in human-object interaction, providing information on action possibilities based on human motor capacity and objects' physical property thus benefiting tasks such as action anticipation and robot imitation learning.
CLRNet: Cross Layer Refinement Network for Lane Detection
In this way, we can exploit more contextual information to detect lanes while leveraging local detailed lane features to improve localization accuracy.
Ranked #1 on
Lane Detection
on LLAMAS
Interact Before Align: Leveraging Cross-Modal Knowledge for Domain Adaptive Action Recognition
Different from previous works, we find that the cross-domain alignment can be more effectively done by using cross-modal interaction first.
Stacked Temporal Attention: Improving First-person Action Recognition by Emphasizing Discriminative Clips
Previous works explored to address this problem by applying temporal attention but failed to consider the global context of the full video, which is critical for determining the relatively significant parts.
Leveraging Human Selective Attention for Medical Image Analysis with Limited Training Data
Then we design a novel Auxiliary Attention Block (AAB) to allow information from SAN to be utilized by the backbone encoder to focus on selective areas.
Ego4D: Around the World in 3,000 Hours of Egocentric Video
We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite.
Spatio-Temporal Perturbations for Video Attribution
The attribution method provides a direction for interpreting opaque neural networks in a visual way by identifying and visualizing the input regions/pixels that dominate the output of a network.
FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learning
In this paper, we propose a talking face generation method that takes an audio signal as input and a short target video clip as reference, and synthesizes a photo-realistic video of the target face with natural lip motions, head poses, and eye blinks that are in-sync with the input audio signal.
EPIC-KITCHENS-100 Unsupervised Domain Adaptation Challenge for Action Recognition 2021: Team M3EM Technical Report
In this report, we describe the technical details of our submission to the 2021 EPIC-KITCHENS-100 Unsupervised Domain Adaptation Challenge for Action Recognition.
Goal-Oriented Gaze Estimation for Zero-Shot Learning
Therefore, we introduce a novel goal-oriented gaze estimation module (GEM) to improve the discriminative attribute localization based on the class-level attributes for ZSL.
Commonsense Knowledge Aware Concept Selection For Diverse and Informative Visual Storytelling
To enrich the candidate concepts, a commonsense knowledge graph is created for each image sequence from which the concept candidates are proposed.
Ranked #19 on
Visual Storytelling
on VIST
Adversarial Robustness of Stabilized NeuralODEs Might be from Obfuscated Gradients
Even replacing only the first layer of a ResNet by such a ODE block can exhibit further improvement in robustness, e. g., under PGD-20 ($\ell_\infty=0. 031$) attack on CIFAR-10 dataset, it achieves 91. 57\% and natural accuracy and 62. 35\% robust accuracy, while a counterpart architecture of ResNet trained with TRADES achieves natural and robust accuracy 76. 29\% and 45. 24\%, respectively.
Improving Action Segmentation via Graph-Based Temporal Reasoning
In this paper, we propose a network module called Graph-based Temporal Reasoning Module (GTRM) that can be built on top of existing action segmentation models to learn the relation of multiple action segments in various time spans.
Ranked #29 on
Action Segmentation
on Breakfast
Towards Visually Explaining Video Understanding Networks with Perturbation
''Making black box models explainable'' is a vital problem that accompanies the development of deep learning networks.
Discovery of Bias and Strategic Behavior in Crowdsourced Performance Assessment
With the industry trend of shifting from a traditional hierarchical approach to flatter management structure, crowdsourced performance assessment gained mainstream popularity.
ON BREIMAN’S DILEMMA IN NEURAL NETWORKS: SUCCESS AND FAILURE OF NORMALIZED MARGINS
A belief persists long in machine learning that enlargement of margins over training data accounts for the resistance of models to overfitting by increasing the robustness.
An Evaluation of Transfer Learning for Classifying Sales Engagement Emails at Large Scale
Our results show that fine-tuning of the BERT model outperforms with as few as 300 labeled samples, but underperforms with fewer than 300 labeled samples, relative to all the feature-based approaches using different embeddings.
Manipulation-skill Assessment from Videos with Spatial Attention Network
Recent advances in computer vision have made it possible to automatically assess from videos the manipulation skills of humans in performing a task, which breeds many important applications in domains such as health rehabilitation and manufacturing.
Mutual Context Network for Jointly Estimating Egocentric Gaze and Actions
In this work, we address two coupled tasks of gaze prediction and action recognition in egocentric videos by exploring their mutual context.
Differentiable Fine-grained Quantization for Deep Neural Network Compression
Thus judiciously selecting different precision for different layers/structures can potentially produce more efficient models compared to traditional quantization methods by striking a better balance between accuracy and compression rate.
Semantic Aware Attention Based Deep Object Co-segmentation
Object co-segmentation is the task of segmenting the same objects from multiple images.
Rethinking Breiman's Dilemma in Neural Networks: Phase Transitions of Margin Dynamics
In this paper, we revisit Breiman's dilemma in deep neural networks with recently proposed spectrally normalized margins, from a novel perspective based on phase transitions of normalized margin distributions in training dynamics.
Predicting Gaze in Egocentric Video by Learning Task-dependent Attention Transition
We present a new computational model for gaze prediction in egocentric videos by exploring patterns in temporal shift of gaze fixations (attention transition) that are dependent on egocentric manipulation tasks.
