Search Results for author:
Found 55 papers, 36 papers with code
Just Shift It: Test-Time Prototype Shifting for Zero-Shot Generalization with Vision-Language Models
Advancements in vision-language models (VLMs) have propelled the field of computer vision, particularly in the zero-shot learning setting.
VideoAgent: Long-form Video Understanding with Large Language Model as Agent
Long-form video understanding represents a significant challenge within computer vision, demanding a model capable of reasoning over long multi-modal sequences.
Ranked #1 on
Zero-Shot Video Question Answer
on NExT-QA
Editing Conceptual Knowledge for Large Language Models
Recently, there has been a growing interest in knowledge editing for Large Language Models (LLMs).
DGL: Dynamic Global-Local Prompt Tuning for Text-Video Retrieval
(2) Equipping the visual and text encoder with separated prompts failed to mitigate the visual-text modality gap.
Describing Differences in Image Sets with Natural Language
To aid in this discovery process, we explore the task of automatically describing the differences between two $\textbf{sets}$ of images, which we term Set Difference Captioning.
Exploring Large Language Models for Human Mobility Prediction under Public Events
Accurate human mobility prediction for public events is thus crucial for event planning as well as traffic or crowd management.
IcoCap: Improving Video Captioning by Compounding Images
Video captioning is a more challenging task compared to image captioning, primarily due to differences in content density.
Ranked #5 on
Video Captioning
on VATEX
(using extra training data)
Editing Personality for Large Language Models
This task seeks to adjust the models' responses to opinion-related questions on specified topics since an individual's personality often manifests in the form of their expressed opinions, thereby showcasing different personality traits.
DiverseMotion: Towards Diverse Human Motion Generation via Discrete Diffusion
We present DiverseMotion, a new approach for synthesizing high-quality human motions conditioned on textual descriptions while preserving motion diversity. Despite the recent significant process in text-based human motion generation, existing methods often prioritize fitting training motions at the expense of action diversity.
Ranked #2 on
Motion Synthesis
on HumanML3D
(using extra training data)
EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Models
Large Language Models (LLMs) usually suffer from knowledge cutoff or fallacy issues, which means they are unaware of unseen events or generate text with incorrect facts owing to outdated/noisy data.
Bird's-Eye-View Scene Graph for Vision-Language Navigation
Vision-language navigation (VLN), which entails an agent to navigate 3D environments following human instructions, has shown great advances.
Methods for Acquiring and Incorporating Knowledge into Stock Price Prediction: A Survey
Predicting stock prices presents a challenging research problem due to the inherent volatility and non-linear nature of the stock market.
JOTR: 3D Joint Contrastive Learning with Transformers for Occluded Human Mesh Recovery
Our method includes an encoder-decoder transformer architecture to fuse 2D and 3D representations for achieving 2D$\&$3D aligned results in a coarse-to-fine manner and a novel 3D joint contrastive learning approach for adding explicitly global supervision for the 3D feature space.
Clustering based Point Cloud Representation Learning for 3D Analysis
The mined patterns are, in turn, used to repaint the embedding space, so as to respect the underlying distribution of the entire training dataset and improve the robustness to the variations.
Kefa: A Knowledge Enhanced and Fine-grained Aligned Speaker for Navigation Instruction Generation
We introduce a novel speaker model \textsc{Kefa} for navigation instruction generation.
Action Sensitivity Learning for the Ego4D Episodic Memory Challenge 2023
This report presents ReLER submission to two tracks in the Ego4D Episodic Memory Benchmark in CVPR 2023, including Natural Language Queries and Moment Queries.
Ranked #1 on
Moment Queries
on Ego4D
Relieving Triplet Ambiguity: Consensus Network for Language-Guided Image Retrieval
We propose a novel Consensus Network (Css-Net) that self-adaptively learns from noisy triplets to minimize the negative effects of triplet ambiguity.
Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models
Given a single test sample, the VLM is forced to maximize the CLIP reward between the input and sampled results from the VLM output distribution.
Whitening-based Contrastive Learning of Sentence Embeddings
Consequently, using multiple positive samples with enhanced diversity further improves contrastive learning due to better alignment.
Action Sensitivity Learning for Temporal Action Localization
Temporal action localization (TAL), which involves recognizing and locating action instances, is a challenging task in video understanding.
Ranked #9 on
Temporal Action Localization
on THUMOS’14
CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Model
With such merits, we transform CLIP into a scene text reader and introduce CLIP4STR, a simple yet effective STR method built upon image and text encoders of CLIP.
Ranked #1 on
Scene Text Recognition
on WOST
(using extra training data)
Gloss-Free End-to-End Sign Language Translation
In this paper, we tackle the problem of sign language translation (SLT) without gloss annotations.
LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunities
We engage in experiments across eight diverse datasets, focusing on four representative tasks encompassing entity and relation extraction, event extraction, link prediction, and question-answering, thereby thoroughly exploring LLMs' performance in the domain of construction and inference.
Continual Multimodal Knowledge Graph Construction
Multimodal Knowledge Graph Construction (MKGC) involves creating structured representations of entities and relations using multiple modalities, such as text and images.
How to Unleash the Power of Large Language Models for Few-shot Relation Extraction?
Scaling language models have revolutionized widespread NLP tasks, yet little comprehensively explored few-shot relation extraction with large language models.
Global-to-Local Modeling for Video-based 3D Human Pose and Shape Estimation
However, using a single kind of modeling structure is difficult to balance the learning of short-term and long-term temporal correlations, and may bias the network to one of them, leading to undesirable predictions like global location shift, temporal inconsistency, and insufficient local details.
Ranked #47 on
3D Human Pose Estimation
on 3DPW
Lana: A Language-Capable Navigator for Instruction Following and Generation
Recently, visual-language navigation (VLN) -- entailing robot agents to follow navigation instructions -- has shown great advance.
MAAL: Multimodality-Aware Autoencoder-Based Affordance Learning for 3D Articulated Objects
Experimental results and visualizations, based on a large-scale dataset PartNet-Mobility, show the effectiveness of MAAL in learning multi-modal data and solving the 3D articulated object affordance problem.
Adversarially Masking Synthetic To Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation
With the help of adversarial training, the masking module can learn to generate source masks to mimic the pattern of irregular target noise, thereby narrowing the domain gap.
Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Models
In this paper, we propose a novel framework called BIKE, which utilizes the cross-modal bridge to explore bidirectional knowledge: i) We introduce the Video Attribute Association mechanism, which leverages the Video-to-Text knowledge to generate textual auxiliary attributes for complementing video recognition.
Ranked #1 on
Zero-Shot Action Recognition
on ActivityNet
EASpace: Enhanced Action Space for Policy Transfer
In this paper, a novel algorithm named EASpace (Enhanced Action Space) is proposed, which formulates macro actions in an alternative form to accelerate the learning process using multiple available sub-optimal expert policies.
Penalizing the Hard Example But Not Too Much: A Strong Baseline for Fine-Grained Visual Classification
Second, we instantiate the loss function and provide a strong baseline for FGVC, where the performance of a naive backbone can be boosted and be comparable with recent methods.
Ranked #27 on
Fine-Grained Image Classification
on CUB-200-2011
Fine-Grained Image Classification
Fine-Grained Visual Recognition
ReLER@ZJU Submission to the Ego4D Moment Queries Challenge 2022
Moreover, in order to better capture the long-term temporal dependencies in the long videos, we propose a segment-level recurrence mechanism.
LambdaKG: A Library for Pre-trained Language Model-Based Knowledge Graph Embeddings
Knowledge Graphs (KGs) often have two characteristics: heterogeneous graph structure and text-rich entity/relation information.
Slimmable Networks for Contrastive Self-supervised Learning
In this work, we present a one-stage solution to obtain pre-trained small models without the need for extra teachers, namely, slimmable networks for contrastive self-supervised learning (\emph{SlimCLR}).
Jointly Harnessing Prior Structures and Temporal Consistency for Sign Language Video Generation
We optimize the two losses and keypoint detector network in an end-to-end manner.
ReLER@ZJU-Alibaba Submission to the Ego4D Natural Language Queries Challenge 2022
In this report, we present the ReLER@ZJU-Alibaba submission to the Ego4D Natural Language Queries (NLQ) Challenge in CVPR 2022.
Ranked #3 on
Natural Language Queries
on Ego4D
CenterCLIP: Token Clustering for Efficient Text-Video Retrieval
In this paper, to reduce the number of redundant video tokens, we design a multi-segment token clustering algorithm to find the most representative tokens and drop the non-essential ones.
Ranked #11 on
Video Retrieval
on MSVD
(using extra training data)
Scalable Video Object Segmentation with Identification Mechanism
This paper delves into the challenges of achieving scalable and effective multi-object modeling for semi-supervised Video Object Segmentation (VOS).
Multi-robot Cooperative Pursuit via Potential Field-Enhanced Reinforcement Learning
It is shown by numerical simulations that the proposed hybrid design outperforms the pursuit policies either learned from vanilla reinforcement learning or designed by the potential field method.
Action Keypoint Network for Efficient Video Recognition
An effective approach is to select informative content from the holistic video, yielding a popular family of dynamic video recognition methods.
Ranked #42 on
Action Recognition
on Something-Something V1
Reasoning Through Memorization: Nearest Neighbor Knowledge Graph Embeddings
Previous knowledge graph embedding approaches usually map entities to representations and utilize score functions to predict the target entities, yet they typically struggle to reason rare or emerging unseen entities.
Ranked #1 on
Link Prediction
on FB15k-237-ind
Large-Scale Video Panoptic Segmentation in the Wild: A Benchmark
In contrast, our large-scale VIdeo Panoptic Segmentation in the Wild (VIPSeg) dataset provides 3, 536 videos and 84, 750 frames with pixel-level panoptic annotations, covering a wide range of real-world scenarios and categories.
A Simple Episodic Linear Probe Improves Visual Recognition in the Wild
In this paper, we propose an episodic linear probing (ELP) classifier to reflect the generalization of visual representations in an online manner.
Ranked #13 on
Fine-Grained Image Classification
on CUB-200-2011
Self-supervised Point Cloud Representation Learning via Separating Mixed Shapes
Albeit simple, the pre-trained encoder can capture the key points of an unseen point cloud and surpasses the encoder trained from scratch on downstream tasks.
Ranked #43 on
3D Part Segmentation
on ShapeNet-Part
PR-RRN: Pairwise-Regularized Residual-Recursive Networks for Non-rigid Structure-from-Motion
As NRSfM is a highly under-constrained problem, we propose two new pairwise regularization to further regularize the reconstruction.
Less is More: Sparse Sampling for Dense Reaction Predictions
Obtaining viewer responses from videos can be useful for creators and streaming platforms to analyze the video performance and improve the future user experience.
Connecting Language and Vision for Natural Language-Based Vehicle Retrieval
In this paper, we apply one new modality, i. e., the language description, to search the vehicle of interest and explore the potential of this task in the real-world scenario.
T2VLAD: Global-Local Sequence Alignment for Text-Video Retrieval
Moreover, a global alignment method is proposed to provide a global cross-modal measurement that is complementary to the local perspective.
Learning to Anticipate Egocentric Actions by Imagination
We further improve ImagineRNN by residual anticipation, i. e., changing its target to predicting the feature difference of adjacent frames instead of the frame content.
Interactive Prototype Learning for Egocentric Action Recognition
To avoid these additional costs, we propose an end-to-end Interactive Prototype Learning (IPL) framework to learn better active object representations by leveraging the motion cues from the actor.
Variable-Viewpoint Representations for 3D Object Recognition
For the problem of 3D object recognition, researchers using deep learning methods have developed several very different input representations, including "multi-view" snapshots taken from discrete viewpoints around an object, as well as "spherical" representations consisting of a dense map of essentially ray-traced samples of the object from all directions.
Symbiotic Attention with Privileged Information for Egocentric Action Recognition
Due to the large action vocabulary in egocentric video datasets, recent studies usually utilize a two-branch structure for action recognition, ie, one branch for verb classification and the other branch for noun classification.
Ranked #4 on
Egocentric Activity Recognition
on EGTEA
Baidu-UTS Submission to the EPIC-Kitchens Action Recognition Challenge 2019
In this report, we present the Baidu-UTS submission to the EPIC-Kitchens Action Recognition Challenge in CVPR 2019.
The Toybox Dataset of Egocentric Visual Object Transformations
In object recognition research, many commonly used datasets (e. g., ImageNet and similar) contain relatively sparse distributions of object instances and views, e. g., one might see a thousand different pictures of a thousand different giraffes, mostly taken from a few conventionally photographed angles.
