Search Results for author:
Found 73 papers, 46 papers with code
SurgiSAM2: Fine-tuning a foundational model for surgical video anatomy segmentation and detection
Conclusion: SAM 2 achieves remarkable zero-shot and fine-tuned performance for surgical scene segmentation, surpassing prior SOTA models across several organ classes of diverse datasets.
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
We introduce our first-generation reasoning models, DeepSeek-R1-Zero and DeepSeek-R1.
Ranked #1 on
Multi-task Language Understanding
on MMLU
(using extra training data)
BIOMEDICA: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literature
The development of vision-language models (VLMs) is driven by large-scale and diverse multimodal datasets.
Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation
The rapid development of vision language models (VLMs) demands rigorous and reliable evaluation.
DeepSeek-V3 Technical Report
We present DeepSeek-V3, a strong Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token.
Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Acceleration
In this work, we examine the popular acceleration approach of early pruning of visual tokens inside the language model and find that its strong performance across many tasks is not due to an exceptional ability to compress visual information, but rather the benchmarks' limited ability to assess fine-grained visual capabilities.
Targeted Learning for Variable Importance
Variable importance is one of the most widely used measures for interpreting machine learning with significant interest from both statistics and machine learning communities.
Zero-shot Action Localization via the Confidence of Large Vision-Language Models
Precise action localization in untrimmed video is vital for fields such as professional sports and minimally invasive surgery, where the delineation of particular motions in recordings can dramatically enhance analysis.
Innovative Thinking, Infinite Humor: Humor Research of Large Language Models through Structured Thought Leaps
Humor is a culturally nuanced aspect of human language that presents challenges for understanding and generation, requiring participants to possess good creativity and strong associative thinking.
RuleAlign: Making Large Language Models Better Physicians with Diagnostic Rule Alignment
Large Language Models (LLMs) like GPT-4, MedPaLM-2, and Med-Gemini achieve performance competitively with human experts across various medical benchmarks.
MetaTool: Facilitating Large Language Models to Master Tools with Meta-task Augmentation
Training on solution paths is also hindered by the high cost of expert annotations and generalizing to new tools.
Video-STaR: Self-Training Enables Video Instruction Tuning with Any Supervision
The performance of Large Vision Language Models (LVLMs) is dependent on the size and quality of their training datasets.
Why are Visually-Grounded Language Models Bad at Image Classification?
Image classification is one of the most fundamental capabilities of machine vision intelligence.
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
MLA guarantees efficient inference through significantly compressing the Key-Value (KV) cache into a latent vector, while DeepSeekMoE enables training strong models at an economical cost through sparse computation.
Interpretable3D: An Ad-Hoc Interpretable Classifier for 3D Point Clouds
We evaluate the performance of Interpretable3D on four popular point cloud models: DGCNN, PointNet2, PointMLP, and PointNeXt.
Just Shift It: Test-Time Prototype Shifting for Zero-Shot Generalization with Vision-Language Models
Advancements in vision-language models (VLMs) have propelled the field of computer vision, particularly in the zero-shot learning setting.
VideoAgent: Long-form Video Understanding with Large Language Model as Agent
Long-form video understanding represents a significant challenge within computer vision, demanding a model capable of reasoning over long multi-modal sequences.
Ranked #6 on
Zero-Shot Video Question Answer
on NExT-QA
Editing Conceptual Knowledge for Large Language Models
Recently, there has been a growing interest in knowledge editing for Large Language Models (LLMs).
DGL: Dynamic Global-Local Prompt Tuning for Text-Video Retrieval
(2) Equipping the visual and text encoder with separated prompts failed to mitigate the visual-text modality gap.
An Interactive Navigation Method with Effect-oriented Affordance
Thus Interactive Navigation (InterNav) is introduced to navigate to the objects in more realistic "messy" environments according to the object interaction.
Imagine Before Go: Self-Supervised Generative Map for Object Goal Navigation
The Object Goal navigation (ObjectNav) task requires the agent to navigate to a specified target in an unseen environment.
A Category Agnostic Model for Visual Rearrangment
This paper presents a novel category agnostic model for visual rearrangement task which can help an embodied agent to physically recover the shuffled scene configuration without any category concepts to the goal configuration.
Describing Differences in Image Sets with Natural Language
To aid in this discovery process, we explore the task of automatically describing the differences between two $\textbf{sets}$ of images, which we term Set Difference Captioning.
Exploring Large Language Models for Human Mobility Prediction under Public Events
Accurate human mobility prediction for public events is thus crucial for event planning as well as traffic or crowd management.
IcoCap: Improving Video Captioning by Compounding Images
Video captioning is a more challenging task compared to image captioning, primarily due to differences in content density.
Ranked #5 on
Video Captioning
on VATEX
(using extra training data)
Editing Personality for Large Language Models
This task seeks to adjust the models' responses to opinion-related questions on specified topics since an individual's personality often manifests in the form of their expressed opinions, thereby showcasing different personality traits.
DiverseMotion: Towards Diverse Human Motion Generation via Discrete Diffusion
We present DiverseMotion, a new approach for synthesizing high-quality human motions conditioned on textual descriptions while preserving motion diversity. Despite the recent significant process in text-based human motion generation, existing methods often prioritize fitting training motions at the expense of action diversity.
Ranked #11 on
Motion Synthesis
on HumanML3D
(using extra training data)
EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Models
Large Language Models (LLMs) usually suffer from knowledge cutoff or fallacy issues, which means they are unaware of unseen events or generate text with incorrect facts owing to outdated/noisy data.
Methods for Acquiring and Incorporating Knowledge into Stock Price Prediction: A Survey
Predicting stock prices presents a challenging research problem due to the inherent volatility and non-linear nature of the stock market.
Bird's-Eye-View Scene Graph for Vision-Language Navigation
Vision-language navigation (VLN), which entails an agent to navigate 3D environments following human instructions, has shown great advances.
JOTR: 3D Joint Contrastive Learning with Transformers for Occluded Human Mesh Recovery
Our method includes an encoder-decoder transformer architecture to fuse 2D and 3D representations for achieving 2D$\&$3D aligned results in a coarse-to-fine manner and a novel 3D joint contrastive learning approach for adding explicitly global supervision for the 3D feature space.
Clustering based Point Cloud Representation Learning for 3D Analysis
The mined patterns are, in turn, used to repaint the embedding space, so as to respect the underlying distribution of the entire training dataset and improve the robustness to the variations.
Kefa: A Knowledge Enhanced and Fine-grained Aligned Speaker for Navigation Instruction Generation
We introduce a novel speaker model \textsc{Kefa} for navigation instruction generation.
Action Sensitivity Learning for the Ego4D Episodic Memory Challenge 2023
This report presents ReLER submission to two tracks in the Ego4D Episodic Memory Benchmark in CVPR 2023, including Natural Language Queries and Moment Queries.
Ranked #2 on
Moment Queries
on Ego4D
Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models
Given a single test sample, the VLM is forced to maximize the CLIP reward between the input and sampled results from the VLM output distribution.
Whitening-based Contrastive Learning of Sentence Embeddings
Consequently, using multiple positive samples with enhanced diversity further improves contrastive learning due to better alignment.
Action Sensitivity Learning for Temporal Action Localization
Temporal action localization (TAL), which involves recognizing and locating action instances, is a challenging task in video understanding.
Ranked #10 on
Temporal Action Localization
on THUMOS’14
LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunities
We engage in experiments across eight diverse datasets, focusing on four representative tasks encompassing entity and relation extraction, event extraction, link prediction, and question-answering, thereby thoroughly exploring LLMs' performance in the domain of construction and inference.
Gloss-Free End-to-End Sign Language Translation
In this paper, we tackle the problem of sign language translation (SLT) without gloss annotations.
Ranked #3 on
Gloss-free Sign Language Translation
on OpenASL
Gloss-free Sign Language Translation
Sign Language Translation
+1
Continual Multimodal Knowledge Graph Construction
Current Multimodal Knowledge Graph Construction (MKGC) models struggle with the real-world dynamism of continuously emerging entities and relations, often succumbing to catastrophic forgetting-loss of previously acquired knowledge.
How to Unleash the Power of Large Language Models for Few-shot Relation Extraction?
Scaling language models have revolutionized widespread NLP tasks, yet little comprehensively explored few-shot relation extraction with large language models.
Global-to-Local Modeling for Video-based 3D Human Pose and Shape Estimation
However, using a single kind of modeling structure is difficult to balance the learning of short-term and long-term temporal correlations, and may bias the network to one of them, leading to undesirable predictions like global location shift, temporal inconsistency, and insufficient local details.
Ranked #56 on
3D Human Pose Estimation
on 3DPW
Lana: A Language-Capable Navigator for Instruction Following and Generation
Recently, visual-language navigation (VLN) -- entailing robot agents to follow navigation instructions -- has shown great advance.
Adversarially Masking Synthetic To Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation
With the help of adversarial training, the masking module can learn to generate source masks to mimic the pattern of irregular target noise, thereby narrowing the domain gap.
MAAL: Multimodality-Aware Autoencoder-Based Affordance Learning for 3D Articulated Objects
Experimental results and visualizations, based on a large-scale dataset PartNet-Mobility, show the effectiveness of MAAL in learning multi-modal data and solving the 3D articulated object affordance problem.
Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Models
In this paper, we propose a novel framework called BIKE, which utilizes the cross-modal bridge to explore bidirectional knowledge: i) We introduce the Video Attribute Association mechanism, which leverages the Video-to-Text knowledge to generate textual auxiliary attributes for complementing video recognition.
Ranked #1 on
Zero-Shot Action Recognition
on ActivityNet
EASpace: Enhanced Action Space for Policy Transfer
In this paper, a novel algorithm named EASpace (Enhanced Action Space) is proposed, which formulates macro actions in an alternative form to accelerate the learning process using multiple available sub-optimal expert policies.
Penalizing the Hard Example But Not Too Much: A Strong Baseline for Fine-Grained Visual Classification
Second, we instantiate the loss function and provide a strong baseline for FGVC, where the performance of a naive backbone can be boosted and be comparable with recent methods.
Ranked #32 on
Fine-Grained Image Classification
on FGVC Aircraft
Fine-Grained Image Classification
Fine-Grained Visual Recognition
ReLER@ZJU Submission to the Ego4D Moment Queries Challenge 2022
Moreover, in order to better capture the long-term temporal dependencies in the long videos, we propose a segment-level recurrence mechanism.
LambdaKG: A Library for Pre-trained Language Model-Based Knowledge Graph Embeddings
Knowledge Graphs (KGs) often have two characteristics: heterogeneous graph structure and text-rich entity/relation information.
Slimmable Networks for Contrastive Self-supervised Learning
In this work, we introduce another one-stage solution to obtain pre-trained small models without the need for extra teachers, namely, slimmable networks for contrastive self-supervised learning (SlimCLR).
Jointly Harnessing Prior Structures and Temporal Consistency for Sign Language Video Generation
We optimize the two losses and keypoint detector network in an end-to-end manner.
ReLER@ZJU-Alibaba Submission to the Ego4D Natural Language Queries Challenge 2022
In this report, we present the ReLER@ZJU-Alibaba submission to the Ego4D Natural Language Queries (NLQ) Challenge in CVPR 2022.
Ranked #5 on
Natural Language Queries
on Ego4D
CenterCLIP: Token Clustering for Efficient Text-Video Retrieval
In this paper, to reduce the number of redundant video tokens, we design a multi-segment token clustering algorithm to find the most representative tokens and drop the non-essential ones.
Ranked #11 on
Video Retrieval
on MSVD
(using extra training data)
Scalable Video Object Segmentation with Identification Mechanism
This paper delves into the challenges of achieving scalable and effective multi-object modeling for semi-supervised Video Object Segmentation (VOS).
Multi-robot Cooperative Pursuit via Potential Field-Enhanced Reinforcement Learning
It is shown by numerical simulations that the proposed hybrid design outperforms the pursuit policies either learned from vanilla reinforcement learning or designed by the potential field method.
Action Keypoint Network for Efficient Video Recognition
An effective approach is to select informative content from the holistic video, yielding a popular family of dynamic video recognition methods.
Ranked #43 on
Action Recognition
on Something-Something V1
Reasoning Through Memorization: Nearest Neighbor Knowledge Graph Embeddings
Previous knowledge graph embedding approaches usually map entities to representations and utilize score functions to predict the target entities, yet they typically struggle to reason rare or emerging unseen entities.
Ranked #1 on
Link Prediction
on FB15k-237-ind
A Simple Episodic Linear Probe Improves Visual Recognition in the Wild
In this paper, we propose an episodic linear probing (ELP) classifier to reflect the generalization of visual representations in an online manner.
Ranked #13 on
Fine-Grained Image Classification
on CUB-200-2011
Large-Scale Video Panoptic Segmentation in the Wild: A Benchmark
In contrast, our large-scale VIdeo Panoptic Segmentation in the Wild (VIPSeg) dataset provides 3, 536 videos and 84, 750 frames with pixel-level panoptic annotations, covering a wide range of real-world scenarios and categories.
Self-supervised Point Cloud Representation Learning via Separating Mixed Shapes
Albeit simple, the pre-trained encoder can capture the key points of an unseen point cloud and surpasses the encoder trained from scratch on downstream tasks.
Ranked #46 on
3D Part Segmentation
on ShapeNet-Part
PR-RRN: Pairwise-Regularized Residual-Recursive Networks for Non-rigid Structure-from-Motion
As NRSfM is a highly under-constrained problem, we propose two new pairwise regularization to further regularize the reconstruction.
Less is More: Sparse Sampling for Dense Reaction Predictions
Obtaining viewer responses from videos can be useful for creators and streaming platforms to analyze the video performance and improve the future user experience.
Connecting Language and Vision for Natural Language-Based Vehicle Retrieval
In this paper, we apply one new modality, i. e., the language description, to search the vehicle of interest and explore the potential of this task in the real-world scenario.
T2VLAD: Global-Local Sequence Alignment for Text-Video Retrieval
Moreover, a global alignment method is proposed to provide a global cross-modal measurement that is complementary to the local perspective.
Learning to Anticipate Egocentric Actions by Imagination
We further improve ImagineRNN by residual anticipation, i. e., changing its target to predicting the feature difference of adjacent frames instead of the frame content.
Interactive Prototype Learning for Egocentric Action Recognition
To avoid these additional costs, we propose an end-to-end Interactive Prototype Learning (IPL) framework to learn better active object representations by leveraging the motion cues from the actor.
Symbiotic Attention with Privileged Information for Egocentric Action Recognition
Due to the large action vocabulary in egocentric video datasets, recent studies usually utilize a two-branch structure for action recognition, ie, one branch for verb classification and the other branch for noun classification.
Ranked #4 on
Egocentric Activity Recognition
on EGTEA
Variable-Viewpoint Representations for 3D Object Recognition
For the problem of 3D object recognition, researchers using deep learning methods have developed several very different input representations, including "multi-view" snapshots taken from discrete viewpoints around an object, as well as "spherical" representations consisting of a dense map of essentially ray-traced samples of the object from all directions.
Baidu-UTS Submission to the EPIC-Kitchens Action Recognition Challenge 2019
In this report, we present the Baidu-UTS submission to the EPIC-Kitchens Action Recognition Challenge in CVPR 2019.
The Toybox Dataset of Egocentric Visual Object Transformations
In object recognition research, many commonly used datasets (e. g., ImageNet and similar) contain relatively sparse distributions of object instances and views, e. g., one might see a thousand different pictures of a thousand different giraffes, mostly taken from a few conventionally photographed angles.
