Search Results for author:
Found 97 papers, 47 papers with code
Proposal-based Video Completion
Video inpainting is an important technique for a wide variety of applications from video content editing to video restoration.
Theoretical Analysis of Near-Field MIMO Channel Capacity and Mid-Band Experimental Validation
With the increase of multiple-input-multiple-output (MIMO) array size and carrier frequency, near-field MIMO communications will become crucial in 6G wireless networks.
Step-Audio-AQAA: a Fully End-to-End Expressive Large Audio Language Model
This work contributes a promising solution for end-to-end LALMs and highlights the critical role of token-based vocoder in enhancing overall performance for AQAA tasks.
Generalizable LLM Learning of Graph Synthetic Data with Reinforcement Learning
We then compare them against existing settings on both in-domain synthetic tasks and out-of-domain real-world tasks with implicit graph structures such as multi-hop QA, structured planning, and more.
Unveiling the Hidden: Movie Genre and User Bias in Spoiler Detection
User bias is calculated through dynamic graph modeling of review history.
Kimi-VL Technical Report
We present Kimi-VL, an efficient open-source Mixture-of-Experts (MoE) vision-language model (VLM) that offers advanced multimodal reasoning, long-context understanding, and strong agent capabilities - all while activating only 2. 8B parameters in its language decoder (Kimi-VL-A3B).
Open-Qwen2VL: Compute-Efficient Pre-Training of Fully-Open Multimodal LLMs on Academic Resources
We open-source all aspects of our work, including compute-efficient and data-efficient training details, data filtering methods, sequence packing scripts, pre-training data in WebDataset format, FSDP-based training codebase, and both base and instruction-tuned model checkpoints.
ROS-SAM: High-Quality Interactive Segmentation for Remote Sensing Moving Object
In this work, we propose ROS-SAM, a method designed to achieve high-quality interactive segmentation while preserving generalization across diverse remote sensing data.
BannerAgency: Advertising Banner Design with Multimodal LLM Agents
Advertising banners are critical for capturing user attention and enhancing advertising campaign effectiveness.
Reward Shaping to Mitigate Reward Hacking in RLHF
Reinforcement Learning from Human Feedback (RLHF) is essential for aligning large language models (LLMs) with human values.
Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction
Based on our new StepEval-Audio-360 evaluation benchmark, Step-Audio achieves state-of-the-art performance in human evaluations, especially in terms of instruction following.
Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model
We present Step-Video-T2V, a state-of-the-art text-to-video pre-trained model with 30B parameters and the ability to generate videos up to 204 frames in length.
CoSER: Coordinating LLM-Based Persona Simulation of Established Roles
It provides authentic dialogues with real-world intricacies, as well as diverse data types such as conversation setups, character experiences and internal thoughts.
Cosmos World Foundation Model Platform for Physical AI
In this paper, we present the Cosmos World Foundation Model Platform to help developers build customized world models for their Physical AI setups.
Fast Prompt Alignment for Text-to-Image Generation
Text-to-image generation has advanced rapidly, yet aligning complex textual prompts with generated visuals remains challenging, especially with intricate object relationships and fine-grained details.
Gotta Hear Them All: Sound Source Aware Vision to Audio Generation
By addressing V2A generation at the sound-source level, SSV2A surpasses state-of-the-art methods in both generation fidelity and relevance as evidenced by extensive experiments.
Real-time and Downtime-tolerant Fault Diagnosis for Railway Turnout Machines (RTMs) Empowered with Cloud-Edge Pipeline Parallelism
In this paper, an edge-cloud collaborative early-warning system is proposed to enable real-time and downtime-tolerant fault diagnosis of RTMs, providing a new paradigm for the deployment of models in safety-critical scenarios.
DINeuro: Distilling Knowledge from 2D Natural Images via Deformable Tubular Transferring Strategy for 3D Neuron Reconstruction
Reconstructing neuron morphology from 3D light microscope imaging data is critical to aid neuroscientists in analyzing brain networks and neuroanatomy.
DAOcc: 3D Object Detection Assisted Multi-Sensor Fusion for 3D Occupancy Prediction
Multi-sensor fusion significantly enhances the accuracy and robustness of 3D semantic occupancy prediction, which is crucial for autonomous driving and robotics.
Ranked #1 on
Prediction Of Occupancy Grid Maps
on Occ3D-nuScenes
BUPTCMCC-6G-CMG+: A GBSM-Based ISAC Standard Channel Model Generator
Integrated sensing and communication (ISAC) has been recognized as the key technology in the vision of the sixth generation (6G) era.
Enhancing Advanced Visual Reasoning Ability of Large Language Models
Recent advancements in Vision-Language (VL) research have sparked new benchmarks for complex visual reasoning, challenging models' advanced reasoning ability.
Explaining Datasets in Words: Statistical Models with Natural Language Parameters
To make sense of massive data, we often fit simplified models and then interpret the parameters; for example, we cluster the text embeddings and then interpret the mean parameters of each cluster.
VMAS: Video-to-Music Generation via Semantic Alignment in Web Music Videos
We present a framework for learning to generate background music from video inputs.
Future Does Matter: Boosting 3D Object Detection with Temporal Motion Estimation in Point Cloud Sequences
Specifically, Motion-Guided Feature Aggregation (MGFA) is proposed to utilize the object trajectory from previous and future motion states to model spatial-temporal correlations into gaussian heatmap over a driving sequence.
Multimodal Causal Reasoning Benchmark: Challenging Vision Large Language Models to Infer Causal Links Between Siamese Images
However, will these causalities remain straightforward for Vision Large Language Models (VLLMs) when only visual hints are provided?
Can LLM Graph Reasoning Generalize beyond Pattern Memorization?
To this end, we propose the NLGift benchmark, an evaluation suite of LLM graph reasoning generalization: whether LLMs could go beyond semantic, numeric, structural, reasoning patterns in the synthetic training data and improve utility on real-world graph-based tasks.
Autoregressive Pretraining with Mamba in Vision
The vision community has started to build with the recently developed state space model, Mamba, as the new backbone for a range of tasks.
Dance Any Beat: Blending Beats with Visuals in Dance Video Generation
This task enables the dance generation of specific individuals without requiring keypoint annotations, making it more versatile and applicable to various situations.
Boosting 3D Neuron Segmentation with 2D Vision Transformer Pre-trained on Natural Images
To address this limitation, we aim to distill the consensus knowledge from massive natural image data to aid the segmentation model in learning the complex neuron structures.
HQ-Edit: A High-Quality Dataset for Instruction-based Image Editing
This study introduces HQ-Edit, a high-quality instruction-based image editing dataset with around 200, 000 edits.
Digital Twin Channel for 6G: Concepts, Architectures and Potential Applications
Digital twin channel (DTC) is the real-time mapping of a wireless channel from the physical world to the digital world, which is expected to provide significant performance enhancements for the sixth-generation (6G) air-interface design.
MMoE: Robust Spoiler Detection with Multi-modal Information and Domain-aware Mixture-of-Experts
For instance, the metadata and the corresponding user's information of a review could be helpful.
Finetuned Multimodal Language Models Are High-Quality Image-Text Data Filters
Our MLM filter can generalize to different models and tasks, and be used as a drop-in replacement for CLIPScore.
Hy-DAT: A Tool to Address Hydropower Modeling Gaps Using Interdependency, Efficiency Curves, and Unit Dispatch Models
As the power system continues to be flooded with intermittent resources, it becomes more important to accurately assess the role of hydro and its impact on the power grid.
DELL: Generating Reactions and Explanations for LLM-Based Misinformation Detection
Large language models are limited by challenges in factuality and hallucinations to be directly employed off-the-shelf for judging the veracity of news articles, where factual accuracy is paramount.
VISTA-LLAMA: Reducing Hallucination in Video Language Models via Equal Distance to Visual Tokens
This amplifies the effect of visual tokens on text generation especially when the relative distance is longer between visual and text tokens.
Video Recognition in Portrait Mode
While existing datasets mainly comprise landscape mode videos, our paper seeks to introduce portrait mode videos to the research community and highlight the unique challenges associated with this video format.
Shot2Story20K: A New Benchmark for Comprehensive Understanding of Multi-shot Videos
A human need to capture both the event in every shot and associate them together to understand the story behind it.
Ranked #1 on
video narration captioning
on Shot2Story20K
Vista-LLaMA: Reliable Video Narrator via Equal Distance to Visual Tokens
This amplifies the effect of visual tokens on text generation, especially when the relative distance is longer between visual and text tokens.
Ranked #2 on
Question Answering
on NExT-QA (Open-ended VideoQA)
InfiMM-Eval: Complex Open-Ended Reasoning Evaluation For Multi-Modal Large Language Models
To mitigate this issue, we manually curate a benchmark dataset specifically designed for MLLMs, with a focus on complex reasoning tasks.
GPT-4V(ision) as a Generalist Evaluator for Vision-Language Tasks
Automatically evaluating vision-language tasks is challenging, especially when it comes to reflecting human judgments due to limitations in accounting for fine-grained details.
Consistent-1-to-3: Consistent Image to 3D View Synthesis via Geometry-aware Diffusion Models
Specifically, we decompose the NVS task into two stages: (i) transforming observed regions to a novel view, and (ii) hallucinating unseen regions.
Resolving Knowledge Conflicts in Large Language Models
To this end, we introduce an evaluation framework for simulating contextual knowledge conflicts and quantitatively evaluating to what extent LLMs achieve these goals.
The Devil is in the Details: A Deep Dive into the Rabbit Hole of Data Filtering
DataComp is a new benchmark dedicated to evaluating different methods for data filtering.
Advancements in 3D Lane Detection Using LiDAR Point Clouds: From Data Collection to Model Development
Advanced Driver-Assistance Systems (ADAS) have successfully integrated learning-based techniques into vehicle perception and decision-making.
Dataset Condensation via Generative Model
Dataset condensation aims to condense a large dataset with a lot of training samples into a small set.
V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models
In this paper, we propose a lightweight solution to this problem by leveraging foundation models, specifically CLIP, CLAP, and AudioLDM.
Ranked #2 on
Video-to-Sound Generation
on VGG-Sound
Exploring Annotation-free Image Captioning with Retrieval-augmented Pseudo Sentence Generation
Recently, training an image captioner without annotated image-sentence pairs has gained traction.
Why Is Prompt Tuning for Vision-Language Models Robust to Noisy Labels?
Vision-language models such as CLIP learn a generic text-image embedding from large-scale training data.
Exploring the Role of Audio in Video Captioning
Recent focus in video captioning has been on designing architectures that can consume both video and text modalities, and using large-scale video datasets with text transcripts for pre-training, such as HowTo100M.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Can Language Models Solve Graph Problems in Natural Language?
We then propose Build-a-Graph Prompting and Algorithmic Prompting, two instruction-based approaches to enhance LLMs in solving natural language graph problems.
Detecting Spoilers in Movie Reviews with External Movie Knowledge and User Networks
We then propose MVSD, a novel Multi-View Spoiler Detection framework that takes into account the external knowledge about movies and user activities on movie review platforms.
Progressive Volume Distillation with Active Learning for Efficient NeRF Architecture Conversion
For instance, PVD-AL can distill an MLP-based model from a Hashtables-based model at a 10~20X faster speed and 0. 8dB~2dB higher PSNR than training the MLP-based model from scratch.
$R^{2}$Former: Unified $R$etrieval and $R$eranking Transformer for Place Recognition
Visual Place Recognition (VPR) estimates the location of query images by matching them with images in a reference database.
PAniC-3D: Stylized Single-view 3D Reconstruction from Portraits of Anime Characters
We propose PAniC-3D, a system to reconstruct stylized 3D character heads directly from illustrated (p)ortraits of (ani)me (c)haracters.
Open-world Instance Segmentation: Top-down Learning with Bottom-up Supervision
Many top-down architectures for instance segmentation achieve significant success when trained and tested on pre-defined closed-world taxonomy.
Temporal Perceiving Video-Language Pre-training
Specifically, text-video localization consists of moment retrieval, which predicts start and end boundaries in videos given the text description, and text localization which matches the subset of texts with the video features.
R2Former: Unified Retrieval and Reranking Transformer for Place Recognition
Visual Place Recognition (VPR) estimates the location of query images by matching them with images in a reference database.
One is All: Bridging the Gap Between Neural Radiance Fields Architectures with Progressive Volume Distillation
In this paper, we propose Progressive Volume Distillation (PVD), a systematic distillation method that allows any-to-any conversions between different architectures, including MLP, sparse or low-rank tensors, hashtables and their compositions.
Ranked #1 on
Novel View Synthesis
on NeRF
(Average PSNR metric)
PointNeuron: 3D Neuron Reconstruction via Geometry and Topology Learning of Point Clouds
In this paper, we propose a novel framework for 3D neuron reconstruction.
TwiBot-22: Towards Graph-Based Twitter Bot Detection
Twitter bot detection has become an increasingly important task to combat misinformation, facilitate social media moderation, and preserve the integrity of the online discourse.
Towards Generalisable Audio Representations for Audio-Visual Navigation
In audio-visual navigation (AVN), an intelligent agent needs to navigate to a constantly sound-making object in complex 3D environments based on its audio and visual perceptions.
Spatiality-guided Transformer for 3D Dense Captioning on Point Clouds
Dense captioning in 3D point clouds is an emerging vision-and-language task involving object-level 3D scene understanding.
Ranked #8 on
3D dense captioning
on ScanRefer Dataset
Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinity
From PA we construct a large set of pseudo-ground-truth instance masks; combined with human-annotated instance masks we train GGNs and significantly outperform the SOTA on open-world instance segmentation on various benchmarks including COCO, LVIS, ADE20K, and UVO.
Canonical Mean Filter for Almost Zero-Shot Multi-Task classification
Motivated by ANIL, we rethink the role of adaption in the feature extractor of CNAPs, which is a state-of-the-art representative few-shot method.
3D Medical Point Transformer: Introducing Convolution to Attention Networks for Medical Point Cloud Analysis
General point clouds have been increasingly investigated for different tasks, and recently Transformer-based networks are proposed for point cloud analysis.
Ranked #1 on
3D Point Cloud Classification
on IntrA
PyTorchVideo: A Deep Learning Library for Video Understanding
We introduce PyTorchVideo, an open-source deep-learning library that provides a rich set of modular, efficient, and reproducible components for a variety of video understanding tasks, including classification, detection, self-supervised learning, and low-level processing.
Searching for Two-Stream Models in Multivariate Space for Video Recognition
We design a multivariate search space, including 6 search variables to capture a wide variety of choices in designing two-stream models.
Voxel-wise Cross-Volume Representation Learning for 3D Neuron Reconstruction
Recently, a series of deep learning based segmentation methods have been proposed to improve the quality of raw 3D optical image stacks by removing noises and restoring neuronal structures from low-contrast background.
Towards Understanding the Effectiveness of Attention Mechanism
Instead, we verify the crucial role of feature map multiplication in attention mechanism and uncover a fundamental impact of feature map multiplication on the learned landscapes of CNNs: with the high order non-linearity brought by the feature map multiplication, it played a regularization role on CNNs, which made them learn smoother and more stable landscapes near real samples compared to vanilla CNNs.
Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentation
Current state-of-the-art object detection and segmentation methods work well under the closed-world assumption.
Beyond Short Clips: End-to-End Video-Level Learning with Collaborative Memories
The standard way of training video models entails sampling at each iteration a single clip from a video and optimizing the clip prediction with respect to the video-level label.
Is Space-Time Attention All You Need for Video Understanding?
We present a convolution-free approach to video classification built exclusively on self-attention over space and time.
Ranked #1 on
Video Question Answering
on Howto100M-QA
Single Neuron Segmentation using Graph-based Global Reasoning with Auxiliary Skeleton Loss from 3D Optical Microscope Images
One of the critical steps in improving accurate single neuron reconstruction from three-dimensional (3D) optical microscope images is the neuronal structure segmentation.
Interactive Prototype Learning for Egocentric Action Recognition
To avoid these additional costs, we propose an end-to-end Interactive Prototype Learning (IPL) framework to learn better active object representations by leveraging the motion cues from the actor.
From W-Net to CDGAN: Bi-temporal Change Detection via Deep Learning Techniques
Traditional change detection methods usually follow the image differencing, change feature extraction and classification framework, and their performance is limited by such simple image domain differencing and also the hand-crafted features.
CJRC: A Reliable Human-Annotated Benchmark DataSet for Chinese Judicial Reading Comprehension
We present a Chinese judicial reading comprehension (CJRC) dataset which contains approximately 10K documents and almost 50K questions with answers.
Region and Object based Panoptic Image Synthesis through Conditional GANs
Our contribution consists of the proposal of a significant task worth investigating and a naive baseline of solving it.
CAIL2019-SCM: A Dataset of Similar Case Matching in Legal Domain
In this paper, we introduce CAIL2019-SCM, Chinese AI and Law 2019 Similar Case Matching dataset.
Incorporating Graph Attention Mechanism into Knowledge Graph Reasoning Based on Deep Reinforcement Learning
Meanwhile, a novel mechanism of reinforcement learning is proposed by forcing an agent to walk forward every step to avoid the agent stalling at the same entity node constantly.
FASTER Recurrent Networks for Efficient Video Classification
FASTER aims to leverage the redundancy between neighboring clips and reduce the computational cost by learning to aggregate the predictions from models of different complexities.
Ranked #29 on
Action Recognition
on UCF101
Video Modeling with Correlation Networks
Motion is a salient cue to recognize actions in video.
Ranked #115 on
Action Classification
on Kinetics-400
Large-scale weakly-supervised pre-training for video action recognition
Second, frame-based models perform quite well on action recognition; is pre-training for good image features sufficient or is pre-training for spatio-temporal features valuable for optimal transfer learning?
Ranked #2 on
Egocentric Activity Recognition
on EPIC-KITCHENS-55
(Actions Top-1 (S2) metric)
Video Classification with Channel-Separated Convolutional Networks
It is natural to ask: 1) if group convolution can help to alleviate the high computational cost of video classification networks; 2) what factors matter the most in 3D group convolutional networks; and 3) what are good computation/accuracy trade-offs with 3D group convolutional networks.
Ranked #1 on
Action Recognition
on Sports-1M
Defeats GAN: A Simpler Model Outperforms in Knowledge Representation Learning
The goal of knowledge representation learning is to embed entities and relations into a low-dimensional, continuous vector space.
Multi-task Learning for Chinese Word Usage Errors Detection
Chinese word usage errors often occur in non-native Chinese learners' writing.
Overview of CAIL2018: Legal Judgment Prediction Competition
In this paper, we give an overview of the Legal Judgment Prediction (LJP) competition at Chinese AI and Law challenge (CAIL2018).
Scenes-Objects-Actions: A Multi-Task, Multi-Label Video Dataset
The videos retrieved by the search engines are then veried for correctness by human annotators.
CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction
In this paper, we introduce the \textbf{C}hinese \textbf{AI} and \textbf{L}aw challenge dataset (CAIL2018), the first large-scale Chinese legal dataset for judgment prediction.
Devon: Deformable Volume Network for Learning Optical Flow
State-of-the-art neural network models estimate large displacement optical flow in multi-resolution and use warping to propagate the estimation between two resolutions.
Text Generation Based on Generative Adversarial Nets with Latent Variable
We propose the VGAN model where the generative model is composed of recurrent neural network and VAE.
A Closer Look at Spatiotemporal Convolutions for Action Recognition
In this paper we discuss several forms of spatiotemporal convolutions for video analysis and study their effects on action recognition.
Ranked #3 on
Action Recognition
on Sports-1M
Concept Drift Detection and Adaptation with Hierarchical Hypothesis Testing
A fundamental issue for statistical classification models in a streaming environment is that the joint distribution between predictor and response variables changes over time (a phenomenon also known as concept drifts), such that their classification performance deteriorates dramatically.
Face Aging Effect Simulation using Hidden Factor Analysis Joint Sparse Representation
Face aging simulation has received rising investigations nowadays, whereas it still remains a challenge to generate convincing and natural age-progressed face images.
A robust and efficient video representation for action recognition
We also use the homography to cancel out camera motion from the optical flow.
Concept Drift Detection for Streaming Data
Common statistical prediction models often require and assume stationarity in the data.
Active Community Detection in Massive Graphs
A canonical problem in graph mining is the detection of dense communities.
Social and Information Networks Physics and Society
