Search Results for author:
Found 324 papers, 116 papers with code
\textrm{DuReader}_{\textrm{vis}}: A Chinese Dataset for Open-domain Document Visual Question Answering
Open-domain question answering has been used in a wide range of applications, such as web search and enterprise search, which usually takes clean texts extracted from various formats of documents (e. g., web pages, PDFs, or Word documents) as the information source.
Learning Progressive Joint Propagation for Human Motion Prediction
Last, in order to incorporate a general motion space for high-quality prediction, we build a memory-based dictionary, which aims to preserve the global motion patterns in training data to guide the predictions.
基于相似度进行句子选择的机器阅读理解数据增强(Machine reading comprehension data Augmentation for sentence selection based on similarity)
“目前常见的机器阅读理解数据增强方法如回译, 单独对文章或者问题进行数据增强, 没有考虑文章、问题和选项三元组之间的联系。因此, 本文探索了一种利用三元组联系进行文章句子筛选的数据增强方法, 通过比较文章与问题以及选项的相似度, 选取文章中与二者联系紧密的句子。同时为了使不同选项的三元组区别增大, 我们选用了正则化Dropout的策略。实验结果表明, 在RACE数据集上的准确率可提高3. 8%。”
Deep Transferring Quantization
Specifically, we propose a method named deep transferring quantization (DTQ) to effectively exploit the knowledge in a pre-trained full-precision model.
LaVi: Efficient Large Vision-Language Models via Internal Feature Modulation
Despite the impressive advancements of Large Vision-Language Models (LVLMs), existing approaches suffer from a fundamental bottleneck: inefficient visual-language integration.
Abstract Sound Fusion with Unconditioned Inversion Model
An abstract sound is defined as a sound that does not disclose identifiable real-world sound events to a listener.
AWP: Activation-Aware Weight Pruning and Quantization with Projected Gradient Descent
To address the enormous size of Large Language Models (LLMs), model compression methods, such as quantization and pruning, are often deployed, especially on edge devices.
SimClass: A Classroom Speech Dataset Generated via Game Engine Simulation For Automatic Speech Recognition Research
The scarcity of large-scale classroom speech data has hindered the development of AI-driven speech models for education.
Uniqueness of phase retrieval from offset linear canonical transform
The classical phase retrieval refers to the recovery of an unknown signal from its Fourier magnitudes, which is widely used in fields such as quantum mechanics, signal processing, optics, etc.
Align is not Enough: Multimodal Universal Jailbreak Attack against Multimodal Large Language Models
Large Language Models (LLMs) have evolved into Multimodal Large Language Models (MLLMs), significantly enhancing their capabilities by integrating visual information and other types, thus aligning more closely with the nature of human intelligence, which processes a variety of data forms beyond just text.
FT-Boosted SV: Towards Noise Robust Speaker Verification for English Speaking Classroom Environments
Creating Speaker Verification (SV) systems for classroom settings that are robust to classroom noises such as babble noise is crucial for the development of AI tools that assist educational environments.
Context-Driven Dynamic Pruning for Large Speech Foundation Models
Speech foundation models achieve strong generalization across languages and acoustic conditions, but require significant computational resources for inference.
$μ$-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Experts
To tackle the huge computational demand of large foundation models, activation-aware compression techniques without retraining have been introduced.
LatentLLM: Attention-Aware Joint Tensor Compression
Modern foundation models such as large language models (LLMs) and large multi-modal models (LMMs) require a massive amount of computational and memory resources.
From Weak Labels to Strong Results: Utilizing 5,000 Hours of Noisy Classroom Transcripts with Minimal Accurate Data
However, classroom Automatic Speech Recognition (ASR) faces the real-world challenge of abundant weak transcripts paired with only a small amount of accurate, gold-standard data.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Scaling Law for Quantization-Aware Training
Through 268 QAT experiments, we show that quantization error decreases as model size increases, but rises with more training tokens and coarser quantization granularity.
Emergent Specialization: Rare Token Neurons in Language Models
Large language models struggle with representing and generating rare tokens despite their importance in specialized domains.
Model Merging in Pre-training of Large Language Models
Model merging has emerged as a promising technique for enhancing large language models, though its application in large-scale pre-training remains relatively unexplored.
Unveiling Knowledge Utilization Mechanisms in LLM-based Retrieval-Augmented Generation
In this paper, we present a systematic investigation of the intrinsic mechanisms by which LLMs integrate internal (parametric) and external (retrieved) knowledge in RAG scenarios.
QVGen: Pushing the Limit of Quantized Video Generative Models
We begin with a theoretical analysis demonstrating that reducing the gradient norm is essential to facilitate convergence for QAT.
End-to-End Vision Tokenizer Tuning
To address this, we propose ETT, an end-to-end vision tokenizer tuning approach that enables joint optimization between vision tokenization and target autoregressive tasks.
RetroInfer: A Vector-Storage Approach for Scalable Long-Context LLM Inference
The growing context lengths of large language models (LLMs) pose significant challenges for efficient inference, primarily due to GPU memory and bandwidth constraints.
Edge-Cloud Collaborative Computing on Distributed Intelligence and Model Optimization: A Survey
Furthermore, the review identifies critical research directions including LLMs deployment, 6G integration, neuromorphic computing, and quantum computing, offering a roadmap for addressing persistent challenges in heterogeneity management, real-time processing, and scalability.
ZipR1: Reinforcing Token Sparsity in MLLMs
Sparse attention mechanisms aim to reduce computational overhead by selectively processing a subset of salient tokens while preserving model performance.
Image Difference Grounding with Natural Language
We introduce DiffGround, a large-scale and high-quality dataset for IDG, containing image pairs with diverse visual variations along with instructions querying fine-grained differences.
Towards Unified Referring Expression Segmentation Across Omni-Level Visual Target Granularities
Referring expression segmentation (RES) aims at segmenting the entities' masks that match the descriptive language expression.
COSMO: Combination of Selective Memorization for Low-cost Vision-and-Language Navigation
RSS facilitates comprehensive inter-modal interactions within a single scan, while the CS3 module adapts the selective state space module into a dual-stream architecture, thereby enhancing the acquisition of cross-modal interactions.
AssistPDA: An Online Video Surveillance Assistant for Video Anomaly Prediction, Detection, and Analysis
To bridge this gap and facilitate the practical deployment of LLM-based VAD, we introduce AssistPDA, the first online video anomaly surveillance assistant that unifies video anomaly prediction, detection, and analysis (VAPDA) within a single framework.
Adaptive Weighted Parameter Fusion with CLIP for Class-Incremental Learning
Class-incremental Learning (CIL) enables the model to incrementally absorb knowledge from new classes and build a generic classifier across all previously encountered classes.
CRCL: Causal Representation Consistency Learning for Anomaly Detection in Surveillance Videos
Video Anomaly Detection (VAD) remains a fundamental yet formidable task in the video understanding community, with promising applications in areas such as information forensics and public safety protection.
Anomaly Detection In Surveillance Videos
Representation Learning
+1
Breaking the Encoder Barrier for Seamless Video-Language Understanding
Most Video-Large Language Models (Video-LLMs) adopt an encoder-decoder framework, where a vision encoder extracts frame-wise features for processing by a language model.
QuartDepth: Post-Training Quantization for Real-Time Depth Estimation on the Edge
Monocular Depth Estimation (MDE) has emerged as a pivotal task in computer vision, supporting numerous real-world applications.
FlexVLN: Flexible Adaptation for Diverse Vision-and-Language Navigation Tasks
In this paper, we propose FlexVLN, an innovative hierarchical approach to VLN that integrates the fundamental navigation ability of a supervised-learning-based Instruction Follower with the robust generalization ability of the LLM Planner, enabling effective generalization across diverse VLN datasets.
AR-Diffusion: Asynchronous Video Generation with Auto-Regressive Diffusion
To address these issues, we introduce Auto-Regressive Diffusion (AR-Diffusion), a novel model that combines the strengths of auto-regressive and diffusion models for flexible, asynchronous video generation.
Winning Big with Small Models: Knowledge Distillation vs. Self-Training for Reducing Hallucination in QA Agents
Using a dataset of questions about a Samsung Smart TV user manual, we demonstrate that synthetic data generated by LLMs outperforms crowdsourced data in reducing hallucination in finetuned models.
Mitigating Data Scarcity in Time Series Analysis: A Foundation Model with Series-Symbol Data Generation
Building on this concept, we introduce a series-symbol (S2) dual-modulity data generation mechanism, enabling the unrestricted creation of high-quality time series data paired with corresponding symbolic representations.
VRoPE: Rotary Position Embedding for Video Large Language Models
Rotary Position Embedding (RoPE) has shown strong performance in text-based Large Language Models (LLMs), but extending it to video remains a challenge due to the intricate spatiotemporal structure of video frames.
BabyLM Turns 3: Call for papers for the 2025 BabyLM workshop
We also call for papers outside the competition in any relevant areas.
Smoothed Embeddings for Robust Language Models
Improving the safety and reliability of large language models (LLMs) is a crucial aspect of realizing trustworthy AI systems.
Community-Aware Temporal Walks: Parameter-Free Representation Learning on Continuous-Time Dynamic Graphs
Dynamic graph representation learning plays a crucial role in understanding evolving behaviors.
A Survey on Diffusion Models for Anomaly Detection
Diffusion models (DMs) have emerged as a powerful class of generative AI models, showing remarkable potential in anomaly detection (AD) tasks across various domains, such as cybersecurity, fraud detection, healthcare, and manufacturing.
Can LLM Generate Regression Tests for Software Commits?
For programs with more compact file formats, like PDF, as expected, it struggled to generate effective test cases.
Few-Shot Learner Generalizes Across AI-Generated Image Detection
Current fake image detectors trained on large synthetic image datasets perform satisfactorily on limited studied generative models.
Advancing the Understanding of Fine-Grained 3D Forest Structures using Digital Cousins and Simulation-to-Reality: Methods and Datasets
The Boreal3D dataset, and more broadly, the synthetic data augmentation framework, is poised to become a critical resource for advancing research in large-scale 3D forest scene understanding and structural parameter estimation.
Efficient Motion-Aware Video MLLM
Most current video MLLMs rely on uniform frame sampling and image-level encoders, resulting in inefficient data processing and limited motion awareness.
FedCross: Intertemporal Federated Learning Under Evolutionary Games
Federated Learning (FL) mitigates privacy leakage in decentralized machine learning by allowing multiple clients to train collaboratively locally.
Channel Merging: Preserving Specialization for Merged Experts
To address the inefficiency, model merging strategies have emerged, merging all LLMs into one model to reduce the memory footprint during inference.
Numerical Pruning for Efficient Autoregressive Models
Transformers have emerged as the leading architecture in deep learning, proving to be versatile and highly effective across diverse domains beyond language and image processing.
AutoSGNN: Automatic Propagation Mechanism Discovery for Spectral Graph Neural Networks
This challenge has led to the manual design of GNNs tailored to specific graph types, but these approaches are limited by the high cost of labor and the constraints of expert knowledge, which cannot keep up with the rapid growth of graph data.
Two Layer Walk: A Community-Aware Graph Embedding
Community structures are critical for understanding the mesoscopic organization of networks, bridging local and global patterns.
TRAIL: Trust-Aware Client Scheduling for Semi-Decentralized Federated Learning
However, in the context of semi-decentralized FL, clients' communication and training states are dynamic.
AniSora: Exploring the Frontiers of Animation Video Generation in the Sora Era
In this paper, we present a comprehensive system, AniSora, designed for animation video generation, which includes a data processing pipeline, a controllable generation model, and an evaluation benchmark.
Benchmarking LLMs for Mimicking Child-Caregiver Language in Interaction
LLMs can generate human-like dialogues, yet their ability to simulate early child-adult interactions remains largely unexplored.
CA-SSLR: Condition-Aware Self-Supervised Learning Representation for Generalized Speech Processing
We introduce Condition-Aware Self-Supervised Learning Representation (CA-SSLR), a generalist conditioning model broadly applicable to various speech-processing tasks.
COAP: Memory-Efficient Training with Correlation-Aware Gradient Projection
While LoRA, a popular parameter-efficient method, reduces memory usage, it often suffers from suboptimal performance due to the constraints of low-rank updates.
Evaluating and Advancing Multimodal Large Language Models in Ability Lens
To address this, we introduce \textbf{AbilityLens}, a unified benchmark designed to evaluate MLLMs across six key perception abilities, focusing on both accuracy and stability, with each ability encompassing diverse question types, domains, and metrics.
Privacy-Preserving Video Anomaly Detection: A Survey
Recently, researchers have focused on privacy concerns in VAD by conducting systematic studies from various perspectives including data, features, and systems, making Privacy-Preserving Video Anomaly Detection (P2VAD) a hotspot in the AI community.
CompetitorFormer: Competitor Transformer for 3D Instance Segmentation
Transformer-based methods have become the dominant approach for 3D instance segmentation.
ID-Patch: Robust ID Association for Group Photo Personalization
Existing methods suffer from limitations such as the reliance on segmentation models, increased runtime, or a high probability of ID leakage.
Self-Calibrated Listwise Reranking with Large Language Models
Large language models (LLMs), with advanced linguistic capabilities, have been employed in reranking tasks through a sequence-to-sequence approach.
Quantum Diffusion Models for Few-Shot Learning
Modern quantum machine learning (QML) methods involve the variational optimization of parameterized quantum circuits on training datasets, followed by predictions on testing datasets.
Deep Insights into Automated Optimization with Large Language Models and Evolutionary Algorithms
Since this synergy enables a more efficient and creative search process, we first conduct an extensive review of recent research on the application of LLMs in optimization.
ChatSearch: a Dataset and a Generative Retrieval Model for General Conversational Image Retrieval
In this paper, we investigate the task of general conversational image retrieval on open-domain images.
AutoRNet: Automatically Optimizing Heuristics for Robust Network Design via Large Language Models
Achieving robust networks is a challenging problem due to its NP-hard nature and complex solution space.
Ada-K Routing: Boosting the Efficiency of MoE-based LLMs
In this paper, we propose a novel Ada-K routing strategy that dynamically adjusts the number of activated experts for each token, thereby improving the balance between computational efficiency and model performance.
EEGPT: Unleashing the Potential of EEG Generalist Foundation Model by Autoregressive Pre-training
First, we propose an electrode-wise modeling strategy that treats each electrode as a fundamental unit, enabling the integration of diverse EEG datasets collected from up to 138 electrodes, amassing 37. 5M pre-training samples.
ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsification
The efficiency of large vision-language models (LVLMs) is constrained by the computational bottleneck of the attention mechanism during the prefill phase and the memory bottleneck of fetching the key-value (KV) cache in the decoding phase, particularly in scenarios involving high-resolution images or videos.
Gap Preserving Distillation by Building Bidirectional Mappings with A Dynamic Teacher
To address this, we propose a Gap Preserving Distillation (GPD) method that trains an additional dynamic teacher model from scratch along with training the student to bridge this gap.
COMUNI: Decomposing Common and Unique Video Signals for Diffusion-based Video Generation
To prevent redundant modeling of common video signals, we propose a novel diffusion-based framework, named COMUNI, which decomposes the COMmon and UNIque video signals to enable efficient video generation.
MM-LDM: Multi-Modal Latent Diffusion Model for Sounding Video Generation
Sounding Video Generation (SVG) is an audio-video joint generation task challenged by high-dimensional signal spaces, distinct data formats, and different patterns of content information.
HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Acceleration
Diffusion Transformers (DiTs) excel in generative tasks but face practical deployment challenges due to high inference costs.
Design and validation of a fuzzy logic controller for multi-section continuum robots
This paper introduces a simple-structured, model-less fuzzy logic controller for the closed-loop control of continuum robots.
MiniVLN: Efficient Vision-and-Language Navigation by Progressive Knowledge Distillation
In recent years, Embodied Artificial Intelligence (Embodied AI) has advanced rapidly, yet the increasing size of models conflicts with the limited computational capabilities of Embodied AI platforms.
RoleBreak: Character Hallucination as a Jailbreak Attack in Role-Playing Systems
Role-playing systems powered by large language models (LLMs) have become increasingly influential in emotional communication applications.
M2OST: Many-to-one Regression for Predicting Spatial Transcriptomics from Digital Pathology Images
The advancement of Spatial Transcriptomics (ST) has facilitated the spatially-aware profiling of gene expressions based on histopathology images.
End-Cloud Collaboration Framework for Advanced AI Customer Service in E-commerce
Specifically, the large cloud model acts as a teacher, guiding and promoting the learning of the end model, which significantly reduces the end model's reliance on large-scale, high-quality data and thereby addresses the data bottleneck in traditional end model training, offering a new paradigm for the rapid deployment of industry applications.
CPT-Boosted Wav2vec2.0: Towards Noise Robust Speech Recognition for Classroom Environments
Creating Automatic Speech Recognition (ASR) systems that are robust and resilient to classroom conditions is paramount to the development of AI tools to aid teachers and students.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Exploring User-level Gradient Inversion with a Diffusion Prior
We explore user-level gradient inversion as a new attack surface in distributed learning.
LSVOS Challenge Report: Large-scale Complex and Long Video Object Segmentation
Despite the promising performance of current video segmentation models on existing benchmarks, these models still struggle with complex scenes.
Cross Fusion RGB-T Tracking with Bi-directional Adapter
To address this issue, we propose a novel Cross Fusion RGB-T Tracking architecture (CFBT) that ensures the full participation of multiple modalities in tracking while dynamically fusing temporal information.
Ranked #6 on
Rgb-T Tracking
on RGBT210
Forget to Flourish: Leveraging Machine-Unlearning on Pretrained Language Models for Privacy Leakage
This approach manipulates a pre-trained language model to increase the leakage of private data during the fine-tuning process.
Analyzing Inference Privacy Risks Through Gradients in Machine Learning
While previous work has studied various privacy risks of sharing gradients, our paper aims to provide a systematic approach to analyze private information leakage from gradients.
A Survey on Facial Expression Recognition of Static and Dynamic Emotions
Facial expression recognition (FER) aims to analyze emotional states from static images and dynamic sequences, which is pivotal in enhancing anthropomorphic communication among humans, robots, and digital avatars by leveraging AI technologies.
Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detection
We call a function "vulnerable" if it was involved in a patch of an actual security flaw and confirmed to cause the program's vulnerability.
The Instance-centric Transformer for the RVOS Track of LSVOS Challenge: 3rd Place Solution
Referring Video Object Segmentation is an emerging multi-modal task that aims to segment objects in the video given a natural language expression.
A Unified Framework for Iris Anti-Spoofing: Introducing IrisGeneral Dataset and Masked-MoE Method
Specifically, we utilize the Mixture of Experts (MoE) to fit complex data distributions using multiple sub-neural networks.
Learning Feature-Preserving Portrait Editing from Generated Pairs
Portrait editing is challenging for existing techniques due to difficulties in preserving subject features like identity.
Diffusion Feedback Helps CLIP See Better
We demonstrate that DIVA improves CLIP's performance on the challenging MMVP-VLM benchmark which assesses fine-grained visual abilities to a large extent (e. g., 3-7%), and enhances the performance of MLLMs and vision models on multimodal understanding and segmentation tasks.
Temporal Feature Matters: A Framework for Diffusion Model Quantization
However, unlike traditional models, diffusion models critically rely on the time-step for the multi-round denoising.
Variational Randomized Smoothing for Sample-Wise Adversarial Robustness
Randomized smoothing is a defensive technique to achieve enhanced robustness against adversarial examples which are small input perturbations that degrade the performance of neural network models.
GPT Sonograpy: Hand Gesture Decoding from Forearm Ultrasound Images via VLM
Large vision-language models (LVLMs), such as the Generative Pre-trained Transformer 4-omni (GPT-4o), are emerging multi-modal foundation models which have great potential as powerful artificial-intelligence (AI) assistance tools for a myriad of applications, including healthcare, industrial, and academic sectors.
Random Channel Ablation for Robust Hand Gesture Classification with Multimodal Biosignals
Notably, the proposed method is also robust to an increase in the number of missing channels compared to other methods.
Exploring Knowledge Transfer in Evolutionary Many-task Optimization: A Complex Network Perspective
The field of evolutionary many-task optimization (EMaTO) is increasingly recognized for its ability to streamline the resolution of optimization challenges with repetitive characteristics, thereby conserving computational resources.
OneDiff: A Generalist Model for Image Difference Captioning
This paper introduces the OneDiff model, a novel generalist approach that utilizes a robust vision-language model architecture, integrating a siamese image encoder with a Visual Delta Module.
ShapG: new feature importance method based on the Shapley value
In this paper, we proposed a new Explainable Artificial Intelligence (XAI) method called ShapG (Explanations based on Shapley value for Graphs) for measuring feature importance.
Explainable artificial intelligence
Explainable Artificial Intelligence (XAI)
+1
PVUW 2024 Challenge on Complex Video Understanding: Methods and Results
Moreover, we provide a new motion expression guided video segmentation dataset MeViS to study the natural language-guided video understanding in complex environments.
2nd Place Solution for MeViS Track in CVPR 2024 PVUW Workshop: Motion Expression guided Video Segmentation
Motion Expression guided Video Segmentation is a challenging task that aims at segmenting objects in the video based on natural language expressions with motion descriptions.
Instance Segmentation
Referring Video Object Segmentation
+5
Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs
In this paper, we propose VideoNIAH (Video Needle In A Haystack), a benchmark construction framework through synthetic video generation.
ME-Switch: A Memory-Efficient Expert Switching Framework for Large Language Models
LLM development involves pre-training a foundation model on massive data, followed by fine-tuning on task-specific data to create specialized experts.
Explore the Limits of Omni-modal Pretraining at Scale
We propose to build omni-modal intelligence, which is capable of understanding any modality and learning universal representations.
Ranked #194 on
Visual Question Answering
on MM-Vet
Efficient Differentially Private Fine-Tuning of Diffusion Models
The recent developments of Diffusion Models (DMs) enable generation of astonishingly high-quality synthetic samples.
OUS: Scene-Guided Dynamic Facial Expression Recognition
By analyzing the Rigid Cognitive Problem, OUS successfully understands the complex relationship between scene context and emotional expression, closely aligning with human emotional understanding in real-world scenarios.
Ranked #5 on
Dynamic Facial Expression Recognition
on FERV39k
Dynamic Facial Expression Recognition
Facial Expression Recognition
MiniCache: KV Cache Compression in Depth Dimension for Large Language Models
In this paper, we present a simple yet effective approach, called MiniCache, to compress the KV cache across layers from a novel depth perspective, significantly reducing the memory footprint for LLM inference.
Automated Loss function Search for Class-imbalanced Node Classification
Class-imbalanced node classification tasks are prevalent in real-world scenarios.
ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification
In terms of efficiency, ZipCache also showcases a $37. 3\%$ reduction in prefill-phase latency, a $56. 9\%$ reduction in decoding-phase latency, and a $19. 8\%$ reduction in GPU memory usage when evaluating LLaMA3-8B model with a input length of $4096$.
Fuse & Calibrate: A bi-directional Vision-Language Guided Framework for Referring Image Segmentation
Referring Image Segmentation (RIS) aims to segment an object described in natural language from an image, with the main challenge being a text-to-pixel correlation.
Cooperative Visual-LiDAR Extrinsic Calibration Technology for Intersection Vehicle-Infrastructure: A review
Camera and LiDAR, the bedrock sensors in autonomous driving, exhibit expansive applicability.
Networking Systems for Video Anomaly Detection: A Tutorial and Survey
The increasing utilization of surveillance cameras in smart cities, coupled with the surge of online video applications, has heightened concerns regarding public security and privacy protection, which propelled automated Video Anomaly Detection (VAD) into a fundamental research task within the Artificial Intelligence (AI) community.
Continued Pretraining for Domain Adaptation of Wav2vec2.0 in Automatic Speech Recognition for Elementary Math Classroom Settings
Creating Automatic Speech Recognition (ASR) systems that are robust and resilient to classroom conditions is paramount to the development of AI tools to aid teachers and students.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Pretrained Optimization Model for Zero-Shot Black Box Optimization
Zero-shot optimization involves optimizing a target task that was not seen during training, aiming to provide the optimal solution without or with minimal adjustments to the optimizer.
Self-Bootstrapped Visual-Language Model for Knowledge Selection and Question Answering
Motivated by the research of retrieval-augmented generation in the field of natural language processing, we use Dense Passage Retrieval (DPR) to retrieve related knowledge to help the model answer questions.
Calibration & Reconstruction: Deep Integrated Language for Referring Image Segmentation
In this paper, we introduce CRFormer, a model that iteratively calibrates multi-modal features in the transformer decoder.
Text Prompt with Normality Guidance for Weakly Supervised Video Anomaly Detection
Further, we propose a learnable text prompt mechanism with the assist of a normality visual prompt to further improve the matching accuracy of video event description text and video frames.
The Promises and Pitfalls of Using Language Models to Measure Instruction Quality in Education
Assessing instruction quality is a fundamental component of any improvement efforts in the education system.
SplatFace: Gaussian Splat Face Reconstruction Leveraging an Optimizable Surface
Our method is designed to simultaneously deliver both high-quality novel view rendering and accurate 3D mesh reconstructions.
SC-Tune: Unleashing Self-Consistent Referential Comprehension in Large Vision Language Models
In this paper, we present and delve into the self-consistency capability of LVLMs, a crucial aspect that reflects the models' ability to both generate informative captions for specific objects and subsequently utilize these captions to accurately re-identify the objects in a closed-loop process.
VL-Mamba: Exploring State Space Models for Multimodal Learning
The extensive experiments on diverse multimodal benchmarks with competitive performance show the effectiveness of our proposed VL-Mamba and demonstrate the great potential of applying state space models for multimodal learning tasks.
Ranked #179 on
Visual Question Answering
on MM-Vet
SuperLoRA: Parameter-Efficient Unified Adaptation of Multi-Layer Attention Modules
Low-rank adaptation (LoRA) and its variants are widely employed in fine-tuning large models, including large language models for natural language processing and diffusion models for computer vision.
Knowledge Condensation and Reasoning for Knowledge-based VQA
We condense the retrieved knowledge passages from two perspectives.
Self-Evaluation of Large Language Model based on Glass-box Features
The proliferation of open-source Large Language Models (LLMs) underscores the pressing need for evaluation methods.
Enhancing Instructional Quality: Leveraging Computer-Assisted Textual Analysis to Generate In-Depth Insights from Educational Artifacts
This paper explores the transformative potential of computer-assisted textual analysis in enhancing instructional quality through in-depth insights from educational artifacts.
An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4
Alternatively, other works have fine-tuned judge models based on open-source LLMs as the evaluator.
SynArtifact: Classifying and Alleviating Artifacts in Synthetic Images via Vision-Language Model
To alleviate artifacts and improve quality of synthetic images, we fine-tune Vision-Language Model (VLM) as artifact classifier to automatically identify and classify a wide range of artifacts and provide supervision for further optimizing generative models.
REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question Answering
By combining the improvements in both architecture and training, our proposed REAR can better utilize external knowledge by effectively perceiving the relevance of retrieved documents.
BASES: Large-scale Web Search User Simulation with Large Language Model based Agents
Due to the excellent capacities of large language models (LLMs), it becomes feasible to develop LLM-based agents for reliable user simulation.
CCFC++: Enhancing Federated Clustering through Feature Decorrelation
In federated clustering, multiple data-holding clients collaboratively group data without exchanging raw data.
Beyond Literal Descriptions: Understanding and Locating Open-World Objects Aligned with Human Intentions
To promote classic VG towards human intention interpretation, we propose a new intention-driven visual grounding (IVG) task and build a large-scale IVG dataset termed IntentionVG with free-form intention expressions.
Why Does Differential Privacy with Large Epsilon Defend Against Practical Membership Inference Attacks?
In practical applications, such a worst-case guarantee may be overkill: practical attackers may lack exact knowledge of (nearly all of) the private data, and our data set might be easier to defend, in some sense, than the worst-case data set.
M2ORT: Many-To-One Regression Transformer for Spatial Transcriptomics Prediction from Histopathology Images
To address this limit, we propose M2ORT, a many-to-one regression Transformer that can accommodate the hierarchical structure of the pathology images through a decoupled multi-scale feature extractor.
Operation Scheme Optimizations to Achieve Ultra-high Endurance (1010) in Flash Memory with Robust Reliabilities
Flash memory has been widely adopted as stand-alone memory and embedded memory due to its robust reliability.
CCFC: Bridging Federated Clustering and Contrastive Learning
Benefiting from representation learning, the clustering performance of CCFC even double those of the best baseline methods in some cases.
Temporal Adaptive RGBT Tracking with Modality Prompt
RGBT tracking has been widely used in various fields such as robotics, surveillance processing, and autonomous driving.
Ranked #12 on
Rgb-T Tracking
on RGBT210
Unveiling Parts Beyond Objects: Towards Finer-Granularity Referring Expression Segmentation
To foster future research into fine-grained visual grounding our benchmark RefCOCOm the MRES-32M dataset and model UniRES will be publicly available at https://github. com/Rubics-Xuan/MRES.
Signed Graph Neural Ordinary Differential Equation for Modeling Continuous-time Dynamics
Modeling continuous-time dynamics constitutes a foundational challenge, and uncovering inter-component correlations within complex systems holds promise for enhancing the efficacy of dynamic modeling.
Unveiling Parts Beyond Objects:Towards Finer-Granularity Referring Expression Segmentation
To foster future research into fine-grained visual grounding, our benchmark RefCOCOm, the MRES-32M dataset and model UniRES will be publicly available at https://github. com/Rubics-Xuan/MRES
ClusterDDPM: An EM clustering framework with Denoising Diffusion Probabilistic Models
In the E-step, we aim to derive a mixture of Gaussian priors for the subsequent M-step.
Efficient Stitchable Task Adaptation
In this work, we present a novel framework, Efficient Stitchable Task Adaptation (ESTA), to efficiently produce a palette of fine-tuned models that adhere to diverse resource constraints.
BAND-2k: Banding Artifact Noticeable Database for Banding Detection and Quality Assessment
The quality score of a banding image is generated by pooling the banding detection maps masked by the spatial frequency filters.
TFMQ-DM: Temporal Feature Maintenance Quantization for Diffusion Models
Remarkably, our quantization approach, for the first time, achieves model performance nearly on par with the full-precision model under 4-bit weight quantization.
Open-Vocabulary Video Anomaly Detection
Particularly, we devise a semantic knowledge injection module to introduce semantic knowledge from large language models for the detection task, and design a novel anomaly synthesis module to generate pseudo unseen anomaly videos with the help of large vision generation models for the classification task.
An Interdisciplinary Outlook on Large Language Models for Scientific Research
In this paper, we describe the capabilities and constraints of Large Language Models (LLMs) within disparate academic disciplines, aiming to delineate their strengths and limitations with precision.
Med-DANet V2: A Flexible Dynamic Architecture for Efficient Medical Volumetric Segmentation
As a pioneering work, a dynamic architecture network for medical volumetric segmentation (i. e. Med-DANet) has achieved a favorable accuracy and efficiency trade-off by dynamically selecting a suitable 2D candidate model from the pre-defined model bank for different slices.
Stabilizing Subject Transfer in EEG Classification with Divergence Estimation
Classification models for electroencephalogram (EEG) data show a large decrease in performance when evaluated on unseen test sub jects.
QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models
Additionally, an adaptive strategy is designed to autonomously determine the optimal number of sub-channels for channel disassembly.
EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models
In this paper, we introduce a data-free and parameter-efficient fine-tuning framework for low-bit diffusion models, dubbed EfficientDM, to achieve QAT-level performance with PTQ-like efficiency.
GLOBER: Coherent Non-autoregressive Video Generation via GLOBal Guided Video DecodER
Specifically, we propose a video auto-encoder, where a video encoder encodes videos into global features, and a video decoder, built on a diffusion model, decodes the global features and synthesizes video frames in a non-autoregressive manner.
Kid-Whisper: Towards Bridging the Performance Gap in Automatic Speech Recognition for Children VS. Adults
Recent advancements in Automatic Speech Recognition (ASR) systems, exemplified by Whisper, have demonstrated the potential of these systems to approach human-level performance given sufficient data.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
PhotoVerse: Tuning-Free Image Customization with Text-to-Image Diffusion Models
Personalized text-to-image generation has emerged as a powerful and sought-after tool, empowering users to create customized images based on their specific concepts and prompts.
GLS-CSC: A Simple but Effective Strategy to Mitigate Chinese STM Models' Over-Reliance on Superficial Clue
To address this issue, it is crucial to analyze and mitigate the influence of superficial clues on STM models.
Machine learning of network inference enhancement from noisy measurements
Inferring networks from observed time series data presents a clear glimpse into the interconnections among nodes.
FG-Net: Facial Action Unit Detection with Generalizable Pyramidal Features
The proposed FG-Net achieves a strong generalization ability for heatmap-based AU detection thanks to the generalizable and semantic-rich features extracted from the pre-trained generative model.
March in Chat: Interactive Prompting for Remote Embodied Referring Expression
Nevertheless, this poses more challenges than other VLN tasks since it requires agents to infer a navigation plan only based on a short instruction.
EAVL: Explicitly Align Vision and Language for Referring Image Segmentation
Our method harnesses the potential of the multi-modal features in the segmentation stage and aligns language features of different emphases with image features to achieve fine-grained text-to-pixel correlation.
AIDE: A Vision-Driven Multi-View, Multi-Modal, Multi-Tasking Dataset for Assistive Driving Perception
Driver distraction has become a significant cause of severe traffic accidents over the past decade.
Spatio-Temporal Domain Awareness for Multi-Agent Collaborative Perception
In this paper, we propose SCOPE, a novel collaborative perception framework that aggregates the spatio-temporal awareness characteristics across on-road agents in an end-to-end manner.
Towards Video Anomaly Retrieval from Video Anomaly Detection: New Benchmarks and Model
In this context, we propose a novel task called Video Anomaly Retrieval (VAR), which aims to pragmatically retrieve relevant anomalous videos by cross-modalities, e. g., language descriptions and synchronous audios.
Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation
In this study, we present the first analysis on the factual knowledge boundaries of LLMs and how retrieval augmentation affects LLMs on open-domain question answering (QA), with a bunch of important findings.
Perceptual Quality Assessment of Omnidirectional Audio-visual Signals
Omnidirectional videos (ODVs) play an increasingly important role in the application fields of medical, education, advertising, tourism, etc.
AIGCIQA2023: A Large-scale Image Quality Assessment Database for AI Generated Images: from the Perspectives of Quality, Authenticity and Correspondence
In this paper, in order to get a better understanding of the human visual preferences for AIGIs, a large-scale IQA database for AIGC is established, which is named as AIGCIQA2023.
Stitched ViTs are Flexible Vision Backbones
With extensive experiments on ImageNet-1K, ADE20K, COCO-Stuff-10K and NYUv2, SN-Netv2 demonstrates superior performance over SN-Netv1 on downstream dense predictions and shows strong ability as a flexible vision backbone, achieving great advantages in both training efficiency and deployment flexibility.
COSA: Concatenated Sample Pretrained Vision-Language Foundation Model
Due to the limited scale and quality of video-text training corpus, most vision-language foundation models employ image-text datasets for pretraining and primarily focus on modeling visually semantic representations while disregarding temporal semantic representations and correlations.
Ranked #1 on
TGIF-Frame
on TGIF-QA
(using extra training data)
Description-Enhanced Label Embedding Contrastive Learning for Text Classification
Then, we propose a novel Relation of Relation Learning Network (R2-Net) for text classification, in which text classification and R2 classification are treated as optimization targets.
VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset
Based on the proposed VAST-27M dataset, we train an omni-modality video-text foundational model named VAST, which can perceive and process vision, audio, and subtitle modalities from video, and better support various tasks including vision-text, audio-text, and multi-modal video-text tasks (retrieval, captioning and QA).
Ranked #1 on
Image Captioning
on COCO Captions
(SPICE metric, using extra
training data)
Rapid Plug-in Defenders
To address this challenge, this paper focuses on the Rapid Plug-in Defender (RaPiD) problem, aiming to rapidly counter adversarial perturbations without altering the deployed model.
ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst
We show that only language-paired two-modality data is sufficient to connect all modalities.
MMNet: Multi-Mask Network for Referring Image Segmentation
However, this task is challenging due to the distinct data properties between text and image, and the randomness introduced by diverse objects and unrestricted language expression.
VLAB: Enhancing Video Language Pre-training by Feature Adapting and Blending
Towards this goal, we propose a novel video-text pre-training method dubbed VLAB: Video Language pre-training by feature Adapting and Blending, which transfers CLIP representations to video pre-training tasks and develops unified video multimodal models for a wide range of video-text tasks.
Ranked #1 on
Visual Question Answering (VQA)
on MSVD-QA
(using extra training data)
CM-MaskSD: Cross-Modality Masked Self-Distillation for Referring Image Segmentation
Referring image segmentation (RIS) is a fundamental vision-language task that intends to segment a desired object from an image based on a given natural language expression.
Enhancing Vision-Language Pre-Training with Jointly Learned Questioner and Dense Captioner
In this paper, we propose a novel method called Joint QA and DC GEneration (JADE), which utilizes a pre-trained multimodal model and easily-crawled image-text pairs to automatically generate and filter large-scale VQA and dense captioning datasets.
TOME: A Two-stage Approach for Model-based Retrieval
Recently, model-based retrieval has emerged as a new paradigm in text retrieval that discards the index in the traditional retrieval model and instead memorizes the candidate corpora using model parameters.
PTQD: Accurate Post-Training Quantization for Diffusion Models
To address these challenges, we propose a unified formulation for the quantization noise and diffusion perturbed noise in the quantized denoising process.
Configurable Spatial-Temporal Hierarchical Analysis for Flexible Video Anomaly Detection
Since the anomaly set is complicated and unbounded, our STHA can adjust its detection ability to adapt to the human detection demands and the complexity degree of anomaly that happened in the history of a scene.
Robust Acoustic and Semantic Contextual Biasing in Neural Transducers for Speech Recognition
Prior approaches typically relied on subword encoders for encoding the bias phrases.
SLSG: Industrial Image Anomaly Detection by Learning Better Feature Embeddings and One-Class Classification
Focusing on more effective and comprehensive anomaly detection, we propose a network based on self-supervised learning and self-attentive graph convolution (SLSG) for anomaly detection.
Ranked #8 on
Anomaly Detection
on MVTec LOCO AD
B2Opt: Learning to Optimize Black-box Optimization with Little Budget
To achieve this, 1) drawing on the mechanism of genetic algorithm, we propose a deep neural network framework called B2Opt, which has a stronger representation of optimization strategies based on survival of the fittest; 2) B2Opt can utilize the cheap surrogate functions of the target task to guide the design of the efficient optimization strategies.
Med-Tuning: A New Parameter-Efficient Tuning Framework for Medical Volumetric Segmentation
Thus, it is of increasing importance to fine-tune pre-trained models for medical volumetric segmentation tasks in a both effective and parameter-efficient manner.
DECN: Evolution Inspired Deep Convolution Network for Black-box Optimization
We design a deep evolutionary convolution network (DECN) to realize the move from hand-designed EAs to automated EAs without manual interventions.
VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset
Different from widely-studied vision-language pretraining models, VALOR jointly models relationships of vision, audio and language in an end-to-end manner.
Ranked #1 on
Video Captioning
on VATEX
(using extra training data)
Video Event Restoration Based on Keyframes for Video Anomaly Detection
Video anomaly detection (VAD) is a significant computer vision problem.
SCMM: Calibrating Cross-modal Representations for Text-Based Person Search
This dual-pronged strategy enhances feature alignment and cross-modal correspondences, enabling accurate distinction of similar individuals while maintaining a streamlined dual-encoder architecture for real-time inference, which is essential for resource-limited sensors and IoT systems.
Dual-Attention Neural Transducers for Efficient Wake Word Spotting in Speech Recognition
We present dual-attention neural biasing, an architecture designed to boost Wake Words (WW) recognition and improve inference time latency on speech recognition tasks.
PROCTER: PROnunciation-aware ConTextual adaptER for personalized speech recognition in neural transducers
End-to-End (E2E) automatic speech recognition (ASR) systems used in voice assistants often have difficulties recognizing infrequent words personalized to the user, such as names and places.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Sounding Video Generator: A Unified Framework for Text-guided Sounding Video Generation
In this work, we concentrate on a rarely investigated problem of text guided sounding video generation and propose the Sounding Video Generator (SVG), a unified framework for generating realistic videos along with audio signals.
OmniAvatar: Geometry-Guided Controllable 3D Head Synthesis
We present OmniAvatar, a novel geometry-guided 3D head synthesis model trained from in-the-wild unstructured images that is capable of synthesizing diverse identity-preserved 3D heads with compelling dynamic details under full disentangled control over camera poses, facial expressions, head shapes, articulated neck and jaw poses.
AgileGAN3D: Few-Shot 3D Portrait Stylization by Augmented Transfer Learning
Capitalizing on the recent advancement of 3D-aware GAN models, we perform \emph{guided transfer learning} on a pretrained 3D GAN generator to produce multi-view-consistent stylized renderings.
Boosting Verified Training for Robust Image Classifications via Abstraction
Via abstraction, all perturbed images are mapped into intervals before feeding into neural networks for training.
Subjective and Objective Quality Assessment for in-the-Wild Computer Graphics Images
Computer graphics images (CGIs) are artificially generated by means of computer programs and are widely perceived under various scenarios, such as games, streaming media, etc.
MOSO: Decomposing MOtion, Scene and Object for Video Prediction
Experimental results demonstrate that our method achieves new state-of-the-art performance on five challenging benchmarks for video prediction and unconditional video generation: BAIR, RoboNet, KTH, KITTI and UCF101.
SMoA: Sparse Mixture of Adapters to Mitigate Multiple Dataset Biases
Recent studies reveal that various biases exist in different NLP tasks, and over-reliance on biases results in models' poor generalization ability and low adversarial robustness.
Graph-based Knowledge Distillation: A survey and experimental evaluation
It then provides a comprehensive summary of three types of Graph-based Knowledge Distillation methods, namely Graph-based Knowledge Distillation for deep neural networks (DKD), Graph-based Knowledge Distillation for GNNs (GKD), and Self-Knowledge Distillation based Graph-based Knowledge Distillation (SKD).
A novel efficient Multi-view traffic-related object detection framework
With the rapid development of intelligent transportation system applications, a tremendous amount of multi-view video data has emerged to enhance vehicle perception.
Tag-based annotation creates better avatars
Compared to direct annotation of labels, the proposed method: produces higher annotator agreements, causes machine learning to generates more consistent predictions, and only requires a marginal cost to add new rendering systems.
Fast Learnings of Coupled Nonnegative Tensor Decomposition Using Optimal Gradient and Low-rank Approximation
Tensor decomposition is a fundamental technique widely applied in signal processing, machine learning, and various other fields.
Generalized Video Anomaly Event Detection: Systematic Taxonomy and Comparison of Deep Models
Video Anomaly Detection (VAD) serves as a pivotal technology in the intelligent surveillance systems, enabling the temporal or spatial identification of anomalous events within videos.
A Survey on Efficient Training of Transformers
Recent advances in Transformers have come with a huge requirement on computing resources, highlighting the importance of developing efficient training techniques to make Transformer training faster, at lower cost, and to higher accuracy by the efficient use of computation and memory resources.
Planning-Assisted Context-Sensitive Autonomous Shepherding of Dispersed Robotic Swarms in Obstacle-Cluttered Environments
Robotic shepherding is a bio-inspired approach to autonomously guiding a swarm of agents towards a desired location.
Discover governing differential equations from evolving systems
Our proposal is competitive in identifying the change points and discovering governing differential equations in three hybrid systems and two switching linear systems.
LoTE-Animal: A Long Time-span Dataset for Endangered Animal Behavior Understanding
To break the deadlock, we present LoTE-Animal, a large-scale endangered animal dataset collected over 12 years, to foster the application of deep learning in rare species conservation.
Ranked #1 on
Action Recognition
on LoTE-Animal
BiViT: Extremely Compressed Binary Vision Transformers
To solve this, we propose Softmax-aware Binarization, which dynamically adapts to the data distribution and reduces the error caused by binarization.
