Search Results for author:
Found 172 papers, 98 papers with code
The Sogou-TIIC Speech Translation System for IWSLT 2018
Our final average result on speech translation is 31. 02 BLEU.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+6
VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generation
Speech large language models (LLMs) have emerged as a prominent research focus in speech processing.
COST: Contrastive One-Stage Transformer for Vision-Language Small Object Tracking
In this work, we propose COST, a contrastive one-stage transformer fusion framework for VL tracking, aiming to learn semantically consistent and unified VL representations.
RARE: Retrieval-Augmented Reasoning Modeling
Domain-specific intelligence demands specialized knowledge and sophisticated reasoning for problem-solving, posing significant challenges for large language models (LLMs) that struggle with knowledge hallucination and inadequate reasoning capabilities under constrained parameter budgets.
Learning to Instruct for Visual Instruction Tuning
We propose LIT, an advancement of visual instruction tuning (VIT).
4DGC: Rate-Aware 4D Gaussian Compression for Efficient Streamable Free-Viewpoint Video
Existing methods typically handle dynamic 3DGS representation and compression separately, neglecting motion information and the rate-distortion (RD) trade-off during training, leading to performance degradation and increased model redundancy.
ChatBEV: A Visual Language Model that Understands BEV Maps
Traffic scene understanding is essential for intelligent transportation systems and autonomous driving, ensuring safe and efficient vehicle operation.
FedMABench: Benchmarking Mobile Agents on Decentralized Heterogeneous User Data
FedMABench features 6 datasets with 30+ subsets, 8 federated algorithms, 10+ base models, and over 800 apps across 5 categories, providing a comprehensive framework for evaluating mobile agents across diverse environments.
Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases
Notably, open-source models like DeepSeek-R1 are narrowing the gap with proprietary systems, highlighting their potential to drive accessible and equitable advancements in healthcare.
DSVD: Dynamic Self-Verify Decoding for Faithful Generation in Large Language Models
The reliability of large language models remains a critical challenge, particularly due to their susceptibility to hallucinations and factual inaccuracies during text generation.
M^3Builder: A Multi-Agent System for Automated Machine Learning in Medical Imaging
In this paper, we make three contributions: (i) We present M3Builder, a novel multi-agent system designed to automate machine learning (ML) in medical imaging.
Contrast-Unity for Partially-Supervised Temporal Sentence Grounding
Temporal sentence grounding aims to detect event timestamps described by the natural language query from given untrimmed videos.
FedMobileAgent: Training Mobile Agents Using Decentralized Self-Sourced Data from Diverse Users
To tackle these challenges, we propose FedMobileAgent, a collaborative framework that trains mobile agents using self-sourced data from diverse users.
WanJuanSiLu: A High-Quality Open-Source Webtext Dataset for Low-Resource Languages
This paper introduces the open-source dataset WanJuanSiLu, designed to provide high-quality training corpora for low-resource languages, thereby advancing the research and development of multilingual models.
MedS$^3$: Towards Medical Small Language Models with Self-Evolved Slow Thinking
In this work, we present a deployable, small-scale medical language model, \mone, designed for long-chain reasoning in clinical tasks using a self-evolution paradigm.
Active Sampling for Node Attribute Completion on Graphs
Node attribute, a type of crucial information for graph analysis, may be partially or completely missing for certain nodes in real world applications.
Towards Omni-RAG: Comprehensive Retrieval-Augmented Generation for Large Language Models in Medical Applications
Large language models (LLMs) hold promise for addressing healthcare challenges but often generate hallucinations due to limited integration of medical knowledge.
A Knowledge-enhanced Pathology Vision-language Foundation Model for Cancer Diagnosis
To derive more nuanced image and text representations, we propose a novel knowledge-enhanced vision-language pre-training approach that integrates disease knowledge into the alignment within hierarchical semantic groups instead of unstructured image-text pairs.
VRVVC: Variable-Rate NeRF-Based Volumetric Video Compression
In this paper, we propose VRVVC, a novel end-to-end joint optimization variable-rate framework for volumetric video compression that achieves variable bitrates using a single model while maintaining superior RD performance.
Drawing the Line: Enhancing Trustworthiness of MLLMs Through the Power of Refusal
Multimodal large language models (MLLMs) excel at multimodal perception and understanding, yet their tendency to generate hallucinated or inaccurate responses undermines their trustworthiness.
Can Modern LLMs Act as Agent Cores in Radiology Environments?
Advancements in large language models (LLMs) have paved the way for LLM-based agent systems that offer enhanced accuracy and interpretability across various domains.
MRGen: Diffusion-based Controllable Data Engine for MRI Segmentation towards Unannotated Modalities
Medical image segmentation has recently demonstrated impressive progress with deep neural networks, yet the heterogeneous modalities and scarcity of mask annotations limit the development of segmentation models on unannotated modalities.
Towards Universal Soccer Video Understanding
As a globally celebrated sport, soccer has attracted widespread interest from fans all over the world.
LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant
With the rapid advancement of multimodal information retrieval, increasingly complex retrieval tasks have emerged.
MambaTrack: Exploiting Dual-Enhancement for Night UAV Tracking
Night unmanned aerial vehicle (UAV) tracking is impeded by the challenges of poor illumination, with previous daylight-optimized methods demonstrating suboptimal performance in low-light conditions, limiting the utility of UAV applications.
Task-Aware Harmony Multi-Task Decision Transformer for Offline Reinforcement Learning
The purpose of offline multi-task reinforcement learning (MTRL) is to develop a unified policy applicable to diverse tasks without the need for online environmental interaction.
ReflecTool: Towards Reflection-Aware Tool-Augmented Clinical Agents
Large Language Models (LLMs) have shown promising potential in the medical domain, assisting with tasks like clinical note generation and patient communication.
Synthesizing Post-Training Data for LLMs through Multi-Agent Simulation
Post-training is essential for enabling large language models (LLMs) to follow human instructions.
Data Quality Control in Federated Instruction-tuning of Large Language Models
To address this gap, we propose a new framework of federated instruction tuning of LLMs with data quality control (FedDQC), which measures data quality to facilitate the subsequent filtering and hierarchical training processes.
KnowledgeSG: Privacy-Preserving Synthetic Text Generation with Knowledge Distillation from Server
The success of large language models (LLMs) facilitate many parties to fine-tune LLMs on their own private data.
CliMedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models in Clinical Scenarios
With the proliferation of Large Language Models (LLMs) in diverse domains, there is a particular need for unified evaluation standards in clinical medical scenarios, where models need to be examined very thoroughly.
LoRKD: Low-Rank Knowledge Decomposition for Medical Foundation Models
To accomplish the above objective, we propose a novel framework named Low-Rank Knowledge Decomposition (LoRKD), which explicitly separates gradients from different tasks by incorporating low-rank expert modules and efficient knowledge separation convolution.
3D Medical Imaging Segmentation
Medical Image Classification
Towards Underwater Camouflaged Object Tracking: Benchmark and Baselines
Based on the proposed dataset, this paper first comprehensively evaluates current advanced visual object tracking methods and SAM- and SAM2-based trackers in challenging underwater environments.
Leveraging Unstructured Text Data for Federated Instruction Tuning of Large Language Models
Federated instruction tuning enables multiple clients to collaboratively fine-tune a shared large language model (LLM) that can follow humans' instructions without directly sharing raw data.
Self-supervised Anomaly Detection Pretraining Enhances Long-tail ECG Diagnosis
Current computer-aided ECG diagnostic systems struggle with the underdetection of rare but critical cardiac anomalies due to the imbalanced nature of ECG datasets.
Towards Evaluating and Building Versatile Large Language Models for Medicine
To promote further advancements in the application of LLMs to clinical challenges, we have made the MedS-Ins dataset fully accessible and invite the research community to contribute to its expansion. Additionally, we have launched a dynamic leaderboard for MedS-Bench, which we plan to regularly update the test set to track progress and enhance the adaptation of general LLMs to the medical domain.
MegaFusion: Extend Diffusion Models towards Higher-resolution Image Generation without Further Tuning
Diffusion models have emerged as frontrunners in text-to-image generation, but their fixed image resolution during training often leads to challenges in high-resolution image generation, such as semantic deviations and object replication.
Med-PMC: Medical Personalized Multi-modal Consultation with a Proactive Ask-First-Observe-Next Paradigm
The application of the Multi-modal Large Language Models (MLLMs) in medical clinical scenarios remains underexplored.
Decoding Linguistic Representations of Human Brain
Language, as an information medium created by advanced organisms, has always been a concern of neuroscience regarding how it is represented in the brain.
AutoRG-Brain: Grounded Report Generation for Brain MRI
To address these challenges, we initiate a series of work on grounded Automatic Report Generation (AutoRG), starting from the brain MRI interpretation system, which supports the delineation of brain structures, the localization of anomalies, and the generation of well-organized findings.
Reconstruct the Pruned Model without Any Retraining
Structured pruning is a promising hardware-friendly compression technique for large language models (LLMs), which is expected to be retraining-free to avoid the enormous retraining cost.
HPC: Hierarchical Progressive Coding Framework for Volumetric Video
Volumetric video based on Neural Radiance Field (NeRF) holds vast potential for various 3D applications, but its substantial data volume poses significant challenges for compression and transmission.
Reprogramming Distillation for Medical Foundation Models
However, due to the gap between pre-training tasks (or modalities) and downstream tasks (or modalities), the real-world computation and speed constraints, it might not be straightforward to apply medical foundation models in the downstream scenarios.
MatchTime: Towards Automatic Soccer Game Commentary Generation
Soccer is a globally popular sport with a vast audience, in this paper, we consider constructing an automatic soccer game commentary model to improve the audiences' viewing experience.
MedCare: Advancing Medical LLMs through Decoupling Clinical Alignment and Knowledge Aggregation
Based on this two-stage paradigm, we proposed a Medical LLM through decoupling Clinical Alignment and Knowledge Aggregation (MedCare), which is designed to achieve state-of-the-art (SOTA) performance on over 20 medical tasks, as well as SOTA results on specific medical alignment tasks.
RaTEScore: A Metric for Radiology Report Generation
This paper introduces a novel, entity-aware metric, termed as Radiological Report (Text) Evaluation (RaTEScore), to assess the quality of medical reports generated by AI models.
Self-Localized Collaborative Perception
However, existing collaborative perception systems heavily rely on precise localization systems to establish a consistent spatial coordinate system between agents.
Towards an End-to-End Framework for Invasive Brain Signal Decoding with Large Language Models
In this paper, we introduce a groundbreaking end-to-end (E2E) framework for decoding invasive brain signals, marking a significant advancement in the field of speech neuroprosthesis.
Emerging Safety Attack and Defense in Federated Instruction Tuning of Large Language Models
Federated learning (FL) enables multiple parties to collaboratively fine-tune an large language model (LLM) without the need of direct data sharing.
Exploring Training on Heterogeneous Data with Mixture of Low-rank Adapters
Training a unified model to take multiple targets into account is a trend towards artificial general intelligence.
Few-Shot Anomaly Detection via Category-Agnostic Registration Learning
At test time, an image and its corresponding support set, consisting of a few normal images from the same category, are supplied, and anomalies are identified by comparing the registered features of the test image to its corresponding support image features.
Diversified Batch Selection for Training Acceleration
The remarkable success of modern machine learning models on large datasets often demands extensive training time and resource consumption.
FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Models
Addressing this, we propose FedLLM-Bench, which involves 8 training methods, 4 training datasets, and 6 evaluation metrics, to offer a comprehensive testbed for the FedLLM community.
WebUOT-1M: Advancing Deep Underwater Object Tracking with A Million-Scale Benchmark
Most existing trackers are tailored for open-air environments, leading to performance degradation when applied to UOT due to domain gaps.
TAIA: Large Language Models are Out-of-Distribution Data Learners
However, in certain specialized domains, such as healthcare or harmless content generation, it is nearly impossible to obtain a large volume of high-quality data that matches the downstream distribution.
Domain-Inspired Sharpness-Aware Minimization Under Domain Shifts
This paper presents a Domain-Inspired Sharpness-Aware Minimization (DISAM) algorithm for optimization under domain shifts.
Federated Learning with Bilateral Curation for Partially Class-Disjoint Data
Partially class-disjoint data (PCDD), a common yet under-explored data formation where each client contributes a part of classes (instead of all classes) of samples, severely challenges the performance of federated algorithms.
Federated Learning under Partially Class-Disjoint Data via Manifold Reshaping
Statistical heterogeneity severely limits the performance of federated learning (FL), motivating several explorations e. g., FedProx, MOON and FedDyn, to alleviate this problem.
Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimization
However, the local loss landscapes may not accurately reflect the flatness of global loss landscape in heterogeneous environments; as a result, minimizing local sharpness and calculating perturbations on client data might not align the efficacy of SAM in FL with centralized training.
HarmoDT: Harmony Multi-Task Decision Transformer for Offline Reinforcement Learning
However, variations in task content and complexity pose significant challenges in policy formulation, necessitating judicious parameter sharing and management of conflicting gradients for optimal policy performance.
Mitigating Noisy Correspondence by Geometrical Structure Consistency Learning
Prior approaches to leverage such data mainly consider the application of uni-modal noisy label learning without amending the impact on both cross-modal and intra-modal geometrical structures in multimodal learning.
Q-value Regularized Transformer for Offline Reinforcement Learning
Recent advancements in offline reinforcement learning (RL) have underscored the capabilities of Conditional Sequence Modeling (CSM), a paradigm that learns the action distribution based on history trajectory and target returns for each state.
Language-Driven Interactive Traffic Trajectory Generation
To produce interactive traffic trajectories, we propose a code-to-trajectory decoder with interaction-aware feature aggregation that synergizes vehicle interactions with the environmental map and the vehicle moves.
JointRF: End-to-End Joint Optimization for Dynamic Neural Radiance Field Representation and Compression
Neural Radiance Field (NeRF) excels in photo-realistically static scenes, inspiring numerous efforts to facilitate volumetric videos.
Awesome Multi-modal Object Tracking
To leverage more modalities, some recent efforts have been made to learn a unified visual object tracking model for any modality.
Fair Evaluation of Federated Learning Algorithms for Automated Breast Density Classification: The Results of the 2022 ACR-NCI-NVIDIA Federated Learning Challenge
The correct interpretation of breast density is important in the assessment of breast cancer risk.
Robust Collaborative Perception without External Localization and Clock Devices
To achieve this spatial-temporal alignment, traditional methods depend on external devices to provide localization and clock signals.
Low-Rank Knowledge Decomposition for Medical Foundation Models
The popularity of large-scale pre-training has promoted the development of medical foundation models.
RadGenome-Chest CT: A Grounded Vision-Language Dataset for Chest CT Analysis
We believe that RadGenome-Chest CT can significantly advance the development of multimodal medical foundation models, by training to generate texts based on given segmentation regions, which is unattainable with previous relevant datasets.
DENOISER: Rethinking the Robustness for Open-Vocabulary Action Recognition
The denoised text classes help OVAR models classify visual samples more accurately; in return, classified visual samples help better denoising.
Ethical-Lens: Curbing Malicious Usages of Open-Source Text-to-Image Models
The burgeoning landscape of text-to-image models, exemplified by innovations such as Midjourney and DALLE 3, has revolutionized content creation across diverse sectors.
Knowledge-enhanced Visual-Language Pretraining for Computational Pathology
In this paper, we consider the problem of visual representation learning for computational pathology, by exploiting large-scale image-text pairs gathered from public resources, along with the domain-specific knowledge in pathology.
MING-MOE: Enhancing Medical Multi-Task Learning in Large Language Models with Sparse Mixture of Low-Rank Adapter Experts
Large language models like ChatGPT have shown substantial progress in natural language understanding and generation, proving valuable across various disciplines, including the medical field.
Anomaly Detection in Electrocardiograms: Advancing Clinical Diagnosis Through Self-Supervised Learning
We introduce a novel self-supervised learning framework for ECG AD, utilizing a vast dataset of normal ECGs to autonomously detect and localize cardiac anomalies.
ReMamber: Referring Image Segmentation with Mamba Twister
In this paper, we propose ReMamber, a novel RIS architecture that integrates the power of Mamba with a multi-modal Mamba Twister block.
M$^3$AV: A Multimodal, Multigenre, and Multipurpose Audio-Visual Academic Lecture Dataset
Although multiple academic video datasets have been constructed and released, few of them support both multimodal content recognition and understanding tasks, which is partially due to the lack of high-quality human annotations.
Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images
Recent advancements in large-scale visual-language pre-trained models have led to significant progress in zero-/few-shot anomaly detection within natural image domains.
Audio-Visual Segmentation via Unlabeled Frame Exploitation
NFs, temporally adjacent to the labeled frame, often contain rich motion information that assists in the accurate localization of sounding objects.
Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulator
Large Language Models (LLMs) have demonstrated remarkable proficiency in human interactions, yet their application within the medical field remains insufficiently explored.
Decentralized and Lifelong-Adaptive Multi-Agent Collaborative Learning
In this paper, we propose DeLAMA, a decentralized multi-agent lifelong collaborative learning algorithm with dynamic collaboration graphs.
Enhancing Data Quality in Federated Fine-Tuning of Foundation Models
In the current landscape of foundation model training, there is a significant reliance on public domain data, which is nearing exhaustion according to recent research.
Post-decoder Biasing for End-to-End Speech Recognition of Multi-turn Medical Interview
End-to-end (E2E) approach is gradually replacing hybrid models for automatic speech recognition (ASR) tasks.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Leveraging Diverse Modeling Contexts with Collaborating Learning for Neural Machine Translation
Autoregressive (AR) and Non-autoregressive (NAR) models are two types of generative models for Neural Machine Translation (NMT).
Towards Building Multilingual Language Model for Medicine
The development of open-source, multilingual medical language models can benefit a wide, linguistically diverse audience from different regions.
M2K-VDG: Model-Adaptive Multimodal Knowledge Anchor Enhanced Video-grounded Dialogue Generation
Video-grounded dialogue generation (VDG) requires the system to generate a fluent and accurate answer based on multimodal knowledge.
DictLLM: Harnessing Key-Value Data Structures with Large Language Models for Enhanced Medical Diagnostics
Structured data offers a sophisticated mechanism for the organization of information.
OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learning
Trained on massive publicly available data, large language models (LLMs) have demonstrated tremendous success across various fields.
Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agents
Furthermore, to unleash the potential of extensive high-quality digital assets, ChatSim employs a novel multi-camera lighting estimation method to achieve scene-consistent assets' rendering.
Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation
Drawing from the sociological insight that acknowledging all parties' concerns is a key factor in shaping human values, this paper proposes a novel direction to align LLMs by themselves: social scene simulation.
An Extensible Framework for Open Heterogeneous Collaborative Perception
In this paper, we introduce a new open heterogeneous problem: how to accommodate continually emerging new heterogeneous agent types into collaborative perception, while ensuring high perception performance and low integration cost?
FedRSU: Federated Learning for Scene Flow Estimation on Roadside Units
In this paper, we introduce FedRSU, an innovative federated learning framework for self-supervised scene flow estimation.
MM-SAP: A Comprehensive Benchmark for Assessing Self-Awareness of Multimodal Large Language Models in Perception
We believe that these hallucinations are partially due to the models' struggle with understanding what they can and cannot perceive from images, a capability we refer to as self-awareness in perception.
Intelligent Grimm - Open-ended Visual Storytelling via Latent Diffusion Models
Generative models have recently exhibited exceptional capabilities in text-to-image generation but still struggle to generate image sequences coherently.
One Model to Rule them All: Towards Universal Segmentation for Medical Images with Text Prompts
Our main contributions are three folds: (i) for dataset construction, we construct the first multi-modal knowledge tree on human anatomy, including 6502 anatomical terminologies; Then, we build up the largest and most comprehensive segmentation dataset for training, by collecting over 22K 3D medical image scans from72 segmentation datasets, across 497 classes, with careful standardization on both image scans and label space; (ii) for architecture design, we propose to inject medical knowledge into a text encoder via contrastive learning, and then formulate a universal segmentation model, that can be prompted by feeding in medical terminologies in text form; (iii) As a result, we have trained SAT-Nano (110M parameters) and SAT-Pro (447M parameters), demonstrating superior or comparable performance to 72 specialist models, i. e., nnU-Nets, U-Mamba or SwinUNETR, trained on each dataset/subsets.
Large-scale Long-tailed Disease Diagnosis on Radiology Images
Developing a generalist radiology diagnosis system can greatly enhance clinical diagnostics.
Multi-Sentence Grounding for Long-term Instructional Video
In this paper, we aim to establish an automatic, scalable pipeline for denoising the large-scale instructional dataset and construct a high-quality video-text dataset with multiple descriptive steps supervision, named HowToStep.
MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models
The emergence of various medical large language models (LLMs) in the medical domain has highlighted the need for unified evaluation standards, as manual evaluation of LLMs proves to be time-consuming and labor-intensive.
Hypergraph Transformer for Semi-Supervised Classification
Hypergraphs play a pivotal role in the modelling of data featuring higher-order relations involving more than two entities.
UniChest: Conquer-and-Divide Pre-training for Multi-Source Chest X-Ray Classification
Vision-Language Pre-training (VLP) that utilizes the multi-modal information to promote the training efficiency and effectiveness, has achieved great success in vision recognition of natural domains and shown promise in medical imaging diagnosis for the Chest X-Rays (CXRs).
Fake It Till Make It: Federated Learning with Consensus-Oriented Generation
FedCOG consists of two key components at the client side: complementary data generation, which generates data extracted from the shared global model to complement the original dataset, and knowledge-distillation-based model training, which distills knowledge from global model to local model based on the generated data to mitigate over-fitting the original heterogeneous dataset.
Federated Learning Empowered by Generative Content
In this paper, we propose a novel FL framework termed FedGC, designed to mitigate data heterogeneity issues by diversifying private data with generative content.
Combating Representation Learning Disparity with Geometric Harmonization
Self-supervised learning (SSL) as an effective paradigm of representation learning has achieved tremendous success on various curated datasets in diverse scenarios.
Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosis
Driven by the large foundation models, the development of artificial intelligence has witnessed tremendous progress lately, leading to a surge of general interest from the public.
Improving the Reliability of Large Language Models by Leveraging Uncertainty-Aware In-Context Learning
In this study, we introduce an uncertainty-aware in-context learning framework to empower the model to enhance or reject its output in response to uncertainty.
MSG-BART: Multi-granularity Scene Graph-Enhanced Encoder-Decoder Language Model for Video-grounded Dialogue Generation
Generating dialogue grounded in videos requires a high level of understanding and reasoning about the visual scenes in the videos.
UniBrain: Universal Brain MRI Diagnosis with Hierarchical Knowledge-enhanced Pre-training
Magnetic resonance imaging~(MRI) have played a crucial role in brain disease diagnosis, with which a range of computer-aided artificial intelligence methods have been proposed.
An Automatic Evaluation Framework for Multi-turn Medical Consultations Capabilities of Large Language Models
A medical consultation training set is further constructed to improve the consultation ability of LLMs.
AttrSeg: Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregation
The results show the superior performance of attribute decomposition-aggregation.
LibriSQA: A Novel Dataset and Framework for Spoken Question Answering with Large Language Models
While Large Language Models (LLMs) have demonstrated commendable performance across a myriad of domains and tasks, existing LLMs still exhibit a palpable deficit in handling multimodal functionalities, especially for the Spoken Question Answering (SQA) task which necessitates precise alignment and deep interaction between speech and text features.
Auxiliary Tasks Benefit 3D Skeleton-based Human Motion Prediction
To work with auxiliary tasks, we propose a novel auxiliary-adapted transformer, which can handle incomplete, corrupted motion data and achieve coordinate recovery via capturing spatial-temporal dependencies.
Ranked #4 on
Human Pose Forecasting
on Human3.6M
Bag of Tricks for Long-Tailed Multi-Label Classification on Chest X-Rays
Clinical classification of chest radiography is particularly challenging for standard machine learning algorithms due to its inherent long-tailed and multi-label nature.
Multi-Scale Memory Comparison for Zero-/Few-Shot Anomaly Detection
Anomaly detection has gained considerable attention due to its broad range of applications, particularly in industrial defect detection.
Joint-Relation Transformer for Multi-Person Motion Prediction
Multi-person motion prediction is a challenging problem due to the dependency of motion on both individual past movements and interactions with other people.
Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Data
In this study, we aim to initiate the development of Radiology Foundation Model, termed as RadFM.
Multi-scale Cross-restoration Framework for Electrocardiogram Anomaly Detection
To address this challenge, this paper introduces a novel multi-scale cross-restoration framework for ECG anomaly detection and localization that considers both local and global ECG characteristics.
Balanced Destruction-Reconstruction Dynamics for Memory-replay Class Incremental Learning
By dynamically manipulating the gradient during training based on these factors, BDR can effectively alleviate knowledge destruction and improve knowledge reconstruction.
Audio-aware Query-enhanced Transformer for Audio-Visual Segmentation
The goal of the audio-visual segmentation (AVS) task is to segment the sounding objects in the video frames using audio cues.
All in One: Exploring Unified Vision-Language Tracking with Multi-Modal Alignment
This approach achieves feature integration in a unified backbone, removing the need for carefully-designed fusion modules and resulting in a more effective and efficient VL tracking framework.
Multi-Modal Prototypes for Open-World Semantic Segmentation
To be specific, unlike the straightforward combination of bi-modal clues, we decompose the high-level language information as multi-aspect prototypes and aggregate the low-level visual information as more semantic prototypes, on basis of which, a fine-grained complementary fusion makes the multi-modal prototypes more powerful and accurate to promote the prediction.
Boost Video Frame Interpolation via Motion Adaptation
Video frame interpolation (VFI) is a challenging task that aims to generate intermediate frames between two consecutive frames in a video.
Zero-shot Composed Text-Image Retrieval
In this paper, we consider the problem of composed image retrieval (CIR), it aims to train a model that can fuse multi-modal information, e. g., text and images, to accurately retrieve images that match the query, extending the user's expression ability.
Ranked #6 on
Zero-Shot Composed Image Retrieval (ZS-CIR)
on CIRR
Exploring Effective Mask Sampling Modeling for Neural Image Compression
Specifically, Cube Mask Sampling Module (CMSM) is proposed to apply both spatial and channel mask sampling modeling to image compression in the pre-training stage.
Intelligent Grimm -- Open-ended Visual Storytelling via Latent Diffusion Models
Generative models have recently exhibited exceptional capabilities in text-to-image generation, but still struggle to generate image sequences coherently.
FedDisco: Federated Learning with Discrepancy-Aware Collaboration
However, based on our empirical observations and theoretical analysis, we find that the dataset size is not optimal and the discrepancy between local and global category distributions could be a beneficial and complementary indicator for determining aggregation weights.
PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering
Medical Visual Question Answering (MedVQA) presents a significant opportunity to enhance diagnostic accuracy and healthcare delivery by leveraging artificial intelligence to interpret and answer questions based on medical images.
Ranked #1 on
Medical Visual Question Answering
on PMC-VQA
PMC-LLaMA: Towards Building Open-source Language Models for Medicine
Our contributions are threefold: (i) we systematically investigate the process of adapting a general-purpose foundation language model towards medical domain, this involves data-centric knowledge injection through the integration of 4. 8M biomedical academic papers and 30K medical textbooks, as well as comprehensive fine-tuning for alignment with domain-specific instructions; (ii) we contribute a large-scale, comprehensive dataset for instruction tuning.
Collaboration Helps Camera Overtake LiDAR in 3D Detection
Camera-only 3D detection provides an economical solution with a simple configuration for localizing objects in 3D space compared to LiDAR-based detection systems.
Multi-modal Prompting for Low-Shot Temporal Action Localization
In this paper, we consider the problem of temporal action localization under low-shot (zero-shot & few-shot) scenario, with the goal of detecting and classifying the action instances from arbitrary categories within some untrimmed videos, even not seen at training time.
EqMotion: Equivariant Multi-agent Motion Prediction with Invariant Interaction Reasoning
In motion prediction tasks, maintaining motion equivariance under Euclidean geometric transformations and invariance of agent interaction is a critical and fundamental principle.
Ranked #1 on
Human Pose Forecasting
on HARPER
Leapfrog Diffusion Model for Stochastic Trajectory Prediction
The core of the proposed LED is to leverage a trainable leapfrog initializer to directly learn an expressive multi-modal distribution of future trajectories, which skips a large number of denoising steps, significantly accelerating inference speed.
Boundary-aware Supervoxel-level Iteratively Refined Interactive 3D Image Segmentation with Multi-agent Reinforcement Learning
Interactive segmentation has recently been explored to effectively and efficiently harvest high-quality segmentation masks by iteratively incorporating user hints.
DiffusionSeg: Adapting Diffusion Towards Unsupervised Object Discovery
However, the challenges exist as there is one structural difference between generative and discriminative models, which limits the direct use.
TBP-Former: Learning Temporal Bird's-Eye-View Pyramid for Joint Perception and Prediction in Vision-Centric Autonomous Driving
Second, a spatial-temporal pyramid transformer is introduced to comprehensively extract multi-scale BEV features and predict future BEV states with the support of spatial-temporal priors.
Ranked #2 on
Bird's-Eye View Semantic Segmentation
on nuScenes
(IoU ped - 224x480 - Vis filter. - 100x100 at 0.5 metric)
DR2: Diffusion-based Robust Degradation Remover for Blind Face Restoration
However, it is expensive and infeasible to include every type of degradation to cover real-world cases in the training data.
PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents
Foundation models trained on large-scale dataset gain a recent surge in CV and NLP.
Ranked #3 on
Medical Visual Question Answering
on PMC-VQA
Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Images
While multi-modal foundation models pre-trained on large-scale data have been successful in natural language understanding and vision recognition, their use in medical domains is still limited due to the fine-grained nature of medical tasks and the high demand for domain knowledge.
K-Diag: Knowledge-enhanced Disease Diagnosis in Radiographic Imaging
In this paper, we consider the problem of disease diagnosis.
Knowledge-aware Bayesian Co-attention for Multimodal Emotion Recognition
Multimodal emotion recognition is a challenging research area that aims to fuse different modalities to predict human emotion.
Long-Tailed Partial Label Learning via Dynamic Rebalancing
The straightforward combination of LT and PLL, i. e., LT-PLL, suffers from a fundamental dilemma: LT methods build upon a given class distribution that is unavailable in PLL, and the performance of PLL is severely influenced in long-tailed context.
Open-vocabulary Object Segmentation with Diffusion Models
The goal of this paper is to extract the visual-language correspondence from a pre-trained text-to-image diffusion model, in the form of segmentation map, i. e., simultaneously generating images and segmentation masks for the corresponding visual entities described in the text prompt.
Integrating features from lymph node stations for metastatic lymph node detection
The branch targets to solve a closely related task on the LN station level, i. e., classifying whether an LN station contains metastatic LN or not, so as to learn representations for LN stations.
MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiology
In this paper, we consider enhancing medical visual-language pre-training (VLP) with domain-specific knowledge, by exploiting the paired image-text reports from the radiological daily practice.
MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training for X-ray Diagnosis
In this paper, we consider enhancing medical visual-language pre-training (VLP) with domain-specific knowledge, by exploiting the paired image-text reports from the radiological daily practice.
Federated Domain Generalization With Generalization Adjustment
Federated Domain Generalization (FedDG) attempts to learn a global model in a privacy-preserving manner that generalizes well to new clients possibly with domain shift.
Distilling Vision-Language Pre-training to Collaborate with Weakly-Supervised Temporal Action Localization
And as a result, the dual-branch complementarity is effectively fused to promote a strong alliance.
Weakly-supervised Temporal Action Localization
Weakly Supervised Temporal Action Localization
FedSkip: Combatting Statistical Heterogeneity with Federated Skip Aggregation
However, in addition to previous explorations for improvement in federated averaging, our analysis shows that another critical bottleneck is the poorer optima of client models in more heterogeneous conditions.
Robust Collaborative 3D Object Detection in Presence of Pose Errors
Collaborative 3D object detection exploits information exchange among multiple agents to enhance accuracy of object detection in presence of sensor impairments such as occlusion.
Unrolled Graph Learning for Multi-Agent Collaboration
To fill this gap, we propose a distributed multi-agent learning model inspired by human collaboration, in which the agents can autonomously detect suitable collaborators and refer to collaborators' model for better performance.
Open-vocabulary Semantic Segmentation with Frozen Vision-Language Models
When trained at a sufficient scale, self-supervised learning has exhibited a notable ability to solve a wide range of visual or language understanding tasks.
Number-Adaptive Prototype Learning for 3D Point Cloud Semantic Segmentation
Instead of learning a single prototype for each class, in this paper, we propose to use an adaptive number of prototypes to dynamically describe the different point patterns within a semantic class.
Ranked #19 on
3D Semantic Segmentation
on SemanticKITTI
A Simple Plugin for Transforming Images to Arbitrary Scales
Existing models on super-resolution often specialized for one scale, fundamentally limiting their use in practical scenarios.
Low-Light Video Enhancement with Synthetic Event Guidance
Low-light video enhancement (LLVE) is an important yet challenging task with many applications such as photographing and autonomous driving.
Multi-level Fusion of Wav2vec 2.0 and BERT for Multimodal Emotion Recognition
The research and applications of multimodal emotion recognition have become increasingly popular recently.
Collaborative Uncertainty Benefits Multi-Agent Multi-Modal Trajectory Forecasting
Further, we apply the proposed framework to current SOTA multi-agent multi-modal forecasting systems as a plugin module, which enables the SOTA systems to 1) estimate the uncertainty in the multi-agent multi-modal trajectory forecasting task; 2) rank the multiple predictions and select the optimal one based on the estimated uncertainty.
Nextformer: A ConvNeXt Augmented Conformer For End-To-End Speech Recognition
To the best of our knowledge, the proposed Nextformer model achieves SOTA results on AISHELL-1(CER 4. 06%) and WenetSpeech(CER 7. 56%/11. 29%).
K-Space Transformer for Undersampled MRI Reconstruction
This paper considers the problem of undersampled MRI reconstruction.
Contrastive Learning with Boosted Memorization
Different from previous works, we explore this direction from an alternative perspective, i. e., the data perspective, and propose a novel Boosted Contrastive Learning (BCL) method.
Self-Supervised Masking for Unsupervised Anomaly Detection and Localization
To extend the reconstruction-based anomaly detection architecture to the localized anomalies, we propose a self-supervised learning approach through random masking and then restoring, named Self-Supervised Masking (SSM) for unsupervised anomaly detection and localization.
Handwritten Mathematical Expression Recognition via Attention Aggregation based Bi-directional Mutual Learning
Handwritten mathematical expression recognition aims to automatically generate LaTeX sequences from given images.
Self-supervised Tumor Segmentation through Layer Decomposition
In this paper, we target self-supervised representation learning for zero-shot tumor segmentation.
Multiscale Spatio-Temporal Graph Neural Networks for 3D Skeleton-Based Motion Prediction
The core of MST-GNN is a multiscale spatio-temporal graph that explicitly models the relations in motions at various spatial and temporal scales.
Cooperative Learning for Noisy Supervision
Learning with noisy labels has gained the enormous interest in the robust deep learning area.
MS-KD: Multi-Organ Segmentation with Multiple Binary-Labeled Datasets
A novel Multi-teacher Single-student Knowledge Distillation (MS-KD) framework is proposed, where the teacher models are pre-trained single-organ segmentation networks, and the student model is a multi-organ segmentation network.
A Fourier-based Framework for Domain Generalization
Modern deep neural networks suffer from performance degradation when evaluated on testing data under different distributions from training data.
H2O: A Benchmark for Visual Human-human Object Handover Analysis
Besides, we also report the hand and object pose errors with existing baselines and show that the dataset can serve as the video demonstrations for robot imitation learning on the handover task.
Collaborative Label Correction via Entropy Thresholding
Deep neural networks (DNNs) have the capacity to fit extremely noisy labels nonetheless they tend to learn data with clean labels first and then memorize those with noisy labels.
Divide and Conquer for Single-Frame Temporal Action Localization
Single-frame temporal action localization (STAL) aims to localize actions in untrimmed videos with only one timestamp annotation for each action instance.
FGraDA: A Dataset and Benchmark for Fine-Grained Domain Adaptation in Machine Translation
Previous research for adapting a general neural machine translation (NMT) model into a specific domain usually neglects the diversity in translation within the same domain, which is a core problem for domain adaptation in real-world scenarios.
Point-Level Temporal Action Localization: Bridging Fully-supervised Proposals to Weakly-supervised Losses
Point-Level temporal action localization (PTAL) aims to localize actions in untrimmed videos with only one timestamp annotation for each action instance.
Ranked #3 on
Weakly Supervised Action Localization
on BEOID
Privileged Knowledge Distillation for Online Action Detection
Knowledge distillation is employed to transfer the privileged information from the offline teacher to the online student.
Ranked #11 on
Online Action Detection
on TVSeries
SAR: Scale-Aware Restoration Learning for 3D Tumor Segmentation
We demonstrate the effectiveness of our methods on two downstream tasks: i) Brain tumor segmentation, ii) Pancreas tumor segmentation.
Defending Adversarial Attacks by Correcting logits
Generating and eliminating adversarial examples has been an intriguing topic in the field of deep learning.
