Search Results for author:
Found 102 papers, 45 papers with code
The Sogou-TIIC Speech Translation System for IWSLT 2018
Our final average result on speech translation is 31. 02 BLEU.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Ethical-Lens: Curbing Malicious Usages of Open-Source Text-to-Image Models
The burgeoning landscape of text-to-image models, exemplified by innovations such as Midjourney and DALLE 3, has revolutionized content creation across diverse sectors.
Knowledge-enhanced Visual-Language Pretraining for Computational Pathology
In this paper, we consider the problem of visual representation learning for computational pathology, by exploiting large-scale image-text pairs gathered from public resources, along with the domain specific knowledge in pathology.
MING-MOE: Enhancing Medical Multi-Task Learning in Large Language Models with Sparse Mixture of Low-Rank Adapter Experts
Large language models like ChatGPT have shown substantial progress in natural language understanding and generation, proving valuable across various disciplines, including the medical field.
Anomaly Detection in Electrocardiograms: Advancing Clinical Diagnosis Through Self-Supervised Learning
We introduce a novel self-supervised learning framework for ECG AD, utilizing a vast dataset of normal ECGs to autonomously detect and localize cardiac anomalies.
Self-Supervised Anomaly Detection
Self-Supervised Learning
+2
ReMamber: Referring Image Segmentation with Mamba Twister
Referring Image Segmentation (RIS) leveraging transformers has achieved great success on the interpretation of complex visual-language tasks.
M$^3$AV: A Multimodal, Multigenre, and Multipurpose Audio-Visual Academic Lecture Dataset
Although multiple academic video datasets have been constructed and released, few of them support both multimodal content recognition and understanding tasks, which is partially due to the lack of high-quality human annotations.
Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images
Recent advancements in large-scale visual-language pre-trained models have led to significant progress in zero-/few-shot anomaly detection within natural image domains.
Audio-Visual Segmentation via Unlabeled Frame Exploitation
NFs, temporally adjacent to the labeled frame, often contain rich motion information that assists in the accurate localization of sounding objects.
Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulator
Large Language Models (LLMs) have demonstrated remarkable proficiency in human interactions, yet their application within the medical field remains insufficiently explored.
Decentralized and Lifelong-Adaptive Multi-Agent Collaborative Learning
In this paper, we propose DeLAMA, a decentralized multi-agent lifelong collaborative learning algorithm with dynamic collaboration graphs.
Enhancing Data Quality in Federated Fine-Tuning of Foundation Models
In the current landscape of foundation model training, there is a significant reliance on public domain data, which is nearing exhaustion according to recent research.
Post-decoder Biasing for End-to-End Speech Recognition of Multi-turn Medical Interview
End-to-end (E2E) approach is gradually replacing hybrid models for automatic speech recognition (ASR) tasks.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Leveraging Diverse Modeling Contexts with Collaborating Learning for Neural Machine Translation
Autoregressive (AR) and Non-autoregressive (NAR) models are two types of generative models for Neural Machine Translation (NMT).
Towards Building Multilingual Language Model for Medicine
In this paper, we aim to develop an open-source, multilingual language model for medicine, that the benefits a wider, linguistically diverse audience from different regions.
M2K-VDG: Model-Adaptive Multimodal Knowledge Anchor Enhanced Video-grounded Dialogue Generation
Video-grounded dialogue generation (VDG) requires the system to generate a fluent and accurate answer based on multimodal knowledge.
DictLLM: Harnessing Key-Value Data Structures with Large Language Models for Enhanced Medical Diagnostics
Structured data offers a sophisticated mechanism for the organization of information.
OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learning
Trained on massive publicly available data, large language models (LLMs) have demonstrated tremendous success across various fields.
Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation
Aligning large language models (LLMs) with human values is imperative to mitigate potential adverse effects resulting from their misuse.
Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agents
Furthermore, to unleash the potential of extensive high-quality digital assets, ChatSim employs a novel multi-camera lighting estimation method to achieve scene-consistent assets' rendering.
An Extensible Framework for Open Heterogeneous Collaborative Perception
In this paper, we introduce a new open heterogeneous problem: how to accommodate continually emerging new heterogeneous agent types into collaborative perception, while ensuring high perception performance and low integration cost?
FedRSU: Federated Learning for Scene Flow Estimation on Roadside Units
In this paper, we introduce FedRSU, an innovative federated learning framework for self-supervised scene flow estimation.
MM-SAP: A Comprehensive Benchmark for Assessing Self-Awareness of Multimodal Large Language Models in Perception
We believe that these hallucinations are partially due to the models' struggle with understanding what they can and cannot perceive from images, a capability we refer to as self-awareness in perception.
One Model to Rule them All: Towards Universal Segmentation for Medical Images with Text Prompts
Our main contributions are three folds: (i) on data construction, we combine multiple knowledge sources to construct a multi-modal medical knowledge tree; Then we build up a large-scale segmentation dataset for training, by collecting over 11K 3D medical image scans from 31 segmentation datasets with careful standardization on both visual scans and label space; (ii) on model training, we formulate a universal segmentation model, that can be prompted by inputting medical terminologies in text form.
Large-scale Long-tailed Disease Diagnosis on Radiology Images
In this study, we aim to investigate the problem of large-scale, large-vocabulary disease classification for radiologic images, which can be formulated as a multi-modal, multi-anatomy, multi-label, long-tailed classification.
A Strong Baseline for Temporal Video-Text Alignment
In this paper, we consider the problem of temporally aligning the video and texts from instructional videos, specifically, given a long-term video, and associated text sentences, our goal is to determine their corresponding timestamps in the video.
MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models
The emergence of various medical large language models (LLMs) in the medical domain has highlighted the need for unified evaluation standards, as manual evaluation of LLMs proves to be time-consuming and labor-intensive.
UniChest: Conquer-and-Divide Pre-training for Multi-Source Chest X-Ray Classification
Vision-Language Pre-training (VLP) that utilizes the multi-modal information to promote the training efficiency and effectiveness, has achieved great success in vision recognition of natural domains and shown promise in medical imaging diagnosis for the Chest X-Rays (CXRs).
Hypergraph Transformer for Semi-Supervised Classification
Hypergraphs play a pivotal role in the modelling of data featuring higher-order relations involving more than two entities.
Fake It Till Make It: Federated Learning with Consensus-Oriented Generation
FedCOG consists of two key components at the client side: complementary data generation, which generates data extracted from the shared global model to complement the original dataset, and knowledge-distillation-based model training, which distills knowledge from global model to local model based on the generated data to mitigate over-fitting the original heterogeneous dataset.
Federated Learning Empowered by Generative Content
In this paper, we propose a novel FL framework termed FedGC, designed to mitigate data heterogeneity issues by diversifying private data with generative content.
Combating Representation Learning Disparity with Geometric Harmonization
Self-supervised learning (SSL) as an effective paradigm of representation learning has achieved tremendous success on various curated datasets in diverse scenarios.
Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosis
Driven by the large foundation models, the development of artificial intelligence has witnessed tremendous progress lately, leading to a surge of general interest from the public.
Improving the Reliability of Large Language Models by Leveraging Uncertainty-Aware In-Context Learning
In this study, we introduce an uncertainty-aware in-context learning framework to empower the model to enhance or reject its output in response to uncertainty.
MSG-BART: Multi-granularity Scene Graph-Enhanced Encoder-Decoder Language Model for Video-grounded Dialogue Generation
Generating dialogue grounded in videos requires a high level of understanding and reasoning about the visual scenes in the videos.
UniBrain: Universal Brain MRI Diagnosis with Hierarchical Knowledge-enhanced Pre-training
Magnetic resonance imaging~(MRI) have played a crucial role in brain disease diagnosis, with which a range of computer-aided artificial intelligence methods have been proposed.
An Automatic Evaluation Framework for Multi-turn Medical Consultations Capabilities of Large Language Models
A medical consultation training set is further constructed to improve the consultation ability of LLMs.
AttrSeg: Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregation
The results show the superior performance of attribute decomposition-aggregation.
LibriSQA: A Novel Dataset and Framework for Spoken Question Answering with Large Language Models
While Large Language Models (LLMs) have demonstrated commendable performance across a myriad of domains and tasks, existing LLMs still exhibit a palpable deficit in handling multimodal functionalities, especially for the Spoken Question Answering (SQA) task which necessitates precise alignment and deep interaction between speech and text features.
Bag of Tricks for Long-Tailed Multi-Label Classification on Chest X-Rays
Clinical classification of chest radiography is particularly challenging for standard machine learning algorithms due to its inherent long-tailed and multi-label nature.
Auxiliary Tasks Benefit 3D Skeleton-based Human Motion Prediction
To work with auxiliary tasks, we propose a novel auxiliary-adapted transformer, which can handle incomplete, corrupted motion data and achieve coordinate recovery via capturing spatial-temporal dependencies.
Joint-Relation Transformer for Multi-Person Motion Prediction
Multi-person motion prediction is a challenging problem due to the dependency of motion on both individual past movements and interactions with other people.
Multi-Scale Memory Comparison for Zero-/Few-Shot Anomaly Detection
Anomaly detection has gained considerable attention due to its broad range of applications, particularly in industrial defect detection.
Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Data
In this study, we aim to initiate the development of Radiology Foundation Model, termed as RadFM.
Balanced Destruction-Reconstruction Dynamics for Memory-replay Class Incremental Learning
By dynamically manipulating the gradient during training based on these factors, BDR can effectively alleviate knowledge destruction and improve knowledge reconstruction.
Multi-scale Cross-restoration Framework for Electrocardiogram Anomaly Detection
To address this challenge, this paper introduces a novel multi-scale cross-restoration framework for ECG anomaly detection and localization that considers both local and global ECG characteristics.
Audio-aware Query-enhanced Transformer for Audio-Visual Segmentation
The goal of the audio-visual segmentation (AVS) task is to segment the sounding objects in the video frames using audio cues.
All in One: Exploring Unified Vision-Language Tracking with Multi-Modal Alignment
This approach achieves feature integration in a unified backbone, removing the need for carefully-designed fusion modules and resulting in a more effective and efficient VL tracking framework.
Multi-Modal Prototypes for Open-Set Semantic Segmentation
In this paper, we define a unified setting termed as open-set semantic segmentation (O3S), which aims to learn seen and unseen semantics from both visual examples and textual names.
Boost Video Frame Interpolation via Motion Adaptation
Video frame interpolation (VFI) is a challenging task that aims to generate intermediate frames between two consecutive frames in a video.
Zero-shot Composed Text-Image Retrieval
In this paper, we consider the problem of composed image retrieval (CIR), it aims to train a model that can fuse multi-modal information, e. g., text and images, to accurately retrieve images that match the query, extending the user's expression ability.
Ranked #1 on
Zero-Shot Composed Image Retrieval (ZS-CIR)
on CIRR
Exploring Effective Mask Sampling Modeling for Neural Image Compression
Specifically, Cube Mask Sampling Module (CMSM) is proposed to apply both spatial and channel mask sampling modeling to image compression in the pre-training stage.
Intelligent Grimm -- Open-ended Visual Storytelling via Latent Diffusion Models
Generative models have recently exhibited exceptional capabilities in text-to-image generation, but still struggle to generate image sequences coherently.
FedDisco: Federated Learning with Discrepancy-Aware Collaboration
However, based on our empirical observations and theoretical analysis, we find that the dataset size is not optimal and the discrepancy between local and global category distributions could be a beneficial and complementary indicator for determining aggregation weights.
PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering
In this paper, we focus on the problem of Medical Visual Question Answering (MedVQA), which is crucial in efficiently interpreting medical images with vital clinic-relevant information.
Ranked #1 on
Medical Visual Question Answering
on PMC-VQA
PMC-LLaMA: Towards Building Open-source Language Models for Medicine
Our contributions are threefold: (i) we systematically investigate the process of adapting a general-purpose foundation language model towards medical domain, this involves data-centric knowledge injection through the integration of 4. 8M biomedical academic papers and 30K medical textbooks, as well as comprehensive fine-tuning for alignment with domain-specific instructions; (ii) we contribute a large-scale, comprehensive dataset for instruction tuning.
Collaboration Helps Camera Overtake LiDAR in 3D Detection
Camera-only 3D detection provides an economical solution with a simple configuration for localizing objects in 3D space compared to LiDAR-based detection systems.
Multi-modal Prompting for Low-Shot Temporal Action Localization
In this paper, we consider the problem of temporal action localization under low-shot (zero-shot & few-shot) scenario, with the goal of detecting and classifying the action instances from arbitrary categories within some untrimmed videos, even not seen at training time.
EqMotion: Equivariant Multi-agent Motion Prediction with Invariant Interaction Reasoning
In motion prediction tasks, maintaining motion equivariance under Euclidean geometric transformations and invariance of agent interaction is a critical and fundamental principle.
Ranked #1 on
Human Pose Forecasting
on Human3.6M
Leapfrog Diffusion Model for Stochastic Trajectory Prediction
The core of the proposed LED is to leverage a trainable leapfrog initializer to directly learn an expressive multi-modal distribution of future trajectories, which skips a large number of denoising steps, significantly accelerating inference speed.
Boundary-aware Supervoxel-level Iteratively Refined Interactive 3D Image Segmentation with Multi-agent Reinforcement Learning
Interactive segmentation has recently been explored to effectively and efficiently harvest high-quality segmentation masks by iteratively incorporating user hints.
DiffusionSeg: Adapting Diffusion Towards Unsupervised Object Discovery
However, the challenges exist as there is one structural difference between generative and discriminative models, which limits the direct use.
TBP-Former: Learning Temporal Bird's-Eye-View Pyramid for Joint Perception and Prediction in Vision-Centric Autonomous Driving
Second, a spatial-temporal pyramid transformer is introduced to comprehensively extract multi-scale BEV features and predict future BEV states with the support of spatial-temporal priors.
Ranked #2 on
Bird's-Eye View Semantic Segmentation
on nuScenes
(IoU ped - 224x480 - Vis filter. - 100x100 at 0.5 metric)
DR2: Diffusion-based Robust Degradation Remover for Blind Face Restoration
However, it is expensive and infeasible to include every type of degradation to cover real-world cases in the training data.
PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents
Foundation models trained on large-scale dataset gain a recent surge in CV and NLP.
Ranked #3 on
Medical Visual Question Answering
on PMC-VQA
Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Images
While multi-modal foundation models pre-trained on large-scale data have been successful in natural language understanding and vision recognition, their use in medical domains is still limited due to the fine-grained nature of medical tasks and the high demand for domain knowledge.
K-Diag: Knowledge-enhanced Disease Diagnosis in Radiographic Imaging
In this paper, we consider the problem of disease diagnosis.
Knowledge-aware Bayesian Co-attention for Multimodal Emotion Recognition
Multimodal emotion recognition is a challenging research area that aims to fuse different modalities to predict human emotion.
Long-Tailed Partial Label Learning via Dynamic Rebalancing
The straightforward combination of LT and PLL, i. e., LT-PLL, suffers from a fundamental dilemma: LT methods build upon a given class distribution that is unavailable in PLL, and the performance of PLL is severely influenced in long-tailed context.
Open-vocabulary Object Segmentation with Diffusion Models
The goal of this paper is to extract the visual-language correspondence from a pre-trained text-to-image diffusion model, in the form of segmentation map, i. e., simultaneously generating images and segmentation masks for the corresponding visual entities described in the text prompt.
Integrating features from lymph node stations for metastatic lymph node detection
The branch targets to solve a closely related task on the LN station level, i. e., classifying whether an LN station contains metastatic LN or not, so as to learn representations for LN stations.
MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiology
In this paper, we consider enhancing medical visual-language pre-training (VLP) with domain-specific knowledge, by exploiting the paired image-text reports from the radiological daily practice.
Federated Domain Generalization With Generalization Adjustment
Federated Domain Generalization (FedDG) attempts to learn a global model in a privacy-preserving manner that generalizes well to new clients possibly with domain shift.
MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training for X-ray Diagnosis
In this paper, we consider enhancing medical visual-language pre-training (VLP) with domain-specific knowledge, by exploiting the paired image-text reports from the radiological daily practice.
Distilling Vision-Language Pre-training to Collaborate with Weakly-Supervised Temporal Action Localization
And as a result, the dual-branch complementarity is effectively fused to promote a strong alliance.
Weakly-supervised Temporal Action Localization
Weakly Supervised Temporal Action Localization
FedSkip: Combatting Statistical Heterogeneity with Federated Skip Aggregation
However, in addition to previous explorations for improvement in federated averaging, our analysis shows that another critical bottleneck is the poorer optima of client models in more heterogeneous conditions.
Robust Collaborative 3D Object Detection in Presence of Pose Errors
Collaborative 3D object detection exploits information exchange among multiple agents to enhance accuracy of object detection in presence of sensor impairments such as occlusion.
Unrolled Graph Learning for Multi-Agent Collaboration
To fill this gap, we propose a distributed multi-agent learning model inspired by human collaboration, in which the agents can autonomously detect suitable collaborators and refer to collaborators' model for better performance.
Open-vocabulary Semantic Segmentation with Frozen Vision-Language Models
When trained at a sufficient scale, self-supervised learning has exhibited a notable ability to solve a wide range of visual or language understanding tasks.
Number-Adaptive Prototype Learning for 3D Point Cloud Semantic Segmentation
Instead of learning a single prototype for each class, in this paper, we propose to use an adaptive number of prototypes to dynamically describe the different point patterns within a semantic class.
Ranked #17 on
3D Semantic Segmentation
on SemanticKITTI
A Simple Plugin for Transforming Images to Arbitrary Scales
Existing models on super-resolution often specialized for one scale, fundamentally limiting their use in practical scenarios.
Low-Light Video Enhancement with Synthetic Event Guidance
Low-light video enhancement (LLVE) is an important yet challenging task with many applications such as photographing and autonomous driving.
Collaborative Uncertainty Benefits Multi-Agent Multi-Modal Trajectory Forecasting
Further, we apply the proposed framework to current SOTA multi-agent multi-modal forecasting systems as a plugin module, which enables the SOTA systems to 1) estimate the uncertainty in the multi-agent multi-modal trajectory forecasting task; 2) rank the multiple predictions and select the optimal one based on the estimated uncertainty.
Multi-level Fusion of Wav2vec 2.0 and BERT for Multimodal Emotion Recognition
The research and applications of multimodal emotion recognition have become increasingly popular recently.
Nextformer: A ConvNeXt Augmented Conformer For End-To-End Speech Recognition
To the best of our knowledge, the proposed Nextformer model achieves SOTA results on AISHELL-1(CER 4. 06%) and WenetSpeech(CER 7. 56%/11. 29%).
K-Space Transformer for Undersampled MRI Reconstruction
This paper considers the problem of undersampled MRI reconstruction.
Contrastive Learning with Boosted Memorization
Different from previous works, we explore this direction from an alternative perspective, i. e., the data perspective, and propose a novel Boosted Contrastive Learning (BCL) method.
Self-Supervised Masking for Unsupervised Anomaly Detection and Localization
To extend the reconstruction-based anomaly detection architecture to the localized anomalies, we propose a self-supervised learning approach through random masking and then restoring, named Self-Supervised Masking (SSM) for unsupervised anomaly detection and localization.
Handwritten Mathematical Expression Recognition via Attention Aggregation based Bi-directional Mutual Learning
Handwritten mathematical expression recognition aims to automatically generate LaTeX sequences from given images.
Self-supervised Tumor Segmentation through Layer Decomposition
In this paper, we target self-supervised representation learning for zero-shot tumor segmentation.
Multiscale Spatio-Temporal Graph Neural Networks for 3D Skeleton-Based Motion Prediction
The core of MST-GNN is a multiscale spatio-temporal graph that explicitly models the relations in motions at various spatial and temporal scales.
Cooperative Learning for Noisy Supervision
Learning with noisy labels has gained the enormous interest in the robust deep learning area.
MS-KD: Multi-Organ Segmentation with Multiple Binary-Labeled Datasets
A novel Multi-teacher Single-student Knowledge Distillation (MS-KD) framework is proposed, where the teacher models are pre-trained single-organ segmentation networks, and the student model is a multi-organ segmentation network.
A Fourier-based Framework for Domain Generalization
Modern deep neural networks suffer from performance degradation when evaluated on testing data under different distributions from training data.
H2O: A Benchmark for Visual Human-human Object Handover Analysis
Besides, we also report the hand and object pose errors with existing baselines and show that the dataset can serve as the video demonstrations for robot imitation learning on the handover task.
Collaborative Label Correction via Entropy Thresholding
Deep neural networks (DNNs) have the capacity to fit extremely noisy labels nonetheless they tend to learn data with clean labels first and then memorize those with noisy labels.
Divide and Conquer for Single-Frame Temporal Action Localization
Single-frame temporal action localization (STAL) aims to localize actions in untrimmed videos with only one timestamp annotation for each action instance.
FGraDA: A Dataset and Benchmark for Fine-Grained Domain Adaptation in Machine Translation
Previous research for adapting a general neural machine translation (NMT) model into a specific domain usually neglects the diversity in translation within the same domain, which is a core problem for domain adaptation in real-world scenarios.
Point-Level Temporal Action Localization: Bridging Fully-supervised Proposals to Weakly-supervised Losses
Point-Level temporal action localization (PTAL) aims to localize actions in untrimmed videos with only one timestamp annotation for each action instance.
Ranked #3 on
Weakly Supervised Action Localization
on BEOID
Privileged Knowledge Distillation for Online Action Detection
Knowledge distillation is employed to transfer the privileged information from the offline teacher to the online student.
Ranked #11 on
Online Action Detection
on TVSeries
SAR: Scale-Aware Restoration Learning for 3D Tumor Segmentation
We demonstrate the effectiveness of our methods on two downstream tasks: i) Brain tumor segmentation, ii) Pancreas tumor segmentation.
Defending Adversarial Attacks by Correcting logits
Generating and eliminating adversarial examples has been an intriguing topic in the field of deep learning.
