Medical Report Generation
27 papers with code • 2 benchmarks • 3 datasets
Medical report generation (MRG) is a task which focus on training AI to automatically generate professional report according the input image data. This can help clinicians make faster and more accurate decision since the task itself is both time consuming and error prone even for experienced doctors.
Deep neural network and transformer based architecture are currently the most popular methods for this certain task, however, when we try to transfer out pre-trained model into this certain domain, their performance always degrade.
The following are some of the reasons why RSG is hard for pre-trained models:
- Language datasets in a particular domain can sometimes be quite different from the large number of datasets available on the Internet
- During the fine-tuning phase, datasets in the medical field are often unevenly distributed
More recently, multi-modal learning and contrastive learning have shown some inspiring results in this field, but it's still challenging and requires further attention.
Here are some additional readings to go deeper on the task:
- On the Automatic Generation of Medical Imaging Reports
https://doi.org/10.48550/arXiv.1711.08195
- A scoping review of transfer learning research on medical image analysis using ImageNet
https://arxiv.org/abs/2004.13175
- A Survey on Incorporating Domain Knowledge into Deep Learning for Medical Image Analysis
https://arxiv.org/abs/2004.12150
(Image credit : Transformers in Medical Imaging: A Survey)
Libraries
Use these libraries to find Medical Report Generation models and implementations
MIMIC-CXR
PathVQA
HistGen WSI-Report Dataset
Most implemented papers
Transformers in Medical Imaging: A Survey
fahadshamshad/awesome-transformers-in-medical-imaging
•
•
Following unprecedented success on the natural language tasks, Transformers have been successfully applied to several computer vision problems, achieving state-of-the-art results and prompting researchers to reconsider the supremacy of convolutional neural networks (CNNs) as {de facto} operators.
A Benchmark for Automatic Medical Consultation System: Frameworks, Tasks and Datasets
In recent years, interest has arisen in using machine learning to improve the efficiency of automatic medical consultation and enhance patient experience.
M^4I: Multi-modal Models Membership Inference
multimodalmi/multimodal-membership-inference
•
•
To achieve this, we propose Multi-modal Models Membership Inference (M^4I) with two attack methods to infer the membership status, named metric-based (MB) M^4I and feature-based (FB) M^4I, respectively.
DeltaNet:Conditional Medical Report Generation for COVID-19 Diagnosis
To reduce the workload of radiologists, we propose DeltaNet to generate medical reports automatically.
Lesion Guided Explainable Few Weak-shot Medical Report Generation
jinghansunn/few-weak-shot-rg
•
•
To this end, we propose a lesion guided explainable few weak-shot medical report generation framework that learns correlation between seen and novel classes through visual and semantic feature alignment, aiming to generate medical reports for diseases not observed in training.
Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation
To address the limitation, we propose a knowledge graph with Dynamic structure and nodes to facilitate medical report generation with Contrastive Learning, named DCL.
Interactive and Explainable Region-guided Radiology Report Generation
While previous methods generate reports without the possibility of human intervention and with limited explainability, our method opens up novel clinical use cases through additional interactive capabilities and introduces a high degree of transparency and explainability.
Automatic Radiology Report Generation by Learning with Increasingly Hard Negatives
bhanu068/ithn
•
•
At each iteration, conditioned on a given set of hard negative reports, image and report features are learned as usual by minimising the loss functions related to report generation.
Multi-modal Pre-training for Medical Vision-language Understanding and Generation: An Empirical Study with A New Benchmark
control-xl/medical-vision-langauge-transformer
•
•
With the availability of large-scale, comprehensive, and general-purpose vision-language (VL) datasets such as MSCOCO, vision-language pre-training (VLP) has become an active area of research and proven to be effective for various VL tasks such as visual-question answering.
ORGAN: Observation-Guided Radiology Report Generation via Tree Reasoning
wjhou/organ
•
•
This paper explores the task of radiology report generation, which aims at generating free-text descriptions for a set of radiographs.
