Medical Report Generation
29 papers with code • 2 benchmarks • 3 datasets
Medical report generation (MRG) is a task which focus on training AI to automatically generate professional report according the input image data. This can help clinicians make faster and more accurate decision since the task itself is both time consuming and error prone even for experienced doctors.
Deep neural network and transformer based architecture are currently the most popular methods for this certain task, however, when we try to transfer out pre-trained model into this certain domain, their performance always degrade.
The following are some of the reasons why RSG is hard for pre-trained models:
- Language datasets in a particular domain can sometimes be quite different from the large number of datasets available on the Internet
- During the fine-tuning phase, datasets in the medical field are often unevenly distributed
More recently, multi-modal learning and contrastive learning have shown some inspiring results in this field, but it's still challenging and requires further attention.
Here are some additional readings to go deeper on the task:
- On the Automatic Generation of Medical Imaging Reports
https://doi.org/10.48550/arXiv.1711.08195
- A scoping review of transfer learning research on medical image analysis using ImageNet
https://arxiv.org/abs/2004.13175
- A Survey on Incorporating Domain Knowledge into Deep Learning for Medical Image Analysis
https://arxiv.org/abs/2004.12150
(Image credit : Transformers in Medical Imaging: A Survey)
Libraries
Use these libraries to find Medical Report Generation models and implementations
MIMIC-CXR
PathVQA
HistGen WSI-Report Dataset
Most implemented papers
On the Automatic Generation of Medical Imaging Reports
ZexinYan/Medical-Report-Generation
•
•
ACL 2018
To cope with these challenges, we (1) build a multi-task learning framework which jointly performs the pre- diction of tags and the generation of para- graphs, (2) propose a co-attention mechanism to localize regions containing abnormalities and generate narrations for them, (3) develop a hierarchical LSTM model to generate long paragraphs.
Auxiliary Signal-Guided Knowledge Encoder-Decoder for Medical Report Generation
mlii0117/COV-CTR
•
•
Firstly, the regions of primary interest to radiologists are usually located in a small area of the global image, meaning that the remainder parts of the image could be considered as irrelevant noise in the training procedure.
Cross-Modal Causal Intervention for Medical Report Generation
Medical report generation (MRG) is essential for computer-aided diagnosis and medication guidance, which can relieve the heavy burden of radiologists by automatically generating the corresponding medical reports according to the given radiology image.
CausalVLR: A Toolbox and Benchmark for Visual-Linguistic Causal Reasoning
We present CausalVLR (Causal Visual-Linguistic Reasoning), an open-source toolbox containing a rich set of state-of-the-art causal relation discovery and causal inference methods for various visual-linguistic reasoning tasks, such as VQA, image/video captioning, medical report generation, model generalization and robustness, etc.
DeepOpht: Medical Report Generation for Retinal Images via Deep Models and Visual Explanation
To train and validate the effectiveness of our DNN-based module, we propose a large-scale retinal disease image dataset.
Inspecting state of the art performance and NLP metrics in image-based medical report generation
Several deep learning architectures have been proposed over the last years to deal with the problem of generating a written report given an imaging exam as input.
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning
Vision-CAIR/VisualGPT
•
•
CVPR 2022
To the best of our knowledge, this is the first work that improves data efficiency of image captioning by utilizing LM pretrained on unimodal data.
Automated radiology report generation using conditioned transformers
omar-mohamed/GPT2-Chest-X-Ray-Report-Generation
•
•
We represent the first work to condition a pre-trained transformer on visual and semantic features to generate medical reports and to include semantic similarity metrics in the quantitative analysis of the generated reports.
FFA-IR: Towards an Explainable and Reliable Medical Report Generation Benchmark
mlii0117/FFA-IR
•
•
Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) 2021
Researchers have explored advanced methods from computer vision and natural language processing to incorporate medical domain knowledge for the generation of readable medical reports.
Weakly Supervised Contrastive Learning for Chest X-Ray Report Generation
zzxslp/wcl
•
•
Findings (EMNLP) 2021
Radiology report generation aims at generating descriptive text from radiology images automatically, which may present an opportunity to improve radiology reporting and interpretation.
