Medical Report Generation

27 papers with code • 2 benchmarks • 3 datasets

Medical report generation (MRG) is a task which focus on training AI to automatically generate professional report according the input image data. This can help clinicians make faster and more accurate decision since the task itself is both time consuming and error prone even for experienced doctors.

Deep neural network and transformer based architecture are currently the most popular methods for this certain task, however, when we try to transfer out pre-trained model into this certain domain, their performance always degrade.

The following are some of the reasons why RSG is hard for pre-trained models:

  • Language datasets in a particular domain can sometimes be quite different from the large number of datasets available on the Internet
  • During the fine-tuning phase, datasets in the medical field are often unevenly distributed

More recently, multi-modal learning and contrastive learning have shown some inspiring results in this field, but it's still challenging and requires further attention.

Here are some additional readings to go deeper on the task:

  • On the Automatic Generation of Medical Imaging Reports

https://doi.org/10.48550/arXiv.1711.08195

  • A scoping review of transfer learning research on medical image analysis using ImageNet

https://arxiv.org/abs/2004.13175

  • A Survey on Incorporating Domain Knowledge into Deep Learning for Medical Image Analysis

https://arxiv.org/abs/2004.12150

(Image credit : Transformers in Medical Imaging: A Survey)

Libraries

Use these libraries to find Medical Report Generation models and implementations

HistGen: Histopathology Report Generation via Local-Global Feature Encoding and Cross-modal Context Interaction

dddavid4real/HistGen 8 Mar 2024

Histopathology serves as the gold standard in cancer diagnosis, with clinical reports being vital in interpreting and understanding this process, guiding cancer treatment and patient care.

6
08 Mar 2024

ICON: Improving Inter-Report Consistency of Radiology Report Generation via Lesion-aware Mix-up Augmentation

wjhou/icon 20 Feb 2024

Previous research on radiology report generation has made significant progress in terms of increasing the clinical accuracy of generated reports.

5
20 Feb 2024

Complex Organ Mask Guided Radiology Report Generation

garygutc/comg_model 4 Nov 2023

The goal of automatic report generation is to generate a clinically accurate and coherent phrase from a single given X-ray image, which could alleviate the workload of traditional radiology reporting.

19
04 Nov 2023

RECAP: Towards Precise Radiology Report Generation via Dynamic Disease Progression Reasoning

wjhou/recap 21 Oct 2023

It then combines the historical records, spatiotemporal information, and radiographs for report generation, where a disease progression graph and dynamic progression reasoning mechanism are devised to accurately select the attributes of each observation and progression.

19
21 Oct 2023

PromptMRG: Diagnosis-Driven Prompts for Medical Report Generation

jhb86253817/promptmrg 24 Aug 2023

To address these challenges, we propose diagnosis-driven prompts for medical report generation (PromptMRG), a novel framework that aims to improve the diagnostic accuracy of MRG with the guidance of diagnosis-aware prompts.

7
24 Aug 2023

Rethinking Medical Report Generation: Disease Revealing Enhancement with Knowledge Graph

wangyixinxin/mrg-kg 24 Jul 2023

Knowledge Graph (KG) plays a crucial role in Medical Report Generation (MRG) because it reveals the relations among diseases and thus can be utilized to guide the generation process.

22
24 Jul 2023

CausalVLR: A Toolbox and Benchmark for Visual-Linguistic Causal Reasoning

hcplab-sysu/causal-vlreasoning 30 Jun 2023

We present CausalVLR (Causal Visual-Linguistic Reasoning), an open-source toolbox containing a rich set of state-of-the-art causal relation discovery and causal inference methods for various visual-linguistic reasoning tasks, such as VQA, image/video captioning, medical report generation, model generalization and robustness, etc.

107
30 Jun 2023

ORGAN: Observation-Guided Radiology Report Generation via Tree Reasoning

wjhou/organ 10 Jun 2023

This paper explores the task of radiology report generation, which aims at generating free-text descriptions for a set of radiographs.

34
10 Jun 2023

Multi-modal Pre-training for Medical Vision-language Understanding and Generation: An Empirical Study with A New Benchmark

control-xl/medical-vision-langauge-transformer 10 Jun 2023

With the availability of large-scale, comprehensive, and general-purpose vision-language (VL) datasets such as MSCOCO, vision-language pre-training (VLP) has become an active area of research and proven to be effective for various VL tasks such as visual-question answering.

15
10 Jun 2023

Automatic Radiology Report Generation by Learning with Increasingly Hard Negatives

bhanu068/ithn 11 May 2023

At each iteration, conditioned on a given set of hard negative reports, image and report features are learned as usual by minimising the loss functions related to report generation.

8
11 May 2023