Search Results for author:
Found 24 papers, 15 papers with code
MiniVQA - A resource to build your tailored VQA competition
MiniVQA is a Jupyter notebook to build a tailored VQA competition for your students.
QIAI at MEDIQA 2021: Multimodal Radiology Report Summarization
This paper describes the solution of the QIAI lab sent to the Radiology Report Summarization (RRS) challenge at MEDIQA 2021.
ViLMedic: a framework for research at the intersection of vision and language in medical AI
There is a growing need to model interactions between data modalities (e. g., vision, language) — both to improve AI predictions on existing tasks and to enable new applications.
CheXagent: Towards a Foundation Model for Chest X-Ray Interpretation
However, developing FMs that can accurately interpret CXRs is challenging due to the (1) limited availability of large-scale vision-language datasets in the medical image domain, (2) lack of vision and language encoders that can capture the complexities of medical data, and (3) absence of evaluation frameworks for benchmarking the abilities of FMs on CXR interpretation.
Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization
Analyzing vast textual data and summarizing key information from electronic health records imposes a substantial burden on how clinicians allocate their time.
ViLLA: Fine-Grained Vision-Language Representation Learning from Real-World Data
The first key contribution of this work is to demonstrate through systematic evaluations that as the pairwise complexity of the training dataset increases, standard VLMs struggle to learn region-attribute relationships, exhibiting performance degradations of up to 37% on retrieval tasks.
RadAdapt: Radiology Report Summarization via Lightweight Domain Adaptation of Large Language Models
We systematically investigate lightweight strategies to adapt large language models (LLMs) for the task of radiology report summarization (RRS).
RoentGen: Vision-Language Foundation Model for Chest X-ray Generation
We present evidence that the resulting model (RoentGen) is able to create visually convincing, diverse synthetic CXR images, and that the output can be controlled to a new extent by using free-form text prompts including radiology-specific language.
Toward expanding the scope of radiology report summarization to multiple anatomies and modalities
We then conduct extensive experiments to evaluate the performance of models both within and across modality-anatomy pairs in MIMIC-RRS.
Improving the Factual Correctness of Radiology Report Generation with Semantic Rewards
To overcome this limitation, we propose a new method, the RadGraph reward, to further improve the factual completeness and correctness of generated radiology reports.
Domino: Discovering Systematic Errors with Cross-Modal Embeddings
In this work, we address these challenges by first designing a principled evaluation framework that enables a quantitative comparison of SDMs across 1, 235 slice discovery settings in three input domains (natural images, medical images, and time-series data).
Modulated Fusion using Transformer for Linguistic-Acoustic Emotion Recognition
This paper aims to bring a new lightweight yet powerful solution for the task of Emotion Recognition and Sentiment Analysis.
Ranked #6 on
Multimodal Sentiment Analysis
on CMU-MOSEI
(using extra training data)
A Transformer-based joint-encoding for Emotion Recognition and Sentiment Analysis
Understanding expressed sentiment and emotions are two crucial factors in human multimodal language.
Ranked #5 on
Multimodal Sentiment Analysis
on CMU-MOSEI
(using extra training data)
Can adversarial training learn image captioning ?
Recently, generative adversarial networks (GAN) have gathered a lot of interest.
Modulated Self-attention Convolutional Network for VQA
As new data-sets for real-world visual reasoning and compositional question answering are emerging, it might be needed to use the visual feature extraction as a end-to-end process during training.
Adversarial reconstruction for Multi-modal Machine Translation
Even with the growing interest in problems at the intersection of Computer Vision and Natural Language, grounding (i. e. identifying) the components of a structured description in an image still remains a challenging task.
Bringing back simplicity and lightliness into neural image captioning
So far, the goal has been to maximize scores on automated metric and to do so, one has to come up with a plurality of new modules and techniques.
UMONS Submission for WMT18 Multimodal Translation Task
This paper describes the UMONS solution for the Multimodal Machine Translation Task presented at the third conference on machine translation (WMT18).
Transformer for Emotion Recognition
This paper describes the UMONS solution for the OMG-Emotion Challenge.
Modulating and attending the source image during encoding improves Multimodal Translation
We propose a new and fully end-to-end approach for multimodal translation where the source text encoder modulates the entire visual input processing using conditional batch normalization, in order to compute the most informative image features for our task.
Visually Grounded Word Embeddings and Richer Visual Features for Improving Multimodal Neural Machine Translation
In Multimodal Neural Machine Translation (MNMT), a neural model generates a translated sentence that describes an image, given the image itself and one source descriptions in English.
An empirical study on the effectiveness of images in Multimodal Neural Machine Translation
In state-of-the-art Neural Machine Translation (NMT), an attention mechanism is used during decoding to enhance the translation.
Multimodal Compact Bilinear Pooling for Multimodal Neural Machine Translation
Recently, the effectiveness of the attention mechanism has also been explored for multimodal tasks, where it becomes possible to focus both on sentence parts and image regions.
