Enhancing Adverse Drug Event Detection with Multimodal Dataset: Corpus Creation and Model Development

Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Aman Chadha, Samrat Mondal·May 24, 2024

Summary

The paper presents the MMADE dataset, a multimodal resource for adverse drug event (ADE) detection, which combines textual information from diverse sources with medical images. By leveraging large language and vision models, the study aims to improve ADE detection, enhancing patient safety and healthcare. The dataset consists of 1,500 ADR cases with paired images and descriptions, annotated by medical professionals. Researchers fine-tune InstructBLIP on the dataset, showing the potential of these models in tasks like classification, caption generation, and summarization. The study highlights the importance of integrating visual cues in ADE detection and addresses the lack of multimodal datasets in the field. Performance comparisons demonstrate the superiority of multimodal models over unimodal ones, with InstructBLIP reducing hallucinations and focusing on relevant visual information. The research contributes to pharmacovigilance by providing a foundation for further studies on ADE monitoring and understanding.

Introduction
Background
[ ] Emergence of multimodal approaches in healthcare
[ ] Importance of ADE detection for patient safety
Objective
[ ] Creation of MMADE dataset
[ ] Goal: Improve ADE detection with multimodal models
Dataset Description
Data Collection
[ ] 1,500 ADR cases
[ ] Paired images and descriptions
[ ] Annotating medical professionals
Data Characteristics
[ ] Textual information: diverse sources
[ ] Medical images: visual cues
Model Fine-Tuning and Evaluation
InstructBLIP
[ ] Model selection: InstructBLIP
[ ] Tasks: classification, caption generation, summarization
Performance Analysis
[ ] Multimodal vs. unimodal models
[ ] Reduction in hallucinations and focus on relevant visuals
Applications and Impact
Pharmacovigilance
[ ] Enhancing ADE monitoring
[ ] Contribution to healthcare research
Future Directions
[ ] Potential for further studies
[ ] Advancing patient safety through multimodal AI
Conclusion
[ ] Summary of findings
[ ] Implications for the field of medical informatics
Basic info
papers
computation and language
computer vision and pattern recognition
artificial intelligence
Advanced features
Insights
What is the size of the MMADE dataset, and who annotated the ADR cases?
How does InstructBLIP perform in comparison to unimodal models in ADE detection tasks?
What is the primary purpose of the MMADE dataset?
How does the study leverage large language and vision models for ADE detection?