Static and multivariate-temporal attentive fusion transformer for readmission risk prediction

Zhe Sun, Runzhi Li, Jing Wang, Gang Chen, Siyu Yan, Lihong Ma·July 15, 2024

Summary

The paper introduces SMTAFormer, a novel model for predicting short-term readmission risk in ICU patients. It combines static demographic data and dynamic temporal ICU monitor data using an MLP network and temporal transformer network to learn useful feature representations. A static and multivariate temporal feature fusion module models the correlation between static and temporal features, enhancing prediction accuracy. The model outperforms advanced methods, achieving an accuracy of up to 86.6% and an area under the receiver operating characteristic curve (AUC) of 0.717. The RRA dataset, designed for readmission risk prediction, demonstrates the model's effectiveness compared to strong baselines. The paper's contributions include the development of SMTAFormer, the utilization of MLP and temporal transformer networks, and the creation of the RRA dataset, which will be publicly available. SMTAFormer is a predictive model designed to forecast inpatients' readmission risk probability using clinical data, incorporating static demographic data, diagnosis, laboratory, and physiology monitoring data. The model's architecture includes four main components: static feature extraction, dynamic temporal feature learning, feature fusion, and readmission risk prediction. A simple Multi-Layer Perceptron (MLP) network is used for static features, while a traditional transformer encoder processes dynamic temporal data. An attentive fusion network combines these features using an intra-temporal multi-head self-attention layer followed by an inter-static and multivariate temporal multi-head self-attention mechanism. Finally, a fully connected layer completes the readmission risk prediction. The paper introduces a method for fusing static and multivariate temporal data, aiming to extract inter-correlation between them and further the intra-correlation within the multivariate temporal data. The proposed method constructs a deep learning model to predict readmission risk within 30 days for ICU patients. The input consists of numeric vectors derived from static features (e.g., age, sex) and multivariate temporal features, while the output represents the readmission probability within 30 days, indicating whether patients are readmitted or not. The study focuses on 10,008 ICU records of patients with essential hypertension, using three ICD 9 codes (4010, 4011, 4019) to identify them. It excludes patients under 18, those who died in the ICU, and those admitted due to pregnancy. The dataset is filtered for stays longer than 24 hours and shorter than 72 hours. The labels indicate whether a patient returns to the ICU within 30 days or dies within that period. The dataset comprises 1110 records readmitted within 30 days out of 10,008. Data preprocessing involves filling missing values using forward and backward methods, with average for continuous data and mode for discrete data. Standardization uses z-score for continuous data and one-hot encoding for discrete data. The dataset is split into a 9:1 training and test set, with 5-fold cross-validation. The text discusses experiments comparing various methods for processing multimodal and multichannel data, focusing on static, continuous temporal, and discrete temporal features. Three baseline methods—LR, LSTM, and CNN+LSTM—are used for single-modality data processing, with static data montage at each temporal step. The proposed SMTAFormer method is contrasted with different models for each data type. For static features, Dense, Multi-scale convolution, and Multi-scale convolution with learning addition models are used. LSTM and GRU are applied for continuous temporal features analysis. For static data with temporal features, CNN and LSTM models are employed. The study found that the introduction of a self-attention mechanism following a gate mechanism improved AUC performance. The SMTAFormer strategically incorporates static data only once in the final step to mitigate redundancy issues. The model achieved promising results with an accuracy of up to 0.866 and an AUC of up to 0.717 on the RRA dataset. Future work aims to expand the feature set for enhanced readmission prediction, adapt the methodology to local readmission datasets, and address the optimization of patient discharge criteria. In conclusion, SMTAFormer is a significant advancement in predicting ICU patient readmission risk, offering improved accuracy and efficiency in resource allocation. The model's development and the creation of the RRA dataset contribute valuable resources to the medical community, facilitating further research and improvements in patient care.

Key findings

2

Introduction
Background
Overview of ICU patient readmission risk prediction
Importance of accurate prediction in healthcare management
Objective
Aim of the research: developing a novel model for short-term readmission risk prediction
Objectives: combining static demographic data and dynamic temporal ICU monitor data
Method
Data Collection
Sources of data: static demographic data, dynamic temporal ICU monitor data
Data types: numeric, categorical, time-series
Data Preprocessing
Handling missing values: forward, backward filling, average, mode
Data normalization: z-score for continuous data, one-hot encoding for discrete data
Data splitting: 9:1 training and test set, 5-fold cross-validation
Model Architecture
Static Feature Extraction
Use of Multi-Layer Perceptron (MLP) network
Dynamic Temporal Feature Learning
Traditional transformer encoder for processing dynamic temporal data
Feature Fusion
Attentive fusion network combining static and temporal features
Intra-temporal multi-head self-attention layer
Inter-static and multivariate temporal multi-head self-attention mechanism
Readmission Risk Prediction
Fully connected layer for final prediction
Experiments
Baseline Methods
Logistic Regression (LR)
Long Short-Term Memory (LSTM)
Convolutional Neural Network (CNN) + LSTM
Data Processing Methods
Static data montage at each temporal step
Comparison with different models for single-modality data processing
Model Evaluation
Metrics: accuracy, area under the receiver operating characteristic curve (AUC)
Comparison with baseline methods
Results
Model Performance
Achieved accuracy of up to 0.866
Achieved AUC of up to 0.717 on the RRA dataset
Contributions
Development of SMTAFormer model
Utilization of MLP and temporal transformer networks
Creation of the RRA dataset for readmission risk prediction
Future Work
Expansion of feature set for enhanced prediction
Adaptation to local readmission datasets
Optimization of patient discharge criteria
Conclusion
Summary of SMTAFormer's significance in predicting ICU patient readmission risk
Contribution to healthcare management and patient care
Potential for further research and improvements
Basic info
papers
machine learning
artificial intelligence
Advanced features
Insights
What are the key components of the SMTAFormer model architecture, and how do they work together to predict readmission risk?
How does SMTAFormer combine static demographic data and dynamic temporal ICU monitor data to learn useful feature representations?
What are the results of the SMTAFormer model in terms of accuracy and AUC, and how do they compare to advanced methods?