Atrial Fibrillation Recurrence Prediction

Building upon multimodal fusion and interpretable learning, this study adapts and extends methods for AF recurrence prediction. It specifically compares the impact of four open-source large language models (LLaMA-7B, Phi2-2.7B, Mistral-7B, and MedGemma-27B) on the representation of Holter ECG reports, echocardiography reports, and surgical records. A convolutional neural network is employed in the structured feature branch for representation learning and classification. Furthermore, a generative adversarial network is introduced to augment categories, mitigating the imbalance caused by the scarcity of recurrence samples. The dataset comprises multimodal information from the perioperative period and follow-up, including 28 structured features and textual data from Holter ECG reports, echocardiography reports, and surgical records. The structured features are: Age, Gender, BMI, Systolic Blood Pressure, Diastolic Blood Pressure, AF Duration, Hypertension, Coronary Artery Disease, Diabetes, CHA2DS2-VASc, HAS-BLED, AF type, LAD, LVEF, HbA1C, FPG, TC, TG, HDL, LDL, Albumin, ALT, AST, ALP, Creatine, eGFR, use of class I/III or class II antiarrhythmic drugs.

Preprocessing begins with systematic data cleaning on structured channels: for continuous variables, a combined outlier detection method based on clinically plausible range constraints and the IQR rule is used, with extreme outliers beyond the threshold truncated at quantiles while preserving order information (Supplementary A5). Missing values are handled using a multiple imputation strategy: continuous variables are predicted and imputed using regression models constructed with multiple imputation chained equations (MICE), with mean and variance adjustments to avoid shrinkage; categorical variables are imputed using mode or conditional sampling under Bayesian smoothed frequency encoding to preserve category co-occurrence relationships. To standardize scales, continuous features are z-score normalized, while retaining scaling parameters for external validation; categorical variables are subjected to target leakage-free one-hot encoding or ordinal encoding (for clearly monotonic ordinal features). All encoders are fitted within the training fold and transformed on the validation fold and test set to prevent information leakage. For text channels, lightweight cleaning and normalization are performed on dynamic electrocardiogram reports, echocardiogram reports, and surgical records, including special symbol unification, unit standardization, date and identifier de-identification, and medical abbreviation expansion; subsequently, fragment-based sentence segmentation and keyword localization are used to enhance key point density.

To implement LLM embedding + structured CNN late fusion, we construct four parallel text encoders, each fine-tuned from a pre-trained LLM (LLaMA, Phi-2, Mistral, MedGemma). Fine-tuning combines continued pre-training and instruction alignment: first, domain-specific continued pre-training is performed on de-identified dynamic electrocardiogram reports, echocardiogram reports, and surgical records from our institution to improve clinical terminology coverage and syntactic robustness; subsequently, supervised contrastive learning with a classification auxiliary objective is used to moderately update the LLMs. To balance computational power and portability, LoRA/QLoRA is used for low-rank adaptation, freezing most of the lower-layer weights and opening up partial rank parameters in the mid-to-high-layer attention blocks and word embeddings. Text representations are uniformly taken from the penultimate layer's [CLS]-equivalent pooled vector and a token-attention-based weighted average, concatenated to form a 1024-dimensional embedding, and then linearly projected to 256 dimensions to match the representation space of the structured branch. For fair comparison, the four LLMs independently train their respective text encoders and downstream fusion classification heads, while the remaining training and evaluation procedures remain consistent, resulting in four comparable multimodal models.

The structured branch uses ResNet1D as a 1D CNN backbone to learn local interactions and hierarchical features from the 30-dimensional features.Specifically, the structured vectors are stacked in a fixed order to form a "feature sequence" of length 30, which is fed into a network containing three convolutional blocks: Conv1d (channels=32, kernel size=3, stride=1) + BatchNorm + GELU + MaxPool, followed by a cascade of Conv1d(64, 3) and Conv1d(128, 3). The pooling stride of each layer is controlled to cover different receptive fields and extract cross-feature interactions. The convolutional output is then subjected to global average pooling to obtain a 256-dimensional structured embedding, which is enhanced with dropout and layer normalization to improve generalization. Considering recurrence is a relatively small minority class, a conditional tabular Generative Adversarial Network (GAN) is established within the training fold for the structured branch to perform data augmentation and balance the class distribution. We adopt a conditional WGAN-GP variant adapted to tabular data, where the generator is conditioned on class labels and encoded categorical features to generate synthetic positive samples consistent with the real joint distribution. The discriminator is trained with a Lipschitz constraint and gradient penalty to improve stability. To prevent the augmented data from introducing distribution drift and unreasonable feature combinations, we apply a triple screening process after generation: first, density filtering based on Mahalanobis distance to remove low-density outliers; second, hard constraints based on clinical rules (physiological relationships between indicators and consistency of scoring calculations); and third, an envelope screening of the positive class manifold using a one-class SVM fitted only within the training fold. By performing fold-wise augmentation within the training set, we achieve stratified sampling alignment, while maintaining the natural distribution of the validation and test sets to avoid evaluation bias.

The fusion stage follows a late fusion strategy with attention weighting. In each model instance, the 256-dimensional embedding from the structured branch is concatenated with the corresponding 256-dimensional text embedding from the LLM, forming a 512-dimensional joint representation. This representation is then fed into a multi-head scaled dot-product attention module for learnable cross-modal weighting, with the number of heads set to 4 and the key/value dimension set to 64. A gating mechanism is incorporated to suppress noisy text segments or weakly relevant structured components. The attention output is mapped to the final binary classification logit via a two-layer feedforward network (hidden dimension 256, activation GELU, dropout=0.2), and the cross-entropy loss with label-balanced weights is used as the loss function. Optimization is performed using AdamW (learning rate 2e-4, weight decay 0.01), with cosine annealing and warmup. Training employs stratified k-fold cross-validation (k=5), with patient-level splitting to prevent sample leakage. Within each fold, a validation set is used for early stopping and hyperparameter selection. Evaluation metrics include accuracy, F1-score, sensitivity, specificity, and area under the receiver operating characteristic curve (AUC-ROC), and 95% confidence intervals are reported. To ensure comparability across the four LLMs, all non-text side components, optimizer settings, training epochs, and early stopping criteria are kept consistent, with only the encoder being replaced and fine-tuned individually on the text side. During inference, deterministic forwarding with a temperature of 0 is used to obtain stable embeddings, and the maximum text length and truncation strategy are fixed to avoid bias caused by differences in context length between models.

For explainability analysis, we calculate SHAP values on the log-odds domain of the fused model output to achieve global and individual explanations. For the structured branch, we use DeepSHAP to approximate the marginal contribution of the convolutional pathway to the output, reporting the global importance and interaction effects of each original clinical feature. For the text branch, we combine attention-based token importance with SHAP's text masking estimation to locate the descriptive words driving the prediction. Based on the probability output of the optimal model, we determine the optimal threshold using the Youden's J statistic to classify patients into high-risk and low-risk groups, followed by survival analysis to compare outcome differences between the two groups.