Prediktion av kronisk njursjukdom och typ 2 diabetes i patientjournalsdata med hjälp av transformator-kodare

This thesis evaluates whether a transformer-encoder trained on full longitudinal EHR sequences improves onset prediction of chronic kidney disease and type 2 diabetes over an XGBoost model trained on aggregated features. Patient histories from FinnGen and Kanta were represented as sequences of diagnoses, medication purchases, and biomarker measurements. Three embedding methods for biomarkers were examined: quantile-based discretization, as is widely used in EHR models, a continuous embedding function and a feature-wise modulated variant of this function. The transformer-encoder was pretrained on sequences using masked event modelling, sequence-level objectives and mortality prediction. It was subsequently fine-tuned for disease onset prediction, by optimising the loss of the binary classification objective and the loss of an auxiliary regression task of predicting the label-defining biomarker. Performance was primarily assessed using AUROC, AUPRC, binary classifica-tion loss and expected calibration error. Models were evaluated on held-out test sets comprising 36,436 individuals (220 CKD cases) and 15,791 individuals (185 T2D cases). The transformer-encoder and XGBoost achieve statistically equivalent performance across both chronic kidney disease (AUROC≈0.91) and type 2 diabetes (AUROC≈0.87), with discrimination and calibration metrics in the same numerical ranges. The continuous embedding methods consistently matched or surpassed the uantile-based embedding, suggesting that continuous encodings preserve clinically relevant information lost during discretization. Ablation experiments show that omitting pretraining produces the largest drop in performance, and that removing the auxiliary regression objective leads to a small but consistent decrease, as well as overfitting earlier in training. The findings suggest that aggregated features already capture a large proportion of predictive information under these conditions, resulting in no statistically significant advantage for models using full longitudinal sequences. Interpreted in light of recent large-scale EHR studies, the results are consistent with the hypothesis that the dataset size, particularly the number of positives, is below the scale at which transformer models show consistent, statistically significant advantages over aggregated-feature baselines for predicting disease onset.