Intrinsic Semiparametric Mixed Effects Model for Longitudinal Manifold-Valued Data

Abstract. This paper introduces a novel intrinsic semiparametric mixed-effects model (ISMEM) to characterize the complex relationships between manifold-valued responses and Euclidean-valued covariates in the longitudinal setting. Manifold-valued data, characterized by their inherent nonlinearity, high dimensionality, and specific geometric structures, present significant challenges for longitudinal analysis. Most existing intrinsic models are developed for cross-sectional data and unsuited for longitudinal studies. To address this limitation, we propose ISMEM, an efficient framework designed for longitudinal manifold-valued datasets that capture such relationships at both group and individual levels. Key features of ISMEM include (i) the integration of fixed and random effects on Riemannian manifolds, enabling analysis at multiple levels; (ii) a semiparametric structure combining parametric and nonparametric components, enhancing both flexibility and interpretability; and (iii) the preservation of the geometric properties of manifold-valued observations, offering improved robustness and interpretability compared to Euclidean-based models. In addition, we develop a two-stage iterative estimation procedure and validate our approach through simulations on the symmetric positive definite (SPD) manifold. Finally, we apply ISMEM to longitudinal data from the Alzheimer’s Disease Neuroimaging Initiative (ADNI), reconstructing and comparing continuous three-dimensional (3D) shape trajectories of the lateral ventricle in Kendall shape space across distinct groups.