Non-boundary covariance matrix estimation in generalized linear mixed effects models using data augmentation priors

Boundary estimates of random effects covariance matrices commonly arise when using maximum likelihood (ML) estimation in generalized linear mixed effects models, leading to numerical challenges and affecting statistical inference. To mitigate this, we introduce penalties to the likelihood function derived from conditionally conjugate priors for the covariance or precision matrices of the random effects. Our choice of penalties (priors) allows representation through pseudo-observations, enabling implementation of the proposed penalized estimator within the existing ML software by augmenting the original data. We derive a procedure for constructing these pseudo-observations, a non-trivial task because their likelihood contribution must match the functional form of the penalty and depend only on the covariance or precision matrix of the random effects. Our method includes penalty parameters that can be set using existing prior knowledge or, when no reliable prior information is available, via a novel fully data-driven procedure that eliminates the need for prior specification. Through simulation studies under realistic scenarios, we illustrate that the proposed approach can provide improved estimates of random-effects covariance matrices compared with competing methods in the settings considered. The approach is further illustrated on real-world data.