Mathematical Foundations of Beta Diversity: Why Common Metrics Fail in Microbiome Analysis

In microbiome studies, beta diversity quantifies dissimilarity between samples and is often visualized using ordination techniques. It enables researchers to characterize ecological structure, compare microbial communities, assess environmental or host-driven heterogeneity, and track longitudinal shifts over time. Although many diversity indices were originally developed with practical goals in mind, they lack a unified framework to ensure theoretical rigor and validity. This gap makes it challenging for researchers to evaluate and select appropriate beta diversity measures for microbiome analyses, potentially leading to biased analyses and invalid conclusions. To bridge the persistent knowledge gaps, we systematically evaluate the commonly used beta diversity measures according to key mathematical properties, including whether they are true metrics, conform to Euclidean geometry, and satisfy conditional negative definiteness. We show that their violations can compromise downstream analyses such as PCoA, PERMANOVA, and kernel-based tests. In addition, drawing on mathematical consensus, we introduce a novel four-category classification of beta diversity measures: scale difference, difference scale, Hamming difference, and distribution difference. Complementing this framework, we build diagnostic tools for assessing Euclidean validity and develop remedial strategies that correct problematic dissimilarity matrices while preserving ordination structures. We demonstrate the effectiveness of these solutions using real-world microbiome datasets. These results establish a unified framework for evaluating beta diversity in microbiome research, supported by an R package, interactive Shiny app, and step-by-step tutorials. The framework provides a clear roadmap for selecting and refining dissimilarity metrics, paving the way for future methodological advances.