Similarity-Distance-Magnitude Activations

We introduce the SIMILARITY-DISTANCE-MAGNITUDE (SDM) activation function, a more robust and interpretable formulation of the standard softmax activation function, adding SIMILARITY (i.e., correctly predicted depth-matches into training) awareness and DISTANCE-to-training-distribution awareness to the existing output MAGNITUDE (i.e., decision-boundary) awareness, and enabling interpretability-by-exemplar via dense matching.We further introduce the SDM estimator, based on a data-driven partitioning of the class-wise empirical CDFs via the SDM activation, to control the class-and predictionconditional accuracy among selective classifications.When used as the final-layer activation over pre-trained language models for selective classification, the SDM estimator is more robust to covariate shifts and out-of-distribution inputs than existing calibration methods using softmax activations, while remaining informative over in-distribution data.