Low-Confidence Nearest Neighbor Substitution for Human-Centric Action Segmentation

We propose a Low-confidence Nearest Neighbor Substitution (LNNS) method to address the critical challenge of inaccurate boundary segmentation in video action segmentation, a problem that severely impacts performance metrics such as F1 and Edit scores. By targeting low-confidence frame predictions, typically located near action boundaries, LNNS replaces them with high-confidence predictions from non-boundary regions, effectively resolving boundary ambiguity. This approach not only reduces prediction errors but also mitigates the common issue of over-segmentation, which often arises in intermediate feature outputs. By providing more reliable action segmentation, LNNS contributes to the development of applications that support human-centric ubiquitous collaboration, such as assistive robotics, smart environments, and collaborative interfaces. To further enhance performance, LNNS incorporates specialized handling of tail classes to account for class imbalance, ensuring robust segmentation across diverse action categories. Additionally, the method employs feature filtering to smooth intermediate features before they are propagated to subsequent processing stages, thereby maintaining temporal coherence in the segmentation output. Experimental validation on three benchmark datasets—50Salads, GTEA, and Breakfast—demonstrates that LNNS consistently improves segmentation quality across various architectures. The results highlight the method’s ability to redefine boundary delineation strategies while offering a com-prehensive solution to persistent challenges in video action segmentation. This work underscores the potential of LNNS to set a new standard for precision and reliability in temporal action segmentation tasks.