Fundamental frequency clustering-driven tensor singular spectrum decomposition for multimodal signal-to-text conversion

Abstract To address the challenge of converting complex multi-sensor signals into text for multimodal large models under noisy conditions, this paper proposes a fundamental frequency clustering-driven tensor singular spectrum decomposition algorithm. The algorithm overcomes three limitations. First, the tensor embedding dimension ratio dynamically searches for the optimal embedding dimension to generate a high-discriminability trajectory tensor, solving the feature confusion problem caused by the empirical dimension setting. Second, a residual energy threshold dynamic stopping strategy based on FTSVD is applied to achieve adaptive hierarchical decomposition of feature components. Finally, a greatest common divisor (GCD)-fundamental frequency clustering mechanism is proposed, which integrates amplitude spectrum peak normalization and the GCD algorithm to complete the structured extraction of fundamental and harmonic frequencies without preset clustering parameters. Results from simulations and experiments demonstrate that the proposed algorithm surpasses traditional clustering algorithms in feature clustering without presetting clustering parameters, effectively solving the problems of modal aliasing and noise robustness in multimodal signals. It offers an efficient solution for complex signal-to-text processing in fields like mechanical fault diagnosis.