Empirical Investigation into New Activation Functions Within LSTM Networks for Time-Series

Activation functions significantly influence the training stability and predictive performance of LSTM models in time-series forecasting. This study empirically investigates three novel activation functions—Normalized ArcSquash, Tangent Squash, and Inverse Linear Squash—explicitly designed to address known limitations of conventional activations, such as gradient saturation and instability. Comparative experiments across five distinct types of challenging time-series datasets reveal that the proposed activation functions consistently achieve competitive or superior performance compared to traditional sigmoid and tanh activations, particularly in handling multi-seasonal patterns and skewed/noisy data. The results highlight improved training convergence and robustness, suggesting that these activation functions could serve as viable alternatives in specific forecasting contexts. However, further validation through broader testing remains essential to fully understand their practical limitations and benefits.