Tunable Softplus-Type Activations for Enhanced Gating in LSTM Networks

We propose an extension of Long Short-Term Memory (LSTM) networks. In our approach, the conventional sigmoid and tanh activations are replaced by parametric Softplus-type functions. We consider two configurations. In the first, Gate-only, the input, forget, and output gates adopt a normalized Softplus. In the second, Gate+Candidate, both the gates and the candidate state generation use Softplus variants. The tunable parameter β controls gating smoothness and sharpness. Experiments on synthetic multi-seasonal and non-stationary time-series datasets (600 time steps each) show consistent improvements over standard LSTM and GRU baselines. On the multi-seasonal series, Gate-only with β=0.6 reduced RMSE by 4.2% compared with the best baseline, while Gate+Candidate with β=0.6 achieved a 2.6% gain on the non-stationary series. Ablation confirms that intermediate β values deliver optimal accuracy, while extremes hinder performance. The results demonstrate that Softplus-based gating enhances predictive accuracy, robustness, and convergence, providing a flexible alternative for recurrent modeling of diverse temporal dynamics.