Optimizing stock market prediction and stock trading strategies with deep learning models enhanced by nonlinear feature identification and robust prediction evaluation

This study investigates cross-sectional stock return prediction in the A-share market from 2013 to 2024 with equity and firm-characteristic data available at databases such as RESSET and CSMAR for more than 5,000 listed firms. We introduce the Diff-RMSE method for nonlinear factor identification and the weighted evaluation index (WEI), a finance-specific performance metric that integrates prediction accuracy with market adaptability. Based on 30 market, liquidity, valuation, profitability, technical and risk factors, we compare linear models, tree-based machine learning and deep learning architectures—including GRU, LSTM and Transformer—within a rolling-window forecasting framework. We further translate return forecasts into long–short portfolios to assess economic performance. Our results show that deep learning models, particularly LSTM and Transformer, deliver superior accuracy, more stable WEI scores and stronger tail-risk control than traditional benchmarks. These findings provide practical guidance for quantitative portfolio managers and enrich the literature on machine-learning-based stock-selection models in stock markets.