Deep Reinforcement Learning for Financial Trading: Enhanced by Cluster Embedding and Zero-Shot Prediction

Deep reinforcement learning (DRL) plays a pivotal role in decision-making within financial markets. However, DRL models are highly reliant on raw market data and often overlook the impact of future trends on model performance. To address these challenges, we propose a novel framework named Cluster Embedding-Proximal Policy Optimization (CE-PPO) for trading decision-making in financial markets. Specifically, the framework groups feature channels with intrinsic similarities and enhances the original model by leveraging clustering information instead of features from individual channels. Meanwhile, zero-shot prediction for unseen samples is achieved by assigning them to appropriate clusters. Future Open, High, Low, Close, and Volume (OHLCV) data predicted from observed values are integrated with actually observed OHLCV data, forming the state space inherent to reinforcement learning. Experiments conducted on five real-world financial datasets demonstrate that the time series model integrated with Cluster Embedding (CE) achieves significant improvements in predictive performance: in short-term prediction, the Mean Absolute Error (MAE) is reduced by an average of 20.09% and the Mean Squared Error (MSE) by 30.12%; for zero-shot prediction, the MAE and MSE decrease by an average of 21.56% and 31.71%, respectively. Through data augmentation using real and predicted data, the framework substantially enhances trading performance, achieving a cumulative return rate of 137.94% on the S&P 500 Index. Beyond its empirical contributions, this study also highlights the conceptual relevance of symmetry in the domain of algorithmic trading. The constructed deep reinforcement learning framework is capable of capturing the inherent balanced relationships and nonlinear interaction characteristics embedded in financial market behaviors.