HRL-GridMM: Hierarchical Reinforcement Learning for Adaptive Delta-Neutral Grid Market Making in Cryptocurrency Spot–Futures Markets

Market making in cryptocurrency spot–futures markets requires simultaneously managing grid order placement, cross-venue hedging, inventory risk, and adverse selection under rapidly shifting market regimes. Existing reinforcement learning (RL) approaches for high-frequency trading treat the problem as directional position-taking on a single venue, ignoring crossvenue delta-neutral hedging—the dominant institutional practice. Conversely, rule-based grid systems rely on static heuristics that cannot adapt to non-stationary dynamics. We propose HRL-GridMM, a hierarchical RL framework that decomposes market making across three timescales: a Transformer-based macro agent (1-minute) for regime detection and hedging mode selection; a Soft Actor-Critic meso agent (10-second) for continuous grid parameter optimisation; and a deterministic rule-based micro engine (tick-level) that preserves structural market-making guarantees. A Temporal Convolutional Network (TCN) auxiliary module provides learned adverse selection predictions that enrich the RL state space. Simulation experiments on four cryptocurrency pairs across two exchanges over three months show that HRL-GridMM achieves 17% higher annualised Sharpe ratio than the hand-tuned grid baseline (3.38 vs. 2.89) while maintaining comparable drawdown, and outperforms state-of-the-art RL trading methods adapted to the market-making setting by 14–32% in riskadjusted returns. A 10-day live deployment pilot on Gate.io (69K fills, 672K USDT volume) confirms production viability and reveals the specific failure modes—adverse selection and regime-shift drawdowns—that the proposed RL enhancements address. Keywords: Market making · Hierarchical reinforcement learning · Grid trading · Delta-neutral hedging · Adverse selection · Cryptocurrency