Privacy-preserved LLM Cascade via CoT-enhanced Policy Learning

Large Language Models (LLMs) have attracted significant attention for on-device applications, delivering strong performance across a variety of real-world tasks. However, hardware constraints on edge devices limit model capacity, often resulting in suboptimal performance. A promising remedy is LLM cascading, where a lightweight local model defers selected hard queries to a more capable server model for response generation. While prior work has primarily optimized the performance--cost trade-off, real-world deployments must also address privacy concerns i.e., user information leakage, a requirement that remains largely overlooked. In this work, we go beyond existing confidence- and logit-based cascade methods and propose P3Defer, a novel Chain-of-Thought (CoT)-enhanced policy learning framework coupled with a private memory for privacy-preserved deferral decision-making. By jointly optimizing for performance, cost, and privacy, our approach improves performance-cost cascade efficiency while mitigating privacy risks. Extensive experiments on three benchmark datasets demonstrate that P3Defer consistently outperforms existing methods in both accuracy and privacy preservation.