ConSSAC: Conservative Sigmoidal Soft Actor–Critic for Offline Dialogue Policy Optimization

Read full text Summarise with VIRAT
Authors: Fatemeh Shamsezat, Saeed Shiry Ghidary, Ali Mohades
Cited by: 0
References: 31

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Optimizing dialogue policies in multi-domain task-oriented systems is challenging because of the limited opportunities for real user interaction. Pipeline-based architectures offer efficiency and interpretability; however, their dependence on user simulators limits scalability and generalization to real-world conditions. Offline reinforcement learning has emerged as a promising alternative to reduce simulator reliance. However, offline settings amplify inherent difficulties, including a distribution shift and Q-value overestimation when training on imperfect data. We propose Conservative Sigmoidal Soft Actor–Critic (ConSSAC), an offline reinforcement learning method for discrete dialogue action spaces. ConSSAC extends Sigmoidal Soft Actor–Critic (SSAC) by integrating a conservative Q-learning objective for actions that are weakly supported or unobserved in the dataset, while retaining the sigmoidal policy architecture for calibrated multi-label action selection. This design enables the reliable exploitation of indistribution behaviors. In addition, we introduce RewardWOZ-21, a turn-level reward–annotated extension of MultiWOZ 2.1, which enables scalable and fine-grained offline training. Experiments on ConvLab-3 indicate that ConSSAC achieves higher performance than SSAC, PPO-based models, and recent LLM-based policies under the evaluated settings. It achieves a 4.1% absolute improvement in F1 by reducing false positives and false negatives in dialogue-act selection, while preserving computational efficiency comparable to that of standard actor–critic architectures. These results suggest that combining conservative offline reinforcement learning with sigmoidal policy parameterization is associated with improved robustness and reliability in multi-domain task-oriented dialogue systems.

Fields of Study & Subjects

Speech and dialogue systems Reinforcement Learning in Robotics Intelligent Tutoring Systems and Adaptive Learning

Publication details

Journal / source
IEEE Access
Volume
14
Pages
52309–52323
Publication date
1 January 2026
Language
EN