LSFusion: Ladder-Side Attribute Composition for Multi-Aspect Controllable Text Generation

Multi-aspect controllable text generation (CTG), opposite to single-aspect CTG, aims to produce texts that align with multiple attributes. Incorporating attribute information through supervised fine-tuning for pre-trained language models (PLMs) on a related task is effective for single-aspect CTG. However, extending PLMs to accommodate new attributes in such an approach presents challenges in scalability, demanding substantial computational resources and often undermining performance on previously established attribute control capabilities. To tackle these challenges, we introduce LSFusion, a parameterefficient ladder-side fusion method tailored for multi-aspect CTG. Concretely, LSFusion consists of two main stages: first, we train several ladder-side networks on corresponding data for distinct attributes with the parameters of backbone PLMs frozen. Secondly, we fuse these individual networks, which hold knowledge for specific attribute control capabilities, in a non-destructive manner to achieve scalable multi-aspect CTG. Empirical results on diverse benchmarks demonstrate that LSFusion can not only enhance the model’s ability to generate text that adheres to multiple, specific attribute constraints but also reduce the memory cost above 60% compared to full-parameter fine-tuning, making it applicable in lightweight computing devices whose memory constraints are strict while demanding less latency.