Towards improved political discourse quality using AI agents

Recently, large language models (LLMs) have enabled an in-depth, large-scale analysis of many complex phenomena. One such domain is political discourse on social media, which can serve as an indicator of many issues in society. In this work, we analyze how fine-tuning LLMs and artificial intelligence (AI) agents on social media posts related to the discussion of specific political commemorations can be used to both aid in discourse analysis and shift the messages produced by LLMs into a desired direction, e.g., changing messages from uncivil to civil. We first present a theoretical framework for analyzing political discourse based on eight discourse-quality indicators, followed by two methodological contributions. First, we show that models finetuned on a small number of manually-labelled examples are better at detecting discourse quality indicators than larger, non-finetuned models. Second, we finetune models on examples that match specific discourse quality indicators and demonstrate how this process can shift the messages produced by those models to more closely align with the desired indicator, e.g., civility. The findings imply that LLM-based AI agents could be used to moderate public discourse and improve its quality.