Agential harms of chatbots: an ethical perspective

Abstract The widespread integration of AI-assisted digital technologies, particularly those leveraging Large Language Models (LLMs) and generative AI in applications like chatbots, presents both significant benefits and novel ethical challenges. While these technologies enhance efficiency and democratise access to services, their rapid deployment necessitates a critical examination of potential harms. There are ambitious forecasts of the revenue the chatbot industry is expected to generate over a 5-year window, and given the accelerated deployment, this paper examines its ethical risks. We introduce the concept of “agential harms”, which subsumes many others (psychological, developmental, recognition) that are often referred to as intangibles. Unlike traditional harms (physical, financial), agential harms subvert a person’s capacity to be an agent—to self-govern, develop, and be socially recognised—caused by an AI system’s convincing imitation of agency. These harms occur when the AI, by acting as a conversational partner or cognitive substitute, subtly undermines the very faculties that define human agency. Using AI assistant chatbots as a test case, we clarify the nature and significance of agential harms, distinguishing them from conventional categories of harms.