Through the looking glass: ethical considerations regarding LLM-induced hallucinations to medical questions

Large language models (LLMs) are artificial intelligence (AI)-based systems which have been trained on extensive records derived from articles, books and other internet-based content (typically from a mix of both publicly available and proprietary content). (1) LLMs have become rapidly embedded in digital health ecosystems, promising to augment healthcare delivery, research, and patient engagement.(2) Their ability to generate coherent, human-like responses positions them as transformative tools for health information dissemination, clinical decision support, and patient self-management. Yet, their probabilistic foundations carry inherent limitations: they are prone to "hallucinations," the confident presentation of factually incorrect or biologically implausible outputs. (3) While such hallucinations may be benign in creative domains, akin to the favourite book by Lewis Carroll 'Through the looking glass', in healthcare the consequences of such hallucinations can be profound. This opinion manuscript examines the key ethical and practical implications of such hallucinations, with particular attention to accountability, transparency, and the shared responsibilities of developers, deployers, and end-users.As a general approach, LLMs generate responses by predicting likely word sequences rather than assessing factual accuracy. Therefore, there are several technical reasons that may affect hallucinations. The primary one is the biases in training data, where for example a model would assume the universality of a pattern, simply because it occurred frequently within a given training dataset. Moreover, when datasets are incomplete or contain lots of noise such as inaccuracies or contradictions, LLMs can learn correlations across these sources, but when asked to reconcile them, they may synthesize a 'fictional' middle ground, opt for a wrong pattern observed in the noise, and thus produce authoritative statements even when no reliable data exists. If a model is trained without explicit grounding in fact-checking or truthfulness, hallucination risks increase, because the model is never penalised for inventing, and only rewarded for coherence and diversity. (4) In turn in clinical practice, this can result in plausible but incorrect clinical content, ranging from misdescriptions of symptoms to fabricated treatment guidelines -reflecting the collective accuracy of symptoms descriptions in the harvested databases. (5) These risks are amplified in medicine where additional to potential misdescriptions, misinformation can undermine evidence-based practice, confuse patients, or-in the worst cases-cause harm. However, not all hallucinations carry the same weight. Unlike overtly dangerous outputs, which may be swiftly identified and corrected be experienced healthcare professionals, subtler hallucinations can often persist undetected. These smaller inaccuracies accumulate in public discourse, shaping patient perceptions and subtly eroding trust in digital health over the longer-term. Their collective impact is nontrivial: while each instance may appear harmless, the long-tail effect threatens the integrity of health information ecosystems.The central ethical questions revolve on fairness and accountability: who is responsible for harm caused by hallucinations? From a technical perspective, accountability may be understood solely as the need for ensuring quality on the algorithm performance and the protection over the data. However, the AI impacting an individual patient is integrated within the healthcare system. Thus, accountability should be viewed through a more comprehensive framework, able to accommodate such a complexity at the level of the product, the process and the actual decision. (6) Five major stakeholder groups warrant consideration:1. Developers -Engineers and companies that design LLMs carry ethical duties to minimize hallucination risk through technical safeguards, rigorous testing, and transparent disclosure of limitations. Steps have been taken to this direction with testing options emerging during LLM development. (7) 2. Deployers -Institutions and platforms integrating LLMs into health systems have the ethical imperative to ensure (as with any equipment or software) that contextual validation, risk assessment, and alignment with clinical standards take place to pre-defined local and national standards (8). Furthermore, the implementation of such systems touches the principle of fairness, so that any improvements in patient service are as equitable as possible.3. End-users -Both clinicians and patients using these systems are anticipated to exercise critical judgment, recognizing that AI outputs are advisory, not authoritative. ( 9) However, to do so, clinicians and patients alike need to acquire sufficient digital health literacy and be educated to use such systems to best effect. (10) 4. Publishing platforms -Publishing platforms for international scientific journals so far face a double challenge of promoting accurate peer-reviewing for LLMs presented as part of submitted manuscripts (11), as well as potentially adopting LLM as an aid for the peer review process itself. (12) The ethical perspective here, is this of trust, and how to maintain the trustworthiness of the wave of new publications that results from the rapid technological advancement.5. Regulators -The efforts to regulate effectively and thereafter maintain control over regulated domains, e.g., in the European Union (EU) with the AI Act (13), the China Cybersecurity Law (14) and the UK pro-innovation, context-specific approach, underscore the regulatory urgency of implementing national and international regulations. (15) However, even more effective would be the creation of an aligned regulatory framework in the near future (16).The ethical complexity lies in the interaction of these key stakeholder groups and their distinct roles. While end-users cannot be expected to independently audit every claim (and due to the technological complexity objectively cannot do so), neither can developers disclaim all liability. Moreover, there is an acute need to translate abstract AI ethical principles into actionable frameworks within healthcare, where the social good is a critical underpinning. This urgency is proportionate to the risk, as AI-driven technologies have the potential to be both powerful and disruptive once implemented. (17) Publishing platforms need to create consensus guidelines for the review and assessment of LLMs both as in-house tools, as well as engines driving scientific discovery. In a similar manner, solid assessment criteria need to be developed and implemented by the deployers. Thus, it is the strong opinion of the authors that a dynamic, multi-stakeholder ethical accountability framework is essential.Transparency is at the foundation of ethical accountability, both a technical and ethical imperative in the healthcare-related LLM implementation, as -at a minimum-users must be clearly informed about the probabilistic nature of LLM outputs and their limitations in terms of generating hallucinations, and in turn be able to inform patients' and clinicians' decisions (18). Several AI guidance documents have been published recently on this basis and refined by many actors (19)(20)(21). Theoretically, this can include the disclosure of data provenance (esp. for training data), the communication of uncertainty (e.g., with confidence scores), and the clear indication throughout that any outputs are non-substitutive for medical advice. In this way, the discourse shifts from blind reliance to informed engagement (22) and allows clinicians and patients to situate LLM-generated information within evidence-based frameworks. However, this is an ideal and static scenario, that anticipates high levels of digital literacy and understanding from all parties, as well as the ability to identify technological nuances. However, in the case of the LLM rapid technological development and logarithmic increase of related publications, it is not sufficiently practical. Thus, the authors support the need for a dynamic ethical accountability framework.In this model, i) the developers ensure ethical governance, transparency and safety by-design; potential quantifiable metrics can include the number/proportion of data sources with fully documented provenance, and/or for existing LLMs the % of new model versions passing pre-deployment safety tests (for both ideally at an eventual 100%)ii) the deployers operationalise safety and fairness in real-world use; potential metrics can include a report for the number of LLM safety or fairness incidents reported per 1,000 clinical interactions (with a targeted decrease by 30-50% within 3 years) and/or a report for the number of clinical departments using the LLM that fully implement required safety policies;iii) the end-users are co-creators of consent models and define acceptable, even personalised, risk thresholds; potential metrics can include a published number of clinical pathways that include LLMspecific informed consent steps, complemented by the reported increase in clinician competency scores in AI/LLM literacy testing (with a targeted improvement over time, relevant to the institutions); iv) publishing platforms provide oversight of the moving field and emerging requirements; potential metrics can include the average time taken to issue updated editorial requirements following major AI regulatory changes (eventual target: