Digital threat in healthcare

AI technologies have significant potential to change diagnosis, treatment planning and operational processes in healthcare. However, the problem known as “AI hallucination” brings new and serious risks alongside this potential.

The tendency of AI systems to produce false or fabricated information can harm patient safety, clinical decision-making and health professionals’ trust in technology.

What is AI hallucination and how does it appear in healthcare?

AI hallucination is when an AI system (especially a large language model or another generative AI system) produces information that appears consistent with the data it was given but is in fact nonexistent or wrong. This can happen when the model makes a logical error while learning complex patterns or filling in missing data.

In healthcare, hallucinations can take the form of seeing lesions that do not exist in medical images, suggesting incorrect diagnoses, recommending treatments with no scientific basis or producing false information in patient records.

Possible negative effects and risks

AI hallucinations can cause harm in several critical areas of healthcare:

• Incorrect diagnosis and treatment planning: Hallucinations can lead to misdiagnosis and, as a result, to inappropriate or harmful treatment plans. This directly endangers the patient’s health and can have irreversible consequences.
• Risks to patient safety: Incorrect information can affect clinical decisions and put patient safety at risk. The consequences can be severe, especially in cases that require emergency intervention.
• Loss of trust among physicians: Unreliable outputs undermine the trust of physicians and other health professionals in these technologies. This can slow the integration of AI into healthcare services.
• Wasted resources and operational inefficiency: Incorrect diagnoses and treatment recommendations can lead to unnecessary tests, wrong medications and unnecessary hospital stays, wasting resources in the health system.
• Legal and ethical responsibility: Wrong decisions caused by hallucinations can create serious legal liability and ethical dilemmas. This is a significant risk for both AI developers and healthcare organizations.
• Damage to patient trust: If patients worry about the reliability of AI systems used in their care, their trust in the health system as a whole can decline.

Ways to reduce the risk

Reducing the risk of hallucination and ensuring the safe use of AI in healthcare requires a multi-faceted approach:

• High-quality, verified datasets: Models should be trained on comprehensive, diverse and up-to-date datasets verified by human experts.
• Showing sources and reasoning: A clinical recommendation should come with its source and reasoning, so that the physician can verify the output.
• Human oversight: AI outputs should be used as decision support. The diagnosis and treatment decision stays with the physician.

For ways to check AI outputs against their sources and reasoning, see the guide Getting a safe second opinion from AI as a physician.