Healthcare AI Under the EU AI Act: What Hospitals, Medtech, and Digital Health Companies Must Do Now

Healthcare AI: Ground Zero for High-Risk Compliance

If there is one sector where the EU AI Act will hit hardest, it's healthcare. From AI-assisted radiology and pathology tools to clinical decision support systems and patient triage algorithms, the vast majority of AI applications in healthcare fall squarely into the high-risk category under Annex III.

This isn't just about compliance paperwork. The AI Act introduces obligations that will fundamentally change how hospitals procure, deploy, and monitor AI systems — and the penalties for non-compliance can reach €35 million or 7% of global turnover.

Why Healthcare AI Is Almost Always "High-Risk"

Under Annex III, Section 5 of the AI Act, the following healthcare AI systems are explicitly classified as high-risk:

• AI intended to be used as a safety component of a medical device — including AI-powered diagnostic imaging, surgical robotics, and patient monitoring systems
• AI intended to evaluate the eligibility of persons for public health services — including triage systems and treatment prioritization algorithms
• AI used for risk assessment and pricing in health insurance — any system that influences coverage decisions or premium calculations

Additionally, many healthcare AI systems are already regulated as medical devices under the Medical Device Regulation (MDR). The AI Act layers additional requirements on top of existing medical device obligations, creating a dual compliance burden.

The "Double Conformity Assessment" Challenge

For AI systems that are also medical devices, organizations face a unique challenge: they must undergo conformity assessment under both the MDR and the AI Act. The AI Act explicitly states (Article 43(3)) that a single conformity assessment can cover both sets of requirements — but only if the notified body has the expertise to assess AI-specific requirements. In practice, most medical device notified bodies are still building this capability.

What Healthcare Organizations Must Do

1. Inventory and Classify All AI Systems

Start with a comprehensive inventory of every AI system in use — whether procured from vendors, developed in-house, or embedded in medical devices. For each system, document what the system does and what decisions it influences, whether it falls under Annex III (most healthcare AI will), whether it is also a medical device under the MDR, and who the "provider" and "deployer" are under the AI Act's definitions.

2. Establish a Risk Management System

Article 9 requires a risk management system that runs throughout the AI system's lifecycle. For healthcare AI, this must address clinical safety risks (false negatives in diagnostic AI, delayed treatment recommendations), bias risks (AI trained predominantly on certain demographics may perform poorly on underrepresented populations), data quality risks (imaging quality variations, incomplete patient records), and integration risks (how the AI interacts with electronic health records and clinical workflows).

3. Implement Data Governance for Training Data

Article 10 requires that training, validation, and testing datasets are relevant, representative, free of errors, and complete. In healthcare, this creates specific challenges around patient consent (training data must comply with GDPR Article 9 special category health data while being representative enough to avoid bias), multi-site validation (models trained at one hospital must be validated against data from different clinical settings), and demographic representation (training datasets must include adequate representation across age, sex, ethnicity, and comorbidity profiles).

4. Ensure Human Oversight in Clinical Workflows

Article 14 mandates human oversight for high-risk AI. In healthcare, this means AI diagnostic recommendations must be clearly presented as decision support, not final diagnoses. Clinicians must be able to override, disregard, or reverse any AI-generated recommendation. The system must include "stop" functionality — the ability to immediately disable the AI component. Staff must receive specific training on the AI system's limitations, known failure modes, and appropriate use contexts.

5. Prepare Technical Documentation

Article 11 requires detailed technical documentation covering the system's design, development, testing, and intended use. For healthcare AI, this documentation should include clinical validation results and performance metrics (sensitivity, specificity, AUC), known limitations and contraindications, training data demographics and geographic origin, integration specifications with clinical IT systems, and post-market monitoring plan with clinical outcome tracking.

Hospitals as "Deployers": Your Obligations

Most hospitals won't develop AI systems themselves — they'll deploy systems procured from medtech vendors. Under the AI Act, deployers must use the AI system according to the provider's instructions, assign competent natural persons for human oversight, monitor the system for risks and incidents and report serious incidents, conduct a Fundamental Rights Impact Assessment (FRIA) as required under Article 27 for bodies governed by public law, and inform patients when AI contributes to decisions about their healthcare.

The Procurement Imperative

Healthcare organizations must update their AI procurement processes immediately. When evaluating AI vendors, require evidence of CE marking under both MDR and AI Act, complete technical documentation as required by Article 11, instructions for use that comply with Article 13 including performance metrics and known limitations, a post-market monitoring plan and contractual obligations for ongoing compliance, and evidence of bias testing across relevant demographic groups.

Timeline: What to Do When

The Bottom Line

Healthcare AI compliance is not optional, and it's not simple. The intersection of the AI Act, MDR, GDPR, and clinical safety requirements creates one of the most complex compliance environments in any sector. Organizations that start now will have a significant advantage — those that wait until the August 2026 deadline risk both regulatory penalties and, more importantly, patient safety gaps.

Need specialized healthcare AI compliance support? Search our directory for consultants with healthcare and medical device expertise.