Miriam and the AI she never uses
Miriam is a nurse at a mental health institution. Her employer recently introduced an AI module that automatically suggests care plans based on client history. The intent: less paperwork, more time for meaningful conversations.
Miriam doesn't use the system. Not because she is unfamiliar with technology — she has been working with it for years. But because she doesn't know how the AI arrives at its conclusions. 'What if it misses something,' she says. 'Then I've signed off on a plan I didn't fully create myself.'
Miriam is not the exception. A significant share of healthcare professionals systematically disregards AI recommendations — not out of resistance to technology, but out of an informed lack of trust. An AI system that isn't trusted won't be used.
“Trust is the real innovation in AI for healthcare.”
What makes AI trustworthy?
Trust in AI is not a feeling. It is the result of concrete technical and organizational choices. A care provider must be able to understand why an AI system makes a particular recommendation. In care contexts, explainability is essential for transparency, GDPR accountability (Art. 13–15 and 22), and — for high-risk AI systems — AI Act compliance.
Human-in-the-loop is not a feature you can disable — it is an architectural choice. In trustworthy healthcare AI, the professional always decides. The system presents options, provides context, and flags risks. The clinician validates, corrects, or rejects.
Patients entrust healthcare organizations with the most sensitive information there is. That data must never be used for purposes the patient has not approved. Privacy-by-design is the minimum standard.
Safe-by-design: the scientific foundation
Yoshua Bengio, Turing Award laureate, is working through his nonprofit LawZero on a new generation of AI: 'designed to be trustworthy and safe.' The premise: current frontier systems are opaque and misaligned with human objectives.
Safe-by-design means: no self-directed autonomy. AI systems that self-correct without human intervention are dangerous in healthcare. Safe AI cannot do more than what it was designed for — operating toward explicit, verifiable objectives.
The European AI Act classifies AI in healthcare as 'high risk' and sets stringent requirements for transparency, auditability, and human oversight. This is the standard healthcare needs — and what regulators require.
How CareHub implements this
The CareHub ecosystem distinguishes between processes where AI adds value and processes where human judgment is irreplaceable. Every AI recommendation in the platform is accompanied by an explanation at an accessible level — not as a technical metric, but as context the professional can act on.
Every AI interaction is logged: what the system suggested, what the care provider decided, and why. This enables both auditing and learning. The log is immutable, retention-compliant, and organizationally decoupled from clinical data, safeguarding both privacy and forensic utility.
Client data stored in the CareHub platform is never used to train AI models outside the client's own organization. Data sovereignty is a hard requirement — not a marketing term, but a contractual commitment underpinning the data processing agreement and system architecture.
The patient perspective: trust starts with transparency
Trust in AI is not solely a matter for professionals. Clients and patients have the right to information about the use of AI in their care. GDPR enforces this legally (Articles 13–15 on automated decision-making); in practice, informed patients demonstrate better engagement and higher satisfaction.
In a human-centered AI implementation, the client is informed upfront that AI is supporting — for example, in matching a care provider or drafting a summary letter. They can ask what the AI proposed, and may object to automated decision-making where applicable.
This may sound burdensome. In practice it is lightweight: a notice in the client portal, a brief explanation on request, and a 'how AI works in our care' page accessible to anyone who wants to learn more. The effect: clients feel taken seriously. The alternative — concealing or downplaying AI — inevitably leads to a breach of trust when it comes to light.
What regulators expect to see
The Dutch Data Protection Authority, the Healthcare and Youth Inspectorate, and at the European level the supervisory authorities under the AI Act all focus on comparable requirements. First: that AI deployment is documented, including a DPIA, conformity assessment, and risk management. Second: that human oversight is operationally effective, not merely on paper. Third: that incidents are detected, reported, and learned from.
For healthcare providers and health tech vendors, this means the burden of proof has shifted. Before the AI Act, 'we gave it careful consideration' was often sufficient. After it, the standard is: demonstrate that the system performs as documented, on the target population it was designed for, with acceptable error rates.
A practical approach: build a logging and evaluation framework from the outset that serves both internal learning and external accountability. The technical registration requirements of the AI Act overlap substantially with what is needed to improve AI in production anyway. It is not additional work; it is the same work, done deliberately.
In practice: how to build a trustworthy AI deployment
Four steps, in sequence. First: scope the problem precisely. Which specific decision or task are you supporting? Not 'reducing administrative burden' (too broad), but for example 'automatically generating a draft care plan following intake at unit X.' The narrower the scope, the more straightforward the evaluation.
Second: design the human checkpoint upfront. Who reviews the AI suggestion, with what information, and what action is required? 'Approve,' 'modify,' 'reject with reason.' That last option is essential: without a stated reason, the system cannot learn.
Third: build logging from day one. Not later, not as a compliance checkbox. Logging is your product improvement tool. Data you do not capture now cannot be retrieved six months from now to evaluate the AI.
Fourth: define success criteria in advance. What metrics must the AI achieve in production? Acceptance rate? Time saved per case? Concordance with expert judgment on a sample? Without predefined success criteria, every pilot outcome becomes grounds for 'we need more research' rather than a clear go/no-go decision.
