Looking into the Black Box: The Risks and Pillars of AI Governance in MedTech
In 2026 the MedTech industry is going to be defined by technical innovation, deeper levels of connectivity, and yes - the dreaded attempted integration of AI into nearly every facet of patient care. Now despite study after study citing the failing of these attempts, the reality is, the monkey is out of the cage when it comes to AI. There is no going back, so if you’re going to do it, you need to know what you’re getting yourself into.
This is especially challenging for people in and around medical device security, governance, risk, and compliance (GRC). Not only because integrations are generally a pain in the ass, but also because the chance getting them seamless and objectively functional without testing in customer’s actual environment is near zero. Keep in mind, when I say this, I’m really only talking about product-to-product integration via services or serialized connections of some sort. When you factor AI into the equation, now we’re dealing with uncharted, dark and dare I say it… risky territory.
The current frameworks we use for governance around the world are built on the basis that the world is deterministic, and it is increasingly more clear that we are all now living in a probabilistic one.
While my time in MedTech itself has been limited, the majority of my time has been spent on deeply entrenching myself in its regulations, getting an understanding of how real R&D is accomplished, how new product development, pre-market and post-market works in the real world. One of the conclusions I’ve come to is that despite best efforts, I believe our industry faces a widening gap between executive policy and technical execution - largely driven by lack of focus on AI safety and risk management.
This should come as no surprise as the companies that make the most popular models and products on the market today don’t want anyone looking underneath the hood. “Just accept what you’re given and move on”, “ignore our revenue vs. valuation claims” or “regulating us will only slow us down them down because if you do that China wins” they say; all of which are absurd statements and that should disturb all of us.
But luckily, things are changing and quickly.
People are starting to realize that not everything is as it seems - both technical and non-technical folks are aligning. This is why the time is now to lead effectively, to move beyond symbolic compliance in the face of these innovations and the challenges they present, and into rigorous technical mechanics of AI safety, observability, and risk management.
The Reality of Non-Deterministic Systems
Traditionally, software governance is based on and around predictability of behavior and usability. This is especially true of medical device software and medical “cyber devices” (going to use the FDA definition for that turn of phrase).
It’s pretty straight forward; if you define a requirement and test it correctly, the software behaves the same way every time it is deployed. This by definition is a deterministic system.
AI fundamentally changes this.
ALL, and I do mean ALL, machine learning models and LLMs are non-deterministic. They are based on and built around stats… probabilities. A model that performs with 99% accuracy in a controlled testing environment can change its behavior immediately when introduced to a new environment, with new data, and with new biases that skew those probabilities. Production environments of all sorts, not just those in a medical setting are filled to the brim with a mushed up amalgamation of legacy protocols, hordes of unsegmented data silos, wildly unpredictable traffic loads, and of course - dirty stinky humans. All of this is a recipe for triggering “model drift”.
When model drift occurs, the baseline performance of a model (what was tested in a controlled setting, what expected outcomes were verified) degrades over time. Data the model was trained on deviates and morphs which causes the model to attempt to adjust, which changes its outputs and general behavior - often without users knowing those changes have occurred.
The other variable of this to note is that it isn’t an unlikely scenario for these changes to happen rapidly and often. Many people believe that behavior changes take a long time, much like it does for people - but that is not the case here. Day to day use of AI tools and systems only interact with a facade on the very tip of the iceberg that is the model itself. We never interact with the actual model, only what the model’s designers have connected an interface and prompting terminal to. That alone is scary enough when you consider it; even scarier when you realize every time / any time you ask the model to do something it hasn’t done before it could instantly change any/all of its biases.
In any case, this causes a real problem for traditional governance strategies and frameworks - mostly because they remain focused on “point-in-time” snapshots. These will all fail against the dynamic non-deterministic behaviors AI products and services present us with; and that is why we have to shift away from static oversight and into continuous operational monitoring capabilities. That is why I am proposing today for practitioners and manufacturers to focus on three core pillars:
Implementing AI Observability
Overemphasizing the use and management of model cards and model base-lining
Building threat modeling capabilities targeted at AI specifically
Pillar 1: Implementing AI Observability
![What's in the box?" - David Mills [Se7en] - Samsung Members](https://substackcdn.com/image/fetch/$s_!f8JZ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F426d36cf-c683-4562-b115-d4781cdfe127_998x524.jpeg "What's in the box?\" - David Mills [Se7en] - Samsung Members")
Above I briefly mentioned the idea of “AI observability”, this is a fancy way of saying “monitoring what the internal state of a model is doing / how it’s behaving against expected behavior and outcomes”. It is - in my humble opinion - one of the least sexy, yet most critical aspects of AI engineering that is getting too little focus. It is vital to governance and the safety of AI because, as I said before, we are only typically interacting with a facade of the model.
The happy little blue circle in the chatGPT app is not the actual model itself. This presents a ton of difficulty if we cannot actually see what’s in the black box that is the actual model. For example, as we’ve covered, in the context of an AI system, app, or service - it can be “up” 100% of the time all while providing entirely inaccurate outputs. In the clinical setting - this again, is beyond scary.
We must go beyond only time to market focus when it comes to AI in MedTech, and put our focus on continuous AI Observability and reporting.
Having the ability to keep tabs on the internal state of a model to detect failures before they impact a patient isn’t a nice to have, it is and should be non-negotiable.
For example, some of the common areas to focus on should be the following (amongst many others):
Data Drift Monitoring: Tracking the statistical distribution of incoming data to make sure that variances in data are not substantial enough to affect the baselined and validated behavior of the underlying model or its outputs.
Concept Drift Detection: Ensure tracking and spot reporting capabilities for detecting if the relationship between inputs and outputs of a model changes logic, behavior, or core functions.
Model Integrity: Continuous monitoring to ensure the model hasn’t been compromised or “poisoned” by external data inputs or prompts.
From a regulatory perspective, observability, reporting, and evidence is and should remain the cornerstone of modern post-market surveillance.
Regulatory bodies like the FDA are increasingly moving toward total product lifecycle (TPLC) oversight, which assumes that “static” software no longer exists. Think about the impact that stance can have against a non-deterministic feature or capability. At minimum if a manufacturer cannot provide real-time evidence that their model is operating within its validated bounds, they will risk regulatory action, denied market access, getting pulled off the market, or the inability to fulfill Predetermined Change Control Plans (PCCPs).
Not to mention that having these capabilities and ability to observe available to customers is going to be essential.
It is no surprise that operational reliability and integrity is the name of the game inside hospital networks. AI is not and should not be immune to the rigor that is involved with understanding operational downtime and issue remediation per customer.
Overall, as the old adage goes, “you don’t know what you don’t know” or as those who’ve been in security cringe at “you can’t protect what you can’t see”. Observability is the key to getting AI done, and done safely.
Pillar 2: The Role of Model Cards in AI Governance
Moving on now to another form of transparency - model cards.
Model cards act as a one stop shop for technical and product specifications for a given AI model. But more than that, they provide a standardized way to track, manage, and communicate about the capabilities and limitations of an AI model to auditors, clients, stakeholders, and engineers.
Having a small standardized model card template and a repository to consume / house them is one of the easiest and most cost effective first steps organizations can take to get their heads around AI product development; and from my perspective as a GRC professional, it makes it easy for teams like mine to then consume information without having to bother engineers or process owners.
At minimum a robust Model Card for a medical device should detail:
Intended Use and Out-of-Scope Use: Explicitly stating where the model should and should not be used and for what use cases.
Training Data Demographics: Detailing the diversity of the training set to identify potential biases. This can also be used as the foundation for observing variations in data drift.
Performance Benchmarks: How the model performed across various use cases and what the baseline optimal performance looks like.
Environmental Factors: Specific hardware or network requirements necessary for the model to maintain its optimal performance.
But what is really interesting about model cards is its ability to be recycled for different stakeholders without having to overhaul the data they contains.
For example, when it comes to leveraging model cards externally, customers can use model cards to glean the necessary context to exercise professional judgement on the efficacy of a given model against a given use case. That sounded confusing, let me parse this a little bit more.
In MedTech, many AI products have uses cases that might classify them decision-support tools. But a clinician cannot effectively use a given tool if they don’t know its “blind spots”, it’s limitations. The model card brings transparency around both what a given tool is meant to do, and not do. This doesn’t make your AI seem shitty, it makes it seem targeted in its purpose, and that does nothing but empower your customers to put themselves in the loop (i.e. why I named this substack the way I did) and to be an effective gut check on the tool’s performance vs. being a bland, passive consumer of its output(s).
Now, when it comes to product implementation and integration, I’ve had enough experience on both the buy and sell side of software to know that if there’s something going into a network, then there is no such thing as “plug-and-play”. Each network is different, therefore each implementation must be different as well. This is especially true with AI, not just for the aforementioned reasons, but also because of the broad spectrum of possible use cases, leakage, and recursive effects AIs can cause to not just products they are hosted in, but also in systems they’re connected to.
By having a standardized and easy to consume model card, hospital administrators and technical staff can determine with reasonable assurance, up front, whether or not a model’s training environment matches or is close enough to their current infrastructure and resource availability to support the model’s technical requirements, that their patient demographics fit within the bounds of data the AI is effective with and has trained on, etc.
This level of detail can help decision makers in our client organizations make informed decisions so that they don’t start deploying high-performance tools in environments where they’re susceptible to fail at worst, under perform at best. This protects the hospital, it’s investment, and most importantly their patients.
Lastly, model cards can be your first line of defense against legal and ethical liability in the event of an AI-related adverse event. Clear documentation of a model’s limitations and intended uses protects manufacturer's by establishing a “technical nutrition label” that was available and provided to the user.
All in all, model cards ensure that the utility and limitations of a given model is well understood by all parties involved and that everyone, from the manufacturers, to regulators, to providers, are operating from a shared understanding of what a given model can do and cannot do.
Pillar 3: A New Framework for AI Threat Modeling
Threat modeling has become one of the central focuses for medical device product security teams, and for good reason. But before we dive into this, let’s quickly recap what threat modeling is, so that we can look a bit more constructively at the short comings of legacy approaches to threat modeling when it comes to AI.
Traditional Threat Modeling
Put simply, threat modeling is an exercise used to identify potential security gaps. This typically involves business folks, product managers, security personnel, architects, and developers of a given product.
The idea is to talk about everything you know about your given product idea before you start building it. What are it’s features and capabilities? Where will it be deployed? What will it be connected to? What kind of data will it handle? What will it do with the data? How is data processed through the various modules within the product? How is it accessed? Who accesses it? Is it on the network or off? Internet connected or not? What regions is it going to be deployed in? Where will it typically be located physically? Etc.
Based on the answers to these questions the team conducting the threat modeling exercise uses their various areas of expertise to determine the kinds of means an adversary would use to exploit and make their way into the product and potentially into a customer’s network.
The key outcomes of this exercise is essentially:
A clear list of the known or identified potential threats & vulnerabilities that can exploit a weakness in a system’s design.
A list of risks that are loosely prioritized based on which have the highest potential to actually occur or, if they occurred would have the highest impact, and which risks are known but are unlikely to be mitigated and why.
Based on the above two points, a list of recommended security controls (technical & non-technical) to be implemented in order to reduce or eliminate the identified threats
A record of the model itself, the identified threats, mitigations, and decisions are all kept for future reference and revisiting.
Traditional threat models and the due diligence and documentation around them are becoming increasingly vital to both pre-market submission and post-market monitoring. Lack of diligence, care, or quality of these are also increasingly becoming a cause of rejection.
So don’t get me wrong, there is nothing “bad” about traditional threat modeling; but, when it comes to AI, many of the threats cannot be identified, mitigated, or identified in the same way. We have to shift focus (more or less) to the logic vs. overall potential external threats.
The AI Variation: Attacking the Intelligence
Where traditional threat modeling focuses on things like features and capabilities, access control, points of entry, attack vectors, data interactions and boundaries, and product design - AI Threat Modeling focuses (almost solely) on the logic of the model. That is, it focuses on how manipulation of the data a model’s logic uses to infer and reason against / with affect the core function of the product. Said another way, If an attacker can influence the data, they can change the behavior of the product to do things it was never designed to do.
Some specific AI threat vectors include:
Data Poisoning: An attacker introduces subtle, malicious data into the training set, causing the model to develop a “blind spot.”
Adversarial Evasion: A person can apply a specific digital “mask” to an image that causes a diagnostic AI to misclassify it, while it might look normal to a human.
Inference Attacks: Malicious actors can query a model repeatedly to reverse-engineer the sensitive patient data used in its training.
Why This Difference Matters
Let me paint with the world’s broadest brush for a second (don’t hurt me):
Traditional cybersecurity’s obsession on “general access control” is insufficient to protect patient privacy in the age of AI. I’m not saying it’s not important, I am saying we have to do more when appropriate.
To dive into one of the examples above for a moment to prove this point, let’s talk inference attacks.
Inference attacks could allow sophisticated actors to extract sensitive, identifiable health information from a model’s weights and outputs without ever “hacking” any core system or accessing the underlying data repositories a given AI works with. If that were to happen, that would mean that literally all the compliant hospital databases / datastores associated to or have granted a compromised AI to would be the source of a HIPAA breach through its AI tools. If that sounds insane, it’s because it is, but it’s real and only getting more common.
This is why there is a need to change the approach for threat modeling when it comes to AI products. We have to force ourselves to think beyond the perimeter, beyond physical access and standard feature manipulation or credential compromises and evaluate the privacy risks inherent in the algorithm of the AI model itself.
The integrity of clinical logic is the second major reason this change in approach is non-negotiable.
In a medical device, a compromised algorithm is far more dangerous than a stolen password.
If an adversary can use data poisoning to subtly, and without warning, alter the dosage recommendations of an infusion pump or the alarm thresholds of a fetal monitor, the result is a direct threat to life. If this sounds made up fear mongering, I can assure you it’s not.
During my time in OT security I saw and heard about things in the wild that industry skeptics said were not possible or would never be done. Just because we haven’t seen widespread, publicly reported incidences of sophisticated adversaries doing some crazy shit with medical devices, doesn’t mean they don’t have the capabilities and the wherewithal to do it when it benefits their goals.
To me this is existential, and unfortunately an inevitability; I think about this every single day. I take it seriously, and I think everyone involved in protecting hospitals and patients should too.
Hammering out objectively thorough well thought out, even paranoid levels of threat models is an easy way to do that. Do it early, do it throughout the development lifecycle, do it as new threats arise against products you’ve already put into the world.
Most importantly pick the right damn tools for the right job - if you’ve got AI going into your product, or you’re building an AI as your product, make sure you’re approaching threat modeling with the same rigor and dedication as you normally would, but please change your approach to be hand tailored towards that particular AI.
In summary here, AI is quickly becoming a tool for both defense and offense, medical devices are just as vulnerable an asset as any other - and they’re typically soft targets to cause systemic disruption (as we see already with ransomware).
Organizations that have implemented threat modeling as a core function are the bulwark against the rising tide.
The Practitioner’s Path Forward: Actionable Takeaways
So what do you do with all this information? Well, I think it boils down to some key things this publication will continuously focus on - and a couple of others, but if I were to distill it all down, it’s the following:
1. Quantify the Probability of Failure
Utilize Cyber Risk Quantification (CRQ) to determine the financial and clinical magnitude of a model failure or the impact of a model drifting beyond its performance bounds.
2. Integrate Observability into the QMS
Update your procedures to require continuous monitoring logs as part of the Post-Market Surveillance (PMS) requirement. If a model is not observable, it should not be considered “in control” and if it is not in control, then you should not be comfortable with it going out the door - nor should your customers want to take on that risk.
3. Standardize Model Cards Across the Portfolio
Develop a template for Model Cards that must be completed, maintained, and regularly reviewed for every AI-enabled product. This ensures consistency for the EU AI Act or the FDA’s PCCP evaluations as well as gives your customers confidence, and your internal security teams actionable intel and data.
4. Conduct Dedicated AI Red-Teaming
Commission “Red-Team” (e.g. offensive security experts) exercises specifically focused on adversarial attacks and data poisoning. Testing a model’s resilience to “noisy” or malicious data is as critical as testing a hardware device’s resilience to a power surge. If your go to pentesters don’t have the expertise in this area, invest in them getting it.
5. Prioritize Technical Literacy in Leadership
Governance cannot be delegated entirely to a single, siloed technical team. Executives must understand the difference between deterministic software and probabilistic models, the risks they pose, the business impact the realization of the risks may have on the overall business.
While the external landscape of medical device regulation is arguably some of the best and most comprehensive in the grand scheme of things, it is the MedTech and device manufacturing organizations that own the accountability. We have no excuses to not do the objectively right thing; and if the threats evolve with our innovations we owe the world to not only recognize we’re behind, but to do something about it.
The organizations that thrive will be those that recognize that managing AI is not a one-time approval process, not a screen shot of logs, not an EU AI Act audit being passed - but a new, exciting, and continuous technical discipline.