Council for Healthcare AI Responsibility & Safety (CHAIRS) hosts first live panel: The Adherence Gap
On September 30, 2026, Verbal was lucky enough to co-host the first live panel discussion from the Council for Healthcare AI Responsibility & Safety (CHAIRS), a council of healthcare leaders, technologists, and compliance experts convened by Verbal dedicated to advancing the conversation on AI safety in medicine. This session, a direct follow-up to the organization's first industry whitepaper, The Adherence Gap: 2026 State of Clinical Quality & AI Oversight Report, featured the paper's authors and CHAIRS founding council members Fred L. Brown, Dr. Mark Chassin, and Todd M. Gower. Verbal CEO Waleed Mohsen moderated the discussion.
The group discussed the challenges already rampant in ensuring care quality before AI came into the mix, and how those issues can only get worse at scale.
Check out the transcript and video here. You should come away with a better sense of what goes unverified in human care today, what is already going wrong with AI in healthcare, and some ideas of what to prioritize to ensure adherence as you build an AI deployment plan or audit your governance policy.
Panel
Fred L. Brown, MBA, LFACHE, Founding President and CEO of BJC HealthCare, Former Chairman,The Joint Commission & American Hospital Association
Fred Brown’s career in healthcare spans five decades. The founding President and CEO ofBJC HealthCare, Brown is a member of the Health Care Hall of Fame and has previouslyheld roles as Chair and Chairman of The Joint Commission and American HospitalAssociation. Brown has advised healthcare organizations at the national level on governance,operations, and public policy throughout his career.
Dr. Mark Chassin, MD, FACP, MPP, MPH, President Emeritus, The Joint Commission
Dr. Mark Chassin is one of the world’s leading experts in healthcare quality, patient safety,and process improvement. He served as President and CEO of The Joint Commission for 14years, where he introduced the concepts of high reliability and zero harm to healthcare. Hehas also served as Commissioner of the New York State Department of Health. He served onthe committee behind the landmark IOM reports To Err Is Human and Crossing the QualityChasm and received a John M. Eisenberg Patient Safety and Quality Award in 2021.
Todd M. Gower, Chief Compliance Officer, L.A. Care Health Plan
Todd M. Gower is Chief Compliance Officer at L.A. Care Health Plan, the nation’s largestpublicly operated health plan, serving more than two million members. There he overseescorporate and regulatory compliance, privacy and ethics, and enterprise risk management.He brings more than 23 years of experience in governance and compliance management,including leadership roles at Ernst & Young, Kaiser, and Blue Shield of California.
Transcript
WALEED: Welcome, everybody. This is a conversation that is convened by the Council for Healthcare AI Responsibility and Safety following our August report on The Adherence Gap. The question we are here to sit with is a simple one. Before AI enters the picture at all, how much of what healthcare leaders believe about whether protocols are being followed they can actually evidence? And what happens when AI starts doing the work?
CHAIRS is convened by Verbal AI, where I am CEO, and Verbal contributed research to this report. Nobody on this panel is here to endorse a product, including ours, and nobody is here speaking for their organizations, past or present. These are their professional views.
So I'll start by introducing our panelists, giving them about 90 seconds each for some opening remarks before we dive into some of the themes that we're looking at covering. The prompt that we've asked [our panelists to use] for opening remarks, though they can cover anywhere that they'd like to go, is: What concerns you most about how healthcare oversees care today, and what changes as AI comes into it?
We're going to start with introducing Mark. Dr. Mark Chassin is the President Emeritus of the Joint Commission, where he served as President and CEO for 14 years and introduced the concepts of "high reliability" and "zero harm" to healthcare. He was also Commissioner of the New York State Department of Health, and served on the committee behind To Err Is Human. Mark.
MARK: Thanks, Waleed. I know we'll get into a lot of the specific risks that AI poses as it's being introduced into healthcare, but I'd like to start by noting a different problem. And that is our greatly diminished capacity in the healthcare system for the safe adoption of technology.
We don't have a great track record in general in healthcare, in safely adopting technologies, whether it's computer order entry, electronic health records, automated medication dispensing cabinets, you name it. And the use of AI is really just a kind of next generation automation. Safe adoption of any automation requires that you have perfected the process that you're proposing to automate. And the reason that that's a necessity is that if you automate a broken process, you do dumb things to patients really quickly.
And that's the primary problem that I have with AI coming into healthcare. In addition to [AI's] own particular risks, we have seen a drop-off in the capacity of healthcare to adopt technology safely and for process improvement in general.
WALEED: Indeed, indeed, thank you, Mark. Next, we'll go to Fred. So Fred Brown is the founding president and CEO of BJC Healthcare, former chairman of both the Joint Commission and the American Hospital Association, and a member of the Healthcare Hall of Fame. [He's spent] five decades in healthcare, much of it advising on governance and public policy. Fred.
FRED: Okay, thank you, Waleed. And just to hitchhike on what Mark has said, I think traditionally as we have focused on clinical care and the continuity of care, and as he talked about, we have not had a good success record in adopting technology. Not only that, we have depended upon manual systems to really monitor our care.
And as a result of that, whether it be in the medical piece or the clinical piece or administrative piece, we really have not done a very good job in terms of overall auditing and being assured of true care that we're receiving within each of the organizations, as well as providing the continuity of care. We're still very fragmented and AI will, as Mark has indicated, provide some opportunity. At the same time, we have to be very careful how it's used in terms of future auditing of care.
WALEED: Thank you, Fred. And now we'll go to Todd. So Todd Gower is the chief compliance officer at L.A. Care Health Plan. LA Care is the largest publicly operated health plan in the country, serving more than 2 million members. He oversees regulatory compliance, privacy and ethics and enterprise risk and he came up through Ernst & Young, Kaiser and Blue Shield of California. Todd.
TODD: Yeah, nice to meet everybody this morning. And to what Fred and Mark mentioned earlier around governance, that's obviously in my area of concern, but we know that AI can scale the way we'd like it to, but it can also scale good processes and bad processes, which is where we have our biggest concern. It's not so much around AI policy, because AI is a tool, but the data around it.
And in healthcare, as both Mark and Fred mentioned, we already operate a number of organizations with technology and the boundaries of which they work in is really critical for us on how we operate AI. For us, can we demonstrate that it's being used appropriately, that its limitations are understood and that someone remains accountable for the outcome? Those are key for us in our concerns, looking at AI.
You've seen some of our health plans look at AI for potential determinations, or maybe looking at medical necessity or coding reviews. It's a tool, not a determinating outcome for us. So for us, my biggest concern, as was mentioned by Mark and Fred, is that AI doesn't become a black box inside an already complicated accountability structure.
WALEED: Let's start to dig into these themes. The first theme we'd love to dig into is what goes unverified in human care today. So let's take the scenario before AI enters the picture at all.
For instance, NCQA is closing the manual chart abstraction route for HEDIS. CMS is moving to audit every eligible Medicare Advantage contract annually, rather than about 60 a year. Enforcement is asking for proof of execution, not just policies in a binder. So the question is what any of us can evidence about whether care is being delivered the way that it's supposed to?
And Mark, we'd like to start with you on this topic. You've argued that the ability to find where clinical processes fail, understand how to fix them, has degraded across delivery systems. Could you say some more about what that looks like on the ground today?
MARK: Yeah, I think since the pandemic, what we've seen is really a retreat by healthcare systems, hospitals, and health systems across the country in their commitment to raising the bar on quality and safety. Before the pandemic, there were a significant number of systems that were seriously trying to make quantum changes in the degree of safety and quality of care they provided, really literally aiming for zero harm in emulating industries like commercial aviation, nuclear power, that have achieved incredibly high levels of safety and maintained them over time.
But since the pandemic, survey after survey shows that CEOs don't put safety and quality at the highest level of their strategic priorities. And the way that's reflected in the way their organizations work is their quality, safety, compliance, organizational improvement programs have been diminished, if not even hollowed out.
One of the most distressing sequelae of the pandemic is quality programs across the country lost a lot of the backbone of what they were providing by nurses being drawn away to clinical care and bedside roles. And those programs have not been reconstituted largely. So you start with much diminished capacity for quality improvement.
And that also reflects itself in the basic starting point of any quality improvement effort, which is to recognize what a broken process actually looks like. That takes training because so many frontline caregivers are engaged in workarounds. They're trying their best to deliver excellent care and they're fighting processes that are getting in their way every day, but they're not recognizing that those are signs that improvement is necessary. So diminished ability to recognize broken processes, diminished capacity to fix them are part of our current landscape in healthcare delivery.
WALEED: And to your point in your opening remarks, adding a technology that can scale only compounds these sorts of issues.
MARK: It absolutely does. The lesson from automation in every other industry is perfect the process first, understand it and then automate it. And then crucially, as we started to get into, monitor whether that process is continuing to work as it was designed and automated.
WALEED: Fred, you ran a health system before chairing the Joint Commission. What was your read about what leadership could actually see about standards being followed and what was being taken on faith?
FRED: Well, I think, going again to what Mark said and echoing what your question is, it really gets down to the commitment of the administrator, the CEO and the administrative team and the organization to be totally committed towards safety, towards clinical care and towards results.
And what is happening is I think that we get overwhelmed with financial reporting, with consolidations, with size, with trying to compete with our neighbors. And as Mark indicated so eloquently that we forget that it's all about the patient and how we treat the patient and the results.
And we had so many processes that he indicated got in the way. They were primarily manual. They were primarily dependent upon physician review or nursing review or clinical review or administrative. And again, then we would circle back to the focus on the financial reporting. And size sometimes has gotten in our way in terms of really recognizing how the focus should be on each individual patient and the results and how we can create an organization that eliminates those barriers to the processes and the total commitment to safety and assurance of appropriate clinical care.
And having been a CEO of a major health system, 14 hospitals plus, which has expanded significantly across Missouri today, but also on the joint commission. And I was appalled, as one on the joint commission board, both as a board member and as a chairman, seeing the number of incident reports that were routinely reported on an ongoing basis with each of our institutions. And we have not elevated compliance and safety, as Mark said, to our primary goal of care to the patients.
WALEED: I hear you. And that's the current state of affairs. And so Todd, talking to you, there's an even more, it's a very interesting perspective from a payer who is having to review care delivered by organizations that you don't run. Against documentation you did not produce. What's your take on what is actually verifiable and what's also being relied upon on trust?
TODD: Yeah, I think what Fred and Mark mentioned earlier about the delivery of care is critical and not using technology to look at the outcomes of the care, but helping and giving tools to the clinicians around that.
For us on the health plan side, for our organization, we look at about 60 different hospitals and various other care providing activities, if you will, from pharmacy to where it may be. And then for us, we look at the verification documentation, data consistency, contractual requirements and patterns and compliance for defined processes. As mentioned earlier, making sure the processes are well-defined.
And a lot of it for us in the payer space, we look at from a playbook that's given to us by the CMS on the Medicare side and Medicaid, but the state also gives their determinations and letters to us to make sure we're verifying what we're doing on care for our members at those care centers. But documentation itself is not the thing that needs to be observed.
And for us on the health plan side, we're encouraging our clinicians who are on our health care services side, which is a large number of folks to go out to these facilities and looking at LA, if anyone's on from LA. It's a big space to cover. And so making sure we've got good coverage over that to make sure members are being cared for appropriately.
But for us, healthcare oversight inevitably comes to the degree of sampling and reliance on others. The nice thing about AI though, it can allow us to look at data broadly and also sample a little better, some cases versus using the old [unclear] stats. I think for us, it gives us an opportunity to examine a much larger population and identify patterns humans might miss.
So I know that it's easier to go and do the look and review, but for us in the plan side, it gives us a targeted activity so we can communicate better with our health systems. Where we have seen some improvement in this space, where we do our oversight, we would normally take weeks and months to review a set of data and reports. AI is helping us take a look at targeted activities or aberrancies or anomalies in the information. That gives us a more pointed discussion and also allows us to check the algorithms and the testing that's being used to look at the data to make sure it's actually accurate too.
So we're working with our clinicians and our hospital systems to validate these outcomes to make sure they're accurate. But allows us a little more time so we can spend working with our members, working with our care providers, which the point made earlier, we're seeing less and less and especially in the public health space, difficult. A lot of health systems will not contract with managed Medicaid plans like L.A. Care or with the others that do manage Medicaid. It becomes really, really hard for us on access. And we're having to go through that whole process right now as a big risk for us and making sure there's access for the folks who need the care most.
But AI can create new verifications. What does a model itself miss? And so we're going through that whole process right now on examining this. We're still in early stages with our plan. For us in compliance, it becomes a very instrumental part of how we do things to audit and look at governance. But for us, what does a model itself miss? And so we're making sure we're testing that and working with our AI partners internally, which we have our own AI team now, help out on that.
So we're excited by the opportunity using AI in the health plan space. But moving from human sampling and automation doesn't in itself render reduced risk. It just gives us better activity for us to understand what's going on within the health system. So, I'll pause there.
WALEED: Yeah, let's use that to move to our, I think segue us nicely into our next theme is, AI is already here. It's not that it's arriving, it's already here. And the evidence base in some ways underneath is thinner than the adoption curve.
And Todd, I'd like to stick with you for just a moment here. You mentioned some of the benefits that you've been seeing around aggregate patterns in what's happening. Is there anywhere thus far where you've drawn a hard line where you don't want to have it implemented yet? You want to see some more defensible evidence on its performance?
TODD: Yeah, I think when technology looks at from claims and medical review, like encounter data, it helps us look at completeness, accuracy, and timeliness, which are great things to measure, which regulators are looking at. Are we being timely in our grievance activities or any allegations of concern?
And I want to be careful about using fraud, waste, and abuse, because this conversation is not about that. It's about the patient, making sure we're giving good care. But more importantly, though, administratively, by having the tool in place, it doesn't mean it lowers our risk. It doesn't necessarily mean that. It actually may find new risk for us. It may be high risk areas of concern, especially on patterns of bad behavior, or care, meaning bad behavior, not giving care where it's needed.
And when we look at AI, we look at what decision does it influence, making sure we don't see it as a tool to influence a decision, but help us give information. What kind of autonomy does it have? And making sure we look at how we leverage it inside, not giving it full autonomy. It still needs some guardrails. And what data is it using? So for us, again, policy around data is really critical for us, especially as the ethics and privacy officer for the organization is making sure that we're assessing that from patient health or private information, and what human oversight exists.
So we do have to have humans oversee this process, and it's a good sized team. It's shifting the skill sets as mentioned earlier by Mark and Fred. We look at care providers, or maybe in the care setting, a certain skill set. But now in the health plan space, we're moving from a skill set that looked at repeatable processes, and maybe, not to say mundane, but they're very repeatable, that make sure the skill set's a little different from thinking creatively or curiously about what the problem is now.
So what happens when it's wrong? And that's what we're looking at now. And so we're making sure we've had instances where it's provided us some guidance, and we've had something run in parallel, and to see and test it where it's been wrong. That's been the aha moment, like, okay, cool. It still needs us, it needs to learn. It's not out of the box, great. It still needs to learn from its mistakes. And so that's been our important part as we talk with our AI providers.
The risk isn't determined by whether we call something administrative or clinical. It's determined by the consequence of the decision. So we're making sure we only use it for certain decision types, especially right now. We're only at a very early proof of concept, with governance and controls to help us with that.
WALEED: Mark, let's go to you. So on this theme of that the evidence base thus far has been thinner than the adoption curve, and I'm wondering, and how quickly things are being adopted, what level of evidence should exist before a tool touches patient care?
MARK: Yeah, I think there isn't a single answer to that question. It's, to my way of thinking, it's a graded response. The closer that an AI tool gets to influencing the way people actually receive care, the higher the bar for evidence that proves that it works with minimal or at least acceptable risk.
In that situation, if AI were a new drug, we would require randomized control trial data that shows that it's safe and effective to use. We wouldn't depend on the pharmaceutical company's assurances. Oh yeah, we tested this, it's fine, don't worry about it. We would really require hard evidence and evidence that actually applies to the specific use that the tool is being put to.
In the case of AI, the adoption requirements, I think are even a little bit more stringent because as these tools learn and maybe improve, but at least change, it's like a drug changing the next time you use it. So how do we know that the test that we did on the original version that got licensed or approved is actually the version that we're using right now?
Before we get to that point, we really need to understand the specific uses and make sure that we've at least got evidence that the AI tool is effective and is producing the right results that we want at acceptable risk just as we would any other patient care intervention.
WALEED: Well, the comeback to that, just to argue the other side is that this stops innovation. This will stifle this progress of this incredible technology. What's your reaction to that? And Fred, if you want to weigh in on that too, I'd love to get your take.
MARK: Yeah, that argument has been around as long as there have been new things introduced in healthcare over at least a millennium. And I guess the answer is it's a balance.
Back in the late 50s, early 60s, there was a wonder drug that was being touted all over the world. It was called thalidomide as a cure for morning sickness in pregnancy, as a safe sedative and a sleeping pill that pregnant women could take, that anybody could take. It wasn't addictive. It was completely harmless and totally effective. One individual in the FDA in the United States prevented its licensure in the United States and saved countless infants horrific birth defects when it proved to be a terrible drug to be used in pregnancy and indeed failed all of the subsequent testing.
So we do need very strict guidelines for what evidence we accept when we are putting interventions into the patient care realm that directly influence how patients get care. And if that slows down innovation, it also prevents disaster. So it's always a balance, but the risks of not having any effective demands for good evidence are way, way too high to let that position take the day.
FRED: Now, I would actually just concur with what Mark said. And if you take that same example with FDA approval of any new technology or any pieces of equipment, the same thing has to apply. That's the clearance at that point in time, but there has to be the evidence and it has to be the review and the audit ongoing of what that piece of equipment does create in terms of patient care.
And that's where again, we get sometimes overwhelmed with new technology or new pieces of equipment and now with AI without realizing the consequences or without clearly demonstrating what the results are and auditing those results on a continuous basis and providing the evidence that they are effectively working.
WALEED: Yeah, it's such an incredible time to have this type of technology and the analogy of if AI were a drug. And if you take it on day zero versus three months from now, could be different is a sharp point on this unwieldy new technology that we're dealing with.
I'm curious, and this is to the group for anyone that wants to jump in on this, beyond the evidence, what is actually driving adoption right now in the market? What's your sense as to what is pulling people forward despite these risks?
FRED: Well, I think there's a competitive nature. And again, we talk about a healthcare system, but we do not have a healthcare system. We have fragmented pieces and we have now the major systems that are being created that are competing with each other in major metropolitan areas across the country. And it's all about competition and being number one or being the first to acquire. And sometimes being first to acquire is not necessarily the best route.
And again, without the evidence, again, forgetting that patient care is still one-on-one. And we forget that at times. And we forget about, we talk about continuity, but we do not have continuity. And as we talked about the processes that interfere, there's a good intent. There's a wonderful intent by the caregivers. But again, processes getting away, competition getting away. Trying to be number one in the marketplace instead of recognizing why we're there is to provide the opportunity to provide the best care to each patient.
MARK: Yeah, and I would add just a little bit on top of that. It has been when health systems compete with each other, as Fred said, they do compete to be the best in their marketplace. And that often involves adopting what everybody thinks is the shiniest new toy. Whether it's really effective, whether it's cost effective, or whether it's just what's in the headlines and what's on the local news, it's a fear of missing out and a fear of not having the latest and what seems to be best tool intervention machine to advertise and say, we're at the head of the pack because we've adopted this.
TODD: Yeah, I think there's a big issue around that to your point, Mark, around the FOMO, especially in the plan space. We've got to actually drive healthcare costs out. So we know the cost of care is going up and we don't want to reduce that care for the individual. So how do we get our administrative costs down? And so for us in the health plan space, it's really important for us to look at admin costs and workforce pressures.
So we know that a large number of our health insurers are using AI for some operational purposes. But the areas we're looking at from a healthcare standpoint and helping our members and providers too, that prior authorization and utilization management, we're seeing more and more that come into the discussion and helping our healthcare services teams with our provider groups that we delegate or direct network with, help them out even more to make sure we have better decisions. Also, things are repeatable and fast decision-making for us.
I think the important part too we're seeing is that because of the AI, it's helping assemble the data a little more easier and more usable. The old days, the process of getting coordination of care or getting data around that from various systems, the ETL process extract transfer load process into another system to review it. It was just so cumbersome, you had to hire a lot of people for that. This is reducing that need and it's become more sophisticated and the data is becoming more easily readable and you can exchange it easier.
So on top of that, then you can look at medical costs and identify waste. And so for us, I guess in the plan space, not being myopic and only plans because I work both sides, at the hospital side and both payer side. But on the payer side, we're looking at how we can drive out more and more admin costs and help support our healthcare providers and our members better.
MARK: Yeah, I don't wanna be completely nihilistic about AI on the clinical side either because I think there is a great promise there. As we've said earlier, we're not very good at monitoring critical care processes over time in healthcare. We tend to believe that once we've designed a new safety process, that it will continue to work the way we wanted it to in perpetuity and that of course doesn't happen. In fact, the opposite happens. Processes deteriorate as people move and training diminishes its impact and all the rest of it.
AI properly designed and properly deployed could actually help keep track of good processes and figure out when they're going off track and signal that it's time to put a corrective action in place. But I don't see that particular usage being designed or deployed very well today. But I do think the promise is there if we know how to bring it about.
WALEED: So let's take that point for just a moment. If a tool demonstrably outperforms a well-trained clinician, is there an argument that not using it falls below the standard of care?
MARK: Well, so I think in specifically referring to how AI could either assist or replace clinicians, the main question that needs to be answered before that could even be contemplated in a scalable way is who gets sued if the machine is wrong and a patient is harmed. We know who gets sued now when institutions and individuals make mistakes and patients are harmed. And whether or not that's a good thing or a bad thing is a whole nother subject, but there's clear liability now. And it's not at all clear what happens if an AI tool has some responsibility for clinical decision-making.
I think like anything else, if a particular new intervention is superior to an older intervention, that it should supplant it. But those were all decisions that are being made by individuals in support of clinicians making decisions and conducting those interventions. If we're imagining an autonomous machine that doesn't depend on individuals, that's a completely different ball game in the superstructure of how that works, liability, who's responsible, what are the appeal processes? All of that needs to be worked out well in advance of deployment.
WALEED: Yeah, on that point, at least we saw in California that a bill was recently passed that said a defendant could not point to an autonomous system and saying, hey, it made that decision. It wasn't us, the vendor or the developer of the system. So we're already starting to see those signals and it's very interesting what's going on with the Hugging Face hack about this agentic autonomous system that did this, who's ultimately responsible. So it's very, very interesting on where all of these things are going.
WALEED: Let's go to our last theme about what do we do about this? So if we're looking to, how can we make this useful? If I'm on the committee deciding whether to bring an AI tool into patient care, what do I actually do?
And so Fred, I'd like to start with you on this thread. You've chaired boards, you've run a system, you've seen both sides of this. What should a board be asking before an AI tool goes live and what should they be asking a year later?
FRED: Well, I think we go back to the report itself, Waleed, and that's plausible deniability. And our boards of governance within the healthcare systems traditionally have been weakened by the selection process, not based upon the expertise of those individuals being brought on that board for particular reasons of a healthcare board. And so this whole concept of plausible deniability and not bringing up certain things until it results either in legal action or results in other types of actions by the organization.
And I think that the board of governance, board of governors, board of trustees of healthcare systems and providers, organizations, really have to be strengthened and really have to ask for evidence-based from the administrative staff.
What we fail to recognize is that the board of governance, the board of trustees have both the fiduciary responsibility and the legal responsibility for those organizations. They've got to take a more aggressive approach in asking for evidence-based or any AI tool or any activity that's taking place in that organization because ultimately they are responsible. And it comes down to, you know, if they do not take an action or become active in legal actions and other implications.
So I think that we have to strengthen that and we still have to look at AI as a tool, not the answer, but as a tool to assist. And again, what's the evidence to be able to use that AI tool and going forward? And then what are the audited results?
WALEED: All right, and Mark, let's go to you next in helping to provide some of those results and that evidence, you know, Fred's referring to. Who should be doing the checking? Today it's primarily been the vendors reporting on its performance. What's your read on what that should look like?
MARK: Yeah, I think like almost anything else that a health system, hospital health system, would deploy in delivering patient care, you can't depend on the vendor who is selling it to you for evidence that it's safe and effective. You, and as I said before, this is a graded system or situation in which evidence is graded.
So I would say since the hospital health system is going to be responsible for the impact of any AI tool as it's deployed into healthcare, and the closer it gets to actually influencing what happens to patients, the more you would want to have independent evidence, that is, evidence independent of the vendor that actually provides you assurance that the tool is going to function and that it's going to be effective. So you would want to have an assurance that the tool is going to function safely and effectively.
As I said before, you wouldn't rely on the pharmaceutical company for assurances and promises that the drug is safe and effective. That would not fly in any developed country in the world. So the closer an instrument gets to affecting the way patients get care, the more you will want independent evidence that is safe and effective.
WALEED: And Todd, in evaluating these technologies as an executive board, what we're hearing from many is that vendors are often indemnifying usage of these technologies if something goes wrong. What's your read on what that actually covers and what it doesn't?
TODD: Yeah, I think indemnification doesn't mean that the risk is covered. It just doesn't. I think what remains is what kind of contract did we sign? And for us, we look at the ownership, evidence, and consequences. Consequences are important for us, making sure that we understand what those are. And someone has to have the decision around that. And depending on the size of the contract and what's impacting for the most critical ones, of course, they get reviewed up to a board level.
We have to make sure what's being covered and actually will be reviewed. What's the governance around it? Policy is one thing, but actually looking at what the consequences afterwards. And I think policy isn't governance either. So governance happens when the policy meets a real decision.
In this case, in helping boards and being on boards myself and presenting especially the compliance and quality or compliance audit committee to our board of directors, it becomes an important thing because most of our board directors are clinicians. And they deliver care to our members and helping them understand what this risk means. Aside from do no harm, we're bringing on vendors to support the members that go to them and making sure we have the right vendors to support that.
So we've changed our whole process from looking at vendors and making sure the vendors understand before they're signing anything what we're doing and what they're going to be using. They're going to be using AI, how they're going to be using it. We've already started the process of letting all of our vendors know if they're using any AI at all to help us understand that what they're using it for. Secondly, looking at all the applications we have signed and all the vendors we've signed who has AI capabilities.
And so we've got a whole process in place now for the next six to 12 months to take a look at this. And we've got a whole committee internally for this, a whole committee aside from my own risk committee. We have a vendor risk management committee and we have a whole AI committee as well. And it's not just me on it. It's a bunch of folks from CFO, the IT, and we have a new position there related to this transformation activity within health plans to support us.
And it's helping us understand better about what it is. Because a lot of folks, as Mark and Fred were making comments earlier, the FOMO piece, folks wanted to get a company in to help them look at stuff. And it would just sit there not being utilized. So user adoption is a big key for us. But if it's left alone on its own, what's it doing? Who's watching it? So we're making sure there's good risk management from risk oversight, vendor risk management, overall up to reviewing, making sure those key vendors are being looked at and validated against their usage of AI and how we're using their AI to help us.
So I think my last component on this is that innovation, we're not stopping innovation, I guess. That's what I'm looking at. We're making sure we know how to deploy it correctly and using AI to help us with that. We just want to make sure who's accountable and what the evidence is to support the outcome, is good or bad, and making sure we limit its use until we really understand the evidence it's providing us to make the right determination to do what we need to do with that information.
So I think for us it's making sure we have the right information, right governance process in place, and the consequences for it running rampant. If someone does allow it to happen, to the point made earlier, it's not the tool's fault. The person who put the tool in did it. So we're just making sure accountability is well defined and forcing us to be understanding what our CEO calls it and who's got the belly button. And I think that's where we're really making sure speed to market, speed to delivery, that's great. However, if something goes awry, who's accountable and what are the consequences for that?
WALEED: There are a couple of questions that have come in. I'll post to the group here. So this one is directed to Fred and Todd. What should a community hospital or a smaller plan do without the resources of a large system or an L.A. Care?
FRED: I'll let Todd kick that off.
TODD: Great. Well, I think depending on the health system itself, if it's a plan or it's a health system, typically the contracts we have with those health systems, working collaboratively together is where it starts. I know with our organization, we have what's called delegation oversight or provider network management, and we work closely with our providers to help them with that. And we've seen that work really well and looking at technology and helping each other out.
So I'd first work with whomever the community health system is and the leadership around that, speak with whoever is running their IT shop or the chief compliance officer. Quite frankly, you could reach out to them. Sometimes they're a really good place to start to, versus talking to the CEO. But provider network management should be a good place to start talking to them if you want to work together with the health plan for those community health systems.
FRED: Yeah, and to capitalize on what Todd has said, I always maintain that there needed to be a balance between the insurers, the providers and the physician community for health care to really function. And I think if you're a smaller community hospital and don't necessarily have the resources, again, as Todd has said, you can reach out to the insurers, you can reach out to the provider. But also you can reach out to your associations and whether it's on a local basis or on a statewide basis or a national basis to assist in that process, to be able to provide additional resources to you to be able to manage. Because again, it's going to be very, very critical going forward to have the ability to create the evidence-based approach to health care.
TODD: Let me touch on one more thing that has been mentioned. Reaching out on the local health plans and health systems, they all have their associations to work with, so we try to help each other out as much as we can.
WALEED: And Mark, there's one for you here that's asking, is voluntary certification for AI likely to become table stakes and on what timeline?
MARK: That's a really good question. If I had a crystal ball that actually worked, I might be able to answer that. I doubt, frankly, given all of the variations in which AI is presenting itself to health care, I doubt that voluntary certification or any external process that is created today could cover enough of those to be reliable.
And because we don't have any agreement on what safe adoption of AI tools actually looks like, it's pretty hard for a volunteer organization to create that framework all by itself. My view will need to have a number of critical stakeholders come together to put a framework together that everyone can adopt. I don't think any single organization is going to be able to do that and cover the AI waterfront as it's expanding so rapidly today. So it's still Wild West days, at least from my perspective, and I don't see that being solved by any single organization anytime soon.
WALEED: And there's another question that came in. It wasn't directed to anybody specifically, so we'll open it up to the group. What do you do about AI tools that were already deployed that weren't evaluated in this way, say, pre-having a formal governance process?
TODD: Well, I'll say this for us. We saw some of that, and we got our arms around it quickly. I think it's not saying they did something wrong. We want to assume good intent, but more importantly, it's making sure there's a risk committee or vendor risk management committee or vendor risk review of the company, be it the BAA, the business associate agreement, or look at the agreement itself to make sure you understand the underlying components that was being asked to implement and then testing and auditing it. Maybe get a third-party opinion, depending on your size, either a health system or plan, but you definitely need to make sure you've got a good set of controls, like who's monitoring and who owns it, and how's it being checked and validated, and where's it being deployed.
MARK: Yeah, I would add that there's a pretty well-worn quality improvement process called FMEA, failure modes and effects analysis, that you can deploy to understand what the risks are and how the process that this tool is overseeing or is operating could fail in ways that you haven't anticipated before, and I would get on that right away, because the risks may not be apparent until patients are harmed.
WALEED: Absolutely.
Okay, so we're just about at the point of wrapping up here. What we want to do as we close is give a last word to each of you, closing remarks roughly in 90 seconds, along this line of if you were advising someone on the committee deciding whether to bring an AI tool into patient care, what is the one thing you would tell them to do in the next 12 months? And why that one? Todd, let's start with you.
TODD: Well, for the next 12 months, I think for us it's like understanding your organization and what it needs to get done, understand what processes that can be done. I think what Mark was mentioning earlier, making sure the processes are done well, and I think that's where if you don't have it done well, properly document it, because that's all it's going to learn. It can't just run willy-nilly, so making sure you get a really good understanding of what you want to accommodate and also accomplish for the organization using the tool, because otherwise you can't let it run amok. It won't be optimized, nor will you get the full benefit from it.
WALEED: Fred?
FRED: Well, along with what Todd had said is that you also have in place the ability to monitor and create evidence base that that tool is effectively doing what's intended to do.
WALEED: Right on. And Mark?
MARK: I would say certainly in the short run, in the next 12 months, one thing to avoid is adopting an AI tool just to say you've adopted an AI tool. Make sure that you're addressing a real need in the organization, and then also at the same time, make sure that you have a handle on the process that you're trying to automate.
It brings to mind an old adage that I have seen very effectively prevent a lot of disaster, and that is the recognition that computers don't make us less stupid. They make us stupid faster. So if you're going to automate a broken process, anticipate disaster, and don't do it.
WALEED: And on that note, thank you. Thank you all, Mark, Todd, Fred, for this time and going through this ever-changing landscape, this very exciting time, but also one that needs to be thought through very, very carefully. I appreciate everyone's comments on this and the questions that came in from our audience. This video will be made available shortly after our session today following the registration link on Zoom and LinkedIn, and we'll make sure everybody has a chance to refer back to this. So thank you, everybody. We'll wrap it there.
