Key Elements of an AI Governance Program in Healthcare

On August 21, 2024, Sheppard Mullin’s Healthy AI team conducted a CLE webinar on what hospitals, health systems and provider organizations should consider in building an artificial intelligence (“AI”) governance program. As they discussed, key elements of an AI governance program include: (1) an AI governance committee, (2) AI policies and procedures, (3) AI training, and (4) AI auditing and monitoring. These components of an AI governance program will help healthcare organizations embrace the complexities of AI use in healthcare by establishing appropriate guardrails and systematic practices to encourage its safe, ethical, and effective use. This post reviews each of the key elements.

AI Governance Committee

The first key element of an AI governance program is the establishment of a dedicated AI governance committee (the “Committee”). Establishing a Committee is a critical initial step towards AI management and oversight. The Committee should be inclusive of members from various disciplines, such as healthcare providers, AI experts, ethicists, legal advisors, patient representatives, and data scientists to ensure that different perspectives are considered in decision-making.

The Committee’s primary function is to provide (i) oversight and decision-making, (ii) risk management, and (iii) continuous monitoring and review of AI workstreams. Specifically, the Committee will oversee the development, deployment, and use of AI technologies within the organization and ensure that AI systems align with the organization’s ethical standards, regulatory requirements and strategic goals. For purposes of risk management, the Committee will be responsible for identifying potential risks associated with AI technologies and developing strategies to mitigate the risks. The Committee must also regularly review AI systems to ensure they remain effective, safe and compliant with evolving standards and regulations.

While the Committee may take on a number of responsibilities, as is uniquely tailored and necessary for the organization’s business model and goals, its key responsibilities will likely center on: (i) policy development (i.e., drafting and approving AI governance policies and procedures), (ii) ethical reviews of AI projects to ensure the projects align with the organization’s values and goals, (iii) approval of AI projects prior to deployment, and (iv) facilitating communication and collaboration among all stakeholders involved in AI initiatives.

AI Policies and Procedures

The second key element of an AI governance program is adopting a set of AI policies and procedures (“Policies and Procedures”) that provide a structured framework on development, deployment, and use of AI technologies in a consistent and standardized manner. The Policies and Procedures should ensure that all AI related activities comply with legal, regulatory, and ethical standards and establish accountability by clearly outlining roles and responsibilities for those who fall within its scope. The Policies and Procedures should create a formalized process that helps identify and mitigate risks associated with AI technologies, including biases, data privacy issues, and clinical safety concerns.

The foundation on which the Policies and Procedures are formed should be those that align with the common ethical principles that guide the use of AI in health care, such as fairness, transparency, and patient centricity. However, effective Policies and Procedures will also establish practical rules and guidance on data management (e.g., data collection, storage, access, sharing, protection, and privacy) and processes for developing, testing, validating, and deploying AI systems. The Policies and Procedures should also layout the steps required for approving AI projects, including committee reviews, stakeholder consultations, and pilot testing and establish incident management protocols related to AI systems (e.g., reporting mechanisms, response strategies, and corrective actions).

AI Training

The next key element in AI governance is training. AI training is a key risk mitigation strategy that a Committee must implement and oversee. Individuals within the organization should be regularly trained, not only on AI more broadly, but also on the specific programs or platforms that will be implemented before they go live. Where appropriate, AI training should be tailored to an individual’s role and responsibilities within the organization. For example, a physician may require additional training on the use of AI to assist with diagnosing patients or writing notes, while a receptionist might be trained on proper use of AI to schedule patient appointments. Irrespective of their roles or responsibilities, all personnel who will have access to or use AI will require training. In addition to tailoring the trainings to the individual’s role and responsibility, trainings should be tailored based on the AI risk category.[1] Specifically, individuals who leverage AI that is higher risk may benefit from more robust trainings.

AI Auditing and Monitoring

Finally, an effective AI governance model requires ongoing auditing and monitoring by the organization. With AI so publicly available, the Committee and the organization may face challenges determining the type of AI used and how it is being deployed within the organization. As such, it is incumbent upon the Committee to conduct an AI audit that targets and seeks to learn (i) which members of its workforce are utilizing AI, (ii) what type of AI is being accessed and used, (iii) how AI is being used, and (iv) the clinical and non-clinical purposes for its usage. Understanding the full scope and breadth of AI usage, as well as the relevant users, will help position the organization to build out a well-informed and reliable governance framework, and will inform the organization’s AI policies, procedures, and trainings.

Once the AI technology and/or software is identified, it should be properly vetted through a documented approval process, and then be consistently monitored by an organized and cadenced process to ensure proper usage and operation that is consistent with its approved purpose and function. Regular AI monitoring practices will be crucial and should involve documenting the AI’s use and risks based on the organization’s AI inventory, with higher risk AI monitored at greater frequency.

Despite the implementation of consistent AI auditing and monitoring, improper AI practices or incidents may still occur. It is important to have an incident response plan in place, which should address (i) who the incident should be reported to, (ii) how the incident is documented, (iii) whether the AI algorithm related to the incident should be suspended, and (iv) whether regulators need to be notified of the incident.

By establishing a dedicated AI governance program, healthcare organizations can better manage the complexities and risks associated with AI, ensuring these technologies are used safely, ethically, and effectively in patient care.

We will keep our readers posted on best practices for an AI governance program. In the interim, please do not hesitate to contact the Sheppard Mullin Healthy AI team if you have any questions.

FOOTNOTES

[1] While the EU AI Act will generally not apply to U.S. activities, it utilizes a risk category approach which may be a helpful for healthcare organizations to consider in evaluating and categorizing risk of different AI products within their organization.