Harmonizing Artificial Intelligence Governance; A Model for Regulating a High-risk Categories and Applications in Clinical Pathology: The Evidence and some Concerns

The Canadian healthcare system, grappling with issues like systemic and intelligently established structural anti-black racism, including indigenous nations; even within Pathology and Laboratory Medicine Communities: and deteriorating outcomes, sees potential in AI to address challenges, though concerns exist regarding exacerbating discriminatory practices. In clinical pathology, AI demonstrates superior diagnostic accuracy compared to pathologists in a study, emphasizing its potential to improve healthcare. However, AI governance is crucial to navigating ethical, legal, and societal concerns. The Royal College of Physicians of Canada acknowledges the transformative impact of AI in healthcare but stresses the need for responsible AI tools co-developed by diverse teams. Despite positive attitudes towards AI in healthcare, concerns about patient safety, privacy, and autonomy highlight the necessity for comprehensive education, engagement, and collaboration. Legal concerns, including liability and regulation, pose challenges, emphasizing the need for a robust regulatory framework. AI application in healthcare is categorized as high-risk, demanding stringent regulation to ensure safety, efficacy, and fairness. A parallel is drawn to drug regulation processes, suggesting a similar approach for AI. The lack of transparency in AI-based decision-making raises ethical questions, necessitating measures to address biases and ensure patient privacy. Social accountability is crucial to prevent AI from exacerbating health disparities and harming marginalized communities. In conclusion, while AI offers potential benefits in clinical pathology, a cautious approach with comprehensive governance measures is essential to mitigate risks and ensure ethical AI integration into healthcare.

The dicey landscape of the Canadian healthcare system

A recent report indicates that systemic racism in Canada’s healthcare system is continuing to fuel inequities and healthcare disparities among Indigenous, Black, and women of colour [1]. The author highlighted how macro policy and meso organizational structures contribute to advancing these inequities among racialized individuals and communities, indicating that the systemic structures, rooted in colonial practices and racial oppression, include Federal and Provincial/Territorial policies that set the stage for educational systems (i.e., admission processes, training, and licensing of health care professionals) and healthcare systems to perpetuate racism experienced by Indigenous, Black, and people of color at point of care [1]. The authors reported that by design, these macro-level structures facilitate opportunities for the dominant group, thereby reinforcing white privilege throughout meso-level institutions and organizations [1].

In addition, Abondi, et al. showed that African, Caribbean, and Black Canadians faced persistent legislative barriers to accessing healthcare services in Canada [2]. The author revealed that exclusionary healthcare policies restricted their access to public resources, including health insurance and HIV healthcare and related services, subjecting them to precarious status. Clearly, it was shown that Healthcare providers and administrative staff worked as healthcare gatekeepers and that these barriers undermined public health efforts to advance health equity and urged continued policy reforms in Canada’s healthcare systems [2].

Looking at the Canadian academic pathology and Laboratory Medicine institutions, there is abundant evidence of discrimination and underrepresentation of Black people and Indigenous communities in medical schools and training fellowship programs across Canada in several specialties including but not limited to Medical Genetics, Clinical Biochemistry, Clinical Microbiology, Pathology, and Dermatology specialties. Data on the impact of this discrimination on their medical education. Johanne, et al. showed that 59% of the Black students had at least one personal encounter with discrimination in medical school [3]. Discrimination was experienced in both clinical and academic contexts, notably from patients, peers, and hospital staff. most respondents had negative experiences relating to reporting discrimination, their well-being, and career advancement [3]. In nursing and Medical Laboratory Science, there are cultural and historical records indicating a legacy of widespread discrimination that continues to impact Black health professionals at work and during training [4].

In fact, a recent systematic review of 26 independent studies from Canada and the United States that sought to understand how black people cope with the effect of racism demonstrates that Black people tend to cope with racism through social support (friends, family, support groups), religion (prayer, church, spirituality), avoidance (attempting to avoid stressors), and problem-focused coping (confronting the situation directly) [5]. Findings suggest gender differences in coping strategies [5].

With increasing expenditure and job allocations but a deteriorating outcome, and resistance to change some unethical practices such as racism, discrimination, nepotism, and ethnocentrisms, including accessibility and high infant mortality, the Canadian health system requires special attention that could be made better or worse with new technology and artificial intelligence, and its regulation. Thus, leaders in healthcare have proposed that AI could have a role in addressing all or some of these problems, however, there is increasing concern about how this technology might aggravate the discriminatory practices and culture that is rampant in the health system.

Experimental evidence for the use of AI in clinical pathology

In clinical Pathology, biological specimens are deposited on a glass slide, followed by some chemical reactions or processing, and analysis for the purpose of detecting, confirming a diagnosis, or evaluating treatment progression. The current challenges are marked heterogeneity and inconsistency among pathologists in reporting and inter-pretations of slides.

Recently, a study that sought whether the application of AI to whole-slide pathology images can potentially improve diagnostic accuracy and efficiency has been published [1]. The study involved training a data set of whole-slide images from 2 centers with (n = 110) and without (n = 160) nodal metastases provided to challenge participants to build algorithms [6]. Algorithm performance was evaluated in an independent test set of 129 whole-slide images (49 with and 80 without metastases). The same test set of corresponding glass slides was also evaluated by a panel of 11 pathologists with Time Constraint (WTC) to ascertain the likelihood of nodal metastases for each slide in a flexible 2-hour session, simulating routine pathology workflow, and by 1 pathologist Without Time Constraint (WOTC) [6]. The author reported that the AI-based whole-slide analyses performed significantly better than the pathologist’s WTC in a diagnostic simulation and that the top 5 AI algorithms had a mean AUC that was comparable with the pathologist interpreting the slides in the absence of time constraints [6].

The study concluded that an AI-based whole-slide clinical pathology algorithm achieved better diagnostic performance than a panel of 11 pathologists participating in a simulation exercise designed to mimic routine pathology workflow; and that the AI algorithm performance was comparable with an expert pathologist interpreting whole-slide images without time constraints [6,7].

Another report in Nature Science Journal, Topol, 2019 strongly indicated that the generation of data in massive quantities from sources such as high-resolution medical imaging, biosensors with continuous output of physiologic metrics, genome sequencing, and electronic medical records – means we have clearly exceeded the limits of analyses that humans can do alone in healthcare [8]. This suggests we have attained our limit, in a background of numerous problems outlined above. Again this calls on the use of technology advancements to solve the ever-expanding challenges. To do this however there is a need for data governance, and establishing a formidable regulatory framework to balance harms or even mitigate them [9].

On this note, the Royal College of Physicians of Canada recently commissioned a task force to help the medical profession in Canada prepare for the profound changes that artificial intelligence and emerging digital technologies will bring to residency training and delivery of care in Canada [9]. They were mandated to conduct extensive research into the current and future states of these technologies and to provide recommendations to the Council about how to meet the challenges and opportunities these technologies present to the Royal College. The key finding of the Task Force was that AI and emerging digital technologies will become more integrated into the practice of specialty medicine, supporting the daily routines of a healthcare team, and could liberate physicians from repetitive tasks, allowing time for more patient care, including compassionate care, and improving the safety and quality of patient care [9]. The team also recommended that Responsible AI tools for medicine should be co-developed by teams that include members of the health care and specialists involved in the development of AI systems, such as computer scientists, engineers, mathematicians, and professionals working in other technology-oriented disciplines [9].

Whether the application of artificial intelligence in this setting would have direct clinical utility will require further evaluations in a clinical setting to mitigate potential harms and a regulatory framework to ensure that other arms of both common law and criminal law are well articulated.

Regulating AI for clinical pathology application

As indicated above, there are reasons to believe that the Royal College of Physicians of Canada, is open to the use of AI technology to advance higher precisions, accuracy, cost saving, and more. However, it is clear, there is no framework currently in place to appropriately regulate the process to protect vulnerable individuals.

AI Applications in healthcare falls under high-risk category

Major concerns and risks associated with the use of AI have resulted in AI applications in healthcare being classified as high risk. In line with the European AI regulatory framework, AI system that has the potential to create an adverse impact on people’s safety or their fundamental human rights are classified as “High-Risk” that would require clear and strong regulation [10]. Under this classification, AI systems that violate fundamental rights such as those used by the Government or others for social scoring, exploitation of vulnerabilities of children, etc. are regarded as unacceptable risk class, and such are completely prohibited.

Implementing a responsible AI in clinical medicine and pathology

A qualitative study in the USA (conducted in 15 focus group discussions with 87 participants) reported that the participants, in general, have a welcoming and enthusiastic attitude towards AI in healthcare, as it has the potential to improve the care they receive at medical facilities, but they felt their patient safety, privacy, and autonomy was at stake with the inclusion of artificial intelligence in healthcare [11]. Because of these concerns, it is important to have the required education, engagements, and collaborations as well as attain an acceptable level of AI maturity, before considering the adoption of the technology.

To do these, several factors are required to integrate the AI system into our healthcare in Canada, given that it is considered a high risk, therefore steps must be taken to ensure that those associated risks are mitigated to the barest minimum. First, if the Royal College of Physicians of Canada, and other health facilities are considering integrating the benefits and applications of AI to advance healthcare delivery, there is a need to establish, educate, and collaborate with other stakeholders as indicated above. Such that the AI governance would exist internally, as an integral part of the College or Hospital. Within the membership of the committee, would be lawyers with specialties in healthcare, Data scientists, bioinformaticians, doctors of various levels, nurses, and pathologists. Within this committee would be other sub-committees, focusing on ensuring that different areas of laws protecting both vulnerable populations as well as fundamental human rights are obeyed. Such a committee will also oversee that the data and other principles by which the technology operates are as pure as possible and devoid of bias. Both the internal and the external regulators would be required to adopt a stringent policy and procedure with well-documented and signed indications that acceptable standards and safety have been met. Practically, they are required to meet the ethical principles of fairness, justice, prevention of harm, and autonomy.

Ethical concerns ranging from data security to data privacy via the misuse of personal data have led to straineddoctor–patient relationships. Achieving unimpeachable control over the risks associated with the use of AI plays a pivotal role. Concerns about using the obtained data, along with data protection and privacy, are important issues that must be addressed for a successful artificial intelligence-driven healthcare administration. The optimum potential of AI in medical care cannot be achieved without addressing these ethical and legal conundrums.

AI-based decision-making lacks transparency

This feature of artificial intelligence called the “black box” element, renders the AI decision-making process opaque. Artificial intelligence should aid in achieving the well-being and safety of patients. Nevertheless, artificial intelligence has been proven to reinforce the existing biases in the healthcare industry. Unreliable and under-representative data sets for AI development led to inequity, data bias, discrimination, and deceptive predictions. Health data are the most sensitive and intimate information. Respecting the privacy of the patient is a critical ethical principle, as privacy is built upon the grounds of autonomy and bound to individual identity and well-being. 

Legal concerns in health-related artificial intelligence

One of the primary legal concerns is liability. AI systems make decisions that affect patients’ lives, and healthcare providers are responsible for those decisions. In the event of a medical error caused by an AI system, determining liability can be challenging. It is unclear whether the healthcare provider or the AI system is responsible for the error. This issue highlights the need for clear guidelines on how liability should be allocated in cases involving AI systems.

Furthermore, there is a concern about the regulation of AI systems in healthcare. Currently, there are no clear regulations in place to govern the use of AI in healthcare delivery. This lack of regulation could result in healthcare providers using AI systems that have not been adequately tested or validated. Therefore, regulatory bodies must develop guidelines to ensure that AI systems used in healthcare are safe, effective, and reliable.

To address these legal and ethical concerns, healthcare organizations must take a comprehensive approach to the development and implementation of AI systems. This includes conducting thorough risk assessments, designing AI algorithms with privacy and security in mind, and establishing clear policies and procedures around liability and regulatory compliance. It also requires a commitment to ongoing monitoring and evaluation of AI systems to ensure that they continue to operate ethically and responsibly.

AI regulation based on clinical trial principles

Regulation of high-risk classification, and application to healthcare - Adopting regulatory perspective of approval of investigational new drug and clinical trial process in Canada: It is well known that drugs are very toxic and can cause even other unknown diseases. However, there are usually good intentions in administering drugs to persons by healthcare professionals to save lives. The absolute reality is that drugs cause both harm and good, as such a regulation exists and is strongly enforced to have a clear control that accounts to mitigate the negative consequences and gain the benefits.

Through the process of clinical trials, which is under the tight regulation by Health Canada, a proposed new investigational drug must receive approval from the regulator, that certain safety standards have been met, that can trigger clinical trial phase 1. To meet this requirement, a sponsor must exist, who must have inhouse clinical trial governance. This body oversees the stages of pre-clinical studies. They are made up team from multidiscipline with background on medical ethics, legal experts with focus on tort law, privacy law, humanities, and toxicologist, and administrators. Their primary aims are to see that prospective participants in the clinical and preclinical trial studies would have their fundamental human rights respected, in line with relevant charters, and must consent with reasonable understanding of the risks and harms they may face in participating. The sponsor would have to file for approval with reasonable and responsible data that indicate the harms and safety and measures in place that mitigate such harms. Only when this is met would health Canada, give approval for the sponsor to move further to Phase 1 trial.

Phase 1 trials are in two stages, Phase1a and phase 1b. these stages evaluates the safety and raise concerns if available or proof that the drug is save in humans. Phase 2 is triggered once the sponsor show that the drug is save and efficacious in the intended population, that is the target population. Following completion of phase 3, if successful, the sponsor can file for approval of a new drug from the regulator, or Health Canada. This stage is further followed by the 4^th phase, also known as the post-market phase. It keeps monitoring the adverse effects associated with the drug since approval. If adverse effects are heavy and concerning, the new drugs would be withdrawn.

From this explanation, it is clear that sufficient effort is in place to ensure that harms are well mitigated, while utilizing the advancement and benefits of a new treatment. Equating a new investigational drug to application of AI in Healthcare – a high-risk category, and adopting similar measures, the benefits can be harnessed in similar manner.

Guiding AI governance in healthcare

Social accountability: Reports from the Task Force on adoption of AI in health care asserted that the need for this technology in health is based on a prediction that it would improve the efficiency and effectiveness of health care, lead to improved patient outcomes. But the challenges here is whether use of AI in Healthcare would widen the current health disparity and serve as a tool to further harm members of marginalized communities, and used to further deny them of access to certain health cares that is available to others. We have seen how some mathematical algorithms have been used for years to deny the black communities access to renal transplant, whereas the white counterparts with less severe illness, had received urgent and immediate intervention. In these cases, the same perpetrators used media and certain medical journals to propose that black people are most likely to have aggravated renal failures, that do not warrant kidney transplantation. While this practice was well adopted for many decades, it was only in 2022 that some health facility reverted, to give black people equal right to chronic kidney disease intervention by throwing away the obnoxious rule in Clinical Pathology.

Again, given that current health data, does not always include data from black and indigenous populations. This is often because these population are aware of the cold war that operate against them with the public healthcare system, and most of them avoid going to the hospital and choose alternative approach, for lack of trust. Thus, if the health system train their AI algorithm with the current data, it will certainly result in bias and harm to these marginalized communities and cause even more harm, therefore there are risks that integration of AI into health care may pose a risk to the safety and quality of care of the historically under-represented populations, or that these populations will be excluded from the anticipated benefits of AI, or both.

While AI application in Clinical Pathology, and Laboratory Medicine appear to be powerful, and can promote accuracy and cost saving, and can pick-up patterns that human cannot, it seems that its use may better appreciated if appropriate measures have been taken to ensure that there is mitigation protocol in place that is robust enough.

The widespread adoption of artificial intelligence (AI) in healthcare, underscores its potential benefits and attendant challenges. The implementation of AI introduces complex legal considerations, particularly regarding the liability of healthcare professionals and technology manufacturers when AI-generated recommendations lack transparency. Facial recognition technology (FRT) within AI applications, used for patient identification and monitoring, and job interview process raises ethical concerns due to its potential to perpetuate discriminatory practices, particularly against marginalized black and indigenous communities. In Canadian healthcare, reports highlight systemic racism, prompting legal actions against discriminatory practices, revealing a pressing need for equity in healthcare systems. For example, In Quebec, a patient was able to film herself before she died in government funded hospital due to fata racism activities from healthcare providers, revealing the endemic and fatality of both anti-black and indigenous practices in the healthcare system that has been ignored by policy makers. There are reports of similar practices worse in Saskatchewan, Alberta and especially in British Colombia, and the entire western Canada, including a long list of healthcare associations. Recently, 10 black doctors took the Provincial Hospital to court over clear discriminations they have suffered in Regina Saskatchewan Canada. This was possible because, they were able to afford the legal cost. Many other victims from these communities abound who cannot speak out for fear, because in many instances, the same perpetrators work to cover one another.

Furthermore, the deployment of “black-box” algorithms in AI introduces challenges related to medical malpractice and product liability, as the opacity of these algorithms impedes users’ ability to provide a coherent explanation for their outputs. Privacy and data protection issues surface with the use of FRT in health applications, demanding careful consideration and robust safeguards. The multifaceted landscape of AI in healthcare necessitates comprehensive governance frameworks to address legal, ethical, and discriminatory implications. Establishing effective AI governance is crucial for maximizing benefits, mitigating risks, and ensuring the responsible and equitable deployment of AI technologies in the healthcare domain. This highlights the imperative for ongoing research, policymaking, and regulatory measures to guide the ethical use of AI in healthcare, promoting transparency, accountability, and the protection of patient rights.

1. Coen-Sanchez K, Idriss-Wheeler D, Bancroft X, El-Mowafi IM, Yalahow A, Etowa J, Yaya S. Reproductive justice in patient care: tackling systemic racism and health inequities in sexual and reproductive health and rights in Canada. Reprod Health. 2022 Feb 16;19(1):44. doi: 10.1186/s12978-022-01328-7. PMID: 35172842; PMCID: PMC8849010.
2. Odhiambo AJ, Forman L, Nelson LE, O'Campo P, Grace D. Legislatively Excluded, Medically Uninsured and Structurally Violated: The Social Organization of HIV Healthcare for African, Caribbean and Black Immigrants with Precarious Immigration Status in Toronto, Canada. Qual Health Res. 2022 Apr;32(5):847-865. doi: 10.1177/10497323221082958. Epub 2022 Apr 5. PMID: 35380883; PMCID: PMC9152595.
3. Mathieu J, Fotsing S, Akinbobola K, Shipeolu L, Crosse K, Thomas K, Denis-LeBlanc M, Gueye A, Bekolo G. The quest for greater equity: a national cross-sectional study of the experiences of Black Canadian medical students. CMAJ Open. 2022 Oct 23;10(4):E937-E944. doi: 10.9778/cmajo.20220192. PMID: 36280249; PMCID: PMC9640166.
4. Jefferies K, States C, MacLennan V, Helwig M, Gahagan J, Bernard WT, Macdonald M, Murphy GT, Martin-Misener R. Black nurses in the nursing profession in Canada: a scoping review. Int J Equity Health. 2022 Jul 23;21(1):102. doi: 10.1186/s12939-022-01673-w. PMID: 35871077; PMCID: PMC9308111.
5. Jacob G, Faber SC, Faber N, Bartlett A, Ouimet AJ, Williams MT. A Systematic Review of Black People Coping With Racism: Approaches, Analysis, and Empowerment. Perspect Psychol Sci. 2023 Mar;18(2):392-415. doi: 10.1177/17456916221100509. Epub 2022 Aug 25. PMID: 36006823; PMCID: PMC10018067.
6. Ehteshami Bejnordi B, Veta M, Johannes van Diest P, van Ginneken B, Karssemeijer N, Litjens G, van der Laak JAWM; the CAMELYON16 Consortium; Hermsen M, Manson QF, Balkenhol M, Geessink O, Stathonikos N, van Dijk MC, Bult P, Beca F, Beck AH, Wang D, Khosla A, Gargeya R, Irshad H, Zhong A, Dou Q, Li Q, Chen H, Lin HJ, Heng PA, Haß C, Bruni E, Wong Q, Halici U, Öner MÜ, Cetin-Atalay R, Berseth M, Khvatkov V, Vylegzhanin A, Kraus O, Shaban M, Rajpoot N, Awan R, Sirinukunwattana K, Qaiser T, Tsang YW, Tellez D, Annuscheit J, Hufnagl P, Valkonen M, Kartasalo K, Latonen L, Ruusuvuori P, Liimatainen K, Albarqouni S, Mungal B, George A, Demirci S, Navab N, Watanabe S, Seno S, Takenaka Y, Matsuda H, Ahmady Phoulady H, Kovalev V, Kalinovsky A, Liauchuk V, Bueno G, Fernandez-Carrobles MM, Serrano I, Deniz O, Racoceanu D, Venâncio R. Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer. JAMA. 2017 Dec 12;318(22):2199-2210. doi: 10.1001/jama.2017.14585. PMID: 29234806; PMCID: PMC5820737.
7. Golden JA. Deep Learning Algorithms for Detection of Lymph Node Metastases From Breast Cancer: Helping Artificial Intelligence Be Seen. JAMA. 2017 Dec 12;318(22):2184-2186. doi: 10.1001/jama.2017.14580. PMID: 29234791.
8. Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med. 2019 Jan;25(1):44-56. doi: 10.1038/s41591-018-0300-7. Epub 2019 Jan 7. PMID: 30617339. 
9. Royal College of Physicians and Surgeons of Canada. February 2020. Task Force Report on Artificial Intelligence and Emerging Digital TechnologiesLinks to an external site.
10. Regulatory framework proposal on artificial intelligence | Shaping Europe’s digital future (europa.eu) accessed 27-03-2023, and lecture slides INQ.
11. Prakash S, Balaji JN, Joshi A, Surapaneni KM. Ethical Conundrums in the Application of Artificial Intelligence (AI) in Healthcare-A Scoping Review of Reviews. J Pers Med. 2022 Nov 16;12(11):1914. doi: 10.3390/jpm12111914. PMID: 36422090; PMCID: PMC9698424.