Regulate AI in Healthcare

AI tools are spreading without oversight or accountability

AI is being tested and used in healthcare for everything from admin tasks like scheduling, communication, and patient and resource management, to more core medical and care tasks such as research, diagnosis, and treatment.

The use of AI in healthcare, especially for core medical tasks, poses a lot of risks. AI tools reinforce the discriminatory biases baked into the healthcare system. These technologies expand the opportunity for medical surveillance, which is being used to target vulnerable groups including abortion seekers and immigrants. And these tools can track and expose medical conditions and procedures that are under political attack and increasingly criminalized such as pregnancy, abortion, or gender-affirming care, putting patients in danger.

Healthcare professionals have extensive training to prepare them for caring for patients. It is misguided to think that these skills and the nuance of human care can be replaced by AI. It is hubris to think that the work of dedicated healthcare professionals can be replaced by AI.

AI systems are underdeveloped and prone to errors—they shouldn’t decide who does or doesn’t get care, or what the care looks like

Despite claims that using AI will ensure speed and accuracy in care of patients, these systems are still underdeveloped and should not be trusted to make decisions with serious health consequences. Algorithms don’t have all the context necessary to make these decisions, and their use has already resulted in false alarms or undetected cases.

AI is not reliable and can be especially harmful in high risk situations such as triage, diagnosing, deciding who gets surgery or specific care, making insurance decisions or recommending treatments. If an AI system misses a symptom, misdiagnoses a patient, recommends the wrong drug or refuses care to a patient in an emergency because it fails to identify the right insurance information, it could cause irreparable damage.

Quote: I don’t think that the enthusiasm around developing tools has been met with the same level of enthusiasm around testing, validating and demonstrating the safety and effectiveness of these tools. I wouldn’t feel comfortable about AI automating clinical decision-making, in diagnosis or treatment. —- Kaiser Permanente AI Chief Daniel Yang

The data used to train AI can be biased, misleading, and even downright racist

Using AI in healthcare settings poses a risk of worsening or replicating existing biases. Bias is often embedded in the data used to train AI models. Using tools with these biases and allowing them to make healthcare decisions will worsen racial, gender, financial and geographic disparities. This paints a target on the backs of communities that are already at risk of harm in the current medical landscape and puts already vulnerable individuals at increased risk. This includes Black and Brown patients, youth, LGBTQ+ individuals, abortion seekers, and patients in need of gender affirming care. Underrepresented populations are at risk as the gaps in data means that they are misrepresented or the care they receive is based on the information that exists for other groups.

Quote: The challenge is that “a generative AI system like GPT-4 is both smarter than anyone you’ve met and dumber than anyone you’ve met. We both assume too much and too little about its potential in health care. — Microsoft Research President Peter Lee

AI use in healthcare raises data privacy, security and surveillance concerns

AI systems depend on large quantities of data harvested from various sources. Healthcare institutions are entering into multi-national data sharing agreements that allow patient data to be shared globally. Despite claims that this data is anonymized, there is always the risk of this data being de-anonymized and sensitive, personal health information being exposed.

The need for more patient data has incentivized developers to find more sources of health information. HIPAA does not cover many apps, websites, data brokers, social media companies, advertising technology startups, and other actors that have access to personal health data, which means they can sell and share this data with third parties.

As more and more personal health data is being collected, it is becoming increasingly challenging for institutions to secure the vast amounts of sensitive and confidential information they have access to. Hackers often target health records because they are so important, and now marketable. This means an increase in targeted attacks and data breaches as more institutions collect, store, and share personal health data.

There is also a risk that healthcare data and AI systems can be used for medical surveillance. AI tools can be weaponized to track and expose targeted medical conditions and procedures such as pregnancy, abortion, or gender-affirming care. ICE is using healthcare data to track immigrants: feeding our data into AI tools could expose us to further abuses and will further threaten our privacy, rights and freedoms.

STOP THE UNREGULATED USE OF AI IN HEALTHCARE

We don’t want machines to make decisions about our health or how we access care. There may be ways that AI can help streamline medical processes, diagnose or develop new healthcare strategies, but it is critical that we are cautious as we explore these opportunities. Decision-makers must put guardrails in place to make sure that we don’t move forward without ensuring our health, privacy, and rights are protected and prioritized above techno-solutions.