2.1 The Legal Concerns of Using AI in Hospitals

While continuous AI monitoring reduces the need for direct patient observation and frees hospital staff from administrative tasks, it is not without challenges.Footnote ²¹ Gerke et al. have mapped out the ethical and legal concerns that broadly arise in AI and health care.Footnote ²² Ethical concerns include informed consent to use, safety and transparency, algorithmic fairness and bias, and data privacy. Legal concerns include safety and effectiveness, liability, data protection and privacy, cybersecurity, and intellectual property.Footnote ²³ Gerke et al. narrow those considerations when examining aspects of ambient intelligence in hospitals to privacy and reidentification risks, consent and liability.Footnote ²⁴ To that, we can add data accuracy concerns.

On privacy and reidentification, the risk is that health care professionals and patients may be identified, which could undermine their privacy. While captured data may be deidentified in one data set (for example, by capturing silhouette images instead of color images), data triangulation makes it possible to reidentify those individuals from other data sets (such as doctor shift rota data).Footnote ²⁵ Additionally, data accuracy risks are associated with smart surveillance due to insufficient data available to train AI algorithms. Developing these algorithms is a cognitively laborious task, which requires obtaining sufficient training examples of scant activities undertaken by the staff and patients around the hospital and manually annotating data from video recordings, which is susceptible to human labelling error.Footnote ²⁶ Data bias is also a potential pitfall, with some studies finding that algorithms demonstrate significant racial bias when predicting aspects of care.Footnote ²⁷

Data accuracy is also affected by practical considerations, such as the camera placement in a hospital ward. The location of smart cameras determines an AI-enabled system’s effectiveness in capturing and processing accurate real-time data.Footnote ²⁸ In this regard, two main concerns arise. Firstly, the architectural design of different hospital units does not allow the hospital to install a network of sensors with sufficient overlapping fields of view, making data association and trajectory prediction across all hospital units a cumbersome task. Secondly, limited visual data affects accuracy because important visual appearance cues are lost as de-identified depth images are used instead of color videos due to privacy concerns surrounding personal health information, making tracking personnel extremely difficult.Footnote ²⁹

These accuracy risks could feed into medical liability concerns. For example, AI systems could make incorrect recommendations that may ultimately harm patients, raising questions about where responsibility lies.Footnote ³⁰ Patients may also require access to data recorded from an AI system for a liability claim, which raises questions about the appropriate rules a hospital implements for retaining and deleting data. Hospital workers could fail to act on a problem detected by an AI system, which could leave the hospital vulnerable to liability claims. AI systems may also record illegal activities, which requires clear policies about the circumstances in which hospitals will retain and share the recordings with law enforcement.Footnote ³¹

On consent, the concern is whether explicit consent should be required from patients for AI systems recording them.Footnote ³² Hospitals have been sued for recording and retaining videos of patients in vulnerable situations in U.S. hospitals.Footnote ³³ Nevertheless, hospitals might escape liability using consent documents containing boilerplate language, which provide authorization for hospitals to record patients for quality improvement and security reasons.Footnote ³⁴ It has been noted that the more explicit and specific the consent is, the better.Footnote ³⁵ For hospital employees, the employer has a “strong claim of a right to record their employee without meaningful consent” owing to legitimate business interest and public interest justifications under U.S. law.Footnote ³⁶ That right to record the employee will be satisfied as long as the recording is taken in a quasi-public space such as an operating room, corridor or patient room, where the goal is to improve patient care, where workers are informed about the recording, and where proper safeguards are in place to protect individual privacy.Footnote ³⁷ As such, some literature has examined the legal concerns arising in the context of AI being used in hospitals. The query here is the extent to which there is a distinction in the mental health inpatient context.

3.1 AI-CCTV Surveillance in Hospitals

The manner in which surveillance is implemented is very important to the patient experience. CCTV cameras are a crucial tool for monitoring communal areas shared by patients and staff and seclusion rooms. CCTV may, in certain circumstances, be used in private areas, including patient bedrooms.Footnote ⁵¹ Continuous CCTV monitoring enables better risk detection in hospital wards, allows patient observations to occur without disturbance, and provides data for training purposes.Footnote ⁵² However, surveillance “within hospital wards is not based on reciprocity” as it subjects patients to constant monitoring without their consent.Footnote ⁵³ Hospitals have adopted numerous policies and guidelines generally addressing surveillance, but additional, more specific considerations on privacy and informed consent apply in psychiatric wards, suggesting that much caution should be exerted when monitoring psychiatric patients with AI.Footnote ⁵⁴ This phenomenon is easily observed when examining the rubric of rules that have developed in England. There, the Home Office has developed a Surveillance Camera Code of Practice that emphasizes CCTV should only be used in the pursuit of a legitimate aim, be necessary, take account of an individual’s privacy and be subject to regular review, among other rules.Footnote ⁵⁵ In the health sphere, the Care Quality Commission (CQC) and Information Commissioner’s Office (ICO) provide guidance and regulatory oversight for care in England. The ICO can investigate and take enforcement action where surveillance breaches a patient’s rights under the Data Protection Act (DPA) 2018 and U.K. GDPR. To assist organizations, it has developed detailed guidance on the use of CCTV and surveillance.Footnote ⁵⁶

Privacy and dignity must always be maintained with surveillance.Footnote ⁵⁷ Cameras can be installed in public and communal areas, but not private areas such as bathrooms. However, covert recordings are permissible if the benefits outweigh other considerations on privacy and dignity.Footnote ⁵⁸ All recordings must be logged and traceable and not retained for more than 28 days.Footnote ⁵⁹ Disclosure of recordings can only be made in strict adherence to the DPA 2018 and U.K. GDPR.Footnote ⁶⁰ Where an individual cannot consent to the use of CCTV, the clinical leads must decide according to the Mental Capacity Act (MCA) 2005.Footnote ⁶¹ Thus, if CCTV is to be used in a patient’s bedroom (especially where a patient lacks mental capacity), permission may be required from the Court of Protection on an ad-hoc basis.Footnote ⁶²

Also of import is the ECHR. Article 8 articulates a qualified right, which means that recordings can be used despite breaching privacy where the recording is lawful, for a legitimate aim, and proportionate.Footnote ⁶³ The least restrictive possible alternatives to CCTV must be considered, such as using call bells or pendants, using sensors that detect movements, improving staff training and supervision, or increasing staff numbers.Footnote ⁶⁴ Depending on the system used, these considerations may be a double-edged sword for AI. CCTV incorporating AI technology may be less likely to gain approval, but an AI-enabled wearable may be more likely to gain approval because it may be deemed less restrictive.

The British Institute of Human Rights (BIHR) provide case studies for these considerations. They discuss a man with autism who sometimes harmed himself. Staff installed a CCTV camera in his room because that was deemed less restrictive than posting a staff member in his room twenty-four hours a day; they determined that would be a legitimate and proportionate response to the circumstances. The staff then installed CCTV in all the rooms of the facility, but they were not switched on. The courts considered the matter and decided that CCTV could be used in the man’s bedroom, but its use was deemed a significant restriction of his Article 8 ECHR rights, requiring that a judge regularly review the use of the CCTV camera in his bedroom. It was held that using CCTV in other bedrooms was not lawful, legitimate, or proportionate, so they were removed.Footnote ⁶⁵

Such examples raise questions about the appropriate limits of surveillance. It is wrong to assume that psychiatric patients require higher levels of supervision. Applying blanket surveillance rules without considering the impact on individuals could cause more harm and distress than help.Footnote ⁶⁶ This was the case for another patient detained under the law and subject to CCTV surveillance installed in her room and every other room in addition to “arm’s length” observations by two staff day and night. The patient found the CCTV to be intrusive, inhumane and degrading.Footnote ⁶⁷ Consequently, surveillance must be carefully planned, considering what is legitimate, necessary, and proportionate. A major trade union has recommended to members that privacy impact assessments be conducted.Footnote ⁶⁸ The CQC has noted that it is up to the providers to decide whether to use surveillance (within the limits of the law), but that “we would be concerned by an over-reliance on surveillance to deliver key elements of care, and it can never be a substitute for trained and well supported staff.”Footnote ⁶⁹

While such legal protections exist in England, their implementation can be problematic; privacy can be quite limited. The Mental Health Act Revised Code of Practice places an obligation on public authorities to respect a person’s right to a private life, including those detained under the Act.Footnote ⁷⁰ However, CCTV monitoring of communal areas used by patients—not all of whom have consented to such surveillance—raises privacy concerns. The right to privacy is further curtailed for involuntarily detained psychiatric patients. Since their access to space outside the ward environment is restricted due to the involuntary nature of admission, such patients have limited areas where they can avoid constant monitoring.Footnote ⁷¹ These risks may be replicated when AI is incorporated into such technologies. The risk of constant surveillance remains and may be heightened if the form of monitoring encompasses wearable devices that constantly track a patient, no matter their location in a ward.

Despite these concerns, the research in this area is not clear-cut. One study in Australia notes that CCTV cameras do not necessarily upset patients or motivate them to spend time in locations without camera surveillance.Footnote ⁷² Some patients may be passive about CCTV monitoring, which could make them feel safe, particularly on open wards.Footnote ⁷³ The views of staff should also be considered. Many staff in psychiatric wards have been victims of assault by patients, and surveillance could protect them.Footnote ⁷⁴ Staff are also concerned about how to use AI systems to identify such risks.Footnote ⁷⁵ The key is to implement security measures that do not provoke aggression or violence. In this regard, the characteristics of the ward are essential, and due consideration should be given to how AI systems could and should fit within that environment. The CQC has noted that many psychiatric units are in old and outdated buildings, which can lead to issues around privacy and dignity for patients. Wards can be “barren, visually impoverished environments dominated by security fencing” that can undermine morale and have a detrimental impact on recovery.Footnote ⁷⁶ The same is true of most psychiatric wards in the world, which are in old buildings that do not allow for proper CCTV surveillance. The use of AI tracking systems in those environments should, therefore, only be done with careful planning.

Another challenge is gaining informed consent for using CCTV cameras due to psychiatric patients’ varying levels of perception and cognition which can result in an impaired or fluctuating capacity to consent. Certain conditions, such as those linked to feelings of paranoia, tend to incite violent reactions towards the existence of cameras.Footnote ⁷⁷ While it is good practice under the MCA to obtain informed consent for CCTV monitoring, how and when such consent should be sought is ambiguous. Although there is a general presumption of surveillance in public spaces at hospitals not requiring consent, the same cannot be said when CCTV is used to monitor communal areas, seclusion rooms or patient bedrooms. These risks are deepened and broadened where AI is used because of the more pervasive and detailed information that such systems may potentially generate. Even if there is consent for using an AI system in a specific way, the autonomous nature of the system could result in recommendations and analyses for which consent was not obtained initially.

Thus, it is challenging to determine where the balance should tilt on using AI-enabled surveillance systems. There are obvious risks and strict legal considerations for the use of CCTV generally, but should those risks entirely preclude their use for psychiatric inpatients? AI-enabled surveillance does not yet exist in mental health wards, but one could consider prison surveillance by analogy. AI surveillance is used on prisoners to detect and prevent harmful incidences in China, Hong Kong, the United Kingdom, the United States, India and South Korea.Footnote ⁷⁸ Some systems can detect changes in parameters up to thirty minutes before an observable act of aggression with greater precision than existing methods.Footnote ⁷⁹ In Hong Kong, AI surveillance systems can detect instances of self-harm and alert staff, and detect if a person has any contraband.Footnote ⁸⁰ Wearable wristbands can track vital signs and flag any signs of injury. Robots in South Korea can monitor violence and suicide risks.Footnote ⁸¹ In the United States, mass-monitoring millions of phone calls through AI has helped officials combat violence, drug smuggling and attempted suicides in near real-time. By searching for keywords, phrases, and prison slang, investigators notify law enforcement officers when the AI system picks up suspicious language. This has prevented dozens of attempted suicides by providing inmates with psychological counselling in the minutes and hours after the inmate was recorded making references to harming themselves, threatening violence, or plotting to smuggle in contraband.Footnote ⁸²

Of course, the analogy cannot be stretched too far. Inpatients are not inmates in prisons, and one should not generalize the characteristics of psychiatric inpatients to be at risk of suicide or harm to others. Some inpatients are voluntarily there. Nevertheless, there are some similar paradigms and risks that exist that are helpful to consider, such as patients being involuntarily detained, those at risk of self-harm or suicide, those possessing contraband that can be a risk to themselves or others, and those posing a risk of violence (a common occurrence on psychiatric wards).Footnote ⁸³ AI may be helpful in those instances, and there is value in further research being conducted in this area. From a legal standpoint, applying England’s laws as an example, the conclusion that can be drawn at present is that, where a dispute or question arises for a court to consider, a surveillance system in mental health wards may require approval from a court and may be subject to constant review by a judge. However, from a practical standpoint, those legal protections may not always manifest, meaning privacy concerns will persist. In most cases, it appears that surveillance is used without questions arising for a court to consider, and there is a risk that AI-enabled systems will fall within the same void despite the greater legal questions or risks those systems may pose owing to their pervasive nature.

3.2 Predictive Behavior Models for Suicide

Many patients are detained under mental health laws for treatment, and many of them pose a considerable risk to themselves or others.Footnote ⁸⁴ Given the frequency of violence and suicide attempts in psychiatric units, accurate prediction of such behaviors is critical for proper treatment.Footnote ⁸⁵ The current structure for treatment relies on predictive models that are rarely specific to an individual, instead “classifying a patient to a certain group and referring to that group’s averages.”Footnote ⁸⁶ Traditional statistical models accommodate a limited number of variables, limiting predictions based on surrogate markers of suicide attempts, suicidal planning and ideation, and self-injury.Footnote ⁸⁷ Psychiatric units also use risk assessment instruments to predict aggressive behavior.Footnote ⁸⁸ These instruments use individual medical and social history data, treatment conditions, behaviors, and psychopathology.Footnote ⁸⁹ Yet, these tools are “based on one-off assessments of risk factors, use electronic or paper forms, and only predict risk over the very short-term (i.e. the next 24 hours)”.Footnote ⁹⁰ Moreover, these factors become clinically irrelevant in an emergency where the medical history of a possibly violent detained patient is unknown.Footnote ⁹¹ Other common methods include analyzing EHRs, national registries and self-reported questionnaires, although the latter’s reliability is questionable due to the patient’s motivation, capacity, and self-awareness.Footnote ⁹²

Overall, the predictive ability of suicide has not improved in fifty years of research.Footnote ⁹³ Suicide prediction remains incorrect in most cases.Footnote ⁹⁴ Yet eighty-three percent of people who die by suicide have contact with health services the year before their death, and forty-five percent have contact the month before.Footnote ⁹⁵ Therefore, there is a significant opportunity to assist doctors in assessing the risk of suicide using AI when those patients present.Footnote ⁹⁶

AI could be used in hospitals as a decision support for assessing suicide risk using surveillance or by examining EHRs, hospital records, and other government data sources.Footnote ⁹⁷ Inpatients may also provide data using smartphones to log their mood, sleep patterns, step count, and interactions with other patients.Footnote ⁹⁸ At the same time, using such systems could pose risks for patients in two realms: first, in research, when AI systems are being developed; second, in the ward, once AI has been deployed.

On research in the mental health context generally (not only inpatients), Marks has examined AI-based suicide prediction algorithms and the risks they pose.Footnote ⁹⁹ Marks highlights AI tools that examine medical records to predict suicide and social interactions that fall outside the health care sphere.Footnote ¹⁰⁰ Focusing on the former, academic medical centres, hospitals, and government agencies usually examine medical records containing information about speciality and primary care visits.Footnote ¹⁰¹ In the United States, suicide prediction is governed by the Health Insurance Portability and Accountability Act (HIPAA) which protects privacy and imposes penalties on entities when patient data is breached.Footnote ¹⁰² Research that uses patient data for suicide prediction is also subject to the Federal Common Rule, medical ethics rules and IRBs, all designed to protect human research subjects.Footnote ¹⁰³ These avenues have their own challenges. AI is autonomous, so dynamic consent may be needed from patients to use their data.Footnote ¹⁰⁴ Consent is also imperative where public data is integrated into predictive models.Footnote ¹⁰⁵

Additional considerations will arise where AI is approved for use for predicting suicide on a mental health ward in the future. Marks highlights specific risks that arise with such technology and groups those risks into concerns about safety, privacy, and autonomy.Footnote ¹⁰⁶ That framework is helpful in the inpatient context. Safety risks include false negatives and false positives.Footnote ¹⁰⁷ Those misclassifications could identify someone as being at risk of suicide (when they are not), or not at risk of suicide (when they are).Footnote ¹⁰⁸ The resulting action will be wrong for that patient. A patient thought to not be at risk may attempt suicide. A patient thought to be at risk may be detained and lose their freedoms when they are not at risk.Footnote ¹⁰⁹ Those patients may also risk having suicidal thoughts as part of their permanent medical record when it is not true. Moreover, involuntary admission in these circumstances could be traumatic and dehumanizing, raising the risk of suicide.Footnote ¹¹⁰ Improper detention or the denial of access to clinical services could form the basis of a legal claim against the clinician, the health care entity, or the developer.Footnote ¹¹¹

A potential factor in this paradigm is the complexity in mental health disorders, which have highly complex aetiology and symptomology that do not lend well to strict classification, thus, affecting the reliability of predictive algorithms.Footnote ¹¹² In practice, small sample sizes for psychiatric disorders are used which limits data for training and testing, leading to overfitting of the data and skewed accuracy measurements. Clinicians are forced to rely on less accurate models, adversely affecting suicide prediction.

While doctors should exercise their judgement, they may be reluctant to ignore the AI system because of concerns about medical liability.Footnote ¹¹³ It is widely held in psychiatric practice that suicide assessment tools should be used mainly to aid clinical judgment based on a clinical, interview-based risk assessment. However, the fear of legal consequences could result in clinicians relying more on AI than clinical judgement, which could threaten clinicians losing psychiatric risk assessment skills. Practice will vary in hospitals and jurisdictions, but the fear of liability may lead some doctors to take extra measures based on poorly assessed risks—measures that are costly in terms of the personnel required. An inpatient deemed high risk for suicide could be placed under constant direct observation, with frequent assessments being completed. That task may be cumbersome and unnecessary, but clinicians may nevertheless “play it safe” without worrying about resource implications. Thus, in a paradigm where AI exists in psychiatric wards, there are misclassification risks that could lead to staff taking inappropriate action for that patient. AI might improve risk prediction when trained properly but could provide poorer predictive capabilities than traditional methods owing to poor training or input data coupled with an overreliance and trust placed in those systems by doctors.

Privacy concerns cover data breaches and the transfer or sale of data to third parties, such as advertisers and data brokers, leading to affected individuals being stigmatized, exploited or discriminated against.Footnote ¹¹⁴ In the United States, HIPAA protects patient privacy by requiring that suicide-related data cannot be transmitted without being deidentified.Footnote ¹¹⁵ Health care providers are also prohibited from sharing non-anonymized health data with advertisers.Footnote ¹¹⁶ In England, strict rules on surveillance were noted covering the length of time that data can be stored, who can access and the restriction on third parties using that data. For AI in a psychiatric ward, strict data rules could significantly restrict many systems from being used or restrict how they handle data if they are used. If AI is used on wards, it is unclear how these factors will manifest on the ground. At present, it is hard to safeguard the confidentiality of data used in digital mental health tools due to apps being poorly understandable and developers having opaque data management policies.Footnote ¹¹⁷ Such a lack of clarity could not be permitted in the mental health setting.

Autonomy covers risks of censorship, unnecessary confinement or civil commitment, and criminal penalties or incarceration in countries where suicide attempts are illegal.Footnote ¹¹⁸ Warrantless searches could be conducted on individuals at risk of suicide based on the AI system’s prediction.Footnote ¹¹⁹ In a psychiatric ward, a patient could be moved into a locked ward based on a misclassification, undermining their autonomy, be traumatic and inadvertently increase the risk of harm. This implicates consent because confinement could occur without consent based on erroneous information. In reality, the patient is not at risk, and it would be an egregious breach of their rights if confined.Footnote ¹²⁰

Overall, AI may help to overcome some existing limitations of predicting suicide. However, its use is not without risk. Doctors may be reluctant to disagree with AI recommendations, but that reluctance could lead to inappropriate decisions for the patient, resulting in the loss of autonomy (where the AI system is wrong). Whether AI could be used in practice owing to privacy considerations restricting the use of patient data is unclear.

3.3 AI and Hospital Administration

In some hospitals, staff can be overburdened with administrative work, reducing the time for direct patient care. Bed and staff shortages often result in suboptimal risk assessments and early discharges, with subsequent high readmission rates. This part highlights these added challenges of inefficient bed management and mismatch of staff competencies to patient needs due to scheduling irregularities. It further explores the use of AI in ward administration and the legal risks associated with its incorporation in mental health.

While these administrative challenges may exist in other wards, one cannot ignore their impact in psychiatry where there is “severe underinvestment” in many parts of the world.Footnote ¹²¹ Bed shortages in mental health wards have increased over the years, and worsened during COVID-19, causing a supply-demand mismatch.Footnote ¹²² Inadequate bed management causes difficulties in procuring beds when needed, resulting in frequent patient transfers that disorient patients suffering from certain mental disorders, leading to stress and loss of trust in services.Footnote ¹²³ Moreover, the risk of homelessness and incarceration of patients with severe mental illnesses and increased risks of physical harm to other patients or members of the public, are all possible consequences of insufficient psychiatric bed capacity.Footnote ¹²⁴

Traditionally, hospital managers oversee capacity and demand based on estimations and historical knowledge. Currently, bed registries and bed tracking systems are used to track bed availability in psychiatric units. However, since the registries are not linked to EHRs or hospital admission/discharge data systems, bed tracking systems are not updated automatically. Instead, they rely on administrative and medical personnel to update manually.Footnote ¹²⁵ Time-consuming meetings are often held to discuss case management, provide updates on predicted discharges, and the current demand for beds. Information is not always up-to-date due to the staff’s focus on other responsibilities. The registries also struggle to match bed availability to patients with complex needs, such as those who exhibit violence or aggression, have co-occurring medical conditions or needs.Footnote ¹²⁶ This leads to suboptimal risk assessment, with the likelihood of patients being discharged before they receive proper treatment, resulting in increased readmission rates.Footnote ¹²⁷

Similarly, staff shortages and a mismatch of staff competencies to patient needs in psychiatric wards pose a significant barrier to treating mental health disorders due to improper utilization of staff expertise. Understaffing and poor staff rota design has led to unclear and overlapping job responsibilities, which adds to the inefficiencies in psychiatric units.Footnote ¹²⁸ According to the BMA, “staff working in mental health care are at ‘breaking point’ as they try to handle rising demand with a continuous staffing rota gap.”Footnote ¹²⁹ The copious administrative work allows little time for one-on-one direct patient care time. Long working days (of up to twelve hours) and inadequate staff scheduling have been identified as a significant threat to employee health and well-being, affecting their mental health.Footnote ¹³⁰ These rota gaps adversely impact staff training, morale, and the quality of care they provide. Consequently, medical personnel are often “forced to act above their competencies, putting patient safety at risk.”Footnote ¹³¹

While bed shortages and staff rota challenges exist in other wards, the problems in psychiatric wards are at “crisis” levels.Footnote ¹³² Hospitals, in general, are shifting towards using AI to assist with these administrative challenges. For bed management, AI is being deployed to predict discharges to free up bed space. A Portugal-based company developed an AI solution that predicts discharges by combining data about bed usage with clinical data to suggest actions five to seven days ahead. The AI system considers ward gender, surgery schedules, patient age and condition, type of bed needed, number of beds and nursing schedules, among other variables, increasing bed manager productivity by thirty to fifty percent.Footnote ¹³³ This reduces stress and enables staff to identify and prioritize the sickest patients, make faster decisions, and allocate hospital resources efficiently.

Similarly, AI is being used to predict practitioners’ workload in ICUs and EDs and ensure that the right skilled workers are linked with appropriate shifts.Footnote ¹³⁴ These systems extract information, match the competencies of health care workers to specific tasks, and help fill available slots, making the planning of rotas more efficient. AI systems are also designed to consider policy requirements when making recommendations, such as law-mandated working hours or the presence of senior doctors for specific shifts.Footnote ¹³⁵

These solutions can be implemented in psychiatric wards, albeit with caution. Developing a successful AI system for psychiatric ward administration requires high-quality data to train the algorithm and validate the model.Footnote ¹³⁶ Given that this sensitive medical data is a regulated asset, risks arise over data accuracy and privacy.Footnote ¹³⁷ A case study involving AI used to improve hospital bed allocation at Kettering General Hospital revealed the challenges of obtaining enough good quality data that cover the complexity of patient needs. These challenges include the need for the technology to be reconfigured regularly with new information about increased beds, changed ward layouts or flu admission peaks.Footnote ¹³⁸

In mental health care, the challenges of obtaining enough data and the use of EHRs to train resource allocation and staff scheduling algorithms are amplified and somewhat circular. AI requiring data on psychiatric inpatients must comply with data protections noted above. For example, in the United States, health care data in EHRs is considered sensitive, with mental health data being even more sensitive due to the high risk of stigmatization and discrimination.Footnote ¹³⁹ The HIPAA Privacy Rule provides special protection for mental health information, including extra protection for psychotherapy notes and personal notes contained within, and requires the patient’s written authorization for its use or disclosure.Footnote ¹⁴⁰

However, even if research is legally compliant, access to data may be difficult owing to the lack of data, the extra-sensitive nature of the information, and the restrictions on who it can be shared with. The quality of the data on the EHR may also be insufficient for adequately training the algorithm. Mental health information is “regularly missing from EHRs, documented in the wrong place, or under documented in specific contexts.”Footnote ¹⁴¹ This is likely due to the subjective nature of information,Footnote ¹⁴² greater reliance on narrative progress notes as opposed to quantitative measures,Footnote ¹⁴³ and stigma surrounding mental health conditions.Footnote ¹⁴⁴ Fragmented use of paper and electronic records also contributes to missing mental health information on EHRs.Footnote ¹⁴⁵ This ambiguous and, at times, incorrect documentation of mental health data may be insufficient to train AI systems for the administration of psychiatric wards. Thus, even if developers can overcome the legal challenge surrounding privacy, they may be limited by the data that is available.