4.1 Introduction

4.2 How Does the EU Medical Devices Regulation Deal with the Cybersecurity of Medical Devices?

The provisions of the EU Medical Devices Regulation (MDR)Footnote ¹⁵ primarily address manufacturers of medical devices who are defined as “the natural or legal person who manufactures or fully refurbishes a device or has a device designed, manufactured, or fully refurbished and markets that device under its name or trademark.”Footnote ¹⁶ No explicit reference to cybersecurity is provided in the main part of the MDR. However, it provides some essential cybersecurity-related requirements that manufacturers have to implement in a medical device.Footnote ¹⁷

When putting a medical device on the market or into service, Article 5(1) of the MDR obliges its manufacturer to ensure that the device is compliant with the MDR obligations when used in accordance with its intended purpose. According to Article 5(2) of the MDR, “a medical device shall meet the general safety and performance requirements” (also including the cybersecurity-related requirements)Footnote ¹⁸ “set out in Annex I [of the MDR] … taking into account the intended purpose.”Footnote ¹⁹ The intended purpose is defined in Article 2(12) as “the use for which a device is intended according to the data supplied by the manufacturer on the label, in the instructions for use or in promotional or sales materials or statements and as specified by the manufacturer in the clinical evaluation.” As part of the general requirements set in Annex I of the MDR, “devices shall achieve the performance intended by the manufacturer”Footnote ²⁰ and be designed in a way suitable for the intended use. They shall be safe and effective, and associated risks shall be acceptable when weighed against the benefits of the patients and level of protection of health and safety while taking into account the state of the art.Footnote ²¹

Moreover, “[m]anufacturers shall establish, implement, document, and maintain a risk management system.”Footnote ²² Part of this system also includes risk-control measures to be adopted by manufacturers for the design and manufacture of a device, and they shall conform to safety principles and state of the art.Footnote ²³ A medical device designed to be used with other devices/equipment as a whole (including the connection system between them) has to be safe and should not impair the specified performance of the device.Footnote ²⁴

Furthermore, a medical device shall be designed and manufactured in a way to remove, as far as possible, risks associated with possible negative interaction between software and the IT environment within which they operate.Footnote ²⁵ If a medical device is intended to be used with another device, it shall be designed so the interoperability and compatibility are reliable and safe.Footnote ²⁶ A medical device incorporating electronic programmable systems, including software or standalone software as a medical device, “shall be designed to ensure repeatability, reliability, and performance according to the intended use,”Footnote ²⁷ and “appropriate means have to be adopted to reduce risks or impairment of the performance.”Footnote ²⁸ A medical device should be developed and manufactured according to the state of the art and by respecting the principles of the development lifecycle, risk management (including information security), verification, and validation.Footnote ²⁹ Lastly, manufacturers shall “set out minimum requirements concerning hardware, IT network characteristics, and IT security measures, including protection against unauthorized access.”Footnote ³⁰ Concerning information to be supplied together with the device, manufacturers must inform about residual risks,Footnote ³¹ provide warnings requiring immediate attention on the labelFootnote ³² and, for electronic programmable system devices, give information about minimum requirements concerning hardware, IT networks’ characteristics, and IT security measures (including protection against unauthorized access), necessary to run the software as intended.Footnote ³³

4.3 Regulatory Challenges Stemming from the MDR Analyzed Through the Lens of the MDCG Guidance on Cybersecurity for Medical Devices

The Medical Device Coordination Group (MDCG) of the European Commission endorsed Guidance on Cybersecurity for Medical Devices (Guidance) in December 2019Footnote ³⁴ where it dealt with the cybersecurity-related provisions embedded in the MDR. Already, it is necessary here to mention that this MDCG Guidance is not a legally binding document. Hence, in case of disagreement, manufacturers could decide not to follow it – which might have an impact on the overall harmonizing purpose of the MDR and lead to a divergence of application of the EU principles and laws on a Member State level. Nevertheless, being the first guiding document on this topic issued by the EC for the medical devices sector, it is an essential step in further elaborating on specific MDR cybersecurity-related provisions.

As already mentioned in the previous section, the MDR does not expressly refer to cybersecurity.Footnote ³⁵ Nor does the MDCG Guidance define the terms “cybersecurity,” “security-by-design,” and “security-by-default.” Instead, the latter document only provides an outline of its provisions relating to cybersecurity of medical devices and points out conceptual links between safety and security.Footnote ³⁶ Leaving these terms theoretical and undefined does not facilitate their implementation in practical terms by the stakeholders concerned.

Moreover, no reference in the MDCG Guidance is given to definitions provided by the Cybersecurity Act (CSA).Footnote ³⁷ Establishing a connection in the soft-law instrument (i.e., the Guidance) with the latter would imply a reference to a hard law definition. This link could serve to reduce the ambiguity of the term, and it might help in achieving more coherence within the EU cybersecurity regulatory framework as a whole.Footnote ³⁸ The proposed approach would be ultimately beneficial for manufacturers as it would bring more clarity in the interpretation of MDR requirements.

The MDCG Guidance stresses the importance to “recognize the roles and expectations of all stakeholders”Footnote ³⁹ on joint responsibility and states its “substantial alignment” with International Medical Device Regulators Forum (IMRDF) Principles and Practices for Medical Devices Cybersecurity.Footnote ⁴⁰ To this end, achieving a satisfactory level of the cybersecurity of a medical device concerns manufacturers, suppliers, health care providers, patients, integrators, operators, and regulators. Manufacturers are bound by the majority of the provisions in the MDR. Integrators of a medical device are, among others, responsible for assessing a reasonable level of security while operators need to ensure the required level of security for the operational environment, and that personnel are properly trained on cybersecurity issues. At the same time, health care professionals are responsible for a device being used according to the description of the intended use, while patients and consumers need to “employ cyber-smart behaviour.”Footnote ⁴¹ All of these stakeholders are an equally important part of the cybersecurity chain,Footnote ⁴² and each is responsible for ensuring a secured environment in which a device could smoothly operate for the ultimate benefit of patients’ safety.

Nevertheless, the MDCG Guidance failed to elaborate on how exactly the joint responsibility of different stakeholders is influenced or conflicted by other applicable laws, in particular, when it comes to the Network and Information Systems (NIS) Directive,Footnote ⁴³ the General Data Protection Regulation (GDPR),Footnote ⁴⁴ and the Cybersecurity Act (CSA).Footnote ⁴⁵ Since the expert group did not tackle them in detail in theory, it is also hard to imagine how the interested stakeholders operating within the medical devices domain are supposed to implement in practice different pieces of legislation divergent in scope and applicability.Footnote ⁴⁶ Hence, the MDCG should consider adopting a more holistic approach in the future when determining the meaning of “joint responsibility” as this would help in analyzing relevant aspects of other horizontal legislation and, eventually, in achieving a more coherent cybersecurity regulatory framework.

Finally, what seems to be heavily overlooked for unclear reasons is the applicability of the Radio Equipment Directive (RED),Footnote ⁴⁷ which has not even been mentioned in the MDCG Guidance. The RED cybersecurity-related provisions and their interaction with MDR as well as the other laws applicable to the cybersecurity of medical devices are explained below.

4.4 Regulatory Challenges Stemming from Other Legal Frameworks Applicable to Medical Devices

Regulation of cybersecurity is a complex task. Cybersecurity is an area in which different policy fields need to be combined (horizontal consistency), and where measures need to be taken at both levels – the European Union and Member States (vertical consistency).Footnote ⁴⁸ Regulation of medical devices is complex, too, as it is a multi-levelFootnote ⁴⁹ legal framework characterized by specialization and fragmentation.Footnote ⁵⁰ Regulating the cybersecurity of medical devices implies bearing the complexities of both legal frameworks. In this regard, we identified four regulatory challenges: regulatory overlapping; fragmentation risks; regulatory uncertainty; and duplication. We clarify the first two challenges as relating to horizontal consistency requirements, the third to vertical requirements, and the fourth to a combination thereof. Finally, we envisage specialization and fragmentation as a common denominator of all four challenges.

4.5 Conclusions and Recommendations

The adequate level of cybersecurity and resilience of medical devices is one of the crucial elements for maintaining the daily provision of health care services. Above all, it is pivotal to mitigate risks relating to patients’ health and safety. On the one hand, the ongoing debate on the topic in the United States and, more recently in the European Union, shows an increasing level of awareness amongst regulators, manufacturers, health care professionals, and other involved stakeholders. On the other hand, the research presented in this chapter shows that the existing EU legal framework dealing with medical devices’ cybersecurity brings significant regulatory challenges. In order to provide a step forward in mitigating these challenges, the EU regulator might consider the following recommendations:

1. 1. Establish a more robust connection of the MDCG Guidance with EU cybersecurity (hard) laws, especially the CSA and its definitions of cybersecurity, security-by-design, and security-by-default. Ensuring consistent use of terminology across different pieces of legislation (binding and non-binding) would also help manufacturers in meeting the requirements as it would bring more clarity in the interpretation of the MDR cybersecurity-related provisions.

2. 2. Clarify the meaning and implications of “joint responsibility” in the intertwining with other applicable laws (in particular when it comes to the NISD, GDPR, and CSA). Further explanations on how exactly the responsibility stemming from one piece of legislation applicable to a specific stakeholder is influenced or conflicted with the responsibility of another stakeholder (stemming from the same or different piece of legislation) would represent a meaningful tool to guide manufacturers in complying with all the relevant laws.

3. 3. Clarify the scope of application of the CSA for certification mechanisms and MDR security requirements. In particular, the EU regulator should explain how the MDR cybersecurity-related requirements apply to an ICT product which also falls under a definition of a medical device, and what type of certification schemes would be relevant.

4. 4. Provide guidance on the application of the RED, its interaction with the MDR and other laws applicable to the cybersecurity of medical devices.

5. 5. Ensure cooperation between competent national authorities (i.e., for incident notifications) in order to achieve timely respect of the requirements, and to avoid compliance duplication.

5.1 Introduction: mHealth Apps: Promise or Threat?

An increasing number of European Union (EU) citizens use mobile apps to monitor their own fitness, lifestyle, or general health to take control over their health outside of a clinical setting.Footnote ¹ This growing trend is reflected in the content of mobile app stores: self-monitoring mobile health (mHealth) apps such as running trackers and medication reminders are omnipresent. While mHealth apps are said to hold great potential for empowering individuals, the apps also constitute threats to users’ fundamental rights in the European Union.Footnote ² The main risk is posed by the extensive processing and sharing of health data with third parties by mHealth apps. Users have limited awareness of, and control over, who has access to their health data.Footnote ³ This leads to a paradox: users turn to mHealth to increase self-empowerment, but at the same time surrender power due to this lack of data control.Footnote ⁴

These risks are further compounded by the lack of effective EU regulation. The EU legal framework on health and protection of patients’ rights does not apply to self-monitoring mHealth app users.Footnote ⁵ Furthermore, while the EU’s General Data Protection Regulation (GDPR) provides a solid legal framework for the protection of health data, in practice, many mHealth apps do not comply with its provisions.Footnote ⁶ When traditional legislative regulation does not lead to the intended effect, complementary alternative forms of regulation may be the solution.Footnote ⁷ In the context of health data protection in mHealth apps, mobile app distribution platforms (app stores) may be well positioned to improve health data protection by means of self-regulation. App stores in the European Union already occupy an important place in this regard by offering a top-down regulation of third-party mHealth apps distributed on their platforms by means of app review procedures. App stores require app developers to comply with certain rules as part of a preapproval process and remove noncompliant apps. This “gatekeeping function” empowers app stores to influence app developers’ conduct: a form of industry self-regulation.Footnote ⁸ Starting from this premise, the purpose of this chapter is to evaluate whether and to what extent self-regulation by app stores may contribute to the level of health data protection in the European Union.

The chapter is structured as follows. First, it outlines health data protection issues concerning mHealth apps (Section 5.2). Next, it describes the EU legal framework governing mHealth apps, focusing on the GDPR (Section 5.3). Subsequently, it discusses the benefits and risks of industry self-regulation as an alternative means to protect data protection rights in light of current mHealth regulation practices by Apple’s App Store and Google’s Google Play (Section 5.4). Finally, this chapter proposes several improvements to self-regulation in this field (Section 5.5), which will provide the basis for conclusions (Section 5.6).

5.2 Health Privacy Issues in Self-Monitoring mHealth Apps

Popular examples of mHealth apps include calorie counters, apps to monitor menstruation cycles, and running trackers. These types of apps continuously monitor users’ behavior over an extended period of time. While the focus of mHealth apps ranges from health to fitness and lifestyle, all of them collect large amounts of health-related data, such as biometric data, data concerning vital body functions, and health indicators. Most of these data qualifies as “data concerning health” within the meaning of the GDPR.Footnote ⁹ Health data should be understood in a broad manner.Footnote ¹⁰ The GDPR’s definition of health data implies that information about users’ weight, blood pressure, tobacco, and alcohol consumption is considered health data because this information is scientifically linked to health or disease risks.Footnote ¹¹ Furthermore, certain types of information may not be health data as such, but may transform into health data when monitoring takes place over a longer period of time (i.e., average steps per month), or the data is combined with other data sources (i.e., daily calorie intake and social media profile).Footnote ¹²

The risk for a violation of the users’ fundamental rights is high, since misuse of health data may be irreversible and have long-term effects on data subjects’ lives and social environments.Footnote ¹³ Several studies show that the extensive processing of health data by mHealth apps poses numerous threats to privacy.Footnote ¹⁴ This is mainly caused by the fact that health data is a valuable commodity: big data companies are increasingly interested in health data as it is scarce because of the expensive collection process.Footnote ¹⁵ Therefore, mHealth apps may encourage users to provide more health data in order to make more profit. Passively collected data, such as calculated overviews of average steps, are regularly collected beyond users’ control.Footnote ¹⁶ Moreover, mHealth apps often use a standard Terms of Service, setting the rules on a “take it or leave it” basis.Footnote ¹⁷ Consequently, users are often unaware of the exact type and volume of collected data.Footnote ¹⁸

Additional concerns are raised with regard to the user’s control over access to the collected health data. Most apps provide for the possibility to disclose information to an “undefined (future) audience.”Footnote ¹⁹ For example, many apps share health data among unspecified users to provide comparisons, and app operators may sell health data to third parties, such as advertisers and insurance companies.Footnote ²⁰ Apps often do not provide the option to consent granularly: users have to consent to all receivers and all types of data at once.Footnote ²¹ In conclusion, the extensive processing and third-party sharing of health data by mHealth apps compromises users’ control and therefore poses threats to users’ privacy rights.

5.3 The Effectiveness of EU Legal Protection of Health Data in mHealth Apps

5.4 Self-Regulation by App Stores as a Solution to Improve Health Data Protection

When traditional (legislative) regulation does not lead to the intended effect, complementary alternative forms of regulation, such as self-regulation, may be the solution.Footnote ⁴⁹ While the important role of app stores in securing GDPR compliance has been recognized by the European Union on several occasions,Footnote ⁵⁰ and the role of digital platforms in protecting fundamental rights online is a popular topic in legal scholarship, the discussion seems to focus mainly on social media platforms and does not elaborate on app stores.Footnote ⁵¹ However, app stores may be well positioned to improve health data protection by means of self-regulation.

5.4.3 Case Studies

5.4.3.1 Apple App Store

In order for app developers to submit apps to the Apple App Store, they must register to the Apple Developer Program, governed by the Apple Developer Program License Agreement.Footnote ⁶⁶ Furthermore, Apple App Store reviews all submitted apps and app updates according to the App Store Review Guidelines.Footnote ⁶⁷ As shown in Table 5.1 above, these Guidelines contain specific rules on mHealth apps and state that these apps may be reviewed with greater scrutiny.Footnote ⁶⁸ The guidelines also contain general provisions on processing of personal data and privacy. First, apps must include a privacy policy, explaining how users can exercise their rights to data retention, deletion, and withdraw consent.Footnote ⁶⁹ Second, data collection must be based on user consent and users must be provided with an easily accessible and understandable option to withdraw consent.Footnote ⁷⁰ Third, apps should minimize data collection.Footnote ⁷¹ With regard to sharing of data with third parties, user consent is required.Footnote ⁷² Furthermore, apps should not attempt to build a user profile on the basis of collected data.Footnote ⁷³ The Apple Developer Program License Agreement also states that app developers must take into account user privacy and comply with privacy legislation.Footnote ⁷⁴

Table 5.1 Health data protection in app store policies

Source: author’s analysis (2020)

Furthermore, as can be seen in Table 5.1, the guidelines contain explicit rules on health data processed by mHealth apps.Footnote ⁷⁵ First, apps may not use or disclose collected health data to third parties for the purpose of advertising, marketing, or other data-mining purposes.Footnote ⁷⁶ In addition, apps may not use health data for targeted or behavioral advertising.Footnote ⁷⁷ However, they may use or disclose health data for the purposes of improving health management and health research, but only with user permission.Footnote ⁷⁸ Second, app developers may not write inaccurate data into mHealth apps.Footnote ⁷⁹ Third, mHealth apps may not store health information in the iCloud.Footnote ⁸⁰

5.4.3.2 Google Play

Google Play’s review criteria are outlined in the Developer Distribution Agreement and Developer Program Policies.Footnote ⁸¹ The Agreement functions as a legally binding contract between the app developer and Google.Footnote ⁸² With regard to processing of personal data, the Agreement states that apps should comply with applicable data protection laws.Footnote ⁸³ More specifically, apps must inform users of what personal data is processed, provide a privacy notice, and offer adequate data protection. Furthermore, apps may only use personal data for the purposes the user has consented to.Footnote ⁸⁴ As shown in Table 5.1 above, the Agreement does not specifically mention mHealth apps or health data.

The Developer Program Policies provide more guidance on processing of personal (health) data. With regard to processing of personal data, the Policies state that apps that are intended to abuse or misuse personal data are strictly prohibited.Footnote ⁸⁵ Furthermore, apps must be transparent about the collection, use, and sharing of personal data.Footnote ⁸⁶ As to sensitive personal data, which probably also include health data, the Policies state that collection and use should be limited to purposes directly related to functionality of the app. Furthermore, an accessible privacy policy must be posted within the app itself. It must also disclose the type of parties the sensitive data is shared with.Footnote ⁸⁷ Moreover, the in-app disclosure must contain a request for users’ consent prior to data processing, requiring affirmative user action. These permission requests must clearly state the purposes for data processing or transfers. Furthermore, personal data may only be used for purposes that the user has consented to.Footnote ⁸⁸ The Policies do not contain explicit provisions on mHealth apps, except for a prohibition on false or misleading health claims.Footnote ⁸⁹

5.4.3.3 Case Study Analysis

The above examination of app stores’ guidelines shows that app stores are indeed concerned with privacy issues. However, it is questionable whether this leads to a higher level of protection of mHealth app users’ health privacy. Both app stores’ guidelines state that apps must comply with privacy legislation and integrate a privacy policy. However, the level of detail of the respective app stores’ privacy provisions differs significantly. While Apple App Store specifically recalls most of the GDPR’s data protection principles and data subjects’ rights, Google Play’s privacy guidelines are formulated in somewhat vague terms and do not mention data subjects’ rights. Therefore, Google Play’s guidelines do not offer app developers the needed guidance on how to protect personal data, specifically with regard to data subjects’ rights. This entails a strong risk that users’ rights will simply end up in the app’s privacy policy fine print and will not lead to better privacy protection in practice.

Furthermore, while Apple App Store has specific guidelines on health data processing, Google Play’s Policies only mention “sensitive personal data.” This lack of specific regulation of health data does not reflect the risky nature of this type of data and therefore does not increase awareness of the need for protection. Most notably, both guidelines miss a provision on “explicit consent” for health data processing, which is required for app developers under the GDPR. While both guidelines contain provisions on user consent, no distinction is made between “regular” and “explicit” consent and thus no clarification on how to obtain explicit consent is offered. This puts privacy at risk, as control over health data is not sufficiently protected.

Both guidelines state that noncompliant apps will be removed, but do not elaborate on the structure of the monitoring process. Therefore, actual enforcement of the guidelines faces risks of uncertainty and inconsistency, which does not ensure compliance with the GDPR. After all, app stores are likely facing the same capacity problems as data protection authorities, and it could take months before noncompliant apps are taken down. Compliance issues also come into play in the differences between the respective guidelines, as this leads to the risk of unequal standards of protection of iOS and Android users.

Taken together, it can be concluded that the current self-regulation practices, Google Play’s especially, do not live up to their potential and do not adequately ensure mHealth app users’ control over their health data. However, due to the central position of app stores, self-regulation by app stores may still contribute to a higher level of health data protection if certain amendments are made to the content and form of their policies. Recommendations on how to improve the policies are touched upon in the next section.Footnote ⁹⁰

5.5 Recommendations to Improve Current App Store Self-Regulation Practices

App stores have a powerful position in the mHealth app sector. By setting requirements for mHealth apps to be listed on and removed from their platforms they hold the most promising means to improve the level of health data protection of users. Their current self-regulation practices could be improved on multiple fronts. First, app stores could provide app developers with clearer guidelines on data processing obligations and data subjects’ rights. This should include stating all applicable obligations and rights under the GDPR and providing practical guidance on how to adequately implement this in apps. For example, app stores could issue technical guidelines on how to include consent withdrawal mechanisms in the apps. Translating privacy rights to technical measures will enhance adequate understanding and implementation by app developers.Footnote ⁹¹ Furthermore, app stores could make data subject rights and principles part of their contractual agreements with app developers to further strengthen compliance.Footnote ⁹²

Second, specific provisions on health data protection should be included, in order to point out its importance and increased privacy risks. These provisions should at least include the requirement to obtain explicit consent on health data processing and provide technical guidance on how to implement this.Footnote ⁹³ There should also be specific provisions on limiting sharing of health data with third parties and possible commercial use. Additionally, app stores can further strengthen users’ control by requiring apps to include user report tools on data protection infringement or provide for these tools in the app store itself.Footnote ⁹⁴ Furthermore, app stores should commit to raising awareness of the risks of health data processing. For instance, a standard text on the risks could be provided for in the guidelines, which app developers would be required to include in their privacy policies. App stores could educate users of the risks by adding “health data processing warnings” to the downloading environment.

Moreover, app stores could strengthen user protection if they would mainstream their policies and engage in a shared EU Code of Conduct under the GDPR.Footnote ⁹⁵ The GDPR codes are voluntary tools that set out specific data protection rules. They provide a detailed rulebook for controllers and processors in a specific sector. Bodies representing a sector – such as app stores – can create codes to aid GDPR compliance.Footnote ⁹⁶ Codes have to be approved by the European Data Protection Board (EDPB) and compliance will be monitored by an accredited, independent supervisor.Footnote ⁹⁷ Consequently, present self-regulation would turn into coregulation, and current guidelines would be replaced or supplemented by this GDPR code. App stores could make adherence to the code by app developers a requirement to offer apps on their platforms. This would have more effect than current self-regulation initiatives as preapproval of the code by the EDPB will give the code greater authority and the monitoring mechanism will lead to better compliance. Moreover, the unequal level of protection and risks of legal uncertainty and inconsistency would be minimized.Footnote ⁹⁸ For mHealth app users’ health privacy, a GDPR code will provide for more transparency regarding apps’ approaches to data processing.Footnote ⁹⁹ For example, the code would have to include specification of all applicable rights related to control over health data, explicit consent included.Footnote ¹⁰⁰

The preceding sections allow for the conclusion that app stores could positively impact GDPR compliance and thus strengthen mHealth users’ health privacy by engaging in a GDPR code with specific health data safeguards. While there is no guarantee that app stores will make these changes, there are compelling reasons for them to do so. Foremost, the increased legal certainty offers app stores a competitive advantage. It reduces the complexity of app developers’ entrepreneurial process, which may positively impact app stores’ businesses.Footnote ¹⁰¹ For app developers, a code would be beneficial because it could be used to demonstrate compliance with the GDPR.Footnote ¹⁰² Furthermore, app stores will benefit from good privacy practices by third-party apps because this will likely also enhance their own trustworthiness. In this regard, privacy can be seen as a positive marketing statement.Footnote ¹⁰³ Moreover, both Apple and Google were stakeholders in the European Commission’s attempt at a voluntary mHealth Privacy Code of Conduct, which shows their interest in such an initiative.

5.6 Conclusion: Improved App Store Self-Regulation Strengthens Health Privacy

Paradoxically, the wish to achieve self-empowerment by using mHealth apps leads to users surrendering power due to a lack of control over their health data. While the GDPR offers a solid solution for the protection of mHealth app users’ health data in theory, it lacks practical effectiveness. Self-regulation of third-party apps by app stores by means of review procedures could fill the regulatory gap and thereby contribute to the level of health data protection in the European Union. However, the performed case-studies show that current self-regulation does not fulfil this promise. None the less, given the platforms’ central and powerful position in the sector, complementary regulation of mHealth apps by app stores may still be the most promising means to improve the level of health data protection of mHealth app users. This conclusion sheds light on the heavily debated role of the European Union in regulating technological phenomena and related fundamental rights risks: in some cases, the sector itself is in a better position to regulate these risks and enforce legal compliance than independent supervisory authorities. This finding is in line with the European Union’s growing tendency to promote and support self-regulation structures to supplement EU legislation.

Despite the important role of app stores in achieving this, in the end, the ultimate responsibility for safeguarding users’ health privacy lies with the mHealth app developers and providers that process health data. mHealth apps should provide users with the adequate means to exercise privacy rights by ensuring concrete and effective opportunities to have control over decisions regarding health data processing. In this regard, effective possibilities for actual enforcement of self-regulation standards are of key importance. While app store self-regulation may steer mHealth app developers in the right direction by translating the GDPR’s privacy provisions into technical preapproval requirements, compliance with the relevant privacy provisions is also aided by increased awareness among both mHealth users, developers, and health data brokers as to the risks mHealth apps entail for individual fundamental rights. The European Union could play a central role in accomplishing this, in order to assist mHealth users to achieve the highly desired self-empowerment by bringing the GDPR to life in mHealth apps.

Stipulations on deidentification and scientific research in the European General Data Protection Regulation (GDPR) help research organizations to use personal data with fewer restrictions compared to data collection for other purposes. Under these exemptions, organizations may process specific data for a secondary purpose without consent. However, the definition and legal requirements of scientific research differ among EU Member States. Since the new EU Medical Device Regulations 2017/745 and 2017/746 require compliance with the GDPR, the failure to come to grips with these concepts creates misunderstandings and legal issues. We argue that this might result in obstacles for the use and review of input data for medical devices. This could not only lead to forum shopping but also safety risks. The authors discuss to what extent scientific research should benefit from the research exemption and deidentification rules under the GDPR. Furthermore, this chapter analyzes recently released guidelines and discussion papers to examine how input data is reviewed by EU regulators. Ultimately, we call for more harmonized rules to balance individuals’ rights and the safety of medical devices.

6.1 Introduction

Artificial intelligence (AI) and big data have a significant impact on society,Footnote ¹ as many aspects of our lives have become subject to data processing.Footnote ² This “datafication” has also led to a rapid transformation in the delivery of health care services.Footnote ³ The new generation of medical devices represents one example of technological advance that could substantially protect and improve public health.Footnote ⁴ Many of these rely heavily on data and AI algorithms to prevent, diagnose, treat, and monitor sources of epidemic diseases.Footnote ⁵

Though opening a world of new opportunities, rapid advances in AI medical devices have resulted in a number of highly complex dilemmas, tradeoffs, and uncertainties regarding the applicability and appropriateness of the current legal framework. Many of these legal and ethical issues relate to privacy and data protection. The European General Data Protection Regulation (GDPR)Footnote ⁶ is of particular importance in that respect. Focusing on the GDPR, the following chapter discusses the risk that AI medical device systems may run afoul of sufficiently informed consents of data subjects since they collect, process, and transfer sensitive personal data in unexpected ways without giving adequate prior notice, choices of participation, and other options.Footnote ⁷ At the same time, such data can be important to ensure the safety and effectiveness of such devices. Considering the consequential need for reasonably sound tradeoffs, we argue that current legal frameworks and definitions need to be harmonized and refined. We refer to the typical lifecycle in the collection and processing of health data via medical devices (Section 6.2) to highlight the challenges and legal risks at each phase. Section 6.3 examines the new EU regulations for Medical Devices (MDR)Footnote ⁸ and In Vitro Diagnostic Medical Devices (IVDR)Footnote ⁹ with a special focus on the MDR. In this section, we seek in particular to identify and iron out the missing links between the GDPR and the MDR. Section 6.4 discusses our main findings and summarizes recommendations. This provides the basis for our conclusions in Section 6.5.

6.2 Collection and Processing of Health Data Under the GDPR

Modern health care systems and medical devices collect and process vast amounts of data, which may enhance an individual’s health care experience directly and indirectly through scientific research and policy planning. Nevertheless, obtaining informed consentFootnote ¹⁰ or authorization from a large number of data subjects can be challenging and result in disproportionate cost and effort.Footnote ¹¹ For instance, the Italian government provided the health dataFootnote ¹² of 61 million Italian citizens to IBM Watson Health, without obtaining patient consent.Footnote ¹³ The agreement between the Italian government and IBM underlined that IBM alone would retain rights to the results of the research, which it could then license to third parties.Footnote ¹⁴ Instead of acquiring consent for the secondary processing, the most realistic option for privacy protection is providing the option to opt-out for the citizens, such as the national data opt-out systemFootnote ¹⁵ in England.Footnote ¹⁶

In general, the processing of sensitive data (e.g., health data) is prohibited under the GDPR. This can be a crucial issue in the case of AI-augmented medical devices since the sensitivity and specificity of an algorithm are only as good as the data that they are trained on. For instance, if an algorithm is only trained on the genetic material derived from European Caucasians, it may not provide accurate information that can be generalized to individuals of other groups. However, the GDPR enables the processing of sensitive data for public interest, public health, and scientific research purposes, if there are appropriate safeguards for the rights and freedom of individuals. While the GDPR does not fully specify what those safeguards are, it indicates that their purpose is to “ensure that technical and organizational measures are in place in order to ensure respect for the principle of data minimization.”Footnote ¹⁷ Such measures may include de-identification methods (for example, anonymization and pseudonymization) provided that the intended use of the data can still be fulfilled. However, differing requirements of national laws toward the application of these exemptions and de-identification methods often hinder the application of AI medical devices at the EU level. In Sections 6.2.1–6.2.3, we consider the most salient problems.

6.3 The EU Medical Device Regulation

To keep up with advances in science and technology, two new EU regulations on medical devices and in vitro diagnostic medical devices entered into force on May 25, 2017.Footnote ³⁶ They will progressively replace the existing directivesFootnote ³⁷ after a staggered transitional period.Footnote ³⁸

The MDR clarifies that data protection rules need to be applied when medical devices process personal data.Footnote ³⁹ Therefore, if a medical device regulated by the MDR collects personal data, it also falls under the GDPR. The MDR differentiates among three classes of medical devices, depending on their level of risk:

1. 1. Class I devices, posing low/medium risk (e.g., wheelchairs);

2. 2. Class IIa and IIb devices, representing medium/high-level risk (e.g., x-ray devices);

3. 3. Class III, high-risk devices (e.g., pacemakers).

In the case of low-risk level (Class I) medical devices, such as a smartwatch, privacy might often prevail over the secondary use of personal data to develop and improve these devices. In the case of high-risk level (Class III), the safety of medical devices might outweigh patient privacy. AI medical devices with a medium risk level (Class II), such as medical image processing software, may be considered to have at least a general level of public interest. However, developing high-risk devices does not mean that manufacturers could automatically process health data without consent. Careful consideration is necessary on a case-by-case basis with strong safeguards, under the oversight of authorities.

Medical devices in the European Union need to undergo a conformity assessment to demonstrate that they meet legal requirements. The conformity assessment usually involves an audit of the manufacturer’s quality system and, depending on the type of device, a review of technical documentation from the manufacturer on the safety and performance of the device.Footnote ⁴⁰ Manufacturers can place a CE (Conformité Européenne) mark on their medical device after passing the assessment. The EU Member States can designate accredited notified bodies to conduct conformity assessments. A notified body within the European Union is an entity designated by an EU competent authority to assess the conformity of medical devices before being placed on the market. Companies are free to choose the notified body they engage with.Footnote ⁴¹ There are more than fifty EU notified bodies in total that can certify according to Medical Device Directives. However, not all of these notified bodies can certify according to all categories of medical device products. When the authorities start to scrutinize the AI/ML medical device during the approval process, it is challenging to know clearly how the AI application and algorithms developed and evolved due to their opaque nature.Footnote ⁴² It is not clear how notified bodies can review the input data of AI medical devices. First, reviewing large and complex datasets requires special knowledge and technical expertise, which might be lacking or not at the same level within all the notified bodies of the European Union. Second, there are medical devices developed outside of the European Union. Reviewing the datasets used for developing them might trigger data protection and data transfer jurisdictional issues. The datasets might contain sensitive data of individuals from countries outside Europe, thus data sharing is challenging, posing a hurdle for part of the review process. For instance, the Health Insurance Portability and Accountability Act (HIPAA) and state regulations in the United States, and Japanese regulations on personal dataFootnote ⁴³ might not allow the sharing of sensitive data with the notified bodies in the EU Member States. Moreover, sharing anonymized data might not be sufficient to review input data thoroughly. Third, there is a great variety of data-processing software and methods among companies operating in different countries, which makes it extremely challenging to review these devices uniformly on the same level.

The European Medicines Agency and several notified bodies are already preparing for the change of AI medical devices. The European Medicines Agency and the Heads of Medicines Agencies (HMA) Big Data Task Force (BDTF)Footnote ⁴⁴ released two reportsFootnote ⁴⁵ recently for the European regulators and stakeholders to realize the potential of big data in terms of public health and innovation. Since the biggest issues in the European Union currently are the decentralization of health data and regulatory tasks, the reports focus on providing guidance and resources for data quality and discoverability to build up computing and analytical capacity. Thus, the most ambitious recommendation of the BDTF is the establishment of an EU platform: Data Analysis and Real World Interrogation Network (DARWIN) to access and analyze health care data from across the European Union. This platform would create a European network of databases with verified quality and strong data security. It is intended to be used to inform regulatory decision making with robust evidence from health care practice. The reports highlight the following actions for the European Union:

1. 1. Ensuring sufficient expertise and capacities within the European network (in all the notified bodies in the Member States), in order to ensure that AI medical devices can be assessed appropriately.

2. 2. Enable regulatory evaluation of clinical data submitted by drug manufacturers for approval where the data has been processed by AI algorithms or if part of the analysis, such as patient selection, involved AI methods.

3. 3. Enable regulatory use of AI in internal processes at authorities and notified bodies. For instance, applying Natural Language Processing of received texts, or reviewing image data submitted to support a clinical claim from a drug manufacturer.

4. 4. Approval of AI-based Health Apps in devices intended for clinical decision making.

The reports also clarify that the European Union cannot accept opaque algorithms performing without checks and balances. Algorithm code should be more transparent (feature selection, code, original data set) and available for targeted review by regulators and notified bodies. The report states that the outcomes and changes to algorithm use (safety and efficacy) need to be subject to post-marketing surveillance mechanisms, in a similar way as monitoring drug safety after marketing authorization. By way of comparison, the European Union’s approach for the assessment of medical devices is slightly different from the FDA’s in the United States. While the reports suggest that the European Union is still focusing on the transparency of AI applications, the FDA also pays special attention to the excellence and trustworthiness of the companies developing AI medical devices during the precertification process.Footnote ⁴⁶ Figure 6.1 below shows the flow of health data for developing AI medical devices in the European Union.

Figure 6.1. The processing of health data for developing AI medical devices

6.4 Discussion

The effective collection and processing of relevant health data is the first step to making AI medical devices that work properly. This is particularly relevant during the COVID-19Footnote ⁴⁷ outbreak as the foreseeable reuse of health data for scientific purposes leads to a rise in the number of organizations manufacturing AI medical devices.Footnote ⁴⁸ The US Sentinel system is a great example of monitoring the safety of medical devices and securely sharing and reusing the collected information.Footnote ⁴⁹ Our analysis suggests, however, that the processing and review of input data for medical devices, as well as the definition of specific data uses, are not fully harmonized in the European Union. This issue stems from the fact that the health care systems and scientific research are mainly regulated by the EU Member States, resulting in diverse legal environments and barriers for processing health data. Thus, the GDPR and Medical Device Regulation have not reached a sufficient level of harmonization in this field. This may result in unnecessary legal risks in the development and deployment of AI medical devices, which is crucial in the case of the “update problem.”Footnote ⁵⁰ Therefore, harmonized rules on the collection and processing of health data, as well as review systems and processes of medical devices, would be necessary in the EU Member States.

The “update problem” is still not sufficiently addressed and little work has thoroughly examined how AI medical devices are developed and built from the perspectives of public interest and data protection law. To build these devices, data-intensive research is necessary. However, at what cost? Strong privacy protection may hinder the development, effectiveness, and precision of AI products and services. Globally, there is a drive to create competitive pharmaceutical and health care industries. As a result, the developers of AI medical devices and services have enjoyed a privileged position since they have been able to further use health data with less restrictions, and sometimes without adequate consent.Footnote ⁵¹ On the one hand, this could save lives and minimize treatment costs.Footnote ⁵² An increased precision due to better and more data, might even help to identify, monitor, and correct potential risks for bias in the data. On the other hand, this situation might lead to the further use of sensitive data with less control and increasing risks for privacy breaches.

To address this dilemma and achieve reasonable tradeoffs, we suggest the following measures to advance the assessment of the safety and efficacy of AI medical devices in the European Union. First, we believe that the expected level of public interest in the case of the secondary use of health data for developing AI medical devices must be clarified for different categories of medical devices, considering both the intended and unintended use scenarios.Footnote ⁵³ Second, we propose to regulate the definition and requirements of scientific research on the EU level to harmonize the secondary use of health data. This would be crucial for providing a sufficient amount of quality data for machine learning in the case of AI medical devices. Moreover, collecting personal data and processing it for a purpose with public interest should not result in a product or service that negatively affects the data subject’s rights. Third, we think that more guidance would be necessary on the safeguards and expected level of de-identification on health data, without overconfidently relying on them. Fourth, we call upon the EMA and notified bodies to be properly prepared for the review of (large) datasets since it is the foundation of AI medical devices. While opening and assessing opaque algorithms is challenging for regulators, we believe that a reasonable level of transparency should be required to allow for sufficient regulatory review of medical device systems.Footnote ⁵⁴ This does not necessarily imply that every single computational step must be traceable.Footnote ⁵⁵ For instance, some algorithms could still be utilized to construct a transparent and trusted AI system “as long as the assumptions and limitations, operational protocols, data properties, and output decisions can be systematically examined and validated.”Footnote ⁵⁶ Fifth, we recommend harmonizing the conformity assessment of notified bodies to provide safety, allow for European-wide reports on unwanted incidents, and avoid forum shopping. Sixth, and finally, we propose to develop special regulation and oversight for AI research to allow for a better coordination and compliance assessment in view of the great variety of separate regulations concerning data protection, health care, and medical research.

6.5 Conclusion

Harnessing the full benefits of AI-driven medical devices offers many opportunities, in particular in health crisis situations, such as the ongoing COVID-19 pandemic. However, many legal risks and lingering questions remain unsolved. The European Union does not yet have the means to fully exploit the benefits of this data due to heterogeneous health care systems with different content, terminologies, and structures.Footnote ⁵⁷ In addition, the European Union currently has no pan-European data network and is lagging behind other regions in delivering answers for health care-related regulatory questions.Footnote ⁵⁸ Although the GDPR and Medical Device Regulations aim to address some of these challenges by harmonizing the processing of data and risk assessment of AI medical devices in the European Union, these areas still remain diversified. To enhance the performance and safety of medical devices, it will be important to improve the dialogue between data protection authorities, ethical review boards, notified bodies, and medicine agencies. The proposed recommendations discussed in this chapter attempt to enhance this dialogue for a better understanding and alignment between the medical device sector, regulators, public research programs, and data protection standards.Footnote ⁵⁹ This could form the basis for a legal debate on the circumstances under which access by researchers to health data by private companies can be justified based on public interest and research exemptions.Footnote ⁶⁰ Considering the increasing importance of public-private partnerships and AI-driven medical devices proactive initiatives to that effect appear more important than ever.Footnote ⁶¹ The ongoing implementation of the EU strategies concerning AI, Data and medical innovation plays an important role in that regard. This has not only resulted in the evolving formation of the European Health Data Space,Footnote ⁶² but also in the adoption of a new EU Data Governance ActFootnote ⁶³ and the proposal of an AI Regulation,Footnote ⁶⁴ which provides for regulatory sandboxes for low-risk devices. It is the hope of the authors that these developments will improve the current situation.

7.1 Introduction

This chapter explores the efforts made by regulators in Europe to develop standards concerning the explainability of artificial intelligence (AI) systems used in wearables. Diagnostic health devices such as fitness trackers, smart health watches, ECG and blood pressure monitors, and other biosensors are becoming more user-friendly, computationally powerful, and integrated into society. They are used to track the spread of infectious diseases, monitor health remotely, and predict the onset of illness before symptoms arise. At their foundation are complex neural networks making predictions from a plethora of data. While their use has been growing, the COVID-19 pandemic will likely accelerate that rise as governments grapple with monitoring and containing the spread of infectious diseases. One key challenge for scientists and regulators is to ensure that predictions are understood and explainable to legislators, policymakers, doctors, and patients to ensure informed decision making.

Two arguments are made in this chapter. First, regulators in Europe should develop minimum standards on explainability. Second, those standards should be informed by the computer science underlying the technology to identify the limitations of explainability. Recently, several reports have been published by the European Commission and the National Health Service (NHS) in the United Kingdom (UK). This chapter examines the operation of AI networks alongside those guidelines finding that, while they make good progress, they will ultimately be limited by the available technology. Further, despite much being said about the opaqueness of neural networks, human beings have significant oversight over them. The finger of liability will remain pointed toward humans, but the technology should advance to help them decipher networks intelligibly. As computer scientists enhance the technology, lawmakers should set minimum standards that are leveled-up progressively as the technology improves.

7.2 Wearables in Health Care

Wearables are devices designed to stay on the body and collect health data such as heart rate, temperature, and oxygenation levels.Footnote ¹ Smartwatches, chest belts, clothing, ingestible electronics, and many others are converging with the internet-of-things (IoT) and cloud computing to become powerful diagnostics for more than seventy conditions.Footnote ² The technology has advanced rapidly, with GPUs, CPUs, and increasing RAM being adopted, opening possibilities for deep learning.Footnote ³ Despite these advances, adoption remains low in the health care setting overall, being in the early stages of the Gartner Hype Cycle.Footnote ⁴ Nevertheless, the trend is moving toward greater adoption. The COVID-19 pandemic in 2020 may accelerate the development of telemedicine, monitoring patients remotely, predicting disease, and mapping the spread of illnesses.Footnote ⁵ An example of the technology’s use can be seen in England under an NHS pilot program where patients were fitted with a Wi-Fi-enabled armband. This monitored vital signs remotely, such as respiratory rates, oxygen levels, pulse, blood pressure, and body temperature. AI was able to monitor patients in real-time, leading to a reduction in readmission rates, home visits, and emergency admissions. Algorithms were able to identify warning signs in the data, alerting the patient and caregiver.Footnote ⁶ This example aligns with a broader trend of adoption.Footnote ⁷ The largest NHS hospital trusts have signed multiyear deals to increase the number of wearables used for remote digital health assessments and monitoring.Footnote ⁸ This allows doctors to monitor their patients away from the hospital setting, both before and after medical procedures.

7.3 Human Neural Networks?

Underpinning such technologies is complex computer science. A device can predict illness, but it cannot explain why it made a prediction, which raises several legal issues. A targeted legal strategy cannot be realistically devised without understanding the technology driving it. Lawyers are unlikely to become master coders or algorithm developers, but they can have a reasonable understanding of where most efforts are needed. By examining what drives AI, more technically aware discussions can be generated in the legal sphere.

AI is an umbrella term used for different forms of “machine learning.” This includes “supervised” and “unsupervised” learning, which entails making predictions by analyzing data.Footnote ⁹ The former involves predefined labels used to assign the data to relevant groups, whereas the latter searches for common features in the data to classify it. A subset of “machine learning” is “deep learning,” which consists of artificial neural networks (ANNs) used for autonomous learning. There are various architectures, but the primary example here is of a deep supervised learning network with labeled data.Footnote ¹⁰ Such networks are the most numerous in operation and can illustrate how deep learning works and where the legal issues may arise.

Figure 7.1 depicts a neural network. An ANN begins with an “input layer” on the left.Footnote ¹¹ The example is an image of a cerebellum, which the ANN converts into many “neurons” (represented by the grid of squares). Each neuron is assigned a value (for black and white images) called the “activation” number. The number could, for example, be higher for brighter neurons (where the cerebellum is) and lower for darker neurons (outside the cerebellum). Every neuron is represented in the input layer. The example shows four neurons, but the ANN will have as many neurons as there are pixels in the image.

Figure 7.1: Example of an ANN

The example also shows two hidden layers in the middle, but there will be numerous in practice. In reality, the layers are not hidden to the programmer, but their numerousness makes the ANN virtually undecipherable – much like a human brain. The activations in the input layer (the black circles) will influence what is activated in the first hidden layer (the light grey circles) which will influence further activations. At the end is the output layer with several choices (Cerebellum, frontal lobe, or pituitary gland). The ANN gives the highest value to its choice (here, Cerebellum, the dark grey circle). Between the neurons are connections called “weights” (represented as lines) whose values are determined by a mathematical function. The sum of the weights in one layer determines which neurons are activated in the next layer. For example, the sum of the weights in the input layer has activated the first, third, and sixth neurons in the first hidden layer. Humans can also influence those activations by adding a “bias” to alter the value required for an activation.

In practice, the numerousness and complexity of the connections create an undecipherable matrix of distinct weights and biases. The choice of output cannot be explained, which is where the term “black box” algorithms arises. Despite this, humans play a central role. They give the network training data consisting of many prelabeled images of cerebellums, pituitary glands, and frontal lobes. The network is trained on that data. The process of data moving from left to right is called “forward propagation,” and the weights between the neurons are initially random, which produces random outputs. To correct the ANN, a validation data set is used with labels indicating the correct answer. In response, the ANN works backward (backpropagation) from the output layer, through the hidden layers to the input layer, adjusting the weights and biases as it moves along. The network becomes more accurate through repetition.

Deep supervised learning networks are well suited to diagnostics. Inputs, such as scans, are in a standardized format, which is a useful source of structured input data, training, and validation. The process becomes highly accurate because of the numerous hidden layers and connections. However, the black-box nature of an ANN should not be overstated. Humans have significant involvement, labeling data, giving it to the network, providing feedback, computing biases, interpreting data, and putting it into practice.

Most studies of wearable data have focused on supervised learning architectures.Footnote ¹² However, data derived from wearables is often unlabeled and unstructured, which benefits unsupervised learning that identifies patterns to make predictions.Footnote ¹³ These techniques raise more complex legal issues because humans are less involved. They are a work in progress at present, but they will become more prominent.Footnote ¹⁴ Indeed, there are increasing studies that analyze wearable data using unsupervised ANNs. One study proposes an unsupervised ANN to classify and recognize human activities.Footnote ¹⁵ It was able to recognize human activities through a combination of data obtained from magnetometers and accelerometers in wearables.Footnote ¹⁶ Another study analyzed data from 3D accelerometers on the wrist and hip, a skin temperature sensor, an ECG electrode, a respiratory effort sensor, and an oximeter amongst others.Footnote ¹⁷ The unsupervised network yielded 89 percent accuracy in detecting human activities (walking, cycling, playing football, or lying down).Footnote ¹⁸ Another approach has analyzed gestures to detect daily patterns that might indicate when older persons require assistance.Footnote ¹⁹

These are a small sample of studies increasingly utilizing unsupervised learning architectures in wearables. The underlying point is that such technologies are being used more frequently, which raises legal issues surrounding explainability. At the same time, humans must still train the networks. The processes within the hidden layers are complex to decipher, but humans pretrain and oversee the process.Footnote ²⁰ Consequently, while the legal implications must be deciphered, the autonomous nature of these systems should not be overstated.

7.4 Explainability and the Law

Explainability refers to ex-ante explanations of an ANN’s functionality, and ex-ante or ex-post explanations of the decisions taken, such as the rationale, the weighting, and the rules.Footnote ²¹ It requires that humans can understand and trace decisions.Footnote ²² However, the regulation of an ANN is as complex as its operation, which is problematic in health care. While shortcomings in explainability of AI systems will not necessarily lead to liability, it is one important factor. The key point of interaction between explainability and liability is at the fact finding or evidence stage. It may be difficult to factually prove the harm caused by a neural network because one cannot explain how a certain input resulted in a specific output, and that a deficiency resulted due to that process.Footnote ²³ The circumstances in which explainability becomes important in liability analyses are broad.

Problems may arise where harm is caused to a patient because the doctor did not follow the appropriate standard of care.Footnote ²⁴ Price notes how, in the current climate, the risk of liability for doctors relying on AI recommendations is significant because the practice is “too innovative to have many adherents.”Footnote ²⁵ Algorithm developers might be liable as well. However, in Europe, the laws are incoherent. The Product Liability Directive (1985/374/EEC) holds manufacturers liable for defective products. Proving that an ANN was defective requires technical expertise, but even experts cannot explain the hidden layers of a network.Footnote ²⁶ There is also the problem of ANNs being autonomous and changing. While the European Union has taken a strict approach on manufacturers being liable for the safety of products throughout their lifecycle, it acknowledges that the Directive should be revisited to account for products that may change or be altered thereby leaving manufacturers in legal limbo.Footnote ²⁷

There are also medical device regulations, but half of developers in the United Kingdom do not intend to seek CE Mark classification because it is uncertain whether algorithms can be classified as medical devices.Footnote ²⁸ There are medical device conformity assessments, but there are no standards for validating algorithms nor regulating adaptive algorithms.Footnote ²⁹ Also, while manufacturers must carry out risk assessments before products are placed on the market, they quickly become outdated because ANNs continuously evolve.Footnote ³⁰ For doctors, they may be negligent when advising patients based on AI recommendations that later cause harm. There are also questions about whether a person can consent to flawed medical advice from an ANN. These challenges are recognized in Europe where several reports were published in 2019 and 2020.

7.5 Guidelines

In the European Union, there are Guidelines, a White Paper, and an Assessment List regarding AI geared toward developing a future regulatory framework. On the guidelines, the EU Commission set up an “independent group” which released the Ethics Guidelines for Trustworthy AI in 2019 seen as a “starting point” for discussions about AI premised on respect for human autonomy, prevention of harm, fairness, and explicability.Footnote ³¹ The White Paper (which builds upon the Guidelines) was published in 2020 and outlines an approach to AI based on “excellence and trust.”Footnote ³² It notes that while AI can improve prevention and diagnosis in health care, black box algorithms create difficulties of legal enforcement.Footnote ³³ The Assessment List for Trustworthy Artificial Intelligence (ALTAI) is a self-assessment list published in July 2020.Footnote ³⁴

7.6 Conclusion

The foundations for setting minimum standards concerning explainability have now been established. However, there are shortcomings in AI-enhanced technology, such as wearables, which undermine informed decision-making for doctors, patients, and others. This is problematic because wearables will become ever more heavily relied upon for a wide variety of medical purposes. Further, doctors and patients ought to know why neural networks produce specific outputs. In time, scientists will develop more sophisticated models of explainability. Regulators, doctors, patients, and scientists should work together to ensure that those advances filter into the relevant guidelines as they develop – a gradual and flexible “leveling up” that keeps apace with the science. In this manner, lawyers and policymakers should take responsibility for better understanding the technology underlying those systems. As such, they should become more familiar with and knowledgeable about neural networks, the use of input data, training data, how data propagates, and how “learning” occurs. This will be key for creating standards that are relevant, sound, and justified. While laws and guidelines in the future will indicate the path to be pursued, some matters will take concerted interdisciplinary efforts to resolve.

The functionality of digital health technologies (DHTs), such as wearable devices and virtual assistants, is increasingly being used to make personal health and medical decisions. If manufacturers of DHTs are able to avoid regulation of their products as medical devices by marketing them as “lifestyle and well-being” devices, the potential harm caused to consumers who use DHTs beyond the manufacturer’s intended purpose will not be adequately addressed. This chapter argues the need for a framework to reclassify and regulate DHTs based on evidence of actual use.

This chapter focuses on how the classification rules and postmarket surveillance system provisions of the EU Medical Devices Regulation (MDR) need to anticipate and address the actual use of DHTs. To date, courts and regulators have not been consistent on the circumstances under which manufacturers are held responsible for known or encouraged “misuse” of their products. By defining a postmarket surveillance requirement for manufacturers of DHTs to acquire knowledge of the actual use of their products, informed regulatory decisions based on impact can be made. Actual use information can also help establish that the risk caused by a reasonably foreseeable misuse of DHTs was known to the manufacturer in a liability claim should consumers suffer harm from relying on statements or representations, made or implied, when using DHTs to self-manage their health. Moreover, if data generated by DHTs will be used to make regulatory decisions under the 2020 revision of the Good Clinical Practice, the MDR must proactively regulate technologies that have an actual impact on public health.

8.1 Introduction

The functionality of digital health technologies (DHTs), such as wearable devices and virtual assistants, is being promoted as essential tools to empower people to take control and responsibility of their own health and wellness. Examples of wearable devices referred to in this chapter include devices that track health and fitness-related data such as heart rate, activity level, sleep cycles, caloric intake, and the like. An example of a virtual assistant includes Amazon Echo with its technology to analyze the user’s voice to detect and determine “physical or emotional abnormality” and provide targeted content related to a particular medicine sold by a particular retailer to address the detected problem.Footnote ¹

There is significant literature on the potential benefits of DHTs in reducing costs and the burden on the health care system, for example, by providing patients with options to self-manage health from home.Footnote ² DHTs are also attributed with the ability to help detect early warning signs of potentially serious health conditions, alerting users to irregularities, leading to investigations to detect illnesses that may otherwise have gone unnoticed with potentially tragic consequences.Footnote ³ While health care providers generally recognize DHTs as useful tools, there is also evidence that these very same technologies are increasingly being used by the public in a manner that potentially increases health care costs in the long run.Footnote ⁴ One study reported an increase in physician-“digitalchondriac” interaction where patients demand immediate attention from medical professionals based on troubling key health indicators detected by wearable devices, which may or may not be accurate.Footnote ⁵ On the other end of the spectrum, some patients elect to by-pass traditional health service structures and formalities and take medical and health decisions into their own hands at great risk to themselves instead of consulting a medical professional. Some doctors recount stories of patients taking prescription medication in response to an irregular reading from their wearable device without understanding or inquiring about the risk of taking a higher than recommended dosage of medication.Footnote ⁶

DHTs have been setting off alarms for users to take note and control of their health, but there are also data and reports that suggest many of those alarms turn out to be false. As consumers increasingly engage in self-monitoring and self-care with the help of DHTs, health practitioners need to respond to patient confusion and anxiety created by data generated by DHTs.Footnote ⁷ Because the accuracy of DHTs can vary greatly with a margin of error as high as 25 percent across different devices,Footnote ⁸ health practitioners have the added burden of treating patients without medical training who nevertheless seek medical intervention for self-diagnosed illnesses derived from the internet by attributing symptoms detected by unreliable DHTs. The next section will examine the applicable regulatory framework in the European Union to better understand what oversight mechanisms are available to ensure the safety and efficacy of DHTs in view of evidence of how consumers actually use these devices to make personal health and medical decisions.

8.2 The EU Medical Devices Regulation

Medical devices are recognized as essential to the health and wellbeing of European citizens and legislation is essential to ensure the safety and efficacy of medical devices for the protection of public health.Footnote ⁹ The new EU Medical Devices Regulation (MDR)Footnote ¹⁰ will come into force in May 2021, replacing the existing Medical Devices Directive (MDD).Footnote ¹¹ The MDR attempts to modernize the MDD by introducing new concepts, definitions, and rules that may be applicable to DHTs. For example, the definition of a medical device in the MDR includes new qualifying language “prediction and prognosis of disease.”Footnote ¹² In principle, this definition should capture the collection, monitoring, processing, and evaluation of physiological data associated with DHTs since they claim to be capable of potentially predicting or providing a prognosis of potential future disease identification from the data collected.

However, the MDR clearly states “software for general purposes, even when used in a health care setting, or software intended for lifestyle and well-being purposes is not considered a medical device.”Footnote ¹³ It is the intended purpose, as opposed to the technological features and capabilities of a device that determines whether a DHT will be regulated under the MDR. Intended purpose is defined as “the use for which a device is intended according to the data supplied by the manufacturer on the label, in the instructions for use or in promotional or sales materials or statements and as specified by the manufacturer in the clinical evaluation.”Footnote ¹⁴ Because the regulatory framework can be challenging for many startups with limited resources, the ability to market the intended use of DHTs as health and wellness devices as opposed to a medical device that requires a higher degree of regulatory compliance is a pragmatic business decision, but at what potential cost to public health? Even for larger companies, Apple CEO Tim Cook stated that the regulatory process and degree of adherence required for the Apple Watch would prevent Apple from continuing to innovate and remain competitive in the medical product marketplace.Footnote ¹⁵

In brief, the classification rules and procedures under the MDR are based on the potential risk a particular device poses to the user, having regard to the technical design and manufacture of the device.Footnote ¹⁶ Currently, a significant number of wearables are classified as Class I noninvasive devices.Footnote ¹⁷ However, under the MDR, the introduction of a more nuanced classification system and a more involved assessment procedure may increase the regulatory scrutiny of DHTs. For example, software intended to monitor physiological processes will be considered Class IIa, and software intended to monitor vital physiological parameters would be classified as Class IIb.Footnote ¹⁸ As the classification level increases, the applicable safety rules and conformity assessments also become stricter. However, the increased classification level applies only to “active devices intended for diagnosis and monitoring,” which again does not include DHTs that manufacturers self-declare as intended for “lifestyle and well-being purposes.”Footnote ¹⁹

While efforts continue to focus on what types of innovations fall into the definition of a medical device and within which classification level, this chapter focuses on how the public actually uses and interfaces with these products, regardless of the regulatory classification. There is increasing evidence to suggest that consumers use DHTs to help with medical care decision making despite the manufacturer’s stated intent.Footnote ²⁰ Although the MDR attempts to establish a contemporary legislative framework to ensure better protection of public health and safety, the point where DHTs “not intended for medical purposes” and the use of pharmaceuticals intersect, raises a myriad of legal, ethical, and policy implications. Pharmaceuticals, which are highly regulated, are reportedly being used by the public to make self-determined medication decisions based solely on information derived from DHTs,Footnote ²¹ which are not as well regulated under the MDR. Understandably, the regulatory framework should focus on technologies that pose the greatest risk to patients and their data security. However, as discussed in greater detail below, “misuse” of lower-risk devices beyond the manufacturer’s intended use could raise significant public health risks not previously contemplated. Some DHTs proclaim medical benefits but disclaim that the device is intended for health and wellbeing purposes only.Footnote ²² If manufacturers of DHTs are able to avoid the higher regulatory burden associated with having their products classified as medical devices, the question is what legal framework exists to hold manufacturers responsible for known “misuse” of their products and whether consumer protection laws will provide adequate redress to the potential harm caused to consumers who nevertheless use DHTs beyond the manufacturer’s stated purpose to make personal health and medical decisions.

Although the vast majority of DHTs pose a very low risk of harm to consumers, there is increasing evidence that many of these devices are not as accurate as described or fail to work at all.Footnote ²³ Without an oversight mechanism to detect and respond to the health risk arising from the actual use of low-risk devices beyond the manufacturer’s stated intended use means many consumers could be adversely affected throughout the lifecycle of the product without recourse. To bring medical devices onto the EU market, the CE approval process is required to verify that a device meets all the regulatory requirements under the MDR. However, for Class I devices, the manufacturer is responsible for self-certification for the CE marking process.Footnote ²⁴ Policy proposals related to permitting lower-risk devices to be brought to market more efficiently on the condition that postmarketing data on safety and effectiveness is collected as part of mandatory renewal or reevaluation process has previously been considered.Footnote ²⁵ While postmarket safety and efficacy data may be used to assess whether a DHT continues to qualify for a low-risk classification level, it falls short of providing an evidence-based reason to reclassify a DHT based on the potential risk arising from how consumers actually use these devices, regardless of their safety and efficacy profile.

8.3 Intended versus Actual Use

While DHTs are intended to modify behavior to improve health and wellness, an unintended consequence of the functionalities of these devices is that consumers are increasingly using them to make personal health and medical decisions.Footnote ²⁶ DHTs offer to collect and monitor physiological data that medical devices do and can be used in combination with apps to interpret such data to provide medical advice. The line between DHTs and medical devices therefore increasingly becomes blurred, particularly to the consumer, as new devices and new improvements of well-established wearables allow the monitoring and assessing of a range of medical risk factors.Footnote ²⁷ According to a recent survey, 71 percent of physicians say they use digital health data to inform their own personal health decisions,Footnote ²⁸ and another survey found that consumers are increasingly using wearables to make critical health care decisions instead of monitoring physical activity and lifestyle.Footnote ²⁹

However, the majority of manufacturers provide no empirical evidence to support the effectiveness of their products, in part, because the applicable regulation does not require them to do so.Footnote ³⁰ Recent reports indicate an increase in incidents of wearables sending otherwise healthy people to doctors due to incorrect and inaccurate readings.Footnote ³¹ Meanwhile, popular consumer devices continue to insist that their product, unless otherwise specified, is not a medical device and should not be held to such a standardFootnote ³² despite marketing their devices as being able to “help improve wellness, disease management and prevention.”Footnote ³³ Experts agree that wearable devices cannot be expected to give medical grade accuracy, nor should consumers demand such high scientific quality from DHTs. However, as users become increasingly more reliant on DHTs that may provide a false sense of security on one spectrum to misguided self-diagnosis on the other, the need for legal solutions and regulatory oversight has been called for to address issues of consumer harm and accountability.Footnote ³⁴

To avoid liability, manufacturers typically rely on disclaimers even though it is known that users tend to ignore such information.Footnote ³⁵ Legal measures are available to address direct-to-consumer marketing practices relating to fraudulent or misleading advertising. For example, in the 2018 dispute against Fitbit’s Purepulse heart rate tracker for being grossly inaccurate, alleging false advertising, common law fraud, and breach of implied warranty among other claims, the court allowed the class action to proceed, stating that “[g]iven the magnitude of the aberrant heart rate readings and multiple allegations that the devices under-report heart rate, [plaintiff] has plausibly alleged an ‘unreasonable safety hazard’ that may arise when users rely on Fitbit heart rate readings during exercise.”Footnote ³⁶ Similarly, the FDA also monitors medical product communications to make sure they are consistent with the product’s regulatory authorization.Footnote ³⁷ However, the FDA has stated that it will only oversee “medical devices whose functionality could pose a risk to patient safety if the [device] were to not function as intended.”Footnote ³⁸ More specifically, the FDA stated that it does not intend to regulate general wellness products.Footnote ³⁹ In other words, if the manufacturer’s stated intention is for DHTs to be used for “life-style and well-being” purposes, then any use by the public outside the intended use falls outside the scope of the FDA regulatory framework.

Courts and policy makers seem to support the consumer demand for reliability of DHTs.Footnote ⁴⁰ However, courts have not always been consistent on the circumstances under which manufacturers are held responsible for known or encouraged “misuse” of their products.Footnote ⁴¹ Nor have they provided clear or predictable guidance on what constitutes reasonably foreseeable misuse that manufacturers should have known that their product is being used for a purpose for which it is not intended.Footnote ⁴² Because of the legal duty to anticipate and take precautions against unintended but reasonably foreseeable use of products, manufacturers have always been expected to be apprised of the potential “misuses” of their products. Generally, under the reasonable foreseeability standard, manufacturers can be held liable for injuries caused by a product even if the consumer fails to use the product as intended, but the consumer must show the actual use rendered the product defective, which was known or should have been known to the manufacturer.Footnote ⁴³ In practice, it can be difficult to determine what unintended uses and what harms arising from such unintended uses are reasonably foreseeable, with some responsibility of prudence being placed on the consumer.Footnote ⁴⁴ It would likely be difficult to establish legal liability under the consumer protection framework for harms arising from the known use of DHTs by consumers who rely on these devices to make medical and health decisions instead of using them for health and wellness purposes only.

There is an opportunity for the MDR to implement a reclassification framework based on evidence of actual use to provide better regulatory oversight, especially as the functionality of DHTs continues to expand their focus toward health care by detecting and measuring an increasing number of physiological parameters associated with health conditions. Manufacturers should not be able to circumvent and avoid higher regulatory burdens by being willfully blind to the increasing evidence of consumers who feel empowered by promotional statements or representations made or implied that they can use DHTs as a means to take control of and responsibility for their own health and wellness.Footnote ⁴⁵ However, according to the Court of Justice of the European Union “[where] a product is not conceived by its manufacturer to be used for medical purposes, its certification as a medical device cannot be required.”Footnote ⁴⁶ In other words, how a device is actually used should have no bearing on how the device is regulated if the stated intention of the manufacturer is that the product is not a medical device. Nevertheless, the postmarket surveillance (PMS) requirement under the MDR may be used to require manufacturers to proactively understand how their products are being used by the public to better align regulatory purposes with public health objectives.

8.4 Postmarket Surveillance (PMS) under the MDR

Under the MDR, the PMS system is a proactive procedure where manufacturers act in cooperation with other economic actors to collect, review, and report on experiences of devices on the market with the aim of identifying any need for corrective or preventative measures.Footnote ⁴⁷ One of the new features of the MDR is the concept of a PMS plan that requires manufacturers to define the process of collecting, assessing, and investigating incidents and market-related experiences reported by health care professionals, patients, and users on events related to a medical device.Footnote ⁴⁸ According to the MDR, the PMS plan “shall be suited to actively and systematically gathering, recording and analysing relevant data on the quality, performance and safety of a device throughout its entire lifetime, and to drawing the necessary conclusions and to determining, implementing and monitoring any preventive and corrective actions.”Footnote ⁴⁹ Because of a growing demand to adopt a more proactive as opposed to the current passive reactive approach to PMS,Footnote ⁵⁰ the implementation of the PMS plan under the MDR may be an avenue to address the concerns associated with the actual use of DHTs beyond the manufacturer’s stated intended use. The ability to identify risks and take corrective measures in a timely manner is vital for any regulatory framework. Clear guidance on the implementation of the PMS plan is essential to improve the delivery of health care to consumers through the help of DHTs.

Arguably, the PMS plan can be interpreted to include an obligation to collect postmarketing data on consumer use of DHTs as part of a mandatory reevaluation process to assess the appropriate classification level and regulatory compliance the DHT must adhere to in order to continue to remain on the market. By defining a PMS requirement for manufacturers of DHTs to acquire knowledge of actual use of their products in order to maintain their lower classification status, informed regulatory decisions based on data and evidence can be made. Actual use information can also help establish that the risk caused by a reasonably foreseeable misuse of DHTs was known to the manufacturer in a liability claim should consumers suffer harm from relying on statements or representations, made or implied, when using DHTs to self-manage their health.

However, the MDR is not particularly clear on the extent of the PMS obligation, stating that the PMS plan should be “proportionate to the risk class and appropriate for the type of device.”Footnote ⁵¹ For Class I devices, a PMS report based on the PMS plan shall be “updated when necessary and made available to the competent authority upon request,”Footnote ⁵² and there is no clarification of how often information should be collected. The elements and type of information that shall be collected for the PMS plan include adverse events, data on nonserious incidents and undesirable side effects, safety updates, trend reporting, relevant specialist or technical literature, and feedback and complaints from users.Footnote ⁵³ Although information about actual use is not specifically mentioned, it may be captured under trend reporting, which is intended to include incidents “that could have a significant impact on the benefit analysis … which may lead to risks to the health or safety of patients, users or other persons that are unacceptable when weighed against the intended benefits.”Footnote ⁵⁴ The reclassification of devices is contemplated for reasons of public health based on new scientific evidence or based on information that becomes available in the course of vigilance and market surveillance.Footnote ⁵⁵ Evidence of actual use collected as part of the PMS plan can therefore be used as grounds for reclassification to anticipate and address the actual use of DHTs. However, even with the ability to reclassify, the classification rules and definition of noninvasive versus active devices based on intended use renders this process a vicious cycle. Furthermore, the reclassification of devices based on PMS requires a request to the commission by a Member State and consultation with the Medical Device Coordination Group,Footnote ⁵⁶ making the process bureaucratically cumbersome and unlikely to be used in practice. Without clearer implementation guidelines and a better alignment of the PMS objectives with the classification rules, the PMS plan could become a toothless oversight mechanism.

PMS allows for continuous vigilance, not only to ensure quality, safety, and efficacy of the devices but to ensure the appropriate level of regulatory adherence based on how a device is actually being used. With the rapid proliferation and advancement of DHTs, it will require a collaborative effort between manufacturers, regulators, health care providers, and consumers to strike the right balance between the appropriate regulatory burden and the benefit that DHTs promise to bring to the public health system. DHTs could prove to be a good secondary diagnostic tool with its ability to constantly monitor and collect data and provide detailed longitudinal data to monitor progress and understand patterns.Footnote ⁵⁷ A deeper understanding of patients through their health data is one of the keys to improving health, especially in managing chronic conditions that are primarily driven by leading an unhealthy lifestyle.Footnote ⁵⁸ As the International Council for Harmonization of Technical Requirements for Pharmaceuticals for Human Use continues to consider revisions to the Good Clinical Practice (GCP) to enable the use of real-world evidence, such as patient data derived from or influenced by consumer use of DHTs, reliable oversight and regulation of DHTs become even more pressing to ensure the data are reliable and appropriately collected and interpreted to serve as evidence for informing future regulatory decisions. The underlying presumption that data derived from DHTs can be transformed into meaningful real-world evidence to be used for the intent contemplated by the GCP is that the data is reliable and that there is a relationship between the use of the DHT and the clinical relevance of the data.Footnote ⁵⁹ A PMS system that is aligned with the classification rules to adapt regulatory oversight of DHTs based on actual use and actual impact on consumer health will better support the use of real-world evidence or data derived from DHTs for the purposes of the GCP. The interpretation of the PMS plan to include a proactive requirement on manufacturers to self-report and be informed of not only the safety and efficacy of their products but also how their products are being used will help users and regulators make more informed decisions. This requirement also aligns with the EU responsible research and innovation policy objectives to ensure the innovation process is interactive, transparent, and responsive to public interests and concerns.Footnote ⁶⁰ The PMS plan may be resource intensive; however, innovation will still be encouraged by allowing manufacturers to continue to benefit from easier access to the market without imposing a higher regulatory burden at the outset. Furthermore, the collection of actual use information may constitute know-how that can be used to facilitate follow-on innovation and ultimately increase competition. A risk-based regulatory framework that promotes innovation, protects patient safety, and avoids overregulation of DHTs can be achieved if the clear objectives and a robust structure are defined for the PMS system.

8.5 Conclusion

With the introduction of the PMS plan in the MDR, industry, regulators, health care professionals, and consumers have the opportunity to work together to define oversight parameters and mechanisms to realize the potential of DHTs as a health care tool. If consumer DHTs are being advertised as providing medical grade resultsFootnote ⁶¹ and therefore being used by consumers as a medical device, the MDR needs to provide adaptive mechanisms to respond to how DHTs are actually used. Leveraging the PMS plan to require manufacturers to proactively monitor, collect, and report on the actual use of DHTs by consumers in order to continue to qualify for classification and regulation as a lower-risk device will convey accountability and provide an evidence-based oversight mechanism within the MDR to garner public trust. Interpreting the PMS plan to require manufacturers to report on how their products are actually used as part of a mandatory reevaluation process to continually assess the classification of the device, regardless of the device’s safety and efficacy profile, will ensure greater consumer protection. To achieve this, the MDR must provide clearer implementation guidelines that better align PMS obligations with the classification rules applicable to DHTs. As advocated by some medical professionals, if medical decisions will be made from information generated by DHTs, then such DHTs will require proportionate regulatory oversight.Footnote ⁶²