Ensuring healthy levels of AI regulation - the latest on the TGA and AI medical devices

The development of AI regulation has been ramping up globally, particularly in healthcare. In the medical device space, Australia's Therapeutic Goods Administration (TGA) has recently published its report on the outcome of its consultation on clarifying and strengthening the regulation of medical device software including artificial intelligence (the Report), as well as updated guidance on digital scribe products.  

While it appears that no dramatic changes are in store for medical device regulation at this time, the TGA's findings indicate that there will be further consideration of targeted changes to legislation to ensure that AI-enabled medical devices are appropriately regulated. The TGA has not indicated a time frame for this.

At a glance

• The Report found that the current regulatory framework for therapeutic goods is largely adequate to deal with AI, but noted that some tweaks to definitions and offence provisions may be required to clarify the role of entities in the AI lifecycle, clarify how new methods of supplying software are regulated, and appropriately assign responsibility for the outputs of AI.
• Sponsors of digital mental health tools should be on high alert that the TGA no longer considers the current exclusion of such products from regulation to be appropriate. Similarly, sponsors of medical scribe products should note that the TGA intends to review whether medical scribes should be regulated as a medical device.
• Medical device manufacturers and developers of health-related AI can anticipate the release of further practical guidance on technical requirements for adaptive and generative AI, and standards for using and validating data sets of unknown provenances.
• On a global level, AI is a rapidly developing field of regulation. The TGA's general approach to regulation is to ensure international harmonisation with comparable overseas jurisdictions to the greatest extent possible.  Medical device manufacturers in Australia should be mindful that changes in larger jurisdictions (such as the USA and EU) may have flow-on effects on the development of regulation in Australia.

Background

In late 2024, the TGA conducted a consultation on clarifying and strengthening the regulation of medical device software including artificial intelligence. The consultation aimed to determine whether existing legislation, regulations and guidance are appropriate to meet the challenges associated with the use of medical software and AI across the healthcare sector, and to identify measures to clarify and strengthen existing regulation to mitigate risks and leverage opportunities associated with AI use in the medical sector.

The consultation was done as part of a broader review by the Department of Health and Aged Care into the safe and responsible use of AI in healthcare, which complements the work done by the Department of Industry, Science and Resources on developing mechanisms to promote safe and responsible AI in Australia (including the potential introduction of mandatory AI guardrails for high-risk applications of AI).

What laws and regulations might change?

The Report found that the TGA's existing framework and technology-agnostic approach to regulation is largely appropriate to deal with the increasing use of AI in therapeutic goods, but found that the following points (amongst others) may require legislative change to address the unique issues arising from AI software medical devices.

1. Changes to definitions to address entities with regulatory obligations
   ‍
   Currently the Therapeutic Goods Act 1989 (Cth) (TG Act) predominantly uses the words "manufacturer" and "sponsor" to describe entities with regulatory obligations. The Report notes that these terms do not entirely align with emerging terminology to describe entities involved in the AI lifecycle, such as "developer", "deployer", and "distributor."
   ‍
   In particular, the Report noted that the definition of "manufacturer" is not recognised by stakeholders in the software sector or clinical practitioners who may be involved in a manufacturing activity as a part of clinical practice.  Stakeholders also indicated that there is a lack of clarity regarding who is considered a "sponsor" of software products under the TG Act, particularly in scenarios where a software product (that is a medical device) is made available through an online platform hosted by an Australian entity.  
   ‍
   In response to these emerging concerns, the TGA has indicated it will review legislative definitions to determine whether refinements are needed (Finding 1).
   ‍
   The Report also noted that there is general confusion about how new methods of supplying software through online platforms are regulated. The Report noted that the current definition of "supply" under the TG Act is linked to the primary role of a sponsor. The TGA will review the definition of "supply" under the TG Act to ensure that access to software-based devices through online, digital or virtual means is appropriately reflected in the Act.
   ‍
   In response to these emerging concerns, the TGA has indicated it will review legislative definitions to determine whether refinements are needed (Finding 2).
   ‍
2. Changes to language used in offence provisions to assign responsibility
   
   ‍The Report noted that there are emerging use cases for AI that involve replacing services and activities traditionally performed by human beings.  The TG Act contains a number of offence provisions that relate to actions done by a "person". The use of AI to replace human involvement therefore raises questions about who should be assigned responsibility for unlawful or prohibited activity done by an AI model or system. The TGA intends to conduct further review and consultation to determine whether language used in the TG Act for offences continues to assign responsibility to the appropriate entity across the AI lifecycle (Finding 3).
   
   In the meantime, health practitioners using AI medical devices in clinical practice should be mindful of the AHPRA guidance on meeting professional obligations when using Artificial Intelligence in healthcare. Health practitioners remain responsible for delivering safe and quality care, and that to meet their professional obligations, they must apply human judgment to any output of AI.  

What's staying the same

As noted above, the Report does not propose sweeping changes to Australia's medical device regulatory framework. The Report found that no changes are needed to existing medical device classification rules at this time (with future consideration to be given to the classification for AI software with predictive or prognostic functionality, which are currently considered low-risk) (Finding 5).  Similarly, no changes are proposed to the Essential Principles, although the TGA acknowledged that more extensive guidance, with examples demonstrating how the existing regulations apply to specific use cases, is required (Finding 6).

Sponsors and developers of software based medical devices should also not expect any regulations to be introduced which deal with specific subtypes of AI, with the TGA indicating that the existing, technology-agnostic framework is intended to be flexible, and will remain as it is.  The TGA acknowledges that various subtypes of AI are "limitless" and in order to deal with the increasing use cases, the TGA intends to provide clarity to varying subtypes by developing guidance, as opposed to trying to regulate based on each "kind" of AI (Finding 7).

More practical guidance

A common theme in the Report is that instead of changing the legislation and regulations and creating provisions that explicitly deal with AI-specific issues, the TGA will instead address emerging questions about new applications of AI through the use of guidance documents.  In welcome news for sponsors and developers of AI medical devices, the TGA has flagged the following two topics as requiring urgent guidance, which they note should be developed "as expeditiously as possible":

1. Guidance on continuous change control, validation and technical requirements for adaptive and generative AI. Currently, the approach to regulating therapeutic goods is based on an assessment of static models and systems where safety, performance and effectiveness of medical devices are assessed before deployment, and ongoing monitoring of the device is done via post-market monitoring and adverse event reporting. This approach may not be effective when regulating adaptive and generative AI, which may change over time without manufacturer oversight.  The TGA also indicated that the guidance was likely to focus on explaining what changes constitute a "significant change" to an adaptive and generative AI medical device for the purposes of fulfilling notification requirements to the TGA (Finding 11); and
   ‍
2. Guidance for the use of open datasets and software of unknown provenance in the development of AI. The TGA has flagged that there is a need for guidance to help sponsors and developers address the risks of using datasets or software from unknown provenance within medical devices as training data sets. The TGA noted that currently, the justification for the use of these resources and demonstration of their validity is a matter for developers, as there are no explicit guidelines or standards available for those who choose to use options of unknown provenance. The TGA acknowledged that developers are seeking more information about acceptable validation for medical devices that incorporate software of unknown provenance or are trained on open datasets (Finding 12).  

These two matters are noted in the Report to be issues that are currently challenging for regulators globally. Noting that Finding 14 of the Report indicates that the TGA will seek to maintain international harmonisation with comparable jurisdictions as far as possible, sponsors and developers seeking to future-proof their products should keep abreast of any potential changes in larger comparable jurisdictions (such as the USA and the EU).

Noting the lack of clear regulatory guidance at this stage, it is advisable for sponsors and developers to comply with recognised international standards for AI (such as ISO/IEC standards in relation to artificial intelligence). The use of such standards is likely to assist in demonstrating that the developer has appropriate justifications in place for the design and development of their medical device.    

Mental health tools and digital scribes in the hot seat

While the TGA has not announced drastic changes for AI medical devices overall, sponsors and developers of AI digital scribes and digital mental health tools should be aware that these two particular types of software medical devices have been flagged for regulatory reform.

Therapeutic goods legislation currently contains 15 different exclusions that exclude software from regulation as a medical device. Software that is a digital mental health tool is currently one of those exclusions, however the TGA has indicated in the Report that it no longer considers this exclusion to be appropriate (Finding 8).

The TGA has also stated that it is reviewing digital scribes to determine whether they should be considered medical devices (Finding 4). Separately to the report, the TGA has also recently released guidance on digital scribes, which clarifies that if a digital scribe analyses or interprets clinical conversations, including by generating a diagnosis, differential diagnosis or treatment recommendation not explicitly stated by the healthcare practitioner, it is considered a medical device.

For software which may fall under the other 14 exclusions from medical device regulation, the TGA has indicated that it will be reviewing the exclusions and releasing further guidance to assist sponsors and manufacturers with understanding whether their product is excluded.  

In the meantime, developers of AI-enabled medical device software can now see how other AI-enabled medical devices are classified on the Australian Register of Therapeutic Goods (ARTG). The TGA has published a list of AI-enabled medical devices currently included on the ARTG. While we do not recommend that sponsors and developers rely solely on this list to determine the regulatory requirements applicable to their device (noting that complexity and nuance in medical device classification rules), this list certainly serves as a helpful starting point to see how others in the industry have classified AI-enabled medical devices.

What should medical device manufacturers keep an eye on next?

The Report canvassed a broad range of AI-related issues, but medical device manufacturer should keep an eye out for:

• Further consultation on the definitions and language used in the TG Act (including in relation to the definition of "manufacturer" and "sponsor", and language used in offence provisions);
• Changes to the regulation of digital mental health tools;
• Changes to the regulation of digital scribes; and
• Additional guidance on excluded software, technical requirements for AI, and the use of open datasets of unknown provenance.

Noting the TGA's general approach to international harmonisation, we also recommend that medical device manufacturers keep abreast of changes made in comparable overseas jurisdictions, particularly developments from the US FDA and the EU EMA. Changes to regulatory approaches in the US and EU are likely to have impacts on Australian regulation (although global manufacturers should be mindful of Australia-specific nuances).

Further, noting that the Report is only one piece of the healthcare puzzle, medical device manufacturers may also want to stay on top of the ongoing work by AHPRA and the Australian Commission on Safety and Quality in Health Care (ACSQHC) (here) on regulating the use and implementation of AI in healthcare settings.  The professional standards and obligations set by these bodies may inform acceptable use cases for AI in Australia, and guide the development of AI products.

Finally, it is evident that whilst the TGA clearly has its sights set on AI regulation and is proactively reviewing legislation and developing guidance, the use of AI in medical devices is a fast-evolving area that is currently outpacing regulation.  While additional guidance as to the TGA's interpretation of how therapeutic goods laws apply to various types of AI will undoubtedly be a helpful steer for sponsors and developers, such guidance may not fully address all of the rapidly expanding number of use cases for AI in healthcare.  In particular, sponsors should be mindful that medical device regulation is complex, and minor nuances in the function and intended purpose of a product may have significant consequences on the classification (and subsequently, the level of regulation) of a particular device.

If you need assistance with understanding and interpreting the current regulatory landscape for software-based medical devices, particularly those incorporating AI, please reach out to our Competition and Regulation team.

‍