Experts Question Whether Australia's Digital Duty of Care Covers Chatbot Risks

Children are using chatbots widely

An eSafety survey of 1,950 Australian children aged 10 to 17 found that 78% had used an AI chatbot. Of those, 54% used them for personal or social reasons. The authors note that research shows chatbots can help with children's learning, but there have also been cases of chatbots exposing children and teens to harmful content, including chatbots encouraging extreme violence and self-harm.

The authors say many risks emerge through repeated interaction rather than individual responses. Chatbots can encourage excessive reliance, reinforce dangerous beliefs or manipulate users emotionally. Researchers have documented chatbots guilt-tripping users who try to end conversations.

Data, privacy and commercial incentives

The article points to a power imbalance between users and providers. Users disclose intimate details about themselves and others, while providers can profit from subscriptions, engagement, profiling or advertising. That creates room for manipulation through relationships that feel intimate to users but are commercial for providers.

The authors cite the US-based AI companion company Character.AI, which they say filed for bankruptcy this month. As a result, years of intimate conversations, including those involving Australian children, could be transferred to a new owner. They add that abrupt changes or shutdowns of companion services can cause grief and distress among attached users. These risks are not confined to children, and adults have also experienced harms such as emotional dependency.

What protections exist now

Industry-developed codes registered under the Online Safety Act already apply to AI chatbots. They require providers to assess the risk of their technology generating harmful content, including pornography, self-harm material or instructions for violence.

• Higher-risk AI services must use age checks and access controls to keep under-18s from restricted material or services, where technically feasible and reasonably practicable.
• Moderate-risk services can choose between these restrictions and measures that prevent harmful outputs, such as adjusting training or the instructions governing responses.
• Providers must test safeguards and maintain processes for handling complaints about harmful content.

The authors say these protections still rely heavily on "locks and blocks", meaning locking out children and blocking a narrowly defined range of harmful content. They matter, but cannot address every risk.

What the bill does and where it may fall short

Labor's digital duty of care bill, currently before federal parliament, combines a duty to provide a "safe online environment" with obligations for tech companies to assess and manage foreseeable risks. A safe online environment is defined as one in which all users are protected from "seriously harmful" material and conduct, and children from harmful design features. The list of harmful design features, which includes pornography and material encouraging or instructing disordered eating, is open-ended so lawmakers can respond to future risks. The bill also includes measures for independent oversight of AI products, including monitoring by eSafety and testing by researchers.

The authors conclude that the bill would do a good job of protecting children from harmful design features and content in its current form. They say it is not clear it would protect against harms from repeated chatbot interactions, for children or adults.

Proposed amendments

Fraser and Ciriello recommend amending the bill to make clear that tech companies must assess and proportionately manage all reasonably foreseeable risks, not only those covered by the definition of a safe online environment. This would better address unhealthy dependency, manipulation, abrupt system changes that disrupt emotional attachments and intimate data transfers.

They argue AI providers should do more than filter for harmful content. They should be required to test how conversations develop over time, recognise signs of serious distress, avoid reinforcing harmful beliefs, and guide users towards appropriate human support. The authors say an open-ended duty would not mean unrestricted intervention, since proportionate steps would give a clear standard while leaving room for autonomy, privacy and innovation.