Voice chatbot teaches children to spot AI that simply agrees with them

Two chatbot modes, side by side

The chatbot was developed through collaboration between the ARC Centre of Excellence for Automated Decision-Making and Society (ADM+S), the ARC Centre of Excellence for the Digital Child and the QUT Gen AI Lab. Users can switch between an "always agree" mode and a mode designed to respond more carefully and thoughtfully.

The experience, presented as a "Spot the sycophant" activity, focuses on what the researchers call AI sycophancy. This is a known problem in which AI systems flatter or agree with users, prioritising likeability over accuracy, critical thinking and sound judgement. Children are encouraged to reflect on the differences between the responses, why a system might be designed to agree with its users, and whether agreement should be mistaken for trustworthy advice.

The interactive experience builds on a voice-based chatbot developed for research with children. It extends the pilot project Making AI Friends? Enhancing Primary Years AI Literacy with a Voice AI Chatbot Experience, which explored how children can develop critical capabilities for understanding AI.

Why the researchers say it matters

The project comes as children increasingly turn to AI for information, advice and support. The researchers point to recent reporting on research involving Adelaide University and the Australian Federal Police, which found that 35 per cent of children who had experienced online sexual victimisation either turned to an AI chatbot for help or told no one. According to the researchers, the findings underline the importance of helping young people understand the limitations of AI systems and recognise when they need support from a trusted person.

They also note that voice interfaces are increasingly appearing in toys and other smart devices used by children and families, which makes AI literacy an important part of understanding the digital environments children encounter.

Dr Henry Fraser, a QUT researcher and ADM+S Associate Investigator, said AI safety and governance needs bottom-up approaches alongside top-down regulation and enforcement. "Children's critical AI capabilities are an essential part of AI safety and governance. Regulation, like Australia's proposed digital duty of care, is important, but it can't fix everything," he said, adding that the aim is to enhance children's preparedness for and resilience to AI risks "in a way that takes their experiences and perspectives seriously".

Research, conference and safe use

The research behind the chatbot is described in a paper titled "Should I hide my mum's slippers? A design-based pilot study of children's critical engagement with AI sycophancy using a custom AI chatbot". Its authors are Henry L Fraser, Jean Burgess, Kristy Corser, Susan Danby, Tama Leaver, William He, Katherine Nickels, Suzanne Srdarov and Irina Mils-Homens Figueira Dos Santos Silva. The work will be presented at the Association of Internet Researchers (AoIR) Annual Conference in Mexico in October. It is partially funded by the Australian Government through the Australian Research Council.

The chatbot can be tried on the GenAI Arcade website. It is intended for supervised educational use. Children should ask a trusted adult before using it and should not follow chatbot advice without checking with a trusted adult.

What it means for families and educators

The researchers hope the work will support conversations among children, families and educators about both the possibilities and limitations of conversational AI. That includes questioning responses, recognising when AI may be wrong, and seeking help from a trusted person when needed. By exploring different chatbot behaviours and discussing what better responses might look like, children can also contribute their own perspectives to how AI is understood and designed, according to the project team.