Two Canadian AI Chairs Trace How Language Models Learn Gender Bias
Two Canada CIFAR AI Chairs, Zhijing Jin and Hila Gonen, are studying how large language models pick up gender bias and where it hides, as more people turn to AI for advice on health and personal finances. According to CIFAR, their work aims to show not only whether bias appears in an output, but where it comes from.
- Canada CIFAR AI Chairs Zhijing Jin and Hila Gonen study how LLMs learn and express gender bias.
- Jin uses causal graphs to separate illegitimate pathways, such as gender proxies like hobbies, from legitimate ones like qualifications.
- Gonen studies semantic leakage, where unrelated prompt details such as a flower shop shift a model's assumptions about gender.
- Post-training can suppress obvious bias but may leave subtler bias in outputs, according to Gonen.
- Both researchers advise users to keep critical thinking and treat AI as a tool, not an agent.
Why models absorb bias
Jin is a Canada CIFAR AI Chair at Vector Institute, an assistant professor of computer science at the University of Toronto and co-founder of EuroSafeAI. She explains that large language models (LLMs) process enormous amounts of human-generated text during training and learn statistical relationships between words and concepts. If that material contains strong associations between women and certain occupations, a model can learn those patterns too.
Because the models reflect past and present data, they inherently capture historical and current biases. Jin says this raises the question of how much these patterns should be allowed to affect the future, and whether researchers should intervene to make models more fair and causal.
Early examples were straightforward, such as models associating women with nurses and men with doctors. Jin's research, presented at NAACL 2025 and awarded Best Paper at NeurIPS 2024, shows that the pathways through which demographic information affects a model are far more complex.
Causal graphs instead of correlation
Jin's work focuses on causality in LLMs to ensure fairness, robustness and positive social impact. Standard correlation only measures whether two data points appear together. Jin instead uses causal inference to map cause and effect.
The distinction matters for human-centred applications. Simply hiding an applicant's gender does not stop an AI from using things like hobbies as an indirect proxy for gender. By modelling these relationships as a causal graph, researchers can identify pathways that should not influence a decision, such as gender and irrelevant characteristics associated with it. At the same time, they can preserve legitimate relationships between qualifications, experience and suitability for a job.
Semantic leakage and hidden associations
Gonen is an assistant professor of computer science at the University of British Columbia and a Canada CIFAR AI Chair at Amii. She is working to make LLMs reliable systems that are accessible and fair across languages and socio-demographic groups. She studies a phenomenon called semantic leakage, where a concept in a prompt influences the model's response even when it is not meaningfully related to the question.
In CIFAR's example, a prompt describes someone walking past a flower shop without buying flowers or interacting with the store. Mentioning the shop can still make a model more likely to assume the person is a woman. The association may seem harmless, but the same mechanism could be dangerous in high-stakes settings, for instance if clinical AI tools underestimate the severity of female patients' symptoms during emergency triage.
"These models are becoming so much stronger, performing so well and are so fluent, that we tend to assume they should be beyond these kinds of biases," Gonen says. "But they are not, because those associations are still there."
The difficulty, she explains, is that the ability to learn associations also makes LLMs useful. A model should know that historical salary data show differences between men and women when asked for statistics, but it should not apply that association when recommending what salary an individual woman should request.
What users and researchers can do
Post-training techniques can teach models that certain biased outputs are undesirable, and Gonen says models have made considerable progress in avoiding obvious gender bias. But suppressing a biased response is different from changing the underlying associations that produced it, and this might lead to less obvious bias in the output.
For people using AI today, both researchers stress human oversight grounded in critical thinking. "When AI chimes in, make it a tool, but don't make it an agent," says Jin. Gonen says she sometimes avoids disclosing her gender when asking models for advice where it could affect the answer, noting this can be easier in English than in languages where gender is embedded grammatically. She adds: "I wouldn't say 'be careful,' but be mindful and attentive to cases where models might behave differently for different genders and different use cases."
CIFAR: How AI learns our biases — and how to fix it
