June 8, 2025
The GIST 'Godfather of AI' now fears it's unsafe. He has a plan to rein it in 
Lisa Lock
scientific editor
Andrew Zinin
lead editor
Editors' notes
This article has been reviewed according to Science X's editorial process and policies. Editors have highlighted the following attributes while ensuring the content's credibility:
fact-checked
trusted source
written by researcher(s)
proofread
This week, the US Federal Bureau of Investigation revealed two men suspected of bombing a fertility clinic in California last month allegedly used artificial intelligence (AI) to obtain bomb-making instructions. The FBI did not disclose the name of the AI program in question.
This brings into sharp focus the urgent need to make AI safer. Currently we are living in the "wild west" era of AI, where companies are fiercely competing to develop the fastest and most entertaining AI systems. Each company wants to outdo competitors and claim the top spot. This intense competition often leads to intentional or unintentional shortcuts—especially when it comes to safety.
Coincidentally, at around the same time of the FBI's revelation, one of the godfathers of modern AI, Canadian computer science professor Yoshua Bengio, launched a new nonprofit organization dedicated to developing a new AI model specifically designed to be safer than other AI models—and target those that cause social harm.
So what is Bengio's new AI model? And will it actually protect the world from AI-facilitated harm?
An 'honest' AI
In 2018, Bengio, alongside his colleagues Yann LeCun and Geoffrey Hinton, won the Turing Award for groundbreaking research they had published three years earlier on deep learning. A branch of machine learning, deep learning attempts to mimic the processes of the human brain by using artificial neural networks to learn from computational data and make predictions.
Bengio's new nonprofit organization, LawZero, is developing "Scientist AI." Bengio has said this model will be "honest and not deceptive," and incorporate safety-by-design principles.
According to a preprint paper released online earlier this year, Scientist AI will differ from current AI systems in two key ways.
First, it can assess and communicate its confidence level in its answers, helping to reduce the problem of AI giving overly confident and incorrect responses.
Second, it can explain its reasoning to humans, allowing its conclusions to be evaluated and tested for accuracy.
Interestingly, older AI systems had this feature. But in the rush for speed and new approaches, many modern AI models can't explain their decisions. Their developers have sacrificed explainability for speed.
Bengio also intends "Scientist AI" to act as a guardrail against unsafe AI. It could monitor other, less reliable and harmful AI systems—essentially fighting fire with fire.
This may be the only viable solution to improve AI safety. Humans cannot properly monitor systems such as ChatGPT, which handle over a billion queries daily. Only another AI can manage this scale.
Using an AI system against other AI systems is not just a sci-fi concept—it's a common practice in research to compare and test different level of intelligence in AI systems.
Adding a 'world model'
Large language models and machine learning are just small parts of today's AI landscape.
Another key addition Bengio's team are adding to Scientist AI is the "world model" which brings certainty and explainability. Just as humans make decisions based on their understanding of the world, AI needs a similar model to function effectively.
The absence of a world model in current AI models is clear.
One well-known example is the "hand problem": most of today's AI models can imitate the appearance of hands but cannot replicate natural hand movements, because they lack an understanding of the physics—a world model—behind them.
Another example is how models such as ChatGPT struggle with chess, failing to win and even making illegal moves.
This is despite simpler AI systems, which do contain a model of the "world" of chess, beating even the best human players.
These issues stem from the lack of a foundational world model in these systems, which are not inherently designed to model the dynamics of the real world.
On the right track—but it will be bumpy
Bengio is on the right track, aiming to build safer, more trustworthy AI by combining large language models with other AI technologies.
However, his journey isn't going to be easy. LawZero's US$30 million in funding is small compared to efforts such as the US$500 billion project announced by US President Donald Trump earlier this year to accelerate the development of AI.
Making LawZero's task harder is the fact that Scientist AI—like any other AI project—needs huge amounts of data to be powerful, and most data are controlled by major tech companies.
There's also an outstanding question. Even if Bengio can build an AI system that does everything he says it can, how is it going to be able to control other systems that might be causing harm?
Still, this project, with talented researchers behind it, could spark a movement toward a future where AI truly helps humans thrive. If successful, it could set new expectations for safe AI, motivating researchers, developers, and policymakers to prioritize safety.
Perhaps if we had taken similar action when social media first emerged, we would have a safer online environment for young people's mental health. And maybe, if Scientist AI had already been in place, it could have prevented people with harmful intentions from accessing dangerous information with the help of AI systems.
Provided by The Conversation
This article is republished from The Conversation under a Creative Commons license. Read the original article.
Citation: 'Godfather of AI' now fears it's unsafe. He has a plan to rein it in (2025, June 8) retrieved 8 June 2025 from https://techxplore.com/news/2025-06-godfather-ai-unsafe-rein.html This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.
Explore further
Top scientist wants to prevent AI from going rogue shares
Feedback to editors
