Who Should Decide AI's Moral Questions? Not Just Tech Companies, Says Pope and Anthropic Co-founder
As artificial intelligence systems make life-altering decisions, a leading AI researcher and Pope Leo XIV are calling for independent regulation and public input to set moral standards — not corporate self-governance.
Artificial intelligence systems now decide whether you will be flagged at a border, whether your mortgage gets approved, or — in military contexts — which neighbourhoods a weapon system targets. These are fundamentally moral choices about harm and fairness that once belonged to people. Today, they are made by AI developers and the corporations that build them.
That concentration of power prompted an unlikely moment at the Vatican: Chris Olah, co-founder of the AI company Anthropic and a self-described atheist, sat beside Pope Leo XIV and directly challenged his own industry's ability to regulate itself.
"Some might believe that matters of AI are best handled by computer scientists like myself," Olah said. "They are mistaken."
The Pope's Warning
Olah was echoing Pope Leo XIV's first encyclical, Magnifica Humanitas: On Safeguarding the Human Person in the Time of Artificial Intelligence, which warns that AI must serve humanity rather than concentrate power in corporate hands.
The timing reflects a global turning point. Governments around the world are writing the rules for AI right now. Canada, for example, has just launched a federal AI for All strategy with little focus on safety or ethics, let alone clear mechanisms for deciding the harder moral questions.
AI systems now make decisions about borders, mortgages, and weapons targeting. These are moral choices — and they used to be made by people.
Why Companies Cannot Police Themselves
According to research from digital technology scholars, AI developers are not necessarily careless or cynical. Many deeply question the moral aspects of their work. The problem is structural: companies built to move fast create pressure that can push ethical concerns to the margins.
Some firms hire what researchers call "ethics owners" — staff tasked with managing moral tensions by responding to outside criticism while remaining wholly inside the company that provokes it. The approach carries a risk: corporate AI ethics can harden into what scholars call an "economy of virtue" — a marketplace of ethical skills, knowledge and approvals that companies procure to disarm critics and fend off regulation.
The Case for Independent Oversight and Public Voice
Experts argue that AI needs an independent regulator with real power — one capable of saying no to deployment in the same way authorities can refuse approval for a new drug or block a nuclear reactor.
Crucially, that regulator must be informed by ordinary people, not just technical experts. The public must have a voice in setting the moral standards that govern AI systems affecting millions.
Some experiments are already underway. Anthropic invited about 1,000 Americans to help write the rules for its AI chatbot Claude in a project called Collective Constitutional AI. Public engagement researchers have also studied citizens' assemblies — randomly selected groups who study an issue and reach shared recommendations — as a model for tackling urgent policy questions. These approaches offer promise for meaningful public participation in AI governance.
Yet researchers also warn of a risk: "participation-washing" — public consultation that is symbolic rather than binding, leaving ultimate authority with corporations and without real checks on power.
The question now facing governments is whether they will build that independent oversight, and whether they will ensure the public has not just a voice, but genuine weight in deciding whose values shape AI systems that touch billions of lives.