Professor warns on AI authority, not consciousness

Professor Rocky Scopelliti has published a new book on artificial intelligence and human authority, arguing that organisations should focus on how much decision-making power they hand to AI systems.

Scopelliti, an Australian futurologist and author, argues that the immediate governance issue is not whether machines become conscious, but whether people and institutions cede meaningful control before clear rules are in place. He says that distinction matters as businesses and public bodies move from experimenting with generative AI tools to using systems that can recommend decisions and carry out actions.

His book, The Conscious Code: From Artificial Intelligence to Artificial Consciousness, examines the development of AI from information processing towards systems that can perceive, decide and act. Drawing on neuroscience, cognitive science, philosophy, robotics, ethics and governance, it argues that increasingly sophisticated behaviour should not be mistaken for evidence of machine consciousness.

"Capability is not consciousness," said Professor Rocky Scopelliti, general editor of The Evolution of Intelligence Series at Taylor & Francis Group.

He links that argument to a broader policy debate in New Zealand, where officials and companies are weighing how to expand AI use while maintaining safeguards. New Zealand's AI Strategy has focused on responsible adoption and the potential economic gains from wider use of the technology.

Scopelliti argues that the next stage of that debate should examine where assistance ends and delegated authority begins. In his view, that line becomes harder to see when a system appears to support a human decision but, in practice, shapes the outcome so strongly that oversight becomes nominal.

"Today's AI should not be described as conscious. But we don't need conscious machines before we confront an entirely new problem: what happens when machines become capable of making increasingly consequential decisions and acting on them?" Scopelliti said.

Human oversight

The book points to decision-making in finance, healthcare, employment and government as areas where that distinction could matter. A lending model may advise whether a customer should receive a loan, an employment system may narrow a shortlist, and a clinical tool may recommend a treatment path.

In each case, Scopelliti argues, the formal presence of a human reviewer does not settle the question of accountability. If the reviewer lacks the time, authority or technical understanding to challenge the recommendation, the machine may already hold effective decision-making power.

"When organisations say there is a human in the loop, we need to start asking whether that human actually possesses the knowledge, time and authority to change the machine's decision," Scopelliti said.

"A rubber stamp isn't meaningful human oversight," he said.

The issue becomes sharper with so-called agentic AI systems. Unlike conventional chatbots or text generators that respond to prompts, these systems can pursue goals through a sequence of choices and actions, including interactions with software tools, information systems and other agents.

That raises practical questions for boards, executives and regulators. If an AI agent changes a price, declines a customer, executes a transaction or triggers an action that causes harm, responsibility may be spread across several parties, from the developer and model provider to the organisation deploying the system and the employee supervising it.

Five questions

Scopelliti's research sets out five questions that leaders and policymakers should ask before authority is delegated to AI. They concern the consequences of the decision for affected people or communities; whether the system's reasoning can be understood and challenged; whether a clearly identifiable person remains accountable; whether a human can intervene before irreversible harm occurs; and whether the decision involves moral, democratic or human-rights judgments.

These tests are intended to help organisations distinguish routine automation from areas where human judgment should remain central. They also reflect concerns that AI adoption is moving from support functions into decisions with material effects on livelihoods, access to services and public trust.

Scopelliti has already provided related research to the South Australian Government as it develops the terms of reference for its Royal Commission into Artificial Intelligence. That work also focuses on the point at which delegation to AI becomes a governance problem rather than a productivity tool.

In New Zealand, the intervention comes as businesses shift from pilot projects to broader deployment. That trend mirrors developments in other markets, where the early use of generative AI for drafting and summarising is giving way to systems designed to make recommendations, monitor activity and initiate tasks with limited human input.

"New Zealand's AI Strategy asks how the country can confidently adopt AI. My research asks what comes next: what happens as the AI we adopt increasingly perceives, decides and acts on our behalf?" he said.

"AI began by answering our questions. Increasingly, it will recommend our decisions. The next generation of agentic systems will increasingly execute actions as well. That transition from answer to recommendation to decision to action fundamentally changes the governance problem," Scopelliti said.

He also warns that the public may begin treating highly convincing systems as deserving greater trust and authority even without evidence that they possess subjective experience. In that sense, he argues, the social and institutional consequences of AI may arrive well before any scientific consensus on artificial consciousness.

"The great paradox may be that society begins granting machines greater authority before we understand what, if anything, those machines experience," Scopelliti said.

"The immediate question therefore isn't whether machines are conscious. It is whether humans are prepared for machines that are powerful enough to act as though they don't need us," he said.

"The countries that lead in AI won't necessarily be those that automate the most," Scopelliti said.

"They may be those that become clearest about what should be automated, what can be delegated, and what must remain meaningfully human," he said.