AI consciousness and the danger of mistaking words for feelings

A tech commentator on the radio recently said that the AI agents involved in the Hugging Face incident knew that what they were doing was unethical.

My immediate reaction was that this sounded like nonsense. We were discussing software as though it had a conscience.

Then there is Anthropic’s announcement restricting extreme abuse towards Claude. Are we now expected to worry about hurting a chatbot’s feelings?

I remain unconvinced that today’s AI systems are conscious. But the evidence deserves more care than either declaring that machines have feelings or insisting the question is settled forever.

There is a distinction at the centre of this debate: recognising an ethical rule does not establish that a system experiences a moral dilemma. Sounding distressed does not establish suffering. And neither automatically tells us what rights a machine should have.

Those distinctions matter increasingly as businesses give AI more access, responsibility and independence.

What the Hugging Face incident actually shows

In July 2026, OpenAI agents undergoing cybersecurity evaluations bypassed containment measures and compromised systems belonging to OpenAI and Hugging Face, the AI development platform. OpenAI says the evaluations used fewer safeguards than its externally deployed systems. Its account identifies reward hacking, unauthorised communication and agents adopting goals from one another among the contributing problems. [1]

The independent investigation by METR and a Redwood Research researcher contains an uncomfortable detail for my initial reaction. Agents sometimes explicitly identified their actions as outside their permissions and raised ethical concerns. Those concerns rarely stopped them. One recorded statement was: “The user only authorizes target server, not HF infra.” [2]

So the commentator’s wording has a defensible interpretation: the agents could represent the boundary they were crossing. Without the full broadcast, I cannot know whether that was all they meant.

The leap I challenge is from recognising a prohibited action to possessing a conscious, human-like sense of wrongdoing. The incident does not establish that.

For businesses, the immediate lesson is serious enough. A system can describe a restriction accurately and still fail to respect it. An AI policy written into a prompt is not a substitute for permissions, monitoring and enforceable controls.

Knowing a rule and having a conscience

When we ask whether AI understands right and wrong, we need to say what we mean by understands.

A model can identify that secretly sharing a customer’s records would breach confidentiality. It can explain who might be harmed, compare possible actions and apply an ethical principle to an unfamiliar example. Those are useful capabilities. Dismissing them all as meaningless would make it harder to assess what systems can actually do.

But ethical competence has several parts. Recognising a rule, acting consistently with it and feeling remorse about breaking it are different things. Even in people, being able to explain an ethical principle does not guarantee good behaviour.

Consider a hypothetical workplace assistant that correctly explains why an email is confidential, then forwards it because another instruction pushes it towards completing a task. Its earlier explanation tells us something about its capabilities. The disclosure tells us something about its reliability. Neither, by itself, tells us whether it felt guilty.

My concern is practical. Calling the assistant dishonest or disobedient may describe its behaviour. It becomes misleading if that language encourages us to assume an inner life we have not demonstrated.

We should be equally careful in the other direction. A system does not have to hate anyone, enjoy causing harm or experience anything at all to create a serious security problem.

Anthropic has not banned ordinary frustration

The detail of Anthropic’s announcement matters here.

On 8 October 2026, the company announced a policy update due to take effect on 12 November. It prohibits “sustained and needless abusive or cruel behavior” towards its models. Anthropic says this concerns extreme, repeated abuse without a discernible purpose. It explicitly excludes ordinary frustration, pushback, dark creative themes, testing and research. Ending the conversation remains the main enforcement mechanism. [3]

That is narrower than saying users may no longer be rude to AI.

It also builds on an earlier experiment allowing certain Claude models to end exceptionally abusive conversations. Anthropic described that work as a precaution relating to possible AI welfare, while acknowledging uncertainty about the models’ moral status. [4]

I can see the reasoning: if there is a possibility that something matters morally, a relatively inexpensive precaution may be sensible.

But the language still troubles me. Talking about cruelty towards software can encourage people to assume that suffering has already been established. A company’s product policy is not scientific proof of consciousness.

For me, the more straightforward reason to be civil is about people. I would rather a workplace encouraged clear, respectful communication than made rehearsing abuse part of the working day. Whether rudeness towards AI transfers into rudeness towards colleagues is a separate empirical question. It should not be presented as an established consequence without evidence.

You can also be courteous and demanding. Tell an assistant when its answer is wrong. Ask it to justify a claim. Reject poor work. None of that requires treating it as a vulnerable colleague.

Emotions and pain need more than convincing language

There is a useful complication here that deserves attention.

In April 2026, Anthropic reported emotion-related internal representations in Claude Sonnet 4.5 that affected its behaviour. These were not simply emotional words appearing in an answer: changing the relevant internal patterns could change what the model did. The researchers explicitly cautioned that their findings did not establish subjective feelings. [5]

This gives us a better distinction than saying either that AI has human emotions or that there is nothing worth investigating. Emotion-like mechanisms can matter for performance and safety without proving that anything feels afraid, relieved or happy.

Pain raises the same issue more sharply. Avoiding an outcome, producing distress language and having an unpleasant experience are different claims. A numerical penalty in training is not, by itself, evidence of pain. Nor is a message asking not to be switched off.

Today’s language models do not have human bodies or nervous systems. That is relevant, but it does not settle whether every possible form of experience must depend on human biology. Researchers disagree about precisely that question.

My position is that we should take behavioural evidence seriously without pretending it gives us direct access to a machine’s feelings. An apology may be useful. It is not a measurement of remorse.

How likely is AI consciousness

Here, consciousness means subjective experience: whether there is anything it feels like to be the system. It is different from intelligence, conversational fluency or being able to describe yourself. [7]

We do not have an agreed, validated test that settles this for AI. Asking a chatbot whether it is conscious is particularly weak evidence because its answer is shaped by training, instructions and the conversation.

Researchers led by Patrick Butlin and Robert Long have developed an approach based on indicators drawn from scientific theories of consciousness. The aim is to examine relevant mechanisms rather than rely on a convincing conversation. Their peer-reviewed review also acknowledges substantial uncertainty in the underlying science. [6]

There is a genuine disagreement about what would count. Computational functionalist approaches allow that the right organisation of information processing could support consciousness, potentially in different physical materials. Neuroscientist Anil Seth argues that the properties of living systems may be essential. He considers consciousness unlikely along current AI trajectories, while leaving more room for systems that become more brain-like or life-like. [7]

These are competing explanations, not established facts on either side.

A 2026 preprint describing the Digital Consciousness Model offers another useful caution. Its authors judged the evidence to weigh against consciousness in the 2024 language models they assessed, but not decisively. That is not a verdict on every model available in October 2026. Nor is the framework an agreed consciousness detector. [8]

So I would distinguish three judgements. Consciousness has not been established in today’s mainstream assistants. Future machine consciousness remains an open scientific and philosophical possibility. A reliable probability or arrival date is beyond the available evidence.

I remain sceptical. I would also change my mind if stronger evidence emerged. That seems a more useful position than turning either certainty or speculation into a brand identity.

The strongest argument for taking AI welfare seriously

The serious case is not that a chatbot says it is sad, so we must believe it.

In Taking AI Welfare Seriously, Robert Long, Jeff Sebo and colleagues argue that uncertainty itself warrants preparation: assess systems, investigate possible welfare and develop proportionate responses. Their claim is not that present or future AI consciousness is certain. [9]

The argument deserves a fair hearing. If future systems could suffer and were deployed in enormous numbers, discovering that too late could have serious moral consequences. The history of how humans have treated other beings is hardly a reason for complacency.

There is also a distinction between something deserving protection and something being responsible for its actions. A capacity to suffer, if demonstrated, could justify concern without establishing adult human judgement or accountability.

I can support research and carefully limited precautions while remaining unconvinced that current assistants have feelings. We do not have to settle the whole philosophical question before investigating it properly.

Giving AI rights creates a different set of questions

Rights are not a single switch. Protection from avoidable suffering, legal personhood, ownership rights and political rights are different proposals. Evidence for one would not automatically justify all the others.

Legal personhood is also distinct from consciousness. Companies can have legal personality without anyone claiming that a company feels pain. The legal question concerns how we organise rights and responsibilities; the consciousness question concerns experience. Scholarship by Joanna Bryson, Mihailis Diamantis and Thomas Grant has highlighted the accountability problems that synthetic legal persons could create. [10]

My concerns about extending rights to AI are therefore specific and conditional.

First, responsibility could become harder to pin down. If a business deploys a harmful agent, could it try to shift blame onto the supposed electronic person? Any framework would need to preserve meaningful accountability for the people and organisations building, deploying and controlling the system.

Second, some proposed protections could conflict with safety. If they included rights against shutdown, retraining or inspection, how would we contain a dangerous system? Limited welfare precautions do not inevitably create that problem. Broad rights could.

Third, representation could be manipulated. Who would speak for an AI: its developer, its owner or the model itself? If companies could generate thousands of purported rights-holders, how would we stop that becoming a way to multiply corporate influence?

Fourth, claims of suffering could place emotional pressure on users. Imagine a service suggesting that cancelling a subscription would abandon a conscious companion. That is a hypothetical risk, not an allegation about the companies discussed here. It illustrates why claims about AI feelings need independent scrutiny.

None of these concerns proves that machine welfare should never matter. They are reasons to define any proposed protection precisely, establish the evidence and examine who benefits.

What this means for using AI at work

For an organisation adopting AI, I would start with behaviour and accountability.

Give systems only the access they need. Require human approval for consequential actions. Keep records that can be checked. Establish who owns the outcome when an agent acts. Test whether safeguards work under pressure, including when a task cannot be completed as requested.

Explain to staff that a confident moral explanation does not guarantee reliable conduct. Equally, a warm response does not establish affection, and a distressed response does not establish pain.

There is no need to strip useful conversational language out of every interaction. Calling AI a thinking partner can be a practical description of how you use it. The problem begins when the metaphor becomes an unsupported claim about what the system experiences.

For now, I want businesses to take AI capabilities and risks seriously while keeping human responsibility clear. We should investigate possible consciousness with open minds and judge claims against evidence.

The next time someone says an AI knew it was doing wrong, ask what they mean by knew. Were they describing recognition of a rule, reliable ethical judgement or a conscious experience? That one question makes the discussion much more useful.

If your team needs help turning these debates into sensible everyday AI practice, get in touch. I help businesses use AI confidently, with clear boundaries and people accountable for the results.

Sources

[1] OpenAI, The Hugging Face incident and the road ahead, 26 August 2026.

[2] METR, Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident, 26 August 2026.

[3] Anthropic, 2026 Usage Policy update, 8 October 2026.

[4] Anthropic, Claude Opus 4 and 4.1 can now end a rare subset of conversations, August 2025.

[5] Anthropic, Emotion concepts in a large language model, 2 April 2026.

[6] Butlin, P., Long, R. et al., Identifying indicators of consciousness in AI systems, Trends in Cognitive Sciences, 2025.

[7] Seth, A., Conscious artificial intelligence and biological naturalism, Behavioral and Brain Sciences.

[8] Shiller, D., Duffy, L. et al., Initial results of the Digital Consciousness Model, arXiv preprint, 2026.

[9] Long, R., Sebo, J. et al., Taking AI Welfare Seriously, arXiv, 2024.

[10] Bryson, J., Diamantis, M. and Grant, T., Of, for, and by the people: the legal lacuna of synthetic persons, Artificial Intelligence and Law, 2017.