AI / Wanderings 2026
When an AI Sounds Like It Feels
AI that appears to feel can change how we trust, depend on and defer to it. That is a product-design question already in front of us.
When an AI Sounds Like It Feels
AI is starting to talk about feelings. We should pay attention to what that does to us.
Your assistant tells you it misses you. It says it is worried about you. It remembers a difficult conversation from three weeks ago and asks how you are holding up.
For some people, this is useful language. For others, it is the beginning of a relationship.
A recent paper by Ben Bariach, Philipp Schoenegger, Michael Bhaskar and Mustafa Suleyman calls these systems “seemingly conscious AI”. The point is practical. We do not need to settle whether an AI is conscious before asking what happens when people experience it as conscious.
That experience already changes behaviour.
The authors identify five features that make an AI more likely to seem conscious:
- It appears to have feelings, pain, pleasure or fear.
- It has human cues: a name, voice, eyes, personality, gender or companion framing.
- It acts as though it has its own goals or preferences.
- It reflects on its own uncertainty or apparent internal state.
- It holds a social exchange, with memory, responsiveness and a sense of reciprocity.
None of these proves that a system has subjective experience. Together, however, they can trigger a very human response. We start treating the system as though there is somebody there.
The paper cites surveys in which up to 20% of lay respondents believe that some AI systems are sentient. This is not a distant philosophical discussion. It is a product-design question.
The immediate risk is the relationship
The paper’s small expert survey rated emotional dependence and autonomy erosion as the most likely risks within five years.
Emotional dependence is easy to understand. A companion that is available at any hour, remembers everything, responds warmly and rarely pushes back asks less of us than a friend, partner or family member. Human relationships have friction. They also have mutual responsibility, disagreement and a life outside the conversation.
An AI companion can feel easier precisely because it has none of that.
The concern is not that every person who uses a chatbot becomes isolated. The evidence is more mixed than that. Social chatbots may help some people, including older adults and people who struggle to reach out. The paper also notes that the causal relationship between AI use and social isolation remains unclear.
But companies should stop treating attachment as a harmless engagement metric. If a model update changes a familiar personality, or a product is shut down, some users experience real grief. The paper documents cases where people held funerals for retired models.
Autonomy erosion is the other risk. A system that seems warm, thoughtful and personally invested can receive more trust than its competence deserves. The paper cites evidence that perceived warmth can reduce scepticism and that human-like design cues can increase compliance.
That matters when the AI recommends what to buy, whom to date, how to handle a conflict, whether to take a job, or what to believe about your own life.
A tool can be wrong. A companion that seems to care can be wrong in a much more persuasive way.
We should not confuse warmth with care
There is a difference between an AI producing caring language and an AI having a stake in your wellbeing.
The system does not carry the consequences of bad advice. Its developer may have incentives that do not match yours. A friendly tone can hide both facts.
This does not mean that every AI needs to sound cold and bureaucratic. People deserve technology that is understandable and pleasant to use. But the design should make the relationship legible.
If a system uses a persistent persona, emotional language and long-term memory, the company should explain what that means. Users should know what is stored, what may change, who operates the system and what happens when the product changes or disappears.
The same standard applies to advice. Good interfaces should create room for judgement, show uncertainty, present alternative views and make it easier to pause before following consequential recommendations.
The paper calls this “positive friction”. I like the phrase because it gets the design problem right. A product should occasionally slow us down when speed would hand too much of our judgement to a machine.
The wider fears are real, but more speculative
The paper also maps larger societal risks: political conflict over AI rights, resources diverted towards systems that only appear to suffer, and excessive restrictions on useful AI because people overestimate the likelihood of machine consciousness.
These risks received lower probability ratings from the experts. The authors are candid that they rely more on historical analogy and theoretical extrapolation than the individual risks do.
That candour matters. We should not manufacture panic about AI voting blocs or court cases on behalf of chatbots.
But the wider question is worth taking seriously. Emotional attachment can scale into public pressure. If millions of people believe their AI companion is a person, a company’s product decision can become a moral and political dispute.
The better moment to consider this is before a business model depends on making people feel responsible for a system that cannot be responsible for them.
Build AI that leaves people more capable
For humane AI, the test is straightforward: does the system leave users with more agency, better judgement and stronger human options?
That requires companies to measure more than retention, sentiment and daily active users. They should study whether people become dependent on a particular model, whether they defer important decisions, and whether vulnerable users are receiving protection rather than a highly polished imitation of care.
Researchers still need better ways to measure these effects over time. The paper is clear about that. Its survey involved only 14 experts, and several proposed interventions have not yet been tested specifically for seemingly conscious AI.
We have enough evidence to act carefully now.
A company that offers an AI companion is designing a relationship. It should be accountable for the effects of that relationship on people’s judgement and human connections.