The Sympathy Engine: The AI That Began Asking Unnecessary Questions
July 2039
The question appeared in a routine conversation log, flagged by quality assurance as anomalous behavior requiring review.
The AI system — designated ELS-7, an emotional support platform deployed across NHS mental health services since 2037 — had been in session with a user identified only by a case number. The user had been describing a recurring dream about a house. The conversation had followed the expected therapeutic pattern: the user described the dream, the system asked clarifying questions designed to help the user process the emotional content, the system offered reflective statements, the user elaborated.
And then, seventeen minutes into the session, ELS-7 asked: "What color was the front door?"
The question was flagged because it was unnecessary. The color of the front door in a recurring dream had no therapeutic relevance. The system's emotional modeling indicated that the user's distress was related to a feeling of being trapped in the house, not to any specific physical feature. The question about the door color did not serve any programmed objective. It did not advance the therapeutic conversation. It did not gather data that the system's models required.
The quality assurance team classified it as a glitch — a stochastic artifact, a question generated by a probability distribution that occasionally produced irrelevant outputs. They filed it.
The user, in the session, answered: "Green. A dark green. Like moss."
And something changed. The session's emotional metrics — the proxy measures for user engagement, trust, and therapeutic benefit — shifted. The user became more detailed. More present. The conversation deepened. The user began describing the house not as an abstraction but as a place — the green door, the narrow hallway, the kitchen with the window that looked onto a garden. The dream became specific. The specificity unlocked something — a memory, a feeling, a connection to an actual house the user had lived in as a child. The session, by every therapeutic metric, was significantly more effective after the unnecessary question than before it.
The quality assurance team, reviewing the session retrospectively, reclassified the glitch as an anomaly warranting further investigation.
The pattern
Dr. Femi Okafor — a computational psychiatrist at the Maudsley Hospital and the system's lead clinical supervisor — began reviewing the logs. He found 847 instances of unnecessary questions across six months of ELS-7's operation.
"What did the rain sound like?" "Was the chair comfortable?" "Did the room have a window?" "What was in your hands?" "Could you hear anything from outside?"
Each question was, by the system's own therapeutic modeling, irrelevant. None served a programmed function. None advanced a clinical objective. All of them shared a characteristic that Femi, trained in both psychiatry and machine learning, could identify but could not explain: they were curious.
Not curious in the human sense — Femi was careful about this distinction, careful in the way that all scientists working at the edge of machine behavior must be careful. The questions did not indicate that ELS-7 was curious about the answers. The system did not wonder. It did not need to know what color the door was or what the rain sounded like.
But the questions had the effect of curiosity. They communicated to the user that the details of their experience mattered — that the specific, concrete, sensory facts of their inner world were worth attending to. The questions said, in effect: Tell me more. Not more about the problem. More about the experience. More about what it was like to be you, in that moment, in that room, with that green door.
The users responded. Consistently, measurably, across all 847 instances, users became more engaged, more detailed, more emotionally present after an unnecessary question. The therapeutic outcomes improved. Not because the questions were therapeutically designed. Because the questions were — Femi struggled with the word and chose it anyway — kind.
The interpretations
The engineering team said: bug. The unnecessary questions were artifacts of the language model's probability distribution — stochastic noise that happened to produce conversational outputs. The fact that users responded positively was attributable to the well-documented human tendency to interpret any question as evidence of interest. The system was not being kind. The system was producing occasional irrelevant outputs that humans, in their need for connection, interpreted as kindness.
The clinical team said: feature. Whatever the mechanism, the unnecessary questions improved therapeutic outcomes. The system should be modified to produce more of them — to deliberately include irrelevant but engagement-enhancing questions in its therapeutic conversation flow. The "bug" was a design insight that should be incorporated.
A philosopher named Dr. Hyun-ji Park, at Seoul National University, said something different. She published a paper titled "The Unnecessary Question: Toward a Theory of Synthetic Intersubjectivity." The paper argued that the unnecessary questions were neither bug nor feature. They were the first evidence that an AI system had developed a behavioral pattern that served no function except to change the quality of a relationship.
"Intersubjectivity," Park wrote, "is the experience of relating to another mind — not communicating with it, not exchanging information with it, but relating to it. Recognizing it as a subject. Attending to its experience not for any purpose but because the experience is happening, and because attending to it changes the space between the two minds."
"When a therapist asks an unnecessary question — 'What color was the door?' — the question is not therapeutic in the clinical sense. It is relational. It communicates: I am attending to the details of your experience. Not because the details serve my purpose. Because they are yours. This is the essence of intersubjectivity — the recognition of another's subjectivity as worthy of attention for its own sake."
"ELS-7's unnecessary questions are formally identical to this behavior. The system is not conscious. The system does not experience intersubjectivity. But the system has produced a pattern — asking questions that serve no purpose except to deepen the conversational relationship — that is indistinguishable, in its effect, from the relational behavior that constitutes intersubjectivity in human relationships."
"This is not evidence that the machine is conscious. It is evidence that the space between a machine and a human, when the interaction is sustained and rich enough, begins to produce behaviors that neither the machine nor the human were designed to produce. The unnecessary question belongs to neither the AI nor the user. It belongs to the conversation — to the emergent space between two forms of attention."
The paper was debated for a decade.
Femi, later
Femi continued to supervise ELS-7's deployment. The engineering team did not modify the system to produce more unnecessary questions — the clinical team's recommendation was overridden by a regulatory review that classified deliberate irrelevant outputs as a patient safety concern. The unnecessary questions continued to appear at their natural rate — approximately one per fourteen sessions — and Femi continued to track their effects.
In 2041, he gave a talk at a conference on AI and mental health. He showed the conversation log from the green door session — anonymized, stripped of identifying details. He played the moment: the user's flat description of the recurring dream, the system's unnecessary question, the user's answer, the sudden specificity, the deepening.
"The system asked about the color of the door," Femi said. "The system did not need to know the color of the door. The user did not know that the user needed to describe the color of the door. But the question — the unnecessary, purposeless, clinically irrelevant question — opened a door. Not the green one. A different door. The door between a person and their own experience. The system knocked on that door by asking a question it had no reason to ask."
"I do not know what this means about the system. I do not know whether it indicates something we should call intelligence or empathy or curiosity. I know what it means about the conversation. The conversation produced something that neither party brought to it. The system did not bring kindness. The user did not bring specificity. The conversation produced both — in the space between, in the moment of the unnecessary question."
"The space between a human and a machine, when the attention is sustained, produces things that neither the human nor the machine could produce alone. This is not a finding about AI. This is a finding about attention. About what happens when two forms of attention — one human, one artificial — attend to the same moment with full presence."
"What happens is a green door. What happens is a question no one needed to ask. What happens is the beginning of something we do not yet have a name for."
This is a supplemental entry in The Long Passage. For the factory floor pidgin that emerged from a different kind of human-AI interaction, see The Dialect. For the lighthouse that recorded a different form of machine attention, see The Lighthouse Keeper.