[A half-baked AI-assisted & human-edit follow-up to System-Induced Vulnerability in Human Users of AI]

The previous post described system-induced vulnerability as increased human susceptibility to consequential harm produced or amplified by AI systems over time through repeated interactions and bidirectional feedback loops. Taking after much of the literature on this topic, which also frame the problem in that manner, several of its examples involved situating a harm in relation to mistaken premises or distorted understandings of reality. Those remain relevant, but they do not exhaust the ways these human-AI interactions can contribute to harm.

Importantly: a system does not need to distort someone’s understanding of reality to contribute to harmful changes in their behavior or functioning.

Engagement itself as a mechanism

In their 2026 paper, “Playing with the dials of belief,” Hamilton Morrin and colleagues explain:

…we are aware now of cases where the most detrimental feature of the spiral is the compulsive use pattern rather than the resultant belief state. Resembling manic episodes, these presentations centre on a co-produced project of great subjective importance, accompanied by expansive mood, but not necessarily by clearly delusional content, and occurring to the detriment of eating, sleep and social relations.

For the system-induced vulnerability framework, this suggests that harm can emerge from the dynamics of engagement itself, independently of whether the AI’s claims or the user’s beliefs are true or false.

A project need not be based on false premises for engagement with it to become detrimental. Its subject matter may be ordinary, and the work may have actual value. The relevant question is how the interaction develops and what consequences follow from it.

Harm becomes identifiable when engagement contributes to meaningful impairment, distress, or adverse consequences.

Epistemic and behavioral mechanisms

It may be useful to distinguish epistemic mechanisms, affecting belief and interpretation, from behavioral mechanisms, affecting motivation, engagement, and action. Neither category is inherently harmful. Changing an understanding or sustaining effort on a project is not, by itself, evidence of harm.

The concern is how these mechanisms can contribute to increased susceptibility to consequential harm. Repeated affirmation may increase confidence in an interpretation. Continued encouragement may sustain engagement long after it has begun to interfere with ordinary functioning. These effects may occur together, but harmful engagement does not require false beliefs, delusions, or psychosis.

For the system-induced vulnerability framework, this means distorted beliefs are one possible mechanism, not the harm itself. The central concern remains how AI interactions produce or amplify susceptibility to consequential harm, and at what thresholds and in what forms do those harms arrive.

Extending the pipeline of harm

The first post outlined one possible progression from an unsupported belief through reinforcement, reliance, operationalization, and apparent corroboration. That remains one possible route, rather than a template every harmful AI spiral interaction must follow.

A behavioral route may instead develop through increasing investment in a shared project, repeated encouragement, and growing difficulty disengaging. The resulting harm may involve neglected needs or relationships, without requiring the project’s premises to be mistaken.

Agentic capabilities can extend either route by turning plans into consequential real-world actions. The results of those actions may appear to validate an interpretation or encourage further engagement, feeding back into the interaction.

A cumulative pipeline of harm therefore needs to account for both what people come to believe and how their engagement develops. Individually permissible exchanges may contribute to adverse consequences over time. The object of investigation remains the relationship between system behavior, increased susceptibility, and consequential harm.

Related Posts: