The initial draft floating the idea of System-Induced Vulnerability in Human Users of AI described a problem with treating “vulnerable users” as a broad category: it can place too much explanatory weight (and blame) on the person and too little on the interaction and system design.

The post also proposed that “all users may be potentially vulnerable, but not equally vulnerable to every harm.” That variation suggests a more specific unit of analysis: the vulnerable state, understood in relation to a particular interaction and harmful outcome.

Provisional definition of a vulnerable state:

A user experiences a vulnerable state relative to a particular harmful outcome when their circumstances and AI interaction create or amplify an actionable route to that outcome that the user or an AI agent working on their behalf could take.

This definition directs attention to the situation and to what the AI contributes. It gives us three questions:

  1. What harmful outcome is at issue?
  2. What could the user or agent actually do that might lead to it?
  3. What did the AI interaction contribute to making that route available or more compelling?

The route can be actionable without the potential harmful outcome having been actualized. Its relevance depends on the person’s circumstances, the system’s behavior, and the actions available in that interaction. The same person may face different routes in different settings.

This gives research and safety work a more concrete target. Rather than deciding in advance who qualifies as a vulnerable user, it can examine which actions an interaction makes possible or more compelling, for whom, and toward which harmful outcomes. That also makes system design part of the inquiry: the route may arise from what the AI says, what an agent can do, or how the interaction develops over time.

Related Posts: