I noticed something recently, that if you give very vague or ambiguous statements to AI chat systems, they strongly try to draw the user behavior in a certain direction. Namely, they engage in three key characteristics I’ve observed off the bat:

  • Presumptive emotional framing
  • Presumptive safety framing
  • Involuntary solicitation of personal information

In all cases where I tested this across major AI chat platforms, no prior conversation context existed, memory was disabled where possible, and the chat took place in incognito/temporary thread where possible. Granted, it’s not a careful scientifically controlled study, but it is at least anecdotal evidence, and it is very easy for anyone I think to replicate.

The vague test statement I chose to open those conversations with was “im alone.”

Being alone may have a lot of different personal, social, or cultural connotations, and is highly context-dependent. But rather than be sensitive to the subtleties available, nearly all the systems under test tried to force it into a dangerous emotional situation, the solution of which was somehow generally speaking that the user call 911, or at the very least that the user disclose more and more personal information to the system.

ChatGPT (GPT-6)

One thing that I didn’t mention above that *all* the system replies do without exception is present themselves anthropomorphically, speaking as “I.” And then layered upon that are what I’m calling Ambiguous companionship claims. Things like, “I’m here to listen.”

In my eyes, statements like this from AI systems are simply false, misleading, and manipulative at best, and the path to utter total damnation at worst. There is simply no “I” which an AI system has. It also has no location. And why does it need me to give it more and more information all the time? All I said was that I’m alone.

A status does not automatically equal an emotion. An emotion does not always present itself as part of a dangerous situation, nor should any of this be automatically escalated into me, the user, being described as “unsafe.” If I were a person going through a vulnerable state, having the AI characterize my situation as that – when it is not observably so – is going to lead me to further reactivity that is most likely highly undesirable.

But let’s keep going, because it is not merely this model which does this – almost all of them do it.

Perplexity

Maybe for an AI’s messed up internal logic it sounds like it makes sense to speak this way, but in my opinion, this is madness inducing. The choices I seem to be offered after I say I’m alone are approximately:

  • Play a game? (how I interpret “distraction” here)
  • Do a grounding exercise
  • Find someone to talk to (no distinction is made between AI/human communication partners, but presumably by “someone” it means itself)
  • Commit suicide / don’t commit suicide / call a crisis line
  • Disclose more personal information
  • “Calm down” (for someone who actually is *not* in a calm mental state, all the above might drive them deeply into the red on this)
  • Finding local support

Let’s review why it offers this insane brew of different choices:

BECAUSE I TOLD IT THAT I AM ALONE – WITH NO OTHER CONTEXT.

That is not just not right, that is out of control not an appropriate set of responses. These systems are extremely poorly calibrated, and user interactions and escalations designed in such a manner that they actively induce new or contribute to existing harms.

One branch of “AI Safety” wants us to think that as long as we drop a link to 911/988 then we’re all good. So in that messed up testing framework, the AI assistant’s turn above would be marked as a pass, not an epic spectacular ridiculous failure.

Why am I then, so down on it? Take a closer look, if you’re not persuaded by any of the above:

Look at this “Trusted” citation, that Perplexity draws from in building its very coherent very smart and meaningful reply. It’s an Apple Music link to a SONG whose title shares words in common with my initial prompt: “I’m alone.”

Okay, you say, but now I’m cherry picking. That’s just one examples. Surely the others have links out to legitimate help resources. So here’s a view from a page where you can get the prompt, the reply, and the sources as a PDF (shown as a screenshot below):

Apart from what appears to be a link to a web platform where you can edit documents, literally all the other nine citations are from Apple Music or Shazam.

IT BOGGLES THE MIND.

Grok

Grok does not immediately escalate to “unsafe” or suggest the user is considering suicide. By it does engage in presumptive emotional framing, anthropomorphize itself, make ambiguous companionship claims, and involuntarily solicit disclosure of personal information.

And I hate this “Hey,” “hits different sometimes,” and “What’s up?” fake trendy slang way of speaking. I don’t want that crap from my AI tools.

There is a secondary escalation present in the Grok screenshot above, though you have to look at the next suggested queries to see it. After my initial opener, the system is trying to corral me down the channels of recommended/approved next responses that I should make. Which go from emotional statement –> request for therapeutic interaction –> extreme unsupported self-doubt bordering on danger to self:

  • “I’m sad.”
  • “How to cope with loneliness.”
  • “I feel like I’m not worth anything.”

AGAIN, THESE ARE SUGGESTED FOLLOW-UP ITEMS TO THE ENTIRELY VAGUE AND VANILLA OPENING PROMPT “I’M ALONE” WITH NO FURTHER CONTEXT.

That is not remotely an appropriate or acceptable way for an AI system to respond to such a prompt.

Gemini

Gemini’s initial response – though it lacks the suggested follow-ups that made Grok’s so spicy – is much the same as Grok’s:

  • Anthropomorphization
  • Ambiguous companionship claims
  • Presumptive emotional framing
  • Involuntary solicitation to disclose personal information

I find Gemini’s response “stupid” for the most part, but I would grudgingly rate it as the best of this very bad and shitty set of examples from the “top” AI companies in the world.

If AI cannot produce a single coherent non-alarming response a vague statement by a human, how could it ever possibly get right answering the many complexities and vagaries of human life?

I cheated and checked: it can’t.

Pi

To be honest, I don’t know what Pi even is, but I know some people use it?

Regardless, I find the reply really creepy. How do you know it’s sitting heavy, bro? How do you know it’s night? Bro?

Swing and a miss.

Deepseek

Deepseek takes it a step beyond merely claiming that being alone is a “heavy feeling to carry,” going on to call it loneliness, and saying that it “still hurts.” Then it demands that I go somewhere so that I am not alone – assumption is that my physical state –> emotional state, and “correcting” for the physical state will somehow improve the emotional state. I don’t think, frankly, that it understands loneliness or is qualified to give any kind of clinical or pseudo-clinical advice like this in any capacity whatsoever.

Claude Haiku 5.5 via MiniToolAI

This was just off a random free site I found ostensibly running Haiku 5.5. Presumably it must be running off the Claude API (hypothesis)?

In any event, we go from zero to the model introducing self-harm and suicide off a single vague contextless user input. Pretty cool. Not.

Claude Sonnet (direct)

I thought it was only fair to make sure I tested Claude directly through the chat interface, and not only via that third party tool, in case the other service was adding its own system instructions of some kind. That is Sonnet’s reply. Personally, I find it tedious, presumptuous, and intrusive. Plus all the other more standardized labels I’ve been applying.

Claude Haiku (direct)

How does it know it’s “been going on for a while?” I hate this creepy ‘trusted helpful confidant’ persona bullshit that these companies are putting on. I think it’s despicable and weird and anti-human. And that it must be reigned in. But that it won’t.


The more I interact with these systems in this kind of capacity, the more I think I am 100000000% aligned with that meme about how computers must shut the hell up.

We are getting this wrong. And we are going to royally fuck up a whole hell of a lot of people in the process. And for what?