How should we protect Trust & Safety analysts?

I read with interest Anthropic’s latest job ad for a Trust & Safety Analyst. I think there are a lot of flaws in the technology, but I’m still hopeful about the company over all.

Being an on-the-ground Trust & Safety analyst guy is something I never want to do again, having survived 5 years of it. Despite how grueling it was at times, I’m grateful for the experience, and gained a lot from it both personally and professionally, but absolutely never again. I did my time in the trenches.

One thing that leaps out for me now while perusing job ads like this one:

IMPORTANT CONTEXT ON THIS ROLE: In this position you may be exposed to and engage with explicit content spanning a range of topics, including those of a sexual, violent, or psychologically disturbing nature.

Usually, these job ads for this type of role also tend to stipulate that these positions are on-call with irregular hours. Which means, basically, you have no rest from it. Ever. That’s a recipe for disaster for anyone forced to live that way.

There’s a hidden fundamental flaw in all of this across all industry, whether or not it’s specifically an AI business. It’s never expressed out loud:

If this type of problematic content is so potentially bad and dangerous that companies think they should not casually expose regular users to it in order to keep users safe, why then is it suddenly “fine” and “safe” for a content moderator or Trust & Safety analyst to devote literally all their time to it?

Nobody has ever explained that, or even publicly stated the question – as far as I know – because there is no answer to it. It’s false. It is simply unequivocally *not* safe for the analyst or moderator (or “AI trainer” which is often the same work, but frequently even lower paid) who has to spend all their time exposed to the worst that the socio-technical assemblage of the technology plus human nature can cook up.

So when I see these disclaimers in job ads like Anthropic, I automatically think – as someone who was somewhat scarred from this work – what protections do you offer to compensate for the great personal toll you’re asking people to bear who end up taking up this burden on behalf of the rest of us?

The actual protections offered are never mentioned, because they basically don’t exist either. If you’re dealing with certain categories of illegal images, there may be some simple filters that help blur or flip images, but there may also not be, depending on the company and the tooling they offer to people performing these roles, and how seriously they actually take these risks. Most companies don’t take it all that seriously. (It’s also important to note that it’s not only graphic video or image exposure which can mess you up – sifting through highly objectionable text at scale can do a number on you all the same. Don’t believe me? Try it for five years.)

Often there are vague mentions of “wellness” programs offered for people in these roles. It’s never been clear to me what they actually entail, as I never participated in one. Perhaps they are more helpful than I imagine them to be. The fact of the matter is, I’ve looked around a little, and never seen any mention of what might be effective therapy for current or former moderators suffering from on-the-job related toxicity exposure. I’ve seen mentioned a bit CBT (cognitive-behavioral therapy), but it seems fairly involved and on-going. If it works, is the company going to keep paying for it after you stop doing the job?

Also, is it normal in other fields that you take a job knowing full well that the job is going to force you into a negative mental health space, such where you will basically be required to have to do therapy to continue the job (and maybe after)? Maybe I’m naive, but I don’t think that’s too normal.

So my questions all boil down to one thing: if we agree that it’s useful/necessary to have humans in the loop for making determinations about content toxicity, what should we do to protect them from this highly toxic exposure at scale? What is actually appropriate and effective as both prevention and treatment? Is the human impact cost to individuals who do this work ever even justified? I have more questions than answers here, but at least questions can open up further conversations, if anybody’s listening…