Lots of good stuff in this essay on two modes of failure in tech by Dave Karpf. Briefly:

One way that a technology can fail is that it can work as intended, but at a much larger scale, with unexpected results…

Assuming that an emerging technology works as-intended can be a huge stretch….

The second failure mode prompts an entirely different set of questions. What if the bugs in the emerging technology are not resolved? What if it the market for it grows, and it gets incorporated into critical social systems, but it continues to fail in ways that are increasingly hard to see?

We ought to pay more attention to the second failure mode when imagining the trajectory of AI. I’m not worried about an imminent future of artificial general intelligence. I’m worried about a future where generative AI tools get baked into social systems and wreak havoc because the tech doesn’t work nearly as well as intended.

This second failure mode with regard to AI is also the one that I am concerned about, because it is more the norm in tech, as far as I’ve seen it. Things get bigger, but not necessarily better. In fact, they often get worse, and no one gives enough of a damn to fix it after release, cause fixing bugs isn’t sexy. And acknowledging even that you built around a bad paradigm resting on a lot of faulty assumptions is even less sexy for companies and developers. And, as usual, it’s users and all the people downstream who are negatively affected by the tech who end up getting stuck with the bill.

Related writing & research