Have written a little bit in the past about AI/ML model cards, and their potential importance in AI attribution. So, I started asking ChatGPT and YouChat (not sure if that’s its “official” name?) to give me information about their underlying models (LLMs), and to point to model cards. And both services seem to make up a variety of responses to these questions, without offering any kind of – so far as I can tell – definitive authoritative responses.
YouChat in particular was hallucinating URLs where it claimed its model card could be found, both on its own domain, and on the Google AI Platform, and it told me different things about its underlying model depending on what questions I asked it, so I took that to mean that the information was unreliable. I didn’t carefully track ChatGPT’s replies (and it was many days ago now), but it was a similar run-around that it gave me.
So I spent a little time earlier sketching out a draft for a proposed set of best practices for how AI & ML tools such as chatbots ought to be able to give accurate and reliable info about their underlying models & their model cards. I published the first version (extremely rough draft) as a Github gist for now until I can spend more time crafting a more comprehensive set of recommendations to replace it.
Eventually, it seems like it would make sense to merge these efforts with the other parallel conventions I’m exploring around markup/markdown attribution and self-identification restrictions for chatbots. This kind of work tends to come in fits and starts anyway, so doing it piece by piece makes sense, as the issues become more plain & potential solutions likewise reveal themselves.