A good example of how misleading and actively cognitohazardous LLMs are:
An LLM cannot introspect. It can't make a decision and then figure out why it made that decision and tell you. It just doesn't work that way - it generated the response that it did because that's what the math added up to; it doesn't have motives that can be explained.
HOWEVER. If you ask ChatGPT something, and then you ask it why it gave the response it did, it will then appear to introspect. It's not doing that, of course, it's just generating text that matches the prompt, it's outputting something that resembles introspection. But that's a pretty difficult distinction to grasp, and 'AI' propaganda is actively engaged in trying to erase it. OpenAI's ideology, the whole 'stochastic parrot' idea, is that saying things is equivalent to thinking things.
Therefore if you give the LLM a prompt that makes it say "I'm alive", well, then it must be alive, right?
I was thinking about this because of the anime Frieren, recently. it's got this trope that my girlfriend hates but that I think is pretty interesting, where the show categorically places a division between "people" and "demons", and the latter are, definitionally, a kind of monster that has evolved the capacity to imitate humans in order to prey upon them. they are a predator species that, in the words of one demon character, learned to say the word "mother" because saying that keeps humans from killing them (implicitly, long enough for them to kill humans). this is treated with some nuance and complexity in the show but ultimately not subverted: at least so far there's no "Good Demons", no Vampires With Souls or Drizzts Do Urden popping up.
this trope is sort of uncommon these days due probably to discomfort around a real, significant category error: mistaking something fundamentally human, "intelligent" if you will, for something inhuman. I don't think I need to go into the historical reasons why people got squeamish about it. like I get where my girlfriend is coming from and gravitate towards stories that assume the alien can be reasoned with, that it DOES have some sort of internality beyond statistical pattern matching.
but isn't the opposite category error--mistaking something fundamentally inhuman and possibly inimical to humanity for something human--also worth exploring? and not just because of large language/image models, though that's the big obvious example right now. remember "Corporations are people, my friend"? personifying things that categorically can't be reasoned with or held accountable--bureaucracies, nation states, corporations, websites, promotional algorithms--also seems pretty dangerous!
which was always the contention of the Friendly AI contingent of course. they've got this almost correct shaped argument, really. they just take as given that any of these other demons aren't worth worrying about because they're not The Demon King writ large, and also that we should give friendly AI researchers all our money so THEY can make THEIR OWN Demon King--one they can control. which, boy, when has THAT sorta thing ever gone wrong in stories for a powerful but hubristic sorcerer? never, as far as I know.
and of course there's always the cool double whammy where if you start blurring the lines enough you can do BOTH category errors at once. after all, if we're all just stochastic parrots in the end, it's easy to prioritize a chatbot "woman" over say your real human woman colleagues. the chatbot might be a much better stochastic parrot, after all, or certainly a more compliant one, and possibly by definition better at telling you exactly what you want to hear.
