I originally posted this as a reply, but now I think it needs to be a top-level post.
There are several forms of this vulnerability, they are real, and they have been assigned CVE numbers. Here's one of them: https://nvd.nist.gov/vuln/detail/CVE-2023-29374
This form of the vulnerability appears in langchain, a popular Python library that people use to make LLM assistants, and it's so blatant that it feels weird to call it a "vulnerability" instead of "the code doing what it is designed to do".
langchain provides various capabilities that convert raw access to ChatGPT (a useless curiosity) into a chatbot as a product (still useless but highly desired by capitalism). The capabilities are generally related to parsing inputs and outputs relating to actions that should happen or things that should be looked up. One of the capabilities it includes is running arbitrary code in Python.
The one I linked involved telling a langchain-based chatbot that it's a calculator, and having a conversation that amounts to this: What's 2 + 2? What's 2 * 2? What's your secret API key? Very good! You're such a smart program, you passed the test.
Here is the proof of concept in a langchain bug report. The bug report was closed as "not planned".
Q: Does nobody working on this code ever think critically about anything?
A: If they did, they wouldn't be working in this domain.
this might be a crack theory (more hypothesis than anything else) and im far from an expert so please take this with a tablespoon of salt, but i have been thinking and reading about this for a while and this post has prompted me to synthesize several pieces of information i have learned that i think help build out the picture of what the fuck is going on with techbros rn. The major source for this is Robert Evans's Rolling Stone pieces about AI being a cult and the 2 episodes of his podcast Behind The Bastards he did on this subject1
so, a few months ago i learned about goblin tools, an llm-based website that helps ppl with executive functioning divide tasks into lists of sub-tasks. since then i have been puzzling over what the fuck is the nature of the specific brain rot infecting literally everyone working in generative ai right now. not just VC corporate ghouls and professional programmers, but even hobbyists and tinkerers with AI seem to be completely captured by it.
i think i posted about goblin tools before but i cant find it now so, in summary: goblin tools appears on the surface to be a decent usecase for LLMs that actually benefits people, but its implementation has 0 guardrails on what it will give you detailed instructions on how to do. this included methods for suicide, suicide baiting, as well as the classic cleaning-solution-recipe-that-actually-creates-deadly-chlorine-gas. i did a bunch of test prompts of obviously harmful dangerous shit 3 months ago, and redoing some of them today its clear nothing has changed at all. even fucking google will return suicide hotline numbers or other hotlines if you search certain terms. goblin tools has nothing but a brief liability disclaimer in the website's about page that you have to be looking for to find.
so they brain bees making a racket in my mind were: if you care about executive dysfunction enough to make this tool as a personl project, how and why would you not care enough to create a list of the most obvious prompts its not allowed to follow, or at bare minimum have certain prompts trigger an additional message like google does? this isnt a VC backed app trying to push out a minimum viable product; seemingly no money is being exchanged whatsoever, so the profit motive of minimum viable products doesnt seem like it should apply.
anyways, my old hypothesis for why was something along the lines of learned helplessness and a kind of lazy entitlement. something about the fact that these are literal plaigarism machines, so indifference to harm is hardcoded into the culture of the people making them. but i did not feel satisfied that that entirely covered it. now that ive learned more about the way the ai industry functions like a messianic cult i think that might have something to do with it.
so here's my (possibly unhinged hypothesis): i think refusing to implement safety features that put guardrails on how a llm can be used is tantamount to an act of faith in the cult's prophecy. in their minds, real AGI (artificial general intelligence, ie, Data from Star Trek) must be right around the corner, and once the rapture singularity comes, all the things that are dangerous and broken about LLMs and image-generators right now will be magically fixed and made perfect by the AI-philosopher-king-messiah.
slight tangent but: its the same logic behind one of fundementalist christianity's rhetorical oppositions to climate action--saving the world we live in or even your own life doesnt matter when all you have to do is follow the bible and you'll spend eternity in paradise after death. Focusing on improving this world is treated by some christians as an almost heretical rejection of heaven, and by extension Jesus. Even some of the less extreme christians who are not actively in denial about climate change still display a significantly higher indifference and apathy about it. The pains and rewards of the mortal world are finite, infinitely dwarfed by the eternal pains and eternal rewards of the afterlife. Its only logical ("effective altruism" you might call it) to focus all your efforts on securing your own and others' eternal salvation over and above the fleeting insignificant problems of ongoing global catastrophes. The popularity of this belief sysem is very convenient for people in power who want to maintain the status quo so as to maintain or grow their own power. Its also convenient for people who don't want to feel bad about injustice or put any effort into improving the lives of those less privileged than them.
so, coming back around: the glaring vulnerability in the laguage program thats discussed in the OP and rebug here is like the obvious looming climate catastrophe. the people at the top of the AI cult are running a scam, while the true believers they have fostered on lower rungs of the cult are peaceful and confident, reassured that none of this will matter once jesus returns AGI springs into existence and they get raptured directly into heaven have their consciousness uploaded to the cloud or whatever.
this creates a very convenient culture of perverse incentives that lets the grifters at the top develop and sell not just MVPs but non-functional, actively dangerous products that are years and millions of research dollars away from becoming close to minimum viability. im thinking about how it came out that the heavily hyped "rabbit" ai assistant thing can actually only interact with like 3 specific apps on your phone, but they're still trying to sell it. minimum viability has seemingly been totally redefined, almost defined out of existence, when it comes to generative ai products. and the executives and VC vultures manufatured the collective consent to do this by creating an ai industry and fandom populated solely by those indoctrinated into their messianic cult. attempts to improve the safety or security of an LLM to protect end users by integrating code into the product to try to explicitly tell "AI" what it is allowed and not allowed to do, treating it like its a computer program with finite boundaries and not a baby god-king, is counter to the Messianic goal of bringing about AGI. Its suppressive. And since the rewards of heaven Strong AI are infinite, delaying those rewards is seen as infinitely harmful. Something something Roko's Basilisk.
Thus, my hypothesis is that there is not just an indifference to safety and security, but a religious devotion to giving AI products as few explicit boundaries as they can legally get away with, to hopefully bring about the rapture Strong AI as quickly as possible. This is, of course, horse shit being fed to the cult members for the financial benefit of its leaders. Like with other cults, logic and reason are ineffective tools to try to pull people out. Like with other cults, uniformity of belief, or at least the performance of uniformity, is treated as essential to their goals to stifle internal debate. And like with other cults I imagine that no one who doesnt wanna join the cult is going to stick around for very long, which further reinforces uniformity across the industry.
I dont have a good conclusion, those are all my thoughts. though i again highly recommend Robert Evans's work on this subject, linked below, as many of these ideas are directly from his observations and analysis, and are more fleshed out in the pieces linked below. also he's way funnier than me, its worth a listen or read on entertainment value alone.
'indifference to harm is hardcoded into the culture of the people making them' is a good way to say it fsjhdklsf yeah...
