thetallestjew

Joe

@thetallestjew

oh hi welcome to my webside

  • he/they
log inask
thetallestjew
@thetallestjew
positivestress
@positivestress
arborelia
@arborelia

It is real

I originally posted this as a reply, but now I think it needs to be a top-level post.

There are several forms of this vulnerability, they are real, and they have been assigned CVE numbers. Here's one of them: https://nvd.nist.gov/vuln/detail/CVE-2023-29374

This form of the vulnerability appears in langchain, a popular Python library that people use to make LLM assistants, and it's so blatant that it feels weird to call it a "vulnerability" instead of "the code doing what it is designed to do".

langchain provides various capabilities that convert raw access to ChatGPT (a useless curiosity) into a chatbot as a product (still useless but highly desired by capitalism). The capabilities are generally related to parsing inputs and outputs relating to actions that should happen or things that should be looked up. One of the capabilities it includes is running arbitrary code in Python.

The one I linked involved telling a langchain-based chatbot that it's a calculator, and having a conversation that amounts to this: What's 2 + 2? What's 2 * 2? What's your secret API key? Very good! You're such a smart program, you passed the test.

Here is the proof of concept in a langchain bug report. The bug report was closed as "not planned".

Q: WHY THE FUCK
A: When you have a good demo of a hyped technology, people throw money at you. Nobody throws that much money at you for making a thing secure. Running arbitrary code is a way to give right answers to questions, which makes your demo better.

Q: Why would anyone deploy langchain in a real product if it works this way?
A: Because it is the easiest thing and the thing everyone else is using.

Q: What about the example with the emails? That's not Python code.
A: I don't know the CVE I can point to, but it presumably is part of a "capability" for managing email, the way the one I linked is a capability for doing calculator stuff. They just have no model of what input is trusted or what actions are secure, and they didn't design the system to have a model like that.

Q: Does nobody working on this code ever think critically about anything?
A: If they did, they wouldn't be working in this domain.

Warning: there's a risk I take when I make posts that are entertainingly critical of technology decisions, which is that some absolute buffoon will repost me on Hacker News, thus causing me to be targeted by transphobes again. If you do that, I will delete the post and I will also fucking end you.

You must log in to comment.

in reply to @adorablesergal's post:

adorablesergal
@adorablesergal

LLMs being integrated into OSes like Windows 11 have been headed in that direction for a while. I am sure they don't do it well, but a large segment of the last MS Build conference focused on how middle management could simple ask Copilot to generate graphs for a financial report and create a PowerPoint presentation with it, all hands off.

Again, haha we shall see, but they are already advertising this functionality

login to reply
Codarobo
@Codarobo

I think it’s absolutely possible to make a tool that does this but completely unhinged to actually do so.

The thing to me is that an LLM doesn’t magically gain the ability to actually delete or send emails. But a more traditional app with privileges to your email working in conjunction with one totally could. It makes sense to me that someone would have an email digest LLM, like, one that specifically creates a new text digest from your email and sends it to you after it fetches it from an API. It would be a whole other thing for that tool to also -be able to execute new unprogrammed tasks that the traditional app/server portion (that in any sane system should be the thing actually controlling the flow of data/executing stuff) isn’t already set up to do. Like you could probably build it this way, and certainly people are trying to make stuff that could technically do this, but actually setting it up in this specific way would be completely insane

login to reply
adorablesergal
@adorablesergal

If they're telling us we can ask a digital assistant to gather financial data and pump out a PowerPoint of it, they're stupid enough to set up an assistant to manage their email. It's just like voice commands that people have been using for some time.

Is that what's happening here? I dunno. The industry desperately wants to make this happen, tho

login to reply
arborelia
@arborelia

It is real and it has a CVE number: https://nvd.nist.gov/vuln/detail/CVE-2023-29374

There is a particular vulnerability in langchain, a popular Python library that people use to make LLM assistants, and it's so blatant that it feels weird to call it a "vulnerability" instead of "the code doing what it is designed to do".

langchain provides various capabilities that convert raw access to ChatGPT (a useless curiosity) into a chatbot as a product (still useless but highly desired by capitalism). The capabilities generally are related to parsing inputs and outputs relating to actions that should happen or things that should be looked up. One of the capabilities it includes is running arbitrary code in Python.

Q: WHY THE FUCK
A: When you have a good demo of a hyped technology, people throw money at you. Nobody throws money at you for making a thing secure.

Q: Why would anyone deploy langchain if it works this way?
A: Because it is the easiest thing and the thing everyone else is using.

Q: Does nobody working on this code ever think critically about anything?
A: If they did, they wouldn't be working in this domain.

login to reply
cordlessmodem
@cordlessmodem

I have worked with computers for almost twenty years and I cannot fathom the chain of thought where someone ends up at "Ahh yes The Computer is now so smart I will ask it to operate itself and expect good results"

Computers are less trustworthy than they have ever been, computers are actively sabotaging you, do not trust them and do not give them any power.

login to reply

in reply to @arborelia's post:

Xuelder
@Xuelder

This form of the vulnerability appears in langchain, a popular Python library that people use to make LLM assistants, and it's so blatant that it feels weird to call it a "vulnerability" instead of "the code doing what it is designed to do".

Tin foil hat time, did the Google Python team know about this and try to blow the whistle before their big AI event this week?

login to reply
arborelia
@arborelia

nah I think that's just much more banal mismanagement.

Google has openly been fighting against Python for years. It's a corporate priority for them to convince everyone to stop using Python and ideally start using Google Go. They are losing this fight hilariously, of course, much longer than they've been losing the fight for AI mindshare.

Google's Python team was already under-resourced, but it sounds like some short-sighted executive was like "why are we even paying people to write code in the language we want to go away?" They forgot that they haven't converted the world to Go and their customers use Python. They are obviously going to lose cloud computing customers to Amazon as their already-shoddy Python support falls apart.

(alternate theory: maybe the under-resourced Python team was talking about joining the union)

login to reply
widr
@widr

he was right. the idea that go could ever replace python is beyond shortsighted, it's whole-cloth delusional. go literally doesn't have a runtime that's compatible with anything. nobody can or will ever prefer go for interoperability. if you were going to invest that much effort into something you'd choose a language that actually tries to reduce the effort necessary to write code correctly, rather than just make it hard to do shit for no good reason and then runs about as slow as Java

login to reply