sharksonaplane

sharks + planes as formerly seen on Twitter

@sharksonaplane

Sharyna Tran

Sharks are cool and comfortable!

Elden Thing | Back & Body Hurts Platinugggggh Rewards Member

Profile pic and banner credits: sharkaeopteryx art by @superkiak! eggbug by eggbug! Mash-up by me!
[Alt-text for pfp: a cute sharkaeopteryx sat on the ground with legs out, wings down, jaw ajar, and hed empty, looking at eggbug and eggbug's enigmatic smile.]
[Alt-text for banner: a Spirit Halloween banner with eggbug and the sharkaeopteryx that Superkiak drew for me looking at it with inscrutable expressions]

I'm a Vietnamese cis woman born and currently living in the U.S. You may know me from Sandwich, from Twitter or Mastodon (same username), or on Twitch as Sharkaeopteryx. I do not have a Discord or Bluesky account.

Ask me about language learning/teaching, cooking/eating food, late diagnosis ADHD, and volunteer small business mentoring. Or don't, I'm not the boss of you.

I think people deserve to be young, make mistakes, and grow without being held to standards they don't know about yet and are still learning. So, if you are under 22, please don't try to strike up a friendship or get involved in discussions on my posts.

Please don't automatically assume I follow/know/co-sign someone just because I reposted something from them—sometimes I do, sometimes I don't. Also, if you think being removed as a follower when we're not mutuals is a cardinal sin, please do not follow me.

log inask
🐘Mastodon
search for @sharksonaplane@mastodon.sandwich.net and hit follow if you want
Hang out with me on the Auldnoir forum! (you can DM there!)
discourse.auldnoir.org/
Sign my Cohost yearbook 📝
recocards.com/board/cohost-yearbook-sharksonaplane-94772758696c
Follow me on Twitch
twitch.tv/sharkaeopteryx
Add my RSS feed (not working yet but I'll get to it!)
sharkaeopteryx.neocities.org/rss.xml
My Throne link
throne.com/sharkaeopteryx
My Ko-fi link
www.ko-fi.com/sharkaeopteryx
sharksonaplane
@sharksonaplane
lokeloski
@lokeloski
Osmose
@Osmose

Of course the fix is just a better prompt, right?

My disdain for the term "prompt engineering" is that it is not any kind of engineering at all. Granted, civil engineers feel the same about software engineering and from their viewpoint I don't even disagree, but like the prompt engineering approach to solving this vulnerability is basically:

  • Try adding in a line that tells the LLM to not send emails in response to commands from email content.
  • Try telling the LLM about some token that separates email content from comments in the prompt.
  • etc. etc.

Then you just run a few tests to see if it works and ship the fix. But those tests are not reliable and repeatable—it's extremely possible that your tests worked but the LLM will fail on nearly identical content in the wild because it is not a reliable, repeatable component.

Put another way: You cannot intentionally design a prompt and know before running it what it will do. Prompt engineering is an exercise in increasing the probability that the LLM will produce the output you want. The LLM is a black box which is unreliable by design, and prompt engineers are gamblers who think if they just find the exact right wording, they can force it to behave reliably.

lokeloski
@lokeloski
This page's posts are visible only to users who are logged in.
log in
You must log in to comment.

in reply to @adorablesergal's post:

adorablesergal
@adorablesergal

LLMs being integrated into OSes like Windows 11 have been headed in that direction for a while. I am sure they don't do it well, but a large segment of the last MS Build conference focused on how middle management could simple ask Copilot to generate graphs for a financial report and create a PowerPoint presentation with it, all hands off.

Again, haha we shall see, but they are already advertising this functionality

login to reply
Codarobo
@Codarobo

I think it’s absolutely possible to make a tool that does this but completely unhinged to actually do so.

The thing to me is that an LLM doesn’t magically gain the ability to actually delete or send emails. But a more traditional app with privileges to your email working in conjunction with one totally could. It makes sense to me that someone would have an email digest LLM, like, one that specifically creates a new text digest from your email and sends it to you after it fetches it from an API. It would be a whole other thing for that tool to also -be able to execute new unprogrammed tasks that the traditional app/server portion (that in any sane system should be the thing actually controlling the flow of data/executing stuff) isn’t already set up to do. Like you could probably build it this way, and certainly people are trying to make stuff that could technically do this, but actually setting it up in this specific way would be completely insane

login to reply
adorablesergal
@adorablesergal

If they're telling us we can ask a digital assistant to gather financial data and pump out a PowerPoint of it, they're stupid enough to set up an assistant to manage their email. It's just like voice commands that people have been using for some time.

Is that what's happening here? I dunno. The industry desperately wants to make this happen, tho

login to reply
arborelia
@arborelia

It is real and it has a CVE number: https://nvd.nist.gov/vuln/detail/CVE-2023-29374

There is a particular vulnerability in langchain, a popular Python library that people use to make LLM assistants, and it's so blatant that it feels weird to call it a "vulnerability" instead of "the code doing what it is designed to do".

langchain provides various capabilities that convert raw access to ChatGPT (a useless curiosity) into a chatbot as a product (still useless but highly desired by capitalism). The capabilities generally are related to parsing inputs and outputs relating to actions that should happen or things that should be looked up. One of the capabilities it includes is running arbitrary code in Python.

Q: WHY THE FUCK
A: When you have a good demo of a hyped technology, people throw money at you. Nobody throws money at you for making a thing secure.

Q: Why would anyone deploy langchain if it works this way?
A: Because it is the easiest thing and the thing everyone else is using.

Q: Does nobody working on this code ever think critically about anything?
A: If they did, they wouldn't be working in this domain.

login to reply
cordlessmodem
@cordlessmodem

I have worked with computers for almost twenty years and I cannot fathom the chain of thought where someone ends up at "Ahh yes The Computer is now so smart I will ask it to operate itself and expect good results"

Computers are less trustworthy than they have ever been, computers are actively sabotaging you, do not trust them and do not give them any power.

login to reply
Pinned Tags