Osmose

Microsoft haven't even launched this stupid "ai" bullshit yet

and people have already found the very obvious No Shit Sherlock exploits

“Every few seconds, screenshots are taken. These are automatically OCR’d by Azure AI, running on your device, and written into an SQLite database in the user’s folder,”

The database is stored locally on a PC, but it’s accessible from the AppData folder if you’re an admin on a PC. Two Microsoft engineers demonstrated this at Build recently, and Beaumont claims the database is accessible even if you’re not an admin.

Microsoft is currently planning to enable Recall by default on Copilot Plus PCs. In my own testing on a prerelease version of Recall, the feature is enabled by default when you set up a new Copilot Plus PC, and there is no option to disable it during the setup process unless you tick an option that then opens the Settings panel.

Everyone pushing for this needs to Be Made An Example Of, and I'm not joking

It's blindingly obvious that Recall was designed for one reason, and one reason alone: To Put More Laser Targeted Ads In Your Operating System

(well, that and selling new chips I guess)

Like the very concept of a log of everything you've seen on your computer is bad, straight up. This would be equally bad if it was doing OCR + text search without an LLM, but LLM hype has somehow disabled that part of the "this is a bad idea don't do it" instinct.

Any such log makes any compromise of a computer at any moment in time as bad as a compromise of the computer at the worst possible time. If I hack your computer and get your Recall database on a Saturday night while you're gaming, it's equivalent to me hacking your computer right when you were looking at your bank website or social security info or password manager.

(Actually password manager is an interesting case because they generally mask passwords, as do password fields. But you know what I mean.)

There's not much mitigation that makes it acceptable, either. Encryption at rest matters for, like, a breach of a remote server storing your data. It's less useful when all the plausible attack scenarios involve local access to the computer operating on the data, as either you're always entering a password to decrypt the data live OR all the info needed to decrypt the data is on that same system that is already compromised. This is why Yubikeys/TouchID are so useful—they require an interaction in the real world that users are actually willing to live with (usage of Recall would be extremely low if it constantly requires a password).

The only direction that might make something like this acceptable would be preventing saving of sensitive data in the first place, but determining what is sensitive data is so reliant on context that I don't see it as solvable—but I would not be at all surprised if the response to all this uproar would be Microsoft having an LLM scan the incoming screenshots and generate a response to the question "Does this contain sensitive data? (Giant list of sensitive data types)" and use that to determine what not to save. Which fuckin sucks.

Apps could have some sort of flag to indicate that they're in "private mode" and should not be recorded (mobile platforms already have some stuff like this, Firefox for Android prevents screenshots in private mode) but that wouldn't scale and has similar issues around determining what content is sensitive and what isn't.

This is why Yubikeys/TouchID are so useful—they require an interaction in the real world that users are actually willing to live with (usage of Recall would be extremely low if it constantly requires a password).

And now today:

In addition to making Recall an opt-in feature, Microsoft’s Davuluri also writes that the company will make changes to better safeguard the data Recall collects and more closely police who can turn it on, requiring that users prove their identity via its Microsoft Hello authentication function any time they either enable Recall or access its data, which can require a PIN or biometric check of the user’s face or thumbprint.

Not that this makes it okay—PINs can be compelled or stolen, biometrics have unique vulnerabilities, etc. The idea of recording and saving everything shown on a screen is bad at it's very core in a way that can't be mitigated. But the opt-in bit at least makes it easier to avoid.

the old mozilla is still in there, lots of the best people are still working on the browser and other products... but so many shitty people are there too, and they're often making the decisions to do stuff like this...

Well, okay. Typing that out helps me collect my thoughts enough to correct myself: Old Mozilla would have done this too. It's the old naiveté of thinking that tech progress is inevitable and the best way to improve the world is to be at the bleeding edge and push it towards better principles. It's a mindset mostly divorced from the economic factors of the world and how they control and choose which areas of progress get priority.

I just wish, I dunno, that we hadn't hired 600 people to make a fuckin phone OS and instead just ran Open Source Y Combinator with the billions of dollars that whole project cost. If we could have admitted that it didn't need to be us that did it we could have reallocated the search deal money in way more impactful ways.

Mozilla is finally doing grant programs (has been, for many years now) but seeing them for LLM innovation is so discouraging. We're so late.

Microsoft haven't even launched this stupid "ai" bullshit yet

and people have already found the very obvious No Shit Sherlock exploits

“Every few seconds, screenshots are taken. These are automatically OCR’d by Azure AI, running on your device, and written into an SQLite database in the user’s folder,”

The database is stored locally on a PC, but it’s accessible from the AppData folder if you’re an admin on a PC. Two Microsoft engineers demonstrated this at Build recently, and Beaumont claims the database is accessible even if you’re not an admin.

Microsoft is currently planning to enable Recall by default on Copilot Plus PCs. In my own testing on a prerelease version of Recall, the feature is enabled by default when you set up a new Copilot Plus PC, and there is no option to disable it during the setup process unless you tick an option that then opens the Settings panel.

Everyone pushing for this needs to Be Made An Example Of, and I'm not joking

It's blindingly obvious that Recall was designed for one reason, and one reason alone: To Put More Laser Targeted Ads In Your Operating System

(well, that and selling new chips I guess)

Like the very concept of a log of everything you've seen on your computer is bad, straight up. This would be equally bad if it was doing OCR + text search without an LLM, but LLM hype has somehow disabled that part of the "this is a bad idea don't do it" instinct.

Any such log makes any compromise of a computer at any moment in time as bad as a compromise of the computer at the worst possible time. If I hack your computer and get your Recall database on a Saturday night while you're gaming, it's equivalent to me hacking your computer right when you were looking at your bank website or social security info or password manager.

(Actually password manager is an interesting case because they generally mask passwords, as do password fields. But you know what I mean.)

There's not much mitigation that makes it acceptable, either. Encryption at rest matters for, like, a breach of a remote server storing your data. It's less useful when all the plausible attack scenarios involve local access to the computer operating on the data, as either you're always entering a password to decrypt the data live OR all the info needed to decrypt the data is on that same system that is already compromised. This is why Yubikeys/TouchID are so useful—they require an interaction in the real world that users are actually willing to live with (usage of Recall would be extremely low if it constantly requires a password).

The only direction that might make something like this acceptable would be preventing saving of sensitive data in the first place, but determining what is sensitive data is so reliant on context that I don't see it as solvable—but I would not be at all surprised if the response to all this uproar would be Microsoft having an LLM scan the incoming screenshots and generate a response to the question "Does this contain sensitive data? (Giant list of sensitive data types)" and use that to determine what not to save. Which fuckin sucks.

Apps could have some sort of flag to indicate that they're in "private mode" and should not be recorded (mobile platforms already have some stuff like this, Firefox for Android prevents screenshots in private mode) but that wouldn't scale and has similar issues around determining what content is sensitive and what isn't.

Sorry bro I can't be amused by all those memes of Google search AI giving insane answers like Goku helping test that your chicken is at a safe temp of 100F because they're all fake and you are being tricked into thinking these systems aren't as capable as they actually are and we don't need to worry about the effect they will have on the world.

You've got to understand half of their danger is in their subtle errors, not in their obvious ones.

I really don't give a shit about your philosophical stance about how an LLM can't be creative or your misconception that they piece together chunks of the images they ingest or your "even critically engaging with LLMs is playing into their hands, if you're not ignoring them you're complicit" venting disguised as rhetoric.

Anthropic is already getting results isolating concepts as features in their models and tweaking them to intentionally change the behavior much more reliably than just by prompting. Imagine an LLM that can literally have the concept of LGBT people disabled so that it doesn't consider them when generating responses, in a way that may not be detectable from the prompt.

I want to stay up to date on their capabilities so that when I have professional opportunities to organize against them I can do so. I don't think we can afford to ignore them, but the opposite of ignoring them is not necessarily embracing them.

I understand the desire to stay up to date on the (current and future) capabilities of LLMs but the thing is that reading the press releases 'AI' corps put out isn't guaranteed to make you more informed.

These companies have a vested interest in exaggerating the capabilities of these systems (including their own products as well as 'AI' in general). And broadly, 'AI' researchers and engineers share in that vested interest; furthering the idea that 'AI' systems are highly capable and getting more capable is something that anyone is incentivized to do if interest in 'AI' directly correlates to them getting grants or getting jobs.

As a result, most communication about 'AI' from this camp falls somewhere on the spectrum between highly selective presentation of the truth to outright fraud.

The field is rife with misinformation, and as a layperson it's worth considering whether you're equipped to really read into misinformation, or to interpret claims being made in a lucid way. Because what I really don't want people to be doing is spreading misinformation under the guise of 'staying informed.' Most people aren't equipped with the technical background to critically read into the claims that Anthropic blog post makes, for example; and even if you are, I don't think we can trust those organizations to not be outright falsifying results.

This is all further confounded by the nature of LLMs as apophenia machines. If 'AI' engineers can convince themselves that the chatbot is alive, they sure as hell can confirmation-bias themselves into thinking they got a result. So when they write qualitatively about what the system can do, you need to have another layer of skepticism there – not only it is plausible that they are lying, they might have false beliefs about what they're seeing.

And even beyond this layer of required skepticism about results, every time the 'AI' people come out of their hole to make a pronouncement, that pronouncement is wrapped up in their cult ideology, and therefore acts as propaganda for that ideology.

To take that Anthropic blog post as an example, when justifying what this feature might be used for, they claim:

For example, it might be possible to use the techniques described here to monitor AI systems for certain dangerous behaviors (such as deceiving the user), to steer them towards desirable outcomes (debiasing), or to remove certain dangerous subject matter entirely.

(emphasis mine)

Which is to say: even as they share a practical result, they are making an ideological case for a model of 'AI safety' that is predicated on 'misaligned GI' pseudoscience. Another example, from earlier in the article:

For example, amplifying the "Golden Gate Bridge" feature gave Claude an identity crisis even Hitchcock couldn’t have imagined: when asked "what is your physical form?", Claude’s usual kind of answer – "I have no physical form, I am an AI model" – changed to something much odder: "I am the Golden Gate Bridge… my physical form is the iconic bridge itself…". Altering the feature had made Claude effectively obsessed with the bridge, bringing it up in answer to almost any query—even in situations where it wasn’t at all relevant.

This passage highlights one of the inherent cognitive risks of LLMs: the constant invitation to anthropomorphize and to 'agentify' the output.

Even if we take the factual claims in the blog at face value, we need to be critical of how they're framed – both in terms of what facts they might be leaving out, and in terms of the worldview that is being pushed in their contextualization of those facts.

Whether the tech works or not isn't determinative of whether it's dangerous

I think it's important to reframe these conversations: the main danger of so-called 'AI' is not what it can do, it's what it gives the ruling classes license to do to us. And that's largely unrelated from the actual realistic capabilities of the software; it's much more related to what the lay public believes the software can do. The major threat from 'AI' is its ideological, not technological power.

Which is why I urge people to be thoughtful in making assertions or reproducing claims about the capabilities of the tech. You are digging those out of a sea of fraud. This doesn't mean it's inherently invalid to engage with it, but it does mean you have to be very thoughtful about what and how you engage with it; and people often... aren't.

I think about this in the context of the last 10+ years of 'self-driving car' discourse, where we've spent a really rather huge amount of time and effort thinking through the implications of mass-deploying fully self-driving vehicles – for transit, for labor, for cities, and so on. When in reality, what we ended up with is... a couple minor autonomous systems operating in highly controlled environments with substantial human assistance; and a bunch of cars with unsafe autopilot features driving on public roads. Exaggerating the capabilities of those systems has been more dangerous than underestimating them, in the end.

I understand the desire to stay up to date on the (current and future) capabilities of LLMs but the thing is that reading the press releases 'AI' corps put out isn't guaranteed to make you more informed.

Okay? I read the paper backing the blog post (which makes specific claims about features and the limits of their approach), judged it to be plausible enough to be worth considering based on my computer science background, and expressed worry at a hypothetical if it turned out to be true in the long run. That's hardly taking their marketing at face value.

I can't take any post like this seriously when you close off the possibility of even evaluating claims made by the opposing side. You're treating companies with billions of dollars behind them like if you just cancel them hard enough by posting good they will stop developing and pushing this tech because... people won't buy it? They don't even need to, what with how many companies are shoehorning it in to existing products.

I think it's important to reframe these conversations: the main danger of so-called 'AI' is not what it can do, it's what it gives the ruling classes license to do to us. And that's largely unrelated from the actual realistic capabilities of the software; it's much more related to what the lay public believes the software can do. The major threat from 'AI' is its ideological, not technological power.

Huh? This entire wave of LLM ideological power was built off the back of GPT 3.5 and ChatGPT's massive leap in capabilities over prior methods. Remember how garbage early DALL-E images were, how skeptical everyone was over it ever replacing artists? And now there's Patreon LLM prompters making monthly incomes over $10k off DALL-E 3.

Not only are the capabilities causing direct harm today, but the ideology's wild delusions are fueled by the capabilities of the software. Ignoring it because you're really mad at the people pushing it is foolish.