Microsoft haven't even launched this stupid "ai" bullshit yet
and people have already found the very obvious No Shit Sherlock exploits
“Every few seconds, screenshots are taken. These are automatically OCR’d by Azure AI, running on your device, and written into an SQLite database in the user’s folder,”
The database is stored locally on a PC, but it’s accessible from the AppData folder if you’re an admin on a PC. Two Microsoft engineers demonstrated this at Build recently, and Beaumont claims the database is accessible even if you’re not an admin.
Microsoft is currently planning to enable Recall by default on Copilot Plus PCs. In my own testing on a prerelease version of Recall, the feature is enabled by default when you set up a new Copilot Plus PC, and there is no option to disable it during the setup process unless you tick an option that then opens the Settings panel.
Everyone pushing for this needs to Be Made An Example Of, and I'm not joking
It's blindingly obvious that Recall was designed for one reason, and one reason alone: To Put More Laser Targeted Ads In Your Operating System
(well, that and selling new chips I guess)
Like the very concept of a log of everything you've seen on your computer is bad, straight up. This would be equally bad if it was doing OCR + text search without an LLM, but LLM hype has somehow disabled that part of the "this is a bad idea don't do it" instinct.
Any such log makes any compromise of a computer at any moment in time as bad as a compromise of the computer at the worst possible time. If I hack your computer and get your Recall database on a Saturday night while you're gaming, it's equivalent to me hacking your computer right when you were looking at your bank website or social security info or password manager.
(Actually password manager is an interesting case because they generally mask passwords, as do password fields. But you know what I mean.)
There's not much mitigation that makes it acceptable, either. Encryption at rest matters for, like, a breach of a remote server storing your data. It's less useful when all the plausible attack scenarios involve local access to the computer operating on the data, as either you're always entering a password to decrypt the data live OR all the info needed to decrypt the data is on that same system that is already compromised. This is why Yubikeys/TouchID are so useful—they require an interaction in the real world that users are actually willing to live with (usage of Recall would be extremely low if it constantly requires a password).
The only direction that might make something like this acceptable would be preventing saving of sensitive data in the first place, but determining what is sensitive data is so reliant on context that I don't see it as solvable—but I would not be at all surprised if the response to all this uproar would be Microsoft having an LLM scan the incoming screenshots and generate a response to the question "Does this contain sensitive data? (Giant list of sensitive data types)" and use that to determine what not to save. Which fuckin sucks.
Apps could have some sort of flag to indicate that they're in "private mode" and should not be recorded (mobile platforms already have some stuff like this, Firefox for Android prevents screenshots in private mode) but that wouldn't scale and has similar issues around determining what content is sensitive and what isn't.
