I wasn't expecting this level of analysis from comedy podcaster Justin McElroy, but he's right:
This is what sucks is if you break it down to any individual human in the chain, and you ask them about the morality of this, then they would probably say "oh yeah, we should try to help people." Living in a capitalist system, they are not just encouraged but required to try to squeeze as many nickels out of everybody as they can. People get freaked out by AI, it drives me crazy. You know we've already been living under the tyrannical reign of an algorithm for years now. It's a bunch of humans serving this algorithm that will give them the most money. Like, we've been doing it. So no one is to blame... the system is built for this. This is what it's supposed to do.
-- "Sawbones" episode 464, "Patents vs. Life-Saving Drugs"
Consciously or not, most of us know that we all live within a system of rules that we have no choice but to follow. We are the computational substrate for an algorithm that incrementally optimizes for the greatest possible redistribution of wealth from the poor to the rich. This is not what anyone wants -- not even the richest, who live in anxiety about what would happen if they lost their wealth and had to become one of us. The system has its own life, requiring no outside intervention for its continued operation, like a daemon running on a server.
What's being called "AI" doesn't have much to do with intelligence. It's a change of degree, not kind. The more of the human computational substrate gets replaced with silicon, the less human discernment and choice can interfere with the efficiency of the optimization process. All the brakes, all the moderating factors, can be taken out of the picture. AI is not, and isn't supposed to, model the mind of a person because a person has a conscience, and a program does not. The absence of the superego is the point; the scary thing about AI is not that it's too much like a mind, but that it's not like one enough. Human "flaws" like the ability to get bored, distracted, frustrated, or angry are exactly why the entire planet hasn't been melted down into paperclips quite yet.
An engine never thinks about its daddy
And an engine never needs to write its name
(Drive-By Truckers, "The Day John Henry Died")
Yes, I get all my information from podcasts, because I can't read a book while I'm driving or scooping the litter box. The other day I was listening to a "Behind the Bastards" episode from last year, "Why Is the Rent So Damn High?" I keep thinking about something Robert Evans said on it about landlords, which is that yes, they are and have always been awful, but in the 1980s and before, small landlords exercised discretion and didn't always try to extract the maximum possible amount of money from their tenants. Once automated software arrived that provided landlords with analytics that showed exactly how much rent they could get away with charging, some were initially resistant, but eventually most adopted the automated price-fixing, and in a regime of deregulation no one cared to stop it. "Oh yeah, we should try to help people." But the computer said we shouldn't. There doesn't even need to be a computer, as Justin McElroy said. But computers make it all faster, meaner, and harder to sabotage. They feel no mercy. And if we still do, there's nothing we can do about it. In an integrated circuit, the gears are too small to throw sand in.
