We sometimes talk about web scraping projects as "ingesting" or "slurping" text from the web but usually we understand that nothing is being actually consumed, the text is still there where it should be. But now it's different.

Large Language Models (LLMs), of which the current most famous is ChatGPT, are eating the web, actually consuming and destroying it.

• Google: overrun by fake sites with unhelpful content generated by LLMs. They have, bafflingly, decided to counter with their own, first party unhelpful content generated by LLMs.

• Reddit: despite its community, it was previously a refuge of meaningful, open, human-written text on topics people care about. Now it's mostly down, because they betrayed their community with drastic changes and the community revolted. One reason given for the drastic changes: their tasty meaningful text was being exploited by Google and OpenAI and their ilk, and they weren't getting any benefit from it. So they decided to charge a gazillion dollars for API access, causing everything else the community built that uses the API, such as accessibility tools and usable interfaces, to go down.

• Twitter: bought by a complete dipshit who (among his many foolish ideas) thought he could save money by replacing workers and essential site features with AI, so essential site features are being replaced with nothing. Also decided "API access and third-party apps should cost a gazillion dollars so LLMs will pay us" before Reddit did.

• Stack Overflow: Rampant use of ChatGPT threatens to turn its questions and answers into nonsense, much like the questions and answers you see on Google. Moderators responded by moderating even more harshly than they usually do. Site owners saw that a declining number of people want to even attempt to use Stack Overflow, and panicked, and told moderators to stop banning people for using ChatGPT, instead of addressing any of the other reasons people don't want to use Stack Overflow. Moderators went on strike. The site and its siblings are now mostly unmoderated, and, confusingly, still working for now.

• Wikipedia: apparently standing strong for now, but their standards of information are threatened as formerly "reliable sources" start generating nonsense with LLMs. Wikipedia is particularly vulnerable, because if LLM output is ever treated as a reliable source, it can create self-reinforcing fake facts that people repeat because they're on Wikipedia.

• Many independent websites: buried under competing LLM nonsense, or bought out by venture capitalists who fire their staff and replace them with LLM nonsense. (I just saw a hecking GeoGuessr tips page destroyed by someone who copied all the work people put into it, pasted it onto a ChatGPT-generated website, and took credit for it.)

They have found a way to scrape the web so hard that there isn't any web there anymore.
I don't know what we can do about it. I hope that a specifically anti-metrics, anti-capitalist website like Cohost can be a refuge, but there's not enough Cohosts.

I'm at the point of considering tracking down and joining niche forums every time I have an issue. Unfortunately, this would come with the difficulty of finding them when search engines aren't usable. Reddit works for most things for now, but the way things are going I'm getting concerned about how long that'll last.

i am not genuinely mad at people for doing it, but god, it is so eviscerating to have someone say "oh man i love your content"

it's like walking up to a cow and going "dude i just want you to know? the burgers i'm gonna make out of you are going to be so delicious. my mouth is watering already. i cannot wait to exploit you. i see you purely as a resource"

i will never stop reiterating this: "content" was a word created by people who think like this

what is the purpose of the text on a website? to increase visitor count! to "convert" "leads"! to generate organic traffic! imagine being so brainwormed that you think the purpose of baking cookies is to make people interested in the cookie jar. "content" is a word created by the people who invented SEO and ruined google.

"content" is a block on a wireframe (derogatory) that's tucked in between "navigation", "banner ad", and "targeted ad," and just to the left of "column ad (animated)." applying that term to someone's creative output is unspeakably demeaning. applying it to your own is horrifying.

For the last eighteen years we've watched the slow rot and death of the web. In the early days, before capital had figured out how remake the web in a way that best serves its interests, it was very much a network for people. People made websites, people kept journals and fan sites, people met other people on weird little forums, people had experiences that were novel and exciting because the web was novel and exciting. Nobody had a timeline, a feed, a bunch of colourful apps circumscribing their online lives. What you had was dependent on you, what you created and who you met. How you decided to spend your days.

i can't believe tld marketing is a real thing

gex.gay is available but gex is definitely not gay so no i did not buy it

i'm the proud owner of butisit.gay which i plan on turning into a search engine focused on good representation in all media formats. but that's a lot of work so it will take me some time. lol

i'm the proud owner of isincredibly.gay for no reason other than the fact that i wanted my matrix username to be @ruby:isincredibly.gay

the most valid reason anyone has ever purchased a domain

lol. from Bing AI Can't Be Trusted

Internet search has become useless as more and more of the results become useless AI garbage generated to machine SEO and trick people into thinking it's real, so clearly the solution is to generate bespoke AI garbage in the search engine itself to trick people into thinking it's real.

Ah, a great illustration of the AI arms race happening in many spheres right now. Thanks, I hate it!

it’s time to bring back yahoo style web directories