like a lot of people on this app, i spend a lot of time thinking about lofty things like "the state of the internet" and how it changes over time. for example, how people say "on this app" now instead of "on this website". one of the things i realized recently is that the most valuable source of information right now, in early 2023, is reddit. i know. i'm not happy about it either.
there's a great post about this. if you search some topic you want to know about, especially if you're making a purchasing decision about it, chances are you'll find lots of SEO spam blogs. what do we do in this situation? we add "reddit" and read comment threads. that's where the real information lives.
the value of information is, at least in part, determined by how many knowledgeable people have contributed to it. a wiki is valuable precisely because it's been looked at and revised by its audience. the seo blog was probably not looked at by anyone but one overworked copywriter, and maybe barely skimmed by that person's boss. outside the content itself, that's the difference.
you know what else reddit is like? the old internet. reddit is not all that different from the initial organization of things, in yahoo directory, in newsgroups, in webrings. searching for information directly through web crawling might be the aberrational state, sandwiched chronologically between these community-driven bookends of internet information.
maybe there's a search engine concept in here; provable contributor count being a score for the quality of information. i sure don't know how to make it though
Before search engines, we had link directories. Topic-specific, or general, or whatever. We even had stupid link directories. Yahoo started as a really big link directory!
Search engines were the thin end of the wedge, in retrospect. That was the first "algorithm" that decided what you got to see; fully automated, no human in the loop. You had to and could learn how to query the old search engines like lycos and altavista to get useful results but you were still more or less at the mercy of the spider and how it indexed things. Google's big "innovation" was applying more algorithm. We should have seen what was coming.
But yeah if we want to unfuck things: Link directories. Bring links back in general! Put link pages on your websites!
they are in direct conflict with search engines because search engines train off them. for younger people who don't know this history, links were the single most important signal in the early history of Google Web Search. the strategy of using links to do citation analysis was the realization that allowed Google to win and kill off the other search engines. (the company's communication guidelines strongly discourage workers from saying things like "kill off", but that's the reality of what happened)
aside: if you're code-inclined we encourage going and implementing the PageRank paper (it ranked pages, and also it was invented in part by Larry Page...), it's a cute trick how it actually collates links. we say "implementing" and not "reading" because frankly it's dense math that makes very little sense if you just read it, when we studied it about twenty years ago we didn't really understand it until we had written our own implementation. it is a lot simpler than the text makes it seem.
anyway! search engines derive their value from these manually curated links. this means that if link directories are placed somewhere that search engines can see them, people end up getting the value of the link directory by using the search engine instead, and not even realize where it originally came from. this is a problem because, even if we ignore the monetary aspect of things, site creators find it worthwhile to make stuff when people actually visit their sites!
search algorithms aren't magic, they need ground truth data to work. we're sitting today at the end of a decades-long process whereby search engines essentially killed, ate, and digested their entire food supply, and now there's nothing left. when we build new community stuff these days we have to remember that it is an adversarial process, that corporations will try to eat us, and we need to have a plan in place to deal with that. rel=nofollow is probably sufficient for link directories, we just also want to make sure people are bearing the general case of that problem in mind in everything they come up with.
in case somebody mentions it: we lived through web rings. web rings were a lot of fun, but they were not very practical. we think the little "my personal five favorite sites!" thing that every page used to have was a lot more useful than web rings, in general, and that's the kind of thing we mean when we talk about link directories.
