robotmascot

@robotmascot

  • he/him

I make poor decisions, I like weird stuff, I have alright taste.

log inask
robotmascot
@robotmascot

Testing out a "Why Websites Are Making Themselves Worse" Explanation for my Less Online friends

If you're reading this on Cohost, this is me inviting you to please comment to be like "here's where you have factual inaccuracy / are probably conceptually wrong!" because this is my working theory and I'm sure I've messed it up somewhere. Putting a "read more" because it's kinda long.

BEFORE:

Humans talking to each other has always been:

  • extremely hard to monetize 1
  • a giant money sink 2
  • a thing that everyone intuitively understands it'll be valuable to hold as property 3

NOW:

  • AI/ML/LLM (pick your term) training makes those conversations worth money by being the cheapest way to speculate on AI 4
  • but public-facing stuff will be autoscraped unless you wall it off
  • basically any customer backlash is gonna be mitigated because you have a monopoly and also
  • everyone else in the sphere is in the same position

SO:

  • Basically every site 5 where people talk is at a point where the "optimal" move in terms of capitalism is to find a way to let robots read it for a fee
  • A whole ecosystem of people-friendly robot stuff (mobile apps, cute bots, etc) existed b/c it was free
  • That now dies and it would be a pr nightmare to use the above, soooo
  • Various bullshit reasons are deployed

  1. ads aren't great value prop for advertisers/sites tbh

  2. moderation! is! miserable! it's hard and has to be done by humans, it does not scale

  3. the last 10-15 years of tech have been increasingly speculation-driven b/c of the way the tech investment ecosystem works

  4. the ways you can improve an LLM relative to the rest of the field are roughly: more super-expensive hardware, more comparatively cheap training data, the extremely expensive (because it's paying devs, almost certainly in the global north) attempt to make a better algorithm. Also check out @corolla94's notes in the comments on this, they're relevant!

  5. with varying degrees of pressure- reddit feels this in a way that Cohost doesn't, I think, b/c of the nature of funding

You must log in to comment.

in reply to @robotmascot's post:

corolla94
@corolla94

I don't think Cory foresaw this so it would be interesting to torture him with it.

The one thing I'd add is to point 4 that making a better algorithm is not expensive in the sense of fixed/equipment costs and more that engineers are expensive in the global north. There's been an internal admission from Google (that I don't care to find right now) that open source is liable to eat both them and OpenAI alive because talented engineers are doing architecture research out of passion and making advances at a higher rate (although the recent spat of cool stuff is based on a model that leaked from FAIR). RNNs are on a different level of speed entirely, and I think the research there is entirely academic/open source.

Furthermore, it gets cheaper to evaluate new arches over time because their appeal is often exactly that they're cheaper to train. Yudkowski has said that he hopes to restrict training-level hardware and concede ground that LLMs will be inferrable on consumer hardware, but Anlatan (who also did the first Danbooru model) deployed a 3B that trades blows with the original Davinci 175B while having 4x the context length. That likely cost a lot of engineering time but not a whole lot of compute.

This is not to discount the importance of data. One of the more significant developments was the discovery that training a smaller Transformer on more data was cheaper. For a while every new arch was tested on a massive open dataset called the Pile but this development might signal that the field has already outgrown it.

login to reply
robotmascot
@robotmascot

That's a very good point re: algo dev, I glossed over why it's expensive (paying people in the US/Europe) but your stuff covers so much. I'll try to edit that in or at least cover it.

login to reply