Interesting computerphile video about a paper (link here) positing that the "explosive" improvement curve of genAI might not happen without diminishing return levels of work.1
-
which is something to keep in mind when you hear companies sell the "it's only gonna get better from here" pitch. It might be true that it might get better, but "how much better" is almost as important of a question especially if a single-digit% improvement requires 10x the work lol
i think most people miss that LLMs are just the result of someone going “oh there’s no point at which more data stops making neural networks better” and not only have we basically run out of all the easily stolen data but we’ve polluted all future data too.
it is hard to express just how much MORE data we will need to make them significantly better. much more likely are improvements in running them but still, neural networks are too simplistic to not need everything humanity has ever produced. any major breakthrough will have to be something in that low level training area.
This is most obvious with image generation, but its true for text as well. If you trace the technology back beyond the point where Dall-e, "ChatGPT" (or even OpenGPT) started to become household names. You'll immediately notice just how incremental the advance of the technologies is, despite a seeming exponential increase in the resources expended.
- Generative Adversarial Networks (GANs) were described in 2014, and I'm pretty sure that weird Cat Generating paper (the idea used in the "Normal Cat Pics" twitter account) showed up soon after. Google Deepdream (the weird dog nose psychodelic images) was even earlier
- Neural Style Transfer (copying the 'style' of a source image onto another using AI) is from 2015.
- Word2Vec (representing a word's semantic meaning as an array of numbers) is from 2013
You've got rudimentary versions of all of today's generative AI there. Everything now is the result of step-by-step improvement on that. (plus an absurd amount of resources)
One of the reasons there's no reasonable chance of another AI explosion in the near future, is because the current explosion never happened. Just a decade of incremental improvement. Claims to the contrary are pure snake oil.
