cat

If you feel like a little schadenfreude, here's an article about Google admitting that they won't be able to keep pace with open-source LMs that are doing more with less:

• https://www.semianalysis.com/p/google-we-have-no-moat-and-neither

Which also mentioned LoRA regarding reducing the parameters needed for LMs:

• https://arxiv.org/abs/2106.09685

Some quotes from the article I appreciated (emphasis theirs):

We have no secret sauce. Our best hope is to learn from and collaborate with what others are doing outside Google. We should prioritize enabling 3P integrations.

People will not pay for a restricted model when free, unrestricted alternatives are comparable in quality. We should consider where our value add really is.

Giant models are slowing us down. In the long run, the best models are the ones which can be iterated upon quickly. We should make small variants more than an afterthought, now that we know what is possible in the <20B parameter regime.

[...]

In many ways, this shouldn’t be a surprise to anyone.