Google “We have no moat, and neither does OpenAI” https://archive.is/U3fus https://news.ycombinator.com/item?id=35813322
Me: https://news.ycombinator.com/item?id=35819915 I think from the perspective of a Google researcher/engineer, it must be alarming to see the crazy explosion going on w/ LLM development. We’ve gone from just one or two weirdos implementing papers (eg https://github.com/lucidrains?tab=repositories who’s amazing) to now an explosion where basically every dev and PhD student is hacking on neat new things and having a field day and “lapping” (eg productizing) what Google Research was previously holding back.
And we’re also seeing amazing fine-tunes/distillations of very useful/capable smaller models - there’s no denying that things have gotten better and more importantly, cheaper way faster than anyone expected. That being said, most of these are being trained with the help of GPT-4, and so far nothing I’ve seen being done publicly (and I’ve been spending a lot of time tracking these https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp…) gets close in quality/capabilities to GPT-4.
I’m always rooting for the open source camp, but I think the flip-side is that there are still only a handful of organizations in the world that can train a >SoTA foundational model, and that having a mega-model is probably a huge force multiplier if you know how to take advantage of it (eg, I can’t imagine that OpenAI has been able to release software at the pace they have been without leveraging GPT-4 for co-development; also can you distill or develop capable smaller models without a more capable foundational model to leverage?). Anthropic for example has recently taken the flip side of the “no moat” argument, arguing that there is a potential winner-take-all scenario where the lead may become insurmountable if one group gets too far ahead in the next couple years. I guess what we’ll just have to see, but my suspicion, is that the crux to the “moat” question is going to be whether the open source approach can actually train a GPT-n++ system.
I also think that anyone that talks to a local LLaMA vs ChatGPT4 knows there’s just no comparison. Abandoning mega-models completely isn’t really an option if you can’t get to something that reaches the bar of general capabilities, because users probably won’t settle for anything less once they get used to having easy access to that. (that being said, like I was mentioning to Andy, most people can’t tell when they’re talking to GPT-4 vs Prometheus - the future is probably going to be smooth LLM handoff between different sized models)
Other Responses: https://twitter.com/sytelus/status/1654357815060992003 https://twitter.com/jayelmnop/status/1654271276385550337 https://twitter.com/EMostaque/status/1654237268305911810 https://twitter.com/labenz/status/1654853321876815872 https://twitter.com/labenz/status/1655092874768179200
No Moat Followup https://twitter.com/sullyomarr/status/1656351249221169152