• 3 Posts
  • 171 Comments
Joined 1 year ago
cake
Cake day: March 22nd, 2024

help-circle









  • Yes, but its clearly a building block of Meta’s LLM training effort, and part of a pattern.

    One implication I didn’t mention, and don’t have hard proof I can point to, is garbage in garbage out. Meta let AI slop and human garbage proliferate on Facebook, squandering basically the biggest advantage (besides cash) they have. It’s often speculated that, as it turns out, Twitter and Facebook training data is kinda crap.

    …And they’re at it again. Zuckerberg pours cash into corporate trash and get slop back. It’s an internal disaster, like their own divisions.

    On the other side, it’s often thought that Chinese models are so good for their size/compute because they’re ahem getting data from the Chinese government, and don’t need to worry about legal issues.


  • The research community already knows this.

    Llama 4 (Meta’s flagship ‘AI’ project) was as bad release. That’s fine. This is interative research; not every experiment works out.

    …But it was also a messy and dishonest one.

    The release was pushed early and full of bugs. They lied about its performance, especially at long context, going so far as to game Chat Arena with a finetune. Zuckerberg hyped the snot out of it, to the point I saw ads for it on Axios.

    Instead of Meta saying they’ll do better, they said they’re reorganizing their divisions to focus on ‘applications’ instead of fundamental research, aka exactly the wrong thing. They’ve hermmoraged good researchers and kept AI bros, far as I can tell from the outside.

    Every top LLM trainer has controversies. Just recently Qwen (Alibaba) closed off their top base models just to spite Deepseek, so they can’t distill them. Deepseek is almost certainly training on Google Gemini traces. Google hoards their best research for API models and has chased being sycophantic like ChatGPT. X’s Grok is a joke, and muddied by Musk’s constant lies about, for instance, open sourcing it. Some great outfits like 01ai (the Yi series) faded into the night.

    …But I haven’t seen self-destruction quite like Meta’s. Especially considering the ‘f you’ money and GPU farm they have. They’re still pushing interesting research now, but the trajectory is awful.







  • Yeah, just paying for LLM APIs is dirt cheap, and they (supposedly) don’t scrape data. Again I’d recommend Openrouter and Cerebras! And you get your pick of models to try from them.

    Even a framework 16 is not good for LLMs TBH. The Framework desktop is (as it uses a special AMD chip), but it’s very expensive. Honestly the whole hardware market is so screwed up, hence most ‘local LLM enthusiasts’ buy a used RTX 3090 and stick them in desktops or servers, as no one wants to produce something affordable apparently :/