In a bold move that signals Meta’s determination to compete in the AI arms race, the company has released its largest open-source AI model to date: Llama 3.1 405B. With a staggering 405 billion parameters, this behemoth of a model represents a significant leap forward in Meta’s AI capabilities and its push for “open” AI development.
New and improved Features
I believe that AI will develop in a similar way. Today, several tech companies are developing leading closed models. But open source is quickly closing the gap. Last year, Llama 2 was only comparable to an older generation of models behind the frontier. This year, Llama 3 is competitive with the most advanced models and leading in some areas. Starting next year, we expect future Llama models to become the most advanced in the industry. But even before that, Llama is already leading on openness, modifiability, and cost efficiency.
Today we’re taking the next steps towards open source AI becoming the industry standard. We’re releasing Llama 3.1 405B, the first frontier-level open source AI model, as well as new and improved Llama 3.1 70B and 8B models. In addition to having significantly better cost/performance relative to closed models, the fact that the 405B model is open will make it the best choice for fine-tuning and distilling smaller models. Said Mark Zuckerberg
The company also used synthetic data to fine-tune Llama 3.1 405B. Most major AI vendors, including OpenAI and Anthropic, are exploring applications of synthetic data to scale up their AI training, but some experts believe that synthetic data should be a last resort due to its potential to exacerbate model bias.
For its part, Meta insists that it “carefully balance[d]” Llama 3.1 405B’s training data, but declined to reveal exactly where the data came from (outside of webpages and public web files). Many generative AI vendors see training data as a competitive advantage and so keep it and any information pertaining to it close to the chest. But training data details are also a potential source of IP-related lawsuits, another disincentive for companies to reveal much.
However, this move towards openness comes with strings attached. App developers with more than 700 million monthly users must still request special permission from Meta to deploy Llama models, giving the company significant control over how its technology is used at scale.
As Meta races to catch up with competitors like OpenAI and Anthropic, the release of Llama 3.1 405B represents both an impressive technical achievement and a snapshot of the current state of AI development. It highlights the industry’s relentless drive for bigger, more powerful models, while also bringing into focus the challenges and trade-offs that come with such rapid advancement.
As the AI landscape continues to evolve at breakneck speed, it remains to be seen how Meta’s bet on “open” AI will play out in the long run, and whether the benefits of these massive models will outweigh their considerable costs – both financial and environmental.
