There is a language model bigger than GPT-3 with a bold ambition: to free AI from the clutches of Big Tech.
Named BLOOM, the Large Language Model (LLM) promises similar performance to Silicon Valley’s leading systems, but with a radically different approach to access.
While tech giants tend to keep their vaunted LLMs hidden from the public, BLOOM is available to everyone and free.
Greetings, Humanoids
Sign up for our newsletter now for a weekly roundup of our favorite AI stories delivered to your inbox.
It’s also multilingual – unlike Google’s LaMDA and OpenAI’s GPT-3 – an unusual feature in an English-dominated field.
These functions can democratize access to technology that will have a deep impact on society.
Powerful AI models can be trained and released in an open manner.
LLMs are proving adept at a growing number of tasks, including essay writing, code generation, and language translation.
But they are also adept at producing malicious content – and their future capabilities include: hard to predict†
BLOOM gives researchers a unique opportunity to explore their risks and benefits.
“BLOOM is a demonstration that the most powerful AI models can be trained and released to the wider research community with responsibility and in a truly open manner, as opposed to the typical secrecy of industrial AI research labs.” BLOOM’s training co-leader BLOOM said in a statement.
open AI
LLMs are prohibitively expensive to create and run. Training GPT-3, for example, was estimated at cost up to $27.6 million†
Inevitably, technology companies will want to protect such large investments, especially if they offer competitive advantages.
Not surprisingly, therefore, LLMs are rarely open source – with some notable exceptions.
Meta caused the most prominent anomaly. In May, the company offered access to the 175 billion parameter OPT system†
However, the full model is only available on request and for non-commercial use.
BLOOM increases accessibility.
The 176 billion parameter model is available for free to any person or institution that agrees the system Responsible AI License†
Everyone can too public viewing the meeting notes, discussions, and code behind the model.
The seeds of BLOOM
BLOOM was created by BigScience, a research project that started in early 2021. The initiative is started and led by AI startup hugging face†
“Large ML models have changed the world of AI research in recent years, but the massive computational costs required to train them have left very few teams with the ability to train and research them,” says Thomas Wolf, the co-lead of BigScience and co-founder of Hugging Face
The training corpus was in line with our values.
Wolf’s team of 100,000 researchers from more than 60 countries and 250 institutions developed BLOOM to promote inclusion and accountability in LLMs.
They trained the model on the Jean Zay super computer in Paris, France.
“We took a data-first approach to ensure the training corpus was aligned with our values,” said Christopher Akiki, a research scientist at the University of Leipzig and a BigScience researcher.
“BigScience’s multidisciplinary and international makeup enabled us to think critically about each step of the process from multiple points of view: ethical, legal, environmental, linguistic and technical.
“That meant we could address ethical concerns without sacrificing performance or scale.”
The size is certainly impressive. With 176 billion parameters, BLOOM is bigger than OpenAI’s GPT-3 and MetaAI’s OPT.
The model can generate text in 46 natural languages and dialects and 13 programming languages. For many of them, this is the very first language model with more than 100 billion parameters.
It is also uniquely affordable. BigScience says: researchers can use BLOOM for less than $40/hour with a cloud provider.
The model probably won’t compete with those built by Big Tech, but at least it offers a way to take a closer look at them.
Contents
