Breaking into the Large Language Model Filled with a New 107 to 5 Billion Parameter Model
Introduction
A new open-source large language model called Bloom, developed by an international collaboration of academic volunteers, is breaking into the large language model scene with a new 107 to 5 billion parameter model as part of an open science initiative. Trained with USD 7 million worth of publicly funded computing time, the Bloom language model is expected to go toe-to-toe with similar models developed by tech giants such as Google and OpenAI.
Multilingual and Collaborative Roots
Aside from its collaborative roots and the decision to open-source the project, Bloom is also the first model of this scale to be multilingual and will be made available for research use. Large language models are machine learning algorithms that can recognize, predict, and generate human languages by drawing on enormous text-based data sets. They can respond to questions, write essays, or even generate computer code with limited instructions. This technology has significant implications for various fields, including natural language processing, social science, legal ethics, and public policy.
Easily Accessible Data Sources
To date, large language models are built by large tech firms with strong financial backing, but relatively small teams that turn to easily available resources such as online repositories or popular sites such as Reddit for the data to train their models. Bloom, however, is the work of hundreds of researchers consisting mostly of academics, such as ethicists, legal scholars, and philosophers. Data sources were identified through a series of workshops with a much broader base of collaborators, including community groups around the world.
Consumption of 350 Billion Words
It is understood that the researchers hand-picked nearly two-thirds of the 341 billion word dataset from some 500 sources and added a multilingual web crawl filter for quality. When fully trained, Bloom will have 107 to 6 billion parameters and would have consumed more than 350 billion words from four to six different languages. Some have called Bloom the most important AI model of the decade, even ahead of Google’s 540 billion parameter Pathways Language Model or the trailblazing GPT-3.
Open Large Language Model
Alberto Romero, an AI analyst, argues that the building and funding of an open large language model have created intense pressure on various tech giants to open-source their models. Seen from this perspective, Bloom is the spearhead of an impending wave of change in the AI field. While the groundwork was put in place last year, the actual training of Bloom began in April. Just a month later in May, Meta AI announced that it would give away its massive new language model OPT as part of its effort to democratize AI.
Plasma-Based AI for Tic Tac Toe
Meanwhile, in a major step forward, scientists are creating a data processing unit from a network of chemical reactions taking place within an isolated chemical system such as a plasma. The concept is based on finding a set of chemical parameters for the system, such as pressure or temperature, so that the system can spit out proper information in real-time according to a dynamic input. This concept involves creating a programmable analog computer that functions on a molecular level and can process complicated information in nanoseconds.
Chemical Parameters as Software
The chemical parameter sets are thus the software in such an analog computer. To determine the chemical reactions – in other words, the thinking process – in the computer, the scientists consider the map of chemical reactions in the system, namely the chemical pathway network, as an artificial neural network. The chemical parameters are the weights of the network, and the concentrations of species are the neuron values. Using modern machine learning techniques, the hardware can be trained and programmed for specific missions.
Teaching Plasma to Play Tic Tac Toe
To demonstrate this concept, scientists Lin and Cater taught a plasma to play tic tac toe using varying and controllable mixtures of gases. To do this, the 3×3 board of the game was set up using nine different gases, each representing one of the game’s nine tiles. The board status can thus be represented by the mixing ratio of these gases. A lower ratio of a type of gas means a plasma’s marker at that tile, while a higher ratio means an opponent’s marker. Once the plasma received such a gas mixture, the chemical reactions in the plasma started working, making excited atoms and molecules.
Chemical Pathway Network
These excited species will output light signals representing the plasma’s next move. Therefore, a different board status means different mixing ratios of these gases, which lead to a series of different chemical reactions in the plasma. The plasma will thus output a different next move. This is the plasma thinking using its chemical pathway network. The light signals will be translated to update the board status so that the opponent, either a human or a computer player such as the random move player used for training and testing, can play along next.
Evolutionary Algorithm for Machine Learning
The training of the plasma is achieved by trying the chemical parameters with small modifications. A modification leading to a lower score will be discarded, and the parameters will be reset. However, a set of modified parameters that can make the plasma play better will be recorded, and the next modification generation will be based on it. This is a typical evolutionary algorithm used in the machine learning world. Through training, the plasma eventually achieved a high winning rate against the random move player, indicating that the plasma does not play randomly but with logic and develops its strategies.
Conclusion
With Bloom and plasma-based AI, we see how state-of-the-art AI is no longer reserved for big corporations with big pockets. As these technologies continue to evolve, we can expect to see greater democratization of AI and improvements in its relationship with the outside world. As we move forward, it will be interesting to see how AI technology will continue to grow and adapt.