Microsoft developed the new Azure cloud-based supercomputer in collaboration and exclusive to OpenAI, a nonprofit research organization founded by Elon Musk and Sam Altman. The company introduced a supercomputer at the largest technology conference Microsoft Build 2020.
Historically, machine learning experts have developed small AI models that use labeled data sets to learn how to perform one single task — recognize an object, translate text. The artificial intelligence community has proven that some of these tasks are best accomplished by one large model – trained, for example, on billions of pages of publicly available texts. As a result, such a model so subtly understands the nuances of speech, grammar, concepts, and context that it can perfectly perform several tasks: summarize long speeches, moderate chat in online games, and even generate program code based on search results on GitHub.
Microsoft, for example, has developed its own family of such models – Microsoft Turing models and the largest of the currently existing language model with 17 billion parameters Turing Natural Language Generation (T-NLG). The company also announced that it will soon open the code for Turing models, as well as methods for training them in Azure Machine Learning. The creation and training of such arrays require a progressive supercomputing infrastructure or clusters of modern equipment connected via high-bandwidth networks.
The new supercomputer has become one of the five most powerful supercomputers in the world. It was created specifically for training the next generation of massive AI models capable of performing a wider range of functions and, accordingly, requiring large computing resources.
The supercomputer developed for OpenAI is a single system with more than 285 thousand processor cores (CPU core), 10 thousand graphic processors (GPUs), and a network connection speed of 400 gigabits per second for each server. Hosted in the Azure cloud, the supercomputer has the benefits of a modern cloud infrastructure, including rapid deployment, reliable data centers, and access to Azure services.
The development of a new supercomputer for OpenAI is the first step towards making the next generation of massive AI models and the infrastructure necessary for their training available to a wide range of organizations and developers around the world.