This week, Cerebras Systems unveiled the 13.5 million core Andromeda artificial intelligence supercomputer for deep learning. According to Cerebras, Andromeda provides multiple 1 exaflop (1 quintillion operations per second) artificial intelligence computing power at 16-bit half precision.The Andromeda supercomputer is essentially built with 16 clusters of Cerebras C-2 interconnected. Each CS-2 contains an 8.5-inch Wafer Scale Engine chip (WSE 2) containing 2.6 trillion transistors with 850,000 cores.
Plate-size processor
Cerebras has built Andromeda for $35 million in a data center in Santa Clara, California. This system will be used for training major language models and for complex AI-based academic and commercial processors. “Andromeda provides near-perfect scaling through simple data parallelism across GPT-class large language models, including GPT-3, GPT-J, and GPT-NeoX,” Cerebras states in the press release.
According to Cerebras, “Near-perfect scaling” means that as Cerebras adds more CS-2 computing units to Andromeda, the training time in neural networks decreases at a “near-perfect rate”. Typically, in systems based on GPU-based systems, returns appear to decrease as the number of units increases. However, Cerebras claims that its supercomputer can perform tasks that GPU-based systems cannot. For example, Andromeda was able to achieve near-perfect scaling on long array sizes at 2.5 billion and 25 billion parameters in GPT-J. However, a GPU-based system with 2,000 Nvidia A100s could not do the same due to “insufficient bandwidth”.How does Andromeda stack up against other supercomputers? Currently the fastest in the world, Frontier is based at Oak Ridge National Laboratories and can perform at 1,103 exaflops in 64-bit double precision. The construction of this computer cost 600 million dollars.
