The power consumption of AMD’s Instinct MI300X card is surprising: we are approaching the 1KW limit!

We used to think that computer hardware would consume less power, thanks to the advancements in technology and the production of smaller transistors. However, recent years have shown that this is not the case. The Instinct MI300X GPU, which AMD introduced a few days ago, reveals exactly this situation. Prepared for AI workloads, the Instinct MI300X is approaching the 1 KW power consumption limit.

AMD moves towards the 1 KW limit

With a rated power consumption of up to 700W, the Nvidia H100 GPU has been the most power consuming data center chip since its launch. But that changes with the Instinct MI300X, which will consume even more power. AMD’s Instinct MI300X GPU will be the most power consuming chip at launch, consuming close to 1KW.

According to AMD’s documentation, the Instinct MI300X OAM GPU accelerator will have a power consumption of 750W. For comparison, Instinct MI250X GPUs based on the CDNA 2 GPU architecture consumed between 500 and 560 Watts. In other words, power requirements have increased by 34-50 percent in a single generation. On the other hand, considering the design of the chip and the performance it offers, this power increase is expected. The chip provides 8x performance gains for AI workloads, while operating 5x more efficiently.

Huge improvements over the previous generation

The AMD Instinct MI300X GPU is expected to have 304 CDNA 3 processing units, compared to 220 processing units in the MI250X. This means an increase of 38 percent. In addition, the chip contains HBM3 memory with a capacity of 192 GB. That’s 50 percent more than the 128GB of HBM2e memory used by the MI250X, and since we’re talking about HBM3, we’ll also see much higher transfer speeds.

The biggest advantage of the Instinct MI300X GPU and the massive 192GB of VRAM will be its ability to run major language models with fewer GPUs than its competitors. The Instinct MI300X can run LLMs up to 540 Billion parameters with less GPU than its competitors’ 80GB H100 chips.

“Nuclear reactor will be needed”

These power requirements continue to increase as GPUs and HPC accelerators become more and more powerful. A recent Gigabyte Server roadmap showed that CPUs, GPUs and APUs would approach the 1000W power barrier. AMD certainly has the most power-hungry chip, but the red team has also invested in a number of chip and packaging technologies that allow them to drastically reduce their power requirements.

During the ISSCC 2023 conference, AMD’s CEO, Dr. Lisa Su stated that if new designs are not adopted, Zettascale computing will need a “Nuclear Reactor” to meet its energy demand, and this is valid for only one system. However, MI300X GPUs will also be available in conservative variants with lower power requirements, especially in PCIe form factors. These have not yet been announced but are expected to arrive by 2024.