Tesla Dojo Supercomputers Will Provide 100 Exaflops in 2024

by

Tesla is starting production of the Dojo AI training Supercomputer next month. Tesla will have 13 Exaflops of AI training by the end of 2023 and 100 Exaflops by the end of 2024.

Jan 2023 3 Exaflops of AI compute, 10,000 Nvidia A100
June 2023 5.5 Exaflops, 17,000 Nvidia A100
Oct 2023 13 Exaflops, 40,000 Nvidia A100
Feb 2024 33 Exaflops, 100,000 Nvidia A100
October 2024 100 Exaflops
Mid 2025 300 Exaflops

AI supercomputers can use floating point 8 precision. This is different regular supercomputers running FP64.

Nvidia announced the DGX GH200 AI supercomputer at Computex in Taipei in May 2023. It uses 256 Grace-Hopper Superchips, connected by 36 NVLink Switches, to provide over 1 exaflops of FP8 AI performance (or nearly 9 petaflops of FP64 performance). The system further touts 144TB of unified memory, 900 GB/s of GPU-to-GPU bandwidth and 128 TB/s bisection bandwidth. Nvidia is readying the product for end-of-year availability, and notes its Grace Hopper Superchips have entered full production.

Nvidia is building a mega-system DGX GH200-based AI supercomputer called Helios. Helios connects four DGX GH200 systems – for a total of 1,024 Grace Hopper Superchips – using Nvidia’s Quantum-2 InfiniBand networking. Nvidia is planning to bring the system online by the end of the year. Helios will provide about 4 exaflops of AI performance (FP8), and while it’s not the intended use case, would deliver ~34.8 theoretical peak petaflops of traditional FP64 performance.