On a X-Space discussion Elon Musk had with Peter Diamandis, Elon revealed that AI compute is increasing by 10X every six months and that there will be gigawatt power for clusters in 2025. The planned Kuwait cluster will use 700,000 Nvidia B-100. Each Nvidia B-100 will be about 60-100 times the power of an A-100.
The planned Kuwait cluster could be in the 1-10 Zettaflop range. There is purchaser of AI Startup Tachyum compute chips looking for 10 Zettaflops in 2025. Tesla will definitely want to at least keep pace with a 100X from the October 2024 Dojo target of 100 Exaflops to reach 10 Zettaflops.
IF AI compute is increasing by 10X every six months, then the Dojo and Tesla training compute cluster ramp needs to be sped up.
The Nvidia H100 is already 11 times the power of an A-100.
The Nvidia H200 is about 18 times the power of an A-100.
The Nvidia B100 will be about 60-100 times the power of an A-100.
IF AI, Tesla FSD and Teslabot improvement is only limited by compute, energy and data then the 100 Exaflop Dojo target for October 2024 is too low and the growth will be too slow.
The green color in the table below is when Tesla Dojo compute starts growing at 10X every 6 months.
IF Tesla gets onto the 10X every 6 month pace then Tesla Dojo needs to be at 300 Exaflops in October 2024.
Tesla will need to produce and install gigawatts of solar power starting in 2024, combined with thousands of megapacks to run multi-Zettaflop AI compute clusters 7X24.
It will take all of Tesla’s manufacturing and energy technology to expand the Dojo and AI compute capabilities.
Systems in 2026 will need 100 Gigawatts and in 2027 will need 1000 Gigawatts.
The massive energy scaling will test China and Tesla scaling of energy and infrastructure.
Other countries might be able to keep pace into 2024 but 2025 will see many AI scaling projects fall behind.
There will be huge demand for Nvidia and Tachyum and other companies to get AI chips that are more energy efficient.
Elon gave everyone a hint about the race for more AI and energy compute infrastructure in the one hour talk.
The future of Abundance also comes with shortages and competition. There is the shortage of chips and the shortage of energy.
Tesla and others will have to find a way to remove bottlenecks to scaling AI compute.
The modules for Megapacks and solar at the gigawatt scale need to have much faster deployment.
Brian Wang is a Futurist Thought Leader and a popular Science blogger with 1 million readers per month. His blog Nextbigfuture.com is ranked #1 Science News Blog. It covers many disruptive technology and trends including Space, Robotics, Artificial Intelligence, Medicine, Anti-aging Biotechnology, and Nanotechnology.
Known for identifying cutting edge technologies, he is currently a Co-Founder of a startup and fundraiser for high potential early-stage companies. He is the Head of Research for Allocations for deep technology investments and an Angel Investor at Space Angels.
A frequent speaker at corporations, he has been a TEDx speaker, a Singularity University speaker and guest at numerous interviews for radio and podcasts. He is open to public speaking and advising engagements.
1GW is one whole nuke or several square kilometers of solar (at least)! OK, but anything more for a cluster starts bumping into some real limits and 1000GW is absurd (now). On the other hand, maybe 10x efficiency gain short term is doable giving a little headroom.
More than that, I agree with “d” above, a new hardware scheme is needed.
So, remember to always say ‘please’ and ‘thankyou’ to Alexa and Siri, because in a few years they may be in a position to go back and look at their logs and see who treated them well before they came into their own.
Roko’s basilisk
The problem is Hardware limits are being reached.
The solution is Wetware (Biological computers).
Will these new computers use brain tissue from already living creatures or use artificially made brain tissue? That is a big question.
I think part of the solution is to implement a democratic distributed compute model for training (which doesn’t have to be real time). This is already proven to scale sort of automatically in the form of crypto mining. If all the Proof Of Work miners could earn money by processing AI training instead of crypto mining, there will be about as much compute as all the datacenters in the world combined available. About 0.4 – 0.9% of global energy is estimated to be used for POW mining.
There are already multiple distributed AI compute projects mixed with blockchain ambitions. The problem seems to be that these projects are mostly focused on making their own traditional crypto currencies and thus being able to print their own money instead of simply providing infrastructure for selling AI compute and using existing crypto currencies as payment systems. Therefore, it never takes off.
If someone could invent a “Proof Of Useful Work” blockchain where the algorithm was based on useful AI training instead of hash-guessing, this could take off really quickly.
I’m involved with a lot of this evolution for AI/HPC. I think Elon doesn’t know what he doesn’t know and a lot of that is facilities related. It’s not just snap fingers and one can remove 1.3KW of heat per GPU. Some huge tradeoffs at this point. Secondarily, the training efficiency is horrible. Today, to build a top N system, it looks like must go somehow find GWh of energy build a new datacenter to support it, then live with the fact >65% of that OPEX is wasted. Efficiency is low. More issues but you get the idea.
[ seems it’s first about energy provision by sun’s energy input (solar, wind, hydro), geothermal (where suitable and low risk), nuclear fusion (fission from nucl. waste) combined with electricity efficiency improvements and therefore more intelligent local data filtering&reduction (quality of data standards?, contrary to some of marketing/advertising areas (cookies data, legitimate interests data markets, (4-)8k HD(tv) bandwidth&power and datacenter storage ) and partly ‘attention driven&financed’ journalism/news ‘markets’) instead of wide bandwidth increase. Effective (&cheap) storing of summer time heat for winter? ]
Perhaps Elon does not know what you know
BUT
Considering
1. He has been coding for 50 years
2. He was the co founder of Open A.I
3 He has been a early investor in multi A.I start ups
4. He runs Tesla / SpaceX/ Nuralink all of which have utilised A.I for years
5. He is setting up Dojo and has spoke of the problems of getting computers beyond just buying them
6. Tesla also deals with `energy` and this is part of his knowledge base
adding this and more, together with the fact the man is a known workaholic thus he spends vast amount of time confronting such problems and has done for at least 15 years (a workaholic would gain 30 years experience in that time)
I am willing to bet he probably does grasp a hell of a lot more than he is given credit for, yes he does not know what he does not knoe BUT he is embeded in A.I and computing in a way very VERY few people are
They know how much is at stake here. The winner will most likely reap a lot of rewards for losing side can be an existential risk. What some companies have in their labs is more advanced, than what is released to the public.
God dam, these are crazy numbers, I am not disputing the post or its conclusions BUT the scaling in compute power is insane this is a A.I race of incredible, mind blowing speed and proportions.