The economics for AI chips has to work for
Chip builders
System builders
Data center operators
Application builders
Paying customer (end users).
The bigger large language models are at 400 Gigabytes or more and mixture of experts is at 1.8 terabytes.
TechTechPotato interviews AyarLabs and makes the case that optical connections between chiplets is needed to scale the 100X-1000X in memory and compute and connectiveness at an affordable architecture
Memory bandwidth and memory capacity are everything right now. Companies are trying to increase memory HBM3 and HBM4.
The network communication layer is too slow.
Rack power requirements keep increasing. The latest Nvidia systems are using 120 kilowatt hours per rack.
The communication fabric can let 8-way ASIC GPU designs talk to each other but there is the demand to scale to 1024-way ASIC GPUs talking to each other in 2026.
00:00 – Who is making the money
01:31 – Economics of AI
04:20 – Tonal shift from AI Companies
06:39 – Moving to an agentic workflow (more llm in series for agents means that 3 agents triples the compute and memory requirements)
08:35 – Current fabrics are too slow
11:27 – What needs to change
13:50 – What the future could look like
18:18 – More Economics
21:40 – Where are future data centres going?
24:03 – Benefits of an optical system
26:09 – Wrapping up
27:50 – Minimum specification
Brian Wang is a Futurist Thought Leader and a popular Science blogger with 1 million readers per month. His blog Nextbigfuture.com is ranked #1 Science News Blog. It covers many disruptive technology and trends including Space, Robotics, Artificial Intelligence, Medicine, Anti-aging Biotechnology, and Nanotechnology.
Known for identifying cutting edge technologies, he is currently a Co-Founder of a startup and fundraiser for high potential early-stage companies. He is the Head of Research for Allocations for deep technology investments and an Angel Investor at Space Angels.
A frequent speaker at corporations, he has been a TEDx speaker, a Singularity University speaker and guest at numerous interviews for radio and podcasts. He is open to public speaking and advising engagements.
I remember reading about optical interconnect in SciAm decades ago, I had almost forgotten about it… Sure took them long enough.
Optical interconnects is a bit like fusion. Always a decade or two away…
My guess is that the electrical connections will improve in such a way that it’s simply not worth it to make the technological leap in the nearest 30 years or so.
Looking at the 2021 presentation from Tesla on the Dojo D1 chip. Supposedly it had 32 TB/s of off-chip communication, to compare with 80 TB from 10 optical chiplets in the talk from Techtechpotatoe. So three years ago, the state of the art for electrical communication was about half of what the optical technique promises. By now, Dojo 2 (volume production end of 2025) most likely has even higher off-chip data capabilities and Dojo 3 would likely represent another step upwards.
From this point of view, the optical chiplets looks like a technical complication and cost driver with near zero benefit.
Optical interconnect is one area where Intel actually leads by quite a lot. Worth looking into their tech.
I do. Intel has long been associated with “connecting the dots”. Perhaps the most difficult aspect of getting different technologies to “play nice” is how hard it is to have one type of technology “communicate with” another. Optical communication is as “physics” is very different from other types.
By the way Intel, your optical interconnect technology is very creative, and efficient. I must add, you must tell your people not to so freely share proprietary information with strangers. Some of us also connect the dots.
Why is this a surprise? Our fighters (along with other aircraft later) went from fly by cable, to fly by wire, to fly by light. This technological development created some very interesting incongruity’s. Our integrated circuit’s work by moving elections within a two dimensional framework in one dimension/direction at any one time. You can only move so many electrons, threw a two dimensional architectures so fast. Even a superconducting Josephson junction does not solve this signal “traffic jam” But laser light does. It generates almost no waste heat, or thermal “haze”. Think of that as white noise in the background, or tinnitus that “ringing in your ears” you get with hearing damage or getting old (I check both of those boxes).
Or communication cross-talk. This is when the information you “want” to hear is corrupted. This is actually quite easy to create, requiring minimal “energy” but also difficult to notice. This is a hallmark of disinformation, the broad contents may be true (or you think so), but the conclusion and possible consequences you may act on is what someone “tweaks” you to believe. Laser communication actually makes you being bull s***** harder. Is that not a good thing?