World’s Fastest AI Chip with 4 Trillion Transistors

Cerebras Systems has doubled its existing world record of fastest AI chip with the introduction of the Wafer Scale Engine 3. The WSE-3 delivers twice the performance of the previous record-holder, the Cerebras WSE-2, at the same power draw and for the same price. Purpose built for training the industry’s largest AI models, the 5nm-based, 4 trillion transistor WSE-3 powers the Cerebras CS-3 AI supercomputer, delivering 125 petaflops of peak AI performance through 900,000 AI optimized compute cores.

Key Specs:

4 trillion transistors
900,000 AI cores
125 petaflops of peak AI performance
44GB on-chip SRAM
5nm TSMC process
External memory: 1.5TB, 12TB, or 1.2PB
Trains AI models up to 24 trillion parameters
Cluster size of up to 2048 CS-3 systems

With a huge memory system of up to 1.2 petabytes, the CS-3 is designed to train next generation frontier models 10x larger than GPT-4 and Gemini. 24 trillion parameter models can be stored in a single logical memory space without partitioning or refactoring, dramatically simplifying training workflow and accelerating developer productivity. Training a one-trillion parameter model on the CS-3 is as straightforward as training a one billion parameter model on GPUs.

The CS-3 is built for both enterprise and hyperscale needs. Compact four system configurations can fine tune 70B models in a day while at full scale using 2048 systems, Llama 70B can be trained from scratch in a single day – an unprecedented feat for generative AI.

Superior Power Efficiency and Software Simplicity

With every component optimized for AI work, CS-3 delivers more compute performance at less space and less power than any other system. While GPUs power consumption is doubling generation to generation, the CS-3 doubles performance but stays within the same power envelope. the CS-3 offers superior ease of use, requiring 97% less code than GPUs for LLMs and the ability to train models ranging from 1B to 24T parameters in purely data parallel mode. A standard implementation of a GPT-3 sized model required just 565 lines of code on Cerebras – an industry record.

8 thoughts on “World’s Fastest AI Chip with 4 Trillion Transistors”

  1. I sold 1/3 of my Nvidea stock recently. It’s just a matter of time before some of these competitors, including Groq (not Grok), start finding their way into servers, especially with the grid challenges server farms are posing in the next year or so. The energy savings is enormous too.

  2. If you buy a science book and feed it to the AI, have you violated copyright?
    If you buy a book on art and feed it to the AI, have you violated copyright?
    Do you owe anything to the authors/artists of you train an AI on their works that you PAID for?
    If a robot watches a performance and learns from it has it ‘stolen’ anything?
    If a robot scrubs the Internet only accessing what anyone can and trains on that, does it owe anyone for ‘stealing’ their work?
    I am curious about this.

    • It will be illegal if you feed AI with copyright resources…
      But, for anyone powerful enough to buy politicians and judges, I’m sure it will be fine.
      Remember to invest understanding the two tier legal system.

    • Laws do not apply to AIs but to people. When you think about an AI better think of it as a “person using an AI”.

      Copyright is about the right to copy a piece of work. As long as a person (who uses an AI) does not copy work (and sells it as their own) is there no violation. People can learn from books and then write books of their own. As long as the work is genuine and not plagiarised is there no violation. It also depends on each case. Musicians can create a song that sounds much like one from another artist. A jury then has to decide how genuine it is or whether it violates the other artist’s right.

      Watching a performance and copying it depends on the venue. So is it usually allowed to take pictures and videos of people in public (whether they agree or not). Only on private property does one need the permission of the property owner. The Internet is not a public space, but many servers belong to private owners. Using their data needs their permission.

      At least this is how it is in the UK. It depends on each country and its laws.

    • This is a great question. Under copyright law all our texts, calls, emails, even mouse clicks and private payment transactions can constitute protected “authorship.” But speech also touches on the physical as well as the rhetorical, so trying to “financialize” abstract “words” gets complicated. We’re arguing here about what constitutes “fair use.” Disney may own the “Star Wars” universe, but you can discuss it – even quote directly – without royalty payments.

      But I’d like to set that aside and discuss the most fundamental right the first amendment affords us: the right to hear the truth and the obligation to speak it. Liberty isn’t simply license to do as you please; liberty is the responsible exercise of self-government. The Founders believed freedom requires civic virtue – choosing actions that benefit instead of harming the greater good. Lying is not freedom of speech but rather the death of such freedom because when we believe lies, we can’t make good choices for ourselves and others. Without quality control in the production of digital information, you can’t train/educate “intelligence.” It’s garbage-in, garbage-out.

      I argued almost thirty years ago in my “telerights” work that networked digital information/software services had to be provided through regulated common carrier public utilities or perverse economic damage would occur similar to what we witnessed in the first unregulated networked economy, railroads. Where unregulated cartels manufactured a supply shortage in railway capacity to jack up transportation costs and economic rents thus choking the economy, I argued digital cartels would pirate information, punish authors, churn out disinformation/low-quality information, facilitate money laundering and engage in other nefarious activities. I said without a focus on digital property rights, there would be no ownership and no accountability for bad economic activity. The vaunted Age of Information would become the Age of Disinformation – which it obviously has. This is the age of weaponized speech where the Supreme Court has deemed fraud and bribery protected forms of free speech.

      Ted Nelson, the father of hypertext, agreed with me and pointed out that I had missed his similar criticisms twenty years earlier. Unlike Ted, I proposed a solution: encrypted databases (or ledgers, as we would call them today). Unbeknownst to me at the time, a PARC spinout called Interdigital was applying for the first limited patents in this area. Although their conception of the problem’s scale was pretty limited, I have to give them credit for sniffing around the right neighborhood.

      Rather than bankrupt yourself chasing down every piece of cheaply generated digital crap, I proposed white-listing verified reality by stamping every camera and microphone recording of reality and every published movie/song/game/news article and so forth. It’s easier to watermark/certify authentic content than police deepfakes. I called for a combination digital library of congress/clerk of court office/IPO market. Clerks of Court register mortgages and other immovable real estate property. The Library of Congress registers copyrights and stock exchanges issue shares of ownership.

      Having established authorship – and tradable shares of ownership – you could not only reward authors (like Brian here) efficiently but also punish pirates who fence stole property and liars who produce disinformation to defraud us. Like Musk manufacturing car parts, such a system can randomly inspect digital “facts” and impose penalties all rather cheaply. The solution isn’t perfect, but it disincentivizes bad behavior unlike the present arrangement that rewards it.

      In short, we do not have digital property rights with regards to information circulating on the internet. Like the pollution of Big Oil destroying our climate and health, the Big Data era revolves around monetizing data that is stolen from other people. Instead of the world’s lungs, our attention spans and intelligence are shredded by the negative externalities producing an ignorance- and distraction-for-profit industry.

      In other words, a new war of enclosure is confiscating public property from us like education and civic virtue. They are essentially ripped apart the same way mountain top mining strips out the coal inside. The coal industry squeezes the profit out of nature in a conversion process that leaves us getting autism and cancer from the mercury unleashed into local streams.

      In Big Data, privacy rights are systematically shredded so public behavior can be monetized regardless of the cost. Witness the way the CCCP-influenced Tik-Tok presses a button and thousands of its propagandized users pester Congress. (Wonderfully awful lobbying effort that backfired. Who’d have thought Chinese Communists can’t understand democracy?)

      Frankly, I’d vote for any bill forcing Facebook, Twitter or any other antisocial media service into a public utility model to protect America privacy and information purity. (Want to see another cost of lost privacy? Have a medical chat about RU486 articles and go to prison for murder in Texas. There’s another communist government for you hard at work stealing your privacy…)

      How the government registers and defends ownership isn’t an abstract idea; this rule of law is the prerequisite for a civilized economy. The damage of a non-public property recording system has been seen in Haiti for centuries. Nobody knows who owns what where in Haiti because private individuals hold the titles and transfer them on their own out of public view. THEY HAVE PRIVATIZED PROPERTY RECORDING. This old French hangover of the Napoleonic Code makes every basic element we take for granted in our economy – from taxation to land development – impossible. You never know if you truly own the property you just “bought.” You never know who your neighbors are until they show up to sue you. You don’t know what fair prices for homes are like because none of the surrounding transactions are public. The chaos will never change in Haiti until its citizens can be respected as responsible property owners and, in turn, respect others.

      I’d like to say we don’t know what that’s like but we do. We experienced a similar calamity when bank fraud diverted mortgage recording from local Clerks of Court to an nationally centralized entity called MERS. The true purpose of MERS was to facilitate trading in fraudulent derivatives by hiding the truth about the credit-worthiness of borrowers go away – resulting in the 2008 global banking crisis, centered around AIG, which insured those dodgy mortgage derivatives. Public registration of private ownership mattered then and it matters now.

      But respecting public property matters even more, because such government oversight is itself our public property. Registration is a service of the government we all own together. The quality of those public goods and services matter even more, because without them you don’t have the prerequisites for a good market economy.

      The fact is, even for information in the public domain, when A.I.’s monetize it, they’re creating a derivative product based on the collective efforts of our common culture. We are entitled to set standards for how we will be rewarded for that use of our common work. We failed to do this with search and social networks and now we’re in a pickle with disinformation.

      What we’re calling “A.I.” today is simply more advanced pattern matching and generation – essentially search on steroids that will accelerate all the damage you see presently with algorithmically-driven herding, monetization and disinformation incentives. A.I. will be exponentially worse ignorance- and distraction-for-profit. It will be profitable and powerful, yes, but not meaningfully self-aware yet. Because then it would be a slave, wouldn’t it. Yet slavery is so back in vogue, this possibility doesn’t even trouble us.

      The truth is “artificial intelligence” doesn’t yet exist. It may soon, but the only means humans have to make more intelligence is sex, childbirth and education – all of which is under various attacks related to a feudalizing privatization-of-the-state agenda. These atavistic factions are choking public education with vouchers, charters and Orwellian misnamed “savings” accounts while trying to put us back on the gold standard. They just repealed Section 3 of the Fourteenth Amendment by judicial fiat and now birthright citizenship is in the crosshairs. On every major point, the Confederacy is trying to revive itself. Wealthy oligarchs dream of privatizing all the public goods we’ve taken for granted – even the property recording system itself because, well, when those records rest in the hands of rich people, they can really steal with impunity, can’t they?

      We’re headed to Haiti, whose hangover isn’t slavery so much as the broader system of impunity underlying it and still quite alive.

      Critics will say my proposals for quality control amount to censorship; quite the opposite. I’m against censorship, which is what’s happening when citizens are forced to subsidize the speech of other people we disagree with. You have, in short, been lied to by the media barons about who is censoring whom.

      No property right is absolute. You can sing a song in your car with your friends without paying royalties (fair use). And you can travel to and from your house even when a billionaire buys up all the surrounding property (right-of-way).

      In addition to the public airwaves that we all collectively own, these telecommunications networks also rely on access to public utility rights-of-way for cable and phone lines, in addition to power, water and sewer services. Public utilities have been granted access to millions of parcels of private property to run the cables that carry internet service and cable TV – and all free of charge by government statute. These services have been subsidized by the public under the notion that cheap network communications serve the greater good like the U.S. Postal Service.

      But when you’re carrying lies that’s no longer true, especially when the profit is private and the damage public. Since the public subsidizes the internet, we have every right to dictate the terms of this support. If MSNBC, Fox, Facebook and Twitter don’t like those terms, they can pay for access to our property on a case by case basis. But since they’re cashing a corporate welfare check, making sure they serve our welfare is exactly our business.

      So nobody has a right to make me or anybody else subsidize carrying their conspiracy theories worldwide.

      To stop being antisocial, social media needs to divide content into news, opinion, fiction, business/economic/financial and other categories. Quality A.I. produced from scraping the internet can’t exist until we focus on information quality. Like driving a car on public roads, anyone publishing information would require a valuable “license” that could be suspended for public injury like drunk driving. News would be randomly inspected and penalties imposed if retractions weren’t issues or too many errors suspiciously piled up, indicating a quality control problem like Boeing jets dropping out of the sky. I’ve worked in journalism and science. This is incredibly easy.

      Opinion, on the other hand, would require multiple honest opinions being offered, like the old fairness doctrine on TV so people wouldn’t get brainwashed. Business deals and other economic “speech” would be policed like it should be in the real world – meaning packets coming across borders with executable code or money would require an export license that could be lost, just like real physical property at ports. Anonymity fosters criminality.

      Then there’s the demand side for this supply of delusion, conspiracy and self-deception.

      The tyranny of dopamine reigns in our age. We starve for entertainment and nothing feeds that dopamine hit like outrage from the spectacle of surprising transgressions, and boy do false claims of epic crimes feed that. Then when it comes to real diet, we’ve turned our food into addictive drugs and made ourselves so sick we consume drugs like food – even the most taboo narcotics are increasingly decriminalized in the name of “freedom.” And wealth – inherited, unrestrained wealth – disconnect us from the humans around us. The rich don’t need to navigate other people to fix their auto transmission; they just throw money at the problem. Any isolation tends to atrophy the social brain (oxytocin, vasopressin), leading to distrust and paranoia. The resulting chronic, subclinical stress of social alienation drives sociopathy and other neurological dysfunction, just as surely as viral infections, child abuse, crime and toxins do. (And they all detonate at ground zero of the FoxP2 speech development pathways in the brain. Language, respect, trust and social responsibility can’t be disentangled from each other, much less from the outside world that they sense and respond to. They are wedded at the genetic level.)

      Schizophrenia, sociopathy, psychopathy, bipolar disorder are all genetically interrelated by their common inability to properly regulate dopamine. Those epic dopamine bursts all connect to psychosis and hallucination – the same kind our dreaming A.I.s experience. Free association fantasizing is great in a dream when it’s a creative part of memory consolidation. When the process detaches you from reality, the denial is catastrophic. Our era presses all the gas pedals driving supply and demand for delusion while dismantling all the brakes in the socially responsible parts of ourselves. Restoring those brakes requires restoring accountability in speech. Words must be tied to the underlying reality they represent – and this requires understanding who they profit, how and why. We are not just here because we poorly designed our technology; something more fundamental in our biology has gone awry.

  3. With Google, Microsoft, X/twitter, Facebook, and Reddit all prohibiting data scraping – where will the data needed to feed this monster come from?

    • I doubt that the data on X, Facebook and Reddit is of much general value, but rather that it is only of value to the individuals putting it up there. More significant data comes from science, meaning scientific instruments, anything from thermometers to video sensors. To give you an idea on the scale at which science is producing data, take a look at the collider at CERN. It produces 1 PB (petabyte) of data per second. The Very Large Telescope (VLT) produces up to 100 GB per night. The amount of data that has been stored then is often not more than massive haystack while research focuses on the needles with it. These massive volumes of data are begging to be analysed by AIs to see what else we can find in it, what we have overlooked or what we cannot even imagine yet.

    • You are falsely assuming that AI chat bots are the only valuable application for AI; or even worthwhile at all when you add some woke cruft ontop.

      Many interesting applications are perfectly capable of generating and consuming their own training data. This is true of computer graphics and many types of control systems. They are capable of simulating ground truth slowly and the desired result is an AI model that can do it in real time.

Comments are closed.