The End of Moore’s Law in Detail and Starting a New Golden Age

John Hennessy talked about the end of Moore’s Law and the start of a new golden age.

He described the specifics of how general computer chips from Intel and others were sped up over decades. Speed came from scaling, parallelism and multiple cores.

John sees the immediate future of faster computing coming from domain-specific languages and architectures. He sees this being able to provide on the order of 63,000 times faster speeds.

Beyond that we will use nanotechnology, quantum computers and other approaches.

He was speaking at a DARPA meeting. DARPA is funding the exploration of technology and approaches beyond Moore’s law.

Reviewing 40 Years of Moore’s Law

• 40 years of stunning progress in microprocessor design
– 1.4x annual performance improvement for 40+ years ≈ 1 million times x faster (throughput)!
• Three architectural innovations:
– Width: 8->16->64 bit (~4x)
– Instruction level parallelism:
– 4-10 cycles per instruction to 4+ instructions per cycle (~10-20x)
– Multicore: one processor to 32 cores (~32x)
• Clock rate: 3 MHz to 4 GHz (through technology & architecture)
• Made possible by IC technology:
• Moore’s Law: growth in transistor count
– Dennard Scaling: power/transistor shrinks as speed & density increase
– Energy expended per computation is reducing

End of Dennard Scaling is a Crisis

• Energy consumption has become more important to users
– For mobile, IoT, and for large clouds
• Processors have reached their power limit
– Thermal dissipation is maxed out (chips turn off to avoid overheating!)
– Even with better packaging: heat and battery are limits.
• Architectural advances must increase energy efficiency
– Reduce power or improve performance for same power
• The dominant architectural techniques have reached limits in energy efficiency!
– 1982-2005: Instruction-level parallelism
– Compiler and processor find parallelism
– 2005-2017: Multicore
– Programmer identifies parallelism
– Caches: diminishing returns (small incremental improvements).

Instruction-level Parallelism Era 1982-2005

• Instruction-level parallelism achieves significant performance advantages
• Pipelining: 5 stages to 15+ stages to allow faster clock rates (energy neutralized by Dennard scaling)
• Multiple issue: less than 1 instruction/clock to 4+ instructions/clock
– Significant increase in transistors to increase issue rate
• Why did it end?
– Diminishing returns in efficiency

Getting More ILP

• Branches and memory aliasing are a major limit:
– 4 instructions/clock x 15 deep pipeline need more than 60 instructions “in flight”
• Speculation was introduced to allow this
• Speculation involves predicting program behavior
– Predict branches & predict matching memory addresses
– If prediction is accurate can proceed
– If the prediction is inaccurate, undo the work and restart
• How good must branch prediction be—very GOOD!
– 15-deep pipeline: ~4 branches 94% correct = 98.7%
– 60-instructions in flight: ~15 branches 90% = 99%

What OPPORTUNITIES Left?

• SW-centric
– Modern scripting languages are interpreted, dynamically-typed and encourage reuse
– Efficient for programmers but not for execution
• HW-centric
– Only path left is Domain Specific Architectures
– Just do a few tasks, but extremely well
• Combination
– Domain Specific Languages & Architectures

Research Opportunity: New Technology

▪ Silicon:
– Extend Dennard scaling and Moore’s Law
– New methods for efficient energy scaling
– Secure supply chains
▪ Packaging
– Overcome TDP limits for high end
– Tighter integration = more performance & less power
– Integrated 3-5s for optical interconnect.
▪ Beyond Si:
– Carbon nanotubes?
– Quantum?

SOURCES- DARPA, John Hennessy Talk

Written By Christina Wong. Nextbigfuture.com

26 thoughts on “The End of Moore’s Law in Detail and Starting a New Golden Age”

  1. The end of Moore’s law may be the best thing to happen to computing since it started. No competing computing technologies were allowed to take off while bound to the tyranny of Moore’s law. Now that it’s reached it’s end, several different computing paradigms have emerged that have the potential to improve computing capability at a pace that puts Moore’s law to shame.

    Reply
  2. I agree with you that good quality data is one of the main bottlenecks to successful AI. Why not train a neural network that can label data and automate that process? Why not synthesize new data that optimizes training of neural networks if you don’t have enough or good quality data? This is already starting to be done.

    AI is getting better all the time in object recognition and once you can do that, I don’t think it’s hard for AI to label data in an image let’s say. We’re already at the point where the tedious task of labeling data should be able to be automated. It’s not just object and image recognition, but NLP is getting to the point where it should be able to automatically generate its own quality training data.

    Reply
  3. Id have to say I disagree , virtual reality environments already exist to train sensory learning, and learning through linguistic medium can be done by online learning programmes , there are an abundance of video based programmes that teach English for example. Remember that AGI wont need an avalanche of data since its machine learning capabilities will be about 1 million times more efficient that todays machine learning neural nets, that’s assuming that we can closly replicate the neural nets found in humans

    Reply
  4. One may argue that efficiency is only relevant if you wish to stay biologically human.
    Based on current technology the following is possible:
    We can see significantly better than any human (lidar+radar+new tech e.g. SPAD camera). We can hear better than humans, we can calculate faster, move faster, are significantly stronger, less impervious to environmental extremes etc.
    New AI algorithms means we can read raw data and derive equations & formulate strategies significantly faster & better than humans (e.g. go, chess, assorted RPG games, table tennis are some examples).
    re: energy to power our transformer-like human is becoming available -from batteries (least efficient) to small scale nuclear (will become more relevant in future)
    Only missing item is consciousness or adding some code that tells human v2.0 to want to live. (I think this will be the secret recipe that finally gets AGI into the mainstream and results in evolution of humanity.)

    Reply
  5. We are reaching the economical (not physical, yet) limits of current silicon technology, that’s all.
    There’s a higher level law going on here, and Moor’s of Information, if you want, that’s been a constant in human (and even biological) evolution so far.
    For what I can see, we are just in the middle of a change of paradigm, and every slowdown is just a temporary bump on the road. Physical limits of computation (per volume and per unit of energy) are nowhere close: heck, we are orders of magnitude just to reach the efficiency of biological systems!
    Recent announcement from Intel about spintronics almost ready for prime time are a huge step in the right direction. Add memristors, graphene, heterogeneous computation, quantum computing, and optical interconnects and you have lay ground for another ~1.000.000 fold improvements.

    Reply
  6. As with all things “machine learning”, the availability of labelled data is the bottleneck for successful applications. It will take much longer than your guesstimates to build training data for AGI. For humans, this is called schools and we all know how long it takes for a human being to develop extraordinary skills.

    A computer will do the actual training quickly but the data must come from somewhere. A reasonable guess is that the “teachers” for AGI will be the previous best practice AGI implementations. For now and some time, this will be humans, which will put a serious brake on the development speed. Once AGIs can train and design next version AGIs, things will move faster until some other bottleneck, probably related to physics.

    Reply
  7. Then Elon musks company tried to use recommendation thumbs up to filter that type of racist stuff out…so it would be trained by better material.. and all they end up doing was training an AI to be an evil politician with hate speech…

    Reply
  8. sadly AIs strength is evil… they keep using internet social media to train AI to talk like humans… the problem is that every time they do that… the AI is also trained to say racist things… because there’s no way to filter out every account … Microsoft had to shutdown their ai talkbot because it keep making people angry with racist talk..

    Reply
  9. Don’t need anywhere near as many transistors as neurons because neurons are millions of times slower. Neuralnets much larger can be simulated using the higher speed. It is the programming we can’t do, the hardware is just fine…especially the new stuff that will be coming out in just a few years. Self learning is the real key…and programmers are getting that now.
    The nets can be self-trained very rapidly to out perform the best humans in virtually any game now.
    And our nets are hardwired and take a lot of space. Computers can load a new one for any task required reusing the same space.

    Reply
  10. I don’t feel very pumped about the furture of semi industry after hennessy’s Speech… errr… well we’ve reached the period of stagnation… physicist working on quntuan carbon nano tubes… don’t need engineers for 10-15 years… errr.. let’s just make the Eda tools faster so they can finish the new design that they don’t need faster because it’s the same speed as the last design…

    Reply
  11. Good talk. Not much to find fault with. He did not really discuss 3D chips. That is one way to get a lot in a small space. It does take longer to get all those layers on, but that might be sped up too. It also requires very low power per transistor, as you have the same amount of surface to dissipate heat regardless of how many layers you put in there. It is a little like trying to power a skyscraper with solar just on the roof.

    He skipped through instructions and bit size pretty quickly, but the increase in instructions and bits has certainly sped up many applications. And it can continue to do so. In chess for example going from 32-bit to 64-bit can double the speed. Encryption instructions can speed up encryption dramatically. When graphics stuff was added they got large increases in graphics computation (MMX and such). More recently bit shifting also helped chess. But all the various new functions helped various applications. I think we will see machine learning extensions, and more.

    He focused on CPU, but GPU has seen quite a bit of advancement. Ray tracing is new, but there are steps beyond that like path tracing. And I think we will need real physics rather than physics shortcuts that they currently use, as things become less about fooling the eye and more about accurately making simulations of things we will want to build in reality.

    If we ever develop Thermoelectric coolers that are >50% efficient, we can have much higher clocks. Best now are about 15% efficient.

    Reply
  12. I guess quantum is the latest incarnation of the molecular level in semiconductors. Generally these days, quantum refers to quantum computation, a totally different thing.

    Reply
  13. The Beckenstein bound puts a limit on how much information can be stored. A human brain is limited to about 2.6×10 to the 42nd bits. We aren’t even a speck of sand on that beach. New technologies, and better ways of utilizing them, will get closer and closer, so long as there is impetus to do so.

    We have enough breathing room that we could double every 18 months for centuries if can find the means. Of course, after getting used to that kind of growth over centuries, what would happen to a culture that suddenly ran slam bang in to the limit? Why it could very well be an existential crisis. Given how quickly a race might get there from where we are now, maybe that’s a reason for the Fermi Paradox.

    Probably hasn’t happened, but it’s interesting to think about.

    Reply
  14. Since pornography is usually the pioneering use for new information technologies, I can’t help but ask: Will AI chips be able to convincingly “talk dirty”, or will that be the last stronghold of human superiority?

    Reply
  15. Moore’s “law” is that the cost of manufacturing a transistor decreases by half, about every 18 months. He said nothing about speed of calculations, or “computing power”. Even if the feature size stops shrinking, new manufacturing techniques can keep the law alive.

    Obviously, new domain specific languages will not make transistors cheaper. I find it amusing that this “expert” does not know what Gordon Moore actually wrote in his famous article.

    Reply
  16. Not true , according to tachyum the Europeans will have a 34 exaflop supercomputer next year. Full scale , real time brain simulations coming in the next 12 months . And considering the fact that researchers will be getting there hands on MRI scanners that can scan a whole human cortical Colum in real time next year ( Open water MRI) , and that they will have 34 exaflops to simulate and reverse engineer the data with machine learning I think were no more than 3 years away from AGI now. And considering the quantum computing roadmap for 100 000 qubit systems perhaps only 5 years away from artificial super intelligence. Then look at neurlinks plans for neurlaces , human 2.0 in 6 years I think. Kurzweil Predicted that wed get his kind of technology in a decade not 2020, so I bet hell be lowering his estimated date of 2029 shortly.

    https://www.globalbankingandfinance.com/tachyum-backs-open-euro-hpc-project-and-its-goals-of-working-in-collaboration-to-deliver-more-efficient-and-economical-data-center-ai-hpc-solutions/

    Reply
  17. You got that right…. we are not even close to the 10 trillion neurons needed to create artifical intelligence of the level of human

    Reply
  18. No discussion of molecular electronics? Obviously if Moore’s Law is at an end a new technology must be developed. I always assumed that new technology would be molecular electronics.

    Reply
  19. It’s easy to get melodramatic with these declarations, but this really is a golden age for AI (not sure about AGI, though) and a whole different historic period for humanity.

    For a long while things have been more or less the same in which concerns the active agents on history: humans and their endless struggle for power, their deeds, their needs and wants.

    Even if AIs don’t become fully sentient any time soon, they will be a really, radically transformative factor acting upon the world since now until the unforeseeable future.

    Because they will unavoidably stay and become better and better with time.

    Reply
  20. I kind of like these disguised ads that NextBigFuture posts among more legitimate articles on crazy futurism ideas. By “I kind of like these ads” I actually mean “I’d kind of like these ads to stop.” But half truths seem to be the way of this site, so don’t fault me for that.

    Reply

Leave a Comment