* a linear programming problem that would take 82 years to solve in 1988 could be solved in one minute in 2003. Hardware accounted for 1,000 times speedup, while algorithmic advance accounted for 43,000 times.
* algorithm speedup between 1991 and 2013 for mixed integer solvers was 580,000 times, while the hardware speedup of peak supercomputers increased only 320,000 times.
* Similar results are rumored to take place in other classes of constrained optimization problems and prime number factorization.
Given the rate and the magnitude of algorithm evolution, many of those alternative AI chip designs may become obsolete even before their commercial releases. AI algorithms of tomorrow might demand different compute architectures, memory resources, data-transfer capabilities, etc.
DeepScale, spun-out of UC Berkeley research, squeezes AI for advanced driver assistance systems (ADAS) and autonomous driving onto automotive grade-chips (as opposed to GPUs). Their neural network models have demonstrated 30 times speedup compared to leading object detection models using algorithms alone while reducing energy and memory footprint enough to run on existing hardware in just a couple of years.
Another example of such algorithmic leapfrogging came from researchers from the Allen Institute of Artificial Intelligence. Using a novel mathematical approach employing binarization of neural networks, they showed that they can drastically increase speed as well as reduce power and memory requirements. This enables even the most advanced deep-learning model to be deployed on a chip as small as a $5 Raspberry Pi. The researchers recently spun out the algorithms and processing tools as XNOR.ai* to deploy AI on edge devices and drive further algorithmic advances for AI.
Brian Wang is a Futurist Thought Leader and a popular Science blogger with 1 million readers per month. His blog Nextbigfuture.com is ranked #1 Science News Blog. It covers many disruptive technology and trends including Space, Robotics, Artificial Intelligence, Medicine, Anti-aging Biotechnology, and Nanotechnology.
Known for identifying cutting edge technologies, he is currently a Co-Founder of a startup and fundraiser for high potential early-stage companies. He is the Head of Research for Allocations for deep technology investments and an Angel Investor at Space Angels.
A frequent speaker at corporations, he has been a TEDx speaker, a Singularity University speaker and guest at numerous interviews for radio and podcasts. He is open to public speaking and advising engagements.
First there was bubble sort (1956), then there was quick sort (1959), then there was heap sort (1964). Heap-sort is similar to quicksort, but with better worst-case performance.
First there was bubble sort (1956) then there was quick sort (1959) then there was heap sort (1964). Heap-sort is similar to quicksort but with better worst-case performance.
Why? Do you expect some sort of programming/SW revolution? I don’t.
And probably OOP and GC. Nowadays ~all programs gobble up and leak memory.
Why? Do you expect some sort of programming/SW revolution? I don’t.
And probably OOP and GC. Nowadays ~all programs gobble up and leak memory.
Improvements in CPU speed have stagnated. Future improvements in solving difficult problems will have to be by improving the algorithms.
Better algorithms can provide orders of magnitude improvements upon the second best. Things of working with exponential vs polynomial execution times. What hardware is very good at, is at improving the cost of constant overhead & operations, like those of doing arithmetic or other operations, moving data from some place to another or to/from a I/O device. The best algorithms couldn’t get us many near miraculous things hardware can do just playing with or giving you more of those constant cost operations per unit of time. But a good new algorithm can open a whole set of applications that weren’t possible before it.
Well, yeah, that’s not a surprise. Doesn’t GLADIS manage to survive in Portal 2, even when she is reduced to a single motherboard and a potato battery?
Improvements in CPU speed have stagnated. Future improvements in solving difficult problems will have to be by improving the algorithms.
Better algorithms can provide orders of magnitude improvements upon the second best. Things of working with exponential vs polynomial execution times.What hardware is very good at is at improving the cost of constant overhead & operations like those of doing arithmetic or other operations moving data from some place to another or to/from a I/O device.The best algorithms couldn’t get us many near miraculous things hardware can do just playing with or giving you more of those constant cost operations per unit of time. But a good new algorithm can open a whole set of applications that weren’t possible before it.
Well yeah that’s not a surprise. Doesn’t GLADIS manage to survive in Portal 2 even when she is reduced to a single motherboard and a potato battery?
Hardware is just a physical representation of an algorithm. If you get algorithm gains you can make hardware to to run it even faster. If you are going to assume fixed hardware tech, like an instruction based CPU, then you’d expect algorithmic gains to be more effective.
If this is true, then we should see a dramatic speed up of software perfornance running on any given hardware platform starting in the near future. In other words, we should see a reversal of the “bloatware” progression that has been the case for the past 25 years. I’ll believe it when I see it.
Hardware is just a physical representation of an algorithm. If you get algorithm gains you can make hardware to to run it even faster. If you are going to assume fixed hardware tech like an instruction based CPU then you’d expect algorithmic gains to be more effective.
If this is true then we should see a dramatic speed up of software perfornance running on any given hardware platform starting in the near future. In other words we should see a reversal of the bloatware”” progression that has been the case for the past 25 years. I’ll believe it when I see it.”””
So-called ‘bloatware’ is a result of using a substandard, spyware program as your operating system. Linux of today can run on computers of a decade ago and more without issue. Windows can’t run on computers of today without issue.
So-called ‘bloatware’ is a result of using a substandard spyware program as your operating system. Linux of today can run on computers of a decade ago and more without issue. Windows can’t run on computers of today without issue.
Bloatware is the ultimate evil.
Bloatware is the ultimate evil.
Several alternatives are improving the computer architecture and running the algorithm in parallel on multiple processors. Note, processors aren’t getter faster, they’re getter smaller… though there is a limit to that too.
The current state of the art deep learning is in the low millions for number of variables. I have mighty big doubts these can be run on small embedded processors. Also, as the state of the art advances and the number of variables continues to grow there is no way an off the shelve processor can keep up. Note, we’ve also been talking about the evaluation and not the determination of the networks. The evaluation is trivial compared to the determination which requires huge server farms worth of GPUs.
Several alternatives are improving the computer architecture and running the algorithm in parallel on multiple processors. Note processors aren’t getter faster they’re getter smaller… though there is a limit to that too.
The current state of the art deep learning is in the low millions for number of variables. I have mighty big doubts these can be run on small embedded processors. Also as the state of the art advances and the number of variables continues to grow there is no way an off the shelve processor can keep up. Note we’ve also been talking about the evaluation and not the determination of the networks. The evaluation is trivial compared to the determination which requires huge server farms worth of GPUs.
We’ve known that for decades. First there was heap-sort, then bubble sort, then quicksort – each step being much faster than the last. Now we have the advantage of both faster hardware and better algoeithms
We’ve known that for decades. First there was heap-sort then bubble sort then quicksort – each step being much faster than the last. Now we have the advantage of both faster hardware and better algoeithms
Why? Do you expect some sort of programming/SW revolution? I don’t.
And probably OOP and GC. Nowadays ~all programs gobble up and leak memory.
First there was bubble sort (1956), then there was quick sort (1959), then there was heap sort (1964). Heap-sort is similar to quicksort, but with better worst-case performance.
We’ve known that for decades. First there was heap-sort, then bubble sort, then quicksort – each step being much faster than the last. Now we have the advantage of both faster hardware and better algoeithms
Several alternatives are improving the computer architecture and running the algorithm in parallel on multiple processors. Note, processors aren’t getter faster, they’re getter smaller… though there is a limit to that too.
The current state of the art deep learning is in the low millions for number of variables. I have mighty big doubts these can be run on small embedded processors. Also, as the state of the art advances and the number of variables continues to grow there is no way an off the shelve processor can keep up. Note, we’ve also been talking about the evaluation and not the determination of the networks. The evaluation is trivial compared to the determination which requires huge server farms worth of GPUs.
Bloatware is the ultimate evil.
So-called ‘bloatware’ is a result of using a substandard, spyware program as your operating system. Linux of today can run on computers of a decade ago and more without issue. Windows can’t run on computers of today without issue.
Hardware is just a physical representation of an algorithm. If you get algorithm gains you can make hardware to to run it even faster. If you are going to assume fixed hardware tech, like an instruction based CPU, then you’d expect algorithmic gains to be more effective.
If this is true, then we should see a dramatic speed up of software perfornance running on any given hardware platform starting in the near future. In other words, we should see a reversal of the “bloatware” progression that has been the case for the past 25 years. I’ll believe it when I see it.
Improvements in CPU speed have stagnated. Future improvements in solving difficult problems will have to be by improving the algorithms.
Better algorithms can provide orders of magnitude improvements upon the second best. Things of working with exponential vs polynomial execution times.
What hardware is very good at, is at improving the cost of constant overhead & operations, like those of doing arithmetic or other operations, moving data from some place to another or to/from a I/O device.
The best algorithms couldn’t get us many near miraculous things hardware can do just playing with or giving you more of those constant cost operations per unit of time. But a good new algorithm can open a whole set of applications that weren’t possible before it.
Well, yeah, that’s not a surprise. Doesn’t GLADIS manage to survive in Portal 2, even when she is reduced to a single motherboard and a potato battery?