New Linear-complexity Multiplication (L-Mul) algorithm claims it can reduce energy costs by 95% for element-wise tensor multiplications and 80% for dot products in large language models. It maintains or even improving precision compared to 8-bit floating point operations.
By replacing complex floating-point multiplication with integer addition
Solution in this Paper
– Approximates floating-point multiplication using integer addition
– Linear O(n) complexity vs O(m^2) for standard floating-point multiplication
– Replaces tensor multiplications in attention mechanisms and linear transformations
– Implements L-Mul-based attention mechanism in transformer models
—–
Key Insights from this Paper
– L-Mul achieves higher precision than 8-bit float operations with less computation
– Potential 95% energy reduction for element-wise tensor multiplications
– 80% energy reduction for dot products compared to 8-bit float operations
– Can be integrated into existing models without additional training
–
Results
– L-Mul with 4-bit mantissa: comparable precision to float8 e4m3
– L-Mul with 3-bit mantissa: outperforms float8 e5m2
– Attention mechanism replacement: 0.07% average performance loss across NLP tasks
– Vision tasks: 0.12% accuracy improvement
– Full model fine-tuning: equivalent results to float8 e4m3 accumulation precision
MASSIVE claim in this Paper for LLM Training 🤯
This new Linear-complexity Multiplication (L-Mul) algorithm can reduce energy costs by 95% for element-wise tensor multiplications and 80% for dot products in large language models, while maintaining or even improving precision… pic.twitter.com/wtZAkUfTc7
— Rohan Paul (@rohanpaul_ai) October 21, 2024
Brian Wang is a Futurist Thought Leader and a popular Science blogger with 1 million readers per month. His blog Nextbigfuture.com is ranked #1 Science News Blog. It covers many disruptive technology and trends including Space, Robotics, Artificial Intelligence, Medicine, Anti-aging Biotechnology, and Nanotechnology.
Known for identifying cutting edge technologies, he is currently a Co-Founder of a startup and fundraiser for high potential early-stage companies. He is the Head of Research for Allocations for deep technology investments and an Angel Investor at Space Angels.
A frequent speaker at corporations, he has been a TEDx speaker, a Singularity University speaker and guest at numerous interviews for radio and podcasts. He is open to public speaking and advising engagements.
Really cool to be living in this time of clear exponential progress. That said, I think that the demand for intelligence will probably gobble up a couple of orders of magnitude without blinking.
Oh well. Guess we won’t need all those Google-funded Kairos FHR reactors to power AI.
Who would have imagined that there were efficiencies to be had in wasteful computing? /s
The way to go is not just building gigantic GPU centers, but way simpler. Better coding, better algorithms. Consume less, train less, but be more efficient with less waste and more smart. Don’t forget how good the human brain works and doesn’t need gigantic clusters of GPUs.
Cool. I did something similar in a PLC setpoint ramping subroutine once long ago. It really unburdened the processor (there were many ramped setpoints).
Will take dedicated hardware to implement this, but seems valuable – more so for inference than training, and will certainly help to make mobile robotics many times smarter for a given power budget – Tesla should chase this hard for FSD and Optimus.
But what it doesn’t fix is the approx 1/3rd of power consumed by memory.
There was another paper along similar lines a few months back using ternary weights -1,0,1 with even greater power and number of weight savings. https://arxiv.org/abs/2402.17764
Guess we’ll see which is going to win, my money would be on ternary – once ASICs are built to optimally implement it.
Oops forgot to proof read above, should be. Ternary logic taking only values -1,0,1 eliminating multiplications with even greater power savings and reduced number of gates to implement.