“We’re working on a next generation of the chip, but what’s most important now is commercial partners,” says John Kelly, a senior vice president at IBM who oversees IBM Research and several business units, including two dedicated to the company’s Watson suite of machine intelligence software. “Companies could incorporate this in all sorts of mobile devices, machinery, automotive, you name it.”
Adding brain-inspired chips to products such as phones could make them capable of recognizing anything their owners say and tracking what’s going on around them, says Kelly. The closest today’s devices come to that is listening out for certain keywords. Apple’s latest iPhone can be roused by saying “Hey Siri,” and some phones using Google’s software can be woken with the phrase “OK Google.”
The TrueNorth chip unveiled last August is roughly the size of a postage stamp and has one million silicon “neurons” with 256 million connections between them that are analogous to the synapses that link real neurons. The chip consumes over 1,000 times less power than a conventional processor of a similar size. IBM has demonstrated how its network of neurons can be programmed to perform tasks such as recognizing different vehicles in video footage in real time.
However, because the TrueNorth chip architecture is very different from those in existing computers it requires new approaches to writing software.
Dharmendra Modha, who leads development of IBM’s brain-inspired chips, counters that spiking is critical if neural networks are to be run in a chip with high power efficiency. His team has begun to create tools that will make it possible to transfer trained-up deep learning neural networks onto a TrueNorth chip, he says.
“This chip was envisioned as a substrate onto which a large variety of neural networks can be mapped for real-time, ultra-low energy, ultra-low volume applications,” he says.
New research from another pioneer of deep learning, Yoshua Bengio of the University of Montreal, suggests that the technique’s accuracy could be easier to transfer to spiking hardware neurons than was previously thought, says Sejnowski. Bengio, who collaborates with IBM on language software, posted a preliminary paper online last week showing that tweaking the simulated neurons used in deep learning in a way that makes them more like spiking neurons didn’t harm accuracy on image processing.
Arxiv - Neural Networks with Few Multiplications
For most deep learning algorithms training is notoriously time consuming. Since most of the computation in training neural networks is typically spent on floating point multiplications, we investigate an approach to training that eliminates the need for most of these. Our method consists of two parts: First we stochastically binarize weights to convert multiplications involved in computing hidden states to sign changes. Second, while back-propagating error derivatives, in addition to binarizing the weights, we quantize the representations at each layer to convert the remaining multiplications into binary shifts. Experimental results across 3 popular datasets (MNIST, CIFAR10, SVHN) show that this approach not only does not hurt classification performance but can result in even better performance than standard stochastic gradient descent training, paving the way to fast, hardware-friendly training of neural networks.
SOURCES- Technology Review, Arxiv