At first glance I thought this was going to be another 'tech X uses energy Y and so tech X is bad' but reading the article shows it focuses mainly on lowering the energy cost of computations using a number of different approaches that look interesting. Sparse matrices where every element doesn't need to be re-computed as most stay zero for example. I'm not sure what actual improvement this would bring, but they do follow a good basic notion: if you're going to discuss a problem, try to discuss possible solutions to that problem, and improving the energy-efficiency of computation is always a good idea.
The one issue is that translating electricity usage into fossil fuel equivalents for this specific application, without contextual information about similar energy demands, such as streaming video, data collection/storage/processing (be it at Google or the NSA), total router energy consumption in the global Internet, etc. might result in a distorted view of the relative importance of energy demand for their particular issue (training complex models).
Furthermore, it's not necessary to generate electric power with fossil fuels, is it? My view is that solar/wind/storage is the optimal global-scale solution, but placing energy-hungry steady-load data centers near baseload nuclear power plants is arguably an efficient solution (ask the insurers first, however). Hydropower is region-specific and as the drought shows, subject to going offline when needed most to run AC etc. It's not inevitable that power demands equate to fossil carbon emissions, in other words.
You're right on the face of it, the added context that's missing is Numenta has been pushing bio-inspired AI for like 17 years and nothing practical has come of it. If using some energy to achieve a practical goal is wasteful, using less energy to accomplish nothing is more so.
I got into AI after a grad school career split between mathematical signal processing and computational neuroscience. I knew folks back in the early 2010's looking at joining Numenta. The ideas are absolutely good to explore, it's the execution that's lacking. Maybe their big breakthrough is just around the corner, but how long do you wait for product 1 alpha build 1 before calling out vaporware?
Additionally, the ideas here are well represented in mainstream deep learning research by the supposed energy offenders. For example, here are links representing practical applications using sparse inference, continuous multitask learning, and specialized hardware -- just by Google.
The article is marketing to folks who care about the environment, but have limited background on modern ML (and perhaps also energy economics).
That's valid. As I was reading through it I was wondering if their claims about how brains actually work internally were all that well supported, things like:
A biological neuron has two kinds of dendrites: distal and proximal. Only proximal dendrites are modeled in the artificial neurons we see today.
Nerve cells in the brain also have what, ~20 different chemical neurotransmitters modulating the state of the neuron in question, and this looks a lot more analog than digital... any kind of one-to-one correspondence between biological brains and digital learning networks never seemed all that realistic to me.
I don't think their scientific claims are all that inaccurate, maybe some stuff is simplified but that's inevitable. To my understanding, the issue is more converting that advanced scientific knowledge into something tangible.
Can I build a Google photos competitor using spiking neural nets for content recognition? Can i train GPT-4 with 50% of the power bill of GPT-3? Can i dare hope for better accuracy or robustness for some production task? Or better compressed nets for edge computing? I'm not seeing anything like that. We've known a long time that ANNs as implemented today are very crude compared to biological neurons; they really just share a name. What's supposed to differentiate a company from a lab is taking that insight and doing something concrete with it.
A neuron in the brain is more like it's own microprocessor than just a memory cell anyway, which is why the current style of ANNs ('bio-inspired' or not) will never come close to the computational complexity involved in anything like a mammalian brain.
Yes he is not keeping current. Modern AI is differentiable functions, taking about "neurons" and their mapping to brain hardware at this point is old school at best.
The one issue is that translating electricity usage into fossil fuel equivalents for this specific application, without contextual information about similar energy demands, such as streaming video, data collection/storage/processing (be it at Google or the NSA), total router energy consumption in the global Internet, etc. might result in a distorted view of the relative importance of energy demand for their particular issue (training complex models).
Furthermore, it's not necessary to generate electric power with fossil fuels, is it? My view is that solar/wind/storage is the optimal global-scale solution, but placing energy-hungry steady-load data centers near baseload nuclear power plants is arguably an efficient solution (ask the insurers first, however). Hydropower is region-specific and as the drought shows, subject to going offline when needed most to run AC etc. It's not inevitable that power demands equate to fossil carbon emissions, in other words.