Reducing the Spike Rate in Deep Spiking Neural Networks

One objective of Spiking Neural Networks is a very efficient computation in terms of energy consumption. To achieve this target, a small spike rate is of course very beneficial since the event-driven nature of such a computation. However, as the network becomes deeper, the spike rate tends to increase without any improvements in the final results. On the other hand, the introduction of a penalty on the excess of spikes can often lead the network to a configuration where many neurons are silent, resulting in a drop of the computational efficacy. In this paper, we propose a learning strategy that keeps the spike rate under control, by (i) changing the loss function to penalize the spikes generated by neurons after the first ones, and by (ii) proposing a two-phase training that avoids silent neurons during the training.