CRNet: A Fast Continual Learning Framework With Random Theory
Depeng Li, Zhigang Zeng
Abstract
Artificial neural networks are prone to suffer from catastrophic forgetting. Networks trained on something new tend to rapidly forget what was learned previously, a common phenomenon within connectionist models. In this work, we propose an effective and efficient continual learning framework using random theory, together with Bayes' rule, to equip a single model with the ability to learn streaming data. The core idea of our framework is to preserve the performance of old tasks by guiding output weights to stay in a region of low error while encountering new tasks. In contrast to the existing continual learning approaches, our main contributions concern (1) closed-formed solutions with detailed theoretical analysis; (2) training continual learners by one-pass observation of samples; (3) remarkable advantages in terms of easy implementation, efficient parameters, fast convergence, and strong task-order robustness. Comprehensive experiments under popular image classification benchmarks, FashionMNIST, CIFAR-100, and ImageNet, demonstrate that our methods predominately outperform the extensive state-of-the-art methods on training speed while maintaining superior accuracy and the number of parameters, in the class incremental learning scenario. Code is available at https://github.com/toil2sweet/CRNet.