Dynamically Constraining Hidden-Layer Representations
to Reduce Catastrophic Forgetting in Connectionist Networks
Dr. Robert M. French
Department of Psychology
University of Wisconsin-Madison
french@head.neurology.wisc.edu
2:30 pm Fri. Sep. 30 in 2310 Computer Sciences and Statistics Bldg.
It is well known that when a connectionist network is trained on one
set of patterns and then attempts to add new patterns to its
repertoire, catastrophic interference may result. The use of sparse,
orthogonal hidden-layer representations has been shown to reduce
catastrophic interference. The author demonstrates that the use of
sparse representations may, in certain cases, actually result in worse
performance on catastrophic interference. This paper argues for the
necessity of maintaining hidden-layer representations that are both as
highly distributed and as highly orthogonal as possible. The author
presents a learning algorithm, called context-biasing, that
dynamically solves the problem of constraining hidden-layer
representations to simultaneously produce good orthogonality and
distributedness. On the data tested for this study, context-biasing
is shown to reduce catastrophic interference by more than 50% compared
to standard backpropagation.