Abstract

The choice of training parameters and training rules is of great significance in on-line training of neural networks. We employ a variational method for determining globally optimal learning parameters and learning rules for on-line gradient descent training of multi-layer neural networks. The approach is based on maximizing the total decrease in generalization error over a fixed time-window, using a statistical mechanics description of the learning process. The method is employed for obtaining optimal learning rates in both realizable and noise-corrupted tasks, for determining the relation between optimal learning rates of different weights and for examining the efficacy of regularizers in noisy and over-realizable training scenarios. Scaling rules for the optimal learning rates are obtained in learning generic tasks by linearizing the dynamics around transient and asymptotic fixed points. The method is further employed for determining the globally optimal on-line learning rule, which is shown to be superior to the locally optimal rule.

Introduction

Feed-forward neural networks have been extensively applied during the last decade for a variety of classification, regression, prediction and control tasks and are the most commonly used neural network architecture. On-line learning is arguably the most efficient way of training large feed-forward networks, especially when the task to be learnt is non-stationary, and is based on instantaneous modifications of the network parameters calculated according to only the latest in a sequence of training examples. This process is inherently stochastic because a new training example is selected at random each time the training error is determined. This is to be contrasted with batch learning, in which all the training examples are used to determine the training error, leading to a deterministic algorithm.