Neural networks

Such a recurrent structure in Figure 6.5 has ability of nonlinear approximation and temporal representation ability of recurrence. It is also possible to have hidden units in the recurrent backward connections, these being known as context units [17]. In case of sequences have small maximum length; unfolding in time can be used to convert an arbitrary recurrent network to an equivalent network. The resulting network can be trained with backpropagation algorithm. The solution is backpropagation through time [19], summing up the different weights changes in time and changing the weight by the average. The problem with this approach is the memory requirement if the length of the sequence is large. For training recurrent networks without unfolding real time recurrent learning [20] is used and it has the advantage of arbitrary length sequences. 6.5.3. Local Models The neural network structures where the first layer contains locally receptive units that respond to instances in a localized region of the input space are called local models. The second layer learns the regression or classifications function for these local regions.