Incremental algorithm for parallel training of the size and the weights in a feedforward neural network

An algorithm of incremental approximation of functions in a normed linear space by feedforward neural networks is presented. The concept of variation of a function with respect to a set is used to estimate the approximation error together with the weight decay method, for optimizing the size and weights of a network in each iteration step of the algorithm. Two alternatives, recursively incremental and generally incremental, are proposed. In the generally incremental case, the algorithm optimizes parameters of all units in the hidden layer at each step. In the recursively incremental case, the algorithm optimizes the parameters corresponding to only one unit in the hidden layer at each step. In this case, an optimization problem with a smaller number of parameters is being solved at each step.