208 Four-layer gradient disappearance problem

208 Four-layer gradient disappearance problem

The gradient vanishing problem is a phenomenon in which the weights are not updated as well as the layers closer to the input when a deep neural network is learning. This problem occurs in the process of learning by the "error back propagation method," in which the neural network propagates the error from the output layer to the input layer in the opposite direction. In that propagation, the magnitude of the error gets smaller and smaller as it passes through each layer, and eventually there is almost no change near the input layer, i.e., learning stops.

In particular, when using conventional activation functions such as the sigmoid function or the tanh function, the output is within a certain range, and when the output is saturated, the response to it is also sluggish. As a result, as one goes deeper into the layer, it becomes more difficult for changes to be transmitted, and the gradient approaches as close to zero as possible. This is the essence of the "disappearing gradient" phenomenon.

For example, consider a network with up to the fourth layer counting from the input layer. By the time the first layer is reached, the change may be so small that it is barely perceptible, which causes learning to stall. In other words, the problem becomes more pronounced with each deeper layer, and the effects already begin to appear at about layer 4.

In this situation, the parameters in the layers close to the input are rarely updated, and the entire network cannot learn effectively. This prevents the model from taking advantage of the expressive power that it should have, even though it has been built up to the deepest layers.

To counter this problem, there are several ways to prevent extremely small changes in output. For example, the activation function called ReLU has the property that when the output is positive, the response does not slow down and the gradient is preserved. In addition, batch normalization, which aligns the output of each layer, and a method of initializing weights devised to keep the gradient within an appropriate range are also effective. In addition, for networks such as RNNs, which have an iterative structure, this problem is particularly severe, so a special structure called LSTM has been devised to allow learning while preserving long-term information.

Thus, the gradient loss problem is an unavoidable and critical issue in training deep networks, but its impact can be mitigated by various innovations. even for relatively shallow networks such as 4-layer networks, one cannot be too careful and it is important to design without overlooking signs of the problem.