Vanishing gradient problem

The Vanishing Gradient Problem is an issue encountered during the training of deep neural networks where gradients become exceedingly small. This phenomenon results in slow or stalled learning, making it difficult for the network to update its weights effectively. The problem primarily affects deep networks and can be mitigated by techniques such as using ReLU activation functions, proper weight initialization, batch normalization, and gradient clipping.

Vanishing gradient problem was discovered on April 22nd 2019 and it currently has a search volume of 14.8K with a growth of +72%.