Learning Speed and Learning Rate

The Learning Rate is a hyperparameter that dictates the speed at which a model updates its weights. If the learning rate is too high, the model can overshoot the optimal solution and diverge, and if it’s too low, it might not reach the optimal solution or take too long to converge.

Learning rates typically range from 0 to 1, but in practice, very small values below 0.1 are often used.

• 0.1 (10%): This is a relatively large learning rate. It aims to train rapidly but carries a high risk of divergence.

• 0.01 (1%): Commonly used but still a relatively large learning rate. There's a possibility of divergence.

• 0.001 (0.1%): A widely used learning rate. It tends to be stable but may converge slowly.

• 0.0001 (0.01%): A small learning rate. It ensures stable learning but can be quite slow.

• High Learning Rate: Allows for rapid learning, but if set too high, it may prevent reducing errors during training and cause divergence by overshooting the optimal solution.

• Low Learning Rate: Promotes stable learning; however, if set too low, training can be excessively slow, and the model might overfit by overly adapting to the training data.

1. Learning Rate Decay: Gradually reducing the learning rate as training progresses. Start with a high learning rate for rapid learning, and then lower it later for fine-tuning. This can be done by decreasing the learning rate by a fixed ratio (e.g., decreasing by 1/10 every epoch) or based on the lack of improvement in a specific metric.

2. Adaptive Learning Rate: Adjusting the learning rate flexibly as the model learns. Popular algorithms include Adagrad, Adadelta, RMSprop, and Adam.

3. Cyclic Learning Rate: Periodically varying the learning rate. By alternating the learning rate between higher and lower values, the model can explore a wider parameter space.

In the practice panel on the right, compare finely tuned models using different learning rates trained on historical documents from the 18th century.

An overfitted model overly adapts to specific training data, leading to poor performance on new data.