The learning rate is one of the most critical hyperparameters in training neural networks. It determines the step size at each iteration while moving toward a minimum of a loss function. If the learning rate is too high, the model may overshoot the optimal solution or diverge. If it is too low, the training process may be too slow or get stuck in a suboptimal local minimum.
While the "initial" learning rate is often set based on experimentation or methods like the "LR Range Test," calculating the current learning rate during training is usually a function of a decay schedule. Learning rate decay (or annealing) reduces the learning rate as training progresses, allowing for large steps in the beginning and finer adjustments towards the end.
Common Learning Rate Decay Formulas
This calculator supports three of the most standard decay schedules used in frameworks like TensorFlow and PyTorch:
1. Time-Based Decay
This schedule decreases the learning rate gradually based on the iteration or epoch number. It is mathematically represented as:
ηₜ = η₀ / (1 + k * t)
Where:
ηₜ: Learning rate at epoch t.
η₀: Initial learning rate.
k: Decay rate hyperparameter.
t: Current epoch iteration.
2. Step Decay
Step decay drops the learning rate by a specific factor at defined intervals (e.g., halving the rate every 10 epochs). This creates a "staircase" effect on the learning rate graph.
A Drop Factor of 0.5 means the rate is cut in half. Epochs Drop defines how frequently this cut happens.
3. Exponential Decay
Exponential decay reduces the learning rate continuously at an exponential rate. It is often preferred for smoother convergence.
ηₜ = η₀ * e^(-k * t)
Here, e is Euler's number (approx 2.718), and k is the decay rate.
Why Use a Learning Rate Scheduler?
Faster Convergence: Starting with a higher learning rate helps the model traverse the loss landscape quickly.
Better Accuracy: Lowering the rate later in training allows the model to settle into deeper, narrower parts of the minimum, often improving final accuracy.
Stability: It prevents oscillation around the minimum, which is common with a fixed, high learning rate.
Tips for Tuning Learning Rate
Start Small: Common initial values range from 0.1 to 0.0001 depending on the optimizer (SGD vs Adam).
Check Gradients: If gradients explode (NaN values), your learning rate might be too high.
Batch Size Relation: There is often a linear relationship between batch size and learning rate. If you double your batch size, you can often double your learning rate safely.
function toggleInputs() {
var type = document.getElementById('decayType').value;
var kGroup = document.getElementById('kDecayGroup');
var dropFactorGroup = document.getElementById('dropFactorGroup');
var epochDropGroup = document.getElementById('epochDropGroup');
if (type === 'step') {
kGroup.style.display = 'none';
dropFactorGroup.style.display = 'block';
epochDropGroup.style.display = 'block';
} else {
kGroup.style.display = 'block';
dropFactorGroup.style.display = 'none';
epochDropGroup.style.display = 'none';
}
}
function calculateLearningRate() {
// Get Inputs
var type = document.getElementById('decayType').value;
var initialLR = parseFloat(document.getElementById('initialLR').value);
var epoch = parseFloat(document.getElementById('epoch').value);
// Validation
if (isNaN(initialLR) || initialLR < 0) {
alert("Please enter a valid Initial Learning Rate.");
return;
}
if (isNaN(epoch) || epoch < 0) {
alert("Please enter a valid Epoch number.");
return;
}
var resultLR = 0;
var decayParamDisplay = "";
// Logic Switch
if (type === 'time') {
var k = parseFloat(document.getElementById('decayRate').value);
if (isNaN(k)) { alert("Please enter a valid Decay Rate (k)."); return; }
// Formula: LR = LR0 / (1 + k * t)
resultLR = initialLR / (1 + (k * epoch));
decayParamDisplay = k + " (k)";
} else if (type === 'exponential') {
var k = parseFloat(document.getElementById('decayRate').value);
if (isNaN(k)) { alert("Please enter a valid Decay Rate (k)."); return; }
// Formula: LR = LR0 * exp(-k * t)
resultLR = initialLR * Math.exp(-k * epoch);
decayParamDisplay = k + " (k)";
} else if (type === 'step') {
var dropFactor = parseFloat(document.getElementById('dropFactor').value);
var epochsDrop = parseFloat(document.getElementById('epochsDrop').value);
if (isNaN(dropFactor) || dropFactor 1) {
alert("Please enter a valid Drop Factor (0-1).");
return;
}
if (isNaN(epochsDrop) || epochsDrop <= 0) {
alert("Please enter a valid Epoch Drop interval.");
return;
}
// Formula: LR = LR0 * DropFactor ^ floor((1 + epoch) / EpochsDrop)
// Note: Implementation varies slightly by library, using standard floor(epoch/interval) here
var exponent = Math.floor(epoch / epochsDrop);
resultLR = initialLR * Math.pow(dropFactor, exponent);
decayParamDisplay = dropFactor + " (Drop) / " + epochsDrop + " (Interval)";
}
// Display Results
document.getElementById('resStrategy').innerText = type.charAt(0).toUpperCase() + type.slice(1) + " Decay";
document.getElementById('resInitial').innerText = initialLR;
document.getElementById('resEpoch').innerText = epoch;
document.getElementById('resDecayParam').innerText = decayParamDisplay;
// Format result to meaningful precision (usually 6-8 decimals for small LRs)
// If result is very small, use scientific notation or fix to sufficient decimals
if (resultLR < 0.000001) {
document.getElementById('resFinalLR').innerText = resultLR.toExponential(4);
} else {
document.getElementById('resFinalLR').innerText = resultLR.toFixed(6);
}
document.getElementById('resultBox').style.display = 'block';
}
// Initialize inputs state on load
toggleInputs();