lasagne.updates

Functions to generate Theano update dictionaries for training.

The update functions implement different methods to control the learning rate for use with stochastic gradient descent.

Update functions take a loss expression or a list of gradient expressions and a list of parameters as input and return an ordered dictionary of updates:

sgd

Stochastic Gradient Descent (SGD) updates

momentum

Stochastic Gradient Descent (SGD) updates with momentum

nesterov_momentum

Stochastic Gradient Descent (SGD) updates with Nesterov momentum

adagrad

Adagrad updates

rmsprop

RMSProp updates

adadelta

Adadelta updates

adam

Adam updates

adamax

Adamax updates

amsgrad

AMSGrad updates

Two functions can be used to further modify the updates to include momentum:

apply_momentum

Returns a modified update dictionary including momentum

apply_nesterov_momentum

Returns a modified update dictionary including Nesterov momentum

Finally, we provide two helper functions to constrain the norm of tensors:

norm_constraint

Max weight norm constraints and gradient clipping

total_norm_constraint

Rescales a list of tensors based on their combined norm

norm_constraint() can be used to constrain the norm of parameters (as an alternative to weight decay), or for a form of gradient clipping. total_norm_constraint() constrain the total norm of a list of tensors. This is often used when training recurrent neural networks.

Examples

Using nesterov_momentum() to define an update dictionary for a toy example network:

>>> import lasagne
>>> import theano.tensor as T
>>> import theano
>>> from lasagne.nonlinearities import softmax
>>> from lasagne.layers import InputLayer, DenseLayer, get_output
>>> from lasagne.updates import nesterov_momentum
>>> l_in = InputLayer((100, 20))
>>> l1 = DenseLayer(l_in, num_units=3, nonlinearity=softmax)
>>> x = T.matrix('x')  # shp: num_batch x num_features
>>> y = T.ivector('y') # shp: num_batch
>>> l_out = get_output(l1, x)
>>> params = lasagne.layers.get_all_params(l1)
>>> loss = T.mean(T.nnet.categorical_crossentropy(l_out, y))
>>> updates = nesterov_momentum(loss, params, learning_rate=1e-4, momentum=.9)
>>> train_fn = theano.function([x, y], updates=updates)

With apply_momentum() and apply_nesterov_momentum(), we can add momentum to optimization schemes that do not usually support this:

>>> updates = lasagne.updates.rmsprop(loss, params, learning_rate=0.0001)
>>> updates = lasagne.updates.apply_momentum(updates, params, momentum=0.9)

All optimization schemes support symbolic variables for their hyperparameters, such as shared variables. This allows to vary hyperparameters during training without recompiling the training function. Note that the dtypes must match the dtypes of the network parameters, which follow Theano’s floatX setting. In the following example, we use lasagne.utils.floatX() to ensure this:

>>> eta = theano.shared(lasagne.utils.floatX(0.001))
>>> updates = lasagne.updates.adam(loss, params, learning_rate=eta)
>>> train_fn = theano.function([x, y], updates=updates)
>>> # we can now modify the learning rate at any time during training:
>>> eta.set_value(lasagne.utils.floatX(eta.get_value() * 0.1))

Update functions

lasagne.updates.sgd(loss_or_grads, params, learning_rate)[source]

Stochastic Gradient Descent (SGD) updates

Generates update expressions of the form:

  • param := param - learning_rate * gradient

Parameters

loss_or_grads : symbolic expression or list of expressions

A scalar loss expression, or a list of gradient expressions

params : list of shared variables

The variables to generate update expressions for

learning_rate : float or symbolic scalar

The learning rate controlling the size of update steps

Returns

OrderedDict

A dictionary mapping each parameter to its update expression

lasagne.updates.momentum(loss_or_grads, params, learning_rate, momentum=0.9)[source]

Stochastic Gradient Descent (SGD) updates with momentum

Generates update expressions of the form:

  • velocity := momentum * velocity - learning_rate * gradient

  • param := param + velocity

Parameters

loss_or_grads : symbolic expression or list of expressions

A scalar loss expression, or a list of gradient expressions

params : list of shared variables

The variables to generate update expressions for

learning_rate : float or symbolic scalar

The learning rate controlling the size of update steps

momentum : float or symbolic scalar, optional

The amount of momentum to apply. Higher momentum results in smoothing over more update steps. Defaults to 0.9.

Returns

OrderedDict

A dictionary mapping each parameter to its update expression

See also

apply_momentum

Generic function applying momentum to updates

nesterov_momentum

Nesterov’s variant of SGD with momentum

Notes

Higher momentum also results in larger update steps. To counter that, you can optionally scale your learning rate by 1 - momentum.

lasagne.updates.nesterov_momentum(loss_or_grads, params, learning_rate, momentum=0.9)[source]

Stochastic Gradient Descent (SGD) updates with Nesterov momentum

Generates update expressions of the form:

  • velocity := momentum * velocity - learning_rate * gradient

  • param := param + momentum * velocity - learning_rate * gradient

Parameters

loss_or_grads : symbolic expression or list of expressions

A scalar loss expression, or a list of gradient expressions

params : list of shared variables

The variables to generate update expressions for

learning_rate : float or symbolic scalar

The learning rate controlling the size of update steps

momentum : float or symbolic scalar, optional

The amount of momentum to apply. Higher momentum results in smoothing over more update steps. Defaults to 0.9.

Returns

OrderedDict

A dictionary mapping each parameter to its update expression

See also

apply_nesterov_momentum

Function applying momentum to updates

Notes

Higher momentum also results in larger update steps. To counter that, you can optionally scale your learning rate by 1 - momentum.

The classic formulation of Nesterov momentum (or Nesterov accelerated gradient) requires the gradient to be evaluated at the predicted next position in parameter space. Here, we use the formulation described at https://github.com/lisa-lab/pylearn2/pull/136#issuecomment-10381617, which allows the gradient to be evaluated at the current parameters.

lasagne.updates.adagrad(loss_or_grads, params, learning_rate=1.0, epsilon=1e-06)[source]

Adagrad updates

Scale learning rates by dividing with the square root of accumulated squared gradients. See [R97] for further description.

Parameters

loss_or_grads : symbolic expression or list of expressions

A scalar loss expression, or a list of gradient expressions

params : list of shared variables

The variables to generate update expressions for

learning_rate : float or symbolic scalar

The learning rate controlling the size of update steps

epsilon : float or symbolic scalar

Small value added for numerical stability

Returns

OrderedDict

A dictionary mapping each parameter to its update expression

Notes

Using step size eta Adagrad calculates the learning rate for feature i at time step t as:

\[\eta_{t,i} = \frac{\eta} {\sqrt{\sum^t_{t^\prime} g^2_{t^\prime,i}+\epsilon}} g_{t,i}\]

as such the learning rate is monotonically decreasing.

Epsilon is not included in the typical formula, see [R98].

References

R97(1,2)

Duchi, J., Hazan, E., & Singer, Y. (2011): Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 12:2121-2159.

R98(1,2)

Chris Dyer: Notes on AdaGrad. http://www.ark.cs.cmu.edu/cdyer/adagrad.pdf

lasagne.updates.rmsprop(loss_or_grads, params, learning_rate=1.0, rho=0.9, epsilon=1e-06)[source]

RMSProp updates

Scale learning rates by dividing with the moving average of the root mean squared (RMS) gradients. See [R99] for further description.

Parameters

loss_or_grads : symbolic expression or list of expressions

A scalar loss expression, or a list of gradient expressions

params : list of shared variables

The variables to generate update expressions for

learning_rate : float or symbolic scalar

The learning rate controlling the size of update steps

rho : float or symbolic scalar

Gradient moving average decay factor

epsilon : float or symbolic scalar

Small value added for numerical stability

Returns

OrderedDict

A dictionary mapping each parameter to its update expression

Notes

rho should be between 0 and 1. A value of rho close to 1 will decay the moving average slowly and a value close to 0 will decay the moving average fast.

Using the step size \(\eta\) and a decay factor \(\rho\) the learning rate \(\eta_t\) is calculated as:

\[\begin{split}r_t &= \rho r_{t-1} + (1-\rho)*g^2\\ \eta_t &= \frac{\eta}{\sqrt{r_t + \epsilon}}\end{split}\]

References

R99(1,2)

Tieleman, T. and Hinton, G. (2012): Neural Networks for Machine Learning, Lecture 6.5 - rmsprop. Coursera. http://www.youtube.com/watch?v=O3sxAc4hxZU (formula @5:20)

lasagne.updates.adadelta(loss_or_grads, params, learning_rate=1.0, rho=0.95, epsilon=1e-06)[source]

Adadelta updates

Scale learning rates by the ratio of accumulated gradients to accumulated updates, see [R100] and notes for further description.

Parameters

loss_or_grads : symbolic expression or list of expressions

A scalar loss expression, or a list of gradient expressions

params : list of shared variables

The variables to generate update expressions for

learning_rate : float or symbolic scalar

The learning rate controlling the size of update steps

rho : float or symbolic scalar

Squared gradient moving average decay factor

epsilon : float or symbolic scalar

Small value added for numerical stability

Returns

OrderedDict

A dictionary mapping each parameter to its update expression

Notes

rho should be between 0 and 1. A value of rho close to 1 will decay the moving average slowly and a value close to 0 will decay the moving average fast.

rho = 0.95 and epsilon=1e-6 are suggested in the paper and reported to work for multiple datasets (MNIST, speech).

In the paper, no learning rate is considered (so learning_rate=1.0). Probably best to keep it at this value. epsilon is important for the very first update (so the numerator does not become 0).

Using the step size eta and a decay factor rho the learning rate is calculated as:

\[\begin{split}r_t &= \rho r_{t-1} + (1-\rho)*g^2\\ \eta_t &= \eta \frac{\sqrt{s_{t-1} + \epsilon}} {\sqrt{r_t + \epsilon}}\\ s_t &= \rho s_{t-1} + (1-\rho)*(\eta_t*g)^2\end{split}\]

References

R100(1,2)

Zeiler, M. D. (2012): ADADELTA: An Adaptive Learning Rate Method. arXiv Preprint arXiv:1212.5701.

lasagne.updates.adam(loss_or_grads, params, learning_rate=0.001, beta1=0.9, beta2=0.999, epsilon=1e-08)[source]

Adam updates

Adam updates implemented as in [R101].

Parameters

loss_or_grads : symbolic expression or list of expressions

A scalar loss expression, or a list of gradient expressions

params : list of shared variables

The variables to generate update expressions for

learning_rate : float or symbolic scalar

Learning rate

beta1 : float or symbolic scalar

Exponential decay rate for the first moment estimates.

beta2 : float or symbolic scalar

Exponential decay rate for the second moment estimates.

epsilon : float or symbolic scalar

Constant for numerical stability.

Returns

OrderedDict

A dictionary mapping each parameter to its update expression

Notes

The paper [R101] includes an additional hyperparameter lambda. This is only needed to prove convergence of the algorithm and has no practical use (personal communication with the authors), it is therefore omitted here.

References

R101(1,2,3)

Kingma, Diederik, and Jimmy Ba (2014): Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980.

lasagne.updates.adamax(loss_or_grads, params, learning_rate=0.002, beta1=0.9, beta2=0.999, epsilon=1e-08)[source]

Adamax updates

Adamax updates implemented as in [R102]. This is a variant of of the Adam algorithm based on the infinity norm.

Parameters

loss_or_grads : symbolic expression or list of expressions

A scalar loss expression, or a list of gradient expressions

params : list of shared variables

The variables to generate update expressions for

learning_rate : float or symbolic scalar

Learning rate

beta1 : float or symbolic scalar

Exponential decay rate for the first moment estimates.

beta2 : float or symbolic scalar

Exponential decay rate for the weighted infinity norm estimates.

epsilon : float or symbolic scalar

Constant for numerical stability.

Returns

OrderedDict

A dictionary mapping each parameter to its update expression

References

R102(1,2)

Kingma, Diederik, and Jimmy Ba (2014): Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980.

lasagne.updates.amsgrad(loss_or_grads, params, learning_rate=0.001, beta1=0.9, beta2=0.999, epsilon=1e-08)[source]

AMSGrad updates

AMSGrad updates implemented as in [R103].

Parameters

loss_or_grads : symbolic expression or list of expressions

A scalar loss expression, or a list of gradient expressions

params : list of shared variables

The variables to generate update expressions for

learning_rate : float or symbolic scalar

Learning rate

beta1 : float or symbolic scalar

Exponential decay rate for the first moment estimates.

beta2 : float or symbolic scalar

Exponential decay rate for the second moment estimates.

epsilon : float or symbolic scalar

Constant for numerical stability.

Returns

OrderedDict

A dictionary mapping each parameter to its update expression

References

R103(1,2)

https://openreview.net/forum?id=ryQu7f-RZ

Update modification functions

lasagne.updates.apply_momentum(updates, params=None, momentum=0.9)[source]

Returns a modified update dictionary including momentum

Generates update expressions of the form:

  • velocity := momentum * velocity + updates[param] - param

  • param := param + velocity

Parameters

updates : OrderedDict

A dictionary mapping parameters to update expressions

params : iterable of shared variables, optional

The variables to apply momentum to. If omitted, will apply momentum to all updates.keys().

momentum : float or symbolic scalar, optional

The amount of momentum to apply. Higher momentum results in smoothing over more update steps. Defaults to 0.9.

Returns

OrderedDict

A copy of updates with momentum updates for all params.

See also

momentum

Shortcut applying momentum to SGD updates

Notes

Higher momentum also results in larger update steps. To counter that, you can optionally scale your learning rate by 1 - momentum.

lasagne.updates.apply_nesterov_momentum(updates, params=None, momentum=0.9)[source]

Returns a modified update dictionary including Nesterov momentum

Generates update expressions of the form:

  • velocity := momentum * velocity + updates[param] - param

  • param := param + momentum * velocity + updates[param] - param

Parameters

updates : OrderedDict

A dictionary mapping parameters to update expressions

params : iterable of shared variables, optional

The variables to apply momentum to. If omitted, will apply momentum to all updates.keys().

momentum : float or symbolic scalar, optional

The amount of momentum to apply. Higher momentum results in smoothing over more update steps. Defaults to 0.9.

Returns

OrderedDict

A copy of updates with momentum updates for all params.

See also

nesterov_momentum

Shortcut applying Nesterov momentum to SGD updates

Notes

Higher momentum also results in larger update steps. To counter that, you can optionally scale your learning rate by 1 - momentum.

The classic formulation of Nesterov momentum (or Nesterov accelerated gradient) requires the gradient to be evaluated at the predicted next position in parameter space. Here, we use the formulation described at https://github.com/lisa-lab/pylearn2/pull/136#issuecomment-10381617, which allows the gradient to be evaluated at the current parameters.

Helper functions

lasagne.updates.norm_constraint(tensor_var, max_norm, norm_axes=None, epsilon=1e-07)[source]

Max weight norm constraints and gradient clipping

This takes a TensorVariable and rescales it so that incoming weight norms are below a specified constraint value. Vectors violating the constraint are rescaled so that they are within the allowed range.

Parameters

tensor_var : TensorVariable

Theano expression for update, gradient, or other quantity.

max_norm : scalar

This value sets the maximum allowed value of any norm in tensor_var.

norm_axes : sequence (list or tuple)

The axes over which to compute the norm. This overrides the default norm axes defined for the number of dimensions in tensor_var. When this is not specified and tensor_var is a matrix (2D), this is set to (0,). If tensor_var is a 3D, 4D or 5D tensor, it is set to a tuple listing all axes but axis 0. The former default is useful for working with dense layers, the latter is useful for 1D, 2D and 3D convolutional layers. (Optional)

epsilon : scalar, optional

Value used to prevent numerical instability when dividing by very small or zero norms.

Returns

TensorVariable

Input tensor_var with rescaling applied to weight vectors that violate the specified constraints.

Notes

When norm_axes is not specified, the axes over which the norm is computed depend on the dimensionality of the input variable. If it is 2D, it is assumed to come from a dense layer, and the norm is computed over axis 0. If it is 3D, 4D or 5D, it is assumed to come from a convolutional layer and the norm is computed over all trailing axes beyond axis 0. For other uses, you should explicitly specify the axes over which to compute the norm using norm_axes.

Examples

>>> param = theano.shared(
...     np.random.randn(100, 200).astype(theano.config.floatX))
>>> update = param + 100
>>> update = norm_constraint(update, 10)
>>> func = theano.function([], [], updates=[(param, update)])
>>> # Apply constrained update
>>> _ = func()
>>> from lasagne.utils import compute_norms
>>> norms = compute_norms(param.get_value())
>>> np.isclose(np.max(norms), 10)
True
lasagne.updates.total_norm_constraint(tensor_vars, max_norm, epsilon=1e-07, return_norm=False)[source]

Rescales a list of tensors based on their combined norm

If the combined norm of the input tensors exceeds the threshold then all tensors are rescaled such that the combined norm is equal to the threshold.

Scaling the norms of the gradients is often used when training recurrent neural networks [R104].

Parameters

tensor_vars : List of TensorVariables.

Tensors to be rescaled.

max_norm : float

Threshold value for total norm.

epsilon : scalar, optional

Value used to prevent numerical instability when dividing by very small or zero norms.

return_norm : bool

If true the total norm is also returned.

Returns

tensor_vars_scaled : list of TensorVariables

The scaled tensor variables.

norm : Theano scalar

The combined norms of the input variables prior to rescaling, only returned if return_norms=True.

Notes

The total norm can be used to monitor training.

References

R104(1,2)

Sutskever, I., Vinyals, O., & Le, Q. V. (2014): Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems (pp. 3104-3112).

Examples

>>> from lasagne.layers import InputLayer, DenseLayer
>>> import lasagne
>>> from lasagne.updates import sgd, total_norm_constraint
>>> x = T.matrix()
>>> y = T.ivector()
>>> l_in = InputLayer((5, 10))
>>> l1 = DenseLayer(l_in, num_units=7, nonlinearity=T.nnet.softmax)
>>> output = lasagne.layers.get_output(l1, x)
>>> cost = T.mean(T.nnet.categorical_crossentropy(output, y))
>>> all_params = lasagne.layers.get_all_params(l1)
>>> all_grads = T.grad(cost, all_params)
>>> scaled_grads = total_norm_constraint(all_grads, 5)
>>> updates = sgd(scaled_grads, all_params, learning_rate=0.1)