What is the principle of pytorch gradient clipping

This article mainly explains "what is the principle of pytorch gradient cutting". Interested friends may wish to have a look at it. The method introduced in this paper is simple, fast and practical. Next, let the editor take you to learn "what is the principle of pytorch gradient cutting"?

Since the gradient disappears / explodes in the BP process (that is, the partial derivative is infinitely close to 0, so that the long-term memory cannot be updated), the simplest and roughest way is to set a threshold. When the gradient is less than / greater than the threshold, the updated gradient is the threshold, as shown in the following figure:

1. The principle of gradient cutting

Advantages: simple and rough

Cons: it is difficult to find a satisfactory threshold

2. Nn.utils.clip_grad_norm (parameters, max_norm, norm_type=2)

This function is measured according to the norm of the parameter

Parameters:

Parameters (Iterable [Variable])-A variable-based iterator that is normalized (original: an iterable of Variables that will have gradients normalized)

Max_norm (float or int)-the maximum norm of gradient

Norm_type (float or int)-specifies the type of norm, default to L2

Returns: the overall norm of the parameter (as a single vector)

At this point, I believe you have a deeper understanding of "what is the principle of pytorch gradient cutting". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!