Dive into Deep Learning Lec7: Regularization in PyTorch from Scratch (Custom Loss Function Autograd)

This video discusses the implementation of a custom loss function in PyTorch and using to compute the gradient of the loss function with respect to the model parameters. This approach has many applications, including regularizing deep learning models and neural networks using l1 and l2 norms. This video discusses the basics of regularization, such as adding penalty terms and regularization parameters that control the tradeoff between model fitting and complexity. We present a simple example using synthetic data to generate training and test data points. We then define a neural network model, initialize its weights, and define various loss functions. We also discuss how to implement Stochastic Gradient Descent (SGD). Here is the code summary: import numpy as np import torch from torch import nn from import data import as plt # Implementation from Scratch def init_params(): “““Parameter initialization“““ w = (0, 1, size=(num_inputs, 1), requires_grad=True) b = (1, requires_grad=True) return [w, b] def l2_penalty(w): “““Defining the penalty term “““ return ((2)) / 2 def l1_penalty(w): “““Defining the penalty term “““ return ((w)) def forward(X, w, b): “““Linear regression model“““ return (X, w) b def squared_loss(y_hat, y): “““Squared loss“““ return (y_hat - ()) ** 2 / 2 def sgd(params, lr, batch_size): “““Minibatch stochastic gradient descent“““ with (): for param in params: param -= lr * / batch_size () def evaluate_loss(net, data_iter, loss): “““Evaluate the loss of a model on the given dataset“““ l = 0 for X, y in data_iter: out = net(X) y = () w, b = init_params() num_epochs, lr, alpha, beta = 200, .001, 5, 1 net = lambda X: forward(X, w, b) train_loss = (num_epochs) test_loss = (num_epochs) for epoch in range(num_epochs): for X, y in train_iter: l = squared_loss(net(X), y) alpha * l2_penalty(w) beta * l1_penalty(w) ().backward() sgd([w, b], lr, batch_size) with (): train_loss[epoch] = evaluate_loss(net, train_iter, squared_loss) test_loss[epoch] = evaluate_loss(net, test_iter, squared_loss) ({'': 14}) ((num_epochs), train_loss, label='train') ((num_epochs), test_loss, label='test') () ('log') l = loss(out, y).sum() return l #python #regularization #machinelearning