Batch Normalization in PyTorch: Understanding Parameters

作者：demo2024.01.08 01:38浏览量：3

简介：Batch Normalization is a technique used in deep learning to improve the training of neural networks. In this article, we will explore the parameters of Batch Normalization in PyTorch and how they are used.

Batch Normalization (BN) is a technique used in deep learning to improve the training of neural networks. It normalizes the activations within each batch during training and applies scaling and shifting to ensure that the distribution of activations remains consistent across different batches. In PyTorch, BN is implemented in the torch.nn.BatchNorm module.
The torch.nn.BatchNorm module has several parameters that control the behavior of BN. Here’s a breakdown of the parameters:

  1. num_features: This parameter specifies the number of features in the input tensor. It must be provided when initializing the BatchNorm module.
  2. eps: This is a small constant added to the denominator in the scaling and shifting operation to avoid division by zero. It is typically set to a value like 1e-5.
  3. momentum: This parameter controls the exponential moving average used for estimation of the mean and variance during training. It is also called the momentum term in BN. It is typically set to 0.1.
  4. affine: If set to True, the BN module will also learn a set of affine parameters (gamma and beta), which are scale and shift factors applied to the normalized activations. These parameters allow for learned scaling and shifting of the normalized activations, enabling or disabling affine transformation.
  5. track_running_stats: This parameter controls whether to track the running mean and variance statistics during training. If set to True, these statistics are updated during forward passes. If set to False, the running statistics are not updated, and the module only relies on the batch statistics for normalization.
    In PyTorch, BN can be applied to any input tensor, typically following a linear layer or convolutional layer. The input tensor undergoes BN by passing it through the BatchNorm module, and then the normalized activations are passed through an activation function such as ReLU or Tanh.
    Here’s an example of how to apply BN in PyTorch:
    1. import torch
    2. import torch.nn as nn
    3. # Define a simple neural network with BN
    4. class MyNet(nn.Module):
    5. def __init__(self):
    6. super(MyNet, self).__init__()
    7. self.conv1 = nn.Conv2d(3, 64, kernel_size=3, stride=1, padding=1)
    8. self.bn1 = nn.BatchNorm2d(64)
    9. self.conv2 = nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=1)
    10. self.bn2 = nn.BatchNorm2d(128)
    11. self.fc1 = nn.Linear(128 * 7 * 7, 256)
    12. self.fc2 = nn.Linear(256, 10)
    13. def forward(self, x):
    14. x = F.relu(self.bn1(self.conv1(x)))
    15. x = F.relu(self.bn2(self.conv2(x)))
    16. x = x.view(-1, 128 * 7 * 7)
    17. x = F.relu(self.fc1(x))
    18. x = self.fc2(x)
    19. return x
    In this example, we define a simple neural network with two convolutional layers followed by two BN layers. The BN layers are applied to the output of each convolutional layer using bn1 and bn2. The input tensor x is first passed through the convolutional layers and then through the BN layers using element-wise addition (+) and activation function ReLU (F.relu). The BN-normalized activations are then flattened using view() before being passed through the fully connected layers (fc1 and fc2).
    BN has been found to significantly improve the training of neural networks, especially in deep architectures where internal covariate shift is a common issue.

最热文章