#

Generative Adversarial Networks (GAN)

This is an implementation of Generative Adversarial Networks.

The generator, $G (z z; θ_{g})$ generates samples that match the distribution of data, while the discriminator, $D (x x; θ_{g})$ gives the probability that $x x$ came from data rather than $G$ .

We train $D$ and $G$ simultaneously on a two-player min-max game with value function $V (G, D)$ .

$G min D max V (D, G) = E_{x x \sim p_{d a t a} (x x)} [lo g D (x x)] + E_{z z \sim p_{z z} (z z)} [lo g (1 - D (G (z z))]$

$p_{d a t a} (x x)$ is the probability distribution over data, whilst $p_{z z} (z z)$ probability distribution of $z z$ , which is set to gaussian noise.

This file defines the loss functions. Here is an MNIST example with two multilayer perceptron for the generator and discriminator.

34import torch
35import torch.nn as nn
36import torch.utils.data
37import torch.utils.data

#

Discriminator Loss

Discriminator should ascend on the gradient,

$\nabla_{θ_{d}} \frac{1}{m} i = 1 \sum m [lo g D (x x^{(i)}) + lo g (1 - D (G (z z^{(i)})))]$

$m$ is the mini-batch size and $(i)$ is used to index samples in the mini-batch. $x x$ are samples from $p_{d a t a}$ and $z z$ are samples from $p_{z}$ .

40class DiscriminatorLogitsLoss(nn.Module):

#

55    def __init__(self, smoothing: float = 0.2):
56        super().__init__()

#

We use PyTorch Binary Cross Entropy Loss, which is $- \sum [y lo g (\overset{y}{^}) + (1 - y) lo g (1 - \overset{y}{^})]$ , where $y$ are the labels and $\overset{y}{^}$ are the predictions. Note the negative sign. We use labels equal to $1$ for $x x$ from $p_{d a t a}$ and labels equal to $0$ for $x x$ from $p_{G} .$ Then descending on the sum of these is the same as ascending on the above gradient.

BCEWithLogitsLoss combines softmax and binary cross entropy loss.

67        self.loss_true = nn.BCEWithLogitsLoss()
68        self.loss_false = nn.BCEWithLogitsLoss()

#

We use label smoothing because it seems to work better in some cases

71        self.smoothing = smoothing

#

Labels are registered as buffered and persistence is set to False .

74        self.register_buffer('labels_true', _create_labels(256, 1.0 - smoothing, 1.0), False)
75        self.register_buffer('labels_false', _create_labels(256, 0.0, smoothing), False)

#

logits_true are logits from $D (x x^{(i)})$ and logits_false are logits from $D (G (z z^{(i)}))$

77    def forward(self, logits_true: torch.Tensor, logits_false: torch.Tensor):

#

82        if len(logits_true) > len(self.labels_true):
83            self.register_buffer("labels_true",
84                                 _create_labels(len(logits_true), 1.0 - self.smoothing, 1.0, logits_true.device), False)
85        if len(logits_false) > len(self.labels_false):
86            self.register_buffer("labels_false",
87                                 _create_labels(len(logits_false), 0.0, self.smoothing, logits_false.device), False)
88
89        return (self.loss_true(logits_true, self.labels_true[:len(logits_true)]),
90                self.loss_false(logits_false, self.labels_false[:len(logits_false)]))

#

Generator Loss

Generator should descend on the gradient,

$\nabla_{θ_{g}} \frac{1}{m} i = 1 \sum m [lo g (1 - D (G (z z^{(i)})))]$

93class GeneratorLogitsLoss(nn.Module):

#

104    def __init__(self, smoothing: float = 0.2):
105        super().__init__()
106        self.loss_true = nn.BCEWithLogitsLoss()
107        self.smoothing = smoothing

#

We use labels equal to $1$ for $x x$ from $p_{G} .$ Then descending on this loss is the same as descending on the above gradient.

111        self.register_buffer('fake_labels', _create_labels(256, 1.0 - smoothing, 1.0), False)

#

113    def forward(self, logits: torch.Tensor):
114        if len(logits) > len(self.fake_labels):
115            self.register_buffer("fake_labels",
116                                 _create_labels(len(logits), 1.0 - self.smoothing, 1.0, logits.device), False)
117
118        return self.loss_true(logits, self.fake_labels[:len(logits)])

#

Create smoothed labels

121def _create_labels(n: int, r1: float, r2: float, device: torch.device = None):

#

125    return torch.empty(n, 1, requires_grad=False, device=device).uniform_(r1, r2)