Generative Adversarial Networks (GAN)
1. f-divergence
Definition
Let be a convex function with . We define:
This divergence measures the difference between two distributions and .
Examples
When , we have the KL divergence:
When , we have the total variation distance:
When , we have the Jensen-Shannon (JS) divergence:
2. GAN
From JS divergence to a discriminator
With an optimal discriminator, the original GAN minimax objective is equivalent to minimizing the JS divergence between the model and real data distributions.
During training, our goal is to minimize , but it is difficult to evaluate and directly.
The ratio lies between 0 and 1 and, with equal class priors, represents the probability that is real. We can train a binary classifier to estimate it.
Adversarial training
We can reformulate the objective as a minimax problem:
Finally, we train both , the discriminator, and , the generator. While training , we maximize ; while training , we minimize it, keeping the other network fixed. This is known as adversarial training.
Counterfeiters and police
An analogy from the original GAN paper helps explain this process. Counterfeiters try to make fake money look real, while the police try to distinguish fake money from real money. The generator plays the role of the counterfeiters, and the discriminator plays the role of the police. The goal is to produce counterfeits that are indistinguishable from real money.
Training difficulties
GAN training is generally nonconvex–nonconcave and sensitive to the balance between the two networks. Mode collapse occurs when the generator produces samples from only a few modes of the data distribution.
3. Wasserstein distance
Intuition: moving probability mass
The Wasserstein-1 distance is the minimum cost of moving probability mass to transform one distribution into another. Moving mass from to costs :
Here, is the set of transport plans with marginals and .
A form we can optimize
For distributions with finite first moments, Kantorovich–Rubinstein duality gives
where is the set of all 1-Lipschitz functions: .
4. WGAN
Training idea
WGAN follows the same adversarial idea as GAN, using a critic to estimate Wasserstein distance. The critic outputs a real-valued score without a sigmoid:
Alternate between maximizing over with the generator fixed and minimizing over with the critic fixed. The critic should satisfy the 1-Lipschitz constraint; two practical approaches are given below.
Weight clipping
The original WGAN clips each critic parameter after an update:
With Lipschitz activations, this bounds the critic's Lipschitz constant, though not necessarily by 1. Clipping can restrict the critic's capacity.
Gradient penalty: WGAN-GP
WGAN-GP replaces clipping with a penalty on the critic's input gradient at points between real and generated samples:
With , the critic minimizes
This softly encourages gradient norms near 1 at sampled points. The penalty is used only for critic updates.


