Training a vanilla GAN for the first time is humbling. Mode collapse, vanishing gradients, and oscillating losses become familiar enemies very quickly. You spend hours tuning learning rates and architecture choices, only to watch your generator produce identical outputs or your discriminator saturate entirely.
During my undergraduate thesis on GAN-based data augmentation for SSVEP recognition, I hit every one of these walls. The task was synthesizing realistic EEG time-series for low-resource BCI tasks — a problem that sits at the intersection of data scarcity, signal complexity, and the inherent brittleness of adversarial training.
The Problem with Standard GANs
The original GAN framework uses a binary cross-entropy loss where the discriminator learns to distinguish real from fake examples. The problem? When the discriminator becomes too good — which happens quickly with complex signal data — it starts outputting near-zero gradients for generated samples. The generator receives no useful learning signal. Training stalls.
This is the vanishing gradient problem in GANs, and it's especially pronounced when your real and generated distributions are far apart, which is exactly the situation during early training when the generator hasn't learned much yet.
Mode collapse was my biggest headache: the generator would discover one "safe" EEG pattern that fooled the discriminator and reproduce it for every input.
Enter the Wasserstein Distance
The WGAN paper (Arjovsky et al., 2017) reframes GAN training as minimizing the Wasserstein-1 distance (also called Earth Mover's Distance) between the real and generated distributions. This metric measures the minimum "work" required to transform one distribution into another — and crucially, it's well-defined even when the two distributions don't overlap.
The practical consequence is profound: the critic (not discriminator — it's no longer binary) produces meaningful gradient signals throughout training, not just when the distributions are close. Even when your generator is producing garbage, it gets useful feedback about which direction to improve.
The Gradient Penalty
The original WGAN enforced the required 1-Lipschitz constraint via weight clipping — limiting the critic's weights to a small range. This works but introduces its own instabilities, particularly limiting the model's capacity.
WGAN-GP (Gulrajani et al., 2017) replaces weight clipping with a gradient penalty — an additional term in the loss that directly penalizes the norm of the critic's gradients with respect to interpolated samples between real and fake data. This enforces Lipschitz continuity in a much more principled way, allowing the critic to use its full capacity.
# Gradient penalty (conceptual PyTorch)
eps = torch.rand(batch_size, 1, 1).to(device)
interpolated = eps * real + (1 - eps) * fake
interp_out = critic(interpolated)
gradients = torch.autograd.grad(
outputs=interp_out, inputs=interpolated,
grad_outputs=torch.ones_like(interp_out),
create_graph=True, retain_graph=True
)[0]
gp = ((gradients.norm(2, dim=1) - 1) ** 2).mean()
critic_loss = fake_score - real_score + lambda_gp * gpWhat Changed in My Thesis
Switching to WGAN-GP had an immediate and dramatic effect on training stability. The mode collapse I had been battling largely disappeared. The loss curves became interpretable — the Wasserstein distance gives you a meaningful number that correlates with generation quality, unlike the adversarial losses in vanilla GANs.
Combined with spectral normalization on the generator layers and careful noise modeling to match EEG acquisition artifacts, the WGAN-GP produced synthetic EEG signals that improved downstream SSVEP classification performance — the entire goal of the augmentation pipeline.
The Deeper Lesson
What WGAN-GP taught me is that the choice of loss function is not a minor implementation detail — it encodes fundamental assumptions about how you measure the distance between probability distributions. Understanding why a loss works, mathematically, is the difference between debugging blindly and making principled architectural decisions.
If you're working with GANs on scientific signal data — EEG, ECG, time-series of any kind — start with WGAN-GP. The stability improvements are not marginal. They're the difference between a training run that converges and one that doesn't.
This post is adapted from reflections during my undergraduate thesis at the HCI Lab, University of Rajshahi. The thesis focused on SSVEP-based BCI data augmentation using conditional generative models.