1. Activations

1-1. Properties

• Differentiable → For backpropagation
• Non-linear → In order to compute complex features

<aside> 🧑‍🏫 Activation functions have to be non-linear and differentiable

</aside>

1-2. Common Activation Functions

• ReLU
  • $max(0,z^{[l]})$
  • Dying ReLU problem
• Leaky ReLU
  • $max(az^{[l]},z^{[l]})$
• Sigmoid
  • $\frac{1}{1+\exp^{-z^{[l]}}}$
  • Vanishing Gradient, Saturation problems
• Hyperbolic Tangent
  • $tanh(x^{[l]})$
  • Vanishing Gradient, Saturation problems

2. Batch Normalization

2-1. Basics

• GANs are very hard to train so every trick that speeds up and stabilizes training is crucial

• Different Distributions

  • Features $x_1$ and $x_2$ might have different distributions → Elongates the cost function making it hard to reach the global minima
  • If a new training or test data with different distribution comes in, the cost function will change too making it difficult to train, inference correctly → coratiate shift

• Normalization and its effects

  • Having similar distributions across different features makes the cost function smoother and easier to train
  • By making new data distribution same as the distribution of the data we used to train, it will reduce the covariate shift effect

2-1. Procedure

• Training

  $$ \hat{z}^{[l]}_i = \frac{z^{[l]}i-\mu{z^{[l]}i}}{\sqrt{\sigma^2{z^{[l]}_i}+\epsilon}} $$

  • $\mu_{z^{[l]}_i}$ : Batch mean of $z^{[l]}_i$
  • ${\sqrt{\sigma^2_{z^{[l]}_i}+\epsilon}}$ : Batch standard deviation of $z^{[l]}_i$

  $$ y^{[l]}_i = \gamma\hat{z}^{[l]}_i + \beta $$

  • $\gamma$ : scale factor
  • $\beta$ : shift factor

  → Learnable parameters to get the optimal dist

  <aside> 🧑‍🏫 Batch normalization gives you control over what that distribution will look like moving forward in the neural network

  </aside>