Gradient Descent

• First-order iterative optimization algorithm for finding a local minimum of a differentiable function

Important Concepts in Optimization

Generalization

• How well the learned model will behave on unseen data

Cross-validation

• Cross-validation is a model validation technique fro assessing how the model will generalize to an independent(test) data set

Bias and Variance

• Bias and Variance trade off →

Bootstrapping

• Bootstrapping is any test or metric that uses random sampling with replacement

Bagging vs Boosting

Gradient Descent Methods

• Stochastic gradient descent
• Mini-batch gradient descent
• Batch gradient descent