Module 09: Neural Networks Basics
Introduction to neural networks. The foundation of deep learning.
What You'll Learn
- Perceptron and Multi-Layer Perceptron (MLP)
- Activation Functions
- Backpropagation
- Gradient Descent Variants
- Building Neural Networks from Scratch
Deep learning curriculum map
| Topic | Where to study |
|---|---|
| Fundamentals, ML vs DL, linear limits, MLP, activations, losses | Neural networks guide |
| Backpropagation and optimizer math (Momentum, AdaGrad, RMSProp, Adam) | Backpropagation, Gradient descent |
| Training challenges; regularization; weight init | Training challenges, Regularization, Weight init |
| PyTorch / Keras pipelines and projects | Frameworks module |
Topics Covered
1. Perceptron
- Single Perceptron: Basic building block
- Linear Classification: Can only solve linearly separable problems
- Limitations: XOR problem
2. Multi-Layer Perceptron (MLP)
- Hidden Layers: Enable non-linear learning
- Architecture: Input → Hidden → Output
- Universal Approximation: Can approximate any function
3. Activation Functions
- Sigmoid: S-shaped curve (0 to 1)
- Tanh: Hyperbolic tangent (-1 to 1)
- ReLU: Rectified Linear Unit (most common)
- Softmax: For multi-class classification
- When to use each: Different use cases
4. Backpropagation
- Forward Pass: Compute predictions
- Backward Pass: Compute gradients
- Chain Rule: How gradients flow backward
- Understanding: Key to training neural networks
5. Gradient Descent
- Batch Gradient Descent: Use all data
- Stochastic Gradient Descent (SGD): One sample at a time
- Mini-batch Gradient Descent: Small batches (most common)
- Optimizers: Adam, RMSprop, AdaGrad
6. Loss Functions
- Mean Squared Error (MSE): For regression
- Cross-Entropy: For classification
- Binary Cross-Entropy: For binary classification
Learning Objectives
By the end of this module, you should be able to:
- Understand how neural networks work
- Build a neural network from scratch
- Implement backpropagation
- Choose appropriate activation functions
- Train neural networks effectively
Projects
- Build Perceptron from Scratch: Implement basic perceptron
- Build MLP from Scratch: Implement multi-layer network
- MNIST Classification: Classify handwritten digits with MLP
- Compare Optimizers: Test different optimization algorithms
Key Concepts
- Weights and Biases: Parameters learned during training
- Forward Propagation: Computing predictions
- Backward Propagation: Computing gradients
- Learning Rate: How fast to learn (critical hyperparameter)
- Epochs: Full pass through training data
- Batch Size: Number of samples per update
Documentation & Learning Resources
Official Documentation:
Free Courses:
- Neural Networks and Deep Learning (Coursera): Free audit available
- Deep Learning Specialization (Coursera): Free audit available
- Fast.ai Practical Deep Learning: Completely free course
Free Books:
- Neural Networks and Deep Learning (Free Book): Interactive online book
- Deep Learning Book (Free PDF): Comprehensive deep learning textbook
Video Tutorials:
- 3Blue1Brown - Neural Networks: Best visual explanations
- Neural Networks (StatQuest)
- Backpropagation Explained
Tutorials:
- Building Neural Networks (TensorFlow)
- Neural Networks Tutorial (PyTorch)
- Understanding Neural Networks
Additional Resources:
- Advanced Topics →: Advanced architectures, optimization, regularization
- Project Tutorial →: Step-by-step neural network from scratch
- Quick Reference →: Quick lookup guide for neural networks
Previous Module: 08-unsupervised-learning
Next Module: 10-deep-learning-frameworks