Module Review: Neural Networks

[!NOTE] This module explores the core principles of Module Review: Neural Networks, deriving solutions from first principles and hardware constraints to build world-class, production-ready expertise.

This review section summarizes the core concepts and mechanics of the module.

1. Key Takeaways

  • Perceptron: The basic unit of a neural network. It performs a linear classification by taking a weighted sum of inputs and applying a step function. It fails on non-linear problems like XOR because it can only draw a straight line decision boundary.
  • Activation Functions: Essential for introducing non-linearity, allowing the network to learn complex, real-world patterns.
  • ReLU: Standard for hidden layers. It mitigates the vanishing gradient problem and is computationally efficient (max(0, x)).
  • Sigmoid: Maps values to (0,1). Good for binary probability output, bad for hidden layers due to the vanishing gradient problem (gradients approach zero at the extremes).
  • Softmax: Used for multi-class classification output, converting raw scores into a normalized probability distribution.
  • Universal Approximation Theorem: An MLP with at least one hidden layer can approximate any continuous function, provided it has enough neurons.
  • Forward Propagation: The flow of data from input to output through layers of neurons, involving dot products and activation functions to compute the final prediction.

2. Flashcards

What is the "Vanishing Gradient" problem?
When gradients become extremely small during backpropagation (common with Sigmoid/Tanh), preventing early layers from learning effectively.
Why can't a Perceptron solve XOR?
Because XOR is not linearly separable. A single Perceptron can only draw a straight line decision boundary.
What is the purpose of an Activation Function?
To introduce non-linearity into the network, allowing it to learn complex patterns.
What is the output range of Tanh?
(-1, 1). It is zero-centered, unlike Sigmoid which is (0, 1).
What is "Dead ReLU"?
A state where a ReLU neuron only outputs 0 because its weights have updated such that the input is always negative. It stops learning.

3. Cheat Sheet

Concept Formula / Definition Key Usage
Perceptron Output y = 1 if w·x + b > 0, else 0 Simple binary classification
Sigmoid σ(x) = 1 / (1 + e⁻ˣ) Binary probability output
Tanh tanh(x) Zero-centered hidden layers (Legacy)
ReLU max(0, x) Default for hidden layers
Leaky ReLU max(0.01x, x) Fixes “Dead ReLU”
Softmax e^z / Σ e^z Multi-class probability output
Update Rule w ← w + α(y - ŷ)x Perceptron learning

4. Next Steps

Now that you understand the architecture, it’s time to learn how to train these deep networks using Gradient Descent and Backpropagation.

Ready for practice? Practice in the Vault