Solving the Puzzle: Systems of Linear Equations

1. Introduction: The Core Problem

The central problem of Linear Algebra is solving the equation: Ax = b

Where:

  • A is a Matrix (The System / Coefficients).
  • x is a Vector (The Unknowns).
  • b is a Vector (The Result).

This is just a compact way of writing a system of equations:

2x + y = 5
x - y = 1

In Machine Learning, we often solve Ax = b to find the optimal weights (x) that map inputs (A) to targets (b).

2. Geometric Interpretation

In 2D, each linear equation represents a Line. The solution to the system is the point where the lines intersect.

Three Possibilities

  1. Unique Solution: The lines cross at exactly one point. (Most common).
  2. No Solution: The lines are Parallel and never cross. (Inconsistent).
  3. Infinite Solutions: The lines are identical (overlapping).

3. Gaussian Elimination

How does a computer solve this? It uses an algorithm called Gaussian Elimination. The goal is to transform the matrix A into an Upper Triangular Matrix (Row Echelon Form).

Operations Allowed:

  1. Swap two rows.
  2. Multiply a row by a non-zero scalar.
  3. Add a multiple of one row to another.

Example Walkthrough:

Step Augmented Matrix Action
1. Start [[ 2, 1 | 5 ]
[ 1, -1 | 1 ]] 		Initial System
2. Swap [[ 1, -1 | 1 ]
[ 2, 1 | 5 ]] 		Swap R1 ↔ R2 to get 1 in top-left
3. Eliminate [[ 1, -1 | 1 ]
[ 0, 3 | 3 ]] 		R2 = R2 - 2*R1

Back Substitution:

  1. From Row 2: 3y = 3 → y = 1.
  2. Substitute into Row 1: x - 1 = 1 → x = 2.
  3. Solution: (2, 1).

4. Interactive Visualizer: Intersection Simulator

Adjust the Slopes (m) and Intercepts (b) of the two lines to see where they intersect. y = m1x + b1 y = m2x + b2

Line 1 (Blue)

Line 2 (Green)

Solution: x = 1.2, y = 3.4

5. Summary

  • Intersection: The solution to a system is where the shapes intersect.
  • Parallel: No intersection means no solution.
  • Gaussian Elimination: The standard algorithm to solve linear systems systematically.

Next: ML Application (Embeddings) →