Lecture 3 — Singular Value Decomposition (SVD)

Theory • 2×2 worked example • ML applications • Interactive calculator

1. What is SVD?

Singular Value Decomposition (SVD) factorizes any real m×n matrix A into three matrices:

A = U Σ V^T

U is an m×m orthogonal matrix (columns are left singular vectors).
Σ is an m×n diagonal matrix with non-negative singular values (σ₁ ≥ σ₂ ≥ ... ≥ 0).
V is an n×n orthogonal matrix (columns are right singular vectors).

Intuitively, SVD expresses A as a composition of: rotate (V^T), scale (Σ), rotate (U). It works for any rectangular matrix and is numerically stable — unlike EVD which requires square matrices.

2. Why SVD matters in ML

Dimensionality reduction: Truncate small singular values to get best low-rank approximation (foundation of PCA).
Noise filtering: Small singular values often encode noise — removing them denoises data.
Latent factor models: Matrix factorization (recommendation systems) uses low-rank SVD-like decompositions.
Numerical stability: SVD helps compute pseudo-inverses for ill-conditioned matrices (ridge regression, least-squares).

3. How to compute (idea)

Compute eigen-decomposition of A^TA (which is symmetric & positive semidefinite). Its eigenvalues λᵢ ≥ 0. Then singular values σᵢ = sqrt(λᵢ). The columns of V are the eigenvectors of A^TA. Finally, U = A V Σ^-1 (for non-zero σᵢ).

4. Worked 2×2 example (step-by-step)

Take A = [[3, 1],[1, 3]]. We'll compute SVD.

Compute A^TA = [[10,6],[6,10]].
Find eigenvalues of A^TA: solve λ² - 20λ + 64 = 0 → λ₁ = 16, λ₂ = 4.
Singular values: σ₁ = 4, σ₂ = 2.
Compute normalized eigenvectors of A^TA → columns of V: v₁ = [1/√2, 1/√2], v₂ = [1/√2, -1/√2].
Compute U = A V Σ^-1 → U columns are u₁ = (A v₁)/σ₁, u₂ = (A v₂)/σ₂.
Check: A = U Σ V^T.

A = [[3,1],[1,3]]
A^T A = [[10,6],[6,10]]
Eigenvalues: 16, 4 → singular values 4, 2
V = [[1/√2, 1/√2],[1/√2, -1/√2]]
Σ = [[4,0],[0,2]]
U = A V Σ^{-1}  (compute column-wise)

This yields the SVD where the first singular vector captures the direction [1,1] (largest variance) and the second captures [1,-1].

Quick facts

Property	Value
Matrix type	Any m×n
Σ entries	σ₁ ≥ σ₂ ≥ ... ≥ 0
Best rank-k approx	Truncate Σ (Eckart–Young theorem)

5. Interactive SVD Calculator (2×2)

Enter numbers for a 2×2 matrix A. Calculator computes AᵀA, eigenvalues, singular values, V, U and Σ (rounded).

a11 a12
a21 a22

Reminders

• A^TA is symmetric → eigenvalues are real ≥ 0.
• Singular values are square roots of eigenvalues of A^TA.
• If A is m×n and rank r, there are r positive singular values.