Lecture 2: Eigenvalues, Eigenvectors & EVD

Eigenvalues and eigenvectors are fundamental concepts in linear algebra that describe how a linear transformation acts on vectors. If we have a matrix A and a vector v, then v is an eigenvector of A if multiplying A by v results in a scaled version of v:

Here, λ is called the eigenvalue, and it tells us how much the eigenvector is stretched or shrunk. Eigenvectors give us special directions where transformations act as simple scalings.

2. Eigenvalue Decomposition (EVD)

- V is the matrix of eigenvectors (columns are eigenvectors).
- Λ is the diagonal matrix of eigenvalues.
- This decomposition is useful in machine learning, physics, and engineering.

3. Example: 2×2 Matrix

Consider the matrix:

    A = [4   2]
        [1   3]

Step 1: Find the characteristic equation

det(A - λI) = 0

    |4-λ   2  |
    | 1   3-λ| = (4-λ)(3-λ) - 2 = λ² - 7λ + 10 = 0

Step 2: Solve for eigenvalues

    λ² - 7λ + 10 = 0 → (λ-5)(λ-2) = 0
    → Eigenvalues: λ₁ = 5, λ₂ = 2

Step 3: Find eigenvectors

For λ₁ = 5: Solve (A - 5I)v = 0 → Eigenvector v₁ = [2, 1]
For λ₂ = 2: Solve (A - 2I)v = 0 → Eigenvector v₂ = [-1, 1]

Step 4: Form EVD

    V = [[2  -1],
         [1   1]]

    Λ = [[5  0],
         [0  2]]

    A = VΛV⁻¹

4. Applications in Machine Learning

Principal Component Analysis (PCA): Eigenvectors of covariance matrix represent principal directions of data; eigenvalues represent variance along those directions.
Dimensionality Reduction: Keep top-k eigenvalues/eigenvectors to project data into lower dimensions.
Stability Analysis: Eigenvalues tell whether systems (like recurrent networks) are stable or unstable.

Lecture 2: Eigenvalues, Eigenvectors & Eigenvalue Decomposition (EVD)

1. Introduction