Lecture 5 — Definite Matrices

1. Definitions

Let \(A\) be a real symmetric \(n \times n\) matrix (symmetry is important for the usual definiteness notions).

• Positive definite (PD): \(A\) is PD if for all non-zero vectors \(x\in\mathbb{R}^n\), \(x^\top A x > 0.\)
• Positive semidefinite (PSD): \(A\) is PSD if for all \(x\), \(x^\top A x \ge 0.\)
• Negative definite / semidefinite: defined similarly with \(x^\top A x < 0\) or \(\le 0\).
• Indefinite: exists \(x,y\) such that \(x^\top A x > 0\) and \(y^\top A y < 0.\)

2. Key properties (symmetric matrices)

• All eigenvalues of a symmetric matrix are real.
• \(A\) is PD \(\iff\) all eigenvalues of \(A\) are positive.
• \(A\) is PSD \(\iff\) all eigenvalues are non-negative.
• Sylvester's criterion: \(A\) is PD \(\iff\) all leading principal minors (determinants of top-left \(k\times k\) blocks) are positive.
• If \(A\) is PD, there exists a unique Cholesky decomposition \(A = L L^\top\) with lower-triangular \(L\) and positive diagonal.

3. Quick tests you can use

1. Eigenvalue test: compute eigenvalues and check signs. This is the most direct and reliable test.
2. Sylvester's criterion: check leading principal minors (fast for small matrices).
3. Cholesky factorization: try to compute it; if it succeeds (no negative square root), matrix is PD.

4. Worked examples

Example A (2×2 positive definite)

\[ A = \begin{bmatrix} 2 & -1 \\ -1 & 2 \end{bmatrix}, \quad x = \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} \] \[ x^\top A x = \begin{bmatrix} x_1 & x_2 \end{bmatrix} \begin{bmatrix} 2 & -1 \\ -1 & 2 \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = 2x_1^2 - 2x_1x_2 + 2x_2^2 = 2(x_1^2 - x_1x_2 + x_2^2) > 0 \quad (x \neq 0) \] Eigenvalues: \(\lambda_1 = 1,\; \lambda_2 = 3\) → \(A\) is PD.

Example B (PSD but not PD)

\[ A = \begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix} \] Eigenvalues: \(\lambda_1 = 0,\; \lambda_2 = 2\) → PSD (but not PD, since one eigenvalue is 0).

Example C (Indefinite)

\[ A = \begin{bmatrix} 1 & 2 \\ 2 & -1 \end{bmatrix} \] Eigenvalues: \(\lambda_1 \approx 2.236,\; \lambda_2 \approx -2.236\) → Indefinite.

5. Applications in Machine Learning

• Covariance matrices: Covariance matrix \(\Sigma\) is PSD (and PD if variables not collinear). PSD property ensures variances are non-negative and allows PCA.
• Kernel matrices: For a valid kernel \(k(\cdot,\cdot)\), Gram matrix \(K_{ij}=k(x_i,x_j)\) is PSD. Kernel methods rely on this property.
• Optimization: If Hessian is PD at a point, it’s a strict local minimum. Helps analyze convexity.
• Regularization: Adding \(\lambda I\) (\(\lambda>0\)) ensures PD and stabilizes matrix inversion.

6. Interactive checker