Lecture 13: Quadratic Programming and KKT Conditions

Lecture 13: Quadratic Programming (QP) and KKT Conditions

1. Introduction

Quadratic Programming (QP) is a special type of optimization problem where the objective function is quadratic and the constraints are linear. It is one of the most important optimization frameworks in machine learning, especially in Support Vector Machines (SVMs), portfolio optimization, and control systems.

2. General Form of Quadratic Programming

Minimize: f(x) = 1/2 x^TQx + c^Tx
Subject to: Ax ≤ b, Ex = d

Q → Symmetric matrix (defines curvature of quadratic function)
c → Linear coefficients vector
A, b → Inequality constraints
E, d → Equality constraints

3. Types of Quadratic Programming

Convex QP: If Q is positive semidefinite → global optimum guaranteed.
Non-convex QP: If Q has negative eigenvalues → local minima/maxima possible.

4. Properties of Quadratic Programming

Convex QPs can be solved efficiently using interior-point or active-set methods.
Non-convex QPs are NP-hard in general.
Solution depends heavily on the definiteness of matrix Q.
KKT conditions provide necessary (and for convex case, sufficient) conditions for optimality.

5. Karush–Kuhn–Tucker (KKT) Conditions

The KKT conditions are first-order necessary conditions for a solution in nonlinear programming to be optimal, under certain regularity conditions.

L(x, λ, μ) = f(x) + λ^T(Ex - d) + μ^T(Ax - b)

Conditions:
1. Stationarity: ∇f(x) + E^Tλ + A^Tμ = 0
2. Primal feasibility: Ex = d, Ax ≤ b
3. Dual feasibility: μ ≥ 0
4. Complementary slackness: μ_i(a_ix - b_i) = 0

6. Example

Problem:
Minimize: f(x) = (1/2)(x₁² + x₂²) - x₁ - 2x₂
Subject to: x₁ + x₂ ≤ 1, x₁, x₂ ≥ 0

Solution Steps:

Formulate Lagrangian with multipliers λ and μ.
Apply KKT conditions for stationarity and feasibility.
Solve system to find optimal x₁, x₂.

This is a convex QP since Q = I (identity matrix) is positive definite.

7. Applications in Machine Learning

Support Vector Machines (SVMs): Training involves solving a convex quadratic programming problem.
Portfolio Optimization: Quadratic objective to minimize risk (variance).
Control Theory: Model predictive control often uses QP for trajectory optimization.
Feature Selection: QP used to minimize error subject to sparsity constraints.

8. Summary

QP = Quadratic objective + linear constraints.
KKT conditions = optimality criteria.
Convex QPs guarantee global optimal solutions.
Applications widely found in ML: SVMs, optimization, control.