Lecture 12: Linear and Non-Linear Programming
Lecture 12: Linear Programming (LP) and Non-Linear Programming (NLP)
1. Introduction
Optimization is the process of finding the best solution (minimum or maximum) for a given objective function
subject to certain constraints.
- Linear Programming (LP): Optimization where both the objective function and constraints are linear.
- Non-Linear Programming (NLP): Optimization where objective function or constraints (or both) are non-linear.
2. Linear Programming (LP)
General Form:
Maximize (or Minimize): Z = cTx
Subject to: A x ≤ b , x ≥ 0
Example of LP:
Maximize profit: Z = 40x + 30y
Subject to:
x + y ≤ 12
2x + y ≤ 16
x, y ≥ 0
Here, x and y are decision variables, and constraints form a feasible region (polygon). The optimal solution lies on the boundary.
Applications of LP:
- Resource allocation (production planning, diet problems)
- Transportation and supply chain optimization
- Portfolio optimization in finance
- Scheduling and workforce management
3. Non-Linear Programming (NLP)
General Form:
Minimize (or Maximize): f(x)
Subject to: gi(x) ≤ 0, hj(x) = 0
Example of NLP:
Minimize: f(x, y) = x2 + y2
Subject to: x + y = 1, x ≥ 0, y ≥ 0
Here, the objective is quadratic (non-linear), but the constraint is linear.
Applications of NLP:
- Machine learning model training (loss minimization, e.g. logistic regression, neural networks)
- Engineering design problems (non-linear stress, fluid dynamics)
- Energy optimization (power flow problems)
- Economics (utility maximization with non-linear functions)
4. Difference Between LP and NLP
| Aspect | Linear Programming (LP) | Non-Linear Programming (NLP) |
|---|
| Objective function | Linear | Non-linear |
| Constraints | Linear | Can be non-linear |
| Complexity | Relatively simple | Harder, may have local minima |
| Solution Methods | Simplex, Interior-point | Gradient descent, Newton, KKT conditions |
5. Relevance in Machine Learning
- LP is used in Support Vector Machines (SVM) for linear margin optimization.
- NLP is used in Neural Networks, Logistic Regression, Deep Learning since loss functions are non-linear.
- Regularization techniques (L1, L2) add linear or quadratic terms, connecting LP & NLP concepts.
6. Summary
- LP deals with linear functions & constraints; efficient to solve.
- NLP handles real-world complex scenarios where non-linearity appears.
- Both are cornerstones of optimization in Machine Learning.