Lecture Notes

This section contains lecture notes and some associated readings.

Complete lecture notes (PDF - 7.7MB)

TOPICS LECTURE NOTES READINGS

The role of convexity in optimization

Duality theory

Algorithms and duality

Lecture 1 (PDF - 1.2MB)  

Convex sets and functions

Epigraphs

Closed convex functions

Recognizing convex functions

Lecture 2 (PDF) Section 1.1

Differentiable convex functions

Convex and affine hulls

Caratheodory's theorem

Lecture 3 (PDF) Sections 1.1, 1.2

Relative interior and closure

Algebra of relative interiors and closures

Continuity of convex functions

Closures of functions

Lecture 4 (PDF) Section 1.3

Recession cones and lineality space

Directions of recession of convex functions

Local and global minima

Existence of optimal solutions

Lecture 5 (PDF - 1.0MB) Sections 1.4, 3.1, 3.2

Nonemptiness of closed set intersections

Existence of optimal solutions

Preservation of closure under linear transformation

Hyperplanes

Lecture 6 (PDF - 1.4MB)  

Review of hyperplane separation

Nonvertical hyperplanes

Convex conjugate functions

Conjugacy theorem

Examples

Lecture 7 (PDF) Sections 1.5, 1.6

Review of conjugate convex functions

Min common / max crossing duality

Weak duality

Special cases

Lecture 8 (PDF - 1.2MB) Sections 1.6, 4.1, 4.2

Minimax problems and zero-sum games

Min common / max crossing duality for minimax and zero-sum games

Min common / max crossing duality theorems

Strong duality conditions

Existence of dual optimal solutions

Lecture 9 (PDF) Sections 3.4, 4.3, 4.4, 5.1

Min common / max crossing Theorem III

Nonlinear Farkas' lemma / linear constraints

Linear programming duality

Convex programming duality

Optimality conditions

Lecture 10 (PDF) Sections 4.5, 5.1, 5.2, 5.3.1–5.3.2

Review of convex programming duality / counterexamples

Fenchel duality

Conic duality

Lecture 11 (PDF) Sections 5.3.1–5.3.6

Subgradients

Fenchel inequality

Sensitivity in constrained optimization

Subdifferential calculus

Optimality conditions

Lecture 12 (PDF) Section 5.4

Problem structure

Conic programming

Lecture 13 (PDF)  

Conic programming

Semidefinite programming

Exact penalty functions

Descent methods for convex optimization

Steepest descent method

Lecture 14 (PDF) Chapter 6: Convex Optimization Algorithms (This resource may not render correctly in a screen reader.PDF)

Subgradient methods

Calculation of subgradients

Convergence

Lecture 15 (PDF)  

Approximate subgradient methods

Approximation methods

Cutting plane methods

Lecture 16 (PDF)  

Review of cutting plane method

Simplicial decomposition

Duality between cutting plane and simplicial decomposition

Lecture 17 (PDF)  

Generalized polyhedral approximation methods

Combined cutting plane and simplicial decomposition methods

Lecture 18 (PDF) Bertsekas, Dimitri, and Huizhen Yu. "A Unifying Polyhedral Approximation Framework for Convex Optimization." SIAM Journal on Optimization 21, no. 1 (2011): 333–60.

Proximal minimization algorithm

Extensions

Lecture 19 (PDF)  

Proximal methods

Review of proximal minimization

Proximal cutting plane algorithm

Bundle methods

Augmented Lagrangian methods

Dual proximal minimization algorithm

Lecture 20 (PDF - 1.1MB)  

Generalized forms of the proximal point algorithm

Interior point methods

Constrained optimization case: barrier method

Conic programming cases

Lecture 21 (PDF)  

Incremental methods

Review of large sum problems

Review of incremental gradient and subgradient methods

Combined incremental subgradient and proximal methods

Convergence analysis

Cyclic and randomized component selection

Lecture 22 (PDF)

Bertsekas, Dimitri. "Incremental Gradient, Subgradient, and Proximal Methods for Convex Optimization: A Survey." (This resource may not render correctly in a screen reader.PDF) Laboratory for Information and Decision Systems Report LIDS-P-2848, MIT, August 2010.

Buy at MIT Press Sra, Suvrit, Sebastian Nowozin, and Stephen Wright, eds. Optimization for Machine Learning. MIT Press, 2011. ISBN: 9780262016469.

Review of subgradient methods

Application to differentiable problems: gradient projection

 

Iteration complexity issues

Complexity of gradient projection

Projection method with extrapolation

Optimal algorithms

Lecture 23 (PDF)  

Gradient proximal minimization method

Nonquadratic proximal algorithms

Entropy minimization algorithm

Exponential augmented Lagrangian method

Entropic descent algorithm

Lecture 24 (PDF)

Beck, Amir, and Marc Teboulle. "Gradient-Based Algorithms with Applications to Signal-Recovery Problems." In Convex Optimization in Signal Processing and Communications. Edited by Daniel Palomar and Yonina Eldar. Cambridge University Press, 2010. ISBN: 9780521762229.

Beck, Amir, and Marc Teboulle. "Mirror Descent and Nonlinear Projected Subgradient Methods for Convex Optimization." Operations Research Letters 31, no. 3 (2003): 167–75.

Bertsekas, Dimitri. Nonlinear Programming. Athena Scientific, 1999. ISBN: 9781886529007.

Convex analysis and duality

Convex optimization algorithms

Lecture 25 (PDF - 2.0MB)