DCAM MIPT course (6 sem)
Home assignments (1 - 9) on optimization within the course "Methods of Optimization at MIPT". Lecturer - PhD A. Beznosikov
Topics of homework assignments:
-
Gradient descent. Steepest descent. Armijo, Wolfe, Goldstein rules. Polak - Ribiere step.
-
Projection gradient descent. Minimizing empirical risk problem. Proximal gradient descent. Regularization. Principal Component Analysis.
-
Logistic regression. Heavy ball method. Nesterov's accelerated method.
-
Mirror descent, Bregman divergence. Frank-Wolfe method, its accelerated version.
-
Stochastic methods: SGD, SAGA, SVRG, and SARAH.
-
Coordinate stochastic methods: coordinate SGD, SEGA.
-
Distributed minimization problem with compression operators RAND k%, TOP k%. Error feedback. DIANA.
-
Saddle point problem. Gradient descent. Extrapolation with projection, its modification. Solving a bilinear problem on the simplex.
-
Second-order methods. Newton's method, its modifications - damped (adding a step) and cubic Newton's method. Quasi-Newton methods: Broyden, DFP, BFGS, L-BFGS.