Summary This post serves as a broad overview of many optimization methods and algorithms. For specifics, please refer to the post of each algorithm. It is advised to read the post about Functional Analysis to better understand the notation used here. Setup We are going to consider a function $$ f = g + h. $$ Depending on the different hypothesis of $f$, $g$ and $h$ we may be able to use different optimization algorithms. ...
The study of optimization of general functions often rely on many Functional Analysis tools, particularly Hilbert spaces. One can think about Hilbert spaces as a generalization of Euclidean spaces. This allows us to define most algorithms in a generalized fashion, without worrying about the underlying structure of the problem’s domain. Of course, most applications consider an Euclidean space $\mathbb{R}^d$, but it is just a special case of a Hilbert space. ...