Section 6.2 Iterative methods
These are popular methods to find maxima and minima of functions. Moreover, at the same time, they also work as methods to solve linear and nonlinear systems! They work as follows: start with some \(\boldsymbol{x_{0}}\text{,}\) that must be decided after some study of the functions (there is no universal recipe!) and at every step \(k\) determine a direction \(\boldsymbol{d}_k\) and a number \(\alpha_k\) so that the new point
\begin{equation*}
\boldsymbol{x_{k+1}}=\boldsymbol{x}_k+\alpha_k \boldsymbol{d}_k
\end{equation*}
gets closer and closer to the minimum \(\boldsymbol{\bar x}\text{.}\) The idea is that, if we choose carefully enough \(\boldsymbol{d}_k\) and \(\alpha_k\) and we choose the starting point close enough to the minimum, this process will indeed converge to the desired solution.