What is steepest descent algorithm

Abstract It is known that the steepest-descent method converges normally at the first few iterations, and then it slows down. Introduction The steepest-descent method SDM , which can be traced back to Cauchy , is the simplest gradient method for solving positive definite linear equations system. The Convergence of New Algorithm Solving 3 by the steepest-descent method SDM is equivalent to solving the following minimum problem: In the SDM, we search the next from the current state by minimizing the functional along the direction ; that is, Through some calculations we can obtain Thus we have the following iterative algorithm of SDM: A straightforward calculation reveals that It means that the iterative sequences of SDM converge to the minimal point.

A Generalized SDM and Its Optimization In Lemma 1 we have proven that the new scheme 16 is convergent, but it cannot tell us what descent direction is the best. A Generalized SDM We suppose that the original steepest-descent direction in the SDM is replaced by a new descent vector : which is an optimal combination of and the -vector. Optimization of Algorithm Let be an matrix, which consists of That is, the th column of is the vector.

For and , the descent direction which maximizes 31 is given by Moreover, the steplength defined in 30 is positive; that is, Proof. If , then one has Proof. Numerical Examples In order to evaluate the performance of the OGSDA, we test some well-known ill-posed linear problems of the Hilbert problem, the backward heat conduction problem, the heat source identification problem, and the inverse Cauchy problem.

Example 1 First by testing the performance of OGSDA on the solution of linear equations system, we consider the following convex quadratic programming problem with equality constraint: where is an matrix, is an matrix, and is an -vector, which means that 74 provides linear constraints.

According to the Lagrange theory we need to solve 1 with the following and : For definite we take and with Under the following parameters , , and and starting from the initial values of and two Lagrange multipliers and we apply the OGSDA to solve the resultant five-dimensional linear equations system, which is convergent with 38 steps as shown in Figure 1 a for the residual. Figure 1. Example 1 showing a residual, b and c optimal parameters, and d steplength and. Figure 2. For Example 1, using 67 the algorithm does not converge and gives a negative steplength.

Solutions Exact 1. Table 1. Figure 3. Figure 4. Figure 5. Figure 6. Figure 7. Figure 8. References J. Barzilai and J. View at: Google Scholar A. Friedlander, J. Molina, and M.

View at: Google Scholar M. Raydan and B. Dai, J. Yuan, and Y. Dai and L. Dai and Y. Qi, K. Teo, and X. Yang, Eds. Bhaya and E. Helmke and J. View at: Google Scholar Y. Hirsch and S.

View at: Google Scholar I. Ortega and W. View at: Google Scholar U. Ascher, K. Huang, and B. View at: Google Scholar C. Dai, W. Hager, K.

Schittkowski, and H. Frassoldati, L. Zanni, and G. View at: Google Scholar W. Hager, B. Mair, and H. Bonettini, R. Zanella, and L. Yu, L. Qi, and Y. Lindblad, and N. Setzer, G. Steidl, and J. Dongarra and F. View at: Google Scholar R. Freund and N. Van Den Eshof and G. View at: Google Scholar V. Method of Steepest Gradient Descent: descents in the direction of the largest directional derivative. This limit on the directional gradient changes the behavior of the descent.

There are other cases where one would favor an alternative norm for specific problems. In steepest descent after each backpropagation, the cost function is calculated. If cost has been reduced it continues and learning rate is doubled. If cost has been increased, the learning rate is halved and weights will be set to values of before backpropagation. The gradient lives in the dual space, i. Herein lies the key difference. The gradient is the directional derivative of a function.

Sign up to join this community. The best answers are voted up and rise to the top. Stack Overflow for Teams — Collaborate and share knowledge with a private group. Create a free Team What is Teams? A New Kind of Science. Champaign, IL: Wolfram Media, p. Weisstein, Eric W. Explore thousands of free applications across science, mathematics, engineering, technology, business, art, finance, social sciences, and more.

Walk through homework problems step-by-step from beginning to end. Hints help you try the next step on your own. Unlimited random practice problems and answers with built-in Step-by-step solutions. Practice online or make a printable study sheet. Collection of teaching and learning tools built by Wolfram education experts: dynamic textbook, lesson plans, widgets, interactive Demonstrations, and more.

Sign up to join this community. The best answers are voted up and rise to the top. Stack Overflow for Teams — Collaborate and share knowledge with a private group. Create a free Team What is Teams? Learn more. What is steepest descent? Is it gradient descent with exact line search? Ask Question. Asked 3 years, 10 months ago. Active 3 years, 10 months ago. Viewed 6k times. In some literature, such as this and this , steepest descent means using negative gradient direction and exact line search on that direction.

But in this note , It seems as well as we are following negative gradient, the method can be called steepest descent. Is the term "steepest descent" loosely defined?