# 梯度下降算法

## 梯度下降算法

* [返回顶层目录](https://luweikxy.gitbook.io/machine-learning-notes/summary#目录)
* [各类梯度下降算法的演化](#各类梯度下降算法的演化)
* [随机梯度下降SGD](https://luweikxy.gitbook.io/machine-learning-notes/broken-reference)
* [动量法Momentum](https://luweikxy.gitbook.io/machine-learning-notes/gradient-descent-algorithm/momentum)
* [牛顿动量Nesterov](https://luweikxy.gitbook.io/machine-learning-notes/gradient-descent-algorithm/nesterov)
* [AdaGrad](https://luweikxy.gitbook.io/machine-learning-notes/gradient-descent-algorithm/adagrad)
* [RMSprop](https://luweikxy.gitbook.io/machine-learning-notes/gradient-descent-algorithm/rmsprop)
* [Adadelta](https://luweikxy.gitbook.io/machine-learning-notes/gradient-descent-algorithm/adadelta)
* [Adam](https://luweikxy.gitbook.io/machine-learning-notes/gradient-descent-algorithm/adam)
* [Nadam](https://luweikxy.gitbook.io/machine-learning-notes/gradient-descent-algorithm/nadam)
* [AMSGrad](https://luweikxy.gitbook.io/machine-learning-notes/gradient-descent-algorithm/amsgrad)
* [AdasMax](https://luweikxy.gitbook.io/machine-learning-notes/gradient-descent-algorithm/adamax)

![optimization-on-loss-surface-contours](https://3298324061-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LpO5sn2FY1C9esHFJmo%2F-LurJHbNKPiaarTTRnqJ%2F-LurJLn_2Peov9ITKLo3%2Foptimization-on-loss-surface-contours.gif?generation=1575033598545052\&alt=media)

## 各类梯度下降算法的演化

![revolution-of-gradient-descent](https://3298324061-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LpO5sn2FY1C9esHFJmo%2F-Lug-9Dm3wMqPuGpCQ0c%2F-Lug-Cj8eWQsYrsBcgKw%2Frevolution-of-gradient-descent.jpeg?generation=1574843768973331\&alt=media)

## 参考资料

* [Deep Learning 之 最优化方法](https://blog.csdn.net/BVL10101111/article/details/72614711)
* [从 SGD 到 Adam —— 深度学习优化算法概览(一)](https://zhuanlan.zhihu.com/p/32626442)
* [深度学习最全优化方法总结比较（SGD，Adagrad，Adadelta，Adam，Adamax，Nadam）](https://zhuanlan.zhihu.com/p/22252270)
* [10个梯度下降优化算法+备忘单](https://ai.yanxishe.com/page/TextTranslation/1603?from=singlemessage)

  [深度学习——优化器算法Optimizer详解（BGD、SGD、MBGD、Momentum、NAG、Adagrad、Adadelta、RMSprop、Adam）](https://www.cnblogs.com/guoyaohua/p/8542554.html)