Add NN inference demo & activation illustration

5_nn/2-mlp_bp.ipynb

@@ -150,6 +150,15 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
+    "\n",
+    "神经网络正向计算的过程比较简单，就是一层一层不断做运算就可以了，动态的演示如下图所示：\n",
+    "![](images/nn-forward.gif)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
     "## 5. 神经网络的训练 - 反向传播算法\n",
     "\n",
     "现在，我们需要知道一个神经网络的每个连接上的权值是如何得到的。我们可以说神经网络是一个模型，那么这些权值就是模型的参数，也就是模型要学习的东西。然而，一个神经网络的连接方式、网络的层数、每层的节点数这些参数，则不是学习出来的，而是人为事先设置的。对于这些人为设置的参数，我们称之为超参数(Hyper-Parameters)。\n",
@@ -302,7 +311,76 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 6. 示例程序"
+    "## 6. 为什么要使用激活函数\n",
+    "激活函数在神经网络中非常重要，使用激活函数也是非常必要的，前面我们从人脑神经元的角度理解了激活函数，因为神经元需要通过激活才能往后传播，所以神经网络中需要激活函数，下面我们从数学的角度理解一下激活函数的必要性。\n",
+    "\n",
+    "比如一个两层的神经网络，使用 A 表示激活函数，那么\n",
+    "\n",
+    "$$\n",
+    "y = w_2 A(w_1 x)\n",
+    "$$\n",
+    "\n",
+    "如果我们不使用激活函数，那么神经网络的结果就是\n",
+    "\n",
+    "$$\n",
+    "y = w_2 (w_1 x) = (w_2 w_1) x = \\bar{w} x\n",
+    "$$\n",
+    "\n",
+    "可以看到，我们将两层神经网络的参数合在一起，用 $\\bar{w}$ 来表示，两层的神经网络其实就变成了一层神经网络，只不过参数变成了新的 $\\bar{w}$，所以如果不使用激活函数，那么不管多少层的神经网络，$y = w_n \\cdots w_2 w_1 x = \\bar{w} x$，就都变成了单层神经网络，所以在每一层我们都必须使用激活函数。\n",
+    "\n",
+    "最后我们看看激活函数对神经网络的影响\n",
+    "\n",
+    "![](images/nn-activation-function.gif)\n",
+    "\n",
+    "可以看到使用了激活函数之后，神经网络可以通过改变权重实现任意形状，越是复杂的神经网络能拟合的形状越复杂，这就是著名的神经网络万有逼近定理。神经网络使用的激活函数都是非线性的，每个激活函数都输入一个值，然后做一种特定的数学运算得到一个结果。\n",
+    "\n",
+    "### 6.1 sigmoid 激活函数\n",
+    "\n",
+    "$$\\sigma(x) = \\frac{1}{1 + e^{-x}}$$\n",
+    "\n",
+    "![](images/act-sigmoid.jpg)\n",
+    "\n",
+    "### 6.2 tanh 激活函数\n",
+    "\n",
+    "$$tanh(x) = 2 \\sigma(2x) - 1$$\n",
+    "\n",
+    "![](images/act-tanh.jpg)\n",
+    "\n",
+    "### 6.3 ReLU 激活函数\n",
+    "\n",
+    "$$ReLU(x) = max(0, x)$$\n",
+    "\n",
+    "![](images/act-relu.jpg)\n",
+    "\n",
+    "当输入 $x<0$ 时，输出为 $0$，当 $x> 0$ 时，输出为 $x$。该激活函数使网络更快速地收敛。它不会饱和，即它可以对抗梯度消失问题，至少在正区域（$x> 0$ 时）可以这样，因此神经元至少在一半区域中不会把所有零进行反向传播。由于使用了简单的阈值化（thresholding），ReLU 计算效率很高。\n",
+    "\n",
+    "在网络中，不同的输入可能包含着大小不同关键特征，使用大小可变的数据结构去做容器，则更加灵活。假如神经元激活具有稀疏性，那么不同激活路径上：不同数量（选择性不激活）、不同功能（分布式激活）。两种可优化的结构生成的激活路径，可以更好地从有效的数据的维度上，学习到相对稀疏的特征，起到自动化解离效果。\n",
+    "\n",
+    "![](images/nn-sparse.png)\n",
+    "\n",
+    "在深度神经网络中，对非线性的依赖程度就少一些。另外，稀疏特征并不需要网络具有很强的处理线性不可分机制。因此在深度学习模型中，使用简单、速度快的线性激活函数可能更为合适。如图，一旦神经元与神经元之间改为线性激活，网络的非线性部分仅仅来自于神经元部分选择性激活。\n",
+    "\n",
+    "\n",
+    "更倾向于使用线性神经激活函数的另外一个原因是，减轻梯度法训练深度网络时的Vanishing Gradient Problem。\n",
+    "\n",
+    "看过BP推导的人都知道，误差从输出层反向传播算梯度时，在各层都要乘当前层的输入神经元值，激活函数的一阶导数。\n",
+    "$$\n",
+    "grad = error ⋅ sigmoid'(x) ⋅ x\n",
+    "$$\n",
+    "\n",
+    "使用双端饱和(即值域被限制)Sigmoid系函数会有两个问题：\n",
+    "\n",
+    "1. sigmoid'(x) ∈ (0,1)  导数缩放\n",
+    "2. x∈(0,1)或x∈(-1,1)  饱和值缩放\n",
+    "\n",
+    "这样，经过每一层时，Error都是成倍的衰减，一旦进行递推式的多层的反向传播，梯度就会不停的衰减，消失，使得网络学习变慢。而校正激活函数的梯度是1，且只有一端饱和，梯度很好的在反向传播中流动，训练速度得到了很大的提高。"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 7. 示例程序"
    ]
   },
   {
@@ -2524,7 +2602,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 7. 如何使用类的方法封装多层神经网络?"
+    "## 8. 如何使用类的方法封装多层神经网络?"
    ]
   },
   {
@@ -4773,7 +4851,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 8. 深入分析与问题"
+    "## 9. 深入分析与问题"
    ]
   },
   {
@@ -4843,7 +4921,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.5.2"
+   "version": "3.6.8"
   }
  },
  "nbformat": 4,

6_pytorch/1_NN/3-nn-sequential-module.ipynb

@@ -1163,7 +1163,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.5.2"
+   "version": "3.6.8"
   }
  },
  "nbformat": 4,

6_pytorch/1_NN/5-param_initialize.ipynb

@@ -462,7 +462,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.5.2"
+   "version": "3.6.8"
   }
  },
  "nbformat": 4,

6_pytorch/1_NN/optimizer/6_1-sgd.ipynb

@@ -296,7 +296,7 @@
     "losses3 = []\n",
     "idx = 0\n",
     "start = time.time() # 记时开始\n",
-    "for e in range(5):\n",
+    "for e in range(5):h\n",
     "    train_loss = 0\n",
     "    for im, label in train_data:\n",
     "        im = Variable(im)\n",
@@ -433,7 +433,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.5.2"
+   "version": "3.6.8"
   }
  },
  "nbformat": 4,

6_pytorch/1_NN/optimizer/6_2-momentum.ipynb

@@ -388,7 +388,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.5.2"
+   "version": "3.6.8"
   }
  },
  "nbformat": 4,

6_pytorch/1_NN/optimizer/6_3-adagrad.ipynb

@@ -256,7 +256,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.5.2"
+   "version": "3.6.8"
   }
  },
  "nbformat": 4,

6_pytorch/2_CNN/1-basic_conv.ipynb

@@ -347,7 +347,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.5.2"
+   "version": "3.6.8"
   }
  },
  "nbformat": 4,

6_pytorch/2_CNN/2-batch-normalization.ipynb

@@ -561,7 +561,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.5.2"
+   "version": "3.6.8"
   }
  },
  "nbformat": 4,

6_pytorch/2_CNN/3-lr-decay.ipynb

@@ -403,7 +403,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.5.2"
+   "version": "3.6.8"
   }
  },
  "nbformat": 4,

6_pytorch/3_RNN/nlp/n-gram.ipynb

@@ -455,7 +455,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.6.3"
+   "version": "3.6.8"
   }
  },
  "nbformat": 4,

6_pytorch/3_RNN/nlp/seq-lstm.ipynb

@@ -501,7 +501,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.6.3"
+   "version": "3.6.8"
   }
  },
  "nbformat": 4,

6_pytorch/3_RNN/nlp/word-embedding.ipynb

@@ -233,7 +233,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.6.3"
+   "version": "3.6.8"
   }
  },
  "nbformat": 4,

6_pytorch/3_RNN/pytorch-rnn.ipynb

@@ -67,9 +67,7 @@
   {
    "cell_type": "code",
    "execution_count": 48,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "data": {
@@ -138,9 +136,7 @@
   {
    "cell_type": "code",
    "execution_count": 52,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "data": {
@@ -166,9 +162,7 @@
   {
    "cell_type": "code",
    "execution_count": 54,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "data": {
@@ -188,9 +182,7 @@
   {
    "cell_type": "code",
    "execution_count": 55,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "data": {
@@ -235,9 +227,7 @@
   {
    "cell_type": "code",
    "execution_count": 33,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "data": {
@@ -278,9 +268,7 @@
   {
    "cell_type": "code",
    "execution_count": 36,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "data": {
@@ -307,9 +295,7 @@
   {
    "cell_type": "code",
    "execution_count": 35,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "data": {
@@ -355,9 +341,7 @@
   {
    "cell_type": "code",
    "execution_count": 41,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "out, h_t = rnn_seq(x, h_0)"
@@ -366,9 +350,7 @@
   {
    "cell_type": "code",
    "execution_count": 42,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "data": {
@@ -395,9 +377,7 @@
   {
    "cell_type": "code",
    "execution_count": 45,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "data": {
@@ -465,9 +445,7 @@
   {
    "cell_type": "code",
    "execution_count": 80,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "data": {
@@ -503,9 +481,7 @@
   {
    "cell_type": "code",
    "execution_count": 59,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "lstm_input = Variable(torch.randn(10, 3, 50)) # 序列 10，batch 是 3，输入维度 50"
@@ -532,9 +508,7 @@
   {
    "cell_type": "code",
    "execution_count": 66,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "data": {
@@ -554,9 +528,7 @@
   {
    "cell_type": "code",
    "execution_count": 67,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "data": {
@@ -576,9 +548,7 @@
   {
    "cell_type": "code",
    "execution_count": 61,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "data": {
@@ -628,9 +598,7 @@
   {
    "cell_type": "code",
    "execution_count": 70,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "data": {
@@ -650,9 +618,7 @@
   {
    "cell_type": "code",
    "execution_count": 71,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "data": {
@@ -672,9 +638,7 @@
   {
    "cell_type": "code",
    "execution_count": 72,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "data": {
@@ -723,9 +687,7 @@
   {
    "cell_type": "code",
    "execution_count": 76,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "data": {
@@ -753,9 +715,7 @@
   {
    "cell_type": "code",
    "execution_count": 75,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "data": {
@@ -775,9 +735,7 @@
   {
    "cell_type": "code",
    "execution_count": 74,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "data": {
@@ -797,9 +755,9 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "mx",
+   "display_name": "Python 3",
    "language": "python",
-   "name": "mx"
+   "name": "python3"
   },
   "language_info": {
    "codemirror_mode": {
@@ -811,7 +769,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.6.0"
+   "version": "3.6.8"
   }
  },
  "nbformat": 4,

6_pytorch/3_RNN/rnn-for-image.ipynb

@@ -183,7 +183,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.6.3"
+   "version": "3.6.8"
   }
  },
  "nbformat": 4,

6_pytorch/3_RNN/time-series/lstm-time-series.ipynb

@@ -381,7 +381,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.5.2"
+   "version": "3.6.8"
   }
  },
  "nbformat": 4,

6_pytorch/4_GAN/autoencoder.ipynb

@@ -478,7 +478,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.5.2"
+   "version": "3.6.8"
   }
  },
  "nbformat": 4,

6_pytorch/4_GAN/gan.ipynb

@@ -1423,7 +1423,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.5.2"
+   "version": "3.6.8"
   }
  },
  "nbformat": 4,

6_pytorch/4_GAN/vae.ipynb

@@ -401,7 +401,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.5.2"
+   "version": "3.6.8"
   }
  },
  "nbformat": 4,