machinelearning_notebook/6_pytorch/0_basic/autograd.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 自动求导\n",
    "这次课程我们会了解 PyTorch 中的自动求导机制，自动求导是 PyTorch 中非常重要的特性，能够让我们避免手动去计算非常复杂的导数，这能够极大地减少了我们构建模型的时间，这也是其前身 Torch 这个框架所不具备的特性，下面我们通过例子看看 PyTorch 自动求导的独特魅力以及探究自动求导的更多用法。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.autograd import Variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 简单情况的自动求导\n",
    "下面我们显示一些简单情况的自动求导，\"简单\"体现在计算的结果都是标量，也就是一个数，我们对这个标量进行自动求导。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([19.], grad_fn=<AddBackward0>)\n"
     ]
    }
   ],
   "source": [
    "x = Variable(torch.Tensor([2]), requires_grad=True)\n",
    "y = x + 2\n",
    "z = y ** 2 + 3\n",
    "print(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过上面的一些列操作，我们从 x 得到了最后的结果out，我们可以将其表示为数学公式\n",
    "\n",
    "$$\n",
    "z = (x + 2)^2 + 3\n",
    "$$\n",
    "\n",
    "那么我们从 z 对 x 求导的结果就是 \n",
    "\n",
    "$$\n",
    "\\frac{\\partial z}{\\partial x} = 2 (x + 2) = 2 (2 + 2) = 8\n",
    "$$\n",
    "如果你对求导不熟悉，可以查看以下[网址进行复习](https://baike.baidu.com/item/%E5%AF%BC%E6%95%B0#1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([8.])\n"
     ]
    }
   ],
   "source": [
    "# 使用自动求导\n",
    "z.backward()\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于上面这样一个简单的例子，我们验证了自动求导，同时可以发现发现使用自动求导非常方便。如果是一个更加复杂的例子，那么手动求导就会显得非常的麻烦，所以自动求导的机制能够帮助我们省去麻烦的数学计算，下面我们可以看一个更加复杂的例子。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-1.5318,  1.5200, -2.1316, -1.3238,  1.0080, -1.0832, -0.2814, -1.0486,\n",
      "          1.0807, -2.2865,  0.6545, -0.3595,  0.4229, -0.9194,  0.1690, -0.3241,\n",
      "          1.8970, -0.8979, -0.7827,  0.3879],\n",
      "        [ 0.1404, -0.8016,  0.1156, -0.8397, -1.8886,  1.1072, -1.0186,  0.2249,\n",
      "          0.5631,  0.4391,  0.7887, -2.3255, -0.4185,  0.6559,  0.7622,  1.6883,\n",
      "         -1.4147,  0.2579, -0.6177,  0.2172],\n",
      "        [-0.4866, -0.0322, -1.2484,  1.1913, -0.6569,  0.0810,  0.2491, -0.1258,\n",
      "          2.5903, -0.8370, -0.0554,  1.2174,  0.4059, -1.0759,  0.6649,  0.1642,\n",
      "         -0.3512, -0.7695,  1.1469, -0.3409],\n",
      "        [ 1.8789, -1.6553, -0.7401, -0.3198, -0.1010, -0.5512, -0.4792, -0.2891,\n",
      "         -0.2655, -0.8132,  0.7210,  1.0885, -0.9557, -0.4472, -1.5340,  0.8093,\n",
      "          0.9349,  0.8352, -0.0774, -0.1728],\n",
      "        [-0.3424,  0.1938, -2.4253, -0.0229,  0.3132, -0.7731,  0.8481, -1.3002,\n",
      "         -0.1595, -0.0364, -1.5733,  0.8882,  0.1909, -0.1404, -1.5673, -1.1809,\n",
      "         -0.7169,  0.7074,  0.3337, -1.0738],\n",
      "        [-0.0501,  1.6210,  0.6854,  0.2216,  0.3034, -1.2762, -0.6216,  1.4884,\n",
      "          0.6078,  2.1512, -0.7141,  0.4110, -0.8187,  0.9474, -0.5978, -0.2679,\n",
      "          1.5315, -2.1550,  2.0969, -1.7669],\n",
      "        [ 1.4505, -0.9497,  2.0269, -1.6402, -0.0047, -0.2716, -0.2727,  0.6795,\n",
      "         -0.7367, -0.3248, -0.5312,  0.0887, -1.4303, -0.8390,  1.5324,  0.3761,\n",
      "         -0.4658, -0.2044,  0.3050, -0.2756],\n",
      "        [ 0.3265, -0.2513,  1.1441,  0.3805, -1.3629, -1.3120, -1.8571,  0.1180,\n",
      "          0.7466, -0.2654, -0.2154,  1.0603, -0.4113, -2.5965,  1.0736,  1.1610,\n",
      "          0.8165,  1.5916,  1.5556,  0.3078],\n",
      "        [-0.4417,  0.1656, -2.1743, -0.1148, -1.2795,  1.0212, -0.7035, -0.8234,\n",
      "          0.3010, -1.0891, -1.0676,  0.8385, -0.2886, -1.1881,  0.5097, -0.5097,\n",
      "         -1.7893,  0.0494, -0.0162,  1.5170],\n",
      "        [-0.6435, -1.8376,  1.0022, -0.0397,  0.7187, -0.0661, -0.8528,  1.3248,\n",
      "         -0.2566, -2.2886,  0.8728, -0.7152,  1.6180,  0.8416,  0.2788,  0.5515,\n",
      "         -0.1266, -1.0025,  0.1767, -0.4987]], requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "x = Variable(torch.randn(10, 20), requires_grad=True)\n",
    "y = Variable(torch.randn(10, 5), requires_grad=True)\n",
    "w = Variable(torch.randn(20, 5), requires_grad=True)\n",
    "print(x)\n",
    "out = torch.mean(y - torch.matmul(x, w)) # torch.matmul 是做矩阵乘法\n",
    "out.backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果你对矩阵乘法不熟悉，可以查看下面的[网址进行复习](https://baike.baidu.com/item/%E7%9F%A9%E9%98%B5%E4%B9%98%E6%B3%95/5446029?fr=aladdin)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.0198, -0.0066, -0.0288,  0.0080,  0.0079, -0.0569, -0.0489,  0.0505,\n",
      "          0.0132, -0.0072, -0.0024,  0.0400,  0.0691, -0.0273,  0.0124,  0.0104,\n",
      "          0.0098, -0.0598,  0.0365,  0.0177],\n",
      "        [-0.0198, -0.0066, -0.0288,  0.0080,  0.0079, -0.0569, -0.0489,  0.0505,\n",
      "          0.0132, -0.0072, -0.0024,  0.0400,  0.0691, -0.0273,  0.0124,  0.0104,\n",
      "          0.0098, -0.0598,  0.0365,  0.0177],\n",
      "        [-0.0198, -0.0066, -0.0288,  0.0080,  0.0079, -0.0569, -0.0489,  0.0505,\n",
      "          0.0132, -0.0072, -0.0024,  0.0400,  0.0691, -0.0273,  0.0124,  0.0104,\n",
      "          0.0098, -0.0598,  0.0365,  0.0177],\n",
      "        [-0.0198, -0.0066, -0.0288,  0.0080,  0.0079, -0.0569, -0.0489,  0.0505,\n",
      "          0.0132, -0.0072, -0.0024,  0.0400,  0.0691, -0.0273,  0.0124,  0.0104,\n",
      "          0.0098, -0.0598,  0.0365,  0.0177],\n",
      "        [-0.0198, -0.0066, -0.0288,  0.0080,  0.0079, -0.0569, -0.0489,  0.0505,\n",
      "          0.0132, -0.0072, -0.0024,  0.0400,  0.0691, -0.0273,  0.0124,  0.0104,\n",
      "          0.0098, -0.0598,  0.0365,  0.0177],\n",
      "        [-0.0198, -0.0066, -0.0288,  0.0080,  0.0079, -0.0569, -0.0489,  0.0505,\n",
      "          0.0132, -0.0072, -0.0024,  0.0400,  0.0691, -0.0273,  0.0124,  0.0104,\n",
      "          0.0098, -0.0598,  0.0365,  0.0177],\n",
      "        [-0.0198, -0.0066, -0.0288,  0.0080,  0.0079, -0.0569, -0.0489,  0.0505,\n",
      "          0.0132, -0.0072, -0.0024,  0.0400,  0.0691, -0.0273,  0.0124,  0.0104,\n",
      "          0.0098, -0.0598,  0.0365,  0.0177],\n",
      "        [-0.0198, -0.0066, -0.0288,  0.0080,  0.0079, -0.0569, -0.0489,  0.0505,\n",
      "          0.0132, -0.0072, -0.0024,  0.0400,  0.0691, -0.0273,  0.0124,  0.0104,\n",
      "          0.0098, -0.0598,  0.0365,  0.0177],\n",
      "        [-0.0198, -0.0066, -0.0288,  0.0080,  0.0079, -0.0569, -0.0489,  0.0505,\n",
      "          0.0132, -0.0072, -0.0024,  0.0400,  0.0691, -0.0273,  0.0124,  0.0104,\n",
      "          0.0098, -0.0598,  0.0365,  0.0177],\n",
      "        [-0.0198, -0.0066, -0.0288,  0.0080,  0.0079, -0.0569, -0.0489,  0.0505,\n",
      "          0.0132, -0.0072, -0.0024,  0.0400,  0.0691, -0.0273,  0.0124,  0.0104,\n",
      "          0.0098, -0.0598,  0.0365,  0.0177]])\n"
     ]
    }
   ],
   "source": [
    "# 得到 x 的梯度\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.0200, 0.0200, 0.0200, 0.0200, 0.0200],\n",
      "        [0.0200, 0.0200, 0.0200, 0.0200, 0.0200],\n",
      "        [0.0200, 0.0200, 0.0200, 0.0200, 0.0200],\n",
      "        [0.0200, 0.0200, 0.0200, 0.0200, 0.0200],\n",
      "        [0.0200, 0.0200, 0.0200, 0.0200, 0.0200],\n",
      "        [0.0200, 0.0200, 0.0200, 0.0200, 0.0200],\n",
      "        [0.0200, 0.0200, 0.0200, 0.0200, 0.0200],\n",
      "        [0.0200, 0.0200, 0.0200, 0.0200, 0.0200],\n",
      "        [0.0200, 0.0200, 0.0200, 0.0200, 0.0200],\n",
      "        [0.0200, 0.0200, 0.0200, 0.0200, 0.0200]])\n"
     ]
    }
   ],
   "source": [
    "# 得到 y 的的梯度\n",
    "print(y.grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.0060, -0.0060, -0.0060, -0.0060, -0.0060],\n",
      "        [ 0.0405,  0.0405,  0.0405,  0.0405,  0.0405],\n",
      "        [ 0.0749,  0.0749,  0.0749,  0.0749,  0.0749],\n",
      "        [ 0.0502,  0.0502,  0.0502,  0.0502,  0.0502],\n",
      "        [ 0.0590,  0.0590,  0.0590,  0.0590,  0.0590],\n",
      "        [ 0.0625,  0.0625,  0.0625,  0.0625,  0.0625],\n",
      "        [ 0.0998,  0.0998,  0.0998,  0.0998,  0.0998],\n",
      "        [-0.0050, -0.0050, -0.0050, -0.0050, -0.0050],\n",
      "        [-0.0894, -0.0894, -0.0894, -0.0894, -0.0894],\n",
      "        [ 0.1070,  0.1070,  0.1070,  0.1070,  0.1070],\n",
      "        [ 0.0224,  0.0224,  0.0224,  0.0224,  0.0224],\n",
      "        [-0.0438, -0.0438, -0.0438, -0.0438, -0.0438],\n",
      "        [ 0.0337,  0.0337,  0.0337,  0.0337,  0.0337],\n",
      "        [ 0.0952,  0.0952,  0.0952,  0.0952,  0.0952],\n",
      "        [-0.0258, -0.0258, -0.0258, -0.0258, -0.0258],\n",
      "        [-0.0494, -0.0494, -0.0494, -0.0494, -0.0494],\n",
      "        [-0.0063, -0.0063, -0.0063, -0.0063, -0.0063],\n",
      "        [ 0.0318,  0.0318,  0.0318,  0.0318,  0.0318],\n",
      "        [-0.0824, -0.0824, -0.0824, -0.0824, -0.0824],\n",
      "        [ 0.0340,  0.0340,  0.0340,  0.0340,  0.0340]])\n"
     ]
    }
   ],
   "source": [
    "# 得到 w 的梯度\n",
    "print(w.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面数学公式就更加复杂，矩阵乘法之后对两个矩阵对应元素相乘，然后所有元素求平均，有兴趣的同学可以手动去计算一下梯度，使用 PyTorch 的自动求导，我们能够非常容易得到 x, y 和 w 的导数，因为深度学习中充满大量的矩阵运算，所以我们没有办法手动去求这些导数，有了自动求导能够非常方便地解决网络更新的问题。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 复杂情况的自动求导\n",
    "上面我们展示了简单情况下的自动求导，都是对标量进行自动求导，可能你会有一个疑问，如何对一个向量或者矩阵自动求导了呢？感兴趣的同学可以自己先去尝试一下，下面我们会介绍对多维数组的自动求导机制。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[2., 3.]], requires_grad=True)\n",
      "tensor([[0., 0.]])\n"
     ]
    }
   ],
   "source": [
    "m = Variable(torch.FloatTensor([[2, 3]]), requires_grad=True) # 构建一个 1 x 2 的矩阵\n",
    "n = Variable(torch.zeros(1, 2)) # 构建一个相同大小的 0 矩阵\n",
    "print(m)\n",
    "print(n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(2., grad_fn=<SelectBackward>)\n",
      "tensor([[ 4., 27.]], grad_fn=<CopySlices>)\n"
     ]
    }
   ],
   "source": [
    "# 通过 m 中的值计算新的 n 中的值\n",
    "print(m[0,0])\n",
    "n[0, 0] = m[0, 0] ** 2\n",
    "n[0, 1] = m[0, 1] ** 3\n",
    "print(n)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将上面的式子写成数学公式，可以得到 \n",
    "$$\n",
    "n = (n_0,\\ n_1) = (m_0^2,\\ m_1^3) = (2^2,\\ 3^3) \n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面我们直接对 n 进行反向传播，也就是求 n 对 m 的导数。\n",
    "\n",
    "这时我们需要明确这个导数的定义，即如何定义\n",
    "\n",
    "$$\n",
    "\\frac{\\partial n}{\\partial m} = \\frac{\\partial (n_0,\\ n_1)}{\\partial (m_0,\\ m_1)}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在 PyTorch 中，如果要调用自动求导，需要往`backward()`中传入一个参数，这个参数的形状和 n 一样大，比如是 $(w_0,\\ w_1)$，那么自动求导的结果就是：\n",
    "$$\n",
    "\\frac{\\partial n}{\\partial m_0} = w_0 \\frac{\\partial n_0}{\\partial m_0} + w_1 \\frac{\\partial n_1}{\\partial m_0}\n",
    "$$\n",
    "$$\n",
    "\\frac{\\partial n}{\\partial m_1} = w_0 \\frac{\\partial n_0}{\\partial m_1} + w_1 \\frac{\\partial n_1}{\\partial m_1}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "n.backward(torch.ones_like(n)) # 将 (w0, w1) 取成 (1, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 4., 27.]])\n"
     ]
    }
   ],
   "source": [
    "print(m.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过自动求导我们得到了梯度是 4 和 27，我们可以验算一下\n",
    "$$\n",
    "\\frac{\\partial n}{\\partial m_0} = w_0 \\frac{\\partial n_0}{\\partial m_0} + w_1 \\frac{\\partial n_1}{\\partial m_0} = 2 m_0 + 0 = 2 \\times 2 = 4\n",
    "$$\n",
    "$$\n",
    "\\frac{\\partial n}{\\partial m_1} = w_0 \\frac{\\partial n_0}{\\partial m_1} + w_1 \\frac{\\partial n_1}{\\partial m_1} = 0 + 3 m_1^2 = 3 \\times 3^2 = 27\n",
    "$$\n",
    "通过验算我们可以得到相同的结果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 多次自动求导\n",
    "通过调用 backward 我们可以进行一次自动求导，如果我们再调用一次 backward，会发现程序报错，没有办法再做一次。这是因为 PyTorch 默认做完一次自动求导之后，计算图就被丢弃了，所以两次自动求导需要手动设置一个东西，我们通过下面的小例子来说明。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variable containing:\n",
      " 18\n",
      "[torch.FloatTensor of size 1]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "x = Variable(torch.FloatTensor([3]), requires_grad=True)\n",
    "y = x * 2 + x ** 2 + 3\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "y.backward(retain_graph=True) # 设置 retain_graph 为 True 来保留计算图"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variable containing:\n",
      " 8\n",
      "[torch.FloatTensor of size 1]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y.backward() # 再做一次自动求导，这次不保留计算图"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variable containing:\n",
      " 16\n",
      "[torch.FloatTensor of size 1]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以发现 x 的梯度变成了 16，因为这里做了两次自动求导，所以讲第一次的梯度 8 和第二次的梯度 8 加起来得到了 16 的结果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**小练习**\n",
    "\n",
    "定义\n",
    "\n",
    "$$\n",
    "x = \n",
    "\\left[\n",
    "\\begin{matrix}\n",
    "x_0 \\\\\n",
    "x_1\n",
    "\\end{matrix}\n",
    "\\right] = \n",
    "\\left[\n",
    "\\begin{matrix}\n",
    "2 \\\\\n",
    "3\n",
    "\\end{matrix}\n",
    "\\right]\n",
    "$$\n",
    "\n",
    "$$\n",
    "k = (k_0,\\ k_1) = (x_0^2 + 3 x_1,\\ 2 x_0 + x_1^2)\n",
    "$$\n",
    "\n",
    "我们希望求得\n",
    "\n",
    "$$\n",
    "j = \\left[\n",
    "\\begin{matrix}\n",
    "\\frac{\\partial k_0}{\\partial x_0} & \\frac{\\partial k_0}{\\partial x_1} \\\\\n",
    "\\frac{\\partial k_1}{\\partial x_0} & \\frac{\\partial k_1}{\\partial x_1}\n",
    "\\end{matrix}\n",
    "\\right]\n",
    "$$\n",
    "\n",
    "参考答案：\n",
    "\n",
    "$$\n",
    "\\left[\n",
    "\\begin{matrix}\n",
    "4 & 3 \\\\\n",
    "2 & 6 \\\\\n",
    "\\end{matrix}\n",
    "\\right]\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = Variable(torch.FloatTensor([2, 3]), requires_grad=True)\n",
    "k = Variable(torch.zeros(2))\n",
    "\n",
    "k[0] = x[0] ** 2 + 3 * x[1]\n",
    "k[1] = x[1] ** 2 + 2 * x[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# k.backward(torch.ones_like(k)) \n",
    "# print(x.grad)\n",
    "# 和上一个的区别在于该算法是求得导数和，并不是分布求解。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([13., 13.], grad_fn=<CopySlices>)\n"
     ]
    }
   ],
   "source": [
    "j = torch.zeros(2, 2)\n",
    "k.backward(torch.FloatTensor([1, 0]), retain_graph=True)\n",
    "print(k)\n",
    "j[0] = x.grad.data\n",
    "\n",
    "x.grad.data.zero_() # 归零之前求得的梯度\n",
    "\n",
    "k.backward(torch.FloatTensor([0, 1]))\n",
    "j[1] = x.grad.data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([13., 13.], grad_fn=<CopySlices>)\n"
     ]
    }
   ],
   "source": [
    "print(k)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " 4  3\n",
      " 2  6\n",
      "[torch.FloatTensor of size 2x2]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(j)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下一次课我们会介绍两种神经网络的编程方式，动态图编程和静态图编程"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}