machinelearning_notebook/6_pytorch/0_basic/2-autograd.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 自动求导\n",
    "这次课程我们会了解 PyTorch 中的自动求导机制，自动求导是 PyTorch 中非常重要的特性，能够让我们避免手动去计算非常复杂的导数，这能够极大地减少了我们构建模型的时间，这也是其前身 Torch 这个框架所不具备的特性，下面我们通过例子看看 PyTorch 自动求导的独特魅力以及探究自动求导的更多用法。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.autograd import Variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 简单情况的自动求导\n",
    "下面我们显示一些简单情况的自动求导，\"简单\"体现在计算的结果都是标量，也就是一个数，我们对这个标量进行自动求导。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([19.], grad_fn=<AddBackward0>)\n"
     ]
    }
   ],
   "source": [
    "x = Variable(torch.Tensor([2]), requires_grad=True)\n",
    "y = x + 2\n",
    "z = y ** 2 + 3\n",
    "print(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过上面的一些列操作，我们从 x 得到了最后的结果out，我们可以将其表示为数学公式\n",
    "\n",
    "$$\n",
    "z = (x + 2)^2 + 3\n",
    "$$\n",
    "\n",
    "那么我们从 z 对 x 求导的结果就是 \n",
    "\n",
    "$$\n",
    "\\frac{\\partial z}{\\partial x} = 2 (x + 2) = 2 (2 + 2) = 8\n",
    "$$\n",
    "如果你对求导不熟悉，可以查看以下[网址进行复习](https://baike.baidu.com/item/%E5%AF%BC%E6%95%B0#1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([8.])\n"
     ]
    }
   ],
   "source": [
    "# 使用自动求导\n",
    "z.backward()\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于上面这样一个简单的例子，我们验证了自动求导，同时可以发现发现使用自动求导非常方便。如果是一个更加复杂的例子，那么手动求导就会显得非常的麻烦，所以自动求导的机制能够帮助我们省去麻烦的数学计算，下面我们可以看一个更加复杂的例子。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 5.7436e-01, -8.5241e-01,  2.2845e+00,  3.6574e-01,  1.4336e+00,\n",
      "          6.2769e-01, -2.4378e-01,  2.3407e+00,  3.8966e-01,  1.1835e+00,\n",
      "         -6.4391e-01,  9.1353e-01, -5.8734e-01, -1.9392e+00,  9.3507e-01,\n",
      "          8.8518e-02,  7.2412e-01, -1.0687e+00, -6.7646e-01,  1.2672e+00],\n",
      "        [ 7.2998e-01,  2.0229e+00, -5.0831e-01, -6.3940e-01, -8.7033e-01,\n",
      "          2.7687e-01,  6.3498e-01, -1.8736e-03, -8.4395e-01,  1.4696e+00,\n",
      "         -1.7850e+00, -4.5297e-01,  9.2144e-01,  8.5070e-02, -5.8926e-01,\n",
      "          1.2085e+00, -9.7894e-01, -3.4309e-01, -2.4711e-02, -6.4475e-01],\n",
      "        [-2.8774e-01,  1.2039e+00, -5.2320e-01,  1.3787e-01,  3.9971e-02,\n",
      "         -5.6454e-01, -1.5835e+00, -2.0742e-01, -1.4274e+00, -3.7860e-01,\n",
      "          6.2642e-01,  1.6408e+00, -1.1916e-01,  1.4388e-01, -9.5261e-01,\n",
      "          4.0784e-01,  8.1715e-01,  3.9228e-01,  4.1611e-01, -3.3709e-01],\n",
      "        [ 3.3040e-01,  1.7915e-01, -5.7069e-02,  1.1144e+00, -1.0322e+00,\n",
      "          9.9129e-01,  1.1692e+00,  7.9638e-01, -1.0943e-01,  8.2714e-01,\n",
      "         -1.5700e-01, -5.6686e-01, -1.9550e-01, -1.2263e+00,  1.7836e+00,\n",
      "          9.1989e-01, -6.4577e-01,  9.5402e-01, -8.6525e-01,  3.9199e-01],\n",
      "        [-8.8085e-01, -6.3551e-03,  1.6959e+00, -7.5292e-02, -8.8929e-02,\n",
      "          1.0209e+00,  8.9355e-01, -1.2029e+00,  1.9429e+00, -2.7024e-01,\n",
      "         -9.1289e-01, -1.3788e+00, -6.2695e-01, -6.5776e-01,  3.3640e-01,\n",
      "         -1.0473e-01,  9.9417e-01,  1.0128e+00,  2.4199e+00,  2.8859e-01],\n",
      "        [ 8.0469e-02, -1.6585e-01, -4.9862e-01, -5.5413e-01, -4.9307e-01,\n",
      "         -7.3808e-01,  1.3946e-02,  5.6282e-01,  9.1096e-01, -1.9281e-01,\n",
      "         -3.8546e-01, -1.4070e+00,  7.3520e-01,  1.7412e+00,  1.0770e+00,\n",
      "          1.4837e+00, -7.4241e-01, -4.0977e-01,  1.1057e+00, -7.0222e-01],\n",
      "        [-2.3147e-01, -3.7781e-01,  1.0774e+00, -7.9918e-01,  1.8275e+00,\n",
      "          7.6937e-01, -2.7600e-01,  1.0389e+00,  1.4457e+00, -1.2898e+00,\n",
      "          1.2761e-03,  5.5406e-01,  1.8231e+00, -2.3874e-01,  1.2145e+00,\n",
      "         -2.1051e+00, -6.6464e-01, -8.5335e-01, -2.6258e-01,  8.0080e-01],\n",
      "        [ 4.2173e-01,  1.7040e-01, -3.0126e-01, -5.2095e-01,  5.5845e-01,\n",
      "          5.9780e-01, -6.8320e-01, -5.2203e-01,  4.9485e-01, -8.2392e-01,\n",
      "         -1.7584e-01, -1.3862e+00,  1.3604e+00, -7.5567e-01,  3.1400e-01,\n",
      "          1.8617e+00, -1.1887e+00, -3.1732e-01, -1.5062e-01, -1.7251e-01],\n",
      "        [ 1.0924e+00,  1.0899e+00,  5.7135e-01, -2.7047e-01,  1.1123e+00,\n",
      "          9.3634e-01, -1.4739e+00,  5.3640e-01, -8.2090e-02,  3.3112e-02,\n",
      "          6.6032e-01,  1.1448e+00, -4.2457e-01,  1.2898e+00,  3.9002e-01,\n",
      "          2.7646e-01,  9.6717e-03, -1.7425e-01, -1.9732e-01,  9.7876e-01],\n",
      "        [ 4.4554e-01,  5.3807e-01, -2.2031e-02,  1.3198e+00, -1.1642e+00,\n",
      "         -6.6617e-01, -2.6982e-01, -1.0219e+00,  5.8154e-01,  1.7617e+00,\n",
      "          3.3077e-01,  1.5238e+00, -5.8909e-01,  1.1373e+00,  1.0998e+00,\n",
      "         -1.8168e+00, -5.0699e-01,  4.0043e-01, -2.3226e+00,  7.2522e-02]],\n",
      "       requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "# FIXME: the demo need improve\n",
    "x = Variable(torch.randn(10, 20), requires_grad=True)\n",
    "y = Variable(torch.randn(10, 5), requires_grad=True)\n",
    "w = Variable(torch.randn(20, 5), requires_grad=True)\n",
    "print(x)\n",
    "out = torch.mean(y - torch.matmul(x, w)) # torch.matmul 是做矩阵乘法\n",
    "out.backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果你对矩阵乘法不熟悉，可以查看下面的[网址进行复习](https://baike.baidu.com/item/%E7%9F%A9%E9%98%B5%E4%B9%98%E6%B3%95/5446029?fr=aladdin)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 0.0034, -0.0301, -0.0040, -0.0488,  0.0187, -0.0139, -0.0374,  0.0102,\n",
      "          0.0337, -0.0249, -0.0777, -0.0868,  0.0132,  0.0042, -0.0627, -0.0448,\n",
      "          0.0221, -0.0324, -0.0601,  0.0048],\n",
      "        [ 0.0034, -0.0301, -0.0040, -0.0488,  0.0187, -0.0139, -0.0374,  0.0102,\n",
      "          0.0337, -0.0249, -0.0777, -0.0868,  0.0132,  0.0042, -0.0627, -0.0448,\n",
      "          0.0221, -0.0324, -0.0601,  0.0048],\n",
      "        [ 0.0034, -0.0301, -0.0040, -0.0488,  0.0187, -0.0139, -0.0374,  0.0102,\n",
      "          0.0337, -0.0249, -0.0777, -0.0868,  0.0132,  0.0042, -0.0627, -0.0448,\n",
      "          0.0221, -0.0324, -0.0601,  0.0048],\n",
      "        [ 0.0034, -0.0301, -0.0040, -0.0488,  0.0187, -0.0139, -0.0374,  0.0102,\n",
      "          0.0337, -0.0249, -0.0777, -0.0868,  0.0132,  0.0042, -0.0627, -0.0448,\n",
      "          0.0221, -0.0324, -0.0601,  0.0048],\n",
      "        [ 0.0034, -0.0301, -0.0040, -0.0488,  0.0187, -0.0139, -0.0374,  0.0102,\n",
      "          0.0337, -0.0249, -0.0777, -0.0868,  0.0132,  0.0042, -0.0627, -0.0448,\n",
      "          0.0221, -0.0324, -0.0601,  0.0048],\n",
      "        [ 0.0034, -0.0301, -0.0040, -0.0488,  0.0187, -0.0139, -0.0374,  0.0102,\n",
      "          0.0337, -0.0249, -0.0777, -0.0868,  0.0132,  0.0042, -0.0627, -0.0448,\n",
      "          0.0221, -0.0324, -0.0601,  0.0048],\n",
      "        [ 0.0034, -0.0301, -0.0040, -0.0488,  0.0187, -0.0139, -0.0374,  0.0102,\n",
      "          0.0337, -0.0249, -0.0777, -0.0868,  0.0132,  0.0042, -0.0627, -0.0448,\n",
      "          0.0221, -0.0324, -0.0601,  0.0048],\n",
      "        [ 0.0034, -0.0301, -0.0040, -0.0488,  0.0187, -0.0139, -0.0374,  0.0102,\n",
      "          0.0337, -0.0249, -0.0777, -0.0868,  0.0132,  0.0042, -0.0627, -0.0448,\n",
      "          0.0221, -0.0324, -0.0601,  0.0048],\n",
      "        [ 0.0034, -0.0301, -0.0040, -0.0488,  0.0187, -0.0139, -0.0374,  0.0102,\n",
      "          0.0337, -0.0249, -0.0777, -0.0868,  0.0132,  0.0042, -0.0627, -0.0448,\n",
      "          0.0221, -0.0324, -0.0601,  0.0048],\n",
      "        [ 0.0034, -0.0301, -0.0040, -0.0488,  0.0187, -0.0139, -0.0374,  0.0102,\n",
      "          0.0337, -0.0249, -0.0777, -0.0868,  0.0132,  0.0042, -0.0627, -0.0448,\n",
      "          0.0221, -0.0324, -0.0601,  0.0048]])\n"
     ]
    }
   ],
   "source": [
    "# 得到 x 的梯度\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.0200, 0.0200, 0.0200, 0.0200, 0.0200],\n",
      "        [0.0200, 0.0200, 0.0200, 0.0200, 0.0200],\n",
      "        [0.0200, 0.0200, 0.0200, 0.0200, 0.0200],\n",
      "        [0.0200, 0.0200, 0.0200, 0.0200, 0.0200],\n",
      "        [0.0200, 0.0200, 0.0200, 0.0200, 0.0200],\n",
      "        [0.0200, 0.0200, 0.0200, 0.0200, 0.0200],\n",
      "        [0.0200, 0.0200, 0.0200, 0.0200, 0.0200],\n",
      "        [0.0200, 0.0200, 0.0200, 0.0200, 0.0200],\n",
      "        [0.0200, 0.0200, 0.0200, 0.0200, 0.0200],\n",
      "        [0.0200, 0.0200, 0.0200, 0.0200, 0.0200]])\n"
     ]
    }
   ],
   "source": [
    "# 得到 y 的的梯度\n",
    "print(y.grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 0.0172,  0.0172,  0.0172,  0.0172,  0.0172],\n",
      "        [ 0.0389,  0.0389,  0.0389,  0.0389,  0.0389],\n",
      "        [-0.0748, -0.0748, -0.0748, -0.0748, -0.0748],\n",
      "        [-0.0186, -0.0186, -0.0186, -0.0186, -0.0186],\n",
      "        [ 0.0278,  0.0278,  0.0278,  0.0278,  0.0278],\n",
      "        [-0.0228, -0.0228, -0.0228, -0.0228, -0.0228],\n",
      "        [-0.0496, -0.0496, -0.0496, -0.0496, -0.0496],\n",
      "        [-0.0084, -0.0084, -0.0084, -0.0084, -0.0084],\n",
      "        [ 0.0693,  0.0693,  0.0693,  0.0693,  0.0693],\n",
      "        [-0.0821, -0.0821, -0.0821, -0.0821, -0.0821],\n",
      "        [ 0.0419,  0.0419,  0.0419,  0.0419,  0.0419],\n",
      "        [-0.0126, -0.0126, -0.0126, -0.0126, -0.0126],\n",
      "        [ 0.0322,  0.0322,  0.0322,  0.0322,  0.0322],\n",
      "        [ 0.0863,  0.0863,  0.0863,  0.0863,  0.0863],\n",
      "        [-0.0791, -0.0791, -0.0791, -0.0791, -0.0791],\n",
      "        [ 0.0179,  0.0179,  0.0179,  0.0179,  0.0179],\n",
      "        [-0.1109, -0.1109, -0.1109, -0.1109, -0.1109],\n",
      "        [-0.0188, -0.0188, -0.0188, -0.0188, -0.0188],\n",
      "        [-0.0636, -0.0636, -0.0636, -0.0636, -0.0636],\n",
      "        [ 0.0223,  0.0223,  0.0223,  0.0223,  0.0223]])\n"
     ]
    }
   ],
   "source": [
    "# 得到 w 的梯度\n",
    "print(w.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面数学公式就更加复杂，矩阵乘法之后对两个矩阵对应元素相乘，然后所有元素求平均，有兴趣的同学可以手动去计算一下梯度，使用 PyTorch 的自动求导，我们能够非常容易得到 x, y 和 w 的导数，因为深度学习中充满大量的矩阵运算，所以我们没有办法手动去求这些导数，有了自动求导能够非常方便地解决网络更新的问题。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 复杂情况的自动求导\n",
    "上面我们展示了简单情况下的自动求导，都是对标量进行自动求导，可能你会有一个疑问，如何对一个向量或者矩阵自动求导了呢？感兴趣的同学可以自己先去尝试一下，下面我们会介绍对多维数组的自动求导机制。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[2., 3.]], requires_grad=True)\n",
      "tensor([[0., 0.]])\n"
     ]
    }
   ],
   "source": [
    "m = Variable(torch.FloatTensor([[2, 3]]), requires_grad=True) # 构建一个 1 x 2 的矩阵\n",
    "n = Variable(torch.zeros(1, 2)) # 构建一个相同大小的 0 矩阵\n",
    "print(m)\n",
    "print(n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(2., grad_fn=<SelectBackward>)\n",
      "tensor([[ 4., 27.]], grad_fn=<CopySlices>)\n"
     ]
    }
   ],
   "source": [
    "# 通过 m 中的值计算新的 n 中的值\n",
    "print(m[0,0])\n",
    "n[0, 0] = m[0, 0] ** 2\n",
    "n[0, 1] = m[0, 1] ** 3\n",
    "print(n)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将上面的式子写成数学公式，可以得到 \n",
    "$$\n",
    "n = (n_0,\\ n_1) = (m_0^2,\\ m_1^3) = (2^2,\\ 3^3) \n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面我们直接对 n 进行反向传播，也就是求 n 对 m 的导数。\n",
    "\n",
    "这时我们需要明确这个导数的定义，即如何定义\n",
    "\n",
    "$$\n",
    "\\frac{\\partial n}{\\partial m} = \\frac{\\partial (n_0,\\ n_1)}{\\partial (m_0,\\ m_1)}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在 PyTorch 中，如果要调用自动求导，需要往`backward()`中传入一个参数，这个参数的形状和 n 一样大，比如是 $(w_0,\\ w_1)$，那么自动求导的结果就是：\n",
    "$$\n",
    "\\frac{\\partial n}{\\partial m_0} = w_0 \\frac{\\partial n_0}{\\partial m_0} + w_1 \\frac{\\partial n_1}{\\partial m_0}\n",
    "$$\n",
    "$$\n",
    "\\frac{\\partial n}{\\partial m_1} = w_0 \\frac{\\partial n_0}{\\partial m_1} + w_1 \\frac{\\partial n_1}{\\partial m_1}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "n.backward(torch.ones_like(n)) # 将 (w0, w1) 取成 (1, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 4., 27.]])\n"
     ]
    }
   ],
   "source": [
    "print(m.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过自动求导我们得到了梯度是 4 和 27，我们可以验算一下\n",
    "$$\n",
    "\\frac{\\partial n}{\\partial m_0} = w_0 \\frac{\\partial n_0}{\\partial m_0} + w_1 \\frac{\\partial n_1}{\\partial m_0} = 2 m_0 + 0 = 2 \\times 2 = 4\n",
    "$$\n",
    "$$\n",
    "\\frac{\\partial n}{\\partial m_1} = w_0 \\frac{\\partial n_0}{\\partial m_1} + w_1 \\frac{\\partial n_1}{\\partial m_1} = 0 + 3 m_1^2 = 3 \\times 3^2 = 27\n",
    "$$\n",
    "通过验算我们可以得到相同的结果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 多次自动求导\n",
    "通过调用 backward 我们可以进行一次自动求导，如果我们再调用一次 backward，会发现程序报错，没有办法再做一次。这是因为 PyTorch 默认做完一次自动求导之后，计算图就被丢弃了，所以两次自动求导需要手动设置一个东西，我们通过下面的小例子来说明。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([18.], grad_fn=<AddBackward0>)\n"
     ]
    }
   ],
   "source": [
    "x = Variable(torch.FloatTensor([3]), requires_grad=True)\n",
    "y = x * 2 + x ** 2 + 3\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "y.backward(retain_graph=True) # 设置 retain_graph 为 True 来保留计算图"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([8.])\n"
     ]
    }
   ],
   "source": [
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "y.backward() # 再做一次自动求导，这次不保留计算图"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([16.])\n"
     ]
    }
   ],
   "source": [
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以发现 x 的梯度变成了 16，因为这里做了两次自动求导，所以讲第一次的梯度 8 和第二次的梯度 8 加起来得到了 16 的结果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**小练习**\n",
    "\n",
    "定义\n",
    "\n",
    "$$\n",
    "x = \n",
    "\\left[\n",
    "\\begin{matrix}\n",
    "x_0 \\\\\n",
    "x_1\n",
    "\\end{matrix}\n",
    "\\right] = \n",
    "\\left[\n",
    "\\begin{matrix}\n",
    "2 \\\\\n",
    "3\n",
    "\\end{matrix}\n",
    "\\right]\n",
    "$$\n",
    "\n",
    "$$\n",
    "k = (k_0,\\ k_1) = (x_0^2 + 3 x_1,\\ 2 x_0 + x_1^2)\n",
    "$$\n",
    "\n",
    "我们希望求得\n",
    "\n",
    "$$\n",
    "j = \\left[\n",
    "\\begin{matrix}\n",
    "\\frac{\\partial k_0}{\\partial x_0} & \\frac{\\partial k_0}{\\partial x_1} \\\\\n",
    "\\frac{\\partial k_1}{\\partial x_0} & \\frac{\\partial k_1}{\\partial x_1}\n",
    "\\end{matrix}\n",
    "\\right]\n",
    "$$\n",
    "\n",
    "参考答案：\n",
    "\n",
    "$$\n",
    "\\left[\n",
    "\\begin{matrix}\n",
    "4 & 3 \\\\\n",
    "2 & 6 \\\\\n",
    "\\end{matrix}\n",
    "\\right]\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = Variable(torch.FloatTensor([2, 3]), requires_grad=True)\n",
    "k = Variable(torch.zeros(2))\n",
    "\n",
    "k[0] = x[0] ** 2 + 3 * x[1]\n",
    "k[1] = x[1] ** 2 + 2 * x[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "#k.backward(torch.ones_like(k)) \n",
    "#print(x.grad)\n",
    "# 和上一个的区别在于该算法是求得导数和，并不是分布求解。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([13., 13.], grad_fn=<CopySlices>)\n",
      "tensor([4., 3.])\n",
      "tensor([2., 6.])\n"
     ]
    }
   ],
   "source": [
    "j = torch.zeros(2, 2)\n",
    "k.backward(torch.FloatTensor([1, 0]), retain_graph=True)\n",
    "print(k)\n",
    "j[0] = x.grad.data\n",
    "print(x.grad.data)\n",
    "\n",
    "x.grad.data.zero_() # 归零之前求得的梯度\n",
    "\n",
    "k.backward(torch.FloatTensor([0, 1]))\n",
    "j[1] = x.grad.data\n",
    "print(x.grad.data)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([13., 13.], grad_fn=<CopySlices>)\n"
     ]
    }
   ],
   "source": [
    "print(k)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[4., 3.],\n",
      "        [2., 6.]])\n"
     ]
    }
   ],
   "source": [
    "print(j)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下一次课我们会介绍两种神经网络的编程方式，动态图编程和静态图编程"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}