OpenI
/
machinelearning_notebook

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adagrad\n",
    "这个优化算法被称为自适应学习率优化算法，之前我们讲的随机梯度下降以及动量法对所有的参数都使用的固定的学习率进行参数更新，但是不同的参数梯度可能不一样，所以需要不同的学习率才能比较好的进行训练，但是这个事情又不能很好地被人为操作，所以 Adagrad 便能够帮助我们做这件事。\n",
    "\n",
    "## Adagrad 算法\n",
    "Adagrad 的想法非常简答，在每次使用一个 batch size 的数据进行参数更新的时候，我们需要计算所有参数的梯度，那么其想法就是对于每个参数，初始化一个变量 s 为 0，然后每次将该参数的梯度平方求和累加到这个变量 s 上，然后在更新这个参数的时候，学习率就变为\n",
    "\n",
    "$$\n",
    "\\frac{\\eta}{\\sqrt{s + \\epsilon}}\n",
    "$$\n",
    "\n",
    "这里的 $\\epsilon$ 是为了数值稳定性而加上的，因为有可能 s 的值为 0，那么 0 出现在分母就会出现无穷大的情况，通常 $\\epsilon$ 取 $10^{-10}$，这样不同的参数由于梯度不同，他们对应的 s 大小也就不同，所以上面的公式得到的学习率也就不同，这也就实现了自适应的学习率。\n",
    "\n",
    "FIXME: need improve. ref: https://ruder.io/optimizing-gradient-descent/index.html#adagrad\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adagrad 的核心想法就是，如果一个参数的梯度一直都非常大，那么其对应的学习率就变小一点，防止震荡，而一个参数的梯度一直都非常小，那么这个参数的学习率就变大一点，使得其能够更快地更新\n",
    "\n",
    "Adagrad 也有一些问题，因为 s 不断累加梯度的平方，所以会越来越大，导致学习率在后期会变得较小，导致收敛乏力的情况，可能无法收敛到表较好的结果，当然后面有一个对其的改进，我们之后会讲到\n",
    "\n",
    "下面我们来实现一下 Adagrad 的算法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sgd_adagrad(parameters, sqrs, lr):\n",
    "    eps = 1e-10\n",
    "    for param, sqr in zip(parameters, sqrs):\n",
    "        sqr[:] = sqr + param.grad.data ** 2\n",
    "        div = lr / torch.sqrt(sqr + eps) * param.grad.data\n",
    "        param.data = param.data - div"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "from torchvision.datasets import MNIST # 导入 pytorch 内置的 mnist 数据\n",
    "from torch.utils.data import DataLoader\n",
    "from torch import nn\n",
    "from torch.autograd import Variable\n",
    "import time\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "def data_tf(x):\n",
    "    x = np.array(x, dtype='float32') / 255\n",
    "    x = (x - 0.5) / 0.5 # 标准化，这个技巧之后会讲到\n",
    "    x = x.reshape((-1,)) # 拉平\n",
    "    x = torch.from_numpy(x)\n",
    "    return x\n",
    "\n",
    "train_set = MNIST('../../../data/mnist', train=True,  transform=data_tf, download=True) # 载入数据集，申明定义的数据变换\n",
    "test_set  = MNIST('../../../data/mnist', train=False, transform=data_tf, download=True)\n",
    "\n",
    "# 定义 loss 函数\n",
    "criterion = nn.CrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/bushuhui/.virtualenv/dl/lib/python3.5/site-packages/ipykernel_launcher.py:31: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number\n",
      "/home/bushuhui/.virtualenv/dl/lib/python3.5/site-packages/ipykernel_launcher.py:33: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 0, Train Loss: 0.404529\n",
      "epoch: 1, Train Loss: 0.243532\n",
      "epoch: 2, Train Loss: 0.201834\n",
      "epoch: 3, Train Loss: 0.176955\n",
      "epoch: 4, Train Loss: 0.159980\n",
      "使用时间: 31.24649 s\n"
     ]
    }
   ],
   "source": [
    "train_data = DataLoader(train_set, batch_size=64, shuffle=True)\n",
    "# 使用 Sequential 定义 3 层神经网络\n",
    "net = nn.Sequential(\n",
    "    nn.Linear(784, 200),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(200, 10),\n",
    ")\n",
    "\n",
    "# 初始化梯度平方项\n",
    "sqrs = []\n",
    "for param in net.parameters():\n",
    "    sqrs.append(torch.zeros_like(param.data))\n",
    "    \n",
    "# 开始训练\n",
    "losses = []\n",
    "idx = 0\n",
    "start = time.time() # 记时开始\n",
    "for e in range(5):\n",
    "    train_loss = 0\n",
    "    for im, label in train_data:\n",
    "        im = Variable(im)\n",
    "        label = Variable(label)\n",
    "        # 前向传播\n",
    "        out = net(im)\n",
    "        loss = criterion(out, label)\n",
    "        # 反向传播\n",
    "        net.zero_grad()\n",
    "        loss.backward()\n",
    "        sgd_adagrad(net.parameters(), sqrs, 1e-2) # 学习率设为 0.01\n",
    "        # 记录误差\n",
    "        train_loss += loss.data[0]\n",
    "        if idx % 30 == 0:\n",
    "            losses.append(loss.data[0])\n",
    "        idx += 1\n",
    "    print('epoch: {}, Train Loss: {:.6f}'\n",
    "          .format(e, train_loss / len(train_data)))\n",
    "end = time.time() # 计时结束\n",
    "print('使用时间: {:.5f} s'.format(end - start))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x1059a1630>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztvXmYXFd95/09dbfaq/dudbekltxaLMmLbHnDZjM2OMEL\nIQEDIUwSAgMBkgzMmwdm5g2GN7x5w2TlnWwm7BAT2+AJZrHBGBtsZGPJtrxIsqxdLbXUe1XXvp35\n495z6tatW9XVa1VX/z7P48fq6uqqc2s53/PbGeccBEEQxNrD0+gFEARBEI2BBIAgCGKNQgJAEASx\nRiEBIAiCWKOQABAEQaxRSAAIgiDWKCQABEEQaxQSAIIgiDUKCQBBEMQaRW30AmrR1dXFh4aGGr0M\ngiCIVcP+/fsnOOfd9dy3qQVgaGgI+/bta/QyCIIgVg2MsVP13pdcQARBEGsUEgCCIIg1CgkAQRDE\nGqWpYwAEQawdcrkcRkZGkE6nG72UVYHX68Xg4CA0TVvwY5AAEATRFIyMjCAUCmFoaAiMsUYvp6nh\nnGNychIjIyPYtGnTgh+HXEAEQTQF6XQanZ2dtPnXAWMMnZ2di7aWSAAIgmgaaPOvn6V4rVpSAL76\n5Ak8eOBco5dBEATR1LSkAHzz6dP40UujjV4GQRAtxle/+lV89KMfXdHnfOyxx3Drrbcuy2O3pAB4\nNQ8yuWKjl0EQBOEK5xzFYuP3qJYUAENVkM4XGr0MgiBWGW9729tw5ZVXYufOnbj77rsBAF/5ylew\ndetWXH311XjyySflfR988EFcc8012L17N2666SZcuHABADA+Po6bb74ZO3fuxB/8wR9g48aNmJiY\nwMmTJ7Ft2za8733vw65du3DmzBl8+MMfxp49e7Bz5058+tOflo/90EMPYfv27bjiiivw3e9+d9mu\ntyXTQA2VLACCWM185sGXcfBcbEkfc0d/GJ++bWfN+3z5y19GR0cHUqkUrrrqKrz1rW/Fpz/9aezf\nvx+RSARvfOMbsXv3bgDADTfcgKeeegqMMfzrv/4rPv/5z+Ov//qv8ZnPfAY33ngjPvWpT+Ghhx7C\nl770Jfn4r776Kr72ta/h2muvBQB87nOfQ0dHBwqFAt70pjfhhRdewNatW/GBD3wAjz76KIaHh3Hn\nnXcu6etgp2UFYDadb/QyCIJYZXzhC1/AAw88AAA4c+YMvvGNb+ANb3gDurvN5pp33nknjhw5AsCs\nW7jzzjsxOjqKbDYr8/GfeOIJ+Ri33HIL2tvb5eNv3LhRbv4AcO+99+Luu+9GPp/H6OgoDh48iGKx\niE2bNmHLli0AgPe+973SGllqWlIAvJqCDLmACGLVMtdJfTl47LHH8Mgjj2Dv3r3w+/14wxvegO3b\nt+PgwYOu9//Yxz6Gj3/847j99tvx2GOP4a677przOQKBgPz3iRMn8Fd/9Vd45pln0N7ejt/93d9d\n8SropowBMMZuY4zdHY1GF/T3hupBmlxABEHMg2g0ivb2dvj9fhw+fBhPPfUUUqkUHn/8cUxOTiKX\ny+G+++4ru//AwAAA4Gtf+5q8/frrr8e9994LAPjxj3+M6elp1+eLxWIIBAKIRCK4cOECfvSjHwEA\ntm/fjpMnT+LYsWMAgHvuuWdZrhdoUgHgnD/IOf9gJBJZ0N8bKlkABEHMj1tuuQX5fB4XX3wxPvnJ\nT+Laa6/FunXrcNddd+G6667D9ddfj4svvlje/6677sI73vEOXHnllejq6pK3f/rTn8aPf/xj7Nq1\nC/fddx/6+voQCoUqnu+yyy7D7t27sX37drznPe/B9ddfD8Ds8XP33XfjrW99K6644gr09PQs2zUz\nzvmyPfhi2bNnD1/IQJg/+4+X8L0D5/D8n715GVZFEMRycOjQobINdrWSyWSgKApUVcXevXvx4Q9/\nGM8///yyPJfba8YY288531PP37duDIBcQARBNIDTp0/jne98J4rFInRdxxe/+MVGL6kqLSkAhupB\nOl8A55x6ixAEsaJs2bIFzz33XKOXURdNGQNYLIbqAedArtC87i2CICppZpd0s7EUr1WLCoACABQI\nJohVhNfrxeTkJIlAHYh5AF6vd1GP05IuIK9m6lomX0Rl7J0giGZkcHAQIyMjGB8fb/RSVgViIthi\naEkBKFkAFAgmiNWCpmmLmm5FzJ/WdAFZFkA6Ry4ggiCIarSmAKiWC4hSQQmCIKrSmgKgURCYIAhi\nLlpTANRSEJggCIJwp0UFwLQAKAZAEARRnRYVALIACIIg5qIlBcCrURooQRDEXLSkAJSygMgFRBAE\nUY3WFABRB0AWAEEQRFVaUwBEJTBZAARBEFVpSQGw9wIiCIIg3GlJAdAVEgCCIIi5aEkBYIzBUD3k\nAiIIgqhBSwoAYGYCkQVAEARRnZYVAK+mUC8ggiCIGrSsABiah7qBEgRB1KB1BUBVkCYLgCAIoiot\nLABkARAEQdRixUZCMsYCAP4RQBbAY5zzby3n85kxABIAgiCIaizKAmCMfZkxNsYYe8lx+y2MsVcY\nY0cZY5+0bn47gPs55x8AcPtinrcezCwgcgERBEFUY7EuoK8CuMV+A2NMAfAPAH4NwA4A72aM7QAw\nCOCMdbdl35kN1YM0uYAIgiCqsigB4Jz/HMCU4+arARzlnB/nnGcBfBvAHQBGYIrAop+3HgyV0kAJ\ngiBqsRwb8QBKJ33A3PgHAHwXwG8yxv4JwIPV/pgx9kHG2D7G2L7x8fEFL8KrUSEYQRBELVYsCMw5\nTwD4vTrudzeAuwFgz549fKHPZ6gKZQERBEHUYDksgLMA1tt+HrRuW1EMzUN1AARBEDVYDgF4BsAW\nxtgmxpgO4F0AvrcMz1MTqgMgCIKozWLTQO8BsBfANsbYCGPs/ZzzPICPAngYwCEA93LOX178UueH\n6AXE+YK9SARBEC3NomIAnPN3V7n9hwB+uNDHZYzdBuC24eHhhT4EDNWDIgfyRQ5NYQt+HIIgiFal\nKVtBcM4f5Jx/MBKJLPgx5FhIygQiCIJwpSkFYCmQg+FpKAxBEIQrLSsAXrIACIIgatKyAiAsABoL\nSRAE4U5TCgBj7DbG2N3RaHTBj2GoNBieIAiiFk0pAEsZBKYYAEEQhDtNKQBLgXQBkQVAEAThSusK\nAAWBCYIgatLCAkBBYIIgiFq0rAB4RR0AWQAEQRCuNKUALE0WkOUCIguAIAjClaYUgCXJAqIgMEEQ\nRE2aUgCWAgoCEwRB1KaFBYB6AREEQdSi5QWALICFc2w8jqlEttHLIAhimWhZAWCMmVPBaCzkgvm9\nrzyDv3/kSKOXQRDEMtGyAgDQWMjFMpPMYoIsAIJoWZpSAJYiDRQADE1BKksWwEJJ54tIZPKNXgZB\nEMtEUwrAUqSBAsCmrgBeHq1PRM5H09jz5z/BS2cXJzqtAuccWRIAgmhpmlIAlorrNnfi5XMxRJO5\nOe/73OlpTMSzeOX87AqsrPkRwfN4hiwogmhVWlsALuoE58CvTk7Ned9Xx+IAgNn03GKxFhCxE7IA\nCKJ1aWkB2L2hDYbqwd5jk3PetyQAtOEBkNlTJAAE0bq0tAAYqoIrN7Zj7/E6BOCC6fqJkQUAAEgL\nCyBLAkAQrUpLCwBgxgEOjcYwXSOdMV8o4vhEAsDSWQCT8QxGppNL8liNQFgA6VwR+QKl0hJEK9Ly\nAnDtRZ0AgKdPVLcCzkynkLWCnsIC4Jzj8PnYgp/3j779HP7ga/sW/PeNJm2rn0hQKi1BtCRNKQBL\nVQcAAJcNtsGnKXjqePVAsHD/6KpHWgC/OjGFW/7uFzh4zhSBU5MJ/F/3HUCujtPw2ZkUnjw6iWPj\n8VV7erZXUFMcgCBak6YUgKWqAwDMTf3SwQieOzNT9T4iAHzJQAQxSwDOzqQAAOes/z/2yjju2z+C\nU5OJOZ/zfz93FgCQK3D5OKsNew8lEgCCaE2aUgCWmss3tOHguWjVzqBHx+Loj3ixLuLFbMp0AYkm\naFPJbNnP0VTtIDHnHA88dxYhrwoAODExt2A0I/bXKk4CQBAtyZoQgN3r25ErcLx8LoZkNo+b/uZx\n/OjFUfn7V8dmMdwbQsirSQtgxioeE8HjaUsIZuYoKnvpbAxHx+L4wGs3AwBO1ikAt/zdz/HNp07N\n78KWkXILgGIABNGKrA0B2NAGwKz2ffTwGI6OxfH4kXEAQLHIcXQsji09QYS9qiwEEyf/6WS5RTCX\nAHzn2RHoigf/6bohBA21Lgsgmy/i8PlZPHt6emEXuAyQBUAQrY/a6AWsBL1hLwbafHj+zAwKpzgA\n4LDV8uHMdBLpXBHDPUFMJbLI5IvI5AuYEQLgsABquYBm0zncv38Ev3ZJHyJ+DUNdfpyYnDsVVIjO\nuSaKF1AMgCBanzVhAQBmHODpE1N49PAYPAw4cmEWxSLHCyNmptElAxHpt59N5zGdKLcEpqyfZywB\nyOaLOB9Nlz3HfftGEM/k8fvXbwIADHUG6nIBicyjczPpOe65tOw7OYXHXhlz/V3GZgHMVQw2NpsG\n53xJ1zYXvzw2gWPj8RV9ToJoNdaMAOxe34bx2Qwy+SLetnsAyWwBZ2dSOHBmBobqwba+EMJeDQAQ\nS+VsPv9ySyBq/fxvT5/CTX/zuHSVFIocX9t7EldubMdl602X0+auAEamk7LGoBqi9mA0mkKxuHIb\n6d//9FV87geHXH+Xtq25lgtobDaN1/zFo/jpIXchWS4+ce8B/K9Hj67ocxLEcvDLYxPY/dkfN6QL\nwdoRgA3tAIC+sBe/fc0GAKYb6MDIDHb2h6EpnnILwJb9wzmXloCwAE5OJhHP5DFqWQE/OzyGU5NJ\n/N71Q/I5h7oCKHLg9FRtN1AsZW6wuQLHRCKzRFc8N9PJLCbi7s8nmsF5WG0X0GQ8i3yR45ULK9dF\nlXOOiXgG47Mr91oRxHJxaHQW08kcLkRX1gMArCEB2NkfRtBQccfufmzrCwMADp6L4cWzUXliDwkL\nIJ2TLqCZZA7JbEGe4kUMYNzaOEctv/0jhy6gza/hLTv75HMOdQUAzJ0JZO9AupJuoOlEDtPJnGtx\nWzpfgK54EDDUmllAKcsCGo2uXPwinsmbYllFvAhiNSG8Co1ItmjKIDBj7DYAtw0PDy/ZY3o1BQ//\nl9ehK6jDUBUMtvvw4AvnkM4VcdmgKQBhn/lynI+mkS0UYaieilOyyAKasE6fotDr9FQSm7sC0JSS\npm7qtARgjuKxWJkApHC5JUjLjd291RP2lv0ukzOvP2ioNT+YaatNxOgKCteMIzOLIFYzwquQbEDL\nlaa0AJayEtjOQJsPhqoAALb3hXDUqgB2WgDCZbPJcuGctDJ5/LoiLQAhCuLEfnoqifUd/rLnaw/o\naPNrstFcNYQLyHy8lTlJZ/NF2eNn3OUknckXYGjCAqguAMICOLeC5qss0rPccwSxmhF7SiMsgKYU\ngJVgW18IABD2qhjqNDduEQMQArC52zzBH7eyTTZ1BWwCYG5Co9EU8oUiRqNprG8vFwDAzAQ6MT63\nBcCYKTAr1TpiJlU6PYtrsZPOFWGoCgJzWACNcAGJeEy+yMvEkyBWI8KibUS69ZoVgK29pgBctr4N\njDEAQFBXwRhwyjrxb+4KAoBMNxzqCmAmmUUmX5BCcHYmhdFoGoUix4aOSgHY3hfCwdFYzZPqbDqP\nkKFioM1XtytlscPu7WMyJ1yCqcICCBpKbQvAWocZK1mZD7C9tfdKBs0JYjkQLiASgBVkuxUIFv5/\nAPB4GIKGKi2Ai3pMC+DYmHmC32y5hE7ZirvOzaTk/Qc7fBXPs2sggmgqh5Hp6ifkWCqHsE/DujYf\nztVxkn7l/Cx23fUwXj638G6p0zYBmHTZRKUFoNcOAtsrhucTwH7opfN457/sXVDaq933P+livRDE\naiImBIBiACvHlp4g/ujGYdx51fqy28NeTW4wwgI4PhGH4mHSxXPMih1s6PDj3EwaZywBcHMBXTpo\nxjFePFt9s46lcwh5NQy0eXFuJoVCkePvH3kVp6tUEZ+eSqJQ5Pj5kYn5XHIZIgAMuLuAMvkCvNrc\nQeCUTQDm4wa6f/8IfnViSp5+5sO0be1TZAEQqxzxXSQLYAXxeBg+/uZtFYFbEQdgzPTfA8CFWAbt\nfjOgC0AGjy8ZjCCVK+DFs1EoHoZ1kfJMGsCMNWgKkxXHbsTSeYS9KvojPkzEs/jBi6P420eO4LvP\njbjeX7ha9tUx7L4awu+oepi7C8jKAgoYas1K4FS2lEJar/sqXyjiaWtM50Jy+acSOSge023nJl4E\nsVooFjkFgZsJUQ0c8WkI+1RoirnRdAQ0RHyWAFgxgcus0/1Txycx0OaDqlS+nIaqYFtfCC/VsgAs\nF1B/m+lC+ssfHQYAnJlyP1ELl8z+09MLrhwWQeChrkDVLCCvptSVBSQ241ruq2PjcfzX+w4glS3g\npXMxzFqPuZBc/ulEVsZbKBWUWM3MZvIQX+FkA7rukgA4ELUA7X4djDG0+XX5s/i3sAAuteIHx8YT\nWO/i/xdcMhDBi2ejVQPBs+k8Ql4V69pMC0JkAp2pMlNYbMgzyRyOTyysH850MgdNYdjY4a/iAhJ1\nAApyBV42IcxOOldAQFfQFTRqWgCPHhrD/ftHcP+zI/jlsZLraiECMJXMoidkIOLTMFnl718+F12x\noDRBLJSYzQUab8DnlQTAgagFaLfcPR3Wpt8RKLmAjo8nENAVXNQdlH/n5v8XiEBwtRN9LJ1D2Kth\nwLIAQoaKt+zslbEFJ3aXzDMnF9ZCeiaZQ8SnoytouG6i6VxBpoEC1WcCpLIF+HQF/W3emhbA+Zgp\nDl/6xXE88eoE+i132UJcQNOJLNr9OjoDOiZcLIDD52N46xeewJ3/8lRVgWgm5uoVRbQu9vbyFANo\nAkQMoN3a+MWm3x7QpQsolSugK2SgM6BDt9w+zliCnUsHTEvhhbOVYymLRY54Jo+wT0NfxIugoeJ3\nrtuIHesiOB9Lu04xS2bNAG1nQMe+BQtAFu1+DV0hHZOJbIUrKZMvwmsVggHVP5ypXAE+TcG6iFf2\nRXLjfDQNxsyiul8em8Sbd/ZBVzwL8uFPJ7NoD+joDOqYcvn7r/3yJHTVgyMXZvGOf9nb1CIwGc/g\nkrsexpNHFx7QX2k459h7bJKK8JYA4YrVFQ8JQDMgYgDtgdLJHzAtAa+mwFDNl6wraMDjYdJtU0sA\ntvYFoSnMNRNoNpMH52ZBmqEq+OknXo9PvHkb1nf4wDlcC8MSmTyChoorNrZj/6mFBYKnk1m0+TV0\nBgwUirwiG0dYAEFLAKoFqFI5M1awLuLD6Eyq6qYwGk3hqqEODLabVs71w13oCurztgA455hO5tAR\n0NAR0CtSWKPJHB547ix+4/IBfPF9e3B8PIGfHLwwr+dYScbjZofa5+oYBtQsG+5PDl7Au7/4FF46\nG2v0UiRjsTSu//8exZEVbEq4FIgA8Lo2b0Mm75EAOChZAOVCIP4vLIKuoPlzf8Tc0Na3V48BGKqC\n7X1h10CwaAQnhKc37IXiYTLI6eYGSmYL8Osq9mxsx8nJ5ILqAWaSObT5dXSFDACVvvh6LYB0ruQC\nSmQLcqSmkwuxDAbbffjoG4cR8Wm4ZnMHukLGvGMAsXQehSI3XUBBoyIIfO++M0jninjfazbKSXCz\nVdbUDIiuqyfnGBz08yPjuPyzPykr4GsU+06ZYmWvJm80xycSODuTqktIF8I//OwovrEMI1uFC6g/\n4ptz7sZy0JQCwBi7jTF2dzS68EKnhRK23Dyl4K/5c6cQAJ/5/66guXGKzJ1aFgBgxgFeHKkMBItW\nBiL4LKglAPFMHn5dwa9fsg6dAR1v/8df4t+ePl3nFZrMJHNo82lSyJypoGYQWEHQUORzupHKCheQ\n+Tq41QIUixwXYmn0hb1419Ub8Oz/fTPCXg1dwfkLgKgC7giYMYApm/uqWOT4xlOncNVQO3b2RxDQ\n3a2XmWQWn7j3QEP6rzvJWp1YT83RMPDYeBzRVA4HRxt/6hab7GKr0ZcScUA5uwxNCYtFjn9+/Bi+\nsffkkj+2sAAG2n3kAhIsVzO4ehAWgHD9iFiAsABEHEAIwBUb2zDcE5QCUY1LByOIpfMVswHEJiSC\nz4LukAFD9bjOEkhmTRfQ+g4/fvQnr8VVQx34bw+8KLOT6mEmZfrRu63rsAdTc4UiCkUOQ/XAb22i\n1ToVihhAv+UKc2tmN5HIIF/ksk5CpI12BfV5C4DoA9RuCUCRl0rpf3lsEqenknjvtRsBmLUeAV2p\nEICnT0zhO8+O4JkTC6+jWCqEBXBiorYFIAruGj0FLZsvypqWlEt8qlGI93h0GXppHRuPYzadx7Hx\nhGtMbjHMJLPwah50BHSqA2gGKrKAAuWWQES4gCzXyW9fsxGPfPz1sp9QNS4ZMMXMWRAm3BNhhwAw\nxrC+w+8qAIlMAX7LNdMT8uKu23cAQN3mbzpXQDpXRMSnSSGzWwBiHrBXqzMGoCuyavrVC5UblBid\n2Rcpd5N1hwxMxisD0LWQFoBfR4e1dhHkvW//GYS9atlMhqC3so5BPEat9hxunJ5MygK2pSJbMDeU\niXimdsW1JcDzEfnl4PD5mPx8NJMFIA4otRIRapHOFaq61561vleFIl/yGEM0lUObT4dfV5DOmQev\nlYQEwMHF60K4dDCCS6wc/zdt78UfvWkLdqwzewe1WRZAd7D2id/J1t4QdMVTEQcQecBOFxBguoHc\nUkeT2TwCuiJ/3tQVREBXalYb2xF+x3a/mdmkeFjZSVyccox6YgCWC6g9oKM3bOCV85VfEPGl7HPM\nHOgKGsi7BKBrMWVzAXVZ4jwRzyKayuGhl87j9sv74dVKr03AUGXRmXyMpBCA2qduJ1949FX8zpd/\nVbX47P79I/j+C+fm9ZjCAgBqu4HEBtdoC+DZU6VDhrAAjo7N4gNf39dQQRCfz3p6abnxtz85gt/6\n51+6/m7/qWlZEPryuaV1wZmxOE0etFY6DkAC4KAn5MX3PnqDzMmP+DV8/OatssrX6QKqF131YPu6\nUMUmXc0FBAgBSFbEDRKZgnTNAKZLZddABC/UqDa2I3rptPs1eDzMzKePu1gAqoKAFQOoKgD5InzW\nhrutL4zDLgJwISYsgEoBAGoXg33r6VP48hMnKtce0NFhifBUIovvv3AOmXwR79xT3tsp5FLJXK8F\n8N1nR/Cb//RL+fqfj6aRzRdxz6/c4y3/9NhRfPzfD8zrlJgt2AWguiCVNtvGCsBzZ2akNSzW9PSJ\nKfzk4IVlC8DWg7CeztXIRKvFy+diVT8Pz56ewQ3DXQgZKg4utQCkcoj4tDkPWssFCcA8KWUBzU8A\nADMQ/NK58kCwCAKL2IOdwXYfZjP5smIRwDwliOCs4LL1bTg0GqurqEg8XsR2LfZ0TLsFYKgKNIUh\nPkchGFAaspN3jJgcjaahKawiTuLmfnLywLNn8cBzZ+XPU4mcOapSV9AZMP/+9FQS33rqNLb1hqSr\nTRAwVMQdWUBT1rjPuQTg63tPYf+paZnZJF6jb+w95TpGM5rKIVso4hP3HnD9vRt2C+BEjcFBKZuL\noxG+YsGzp6dxzaZOMFaaBifWVqvh4XIjNs50rljxfamH01NJpHKFis/uTDKLo2Nx7BnqwMXrwovq\nwOtGjARgdXHlxg7s3tAm8//nw6UDEcym82Unvdl0Dn5dKRslKRCZQM44QNIWAxBcMhBBNl+s6/Q5\nIy0Ac0O+dDCCXx6blAFcsSmJmgezI2jll4pzLusAAGBbbwjZQrFiI7sQTaMn5IXHUx4n6Q6Zz+/W\ni0gwm86XtXSYTmTRHtDAGJMn0c8/fBiHzsfwn1+/uSIW49bNdLoOF9D5aBrPnzEL94QFMzabxvoO\nH87H0nj45fMVr8VMMoftfSG8eDaKrzx5ouIx3chYG46msJouILt75VgNK+Bnh8fKLKalZHw2gzNT\nKVyxsQ0+TZFuKZG/3lABsL0+83UD5QtF+dl35uI/Z30Gdm9ow45+08JdSj+9cAEJl+5K1wKQAMyT\n6y7qxAN/eL0cLTkfdolAsO2LItpAuLHBmlR2yiYA2XwR2UKxLAYAlOYa1BMHED53Yc185I3D4Nz0\ngwKQfX8Ma2MPeitP0eb9zM1LuIC2rzOH7DjdQKPRtGun1JILqHo++Ww6V5aBNJXMSuFSFQ+Ge4K4\nqDuI+z90Hd5+xWDF37sJgPDhTydzVU/TPz5Y2uAvxEzXz3Qyh7fvHsTGTj++6cgJT2QLyBc5fmP3\nAIZ7gnVXaAuLbXNXsGYtQDJXkFZirTjAvfvO4J8eP1bXc8+X01OmQG3tDcGnKdIFlMyZr2GthofL\njf3kPN/51KPRNPLWpj7rOOg8d2oaiofhskFTAJLZwpwzvufDTCqLNr9OFsBaYGtvCLpaHgiOpfKu\n7h/AHEHp1TzlgTdrM7THAABgfYcPbX4NL4xUtptwMu2wANZ3+PG+6zbiO8+O4JXzs0g7LICQoblu\nlGItPs2833BPEIqHVQSCz8fS6HURgIhPg6awmtXApgVQEoDpRFZmZgHA9z92Ax7+k9fhyo0drn8f\n9LpbALp1bWeruIEefvm8fF8uxDKy4rg37MX1w10V2U7CqjKrq/W63RBCbLf0BuewAPLY3heC6mE1\n4wDRVK6sNmIpyebNxzRUBV67AFin1pOTyYbVViQyeXmgmK8FYG+66PysHBiJYmtvCAFDxc5+MxFk\nqeIA9my8ubLtlgsSgBVEVz3Yvb4NP3xxVPqIZzM5WXzmxFAVXLOpE794dVzeJjoGBhwxAMYYLhmI\n1GcBJHMwVE9ZtsxH3jiMgKHinx8/Jjclr80CcKumFRuAiAEYqoJNXYEyC4BzjvPRNNaFKwWAMVaz\nGKxY5Ihny11AM6mctFzEGhVP9RTcoBUEtsddphJZXGxldZ2dqTx1TyeyeOr4FN5xpRlQvhBLYyxm\nrrEnZGBd2IvJRLasQ6qMq/jMpoH1VskKC2BrbwgXYpmqHUxTuQLCXg0bO/01BSCWzqFQ5EuyEecK\nRXzyOy/IYsR8seSu8umKjBXZBXohVsDRsdlFZzfFM3ls7PRDU9i8JtMB5cWWTkt3OplFb9gUli09\n5myPpcqljKO4AAAgAElEQVQEEhmAZTEAygJqbT7w2s0YmU7hP5430wVjKXMYTDVeu6ULx8YT0keZ\nzAgBqPybywbb8MqF2TmLVWZsbhRBe0DHFRvacXQs7mIB1BYAu5Bs6wvhlQulL0gslUcqV6jIABLU\nEoB41uyTlCtwuVEmMnlZ4VsPAUO12lmbf58rFDGbzuNSyx3nFgh+9PAYCkWOt+3uR9irYiyWllZK\nd8iQ1owQBfM6S261Nl91CyCTL+Dqzz2CH744av1chKYwbOoyhw9VywRKZs16i+GeoJxH4YZIKlhI\nk71ikeP3vvIr2Zju1GQC337mDPYeM2sfxKFFVTzw64q0AJPZvKwoX4gA/LfvvoS7vvfyvP/OTiJj\nusj6It55TaYDyuduOFOGZ9N5eTrXVQ+29obqsrLrwe6KpRjAGuFNF/dgx7ow/vFnR+VJzS0FVHDD\nli4AwBOvml9KEexy2wS39YVQKPKa2SSAGczrdKlj6G8zvzwyBmDFOUIubhTA7gIqCcDFfSGcmUrJ\n+5+vkgIqqFUNbD+NieeKZ/Ku4lcN4cYRvlXh/traG4ShelwF4MDIDIKGiksGIuiLeHE+lpaB6u6Q\nIesZxLUB5V9m0wLIuaYjRlM5jM1mcNzaxLNWyw0hANVOwulsAX7NFIBTk8mq2V6itYCzVuF7B87h\nR5boVGM2k8fPXhmXxW4i+ylXFOJpXo/qYeUuoGwBA20+DLT58OICGsRNxDNlYz4XQiJrfi7MpoTz\nswBOTyUhcgecFoA5q6P0/dyzsR3Pn5mpyBZaCOKQ0OajGMCagTGGj904jOMTCbzl736OU5NJ181Y\nsK03hO6QgV9YpzJhAfj1yiC02EROziEAZlC2snndOmskpTjtey3fvpsfHSili/p0uwVgulZENpI4\njbkFgQHLAph1//LbrY5E1nTjiE6o9eLsBzRtpYB2BAwMtPtcM4FeOT+LLb1BMMbQG/biQiwjT/td\nQUOKmb3q1P5lbvPryOaL0pKyI/zl4neZfAG66sGW3iBUT3X3QtJqujfcE0ShyF3jBcUil80FnbOS\n/+mxY/j4vQdkVbYbQmTFxi9e/7y18Yv/a4rHDALbLACfrmBnv3vDw7mYSeVckwxqkckX8Iff2i/j\nTYlMHkFdRX/E69pBtxanp5Lyu+P8nMczubIY3ZVDHUhmCzg0uviKYBE3ivg0+HUFjJEArAnesrMP\neza2w8OAP71lG/74TVuq3pcxhhuGu/Dk0Qk5OwBwdwENWR/iE3NkKVTLyhG3CQERFkDQ0Fy/oDIG\nYLMARE+gMet0LF0nQXcB6LY6groFLWdtfuxk1gyYFbn7tVcjaH15xWYmTsbtAQ2D7f4KC4Bzs9x/\nW6+Z0dQT8pouoHga7X4NuuqRAnDBLgCp0pdZxCjc4gDCxyvEM2tNXjNUBVt7q48OFfUWw93mutws\nhUS2NF7Q6QKaTeeQyhXw+YcOuz6+fW0ifiBef+H6ETEAVWHlWUDZAgK6aTGdmEiUvW9zwbk5E3e+\nwc+TE0n88MXzeMI6GCUyBdMCaPPhQiw9r1TNkemkjAnZP+e5gini9gPHno3tAIB9C2zDbidqsxoZ\nYwjoatV6m+WCBKABeDwM93/4Nfjxf3k9/vANw7LzaDVuGO7CVCKLg6MxGXBzswCChoquoFHTAkhm\n84imcq51DKKzqXAhCQsg5FWRLRQrYgviBGiPAYjirElroxX/r2blrIt4kS9yTCQq3UB2CyCZzctN\nwlkEV4ug4e4C6gjoGGz3VQjAeDyD6WQOWy0B6A0bGJvN4EIsg26r/1PIUOHXlTIXUDSZg6564NU8\nsl2IWxxA+HjF5pnJF2VG0iUDEbzkMjq0UDRjGD5NweZuU+TdAsFRW0sNpwsolsrBq3nw3efOVq3Y\nFdbJrNMCsDZT4QLSPB749HIB8OkKtvaZr9nxcffP3337zuC/P/Bi2W3xjNne2y3G9Lc/OYKfHnKf\n5SBe++lE1rQMs3kEDAX94vNUZ5PBRCaPiXgWF1trt8cAEvLzVhKA/jYf+iNe2RJ7MYj3SxRk+nVl\nxceYkgCsAvYMmaeOg+di8pRWzQ2yqcuPkzU6S4oMiX4XF5AUgMlyC0CYwM5TmjMLCDBP1gDkpK6p\nRNbqKuq+aYsGcW6uCXsmSyJTkF/IeVkAjvS6KVszucF2H6YS2TKz+8h5c2PdZm0IfdaG8sr5WSkA\njDH0hb1laxbttRlj8gvt5tcuWQDmaTqbL8qpcrsGwphO5nDO8VqI19mvmyM6+yNeVwEQAWAAZVPQ\nikWO2Uwe77l6IzoCOr7y5EnX10psPiKgHZcuoGLZ/1UrCyiVLcq/C+iqHJFaLY7x2CvjuH//SJnA\niU0wky9WxDW+/OQJ/Nl/vOzqbxfW11Qyi1SuAG5ZhsK16daV1g1xANjQGTBrRmxCJEQp6EjSuHKo\nA/tPTi96QM9MMgcPA4KWm9KtZmW5IQFYBQiXw7loSp7SnJXAgqHOQE0XUC2fvLjt9KQZFBMNsOQm\n6jilpV1cQIaqIGSoJQsgnkVnQK/aLVUEVN26ONpPhalcvqb7qxpBrzMGIPL1dQxac5ztPmMRu9hq\ncwEBpp9Y/Bsw6wHKg8BZ6foRMyPcukvKGEDe5gKyLK2dVmaS0w3kDLZfVCUTyG4BTNosgISVTbUu\n4sX2vlBVH7mwLksWgPl4Wevkn7MsAeECSjssgI2dfqgeVlUAZlJZZPLFMveUfc32zY9zjlS2gLMz\nKTzkqLoGyi0A++dCWJpCfF86G8VrP/8ovr73pKubUVTZb+jwW8kOlesJOT5veza243wsPe9Yg5Oo\n1QZCVMgHXPpWLTckAKsAQ1XQFTQwOpOWJ0j7pmtnqCuA8dnqrYXFRusWBPZqCjoCOvJFDq+qyE27\nWpGKWxYQYKaUii/gVCIjm7a5IcTNzQIoCwLbLID5BIErLIBkFiFDha56MGC5wZwC0BHQZVqjyAEH\nIC0AwNxMKy0AMTtCxABcXEDW+5exu4AsC+DivjA8rIYAWCfF4Z4gjo0lKjY0YTH5NKXMBSSCumGf\nip6QgbFZ90CwWJuoho1VsQA0j8dqBWEG5pPZAgKG2c5kQ6e/qgtIbPb2wLtdJO0HjGyhKF1PX/zF\niYrTthCAyURWutWChiKbNYrnOjAygzNTKfzZf7yMO+/eKwOvAlEDsL7dV3ECF58/Z5belVYcYP8i\n3UBmTUvpuxEwFEoDBRo7EaxZ6W/zYjSWRiKTh69G8dNcmUAiRa434t7MTlgB4lQKVAZSBalcaW6A\nnQ5rUhdgulw6AtUb53UGdGgKq2IBlDaHVLYgN6gFuYDSJQtADPeRU8xsaYOvXJjFVisDCDBP+oJu\nWwPA3ogXF2JpuQlHUznp+hFC4BYDEFlcqbIgsPn6+XQFW3oqA8F2FxBgCkAqV8BorPw1E66boa4A\nJm2nbNly3KvJrCY394WwAIQryRkDEFlAwgVU5CUfvqhM39wVrG4BWK+HXXDtFoC9DYMQvYu6Azhw\nZqbC5z5mswDEwcCvqyUBsJ5LPOdfvP0SHDgTxfu/Vt62emQ6BZ918HEWPAprwOkCunhdGAFdwXOn\nF1cPMJPMlhWBBnSVCsGAxk4Ea1bWRbwYnUkhkS3U3ACHOi0BqOIGGo2m0BU0qvYyEpui1/Z70avI\nmd0hNiZRMCboDOhyA5qwXEDV8HhEqqW7BSCELpHNywwJZx+kWjjT66aSOSkAPSEDHlZyi3HOceR8\nKQMIME/9wntltwD6wmZsQLhaRAwAMIPnuuqpkgXkngYq2DkQxkuOVNCkw+obtnztzjiA2Ew3dwXK\nXEBCAEJeDd0hA9l8sSxeIJ/Heo3iVuW0MwsoJyuBS1Xk4n0W4nRRTwAnJ5KufnuxKdsD73YrKV4W\n9Ddfp/deuxGG6sHDL5W7gaQLKJm1JQeockONpkrxDEP14N1Xb8AX3n05njs9jY/d86wUwIl4Bj1h\nA4yxqhaA0+JUPAxdIWPRtQvmMBibAJALiKjGuogPo9E0kpl8RRsIOxutBnInJxI4Ph7HV588URZc\nOxdNy1RNN8TvyiyAKi6gdK4Ar+ap6PJZaQHUznJaV6V6czadk6fuZHZhQWDGGIJ6aSjMdCKLDuuk\nrioe9IVLeeNnLYEV2SyAudmJzKYemwAIy0AIlz0GwBhDm0+TG95HvvWsrPwttS0uuYDsArqrP4Lx\n2Yw84QKVwfbhHncBiKXzYMz8DEwnS/2AZm0uILluFzeQEKdC0XTrVKsDUD1MipEQGikA3UFkC8WK\n7Kp8oSjfA3v/pWoxACF6nUEDvWFvRVbP+aj583SyVEMQMFRoVqtw8bii2yYA3LJrHT7yxmE8cmhM\nxiEm4hnZQyjkaHooYwAulfr2bqhzkc4V8OffP1gWmBfXbm9rEjAoDZSowrqIF/FMHudj6YpGcHYC\nhunnPTaewMfueQ53PXgQ7/niU3KjGp1JVS3KMp/HtADsm5IzkCoQA+GddARNAUhlC0jlCnMKQF/E\nVzUG0BnUoXgYktn8ggRA3F9aADYXEACsaytVjjoDwAIRB3DGAAAzpiKaetn9uW1+DTPJHGLpHH7w\n4qhsr5CUFkDJBWS3ALZb4nPE1mzOGWvpCJj9hioEIJWTqcCFIpeboIgNhL2aFDF7Gwvn84i/ES4Z\nkf8vTvWKh8kNX2xq4jNZLRMoZttY7TEAu5ts1sUC8GsKOoN6mUWTKxQxmcgg5FVRKHLpChPpwRGf\nJq9dBFoFYrKfiIOYAqBbf19uAcSrWADm9Sp1T0Dbd3Ia//rECfzIYcXYrUaxfrIACFfWWSmaR8cS\nc7pAhroC+P4L5/DyuRjec80GHByN4R3/vBeFotWYzSUALBAWgN2vL74AlTEAdwHoDOjIFooyw6Jr\njvGZfWEDo9F0hV/aLMM3c+4TmUIp22MeLiCgvJJ5OplFh22jtlsfr1gpoFt7nAJgviZlLqBIqR2E\nvamXoM1nBsJFLEZsgPbBJUB5HQBgnnjFOgXO2g/GGIa7gzg2FsfYbFpWBYvhIiITRnQwLY0d1Sos\nFzt2//NsOi/f75wtC0hTGBhj8vMxVWEBmC5IZyBYBF8ZK3cBRVM56eYrz8EvXbM5sa70eozNZsC5\nGTQHgBHrcyYOBmGbAMyksjImAwA9lpiPWQWKE/GstACcBY/xjGlRuaUw+2v469O5Au74hyfxlNVS\n49Ux82BxcLTk2itabWAiDhdQKldY0bnAJACrhH5rw5mIZ6qmgAo2dQaQK3Bct7kTn3vbLvzlb16K\n01NJPHLoAmYz+ZoWgKgFsFsAXk2BrnjkhsA5Lw2DcflyiKCv+ODXCgIDpgWQyVdOcoqlcwgamjxt\nJTJ5eDWPHM9ZL8K0TucKSGYLZRbAQJsP5yzxOXJhFn1hrwzmCnrDXuiqp+zL2hU0oHgYLkTTFfMV\nxL+jqZzs8S82YacFkLEFgQHIITf2bBW3pnvDPUHsPz2Na//fn+K2//8JuaGEvVqpGM/aNEvZLGrF\nBmgnaXM/zKZLrpWcLQtI9ZivvXBHlVxAqnXdOjoDeoUFIDbkTV0BnLWNbYymsjIVOO5I+wXMdOfO\ngFHmPhHidbE1f0K0cxZriPg0+XpHU/myQKtI5R2PZZAvFDGdtAmAV0U8my9znQUN1TWF2VfDArgQ\nS+PAmRk5NOhVy1I7ZBOA2bSZmhuxHUaqpVsvJyQAqwRhAQBzV8Lu6A9DUxg+c8dOMMZw08W98OsK\nvvjz4xWPVfE8kUoLABCn6BzSuQKu+twj+N/Pn5UD4Z2IoK/omV9PDAAw3Sn//sxpfPbBgwDML0nY\nq8rsiHimMK8UUEHIUBFP58qqgO3Pnc0XMZnI4siF2TL/v+D3rx/C//ytS8s2AsXD0BMyLRd7HyCB\ncAEJC0BsgJWtIAplYivcSHYxTDksAAB4885e7FgXxmsu6kIsnceF2TRiqTwiPk1enzidx9I5+DQz\nTdOvqwgZqqsFkLRVesdS+YoYQK7AoVq1IeJ9n5AuoNLaLuquzAQSIrmzP4JktoDpZMlF0xs2oHpY\nWQ6+3erptFyKYmMWRWCifYOwKIRlWOYCSmbLhFtYcWOzaUwlsuAc6LJVeHNuS4dN56sOazKrdkuv\n12g0Jd9T8d69bDXGe9VyLb5yflZeg0gQsLuAtlsWzTMnF99mol5IAFYJImMFqBwG4+Q912zAL/70\nRunL9ukKbt7RK1Pp+mtYAL1hLxirzOwRVZJnZ1KYiGfxiyMTSOfdBUCcsIWPulYWEGB3p6TwL48f\nx7/96pTsexTyqvAbJQtgvv5/ufZMXp6I220ndWHxjEyn8OpYHNt6gxV/v6U3hDsuH6i43SwGS5U1\n9RK0+XXMpOwuIMsCkIVgRXDOK4LAumoGMaftAuBScX3j9l48+LEb8KHXXwTA7I0TTeUQ9qnS5TYh\nBCCVR9hXet26w4brEJ6kZWEB5sYuBtbbewGJ0aU+RxaQPTHhop4AjjlcQCIgLoaqiECwGaTVK6bO\nJe0uoKCBvG3GwflYuQCcmUqWWYZhRwzAOT8i7FUxNpspdXgVMQBHrCueyVU9cNgFgHOOW/7uF7LC\nWjz3wdEYikWOIxfiCBoqktmCnO5Xmh9RWtvVmzoQMlQ8UqX9xXJAArBK0BSPPL3M5QPXFE9F++Vb\nL+2X/67Wmln8bW/IK8dBCsQmKkrsXzoXLRsIb0daAMIFNEcMQFgATx6dxPGJBNK5Is7H0pYAaPBr\npgUgWg7MFzMIXJBBXtE0DygJwN5jk8jmixUB4Fps7Q3ihZGo3EjsG03EpyGdK8rhOCLtUmwuhSI3\n5xwUymMA5uPoZS4gsdF4XVJ3RdbX6amEdAEJARbtOGYz5S3He0PuabeJbF66Y+xDVex1AKp1ChHv\nu7AyfLb3ZWNnAFO2Cl2g5NISAiACwSJIGzTUshiAyALy6zZBs67nfCwNXfHgIisbajpZvlFHfBpi\n6RxyhSIS2ULZJgsAPWEvxmIZ+XilGEC5CyaeyVfUAAh8mopUtlTTEU3lpCtKWDvxTB77T08jmsrh\nLTv7AJTcQG5uQ1314HVbu/HTw2PLMtHNDRKAVYQI3s4VA3DjdVu7EPKqYKy8uMmNP3/bLnzwtZvL\nbgt5VcTSeXlyOzoWx3QyV+EqAkoulhMTCWgKqyild9IdNK2b+/adkbcdGo2hUOTSAkhmzSDwglxA\nXhWz6RyePzODgFVsJRDi89grYwBKPYDq4ZZdfZhN5/H9A2aKZ8QRAwDMwjLAZgFky4OMnFdaW2Ke\ngCCVNYv/nOm2Yv2awsxxjNZmqikehL2qLQhcPnSox2pw5ySVLaAnLKqjS5k6sg6gwEsWgF7uArIf\nSjodAgSU8vJ3yElsKet2mwCUtf+2B4FFTMN8rgvRNHrCBgK6IsUz4BCAZLYgrZM2R0xHVENPzJZa\nfAO2gkdhAaSrf978uoJkrgDOSx16hchFbeL9wHNnAQBvvbQPiofJcZJRFwEAzHkh47MZvLhC85VJ\nAFYRIkNnvlkwgNlO4o7L+zHcHZRf4mrctKMXl61vK7tN5EgLC6DIzQ3ezQXk1xUYqge5AkdHjT5A\nAtWybmLpvLRyxGjLkFeT5rbZ8nf+1x4wFCSyBTx/ZgaXDEbKqqg7AjoM1YP9p6bBWCnHvh5uGO5G\nyKti7/FJKJ5yoRMT1wpFjja/Zs0HKMiNDSgFhp0WQLtfL8sCSuXcLS3AfO0G2/04NhZHIluQAc+u\noCEDtLF0+dhRUXjnzLpKZAvoCupQPeVjFWUdQLFYEQOYlBaATQAcWUiA6fMOGSo6AjpChoqR6ZTs\nAhrxaQh7tYoBQB7LFVl6vJIFYLoqmczosrtFxYlfZKFVWAAhUwCFeInHDzksgNkaFoDfUMC5GcQX\nGUtTtqJAwKyX+MEL5uFgV38EF3UHpAUgRMI5DvaN23rgYajaBXWpIQFYRQgLYCF+cAD4s1t34oGP\nXL+gvxUuoJGZVNmJ1U0AGGPyFDhXBpBAdAV991XroaseeQIy00BVJDP5RcQANBSKHC+djeLy9e0V\na+1v8yFf5NjQ4Z8zvmJHVz148w7TtI9YnUAF9uDepYOmmMZSOSQzeXnqE6dAZ1V2xK+V9chJVgm2\nCzZ2+mX7CHHS7wjoJReQY6pVT8hAJl8sy80HzBiAX1cR8qpS6HXFY8sCsrmAbGmgqofJfkZASfzs\n/YiiSbNVBmNMDuKxj9F0Dh1KZgvw62YGjhQAa8Mei2Wkq0q4u+yJEWLDF+mxri4gSwAM1SNP+RUx\ngHS+qvXqt67fXqAohg3NpHII6Aq29YXMuIxXRXfIwMXrwiUXkEsMQFzPlRvb8cihMdfnXWpIAFYR\nwl2xED84YG5YC3GhAOZJXMQALhmIyA9utZOp8PvPFQAWiKHxb97Zh40dfpsFoCJgmdvxec4DFojN\nociByx2WDVB6Xbf01O/+Ebz1UlMA2hxfZLs76LJBs6XJTCqHZK4gX5NoVQtAK7MA0jUsAADY2OGX\nLaTF83aHDOnnj1mbkEC4ecYccYBkroCAriDk1aSLpj2gyS6guUIpCCwOAYUih09XysTPORNCXKsQ\nvsF2H05NJqWbS7iAnJXAIrNInPIn4mbvf2EBAECH1XjP6QICSo3e3CyAbL6I4+MJdAWNyqaHthiA\nWxUwULI47HMqxHsmAtu7+s33fWtvCIwxXLwujHPRNGaSWURTOctSrnxfb9zei1fHZitmOiwHJACr\nCBGw9C/ADbJYgpYf/exMCgPtPuwaMH25bjEAoHTynysFVHDFxjbs7A9jZ38YQ10BaZ6HvBp8uopk\nZhFZQLYvsZsAiNd1W1/97h+BcAM5awfsVcE7rY1gLGYWMIkNUgqAUukCiqZyMhBonoZrWQCloLZI\nW9zQ6ceZ6aScO12eC+9eC5DMFOA3VIR9KjJW+5B2v17qBlospYF6bO0gnKIsxN++gc3YKnJ3DURw\ndDwuA8HCAnBWAotrVhUP2v0aJhMZRFM5JLMF9FnNDIW1Yf9ciGsVGTfOgUvCzXhoNCZTQAEgZFg9\nrzJ55AtFJLMFBA33NFAhyCm7BZDMyglnEZ+GndZ3ZIuVWSbiHwfPxcxOoD73x37P1Ruw73/cXPd3\nZzGQAKwitlj+6VqVvMtF0FCRK3CcnU6hv80nTzfVXBOi3069H+IPvu4i/OCPXgvGmOxoCsCqA1CQ\ntTI65jMNrLR2cy19Ya9rBpRIi51PBpBAVz34779+MX7n2o1lt4svd3/EK4uvzlkVx+I1EYFhe98l\nwDyxFnmpgCuVLVQVWqCUCQSUNj9RDHh8PI5cgZedZN2qgXOFIrKFIvyaIjdCQAhAyQIQhWBAaRN0\nipMIzpYJQLJUkXv1UAc4Bx49PCavN2SojhGg+TJ3XGfQwGQ8K7OqtljvlXgtg/OKAYj5GmmZAgqU\nUlnj6XypxXRVC8C8byJbqlDPFcyAcNTqCyWEf9iyLHdZ8x5eOBvFTDJX4f+X6/drFWteLhbmDyAa\nwpbeEJ761JtqpnEuF2IDKXKzejYsXUDuZwhhAczVBsKNIduJNuhVy9wfC7EAxBfb7fQPAIMd5gYq\n8srny7uu3lBxm19XoCkMQ10BeSoXPYeET7uWBQCYJ8qIX5uzn5LdAhAbh0h1Fa60sLe2BSALrwy1\nTCza/JpsHJcvcDkkCCiJv9MiFTGgScfgF2ElXb6hDaqH4ScHL1hr1hE0VDkVTFc9FVZPZ8DsByR8\n6Dut90q8Ln6XGMBpqwo77NjEe2wzHrpsLb5Va9h9PFPqg1QtBiA+k2aPqlJgfyaZw0wyh+GeIC5f\n34aP37wVd1zeL9e6vsOHF0eiUiQaDQnAKqMRmz9Q3hFxoM0nN5hqJrLY5OoNAtsZ6iqdaENerWzT\nX4gAiBPt5RvcBeD2y/rRFdQXZAFUgzGGoc4ALhksxUvOx0wLQPT7kUHgioE6pZGSQwggmS1goK26\nBbC+wwfGAM5LG70QURFML+s7b6gIOqqBRXqqiAGIf3s1pTwLyGYBiKIxv1b5nnTYhgIJt4iwivy6\nip0DERw4Y/bTj/g0edJOZPLQVR3JbKHsM9cVNHD4fAwHz8XQGdClG0daAGUuIPPfkwkz88jZOsTe\n08kuAECpb5RsMT1HDMDuAgJMt9eMFe9QPAx/9KYtZX936UAbDozMwK8r2Nw1f5fjUkMuIKIu7Bv9\nQLsPm7oC+JffuRK3XrbO9f4dMgto/haAcAF5mLkJ2U+CCwlib+kN4vbL+nHrpe5r9WoKbtzeO+/H\nnYvv/uFr8Imbt8mNTAy9EUFgURzmtAAiYqCMJRDVCu4EhqrIGc9CbHrDBnyaghdGzE3WGczsCpY3\nWBOnWJ+uyA005NWgelhZHYBqtwB0dwsAMN93EQROZgvIFXh51etQKRsr4tOk6IiN11n0JzqCHjof\nw8XrwjJw6xYDMFRFipMzNgOYp3rxe6eFGrLqEWp1AgVKLqCkzQUEmBPnosmcfA+dXDoYwch0Cmem\nUivm5qkFCQBRF/YvggiavmVnX9VeKaKPv/20VS+9IS+8mkc24rL7ghdiAXg1BV949245A3ilCHk1\n6Ko5PMVQPdIFJGMAKfcYgLMhXCpXOwgMmDNtNYXJjY0xho2dfrxsFR4536fOoIEpW56+6DcU0FW5\nGYe85uk5Z7MA7DUk4uTvtrbOgC4f363qdc9Qh/xbe3aaiHtUuoAMzCRzOHI+jh39JVedeC2dnwux\nubptsowxGQfoCrlbAKIYrFoWkHB/OS2AczMpZAvFqu4dkRKcyhWawgVEAkDUhfgiiJS9uXj9tm78\n7Z2XYXcVv3stPB7TfWJ3RQgWUgjWDIR9mmw7XXcMIGGzAGoEgQEzfrEu4itLxxzqDMhsnoiv/D3r\ncPjoRQM0v6FIn3nIq0JTmG0eQKkOAIDsBOtWO9ERMGQdQlTmvJdOxXusubrCLRRy5OAns4Uyy0K8\nZtlCUXYBFdcBVPrqxcZfbZMVcZAKF5DV80pYANXTQG0xgGxepsWK3k/VMnwuGYzICXNu1slKQwJA\n1J4dQ7UAABBsSURBVIX4IvTX6CRqR1M8+I3dg67tC+rh8vVtMhbgW6QLqBkw+9NYU64caaBehwUQ\n9mlgzDw5F4vcqgSufd2fePNW3Pufryu7zd7zyDnYvDNQPmTF3nsn5LW7gDxlWUB2C8AnYgBuFkBQ\nRyJrtuCWnS9tG15n0MBF3QEZmyhZAKWWGXZhsbtqdqwrjYrd3hfCp2/bgZt2lLvwalkAQCkQ7CYA\n9lkI1WJcsg4gV0A8U0BfxAsPM6vjaz1v0FDl0JxmcAGtzm8TseKIL+hAnQKwWD57xy5wmBvPYoPA\nzYA9E8WZBqor5Ruo4mEIezXMJLPyBD+XBRAw1IrXZpMtmO50AXUEdEwnzLx1xpjMAgroirxv0LIA\nci51APY1uVsApVqAaJWq1z+9Zbu8PnsVbqHIkc4Vy65ZBM51xYPN3SVhY4zh967fVPH8JQFw98UL\nF1C3QwA2dQfw6OExnJiIl63LiVfzgLGSCyjkVdHu10sCUON0f+lgBEfH4mXtwxsFWQBEXYgvwkCN\necJLia56ZJWk/YS50CroRmPPwhGulWqVwIAZB5hJ5mwn8/m7vkR6qD02IJAtlq1AtGy/bJRiAGGv\nClVhjm6gc9cBAA4BqNL47C07+3D7ZWaKZMgWAxDtr+3uPhE439I7dy8roPR6Vztl3zDchddt7S5r\nkw0A79yzHvkix78/c8acBlZFeBljci6wqFBv82uy9qDW5n6ZFQegGACxajBUBb/7miHceln/3Hde\nYsqDwKs0BmBtqqqHwbACwzII7CIAEashnDiZz2UBuCGyqUJeraIhn9hQJ6xArYwBaEqZC0hTPCgU\nOYpFbrmA7BZA7SAwYKZiyiBwjU3RbgEI0fM5CsGAUjXtXMwVA7hpRy++/vtXV7wuF3UHcd3mTsTS\neQR1taYL06+rshdQ0Gp0JwLmtTb3m3b04rVbuuq+luWEBICom7tu34mrrOyNlaQ1LIDSZilm6oqW\n77UsgLTLMJh66QmZqaDOQiig/IQO2AvBbAJgqPK0nSsWK11AuogBVD6+nEmQyGAmmYOueCqsEDs+\nTYHiYYin89IasQf/w14Vv35JH26r8wAyVwygFr99rVnYV839IzBHlZaaFLbbWk7UEoCBNh++8f5r\nykaTNorV+W0i1hSG6oGHmemcCw0qNxqxEQk/vX0zdLMA2v3mXN3FWAAiFdRNYIQAiEygZDYvu3p2\nBg1oCkNvxItpSyDyBV7ZCkL0AnKxyuRMgEQOB0dj2NDpr9kWnDFmBWBzZeMg7b//x9++su5rlxbA\nAgTgzTv60BXU50w48OuK1QqigIChytdDU9iC3q9GQAJAND2MMQR01XUA/WpBuIDEpiame3kYXIfc\nR3waZhI56Q9fSAwAAP7gtZtRsNI47XQ6GrYlMgXZ1TPi0/DQn7wOGzr8+PreUwBMAXC2ghD9idw2\nu7DXrIQdnUnh6eOTeM81le0ynIipYGIg/FyZT7VYjAWgqx78P3fsKivwckMMhjddQAoUSxwjvrln\nYDQLJADEqsBvKPPq1d9shB0WgHDpuLUDBkwLYDZTSkdciAsIAH7rykHX20sWgBkDSGULZe41kaoo\nNnzTBVQsEyuxJrfMLI+Hod2v4+GD55HJF/G6Ld1zrjXsM+cgJFxcQPNluCcIXfFgoy0Vdj782iXu\nVeN2/LqCeMYMWgcMVYp0MwR362XFYgCMsc2MsS8xxu5fqeckWge/rq7aADBQsgDEJissADf3DFDq\nB/T4EbNj5kIFoBqGqiBkqLIWIJHNu7Z0EC4f0wXEodlccGLDq7a2zoCOM1Mp6IoH12yeO3a0vt2H\n01PJkttrEdd86WAbDn72LcuatuzTVNm2PGiosu30QtxOjaIuAWCMfZkxNsYYe8lx+y2MsVcYY0cZ\nY5+s9Ric8+Oc8/cvZrHE2sWvK6s2AAzYYwDWyd+KAVQTALGZfPOp07h2c4c8kS8lHUG9LAjs5mYS\nFoAIRtstgIE2P1QPk9O5Kh7fsjL2DLXXZb1t6g7g1GRSul4W+367udaWEr+uYNzqqBowVDm4ZjVZ\nAPW+wl8F8L8AfF3cwBhTAPwDgJsBjAB4hjH2PQAKgL9w/P3vc85XZsYZ0ZK86+oNqyaw5kYpC0gE\ngYULyH2TumJDG64aasedV23A23cPLEvw22zYZm5gzspbgcgCSkkBKK3j6k0d2Pc/bqoYuCIf34oz\nvLYO9w9gzjDIFoo4OmYWYS007rFS+HVFFrIFDFVm9VQrPmtG6hIAzvnPGWNDjpuvBnCUc34cABhj\n3wZwB+f8LwDcutAFMcY+COCDALBhw9yBI2Jt4By4stqQLiCjPHBazQIYbPfjvg+9ZlnX1Bkw5FSu\nZLbgOr5TbPhCADRP+Xqrbf7m45u/e93WrrrWI+oWXj5ntrBearfXUlPeokSRFk8ztHiol8XYSAMA\nzth+HrFuc4Ux1skY+2cAuxljn6p2P8753ZzzPZzzPd3d9Z0cCKLZEUHgkgUgZus2bpMzO3aKLCB3\nC0DEANLZSgtgLl5zURfeuK0bF/fVV/C0qVsIgNnBtNmD/s76lFZ2AS0azvkkgA+t1PMRRDMR9qrQ\nVY88JXrnsABWAhED4JwjNUcMIOUSA5iLW3b14ZZdfXXfvztoIKArmEpkYageKE1e8+FsUx7xa/js\nHTtx4/aeBq5qfixGAM4CWG/7edC6jSAIB6riwXc+9BpstBq0zRUDWAk6A7rsB5SoIgCqIwagLeOm\nzBjDpu4AXjobWxVN/+wxKVE09r7rhhq0moWxmE/fMwC2MMY2McZ0AO8C8L2lWRZBtB6XDEZkLKAp\nBMAK0o7MmJk3bq0PhAWQzM7fAlgIm6wxiash4F/mAloFguVGvWmg9wDYC2AbY2yEMfZ+znkewEcB\nPAzgEIB7OecvL8WiGGO3McbujkajS/FwBNF0lGIADXQBWXMJvvnUKRSK3LVYS2YBWQKgzSMGsBA2\ndZoW0mqo+fDbNv3VOqei3iygd1e5/YcAfrikKzIf90EAD+7Zs+cDS/3YBNEMzFUIthKILJ37949g\nc3cAV2+qLNYSE8BkDMCzzBaAFQheTBuIlUK0ijb7VK3Ovpqrc9UEscqRQeBldqnUQgSkcwWOd121\n3rV/jdMCmE8W0EIQLqBqffibCb+tFcZq6f3jhASAIBqAaKXcyDRQIQCawvCbV7j3DFIdlcDL7wIy\nLYDV4AISdQCr1f0DUDM4gmgIzeAC8mpm8dJ1F3XKgStOhMtnpVxAEb+GjoDe9DUAQCkNdLUGgIEm\nFQDG2G0AbhseHm70UghiWWiGLCAAuOcD11bt5QPY6gBWyAUEAH/5m5diXWRlRo8uBn+NbqirhaZ0\nAXHOH+ScfzASiTR6KQSxLMzVDG6l2NYXqjnAXMQAktIFtPzrvXlHL3YNNP93v+QCan53VTWaUgAI\notXxSQuguTcPGQMQFkCTV+euJNICWAXuqmqQABBEA2iGVhD1oDliACthAawWRBxnNQeB6d0kiAbQ\nLDGAuXB2A12JGMBqweNhCOgKQnMMj29mVu/KCWIVE/KqYAxNv3lU1AEscxbQauN/vuMybOsLNXoZ\nC6YpP32UBUS0Ol1BA/d84Fpcvr6t0UupifD5r1QdwGrj1+uYHdzMNKWcUxYQsRa4dnOndAU1K4qH\ngbGVawZHrCz0bhIEURXGGDSPZ0XaQRMrDwkAQRA1URVmcwHRltFK0LtJEERNVA9DrsDNf1MMoKUg\nASAIoib2Uz9ZAK0FvZsEQdTEfuqnSuDWoikFgCaCEUTzYD/1N/ugdmJ+NKUAUBooQTQPQgA0ha3a\nwSeEO00pAARBNA/C7UNVwK0HvaMEQdREFH9RBlDrQQJAEERNRPsHygBqPegdJQiiJiUXEFkArQYJ\nAEEQNVFlEJi2i1aD3lGCIGqiUwygZWlKAaA6AIJoHsTGTy6g1qMpBYDqAAiieRDpn+QCaj3oHSUI\noiYiC4hcQK0HCQBBEDWRdQBUCNZy0DtKEERNxBAYGgfZepAAEARRE40sgJaF3lGCIGqiUgygZSEB\nIAiiJhoVgrUs9I4SBFETagXRupAAEARRE2oF0bo05TtKlcAE0TxQHUDr0pQCQJXABNE8UBZQ60Lv\nKEEQNVEVqgNoVUgACIKoieahbqCtCgkAQRA1KXUDpe2i1aB3lCCImpSygMgCaDVIAAiCqInoBaRS\nGmjLQe8oQRA1kRYAFYK1HCQABEHUpFQHQNtFq0HvKEEQNdFoJnDLQgJAEERNRA8gjbKAWg56RwmC\nqAlZAK0LCQBBEDVRKQbQsjTlO0rN4AiieRAFYJQF1Ho0pQBQMziCaB50lSyAVoXeUYIgaiItAIoB\ntBwkAARB1ITaQbcu9I4SBFGTrb1BfOj1F+H64c5GL4VYYtRGL4AgiOZGVTz45K9tb/QyiGWALACC\nIIg1CgkAQRDEGoUEgCAIYo1CAkAQBLFGIQEgCIJYo5AAEARBrFFIAAiCINYoJAAEQRBrFMY5b/Qa\nqsIYGwdwaoF/3gVgYgmXsxqga14brLVrXmvXCyzumjdyzrvruWNTC8BiYIzt45zvafQ6VhK65rXB\nWrvmtXa9wMpdM7mACIIg1igkAARBEGuUVhaAuxu9gAZA17w2WGvXvNauF1iha27ZGABBEARRm1a2\nAAiCIIgatJwAMMZuYYy9whg7yhj7ZKPXsxIwxr7MGBtjjL3U6LWsBIyx9YyxnzHGDjLGXmaM/XGj\n17TcMMa8jLFfMcYOWNf8mUavaaVgjCmMsecYY99v9FpWAsbYScbYi4yx5xlj+5b1uVrJBcQYUwAc\nAXAzgBEAzwB4N+f8YEMXtswwxl4HIA7g65zzXY1ez3LDGFsHYB3n/FnGWAjAfgBva+X3mTHGAAQ4\n53HGmAbgCQB/zDl/qsFLW3YYYx8HsAdAmHN+a6PXs9wwxk4C2MM5X/bah1azAK4GcJRzfpxzngXw\nbQB3NHhNyw7n/OcAphq9jpWCcz7KOX/W+vcsgEMABhq7quWFm8StHzXrv9Y5vVWBMTYI4K0A/rXR\na2lFWk0ABgCcsf08ghbfGNY6jLEhALsBPN3YlSw/livkeQBjAH7COW/5awbwdwD+FECx0QtZQTiA\nRxhj+xljH1zOJ2o1ASDWEIyxIIDvAPgTznms0etZbjjnBc755QAGAVzNGGtpdx9j7FYAY5zz/Y1e\nywpzg/U+/xqAj1gu3mWh1QTgLID1tp8HrduIFsPyg38HwLc4599t9HpWEs75DICfAbil0WtZZq4H\ncLvlE/82gBsZY99s7JKWH875Wev/YwAegOnaXhZaTQCeAbCFMbaJMaYDeBeA7zV4TcQSYwVEvwTg\nEOf8bxq9npWAMdbNGGuz/u2DmehwuLGrWl4455/inA9yzodgfpcf5Zy/t8HLWlYYYwErsQGMsQCA\nNwNYtuy+lhIAznkewEcBPAwzMHgv5/zlxq5q+WGM3QNgL4BtjLERxtj7G72mZeZ6AL8D80T4vPXf\nrzd6UcvMOgA/Y4y9APOg8xPO+ZpIi1xj9AJ4gjF2AMCvAPyAc/7Qcj1ZS6WBEgRBEPXTUhYAQRAE\nUT8kAARBEGsUEgCCIIg1CgkAQRDEGoUEgCAIYo1CAkAQBLFGIQEgCIJYo5AAEARBrFH+DyFfFiF7\n3wSkAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1059a14a8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x_axis = np.linspace(0, 5, len(losses), endpoint=True)\n",
    "plt.semilogy(x_axis, losses, label='adagrad')\n",
    "plt.legend(loc='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到，使用自适应的学习率跑 5 个 epoch 可以得到比随机梯度下降得到更小的 loss，学习率能够自适应地降低，所以能够有着更好的效果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "当然 pytorch 也内置了 adagrad 的优化算法，只需要调用 `torch.optim.Adagrad()` 就可以了，下面是例子"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 0, Train Loss: 0.408064\n",
      "epoch: 1, Train Loss: 0.262110\n",
      "epoch: 2, Train Loss: 0.219893\n",
      "epoch: 3, Train Loss: 0.192386\n",
      "epoch: 4, Train Loss: 0.173119\n",
      "使用时间: 56.94233 s\n"
     ]
    }
   ],
   "source": [
    "train_data = DataLoader(train_set, batch_size=64, shuffle=True)\n",
    "# 使用 Sequential 定义 3 层神经网络\n",
    "net = nn.Sequential(\n",
    "    nn.Linear(784, 200),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(200, 10),\n",
    ")\n",
    " \n",
    "optimizer = torch.optim.Adagrad(net.parameters(), lr=1e-2)\n",
    "# 开始训练\n",
    "\n",
    "start = time.time() # 记时开始\n",
    "for e in range(5):\n",
    "    train_loss = 0\n",
    "    for im, label in train_data:\n",
    "        im = Variable(im)\n",
    "        label = Variable(label)\n",
    "        # 前向传播\n",
    "        out = net(im)\n",
    "        loss = criterion(out, label)\n",
    "        # 反向传播\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        # 记录误差\n",
    "        train_loss += loss.data[0]\n",
    "    print('epoch: {}, Train Loss: {:.6f}'\n",
    "          .format(e, train_loss / len(train_data)))\n",
    "end = time.time() # 计时结束\n",
    "print('使用时间: {:.5f} s'.format(end - start))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}