{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 学习率衰减\n",
    "对于基于一阶梯度进行优化的方法而言，开始的时候更新的幅度是比较大的，也就是说开始的学习率可以设置大一点，但是当训练集的 loss 下降到一定程度之后，继续使用这个太大的学习率就会导致 loss 一直来回震荡，比如\n",
    "\n",
    "![](https://ws4.sinaimg.cn/large/006tNc79ly1fmrvdlncomj30bf0aywet.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这个时候就需要对学习率进行衰减已达到 loss 的充分下降，而是用学习率衰减的办法能够解决这个矛盾，学习率衰减就是随着训练的进行不断的减小学习率。\n",
    "\n",
    "在 pytorch 中学习率衰减非常方便，使用 `torch.optim.lr_scheduler`，更多的信息可以直接查看[文档](http://pytorch.org/docs/0.3.0/optim.html#how-to-adjust-learning-rate)\n",
    "\n",
    "但是我推荐大家使用下面这种方式来做学习率衰减，更加直观，下面我们直接举例子来说明"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-24T08:45:34.293625Z",
     "start_time": "2017-12-24T08:45:33.834665Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append('..')\n",
    "\n",
    "import numpy as np\n",
    "import torch\n",
    "from torch import nn\n",
    "import torch.nn.functional as F\n",
    "from torch.autograd import Variable\n",
    "from torchvision.datasets import CIFAR10\n",
    "from utils import resnet\n",
    "from torchvision import transforms as tfs\n",
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-24T08:45:35.195093Z",
     "start_time": "2017-12-24T08:45:35.063610Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "net = resnet(3, 10)\n",
    "optimizer = torch.optim.SGD(net.parameters(), lr=0.01, weight_decay=1e-4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里我们定义好了模型和优化器，可以通过 `optimizer.param_groups` 来得到所有的参数组和其对应的属性，参数组是什么意思呢？就是我们可以将模型的参数分成几个组，每个组定义一个学习率，这里比较复杂，一般来讲如果不做特别修改，就只有一个参数组\n",
    "\n",
    "这个参数组是一个字典，里面有很多属性，比如学习率，权重衰减等等，我们可以访问以下"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-24T08:22:59.192905Z",
     "start_time": "2017-12-24T08:22:59.187178Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "learning rate: 0.01\n",
      "weight decay: 0.0001\n"
     ]
    }
   ],
   "source": [
    "print('learning rate: {}'.format(optimizer.param_groups[0]['lr']))\n",
    "print('weight decay: {}'.format(optimizer.param_groups[0]['weight_decay']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "所以我们可以通过修改这个属性来改变我们训练过程中的学习率，非常简单"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-24T08:25:04.767090Z",
     "start_time": "2017-12-24T08:25:04.762612Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "optimizer.param_groups[0]['lr'] = 1e-5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了防止有多个参数组，我们可以使用一个循环"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-24T08:26:05.142183Z",
     "start_time": "2017-12-24T08:26:05.136955Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for param_group in optimizer.param_groups:\n",
    "    param_group['lr'] = 1e-1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "方法就是这样，非常简单，我们可以在任意的位置改变我们的学习率\n",
    "\n",
    "下面我们具体来看看学习率衰减的好处"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-24T08:45:40.809459Z",
     "start_time": "2017-12-24T08:45:40.803993Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def set_learning_rate(optimizer, lr):\n",
    "    for param_group in optimizer.param_groups:\n",
    "        param_group['lr'] = lr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-24T08:45:48.006789Z",
     "start_time": "2017-12-24T08:45:46.738002Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 使用数据增强\n",
    "def train_tf(x):\n",
    "    im_aug = tfs.Compose([\n",
    "        tfs.Resize(120),\n",
    "        tfs.RandomHorizontalFlip(),\n",
    "        tfs.RandomCrop(96),\n",
    "        tfs.ColorJitter(brightness=0.5, contrast=0.5, hue=0.5),\n",
    "        tfs.ToTensor(),\n",
    "        tfs.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])\n",
    "    ])\n",
    "    x = im_aug(x)\n",
    "    return x\n",
    "\n",
    "def test_tf(x):\n",
    "    im_aug = tfs.Compose([\n",
    "        tfs.Resize(96),\n",
    "        tfs.ToTensor(),\n",
    "        tfs.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])\n",
    "    ])\n",
    "    x = im_aug(x)\n",
    "    return x\n",
    "\n",
    "train_set = CIFAR10('./data', train=True, transform=train_tf)\n",
    "train_data = torch.utils.data.DataLoader(train_set, batch_size=256, shuffle=True, num_workers=4)\n",
    "valid_set = CIFAR10('./data', train=False, transform=test_tf)\n",
    "valid_data = torch.utils.data.DataLoader(valid_set, batch_size=256, shuffle=False, num_workers=4)\n",
    "\n",
    "net = resnet(3, 10)\n",
    "optimizer = torch.optim.SGD(net.parameters(), lr=0.1, weight_decay=1e-4)\n",
    "criterion = nn.CrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-24T08:59:49.656832Z",
     "start_time": "2017-12-24T08:45:48.556187Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0. Train Loss: 1.872896, Valid Loss: 1.798441, Time 00:00:26\n",
      "Epoch 1. Train Loss: 1.397522, Valid Loss: 1.421618, Time 00:00:28\n",
      "Epoch 2. Train Loss: 1.129362, Valid Loss: 1.487882, Time 00:00:28\n",
      "Epoch 3. Train Loss: 0.962217, Valid Loss: 2.095880, Time 00:00:28\n",
      "Epoch 4. Train Loss: 0.859332, Valid Loss: 1.686056, Time 00:00:27\n",
      "Epoch 5. Train Loss: 0.786428, Valid Loss: 1.348701, Time 00:00:27\n",
      "Epoch 6. Train Loss: 0.730535, Valid Loss: 1.568454, Time 00:00:27\n",
      "Epoch 7. Train Loss: 0.682074, Valid Loss: 1.230555, Time 00:00:28\n",
      "Epoch 8. Train Loss: 0.643144, Valid Loss: 0.878328, Time 00:00:27\n",
      "Epoch 9. Train Loss: 0.609817, Valid Loss: 0.869068, Time 00:00:27\n",
      "Epoch 10. Train Loss: 0.585312, Valid Loss: 0.794440, Time 00:00:27\n",
      "Epoch 11. Train Loss: 0.553877, Valid Loss: 1.900850, Time 00:00:27\n",
      "Epoch 12. Train Loss: 0.526790, Valid Loss: 0.752651, Time 00:00:27\n",
      "Epoch 13. Train Loss: 0.505155, Valid Loss: 1.112544, Time 00:00:27\n",
      "Epoch 14. Train Loss: 0.486104, Valid Loss: 0.942357, Time 00:00:27\n",
      "Epoch 15. Train Loss: 0.468512, Valid Loss: 0.729420, Time 00:00:27\n",
      "Epoch 16. Train Loss: 0.449669, Valid Loss: 0.842467, Time 00:00:27\n",
      "Epoch 17. Train Loss: 0.429664, Valid Loss: 0.792635, Time 00:00:27\n",
      "Epoch 18. Train Loss: 0.420088, Valid Loss: 1.025345, Time 00:00:27\n",
      "Epoch 19. Train Loss: 0.404701, Valid Loss: 0.651725, Time 00:00:27\n",
      "Epoch 20. Train Loss: 0.326198, Valid Loss: 0.491904, Time 00:00:27\n",
      "Epoch 21. Train Loss: 0.294623, Valid Loss: 0.478969, Time 00:00:27\n",
      "Epoch 22. Train Loss: 0.284980, Valid Loss: 0.455259, Time 00:00:28\n",
      "Epoch 23. Train Loss: 0.273168, Valid Loss: 0.465930, Time 00:00:27\n",
      "Epoch 24. Train Loss: 0.270120, Valid Loss: 0.455458, Time 00:00:27\n",
      "Epoch 25. Train Loss: 0.261299, Valid Loss: 0.462319, Time 00:00:27\n",
      "Epoch 26. Train Loss: 0.258373, Valid Loss: 0.442525, Time 00:00:27\n",
      "Epoch 27. Train Loss: 0.251803, Valid Loss: 0.457620, Time 00:00:27\n",
      "Epoch 28. Train Loss: 0.247022, Valid Loss: 0.451055, Time 00:00:27\n",
      "Epoch 29. Train Loss: 0.246816, Valid Loss: 0.448706, Time 00:00:28\n"
     ]
    }
   ],
   "source": [
    "train_losses = []\n",
    "valid_losses = []\n",
    "\n",
    "if torch.cuda.is_available():\n",
    "    net = net.cuda()\n",
    "prev_time = datetime.now()\n",
    "for epoch in range(100):\n",
    "    if epoch == 20:\n",
    "        set_learning_rate(optimizer, 0.01) # 20 次修改学习率为 0.01\n",
    "    elif epoch == 60:\n",
    "        set_learning_rate(optimizer, 0.005) # 60 次修改学习率为 0.01\n",
    "\n",
    "    train_loss = 0\n",
    "    net = net.train()\n",
    "    for im, label in train_data:\n",
    "        if torch.cuda.is_available():\n",
    "            im = Variable(im.cuda())  # (bs, 3, h, w)\n",
    "            label = Variable(label.cuda())  # (bs, h, w)\n",
    "        else:\n",
    "            im = Variable(im)\n",
    "            label = Variable(label)\n",
    "        # forward\n",
    "        output = net(im)\n",
    "        loss = criterion(output, label)\n",
    "        # backward\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "        train_loss += loss.data[0]\n",
    "    cur_time = datetime.now()\n",
    "    h, remainder = divmod((cur_time - prev_time).seconds, 3600)\n",
    "    m, s = divmod(remainder, 60)\n",
    "    time_str = \"Time %02d:%02d:%02d\" % (h, m, s)\n",
    "    valid_loss = 0\n",
    "    valid_acc = 0\n",
    "    net = net.eval()\n",
    "    for im, label in valid_data:\n",
    "        if torch.cuda.is_available():\n",
    "            im = Variable(im.cuda(), volatile=True)\n",
    "            label = Variable(label.cuda(), volatile=True)\n",
    "        else:\n",
    "            im = Variable(im, volatile=True)\n",
    "            label = Variable(label, volatile=True)\n",
    "        output = net(im)\n",
    "        loss = criterion(output, label)\n",
    "        valid_loss += loss.data[0]\n",
    "    epoch_str = (\n",
    "        \"Epoch %d. Train Loss: %f, Valid Loss: %f, \"\n",
    "        % (epoch, train_loss / len(train_data), valid_loss / len(valid_data)))\n",
    "    prev_time = cur_time\n",
    "    \n",
    "    train_losses.append(train_loss / len(train_data))\n",
    "    valid_losses.append(valid_loss / len(valid_data))\n",
    "    print(epoch_str + time_str)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面我们画出 loss 曲线"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-24T09:01:37.676274Z",
     "start_time": "2017-12-24T09:01:37.439613Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-24T09:02:37.432883Z",
     "start_time": "2017-12-24T09:02:37.244995Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7fe8a60d4e48>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEKCAYAAAAVaT4rAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xd4XOWV+PHvUe9dtmzJsuQG7r0b\nbDDGhiw11ACBhISQwJKEzW5Idjd197fZtE0IBGIIATaUJYYEktCLMcXGvRv3Jrmo2Fazut7fH+8d\neSyrjDR3NJrR+TzPPDNz27zXY51759z3nleMMSillOo/IoLdAKWUUr1LA79SSvUzGviVUqqf0cCv\nlFL9jAZ+pZTqZzTwK6VUP6OBXyml+hkN/Eop1c9o4FdKqX4mKtgNaE9WVpYpKCgIdjOUUipkrFu3\nrswYk+3Lsn0y8BcUFLB27dpgN0MppUKGiBz0dVlN9SilVD+jgV8ppfoZDfxKKdXP9Mkcv1JKdUdj\nYyNFRUXU1dUFuykBFxcXR15eHtHR0T3ehgZ+pVTIKyoqIjk5mYKCAkQk2M0JGGMM5eXlFBUVUVhY\n2OPtaKpHKRXy6urqyMzMDOugDyAiZGZm+v3LRgO/UioshHvQ93BjPzXwu6XqGGx9KditUEqpLmng\nd8vqpbDsC1BRFOyWKKV62alTp/jtb3/b7fUuv/xyTp06FYAWdU4Dv1tKPrXPB1cGtx1KqV7XUeBv\namrqdL1XX32VtLS0QDWrQxr43VK20z4f+ji47VBK9boHHniAvXv3MmnSJKZPn84FF1zAlVdeyZgx\nYwC4+uqrmTp1KmPHjmXp0qWt6xUUFFBWVsaBAwcYPXo0X/7ylxk7diyXXnoptbW1AWuvdud0Q1M9\nnNhvX+sZv1JB9cO/bmP7kUpXtzlmcArfv2Jsh/N/8pOfsHXrVjZu3Mjy5cv5zGc+w9atW1u7XD7x\nxBNkZGRQW1vL9OnT+exnP0tmZuZZ29i9ezfPPfccjz32GDfccAMvvvgit956q6v74aFn/G4o3wum\nGQaMgdIdcPpEsFuklAqiGTNmnNXP/sEHH2TixInMmjWLw4cPs3v37nPWKSwsZNKkSQBMnTqVAwcO\nBKx9esbvBk+aZ+od8Nq/wKGVcP5ngtokpfqrzs7Me0tiYmLr6+XLl/P222+zcuVKEhISWLBgQbv9\n8GNjY1tfR0ZGBjTVo2f8bijdBQhMuAEiY+Gg5vmV6k+Sk5Opqqpqd15FRQXp6ekkJCTw6aefsmrV\nql5u3bm6DPwiMkRE3hOR7SKyTUS+3s4yIiIPisgeEdksIlO85t0uIrudx+1u70CfUPoppOVDfDrk\nTtXAr1Q/k5mZydy5cxk3bhz//M//fNa8JUuW0NTUxOjRo3nggQeYNWtWkFp5hi+pnibgn4wx60Uk\nGVgnIm8ZY7Z7LXMZMNJ5zAQeAWaKSAbwfWAaYJx1XzHGnHR1L4KtbBdkn29fD50NH/4K6qshNim4\n7VJK9Zpnn3223emxsbG89tpr7c7z5PGzsrLYunVr6/RvfetbrrfPW5dn/MaYo8aY9c7rKmAHkNtm\nsauAp421CkgTkUHAYuAtY8wJJ9i/BSxxdQ+CraUZynZD9ij7fugce6G3aE1w26WUUh3oVo5fRAqA\nycAnbWblAoe93hc50zqa3t627xKRtSKytrS0tDvNCq6TB6C5HrLOs+/zZoBEaLpHKdVn+Rz4RSQJ\neBH4hjHG3U6ygDFmqTFmmjFmWna2T+MF9w1lu+yzJ9UTlwI5423PHqWU6oN8CvwiEo0N+s8YY9qr\nRFYMDPF6n+dM62h6+Ch1unJ6Uj0AQ+faVE9TQ3DapJRSnfClV48Avwd2GGN+2cFirwCfd3r3zAIq\njDFHgTeAS0UkXUTSgUudaeGjdCck5UBc6plp+bOhqQ6ObAheu5RSqgO+9OqZC9wGbBGRjc607wL5\nAMaYR4FXgcuBPcBp4AvOvBMi8mPAc6XzR8aY8LqttWwnZJ939rT82fb50MeQP7P326SUUp3wpVfP\nh8YYMcZMMMZMch6vGmMedYI+Tm+ee4wxw40x440xa73Wf8IYM8J5/CGQO9PrjLE3b7UN/EnZkDVK\n6/YopTqUlGS7ex85coTrrruu3WUWLFjA2rVr253nD71z1x+VR6Chygb5tvJnw6FVtrunUkp1YPDg\nwSxbtqxXP1MDvz88NXo8PXq8DZ0D9RVQsv3ceUqpsPPAAw/w8MMPt77/wQ9+wH/8x3+wcOFCpkyZ\nwvjx43n55ZfPWe/AgQOMGzcOgNraWm666SZGjx7NNddcE7B6PVqkzR+tPXrOO3fe0Dn2+eBK271T\nKdU7XnsAjm1xd5s54+Gyn3S6yI033sg3vvEN7rnnHgBeeOEF3njjDe677z5SUlIoKytj1qxZXHnl\nlR2Om/vII4+QkJDAjh072Lx5M1OmTGl3OX/pGb8/SndCXBoktnPfQVo+pOTBwY96v11KqV43efJk\nSkpKOHLkCJs2bSI9PZ2cnBy++93vMmHCBC655BKKi4s5fvx4h9tYsWJFaw3+CRMmMGHChIC0Vc/4\n/eGp0dPRqPdD58D+9+1F4I6WUe07vh32vgtz7g12S1So6eLMPJCuv/56li1bxrFjx7jxxht55pln\nKC0tZd26dURHR1NQUNBuSebepmf8/ij99Owbt9oaOhuqj8OJfb3XpnCx8Rl481+hMfh/JEr56sYb\nb+T5559n2bJlXH/99VRUVDBgwACio6N57733OHjwYKfrX3jhha3F3rZu3crmzZsD0k4N/D1VUw6n\ny8/U6GlPvifPr3V7uq3Gqdd0uiy47VCqG8aOHUtVVRW5ubkMGjSIW265hbVr1zJ+/Hiefvppzj+/\nnY4gXr761a9SXV3N6NGj+d73vsfUqVMD0s6wSfU0NLXwnZe2cMHILK6e3G4dOHd11qPHI/s8SMi0\ndXum3Bb4NoWT6hL7XFMKqXnBbYtS3bBly5kLy1lZWaxc2f79PNXV1YAdcN1Tkjk+Pp7nn38+4G0M\nmzP+mKgIPtpTxns7S3rnA0s/tc+dpXpEbH9+PePvPs8Zf42e8SvltrAJ/AAT8lLZXFTROx9Wugui\nE2zPnc7kz4aT+6HyaO+0K1xo4FcqYMIq8E8cksb+shoqahsD/2FlO+0duxFd/BN6+vMf0rN+n7W0\nnAn4NSE0NoMKKmNMsJvQK9zYz7AK/BPybIXMLb1x1l/aTnG29uRMgJgkrdvTHbUn7ChmoIFf+SQu\nLo7y8vKwD/7GGMrLy4mLi/NrO2FzcRdgfK4N/JuLTzFvZFbgPqi+CiqL26/R01ZkFORN1zx/d1R7\nXafRVI/yQV5eHkVFRYTU6H09FBcXR16efx0ewirwpyXEMDQzgc2HA3zG33bUra4MnQvv/SfUnoT4\n9MC1K1x4n+XrGb/yQXR0NIWFhcFuRsgIq1QPwIS8NDYXnQrsh3RWo6c9Q2cDBg61HapYtcsT7NOG\nauBXKgDCJ/A31sELt3NdxPscqaijtKo+cJ9VuhMioiHdxzOM3Kl2ea3b4xtPqmfAGE31KBUA4RP4\no+Pg6EYmVH8IENiz/rJdkDnC5u99alu8Df46ALtvakrsgTJrhD3jD/MLdkr1tvAJ/ACFF5JW8glR\n0sKmQPbs6apGT3uGzrZj8DacDkybwklNqa14mjgAmuvtxXSllGt8GWz9CREpEZGtHcz/ZxHZ6Dy2\nikiziGQ48w6IyBZnnvvjh7VVOB+pr+SyjOOBO+NvrIOTBzqv0dOe/DnQ0gRFa7petr+rLoXErDPl\nrrVej1Ku8uWM/0lgSUczjTE/84zFC3wHeL/NgOoXOfOn+ddUHxReCMBliTvZXFQRmD69J/aCafH9\nwq5H/kxANN3ji5oSSBpwJvBrnl8pV/ky2PoK4ERXyzluBp7zq0X+SBoAA8YwqXkzJ2oaKD4VgGHL\nWmv0dDPwx6VCzjjtz++LmjKb5kl07sXQnj1Kucq1HL+IJGB/GbzoNdkAb4rIOhG5q4v17xKRtSKy\n1q+bMArnk3NqIzE0BqZuT+kuQOzF3e7Kn2NTPc29UFIiVBlje/UkZWvgVypA3Ly4ewXwUZs0zzxj\nzBTgMuAeEbmwo5WNMUuNMdOMMdOys9sZytBXw+YT0VzH9Kg9bApEnr9sJ6QX2J463TV0DjSehqOb\nXG9W2KivtBd0E7MhQQO/UoHgZuC/iTZpHmNMsfNcAvwZmOHi57Vv6ByQCK5I3h2YO3h9rdHTnqE6\nMEuXqp0gnzjAdtGNTdEcv1IucyXwi0gqMB942Wtaoogke14DlwLt9gxyVVwqDJ7C7IhtbC2uoKXF\nxQu8zU1Qvse3Gj3tSRoAGcM18HfGc3af5PzqS8zSM36lXOZLd87ngJXAeSJSJCJ3isjdInK312LX\nAG8aY2q8pg0EPhSRTcBq4O/GmNfdbHyHhs1nyOkdtNRXsa+spuvlfXXqIDQ3+F6jpz1D59iePS0t\n7rUrnNQ4d+16evQkZmvgV8plXd56aoy52YdlnsR2+/Setg+Y2NOG+aXwQiI++AUzIj5lc9EsRgxI\ncme7Pe3R423oHNjwv3ZbA8e4065w4inXkDjAec7WweqVcll43bnrMWQmJjKW+dHb3e3Z4ynO1tNU\nD9gRuUDr9nSkphQQO1YxOKkezfEr5abwDPzR8Uj+TBZE73D3Dt6yXZA8GOJSer6N9AJIzYd9y91q\nVXipKbVB31MHKTHb3rmrqTGlXBOegR+gcD4FTfs4cqSIxmaXgkZPavS0JQIjF8He96ApgBVEQ1V1\nyZn8PtjXpsWOZaCUckVYB36AKS1b2XXchSJfxtibt/y5sOsx8lJorNHePe2pKT3TowfOpHz0Aq9S\nrgnfwD94Mi0xycyN2OZOnr+iyAZrf/L7HoUXQmQs7H7T/22Fm5rSMxd2watejwZ+pdwSvoE/Mgop\nmMu8qG3u5PnLujnqVmdiEqDwAg387akuPTfVAxr4lXJR+AZ+QArnM5RjHDm42/+NlXZznN2ujFxs\nbwYr3+vO9sJBYy00VJ2d6tEKnUq5LqwDP8Nsnn9g+WrqGpv921bppxCfcaZwmL9GLrLPu99yZ3vh\noG0ffoCEDED0jF8pF4V34B8whvrYDGbJVrYdqfRvW2W73EnzeGQU2usFu99wb5uhznNWn+QV+CMi\n7QVeDfxKuSa8A78ILUMvYG7EVrYc9qM7oDFOV04XAz/Y3j0HPoQGF8tKhLLWcg1tflVp2QalXBXe\ngR+IG3UxA+UUx/Zt6flGaspsP/LuDrfYlZGX2to/+953d7uhqr1UD9gDweny3m+PUmEq7AO/OHn+\nhOIPe76R1h49LnTl9JY/G2KStXePh+esPrHNeAx6xq+Uq8I+8JNeQGXsIEad3kBVXQ9HvvLU6HGr\nR49HVAwMX2ADv7/jAzfWhn5Zg5pSW38/Ou7s6VqaWSlXhX/gF6Emdy6zIraz5bCvQwe3UboTYpIg\nJdfdtoFN91QWQ8n2nm+jrgL+Zxx8/Gv32hUMbcs1eCRm231sauj9NikVhsI/8APJoy8hTWo4snN1\nzzZQthOyRto6O24b4XTr3OVH7571T9tCZtv+4k6bgqWm9OwePR6ei72ntS+/Um7oF4E/6fyLAIjc\nv6JnG3CrRk97UgZBzoSe9+dvboJVj4JEwNGNUHXM3fb1pprSjs/4PfOVUn7rF4Gf5ByOxgxl0Mk1\n3V+3rgKqjrhTo6cjoxbD4U96VoFyx8tQWQQLvmvf73nb3bb1ps5SPaCBXymX+DL04hMiUiIi7Y6X\nKyILRKRCRDY6j+95zVsiIjtFZI+IPOBmw7urfMBsJjRvp7yim5U6y5xyD2734fc28lIwzbD33e6t\nZwx8/JAdx/eC++1YAaHaQ6i5EWpPdJDq0bINSrnJlzP+J4ElXSzzgTFmkvP4EYCIRAIPA5cBY4Cb\nRSRoYw1GjbiIBKnn4OZu9pkPVI8eb7lTbTmI7qZ7Dq2CI+th9tfsHa6eOv/NPey9FEyeoN7uGb+T\n49czfqVc0WXgN8asAHrSHWYGsMcYs88Y0wA8D1zVg+24Im/yIpqN0LhrefdWLP0UImMgbWhA2gXY\noD3iEhv4u9Mlc+VDEJ8OEz9n34+8FOor7QEh1HTUhx9sF8/IGD3jV8olbuX4Z4vIJhF5TUTGOtNy\ngcNeyxQ504IiKTWT3ZEjSC9Z2b0Vy3ZB5ogzQwEGyqjFttfKkQ2+LV++Fz79O0y705Z5Bhi2ACKi\nQzPd4ynX0F6qR8S5iUsDv1JucCPwrweGGmMmAr8BetSnUETuEpG1IrK2tDQwP+mL02cwrH4Hpt7H\nPH9zk+1fH8j8vsfwi23PHF+Ltn3yKERGw4wvn5kWmwQFc0Mz8Fd3csYPWqhNKRf5HfiNMZXGmGrn\n9atAtIhkAcXAEK9F85xpHW1nqTFmmjFmWnZ2B3/8fmouuIAomjmx3Yc8f+lO+P0iOHWodRjHgErI\ngLwZvgXt0ydgwx9h/PWQnHP2vJGX2vTUyYOBaWegeIJ6e2f8oGUblHKR34FfRHJE7J1NIjLD2WY5\nsAYYKSKFIhID3AS84u/n+WPA2PnUmygqd7zT8UItzfDRg/DoBXDyAFz3B5j2hd5p4MhFNtVTdbzz\n5dY9CY2nYdbX2tnGYvscamf9NSUQFWfvkG6PpnqUco0v3TmfA1YC54lIkYjcKSJ3i8jdziLXAVtF\nZBPwIHCTsZqAe4E3gB3AC8aYbYHZDd+cP2QgG8woEoo+an+B8r3wh8vgrX+3F1vv+QTGXdt7DRzl\nBO3O+uI3NcDqpTDsIsgZd+78zOGQXhh6A7xUO2PtdnR3tKdej781jZRSdHnF0hhzcxfzHwIe6mDe\nq8CrPWua++KiI9mTOJkZp5+16ZKEDDujpcUG07d/YAunXbMUJtwQmBINnRk4zumL/wZMvqX9Zba9\nBFVH4cp2/8ltm0cthnVP2cJt0fGBa6+bakrOHnKxrcRsaKq1YxfEdvCrQCnlk/5x566X6tx5RGAw\nnvINJw/A01fC69+GgnnwtVUw8cbeD/pgP7OzvvjG2C6c2efDiIUdb2fkIhskD/hRirq3dVSuwUPv\n3lXKNf0u8KePmEm1iaNqxzuw5vfw2zlwZCNc+Ru45U+QMji4DeysL/7+FXBsC8y+p/MD09B5EJ0Q\nWnn+al8Dv+b5lfJXvwv84/OzWd1yPslb/xf+fj8MmQ5fWwlTPh+cs/y2hi3ouC/+yodtABx/Q+fb\niI6zPZHcqPPfG1paOq7M6aEVOpVyTb8L/KMGJvGmzKYhIh4+8wu47S+QNqTrFXtLR33xS3fZ3P/0\nL587UEl7Ri6yaSxPraG+rPakrVXUdshFb5rqUco1/S7wR0VGsDvnCm7JWgbTv9Q3zvLbGrn43L74\nqx6GyFiYfqeP27jUPodCuqe1XENWx8tovR6lXNPvAj/AhCFpbD1aSUNTHx2qsG3QrimDTc/DxJs6\nD47e0obAgDEhEvg7KdfgER1v+/hrjl8pv/XLwH/ByCzqGlt4f1cfPXvMGgEZw870xV/ze2iqsxd1\nu2PkIjj4MfhaoiJYqp3A31mqB3TsXaVc0k8DfzZZSTG8tL4o2E3p2MhLbS+e2pOw5jH7vrs1g0Yu\nhpZG2Lc8IE10TWclmb1p2QalXNEvA390ZARXTszlnR0lVJzuo7XrR15q++K/cp8NdrPv7f42hsyA\n2FT/xvPtDTUlIJG2xHRntGyDUq7ol4Ef4NopuTQ0t/C3LUeC3ZT2DZ1r++LveAUGjofCC7u/jcho\nGH6RTRn15W6dniEXI7r476ipHqVc0W8D/9jBKYwamMRL6zssGBpc0XG2Tz90fcNWZ0Ythupj9sav\nvqqmtPNyDR6eM/7uDFajlDpHvw38IsK1U/JYd/AkB8pqgt2c9k37IoxYBOM+2/NtjLjEPvta5z8Y\nuirX4JGYbfv7150KfJuUCmP9NvADXDVpMCLw5w199Kx/5CK4dZktHNdTSQNg8OS+Xa3TU5mzKwme\nvvya51fKH/068A9KjWfu8Cxe2lCE6cs5cH+NXAxFa2xF0r7GmK4rc3roTVxKuaJfB36wF3kPn6hl\n7cGTwW5K4Iy8FEwL7OlkAJpgaai29yj4muoBDfxK+anfB/7FY3OIj47s2336/TV4sk2T9MW7eH29\neQs08Cvlkn4f+BNjo7hsXA5/23yUusbmYDcnMCIi7PWCPW/boSX7ktaxdn0440/IdNbRHL9S/uj3\ngR/g2il5VNU18c6OkmA3JXBGLoLaE1C8LtgtOVt3zvgjoyA+Q8/4lfKTL2PuPiEiJSKytYP5t4jI\nZhHZIiIfi8hEr3kHnOkbRWStmw130+zhmeSkxIV3umf4xfbu2L6W7mmtzOnDGb9nOQ38SvnFlzP+\nJ4ElnczfD8w3xowHfgwsbTP/ImPMJGPMtJ41MfAiI4SrJg9m+a5Syqrrg92cwIhPhyEz+175Bl9K\nMnvTsg1K+a3LwG+MWQF02A/QGPOxMcbTJWYVkOdS23rVtZPzaG4x/HVTHy3h4IaRi+DYZqg8GuyW\nnFFdYtM3kdG+LZ+YpaNwKeUnt3P8dwKveb03wJsisk5E7nL5s1x1Xk4y43JT+m4JBzeMWmyf97wd\n3HZ4qynxPc0DWq9HKRe4FvhF5CJs4P+21+R5xpgpwGXAPSLSYaUxEblLRNaKyNrS0uD8YV8zOY8t\nxRXsPt7H69f31IAxkJLbt8o31JR1PgBLW4nZtlR1cx+tqqpUCHAl8IvIBOBx4CpjTLlnujGm2Hku\nAf4MzOhoG8aYpcaYacaYadnZ3TgDdNGVEwcTGSG81FdLOPhLxKZ79i6HpoZgt8aq7sEZP8Dp8s6X\nU0p1yO/ALyL5wEvAbcaYXV7TE0Uk2fMauBRot2dQX5GdHMv8Udn8ZUMxzS1hWsJhxCJoqIKi1cFu\niVVT2v0zfs96Sqke8aU753PASuA8ESkSkTtF5G4RudtZ5HtAJvDbNt02BwIfisgmYDXwd2PM6wHY\nB1ddMzmXoxV1rNoXpmeUhRdCRFTfKN/QWAf1lb736AEN/Eq5IKqrBYwxN3cx/0vAl9qZvg+YeO4a\nfduiMQNJjo3ixfVFzB3RjYAUKuJSbLfOPW/DJd8Pbltau3L25Ixfe/Yo1VN6524bcdGRfGbCIF7f\neozTDU3Bbk5gDL/YduusDvKdyjXO53cr1aMVOpXylwb+dlwzOZfTDc28se1YsJsSGCMW2ue97wW3\nHdXdvGsXIC7Npqo08CvVYxr42zG9IIO89Pjw7dOfM9FW6wx2f/7ulmsA2zNJ795Vyi8a+NsRESFc\nOzmXD/eUcayiLtjNcV9EhE337H03uOPX9iTVA/agpYFfqR7TwN+Ba6bkYQy8vDFMz/pHLLSlD45t\nDl4bqkshJhmi47u3nt69q5RfNPB3oDArkcn5aby4PkyHZRx+sX0OZrqnpqR7XTk9tEKnUn7RwN+J\na6fkset4NduOVAa7Ke5LGgA5E2y6J1i6e/OWh+b4lfKLBv5OXDFhEDGRETz58YFgNyUwRiyEw59A\nXZAObNWl3buw65GYBY010FDjfpuU6gc08HciLSGGO+YWsGxdERsPnwp2c9w34hJoaYL9K4Lz+TUl\nPT/jh+6f9Z8+AYc+6f7nKRVmNPB34R8vHkF2cizff2UbLeFWvydvBsQkwd4glG9obrKBuEdn/D0M\n/O/9P3hiMRwN4gVtpfoADfxdSI6L5oEl57Pp8CleDLehGaNibO2ePW9Db1/APl0OGD8Dfzcv8O55\n237m69/p/f1Vqg/RwO+DaybnMjk/jf9+fSeVdWFWB37EQjh1CMr39u7n9rQPP/SsbMOJ/XByPwye\nDAc/hO0vd/9zlQoTGvh9EBEh/PDKsZTX1PPg27uD3Rx3DfeUb+jldI+nTlBPL+5C94Zg3OeUp7j6\nURg4Dt78d2is7f5nKxUGNPD7aEJeGjdOG8KTHx9gT0kYjdCVUQgZw3q/P78nP9+dypweMYkQndC9\nHP/edyF1CGSfB0v+CyoOwcqHuv/ZSoUBDfzd8K3F5xEfE8kP/7o9vG7qGnEJHPgQmup77zNbUz09\nHG2tO3fvNjfBvhUw/CJb66fwQhh9BXzwS6g80rPPVyqEaeDvhqykWO5fNIoPdpfx5vbjwW6Oe4Yv\nhMbTcGhl731mdQlExkJsSs/W787du0fWQ33FmbuVARb9GFqa4e0f9OzzlQphGvi76dZZQxk1MIkf\n/207dY3NwW6OOwrmQUR076Z7asps8Bbp2frdCfx73wUECuefmZZRCLPvgc3/B4fX9KwNSoUoDfzd\nFB0ZwQ+uGEvRyVqWrtgX7Oa4IzYJhs6GPb1YvqGmpOdpHnBSPT7m+Pe+Z3vzJGScPf2C+yEpB17/\ndnCrlPojnFKOqtf4FPhF5AkRKRGRdgdLF+tBEdkjIptFZIrXvNtFZLfzuN2thgfTnBFZXD4+h98u\n30PxqTDpGTJ8IZRs672cd3VJzy7senjO+LsKfHUVULTm7DSPR2wyXPIDKF5nz/xDzekT8PORsPHZ\nYLdEhRhfz/ifBJZ0Mv8yYKTzuAt4BEBEMoDvAzOBGcD3RSS9p43tS757+WgA/t/fdwS5JS5pHZWr\nl876a3pYp8cjMduWm6jropTG/g/ANLcf+AEm3Ai5U22uv7665+0Jho3P2H/H7a8EuyUqxPgU+I0x\nK4ATnSxyFfC0sVYBaSIyCFgMvGWMOWGMOQm8RecHkJCRl57AV+eP4O9bjvLx3jCoFDlwHCQNhD29\n0J/fGKcyp5+BH7pO9+x915alyJve/vyICFjy31B9DD78Zc/b09tammHN4/b1gQ9tzyWlfORWjj8X\nOOz1vsiZ1tH0c4jIXSKyVkTWlpaGRq31r8wfRl56PD98ZTtNzSGaI/YQsemeve/aoBJItSft2bo/\nqZ6ETPvsS+AvmGfLU3RkyHR75v/xQ/YO31Cw5204eQDGXAUNVbbnklI+6jMXd40xS40x04wx07Kz\n/TgT7EVx0ZH822fGsPN4FX9cdTDYzfHfiIU2dXJkQ2A/x9MbpyflGjx8qdfjKdPQUZrH2yU/gIhI\neOvfe96m3rR6qb0wfdlP7ft97we3PSqkuBX4i4EhXu/znGkdTQ8bi8cOZN6ILH751i7Kq3vxBqhA\nGH4xIIFP97SWa+jB6FsevgQq+GZGAAAgAElEQVR+T5kGXwJ/ymCYdz/s+GvwylT7qnyvPeOf9kVI\nzoGc8bBvebBbpUKIW4H/FeDzTu+eWUCFMeYo8AZwqYikOxd1L3WmhQ0R4ftXjOF0QzM/f3NnsJvj\nn4QM2+0x0P35PcE60KkeT5mGzBG+bXPOvZCab6t39uWc+ZrH7X0XU++w74ctgKLV0HA6iI1SocTX\n7pzPASuB80SkSETuFJG7ReRuZ5FXgX3AHuAx4GsAxpgTwI+BNc7jR860sDJyYDJ3zCngudWHeWHN\n4a5X6MtGXALFa20ePlDcSPVExUBcWsdn/G3LNPgiOh4u/TEc3wrrn+p52wKpvho2PGNz+8kD7bTC\nBdDc0Lt3XquQFuXLQsaYm7uYb4B7Opj3BPBE95sWWv55yXnsKqnm2y9tJjY6gqsmtXsNu+8bsRBW\n/NTmjMdeHZjPqC4BiYB4P3v2dnb3bntlGnwx5ioYOg/e/Q8Yd63/bXTblhfsfs2468y0obPtL4D9\n75/plqtUJ/rMxd1QFxsVye9uncr0ggzuf2ETb247Fuwm9UzuNIhNDWy6p6YUErLsxVR/dDboentl\nGnwhYqt31p6Ejx70r31uMwZWPwY5E2DIjDPTYxLte83zKx9p4HdRfEwkT9wxnXG5qdz77AY+2B0a\n3VLPEhkFw+bbwBmocgA1pf6leTw6q9DZUZkGXwyaYM/81/we6vtQCe6DH0HJdnu23zZ9VTjfDil5\nOuwyqSoANPC7LCk2iqe+MJ1h2Yl8+em1rN4fgn+IIxZCZTGUfhqY7VeX+HfXrkdHqZ7OyjT4au59\nNqWyrg/l+lcvtamn8dedO2/YfMDAgQ96vVkq9GjgD4C0hBj++KWZDE6L54tPrmHT4S7KCvQ1nlG5\nAtWts8atwJ9lUzJte+B0VabBF7lToeACWPVbaGrwr51uqCiGHX+DybfZi9Bt5U61dyhrf37lAw38\nAZKVFMuzX5pFemI0n39iNTuOVga7Sb5LGwJZ5wVuOMaaMpdSPdmAgdo2v6q6KtPgq7nfsL98tr7o\n33bcsO4PYFpg+p3tz4+MhqFzNc+vfKKBP4ByUuN49kuziI+O5Lbff8K+0hAqAjZiIRz4yP2+4fXV\ndtAXt8744dx0jy9lGnwxYiEMGAsf/Tq45Y+b6mHdkzBqCaQXdLzcsPlwYi9UFPVWy1SI0sAfYEMy\nEvjjl2ZiDNzy+CccPhEiN9kMXwjN9bD2CXvRsLrEnRo+NX4Mst5We3fvdqdMQ1dEYO7XoXQH7H7L\n/+311PaX7T7O+HLny3l6MGm6R3XBp378yj8jBiTxv3fO5ObHVnHL45/wwldmk5MaF+xmda5gru3W\n+ea/npkmETbYJg2wlTyTBjqvc2DgGDuWbVc83S9dS/VwdpfO7pRp8MW4a+GdH8FHv4JRl7qzze5a\nvdTefTzsos6XGzDGdpPdtxwm39IrTVOhSQN/LxkzOIWnvjiDWx5bxS2Pr+L/vjKbrKTYYDerY9Hx\ncN96Wxem+nibR4l9Ltlhn1uci6tT77AljqM7OahVB/iMv7tlGroSGW2HaHzjO3aIxiF+XjforuL1\ntofSZT+1JaQ7ExFh0z3737epqZ4Oa6nCngb+XjRpSBpP3DGd2/+wmit+8yEP3jyZ6QU96GfeWxKz\nui6k1tJie9asfMjWsy9eBzc8DRnD2l/ek+px44w/Lg0k8kzg95RpGHuVu0Fvyufh/f+Gj38NN/7R\nve36Ys3j9kL1xE5vnj+jcL69GF26EwacH9i2qZClOf5eNnNYJsvunkNMVAQ3LV3Fw+/toaUlhMdN\njYiAxEy45PvwuRfg1GH43fyOR4WqdoJ0gh+VOc/6bK+buHpapqErsUkw/Uu2O2XZnp5twxhY/7Q9\nMPqqphy2LIOJN0Fcim/rDHPy/Ps1z686poE/CMblpvK3f5zH5eMH8bM3dnL7H1ZTWhXiJZ0BRi2G\nuz+ArJHwwm22ymXbPvA1pfZM3d8eNx7eZRt6WqbBFzO/ApExsPI3PVt/+X/BK/8Ij10Mz94IRzZ2\nvc6Gp+0F9uldXNT1ll4AaUO1W6fqlAb+IEmOi+bBmybxX9eOZ/X+E1z+4Ad8vCcMhnBMy4cvvA4z\n77Y3Pz15uf0V4FFT4k6axyMh0yvw+1GmoStJA2DS52Djc1B1vHvrbvijTRVN/Bws/B4cWgVL58Pz\nt8CxLe2v09JsS0YUXtj9lM2wBToco+qUBv4gEhFunpHPy/fOJSUuilt+/wm/fGsXzaGc+gF7Nn/Z\nf8P1T0HJp/C7C850h6z2c5D1tjxlG9wo09CVOf9oyx+v/p3v6+x9D/76dRuMr3wQLvgn+MZmuOhf\n7R3Gj86DFz4Px7efvd6u16Hi8NlVOH01bD7UV8JRH35VqH5JA38fcH5OCn/9x3l8dkoeD76zm889\ntopjFXXBbpb/xl4NX3kfUnLhmetst8jqYwEI/GXulGnoSuZwGH2FveDqS/G249ttUM8aZS94R0bb\n6XGpMP9f7AFg/rdhz7vwyBz40xfsRVmwXThT8mDUZd1vZ2t//uXdX1f1Cxr4+4iEmCh+fv1EfnH9\nRLYUV3D5gx+wfGdJsJvlv8zh8KW3bc+YD34BJ/a5m+pJzLKDje981Z0yDV2Z+3X762L9050vV3kU\nnrkeohPglj/ZYN9WfBpc9F17ALjgftj1Bjw806aA9i2H6V+01VK7KzELBo7TwK86pIG/j/ns1Dxe\nuXceA5JjueMPa/h/r+7gdEOI52qj4+HK38DVj9hAmDPevW17fj1sf8WdMg1dyZtmB2pZ+TA0N7a/\nTH01PHej7eZ6ywuQmtf5NhMybO7/G5ttVdC970JUHEy5veftHLYADq+Gxtqeb0OFLV+HXlwiIjtF\nZI+IPNDO/P8RkY3OY5eInPKa1+w1r4M+fsrbiAFJ/OWeudwyM5+lK/Zx0c+X89L6otDu9gn24uh3\nimDSre5t0xP4G6oCm+bxNvfrHRdva26CZV+0F22vfxIGTfR9u4lZsOhH8PXN8JUV/g1GXzjf9gg6\ntKrn21Bhq8vALyKRwMPAZcAY4GYRGeO9jDHmm8aYScaYScBvgJe8Ztd65hljrnSx7WEtLjqS/7xm\nPMvunk1OShz3v7CJa377EesOhmB9f28RkV3fgdod3tcLeivwj1xkyyO0Ld5mDLz2L7D7Dbj85z0v\n8ZCUDdnn+dfGoXMgIkrTPapdvvwFzgD2GGP2GWMagOeBqzpZ/mbgOTcap2BaQQZ//tpcfnnDRI5V\n1vHZR1Zy77PrKToZIsXeAs1zVuxmmYauiMCc++xoWN5DVK58CNb+3v4i6Kh8cm+JTbLDaOqNXKod\nvgT+XMCrIzZFzrRziMhQoBB412tynIisFZFVIhKg0bvDW0SEcO2UPN771gK+vnAkb+84zsW/eJ+f\nvfEp1fUhnv/3l+eMf/hFvVubZtxnbW+lj35t32/7C7z5bzDmalj4g95rR2eGLbA3itWeDHZLVB/j\n9sXdm4Blxhjv+r1DjTHTgM8BvxKR4e2tKCJ3OQeItaWlIThWbS9IiInim4tG8e4/LeDycTk8/N5e\nLvr5cl5Yezj08/89FZsEn/klzPtm735uVAzM+pod6nDVo/DSXTBkJlzzO3dTWf5oHY7xw2C3RPUx\nvvwPLQaGeL3Pc6a15ybapHmMMcXO8z5gOTC5vRWNMUuNMdOMMdOys13s5x2GBqfF86ubJvPnr80h\nLz2ef1m2mSse+pAVu0oxwRwwJFim39lxUbhAmnq7LV39+rchNRdueq7zyqS9LXea7UWleX7Vhi+B\nfw0wUkQKRSQGG9zP6Z0jIucD6cBKr2npIhLrvM4C5gLb266remZyfjovfXUOv75pEqdON/L5J1Zz\n7SMfs3xnSf88APS22GS44JuQPBhuWWaL1fUlUTHOcIya51dn6zLwG2OagHuBN4AdwAvGmG0i8iMR\n8e6lcxPwvDk74owG1orIJuA94CfGGA38LhIRrpqUy7vfms9/XjOOksp67vjDGq5++CPe2XFcDwCB\nNu+b8M2t9ka1vmjYfCjfbQdrV8ohfTEwTJs2zaxduzbYzQhJDU0tvLS+iIeX7+HwiVrG5aZw38Uj\nWTRmIKIDc/Q/RzfbWklXP2Lvo1BhS0TWOddTu9RHrkIpt8RERXDTjHze/acF/Oy6CVTVNXHX/67j\n8gc/5LUtR/vvReD+auA4W8FU0z3Kiwb+MBUdGcH104bwzv3z+eUNE6lvbOarz6znsl9/wCubjtDQ\n1BLsJqreEBFhSzt7hmNUCg38YS8qMoJrp+Tx1v3z+fVNk2hqaeG+5zYw+7/e4b9e28H+sppgN1EF\nWuF8qDoKZbuD3RLVR+iYu/1EZIS9CPwPEwazYncpz68+xOMf7Od37+9j9rBMbp6Zz+KxA4mNigx2\nU5XbhnmVac4eFdSmqL5BA38/ExkhXHTeAC46bwAllXX8aV0Rz685xH3PbSA9IZrPTsnjphn5jBiQ\nFOymKrekF0Jqvk33zOzBwC4q7GivHkVLi+GjvWU8v/owb24/RmOzYXpBOjfPyOeycYOIj9FfASHv\n5XttWYmvb/Sv6qfqs7rTq0cDvzpLWXU9L64r4rnVhzhQfpr46EgWjh7AP0wYzILzsomL1oNASCr5\nFB6dCxNuhKt/G+zWqADQwK/8Zozhk/0n+OumI7y29RgnahpIio1i0ZiB/MOEQVwwMpuYKO0bEFLe\n+j589Cv4wmu2bLMKKxr4lauamltYua+cv206yuvbjlFR20hKXBSLx+bwDxMHM2d4JtGRehDo8xpq\n4OFZEJMId39wZgxgFRY08KuAaWhq4aM9Zfx18xHe2nacqvom0hOiWTRmIHNHZDGzMJOc1D5UqEyd\nbedr8NxNcMkPYd43gt0a5SIN/KpX1DU2s2JXKX/bfJT3dpZQVWfHBijMSmRmYQazhmUya5geCPqc\n5z4H+96Dez6BtPxgt0a5RAO/6nXNLYYdRytZta+cVfvK+WT/idYDQUFmQutBYOawDAalxge5tf3c\nqcPw8Aw7UMvNOlheuNDAr4Lu7APBCT7ZX956IDg/J5nFY3NYMi6H83OStXhcMHz4K3j7+3YMgfMv\nD3ZrlAs08Ks+x3Mg+HhvGW9vL2HNwRMYA0MzE1gyNofF43KYlJdGRIQeBHpFcyM8egE0VNuUT0xi\nsFuk/KSBX/V5pVX1vLX9OK9vO8bHe8poajEMTIm1vwTG5jCjMIMo7SkUWAc/hj9cBnO/AYt+GOzW\nKD9p4FchpeJ0I+/uPM7rW4/x/q5S6hpbSE+I5uLzBzL/vGzmjcgiIzEm2M0MT3+5BzY/D1/5AAaO\nCXZrlB808KuQdbqhiRW7Snlj23He/bSEitpGRGB8bioXjszmwlHZTM5P0/sG3FJTDg9NhezR8IVX\nQa+3hCwN/CosNLcYNhedYsWuMj7YXcqGw6dobjEkxUYxe3gmF47M4sJR2QzN1Py0X9Y9BX+9D676\nLUy+JditUT3keuAXkSXAr4FI4HFjzE/azL8D+BngGdjzIWPM486824F/c6b/hzHmqa4+TwO/ak9F\nbSMr95axYncZK3aVUnSyFrAXiGcUZDBlaDqT89MYOSCZSL1I7LuWFnhiMZzYC/euhYSMYLdI9YCr\ngV9EIoFdwCKgCFgD3Ow9aLoT+KcZY+5ts24GsBaYBhhgHTDVGHOys8/UwK+6Yoxhf1kNH+y2vwbW\nHTzJydONACTFRjFxSCqTh9gDweT8dL1G0JVjW+F3F9oz/it/E+zWqB7oTuD3pR7/DGCPMWafs/Hn\ngauA7Z2uZS0G3jLGnHDWfQtYAuhdI8ovIsKw7CSGZSdx+5wCjDEcKD/NhkMn2XDoFOsPneSR9/fS\n7IwxXJCZwJT8dCblpzF2cCqjByWTEKPDUbTKGQezvgorH4JJt0L+zGC3SAWQL//zc4HDXu+LgPb+\nV3xWRC7E/jr4pjHmcAfr5vawrUp1SEQozEqkMCuRa6fkAfZC8eaiitYDwYrdpby0wWYjIwSGZScx\nbnAK43JTGTM4hbGDU0mN78eFyxZ8B7b9Gf5+P9z1PkTqgTFcufXN/hV4zhhTLyJfAZ4CLu7OBkTk\nLuAugPx8rR+i/JcQE9VaKgJseuhIRR3biivYeqSSbcUVrNp3gr9sPNK6Tn5GAuNy7UFgzKAUzh+U\nTE5KXP+4uzg2CZb8BF64DX46DNKHQnqB13MBpBVA2hCIig1uW5VffAn8xcAQr/d5nLmIC4Axptzr\n7ePAT73WXdBm3eXtfYgxZimwFGyO34d2KdUtIkJuWjy5afFcOjandXppVT3bjlSw7Ugl245UsLW4\nkle3HGudnxofzfk5yfYxKIXzc5I5LydMU0Wjr4BrH4PDq+HkASj9FHa9Ac31XgsJpOTaA0HWSMif\nbev7pw3pYKOqr/Hl4m4UNn2zEBvI1wCfM8Zs81pmkDHmqPP6GuDbxphZzsXddcAUZ9H12Iu7Jzr7\nTL24q4KtoraRXcer+PRoJTuO2eedx6qoaWgGbHf3oRkJnJ9jfxWMHZzK2MEpDEoNw18HLS1QfQxO\nHrQHg5MH4JTz+vh2qK+wy6Xm2wOA55E5wvf7AoyB0+VQWQyNdZCcA8mDIEovyvvK1Yu7xpgmEbkX\neAPbnfMJY8w2EfkRsNYY8wpwn4hcCTQBJ4A7nHVPiMiPsQcLgB91FfSV6gtS46OZXpDB9IIzXRtb\nWgxFJ2v59Fglnx6rss9Hq3hj+zE8508ZiTGMGZTCWCddNHZwCoWZiaFdgygiAlIG28fQ2WfPa2mG\nku22/MPBj2DvO/ZOYIDEbOcgMBeGzLAHkMpiqDwCVUfsc+URZ9rRNr8qOLON5EFnPj95MKQ475Ny\n7Dq1J9s8TrV5PgkRkTBoIgyeDIMm2YvZ0T2oEtvSAhWHoXy33X5cKsSlQXzamecQGOBGb+BSyk+n\nG5rYcbTKpouKK9l2tIJdx6ppaG4BICEmktGDUhg7OIXxualMyEtjeHZieNYiMgbK99qDwMGP7aPi\n0LnLRcY4wTy3TVAfbANy1VF7MGg9QDivT5efu622ohMhPt0GYc9zYx0c3Qg1pXYZiYQBY2Cw52Aw\nGQaOhWhn7Ij6KijbDeV77HPZLvu6fA801fn++XFp9uAQHQeRsfagEBnjPLxeRznPsckw+dbu/Zs7\n9M5dpYKsoamFPSXVrdcOth+pZPvRSqrrbWnq+OhIeyDIS2VCXirjc9MYlhXivww6cuoQFK2F6IQz\nQT4hs2flIRrr7EGh6ihUHYOoOCfIpp8Jth1deDbG/ro4shGObLAHgiMbzhxMIqIga5T9hVB19Mx6\nEuFczxhl01dZI+3rhCyoq4A6r18YdafOfq49aZdpqrMVUZsbnEej/bXS3HB2G5MGwrd2df/fBQ38\nSvVJLS2G/eU1bCmqYFPRKbYUVbD1SAV1jfaXQVJsFONyU5iQl8Z5A5MZmBLHgJRYBiTHkhofHX7X\nDvoCY2zqxnMwOL7NHpSyRtpH5kjIKAxcLyZjoKXpzAGhpRkSs3q0KQ38SoWIpuYW9pbWtB4INhdX\nsONIZWuayCMmKoLspNjWA8GA5DgGJMcyMCWO/MwEhmcnkZUUoweHfsztO3eVUgESFRnBeU730Bum\n2e6QDU0tHD55mpLKekqq6iitqqekqp6SyjpKq+vZV1rDqn0nqKhtPGtbqfHRDM9OZHh2EsMHJNnn\n7ETyMxLC83qC6jEN/Er1MTFREU7QTup0ubrGZkqr6tlfVsPe0mr7KKnh/V2l/GldUety0ZFCQWYi\nMwoz+M7lo0mK1T/7/k7/BygVouKiIxmSkcCQjAQuHJV91ryK2kb2lVazt9QeFHYfr+b5NYf5ZP8J\nlt42lWFdHFRUeNMcv1L9xMd7y7j32Q00NrXwq5smsXD0wGA3SbmoOzl+Tfwp1U/MGZ7FK/fOZWhW\nAl96ei0PvrOblpa+d+KnAk8Dv1L9SF56AsvunsPVk3L55Vu7uPuP61rvLVD9hwZ+pfqZuOhIfnnD\nRL73D2N459MSrn74I/aWVge7WaoXaeBXqh8SEb44r5D/vXMGJ2oauPqhj3hnx/FgN0v1Eg38SvVj\n3nn/O59ay6/f1rx/f6CBX6l+zpP3v2ZyLv/zts37H62oDXazVABpP36lVGvef3xuKv/56g7e3H6c\nSUPSuGxcDkvG5TA0MzHYTVQu0n78Sqmz7C+r4dUtR3l96zG2FNtBVs7PSWbJuBwuGzeIUQOTtCZQ\nH6RF2pRSrig6eZrXtx7jjW3HWHvwJMZAYVYii8fmcNm4HCbkpepBoI/QwK+Ucl1JVR1vbjvOG9uO\nsXJvOU0thtT4aHKc8tHZTtXQgSlO9VCvSqLxMZHBbn7Y08CvlAqoU6cbeHtHCRsOnbSVQ6vqKXWq\nhzY2nxtTkuOiGJwaz+C0OHLT48lNS3Ce48lLjyc7KTY8B6HpRa6XZRaRJcCvsWPuPm6M+Umb+fcD\nX8KOuVsKfNEYc9CZ1wxscRY9ZIy50qe9UEr1WWkJMVw3NY/rpuadNb2lxXCqtpHjlXWtpaQ9z0cq\n6ig+Wcv6Q6fOKSkdHSkMSrUHguzkWOKiI4iNiiQ2KoJY79dREcRGe15Hkp4QTVZyLFlJsaTFR+vB\nw0ddBn4RiQQeBhYBRcAaEXnFGLPda7ENwDRjzGkR+SrwU+BGZ16tMWaSy+1WSvVBERFCRmIMGYkx\njB7U8XLV9U0Un6zlyKlaik7VUnyyluJTtRSfPM2molM0NLVQ39RCfWMz9U0tNPlwb0GU89lZSbHO\nwSCG7CTnoJAQ3XrAiPEcQLwOJjFe75PioogO8/ELfDnjnwHsMcbsAxCR54GrgNbAb4x5z2v5VUDP\nRgtWSvULSbFRrQPQ+KKpuYWG5hbqG50DQlMztY3NnKhpoKy6gbKqesqqPY8Gyqrr2XO8irLqhnNG\nM/NFYkwkqfHRpMRHk9reIyGalLhoEmIiSYqNIiE2iqTYSBJiokiMjSIxJrJPD37jS+DPBQ57vS8C\nZnay/J3Aa17v40RkLTYN9BNjzF+63UqlVL8WFRlBVGQECTHdW88YQ2VdE5W1ja0HjNZfE00tzusz\n0+oam6mqa6KitvGsx8Hy062vaxubffrs2KgIEmOjSIiJJMpJQXl6QLUmpOSsJzITY3nh7tnd28ke\ncPUGLhG5FZgGzPeaPNQYUywiw4B3RWSLMWZvO+veBdwFkJ+f72azlFL9lIi0nqW7pb6pmcraJirr\nGjld30xNQxM19U3UNDTb5/omauqbOd3QRHV9E6cbmmkxBk8/Gk/SytOxpjWJZexF8N7gy6cUA0O8\n3uc5084iIpcA/wrMN8bUe6YbY4qd530ishyYDJwT+I0xS4GlYHv1+L4LSinVe2KjIslOjiQ7OTbY\nTekxX5JQa4CRIlIoIjHATcAr3guIyGTgd8CVxpgSr+npIhLrvM4C5uJ1bUAppVTv6/KM3xjTJCL3\nAm9gu3M+YYzZJiI/AtYaY14BfgYkAX9ycliebpujgd+JSAv2IPOTNr2BlFJK9TK9gUsppcKAjrmr\nlFKqQxr4lVKqn9HAr5RS/YwGfqWU6mc08CulVD/TJ3v1iEgpcLCHq2cBZS42J9jCbX8g/PYp3PYH\nwm+fwm1/4Nx9GmqMyfZlxT4Z+P0hImt97dIUCsJtfyD89inc9gfCb5/CbX/Av33SVI9SSvUzGviV\nUqqfCcfAvzTYDXBZuO0PhN8+hdv+QPjtU7jtD/ixT2GX41dKKdW5cDzjV0op1YmwCfwiskREdorI\nHhF5INjtcYOIHBCRLSKy0RnFLOSIyBMiUiIiW72mZYjIWyKy23lOD2Ybu6OD/fmBiBQ739NGEbk8\nmG3sDhEZIiLvich2EdkmIl93pofyd9TRPoXk9yQicSKyWkQ2OfvzQ2d6oYh84sS8/3PK5vu2zXBI\n9TgDwu/Ca0B44OZQLwEtIgewg9iHbP9jEbkQqAaeNsaMc6b9FDhhjPmJc5BON8Z8O5jt9FUH+/MD\noNoY8/Ngtq0nRGQQMMgYs15EkoF1wNXAHYTud9TRPt1ACH5PYmvdJxpjqkUkGvgQ+DpwP/CSMeZ5\nEXkU2GSMecSXbYbLGX/rgPDGmAbAMyC8CjJjzArgRJvJVwFPOa+fwv5RhoQO9idkGWOOGmPWO6+r\ngB3YcbZD+TvqaJ9CkrGqnbfRzsMAFwPLnOnd+o7CJfC3NyB8yH7RXgzwpoisc8YkDhcDjTFHndfH\ngIHBbIxL7hWRzU4qKGTSIt5EpAA7NOonhMl31GafIES/JxGJFJGNQAnwFnb42lPGmCZnkW7FvHAJ\n/OFqnjFmCnAZcI+TZggrxuYaQz3f+AgwHJgEHAV+EdzmdJ+IJAEvAt8wxlR6zwvV76idfQrZ78kY\n02yMmYQd83wGcL4/2wuXwO/TgPChxmug+hLgz9gvPBwcd/KwnnxsSRfL92nGmOPOH2YL8Bgh9j05\neeMXgWeMMS85k0P6O2pvn0L9ewIwxpwC3gNmA2ki4hk+t1sxL1wCf5cDwocaEUl0LkwhIonApcDW\nztcKGa8AtzuvbwdeDmJb/OYJkI5rCKHvyblw+HtghzHml16zQvY76mifQvV7EpFsEUlzXsdjO7Hs\nwB4ArnMW69Z3FBa9egCcrlm/4syA8P8Z5Cb5RUSGYc/yAaKAZ0Nxn0TkOWABtpLgceD7wF+AF4B8\nbBXWG4wxIXHBtIP9WYBNHxjgAPAVr/x4nyYi84APgC1AizP5u9iceKh+Rx3t082E4PckIhOwF28j\nsSfrLxhjfuTEiOeBDGADcKsxpt6nbYZL4FdKKeWbcEn1KKWU8pEGfqWU6mc08CulVD+jgV8ppfoZ\nDfxKKdXPaOBXykUiskBE/hbsdijVGQ38SinVz2jgV/2SiNzq1DjfKCK/c4pgVYvI/zg1z98RkWxn\n2Ukissop7vVnT3EvERkhIm87ddLXi8hwZ/NJIrJMRD4VkWecO0mV6jM08Kt+R0RGAzcCc53CV83A\nLUAisNYYMxZ4H3tXLmER1uUAAAFDSURBVMDTwLeNMROwd4N6pj8DPGyMmQjMwRb+AlsN8hvAGGAY\nMDfgO6VUN0R1vYhSYWchMBVY45yMx2OLkLUA/+cs80fgJRFJBdKMMe87058C/uTUUco1xvwZwBhT\nB+Bsb7Uxpsh5vxEowA6eoVSfoIFf9UcCPGWM+c5ZE0X+vc1yPa1n4l0vpRn9O1N9jKZ6VH/0DnCd\niAyA1vFlh2L/HjzVDj8HfGiMqQBOisgFzvTbgPedkZ2KRORqZxuxIpLQq3uhVA/pmYjqd4wx20Xk\n37Cjm0UAjcA9QA0ww5lXgr0OALbk7aNOYN8HfMGZfhvwOxH5kbON63txN5TqMa3OqZRDRKqNMUnB\nbodSgaapHqWU6mf0jF8ppfoZPeNXSql+RgO/Ukr1Mxr4lVKqn9HAr5RS/YwGfqWU6mc08CulVD/z\n/wHZSMROwDmVJgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe8a64ac1d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(train_losses, label='train')\n",
    "plt.plot(valid_losses, label='valid')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(loc='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里我们只训练了 30 次，在 20 次的时候进行了学习率衰减，可以看 loss 曲线在 20 次的时候不管是 train loss 还是 valid loss，都有了一个陡降。\n",
    "\n",
    "当然这里我们只是作为举例，在实际应用中，做学习率衰减之前应该经过充分的训练，比如训练 80 次或者 100 次，然后再做学习率衰减得到更好的结果，有的时候甚至需要做多次学习率衰减"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}