{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# PyTorch快速入门\n", "\n", "PyTorch的简洁设计使得它入门很简单，在深入介绍PyTorch之前，本节将先介绍一些PyTorch的基础知识，使得读者能够对PyTorch有一个大致的了解，并能够用PyTorch搭建一个简单的神经网络。部分内容读者可能暂时不太理解，可先不予以深究，后续的课程将会对此进行深入讲解。\n", "\n", "本节内容参考了PyTorch官方教程[^1]并做了相应的增删修改，使得内容更贴合新版本的PyTorch接口，同时也更适合新手快速入门。另外本书需要读者先掌握基础的Numpy使用，其他相关知识推荐读者参考CS231n的教程[^2]。\n", "\n", "[^1]: http://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html\n", "[^2]: http://cs231n.github.io/python-numpy-tutorial/" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Tensor\n", "\n", "Tensor是PyTorch中重要的数据结构，可认为是一个高维数组。它可以是一个数（标量）、一维数组（向量）、二维数组（矩阵）以及更高维的数组。Tensor和Numpy的ndarrays类似，但Tensor可以使用GPU进行加速。Tensor的使用和Numpy及Matlab的接口十分相似，下面通过几个例子来看看Tensor的基本使用。" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "from __future__ import print_function\n", "import torch as t" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([[1.2563e-37, 0.0000e+00, 5.7453e-44],\n", " [0.0000e+00, nan, 4.5814e-41],\n", " [1.3733e-14, 6.4076e+07, 2.0706e-19],\n", " [7.3909e+22, 2.4176e-12, 1.1625e+33],\n", " [8.9605e-01, 1.1632e+33, 5.6003e-02]])" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 构建 5x3 矩阵，只是分配了空间，未初始化\n", "x = t.Tensor(5, 3) \n", "x" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([[0.7149, 0.6065, 0.8056],\n", " [0.2450, 0.1942, 0.5305],\n", " [0.6735, 0.7798, 0.6060],\n", " [0.1072, 0.8325, 0.8617],\n", " [0.5117, 0.2246, 0.4984]])" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 使用[0,1]均匀分布随机初始化二维数组\n", "x = t.rand(5, 3) \n", "x" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "torch.Size([5, 3])\n" ] }, { "data": { "text/plain": [ "(3, 3)" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "print(x.size()) # 查看x的形状\n", "x.size()[1], x.size(1) # 查看列的个数, 两种写法等价" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`torch.Size` 是tuple对象的子类，因此它支持tuple的所有操作，如x.size()[0]" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([[1.6605, 1.1155, 1.2724],\n", " [0.6727, 0.6428, 1.0969],\n", " [1.4898, 1.7437, 1.3258],\n", " [0.8030, 1.5725, 1.4709],\n", " [0.6847, 0.4828, 0.6183]])" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "y = t.rand(5, 3)\n", "# 加法的第一种写法\n", "x + y" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "\n", " 0.4063 0.7378 1.2411\n", " 0.0687 0.7725 0.0634\n", " 1.1016 1.4291 0.7324\n", " 0.7604 1.2880 0.4597\n", " 0.6020 1.0124 1.0185\n", "[torch.FloatTensor of size 5x3]" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 加法的第二种写法\n", "t.add(x, y)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([[1.7112, 1.2969, 0.3289],\n", " [0.7841, 1.0128, 0.7596],\n", " [1.1364, 1.1541, 0.8970],\n", " [0.8831, 0.7063, 0.3158],\n", " [1.5160, 1.3610, 0.8437]])" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 加法的第三种写法：指定加法结果的输出目标为result\n", "result = t.Tensor(5, 3) # 预先分配空间\n", "t.add(x, y, out=result) # 输入到result\n", "result" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "最初y\n", "tensor([[0.9778, 0.9240, 0.0337],\n", " [0.7461, 0.8548, 0.5141],\n", " [0.5364, 0.9908, 0.1078],\n", " [0.6880, 0.1675, 0.0010],\n", " [0.9120, 0.5539, 0.2896]])\n", "第一种加法，y的结果\n", "tensor([[0.9778, 0.9240, 0.0337],\n", " [0.7461, 0.8548, 0.5141],\n", " [0.5364, 0.9908, 0.1078],\n", " [0.6880, 0.1675, 0.0010],\n", " [0.9120, 0.5539, 0.2896]])\n", "第二种加法，y的结果\n", "tensor([[1.7112, 1.2969, 0.3289],\n", " [0.7841, 1.0128, 0.7596],\n", " [1.1364, 1.1541, 0.8970],\n", " [0.8831, 0.7063, 0.3158],\n", " [1.5160, 1.3610, 0.8437]])\n" ] } ], "source": [ "print('最初y')\n", "print(y)\n", "\n", "print('第一种加法，y的结果')\n", "y.add(x) # 普通加法，不改变y的内容\n", "print(y)\n", "\n", "print('第二种加法，y的结果')\n", "y.add_(x) # inplace 加法，y变了\n", "print(y)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "注意，函数名后面带下划线**`_`** 的函数会修改Tensor本身。例如，`x.add_(y)`和`x.t_()`会改变 `x`，但`x.add(y)`和`x.t()`返回一个新的Tensor，而`x`不变。" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "\n", " 0.2522\n", " 0.7138\n", " 0.6019\n", " 0.3675\n", " 0.5104\n", "[torch.FloatTensor of size 5]" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Tensor的选取操作与Numpy类似\n", "x[:, 1]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Tensor还支持很多操作，包括数学运算、线性代数、选择、切片等等，其接口设计与Numpy极为相似。更详细的使用方法，会在第三章系统讲解。\n", "\n", "Tensor和Numpy的数组之间的互操作非常容易且快速。对于Tensor不支持的操作，可以先转为Numpy数组处理，之后再转回Tensor。" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([1., 1., 1., 1., 1.])" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "a = t.ones(5) # 新建一个全1的Tensor\n", "a" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([1., 1., 1., 1., 1.], dtype=float32)" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "b = a.numpy() # Tensor -> Numpy\n", "b" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[1. 1. 1. 1. 1.]\n", "tensor([1., 1., 1., 1., 1.], dtype=torch.float64)\n" ] } ], "source": [ "import numpy as np\n", "a = np.ones(5)\n", "b = t.from_numpy(a) # Numpy->Tensor\n", "print(a)\n", "print(b) " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Tensor和numpy对象共享内存，所以他们之间的转换很快，而且几乎不会消耗什么资源。但这也意味着，如果其中一个变了，另外一个也会随之改变。" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[2. 2. 2. 2. 2.]\n", "\n", " 2\n", " 2\n", " 2\n", " 2\n", " 2\n", "[torch.DoubleTensor of size 5]\n", "\n" ] } ], "source": [ "b.add_(1) # 以`_`结尾的函数会修改自身\n", "print(a)\n", "print(b) # Tensor和Numpy共享内存" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Tensor可通过`.cuda` 方法转为GPU的Tensor，从而享受GPU带来的加速运算。" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tensor([[1.6605, 1.1155, 1.2724],\n", " [0.6727, 0.6428, 1.0969],\n", " [1.4898, 1.7437, 1.3258],\n", " [0.8030, 1.5725, 1.4709],\n", " [0.6847, 0.4828, 0.6183]], device='cuda:0')\n" ] } ], "source": [ "# 在不支持CUDA的机器下，下一步不会运行\n", "if t.cuda.is_available():\n", " x = x.cuda()\n", " y = y.cuda()\n", " x + y\n", "print(x+y)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "此处可能发现GPU运算的速度并未提升太多，这是因为x和y太小且运算也较为简单，而且将数据从内存转移到显存还需要花费额外的开销。GPU的优势需在大规模数据和复杂运算下才能体现出来。\n", "\n", "### Autograd: 自动微分\n", "\n", "深度学习的算法本质上是通过反向传播求导数，而PyTorch的**`Autograd`**模块则实现了此功能。在Tensor上的所有操作，Autograd都能为它们自动提供微分，避免了手动计算导数的复杂过程。\n", " \n", "`autograd.Variable`是Autograd中的核心类，它简单封装了Tensor，并支持几乎所有Tensor有的操作。Tensor在被封装为Variable之后，可以调用它的`.backward`实现反向传播，自动计算所有梯度。Variable的数据结构如图2-6所示。\n", "\n", "\n", "![图2-6:Variable的数据结构](imgs/autograd_Variable.svg)\n", "\n", "\n", "Variable主要包含三个属性。\n", "- `data`：保存Variable所包含的Tensor\n", "- `grad`：保存`data`对应的梯度，`grad`也是个Variable，而不是Tensor，它和`data`的形状一样。\n", "- `grad_fn`：指向一个`Function`对象，这个`Function`用来反向传播计算输入的梯度，具体细节会在下一章讲解。" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "from torch.autograd import Variable" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "tensor([[1., 1.],\n", " [1., 1.]], requires_grad=True)" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 使用Tensor新建一个Variable\n", "x = Variable(t.ones(2, 2), requires_grad = True)\n", "x" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "tensor(4., grad_fn=)" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "y = x.sum()\n", "y" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "y.grad_fn" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "y.backward() # 反向传播,计算梯度" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([[1., 1.],\n", " [1., 1.]])" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# y = x.sum() = (x[0][0] + x[0][1] + x[1][0] + x[1][1])\n", "# 每个值的梯度都为1\n", "x.grad " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "注意：`grad`在反向传播过程中是累加的(accumulated)，**这意味着每一次运行反向传播，梯度都会累加之前的梯度，所以反向传播之前需把梯度清零。**" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([[2., 2.],\n", " [2., 2.]])" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "y.backward()\n", "x.grad" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "tensor([[3., 3.],\n", " [3., 3.]])" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "y.backward()\n", "x.grad" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([[0., 0.],\n", " [0., 0.]])" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 以下划线结束的函数是inplace操作，就像add_\n", "x.grad.data.zero_()" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([[1., 1.],\n", " [1., 1.]])" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "y.backward()\n", "x.grad" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Variable和Tensor具有近乎一致的接口，在实际使用中可以无缝切换。" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tensor([[0.5403, 0.5403, 0.5403, 0.5403, 0.5403],\n", " [0.5403, 0.5403, 0.5403, 0.5403, 0.5403],\n", " [0.5403, 0.5403, 0.5403, 0.5403, 0.5403],\n", " [0.5403, 0.5403, 0.5403, 0.5403, 0.5403]])\n" ] }, { "data": { "text/plain": [ "tensor([[0.5403, 0.5403, 0.5403, 0.5403, 0.5403],\n", " [0.5403, 0.5403, 0.5403, 0.5403, 0.5403],\n", " [0.5403, 0.5403, 0.5403, 0.5403, 0.5403],\n", " [0.5403, 0.5403, 0.5403, 0.5403, 0.5403]])" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "x = Variable(t.ones(4,5))\n", "y = t.cos(x)\n", "x_tensor_cos = t.cos(x.data)\n", "print(y)\n", "x_tensor_cos" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 神经网络\n", "\n", "Autograd实现了反向传播功能，但是直接用来写深度学习的代码在很多情况下还是稍显复杂，torch.nn是专门为神经网络设计的模块化接口。nn构建于 Autograd之上，可用来定义和运行神经网络。nn.Module是nn中最重要的类，可把它看成是一个网络的封装，包含网络各层定义以及forward方法，调用forward(input)方法，可返回前向传播的结果。下面就以最早的卷积神经网络：LeNet为例，来看看如何用`nn.Module`实现。LeNet的网络结构如图2-7所示。\n", "\n", "![图2-7:LeNet网络结构](imgs/nn_lenet.png)\n", "\n", "这是一个基础的前向传播(feed-forward)网络: 接收输入，经过层层传递运算，得到输出。\n", "\n", "#### 定义网络\n", "\n", "定义网络时，需要继承`nn.Module`，并实现它的forward方法，把网络中具有可学习参数的层放在构造函数`__init__`中。如果某一层(如ReLU)不具有可学习的参数，则既可以放在构造函数中，也可以不放，但建议不放在其中，而在forward中使用`nn.functional`代替。" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Net(\n", " (conv1): Conv2d(1, 6, kernel_size=(5, 5), stride=(1, 1))\n", " (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))\n", " (fc1): Linear(in_features=400, out_features=120, bias=True)\n", " (fc2): Linear(in_features=120, out_features=84, bias=True)\n", " (fc3): Linear(in_features=84, out_features=10, bias=True)\n", ")\n" ] } ], "source": [ "import torch.nn as nn\n", "import torch.nn.functional as F\n", "\n", "class Net(nn.Module):\n", " def __init__(self):\n", " # nn.Module子类的函数必须在构造函数中执行父类的构造函数\n", " # 下式等价于nn.Module.__init__(self)\n", " super(Net, self).__init__()\n", " \n", " # 卷积层 '1'表示输入图片为单通道, '6'表示输出通道数，'5'表示卷积核为5*5\n", " self.conv1 = nn.Conv2d(1, 6, 5) \n", " # 卷积层\n", " self.conv2 = nn.Conv2d(6, 16, 5) \n", " # 仿射层/全连接层，y = Wx + b\n", " self.fc1 = nn.Linear(16*5*5, 120) \n", " self.fc2 = nn.Linear(120, 84)\n", " self.fc3 = nn.Linear(84, 10)\n", "\n", " def forward(self, x): \n", " # 卷积 -> 激活 -> 池化 \n", " x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))\n", " x = F.max_pool2d(F.relu(self.conv2(x)), 2) \n", " # reshape，‘-1’表示自适应\n", " x = x.view(x.size()[0], -1) \n", " x = F.relu(self.fc1(x))\n", " x = F.relu(self.fc2(x))\n", " x = self.fc3(x) \n", " return x\n", "\n", "net = Net()\n", "print(net)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "只要在nn.Module的子类中定义了forward函数，backward函数就会自动被实现(利用`Autograd`)。在`forward` 函数中可使用任何Variable支持的函数，还可以使用if、for循环、print、log等Python语法，写法和标准的Python写法一致。\n", "\n", "网络的可学习参数通过`net.parameters()`返回，`net.named_parameters`可同时返回可学习的参数及名称。" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "10\n" ] } ], "source": [ "params = list(net.parameters())\n", "print(len(params))" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "conv1.weight : torch.Size([6, 1, 5, 5])\n", "conv1.bias : torch.Size([6])\n", "conv2.weight : torch.Size([16, 6, 5, 5])\n", "conv2.bias : torch.Size([16])\n", "fc1.weight : torch.Size([120, 400])\n", "fc1.bias : torch.Size([120])\n", "fc2.weight : torch.Size([84, 120])\n", "fc2.bias : torch.Size([84])\n", "fc3.weight : torch.Size([10, 84])\n", "fc3.bias : torch.Size([10])\n" ] } ], "source": [ "for name,parameters in net.named_parameters():\n", " print(name,':',parameters.size())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "forward函数的输入和输出都是Variable，只有Variable才具有自动求导功能，而Tensor是没有的，所以在输入时，需把Tensor封装成Variable。" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "torch.Size([1, 10])" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "input = Variable(t.randn(1, 1, 32, 32))\n", "out = net(input)\n", "out.size()" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [], "source": [ "net.zero_grad() # 所有参数的梯度清零\n", "out.backward(Variable(t.ones(1,10))) # 反向传播" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "需要注意的是，torch.nn只支持mini-batches，不支持一次只输入一个样本，即一次必须是一个batch。但如果只想输入一个样本，则用 `input.unsqueeze(0)`将batch_size设为１。例如 `nn.Conv2d` 输入必须是4维的，形如$nSamples \\times nChannels \\times Height \\times Width$。可将nSample设为1，即$1 \\times nChannels \\times Height \\times Width$。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 损失函数\n", "\n", "nn实现了神经网络中大多数的损失函数，例如nn.MSELoss用来计算均方误差，nn.CrossEntropyLoss用来计算交叉熵损失。" ] }, { "cell_type": "code", "execution_count": 39, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "tensor(28.3834, grad_fn=)" ] }, "execution_count": 39, "metadata": {}, "output_type": "execute_result" } ], "source": [ "\n", "output = net(input)\n", "target = Variable(t.arange(0,10).float().unsqueeze(0)) \n", "criterion = nn.MSELoss()\n", "loss = criterion(output, target)\n", "loss" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "如果对loss进行反向传播溯源(使用`gradfn`属性)，可看到它的计算图如下：\n", "\n", "```\n", "input -> conv2d -> relu -> maxpool2d -> conv2d -> relu -> maxpool2d \n", " -> view -> linear -> relu -> linear -> relu -> linear \n", " -> MSELoss\n", " -> loss\n", "```\n", "\n", "当调用`loss.backward()`时，该图会动态生成并自动微分，也即会自动计算图中参数(Parameter)的导数。" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "反向传播之前 conv1.bias的梯度\n", "Variable containing:\n", " 0\n", " 0\n", " 0\n", " 0\n", " 0\n", " 0\n", "[torch.FloatTensor of size 6]\n", "\n", "反向传播之后 conv1.bias的梯度\n", "Variable containing:\n", "1.00000e-02 *\n", " -4.2109\n", " -2.7638\n", " -5.8431\n", " 1.3761\n", " -2.4141\n", " -1.2015\n", "[torch.FloatTensor of size 6]\n", "\n" ] } ], "source": [ "# 运行.backward，观察调用之前和调用之后的grad\n", "net.zero_grad() # 把net中所有可学习参数的梯度清零\n", "print('反向传播之前 conv1.bias的梯度')\n", "print(net.conv1.bias.grad)\n", "loss.backward()\n", "print('反向传播之后 conv1.bias的梯度')\n", "print(net.conv1.bias.grad)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 优化器" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "在反向传播计算完所有参数的梯度后，还需要使用优化方法来更新网络的权重和参数，例如随机梯度下降法(SGD)的更新策略如下：\n", "```\n", "weight = weight - learning_rate * gradient\n", "```\n", "\n", "手动实现如下：\n", "\n", "```python\n", "learning_rate = 0.01\n", "for f in net.parameters():\n", " f.data.sub_(f.grad.data * learning_rate)# inplace 减法\n", "```\n", "\n", "`torch.optim`中实现了深度学习中绝大多数的优化方法，例如RMSProp、Adam、SGD等，更便于使用，因此大多数时候并不需要手动写上述代码。" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [], "source": [ "import torch.optim as optim\n", "#新建一个优化器，指定要调整的参数和学习率\n", "optimizer = optim.SGD(net.parameters(), lr = 0.01)\n", "\n", "# 在训练过程中\n", "# 先梯度清零(与net.zero_grad()效果一样)\n", "optimizer.zero_grad() \n", "\n", "# 计算损失\n", "output = net(input)\n", "loss = criterion(output, target)\n", "\n", "#反向传播\n", "loss.backward()\n", "\n", "#更新参数\n", "optimizer.step()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "#### 数据加载与预处理\n", "\n", "在深度学习中数据加载及预处理是非常复杂繁琐的，但PyTorch提供了一些可极大简化和加快数据处理流程的工具。同时，对于常用的数据集，PyTorch也提供了封装好的接口供用户快速调用，这些数据集主要保存在torchvison中。\n", "\n", "`torchvision`实现了常用的图像数据加载功能，例如Imagenet、CIFAR10、MNIST等，以及常用的数据转换操作，这极大地方便了数据加载，并且代码具有可重用性。\n", "\n", "\n", "### 小试牛刀：CIFAR-10分类\n", "\n", "下面我们来尝试实现对CIFAR-10数据集的分类，步骤如下: \n", "\n", "1. 使用torchvision加载并预处理CIFAR-10数据集\n", "2. 定义网络\n", "3. 定义损失函数和优化器\n", "4. 训练网络并更新网络参数\n", "5. 测试网络\n", "\n", "#### CIFAR-10数据加载及预处理\n", "\n", "CIFAR-10[^3]是一个常用的彩色图片数据集，它有10个类别: 'airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck'。每张图片都是$3\\times32\\times32$，也即3-通道彩色图片，分辨率为$32\\times32$。\n", "\n", "[^3]: http://www.cs.toronto.edu/~kriz/cifar.html" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "import torch as t\n", "import torchvision as tv\n", "import torchvision.transforms as transforms\n", "from torchvision.transforms import ToPILImage\n", "show = ToPILImage() # 可以把Tensor转成Image，方便可视化" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Files already downloaded and verified\n", "Files already downloaded and verified\n" ] } ], "source": [ "# 第一次运行程序torchvision会自动下载CIFAR-10数据集，\n", "# 大约100M，需花费一定的时间，\n", "# 如果已经下载有CIFAR-10，可通过root参数指定\n", "\n", "# 定义对数据的预处理\n", "transform = transforms.Compose([\n", " transforms.ToTensor(), # 转为Tensor\n", " transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)), # 归一化\n", " ])\n", "\n", "# 训练集\n", "trainset = tv.datasets.CIFAR10(\n", " root='../data/', \n", " train=True, \n", " download=True,\n", " transform=transform)\n", "\n", "trainloader = t.utils.data.DataLoader(\n", " trainset, \n", " batch_size=4,\n", " shuffle=True, \n", " num_workers=2)\n", "\n", "# 测试集\n", "testset = tv.datasets.CIFAR10(\n", " '../data/',\n", " train=False, \n", " download=True, \n", " transform=transform)\n", "\n", "testloader = t.utils.data.DataLoader(\n", " testset,\n", " batch_size=4, \n", " shuffle=False,\n", " num_workers=2)\n", "\n", "classes = ('plane', 'car', 'bird', 'cat',\n", " 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Dataset对象是一个数据集，可以按下标访问，返回形如(data, label)的数据。" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "ship\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAGQAAABkCAIAAAD/gAIDAAALVElEQVR4nO1cW3MVxxGe2d1zk46EhEASlpCEBaEo43K5UqmUKz8jpCo/MQ/Jj0j5JQllLGIw2NxsK4iLERJH17PXPEz316OZxdLoeb4XWruzvbPDfNOX6Tn64cuRUkopVde1OomqEbmsaqcZhIKbFTVJVVV5jRsWRGdRlaRc8d2Gbmtu3/ADTdM4Ql4m0tXavYs+NI1mVepjX9pUckUXlXMX7RMVcWbEwQpAppkCEACttMgsJiw1uOK1EYHvJWhtvQWqUhY0s0FrJqbGYy5V00S6Bwjf5RpdSZKU/vaorRrpldau2oRfFGdWAOJgBSBLZMLy5OS/Ey1DCakRXqAZBDZJunEayRVrkguNEpe3iSwO3DkxWCCv9R0ed1J0hvsO9uFtYLTSNg3d5QhsjTMrAHGwApBZfHJnvj2zwYtaYTLz5GzQ5qQiS2w8U6tsHmm3WeL1wmK9e+VEJ+C7ijlkN5VvZQluJM5HqTbvF0Y6zqwAxMEKQBysAGRwWH0XO7GMqCxVWNr4AawFvivQ1O6CaLvdWpNLXXNEnYgn7boC0r0Ga6ulSvrAjgVPg6pkj58vQUOt8S2W68APIhiIHvx5EAcrAJk18wkWc+ygF3EsLnLuSey9qwNudC0+hJU5Ep/Ddf0tEvqeDFwHiztYHBpXgzg0csv3VERV43EzevDnQRysAEg+S8JgvldbbEgSmBKka+mWzHyhFWJyNIJOC4hs4VJLPotttJDPzZbZy0cj7V2zqFJXuZh7lmp7zqAPSMYpV4g4HXGwAiCBdIt32tjWENlk7d/FE87f2mOHnSxqGtCQA1rtPqi8KxBOmjD0wPOQvTy45fm2JA1ASY3eeHyMOB1xsAKQiY/n08q68BteX4vQss9DQm3lfxEb4lErGhVb62mSXLXVP5ek1h0v2e0pP/HtkuJ2I9w4swIQBysAWcPTrvIqAM6I1CcdiMIzuYBHmGR4MEFWl1mQ8pNlUzhv0QolCOzxCotV3fD/OvIwKLNgVbVmd9oLRWtRLouD1q6vHGdWAOJgBUBI0VKMcDYknt8olUaNn8axd114ejeunRKC1O5mkrWDeyI6dL7DN82+oYQ1TKz8rV8A4Wd7Ik5HHKwAZFq5+/26JegTaEmQIg/jjnjLhEey0dr8hJXBbgjebO3XukkebM3aqkBkLQkc3PXKJrxMqd3hunYXkxQ+s/tVER9HHKwAZLLm+5nEVrRUyzGJavdOW+7UripgS4dSPNGIJCZtKaZI47CG1OonzC7KpPx9B+isGrwOHRUaViBpTZ5qmqb8FRFnRhysAMTBCoCUSYK24H9b4liWNvHFUZXfkrjlhYP5n1mrQ8aBcCUmPOFuUbMclf68q4QOpFYVJ+JorINwgLAdlXiOQtW6THsZ7RhInwdxsAJgbbJaRbzm38r2tgE/Nhan2fWMMZNB0IP9D9C0vf3OCEXB2StW1ZuYcl47nBxSr/hQTpL15TO482VJroZfoCDuiFecUdvBAF/WnBqLZ3fOgzhYAcjatm1cwYb2JrNVe8R/s+1Dk4QLfp/98BCq7t69a4TxeGyEPCc+Fg1Zyi++/NIIn9++bQTQcHK2B1U4QqekNApW3k1tV6UbFdi5A1hPmGZvszXiDIiDFYAsscp86N/W3BPguay1xmRmTR5/Gy6xXbh0ERdXlz+hFzEdtt+/N0JeEw0zVvr4+wdGuH79Bt868Qb+CPSKbTrTFoF3guJcvlLZpcbMOkmXt5Q2RpyGOFgBaNndCd/fYQ1ylo6Jyf8X+TGZuV5X3njzxroRpqbIBf3mm3tG6A5njXBwdER9YtZfnL3g99M6g4cKRWTZvEIoT0r8PLhStXcsPs6sAMTBCoCYlMqLqqQw1vb6pJCBnTdVOQ+CAjjR8fbtKyN8d/9b6Dw+PjbC5i+/GCHNiKTXrpOw9XLLCF999SfuFPWqKqQeIvUOi9f8OR22ffiZCvldB8mQW7UOqPzDOHBqO86sAMTBCkBWeT+XIlV6lt2QX3GQ/U9qX1aF00YOjLEvOneZrJvqiDVMFQV3U3Nz1GyOXNa8yo2w9YpoOL+wyMq5JMi22rUwivopd9wtnFq5YeOJPSfv5EyTRGsYjjhYAcgQOllzklDVYiPQLFMwgkix8hlLsaL0f3BhetoIPzx5YoT5K8vQeXBwYISpGaLh/v6+EV5vEfuevPjJCH/7+z+M8Jc7fzVCryuZUuvnlOhKXoBE2hFg2cUVtew+fNESzWKtwzkQBysAcbACkB0XpXNJ9kUsM4/cccXubJmT/52mXW5BQ//zTz8b4e3bX42wf3hohPxEJRScD96w6Q2MsLh01QhXr103wmBIy193YpJ7YvWZ/Ymyoe6N+St6aYe/y1udJeQQVVhwk9oNSOLMCkAcrABk9+7/10jwtuEldKzcU6/DfnNN/vrkgPzvJCEaNglduXdvwwgbG/eNsLu3Z4SF1TXoXF4mN+Lp06dGmGNXfmVlxQjrN24aYW2Nkl9vft02wrgQHoJZ45w2ipBTyziQxg6TtfdLRCtKey1q4SZpcC9EfBxxsAKQvf+wa6TBgCxRxkmlzLKGmoPJNSbIzDTlgvsDqkJ49uJ/dGuGMr/r69eMsDMi13x6fhE6//Xv/xhhc3PTCCWnqO7c+bMRZmcptH786LER3rwmGua2OWQTdshmt9MhIwinPpX9Hg6k4dNbNMTeKtYlv4Y64nTEwQpABpNSHNAEnp2l3FOv30W7hUt0scPcHI12jbC3T/Gw4jNqv7tJlmtpiUi3u0c03DnMofOPf/i9Eb74/DNqtks6+/zqmRnyRY8OaJvnYH/EfWeiWdVRiIgrzohhdwe0bbyAv2yj4W9UL0WcjjhYAcgSnszb22Rl9njCPzvaQbseVwpcmiVepFLaQCPe53I9mNGq5NxQ2bJBsrJ8hVRxVT4MMRzjfEz28ZPFy0bY3KRUV29yILqYUKMRkTTPmYZcnIsMV8qVvzCCRdFCQ+tcbsxnhSMOVgCyhmfdxUs0z1EOW42lWLbhY9mDASVzUQePCp5KUZuDQ7KPBVfyjXMOPGsxYTnzGDSE3cmYKSknWLocga6vXnUeV0qV7HlWnDhqeM8JDNOpe1K8kjNDkjgqeenAmlDHFM05EAcrABkog1mHdAccQqWULjkvyns5OVfN9jPKzHSEO8je8OOY+aX1Yww1NjvlPdyM+ctv2d+jDmRMzP60dC/nOG5+boaUF2TT9yoUPXT4HbKBRVcSoXQxphdVXAQMWxlnVgDiYAUgO2YaznEyBDwBv5RSyyuU1ex1aTI/evS9EV5uvTHCYEhbCUh4dlLyG3WXnUxl5yS50LxyDWuGA6mcGtIDEsbwNot9UcQBYMo1VDOTE0Y4PqRDL3VO2VosF3ND3h9ZmIcq1Dq8eU0PVtXgRHcjzoI4WAGIgxWAbOEy0fWIyzQS9iFu3/4M7VaWKTO1NyLmT0xQNvnwmIz00xfPjfDkx2eknVUhRzbJJ+GU5a9P8PrS4aieM2MSig/6tHCguPKoOIYq/KbTaIeC//l5itKHvJIOp+gtV68sGGHpCn17t2M5NLwX++7dB/5k+sA4swIQBysAGfI+MMljrtPf2JDK4offkYBULJJWq2trRrh165YRUGb14AEduHn+nBi6s7MLnb0eu/68EwNh0KFb3Q7Fz91u12lTWbWNSUqdQeHFCgf8K4urRri6St7PBU6E9bFzbKnCNm2vR+m50ZAS7nFmBSAOVgAyJGum+QDN+JBouPVqE+0O93aNAIp1mBf//PprI3Q9WoE7S0tLRsjzH6ETaazhkExkxldqjl1hm0bcAcTkCJ6VUkfHtIZ8yiVKO2wWYaw7XVI+9SkRM0mQ/hYavt+mF/X7ZD3n5siUx5kVgDhYAfg/pQ4eZ65sAxcAAAAASUVORK5CYII=\n", "text/plain": [ "" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "(data, label) = trainset[100]\n", "print(classes[label])\n", "\n", "# (data + 1) / 2是为了还原被归一化的数据\n", "show((data + 1) / 2).resize((100, 100))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Dataloader是一个可迭代的对象，它将dataset返回的每一条数据拼接成一个batch，并提供多线程加速优化和数据打乱等操作。当程序对dataset的所有数据遍历完一遍之后，相应的对Dataloader也完成了一次迭代。" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " cat deer horse plane\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAABkCAIAAAAnqfEgAAA09UlEQVR4nO19SZAk53Xel0tl1l7V1dvs0wBmAAx2AiAEimCQIilrs6wIy3JYctjhi0+KUIRPvvrkcPhqRVg32WFH2AeFFZYoOyTKpCRCJkWCBEgAxDL7TE/3dE8vtW+5+fC/72VNVTWla9P/u/Trqsw///wzK/N9b/keYMWKFStWrFixYsWKFStWrFixYsWKFStWrFixYsWKlf+/xFn86Lf/2T+W75zMKNFkDGDc78i/45FR0mQq+6Ty13VdjitKPI2NksHnMV0eKpvdOQgLRikWQ6P4BdnF90UJAtmmVCoZpcCvCp4MW3AdAFkmw2ZpwoOlPL5sWQzkQOVi0SiVUpHnLisTp7II//zf/AfMyMVXNo1SXz1rlO7BUHaJZBcvkANVm3KgUi2QyfgyfhRnAApcyEd7B0bZuHDeKGkk3/WPxrJv7HK2Mlp/0DOK68n61Os1AFEki58kMScuc5tOp3Nn6vuebOLJJ54vB/ID+aRYCgCMB3KmDu+feCrDfvD2D/G4/M6//FdG+fJrbxmlfSyzvX94DOCv3v66+ff2w4dGefPNrxjlaedYxu+Jcu7zv26Ug0jO/c7tD41yeHDDKN3jXQD1alXWpCpX6v1P7hilc/xAvsoio3zx818wyo3rd41y4dIlo2zfet8ov/qLbwHYurhl/r11Z1+mtLpqlJsc/79+7S+M8upXvmyU15590ijR9n2j/Nvf//eYkd/+F79jFL3VPU8uhxvImTr8KsuyOSXgnex4sk2SJLODeK4oSSw/h6AgP4d4MpEDpforczm+bDOZyP1jfkxONuHEZQM3rMmBQln2zJNpT/UmTKc8Ndk5iuSWTibR7OnEsVyX3/29f4fHxYUVK1asnBLxl3wWy9ssLPKJjgEAL+ubfx1XHpkFvpML+oB39QEvg43G8liNJzRweJwgDDBjb4Uh7Z1yyA3EXnBdtQJ8blyYG81xOL6TAfByO27+lRUEYkYVaa8FtCmcTGbrOnw1LTNCAaRTOdz+jthEDt/5NFmQuLJN4vBt78uhx7EYKYVKCKBcF4OxnkRcBNkygyhRJGfUO5R9hzR11WZUu2k4HIGvWQApzUzP05dnxtk6/ETPja9NDuupFea4AMo0b8OCLGC708cJctyVrz65c8soMV+5e8dHAFpnG/J5IMe9+f4PjHKxJed+jldzjdbZ+lf/vlE2X31OTqSzY5S/+J9/AKDIpdgdyXJ1Y4EIo1he7Jc3zxjllRc+I1O6LVbed97+c1kMyBX5zrsfA3ju+c/KuTuPjPLBt75plMBvGWV15aJR9g9l38++9kWjTNY/kXX5fcxKgSupii6+S5zh+PP3fERLJDeTaUl5jgcgpYnk8YeZ8cZOE7GS9GrGEUdz5VYo0GD3qGSpAwCp/ELVcnf5401dudNcXw5d8NXUAkeT2QZ8gMTeFECaqkF3oiFlLSwrVqycGrEPLCtWrJwaWQIJ6011bItSqzoAJiX67cZi29H6QyEoGyXjE3A8IRih0z1OxnMHCn0fQKFApBaKoUhXMqpVOt2J5tT0dR0ZP8l96iKu5wFwGTFIErEzpxOZ9ngkSMqt1eXQBKGuR2OYs/LS5c90n+7GCaFHRsjmevKVz+jBeBRxMrKxH8qy+BUPgEuvdq0l4DGhP3hIS9ohCi5VK0bpHQsqqVYqevbmT7/fBxDRyC8QVqiikDCHHlSCgLibG8eED+PhCECrIeumoY8pgf+iXL99xyiNspzaG6+/bpRPv3ULwL37AhVrZTmLF56/YpTi4Z5RDnZ3jbLaEVd3CXLE5oogyj//+h8ZZWf7AYDLq/L5rR+/Z5RWTQDmL/7SV43y5OUto9y9e90od7YFsg0GgvTf+rmfN8pzz70MoN2RmycZDoxybiKfxEdyOYpDXjLePA8PZNqF6QDLRH0deofnCiGg3oger1QKdWWAG4sYL0rG+E+qPhqNhjEM4/De0/s2pmNef3cakJlOYgCOw9splN9qyhlkKc89476hukqIRhnIcghgUSgASDgnxcWLYi0sK1asnBqxDywrVqycGlkCCTvdtlEGPTERy2EAoMCYWsEXIKAJQUFBsE9EY9IlUitWxCB0mdyhIa1yuQSgWhM4WQg0nMc8KaZHKb5ziPw0bqUBRM9T1FMAgFRtVDmLlHPTUJpLWDEYCFyNmPRUDJmZlaOtx+TlF58wykFHhr13QxBBjZh2TMw8ZKCqALGB11uSt1IsVwAEFYYCaQyPOKXpVAaJafa7BVmfUlmWztXV4O4m4BLHmoelIVReIIIHDc2oOD9ByYCZ6HBCyKlJT4tS1qw6Xt82b7B+rwPgledeMP+eXZEo29OXn5adCbI0D2sqgyG+I+lXLhfh7gcSW5xM2gDe/sF75t8Wces/+YVfNcoLr71mlGuvvmKUvY6M75bkjP7HH/yhTOaZa0b5/OfeAuARAP7NLUn72rwuKPJi7YJR1mJZsG4iMc3f+0+/a5RnN+tYKry4izlWWaIXiBdR8+MYPNWL6Co2dBzMYDoN52kcX38vE8L5IJCNM41up3Rc8CfpSf4gcx75w9cDezz3mEeMuGKe5pEl+tvk3o4DIE71Ll0emoe1sKxYsXKKxD6wrFixcmpkCSTUGhePj7MszgBMIwkVqcHm+VpownBSLNtMUwFBil+qRWYbEm2Vy2UAtTqT+mm+akGAT6tVUxlT2pmaCanpjppKWqvXMRNJUYs6Yb6ixs6iCZNa+ZXm14UEMmFYwjJ57bWrRrmz3ZUtCfeePHfOKPcZG7qx25ZTpjF/9axsY1Lw2jyv9khAhKafJoTDKVGcmvdTAtgC1zbT+ItjJi9nMWH5xXSqRRXgtEVKCsA19sSvKiWBXWaRc5RNyOA4J4Z11taaRtnbk8TOo0MBehurGwDOr0tpy4U1Ua5eFUjoeM8aZTiVZfnff/onRgnee9soFy8JNj/bkEW4cPYqgKeeF4B2sSwobIPKB++8Z5TWplTtXHh2yyi/9Gu/bJRvf//HRrlzS4p1PnttD0CLAeWtp+S4hSPJNT18IOfVLAu2vX5fynp2+nIn7N1hCPxx0QxPF/orYBhXf27O/HXRH4i6YnLJzC5yv2nQOckYIs9kJhEjiQoJC76WyLCujuE842+JCeXiTBMGOKlMA5e8b1mRA24TEkhquHASjwGkC3B4UayFZcWKlVMjSyysp69KFsx0SL/vcALkZo4aL+omd7W0kqk0SWNFNuZLQA0rL3cEZgCCAr3yno7G8TWDA+qApBeZVpLPhCn16ZpH/JiedbXBtAp6PJHzcjL17osFUVY39kJ20pyUCzLa5qoct/WqrJvHhKyMGS4x1ydgOkyDFa2mlGHKtQ19pm5FNDNpu3hMfklY5JSFjDPQwp3wkrlIAUz53o743vNSrRti+hg96C6vS5V5XuD7s8BainIpBICMeTq+XtMTLazzGy1OUk7t1vU7MoVRAqDsyOdFKh6jOgnN2wlfxccDyWOaHom91uu1jaLBnEmcAXj9rc+bf+OurMlfv3fTKJvnxfjqj2Tdxn1ZqJXqulG+8BUp/dn96AM50MN7AGqbcjqdqWCIqCEJX/sPxIzqemLFNBixiVzZ6+jhPpZJHjtayMNSh7pDsJJnI/KUk3T+JpfymnxLOVCs151GjAZdtHSmzPvK02xHpTlABoCAJI8IpTS6tKbNDzTHivZaHhAghuNt4yYuZlHRQiBIxVpYVqxYOTViH1hWrFg5NbIEEl48R4Kno7ZRBt4AAOjXU8tNszOU8ibjeG6gSf2CfRIsFNNkGYBuR/ypmtNRqQouU5+uFotosYvrBJyDAkwxI0fjCYBBn9wSPK+AhSYBvcsFzUnJa9mVBouwa9GdaY7Ls2hWiETo9e/t0xPJ9XninACNpkcfN1FVr9OfPS6JMBCPGAcgmovpL89YKKPrr1k8RZ7j+uoKgDt7h+bfPmuDWEKPDVa0ZDnTgxJFcJGhaJHu0sDHDLWZOk0znOglnXZlDm984UtG0WS07bu7AA6PJAfq/v1t2aAo3vGXPvumTJuXu5pwShW5SWLClg8/IRFCWAZQKEswZ0jYcv+RHKgliA3PvS4reQUyWi0UV0ahQIINrn8hiwFUSrI4RyQF+6Pvfs8o62QBubb1lFE+/8JLRvnu9z82SufBMZaJGg6aH6cgUf3xM84N3pz5AKw/Y2TMoEV1v+TYkIM5iTpGZJsolhs4JvFJyF/xJGJBUpICcDOlOZGJRwpF9TGhhHTMuko53+FYf9d0KTguAF/DSpmFhFasWDn9Yh9YVqxYOTWyBBJutsQqrjOLZ1gdAQiVpY9W64Qg5ZjgUaMGoNmvgYAiI0qaQjUZTQAMmBg0HIjF2Okz4csXJSxLtKhcEaVGIgeHkE0Vkz526axQDIdFmYCvBQ2F+eBjzuhAbDshPoqTEyAhUSSYgtTtMhoyFmXcE0N6tSX462yzKQciJOz2RwAO9yWXxyNcbTE6eNBlNGogkZppVyJlIZc0cpQxQuz8V178WQDXrslR/uzP/8ooTcLtr35JCIvffV9yhXZ5EZXRQa94HMv43V4XyyK/IxJgLEp1Iti8vyO1LI/274lyuA9gvSHpVy1Glm/cFwrju4/+l1HWGGlqEk0cfyQAcJ+h2CtPXjbKOC0AuPup5E997itfMsozz79olHu3BHse9+WUD1mak2Yy/kZVbpLzrwmsazRCAB3SkHz5F/+BUb7+7W8bpcKg9sULQuC32VozyirvWxKgoD2XD6d8ivzAXQjwAcpznW9l/ii7tSZkmWRGdb8U+FtWMmWP/paE0buUSFAT/RLNBeMVTxOjsMCLBA8e102nnSmDZjYf+5shj5RNTA7gdKKELjYPy4oVK6df7APLihUrp0aWQMLGStMoTSomUKI1LlofEzE5U/vcDAiCtM1GwoyyiEUnEakLsmkEIIsFpAxisai7EzFwh33BPqORlL8orNvckCDOuTVJydtsSWhppekBWFlhNUlRwamS/ykdIPgVo5BEi2pa/62QUDnX2QcHGZNaPe66UiVToJZK5PTVCYAC3xxPnBWWcZcTOFORfQ/asm43yCE3JnauNORkJ7TM723fA/Brv/Qr5t/L6wxTrgjsOmoLCDo8FKa6ep1xQ8bF+gMCPYIFzw8xA6Vz3osT0msBDNvCpH58IJUrF87IZFbX1wEM23Jj/Pjjj4xy8drLRgm5Stu7km/5Aq/y+gtStTM+FG6/ClM6G/VNAPfvyS4txhM3tqSY5skLW0aJSSTwta8J+V/GOpJ2h0x7vPf+8v0fALjy7PPm31/49d80ys+Qj/DqmQ2ZCQQA3iSjQ28oi1Cpyt2Cx0nwc6597UDlqAdDy2Lm+fg11qxMeCVl1EvS2S2VfV+pH/w8XEhXA+vqYtJYxmQiVEY9zykASBhhVEioPyUFgooN9YhT1oeVeEs3ecsdHR4AmE5l2FTZBRfEWlhWrFg5NWIfWFasWDk1sgQSBsRHga9+/hTAhFTuWn0U0Fht1gSghUwTHXPjEWHFhJGFlEXbncEIwEFfzL9+KltGzHabJGTL5r6ahloo+3OKdspaWasACAKdpMu5Efcpk3Ruxyoduxask8RaK6weF4e4KSyIkR/4smWzLhVkBbLgN0ge7zCaNuqLsr7SANBsyuQ3G7KSJVrUXeZ8VmmW7+9ISHFCtgYFmCBQ/fCj6wCe2RKQ9dqLAmQ0IfYP//hrMj7DtAGp5fpMiewTxWsqabNRx0x3Jq1cUwKMRVk7tyXbMEHx6JEwtT/zwmcArL8mKHhIlNLcbBrl6lXBfUe3BVupYwEs+mutCuLbPCMVgoNHfQCIBNPppdRwVUFf05z//o7EDT/4kbAA3r4tHU/diYyzUgkxUzR35ZrEHL/0pS8Z5RxzcUMSW97/xjeMsncol2xtQ6Z94yHTXAHMdNZKcyYMzlEDfwTmmqY7Q/KnsTm9tzOzhflvRAZ6OJojrSx9DAWy2DN25iczJZ1k4DqYCTVqxF+5GWLmCSRsWJdxtJjg8Ec/lG67AaPMzVoNQIMI0bMEflasWPkpkCUWVveR1FIUlbPYzTDrDsyrrlnfz66f2vx9HMrzNSQtwTGrUvYGsteN4zGAg2N59mfkBigVWQwB2bdFTiUlzyoVlZ2ZBlQow5arRQAVdoEN6EHkqycnCdLnOK2l3EGoTlD3BKanUN3zXJZqhb587YBCn3VA0uRqIAZUzFfF8xdXATw6Eg9xoFSztOw89nbd3tnj3MQmrbXElJuQv0EpKiJkAO7tyS4/87q4sTsHbaPceyi+9pjXZcAUJ59xhipjBaOxuIh39/cAnKMLX2uDFnmWVe48POQpy+Sikfj7G80VAFvPihn11pelmU2Bfc+P9o+Mcv26dNaZHstoEdOIGrRZBgOxkh492AOQ0L1drMuaH+2Ie/5od4erIZ/84Pv/1yiffCo2qVaYFJj9FDoJgJ0dyRF793t/Y5Q3f+ErcoJchAJtijL79PTpud9Y28Qy0Uw3ZTLIHetsvaNtcd0FA0T/z+iGN/VzZabdqVc+oZmpju2Ye2esRtKKnIIrN2FF+c3jDEDGQfo03LS5vNKaJ3ThJ+yjo7/QJxn9UBe9AXPa2iezpTlWrFj5KRD7wLJixcqpkSWQ8OE9Jraw4U29XsEMd4JaoB7tTFf9mjn9mJL8ySCHXUlF2TkQn+4kdgFkNLmVXnk0lEFYz4BaILZinYqWzrgZU0Los0/HAYBxJIOMdNp0EIbMVamw14vOX7vmKEeadricE/XcT2kMZxBlQst2TKu7Q9q51qrglwsMUxS9NoBGRY4yUpIJ+kRbdZntE5fFOd0niPtgW/DRtM9UmqkACdNsdUKs21yT4/aYujVgrxS3JPhrQndph/wZDqvntao+jscAuj0SbHgnOkdV3v1E/OUXL8ocilPBR+P+IQA/kMt8/oKcYIEgaLIr94w/0YagbL1z2DZKEpP3cVUmY9agS6dvfyizPbwpxTo3PpRqpEe7UiT0YPcux+d9lTunZTKDKAZQZMnU994RkobLL0lA41VGNvbv3OTZs6sokf5owKDB4zLT1xZU9KfEHxeXJVuAhOrYVn+N2SLn2mTExuX8yxXxcPdJ0nA8ZksnJmAWMrkbpwPJoZtMepipxNLSH522/pzdPIyQdwQyf6ukyUTOYx4BmHLxNb9sUayFZcWKlVMj9oFlxYqVUyNL8M5wKHZgmpCeAT6APgtl1GwOGUasVNkRh00r4bIhKLP7t9lT5O49ApnYx0w0pErGtXpJAEhYEPNyheOvr0v5Ra0lympd7NiiJwca9DuYyb5xaSS7uR0tn0TEbhrjUv4JNdE15WROYkKqjKFMJVHrs+i8Tf6G47aE5FbKZzltOaNPbj4AkKRy3GaDsa0SQ7S0qMuMh5YZsskmpOXjqa0QxSfRBMDgSC7lkPHKPRbipAPBZR6nXaoKa0KayCC72+IcWGnIpVlZqQPod2XYFjOPPP/EN19Kn0BQk5jmxaKUB7368lUAXk4OIUihvy1RvP2PpHVNlTU0Zd4bet88uCf46+HtT2WbcgXA5nPS1ijal7M4vCmjjXYl5lgjkirk/HOcNjPyXIa9xnEEoEs+jyIjvy5z36rsLXS71+EocoGe3LpklEYoi4wf4zFR3OTMZzYpJMwWOtvqXZryk1SDjCkw08HX4z3pE4WlU1nSfibeiaQoJBNeWcD7MdPTpt0dzsoDWfMB+FzAIG94Qy5PbX7MWyPlJ0oCkUQKhF3M9HBwbWmOFStWfgrEPrCsWLFyamQJJFxdbxqlxMR5x/EBDIdiXo6HgobG2nNpLAZnpaaEYWVurNx4rDmg+Voo+gB6x5IcyARJBJkAkEpLBvGVfqws5muT4bZGWY5Y8hjgy0IAYcCET1r7mimX94MipcSI+G48Fmim/b6Uo25Oxiw9B+uTQmZR+oyd1aoy7SQRADhiPbpWSoxHCYAS8UWZLVFdBpjiiIEYZuudYRDncy+1eEayF0OL2Hv4CMBKUzbY2xM83u4IWllbkaRQXhYMGMYtsW9VifzlKRNSjx4dANjYEFyjeDnDyTZ8UU7tY1IX/PyvCb3BM09fBrDNrNRGQ67pp9els1bnjqRxrvDmCIkmzgSykmNfoM0x+24V2n0Aq4cyyZU+Q43rcl+V+3KCh8RuVfLxt+klGKrfgBjQdKXLpszaXZdbUdNre0cyWkz81Scv4OsvkwWwJIwOf/zNP55ZJBQYdFYSdk3K1XCbo9wM2gEMWqyz0McUKQCf9wxIrK5ga8LSpQ6rnZIVuVsmKUGoNh9jr4EbN+4B+PimZM8+T9LEl648weMwYJ0uJKbm1WyEfgzBGzBYrzNRmY+aRbEWlhUrVk6NLGtVTxdusajPeAeAS3unHDIzpU/vMsmV2m15WyZa6co2lnsP5f0zINVvIZwAQMYu6rR3jpkMNY3kQTukk2/EYT2+1jbpfW+xbNjHFDPe04h+9DEf2wMahuOpvkW1YnPB05ktf6Z7bDZZJhfYZExCZLLrliraKV4+GfJt36CX+vlnrwHotcUC8hjQ8D368nmJGnWx9Wor8tXZqUwy9cV26LFItRIWATSaq/xc3qJHtLBSfRXz3a6l1HFXNl5pyiIXWREVTX3M0IepBEGAE+RzbwkXc78v5/jaF6UEp9hcBbAK8cEXQ5nShz+WCuSDB2KU1WM5rybtWY0VXGjKsly5JMbL6HAA4AKr4je0UQz9watk3O522kZpVeUED5iMdsyrGUdaWRIAWNuUJf3q3/uyUc6fER7k40dSX6U5ekX1x9PMLCz5wQHIfe1qAemt6LustqHTWkv01dQKNKGJ47lZBlbSYLbih9dOMUSFRWvdqfwutCYPzNV6+1ti6t69swPgeCA/80PS0o0yiSo4tIG8sCljVGTFFPRo/yo1JzuDIwD7exIeGVoLy4oVKz8FYh9YVqxYOTWyLA+LVE36nfH/RYRsKZ1/PtmHayTeTRJ5Ao7HtMO72leDdiyrakajPoACwVFIh3qpqA5vgVRFuqKLbIFTKimbAksB+PA1BmfO0KTVQvTweXQVV7WhjnJaaWcgxZQn1I0XmRDEyvac+6FAtDgZCpCZkHOqEgqAvXRRmvrEgzaAY3p2o5FsqcsC1lJoN1kFAjFd+IMRIxsjjYf0AJw7K2lfIZ3KUawxEIYgCoovmHlERDMecfyuzGpzvQWgypImbWIUs7frovzWP/0tDivw5PKWnLvpxXKmIJf71qfSbfSDG8IVdbQtFTMXuRq1TWE7SOl9L5NOY60mN2FcrAGo1mXYR7vCbPXJR5KoNSTFmMPLvcbOPTu8ZC4LxTKWXpkb7CyJmL/yxZ81isKx7i69/hz2+avXjNLYENh4i3Vvc6K9ncplpra5giJz9iulqUoUPyoReTz3leFWiXir5B1VSUhy3GOOHityEJA3hX6JoChIOSzLWZ895wC4XJGUukuXBAm2zoj3vV6Se0OxZ0TqBeVGPz4SEpFety0nkk0xQ0euNFuLYi0sK1asnBqxDywrVqycGlkCCaeMLEwZp5hGJu5Gur5qg4pYjIk++BjbirnxOqOE/cJ1o7Sj20YZPRoCiMasHqDpWGQNxyYN6SAv1pHxNRDj5HSxGukDZiChdnZR/jNtFAqCIKa8zNQ+6Pje8md6ljebFCVgxUycchhGmlihgc2LckYrqxKA++DuJwAc2s8xSSYU4RbyfBzwE05b6yEYCa2xjmRSLgKoMkx5/rxgww8/YR1GprhSJjemHc7OpFhZkdk6KYno+gMA00gOd+GCgLufUJrzMlOQVBSemHKUkP8ePJJY832SC2rRfodT6uXRWy7CiJlTd6R8pLXaALC+Lud+f1foie9QmTAoHDJPLSSbhcfqkFIsaDfq0wHi+QCeufK0+XeFfL5dcgpOWD80Q6cnylki2fF0OdjJi2wSjQmyIQ3ZLHymBCqDiLLl5UuaQ0IHM6zEE6L7eCJh4hFJLHoD+SR02LSGkLDWkBtg66nnjLK5UgbQWJHlqnEBU57X8EhKvro9iZl2hqLE7PLravkOc7WMOyK/1W0jVStWrPwUiH1gWbFi5dTIEkhYaYiZp3mPpnC835MgQo+FLKknNl6ZXWEKRY2yMWanXWccBpIcEkVXygDqbH8CQsIeCyaG7LhTqgj2PJ9IZ5SnrjA/TUNmkQbIRpjpdqMxNeUwU0M6TeYTR7W5iEOSvDhdbp0mLM1xq5oTyCjeWJQaeeWnicIKmczDPSluMB1KHH6eKjbUYn0m1irPtaN5gwqQtXyHgaRi4AGokGu/wRTQ7QeCm7S0KCI2TPz5dESfk9GoTbFUAqB0jT0y+fmFE5n8ZhgQFbyzVsMMS/Q9ILlgn6nFLinrHvEE7/PemxKDTAl2AtKwnxuPAFRYYHTILMQ9Vswocz+YRhtqDQoD0zXSjYwnMmytXAPwmZdfNf+WQwHX/Uxqy3bJGZ8yAqvhVPVLrDKePicjlrsNSJWn6+b6RIKFcO6rvL8pu0npNsVyDUC5JouwvnnOKDFTcDUMff5YLuLNmwKZXaLR1TXZ68ozguvrwRTAwf4+j0soSq/HjRtCjuizWssrsH+VEmDwvsp46Q0dvkYJs5wrdF6shWXFipVTI/aBZcWKlVMjSyBha0MiGgGLCk0VXrE2nyLps5pfa80URGh4osoenGsrEvtrkMPg3nEXQJDI59WKZPoVmER3/UOpYAqKrBPUNmJklY6mDKuR8KBWbwDItCElZ5Jno/Er7QPqERFUtS0SIaEyEc5JwvaWU6KVKGbCbSZnVCTl3gpP+fBQAmG9EakRyg4AXztWMripHGlFXW1meCYsLnOYSlrOiD0JPQyTX5GZvT96/0dUPpRzpzVeIe1crMHBFiOJiYw2YcrfpQvnMVMC1meAyTmhGRoe61u1PAdXIfeEfIQZ+dR9ZiEeD9pG2SbSr69K5eDxkSCy8ZHEFgemQnBbQPeAV7nHINR4QmKQVPBRyHju+aclCHh4Q+jetWOu63gAfN4PDotMA8IxveV6LLUrsl5vRNxaZnb0nGgIPmAisubZehnD6MwTBmtgx5pByls744VIHR/AyrrQ5DdfFEz37o/lN9UhOYdDbgY3oxuEwxUIkDukbPw/3/oagDNrgjQvXdkySo3Ek00y62uPu4x8kBPS9UWMoyqT3zQ2bhxmTRdOvJ2shWXFipVTI0ssLJ/v7YSpRqaopappIMxj8vMHIX2W9PkFZPhVuqXzZ8VwOzqSF8XBcQqgTkaBc2eY3lWUfQ535SXplyQOUG+JzzII+P7RPClNLIoed5PzP+2fqvOPeYLq7VObAvxqkZmAW8poQ+YBuXQ8O+zfE9CKKfNNtXOk1Q8yzsVKFYBHXoqiGGfodpmrQusppYUy1nwWR17gzUbTKJ22mBuGQ0K7zH7ne9Lipc+Xf4vkR/lFpIWlRnG/Kzk1SowlPv085Y1lK9GJiTN5t1ptV6udXRwHM+TAA1rN2sF8dUXskfvbsj7PvC5cWr/8G79plDY5p771Z39qlFsffQDgJkejqQqfpxzFskrKdfUiG95ceU6U778vdULK1mAS1pT6LclLdkS2traMcuO2UDCnGmeg8RWcQNeQ0WmtjWM0OVDtWbAlsNaN+SzbKigxVvaY4pE3/PZtNoiNZLn2yRP9YJt9betijqW1FgeRr/a2JYmy23kE4OJ5CZR98y++ZZRPb0rZ0xfefNMoRdqMMR8Bak9hoZOxk4wxE7ZSo3VRrIVlxYqVUyP2gWXFipVTI0sMVM9TjEAn5WgCYMrKA/Vea7HLiMzCpYrY8DWCFC2mqZEwoMgsj+7+DoCDHSmmP96TXKFzZwQklpgRVmeNSIMl+IhkMt0DMXojlm8Y5OEzzaRE7rQiqy6mdO4qZV2SKsCUSarBDxAkPi5JTN47JjRVGxUOIrUUiYI4El1o2VM2JmXtxAXgaZEQL0hQ0katrJhhMYq6+VdYGtWoNY3S6YkNX2utAUg47IiTvHhJMmsmpIXQNKkmmZd9+n11ETZWZf3rlSqATlcgVZX5cScB57+LKNTtD7v8hPk4BBFrLG351X/0D43y87/8K0bRhLKXX37FKDc++QSzLUs5Pr3Y+O//+b8Y5cMPhYv5lVdfM8rVa8KvsPonAjAPD/Y5TISZihZVUl2ldQFKD/aFjWAGDosSx8tvp8XPtW1PzIuo4Qv9ATo5piISZIjJFEtd/5HwXnz3h+8Z5a03XjRK90gSvrRipn0keVjOnixLWuEV6chX9+7fw0xIbUhyizLb8e4+pFNC64c0NMSomrY+Ukq/glfHDHDWB8uiWAvLihUrp0bsA8uKFSunRpZY8iOChYjG9mgwwkzqk/Ii+AuJLRMqI9qKdW3BUhe0ePZJ6dd4aecQQLEkSHBtVSKATbI1BDtijddr3Jdoca1JnjCSPYxGZPIreAAyzS5h+xm1ltXgnEw1SqgpQnJGWqwznSy34UfsBpQXEaQCOUOaxwViq/5EJqnceJoL5qMEoFRhIlUkUTxXg1u8Rg4JwmtVpgIx5DRhbKnclDVcPX8ZwHc/EESg+V+tVtMoY22xqaEZzlZLf0psi6vT3tnZBeCQgHxtQ0BQTpf4E2Q+SCiirV/G2tqzTyTCRjjPPifpUS8xn6jKQJIi5Zeek6+effp5zOB9h3ApYbLVN77+TaPcuCWRsvNnBSmvMgx99oyc2kcfCz7yvQxApyOQp0dc7CXKPSAHWueyzESfZRtd9jnRBYwXeoimjEdnZAGJCEJTHjHWPD5l8osdAH0WMF08L11tblwXTsTDQ0F5CZFmSAK/2lh+s1caTxnl+IEsnan9ijkBpVpZWyMzJXlB1AWkzVZd8o142sxV6UbSDLMI92SxFpYVK1ZOjdgHlhUrVk6NLIGEHZawZzRoo8kUQMwsu5wXQbEhYYVyJIz67NDVF4xz1BGrcpv9vuLxBMCjtuC+aCgINF4V3DcmFfpGS2zUFiNxTaYURgPS5hG7mRiKBo+0KaOam1pi7mtDR9ZbpAwXDtkKbMDGWXOiTVgdFjBpeYrLlDmXxrYW3Kh5XGLIcnN9HUDsCmTrH7MZGgOvASRps8E6D4+xlTEJ2h02Xitkss3b338PwDvvCyTM6foIGVoM/IWsdtIFGjIcXCYkVDR98OgRgFJVPi+0BbtVSPK9KEvq7vPYWQZygQB5y9sJJ6mVRltbV41yYUPoOlzG/pgbq61CUQoKAAKukuZksogLHjNjq+wm22LtlM+uoitapMVZu14AYEpc2dkX3gufrUO7rL/RWHOFhTjxWIZtd9hB93HRYhRtmKYAKl8uLSnLZBuFhFONr2k4chwDOLorBUYTpRJkSdPmatMoh20paXqwL8rHJJ4/c5G/Tf7233j5eQAhS6b2HgkuBoPyRToWfA1gav9XXjJHyRd5juZJk+Sc7id0UrAWlhUrVk6RLM3DIj/RQGyTbqcLoMvXqVYnqPu0yNIT/UTrJ3Obhf4/ZVNdqxUB7IzknbPP7JXjjmSInFkXD2id7+/QE7tj/768OpQmOPdnuz6ASCegLx0+8kO1sGhYaWdKbc9ZY4qT75ewTHxlj/LnncmZmqJa7pvKS7hZkjM5uymnVi07AHZph47YrEVZooZkcRrQc18iD3WfVt4KU6iOyDL8V995B8DukQyrPFytGluudnlxM1l/bRSuSu5bpfVaa9TwWJoM3cDREkNKziMn/9VaFu1INAUQs+TV4QaafhWSuezqk+L9BQ3nw35bRmNNeN5D1HVnJ6l+7knu2JYlrdVYbcYq6AKXv0Ur0uNL3bBmu3QMj5nyFrIWvUvryWGPmUZNzLQJk8J6h4+wTPRHl3umlfWM47MwKk/IAk2VYu6t5ldxCiDJ2EmXPGLrLfmkHsq+jYZMcr8jk/zwUKzI47/8vlFefEYM29VGFUDKwu8tpvVlnLY+UALaXA5o6ualOWo0yl/jfU9zE9JSJFuxYuX0i31gWbFi5dTIEkioTLUHB4IBb16/iRnvrKIhNek6tHi1EalyAAyHYl5qr0e1YyvlCoC1NTIE0XNfrYqyUWGy0lTGf3R/wG3FqiwRNVSqgrbKUh5EWKGkXdwyoK9dG4X26GKPSDlUoc+10SximWifGIWEE0Kbiie7aCJbysILNyaJkiuwq9PbA7B/LG7OAfPXnKlAEoeYutsXDMIoAiL6leNMwM4hYXtvNAYwJJSbEt17ZL2aMnIyg+ZkDk9ckUS5IcGUo61cXB/AiOGR7W1xPNcbNZwg47Yg/THpp5UJw/TR7LYlznBM2miHkK1MX3gpkAnc+OgdGY0FJdrRNucUdlzMZDYleqZ6FlNByg2WbSWE2yMeeqXG24bNkEajAYBeX27FB7usaNlnjYsr98zWNQFKBV/u7bsPhcygTz7iOUkXilE0KSkn5FJqNq3RySnGFC3K/27qADhHF3vKlL1Of/z4Hqhq32LGWPq8vg+74nS/PGKRXHMdgM/DuEqRTEY5ZSjxmTaYOepZZ47Y3wb9LCS0YsXKT4PYB5YVK1ZOjSyBhNFUSw3kk+PjNoCAbGGanlNmUxYNBUYsKHFon9eaYpSC3RPHTK7pDaYAWi0BRyXmGdVrsm8rZJ19qlPS9jDgHAQJhgShyWQMwGfYokDb1+W+MZFOxjNFXrXDE2FCWfEExrUkD3URt0ZkUygxdsaErNSfhy39RJDyoNcDMCJw9hh/ySBmuWaNOZybl+fasBCKCWsZj2h6uBbZSKZODmuPAHDKuKT2wKmtSLJbnxlzfRLjalzYxLA0qKr9XzV7aFGuv/vXRkmI0Vwm5kzGQwCTMct6xuQLZFyswlFr5HQcdgWE6rLHE+bQcVYGekyYyKZdOZWTo8rksIwJWZ5OiTHZBo/dZLhwHE8AdFg25LHl7bvvvCuzbcit3jy7ZZSDA8lsuvERmamnyyFhlvMu8COFhGRrcB0t0qI7gmeUsfxIT9ZxUzxGC671N3LKEUPMHj0YdbJH9pXogkBVHQiOHwLwGK/UnsT5/JWaXHOpPA2zEsny55bp/CVKyF/oQn2SirWwrFixcmrEPrCsWLFyamRZI1WS5Gkk7pXPvIgZ3oJUe3Qo7UGslQGa0iZ/tSC7wCBdic1Ws8IYQFhg8C5gx1OPliFHc8lrrqGHcok5nzQ8p0RVJssxb2vKKBKI8rRqR0kaHKLdSs5Ip71elkPCkK19qkwOTBMx3ccMF3oMQq1tbHC2emoCSyvlMmZab2ptReqzoEFT8rjIgXb91MVmbGuDDW+ubm0CeLB/aP6tV5syPKk4XJfx0IbsEoTkHSTBYeqx/IWXpl6vAoi4bpW67FIqLV8lAMODB5wlV1vbZGYJgIA34dkVWfzVCkt/tJssM28TBvg8oonRQCnrGJ/yPACOEkhwECX1X+Et4RG2qIchY4ZtsymTWV+XTMuHhx0AD/eF5ODlt94wCsuicMAI4MGekEDssLVqiaHGp68xA/Ybb2NGlP1dZ+uoK8NR3KTrNs9qsEiEkbkOgIxwMqf+YDuoIe80dTVcZLsslwdSnKd0KVkSAcjyDrvzWdk6t9SZd8Vozq3iVs2szpDh5L5Ks2ItLCtWrJwasQ8sK1asnBpZYskrqnMJCTfPbQJot7UwTdBQoy5xpZ1dyZ3rkaRBbdAG8zkd8hNoALHZdAD4OafdfNGZlu9rhDEhmhgw/22qQTRoDMIBEBJSpUQTXkHbZNHi5ZELxD4BdC+GlpYtEYA6EwtLFe2syVxZwi7NxDt3ecsovbZgtM6xzL/klQAEFZl8maukJBNqJXtcU8IL1IjmylWpTHSZqfjsM78BYO9IcjKnrN8cMvA3JB1jSHykSLlA3D3oS6rnmFHCMAwBHLPozPVlSrW6DPIH/+3reFzikRwx0ppBhuRMBNbl7bRSlUEukU4v1jCrImaGoRXOK0O/r4Wb6RRAwg0KmspIIOMxAzMM5AL5gRw64d2ysiokfBvrAufbgymAARewUSPx5LUrcoIsvVSqxatba5y2fFUqaNL1Y6LxMjUhNHlSs0O1u5cy+TkLRXnOYrTx8f983lBBoDFH+WqzJqe8slqbO2KFHhjfhCM5pZh5yDPJ5HpK88fWuWmZpPbZmyd1xDzmVbEWlhUrVk6NLDEf1ljPPWHByt7DfQADZu7oO63Dogpt7F5mLlVeWEBFa2jU+26eqpowpalPE3aUSflujPle1eSaESdTpAvf44FMmb7yE+hxS2W+Tlma4+hbekR7iqU5+Ymc4AgMQy4drb8Se757ZO91QvlEE5fUQoy0z1ChBODSplTDVMqy7/GhpPBoX1uPZ+TrW4YvKG04OuYbb+3sWQDPPCsvfyWDznKfLp2jmhSTc0vQn61FQBSz+1iJN5jQ5NHU+tf4j3O7KGmvu5A9NJA4iRxFAyk/88bnjPLUM6/I6azJSg5GbaNo59aU5FDawF1KWHg/jHjzuMycGkznm+Devrctn0zkvuonTOvjbyQMygAmzH2bcnGqVZlAhaTAai/UAjFVcnPphH4wGdSe4mwfpw8G/eiYdXDr1fzbyIXVhe8xx62gRGI0sYplTbTkiXC2ahQbA01ZrZW2QSW/yksm4cxt4/EHPo2NUTzPArIo1sKyYsXKqRH7wLJixcqpkSWQsH0otKeDriC+9tExAJcZFgVmLc0Yq4RU2s9D+9DQTk66ZHQgPDEpSGO6qDUfX/cNyQsYEmSVWFleq4ubWV25WrUTGuhEj3pKi3fI7iNePjXln+A60Fg96Iw57PI2J0WlfCChcFBjp9hQgEB7KGd08/odOUetzCBO8ZwAQMBkNGWbW2mIT1eRoDLVVZmmNGoLf4NHE1orS/b39gCUalJkr+BaO+Jo5lfAkiYdRJdUs5OUwNe4S0tlWa6EHYm0M82i+FxkZdbWxCK3EABIIjncgC4ILZCaktn3xg2BbN/+tjj1xwyPKBmA8mf4foCZOo8h4zPquZ+SpUNb43zjm38p585hP70v4ZHDHrPqPB9AkYTLMUG9x8vhaOegHNaBc5NFSE7IV8tyAEiEzltR0eLfBQAu8ByoZ0bxsnxRoJt8xAZRY/4wA3ZFcjkZ7/FQQH4MDqtoTieg9Mf5qS0oM/N+7LyyhUQzFWthWbFi5dSIfWBZsWLl1MiyrjmM/WlOfau5AqBUJkkb8cWI6Tl37kkNvVZoF0NWP3BjBXoTbcgaRZjJeMoWaL2SvBfm/INVTdARu5j4LBxv1JoAiuU6R2P0cDrP6DYiLMpYxqHYJ+XKuCc80stsHKLkFvu7Eter1gUtDsdyRsdMaAqZC6YVRQdHHQAeU94aZBlXuBoSgCsRQ0QMpTxqEcN25Yog5cy0eGEsLVsICU0nMoMeoXqRjAsaYJ2yoH8SycaGBNElQkjIxZhmy4NfmCl/yQj04pwOJAWQMWvpnXd/aJT9fQnV1Vj6UyEvyOYq+ydtSCxbqSLVpWBSzLTPwJAAcMAbr9tuG2XrkuRJXb78ihyxKbfNjbtSVfOjDz+SM8kyAHXGmissySoSEiakIdGUQM/TXCdNblwOdtT7MVHSRy0byuGYIqb5Op5sIQAnx104mtIBelx2TVjrHsuyh3SahLxtQrojzC3nsnguvxMWIGF+RCpLsKGWH7kuZhwLP6FGx1pYVqxYOTViH1hWrFg5NbIEEmpfLLXLgrAAYGdPCJ6Vpj1j4qKnJRS0Y7URaUJsmBN6KVkaEgBBUcsj2FiJikYhnbz3kTI6EOjR4lQO7/ZRB0DQI0sfT2dJBDMPMLH6nLAoL1w4ISKjjTw1xTQmy3iv05YjssFRmXA14V5TZnjGaQRgwFBpleUpYGA0dQUsjIZisWv5SImRSo+NvxI2JTNWd85LRxSmDPQjksdrC6+UxnuR5UFaGVNg9Uzq+AASrb/3tKxKcybnpXskoUxtKxvzSvV7PQAxP3/9jTdl8rzu2p+1we5kF84J5i2WeO6MdmU5i2SEmb6/PpnvC7y4ffKyq5RYRKUg6oWrTxjl2ScvzW7p85RzzkJ2ZkvIrK/gtEhuPJeWgeadzom2jE0XCPBmwDYh4dz/M6KxcpMY7CykceqvO81RpLZllVsi1kgiKQPTPOZuRluMYM5PSSuK3Lwp2QJa5FeOm83ukthGqlasWLFixYoVK1asWLFixYoVK1asWLFixYoVK1asWLFixYqVUyf/D3PcGe48X+nJAAAAAElFTkSuQmCC\n", "text/plain": [ "" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dataiter = iter(trainloader)\n", "images, labels = dataiter.next() # 返回4张图片及标签\n", "print(' '.join('%11s'%classes[labels[j]] for j in range(4)))\n", "show(tv.utils.make_grid((images+1)/2)).resize((400,100))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 定义网络\n", "\n", "拷贝上面的LeNet网络，修改self.conv1第一个参数为3通道，因CIFAR-10是3通道彩图。" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Net(\n", " (conv1): Conv2d(3, 6, kernel_size=(5, 5), stride=(1, 1))\n", " (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))\n", " (fc1): Linear(in_features=400, out_features=120, bias=True)\n", " (fc2): Linear(in_features=120, out_features=84, bias=True)\n", " (fc3): Linear(in_features=84, out_features=10, bias=True)\n", ")\n" ] } ], "source": [ "import torch.nn as nn\n", "import torch.nn.functional as F\n", "\n", "class Net(nn.Module):\n", " def __init__(self):\n", " super(Net, self).__init__()\n", " self.conv1 = nn.Conv2d(3, 6, 5) \n", " self.conv2 = nn.Conv2d(6, 16, 5) \n", " self.fc1 = nn.Linear(16*5*5, 120) \n", " self.fc2 = nn.Linear(120, 84)\n", " self.fc3 = nn.Linear(84, 10)\n", "\n", " def forward(self, x): \n", " x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2)) \n", " x = F.max_pool2d(F.relu(self.conv2(x)), 2) \n", " x = x.view(x.size()[0], -1) \n", " x = F.relu(self.fc1(x))\n", " x = F.relu(self.fc2(x))\n", " x = self.fc3(x) \n", " return x\n", "\n", "\n", "net = Net()\n", "print(net)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 定义损失函数和优化器(loss和optimizer)" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "from torch import optim\n", "criterion = nn.CrossEntropyLoss() # 交叉熵损失函数\n", "optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 训练网络\n", "\n", "所有网络的训练流程都是类似的，不断地执行如下流程：\n", "\n", "- 输入数据\n", "- 前向传播+反向传播\n", "- 更新参数\n" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/home/bushuhui/.virtualenv/dl/lib/python3.5/site-packages/ipykernel_launcher.py:25: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[1, 2000] loss: 2.210\n", "[1, 4000] loss: 1.958\n", "[1, 6000] loss: 1.723\n", "[1, 8000] loss: 1.590\n", "[1, 10000] loss: 1.532\n", "[1, 12000] loss: 1.467\n", "[2, 2000] loss: 1.408\n", "[2, 4000] loss: 1.374\n", "[2, 6000] loss: 1.345\n", "[2, 8000] loss: 1.331\n", "[2, 10000] loss: 1.338\n", "[2, 12000] loss: 1.286\n", "Finished Training\n" ] } ], "source": [ "from torch.autograd import Variable\n", "\n", "t.set_num_threads(8)\n", "for epoch in range(2): \n", " \n", " running_loss = 0.0\n", " for i, data in enumerate(trainloader, 0):\n", " \n", " # 输入数据\n", " inputs, labels = data\n", " inputs, labels = Variable(inputs), Variable(labels)\n", " \n", " # 梯度清零\n", " optimizer.zero_grad()\n", " \n", " # forward + backward \n", " outputs = net(inputs)\n", " loss = criterion(outputs, labels)\n", " loss.backward() \n", " \n", " # 更新参数 \n", " optimizer.step()\n", " \n", " # 打印log信息\n", " running_loss += loss.data[0]\n", " if i % 2000 == 1999: # 每2000个batch打印一下训练状态\n", " print('[%d, %5d] loss: %.3f' \\\n", " % (epoch+1, i+1, running_loss / 2000))\n", " running_loss = 0.0\n", "print('Finished Training')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "此处仅训练了2个epoch（遍历完一遍数据集称为一个epoch），来看看网络有没有效果。将测试图片输入到网络中，计算它的label，然后与实际的label进行比较。" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "lines_to_next_cell": 2 }, "outputs": [], "source": [ "dataiter = iter(testloader)\n", "images, labels = dataiter.next() # 一个batch返回4张图片\n", "print('实际的label: ', ' '.join(\\\n", " '%08s'%classes[labels[j]] for j in range(4)))\n", "show(tv.utils.make_grid(images / 2 - 0.5)).resize((400,100))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "接着计算网络预测的label：" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "预测结果: cat ship ship ship\n" ] } ], "source": [ "# 计算图片在每个类别上的分数\n", "outputs = net(Variable(images))\n", "# 得分最高的那个类\n", "_, predicted = t.max(outputs.data, 1)\n", "\n", "print('预测结果: ', ' '.join('%5s'\\\n", " % classes[predicted[j]] for j in range(4)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "已经可以看出效果，准确率50%，但这只是一部分的图片，再来看看在整个测试集上的效果。" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "10000张测试集中的准确率为: 54 %\n" ] } ], "source": [ "correct = 0 # 预测正确的图片数\n", "total = 0 # 总共的图片数\n", "for data in testloader:\n", " images, labels = data\n", " outputs = net(Variable(images))\n", " _, predicted = t.max(outputs.data, 1)\n", " total += labels.size(0)\n", " correct += (predicted == labels).sum()\n", "\n", "print('10000张测试集中的准确率为: %d %%' % (100 * correct / total))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "训练的准确率远比随机猜测(准确率10%)好，证明网络确实学到了东西。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 在GPU训练\n", "就像之前把Tensor从CPU转到GPU一样，模型也可以类似地从CPU转到GPU。" ] }, { "cell_type": "code", "execution_count": 44, "metadata": {}, "outputs": [], "source": [ "if t.cuda.is_available():\n", " net.cuda()\n", " images = images.cuda()\n", " labels = labels.cuda()\n", " output = net(Variable(images))\n", " loss= criterion(output,Variable(labels))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "如果发现在GPU上并没有比CPU提速很多，实际上是因为网络比较小，GPU没有完全发挥自己的真正实力。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "对PyTorch的基础介绍至此结束。总结一下，本节主要包含以下内容。\n", "\n", "1. Tensor: 类似Numpy数组的数据结构，与Numpy接口类似，可方便地互相转换。\n", "2. autograd/Variable: Variable封装了Tensor，并提供自动求导功能。\n", "3. nn: 专门为神经网络设计的接口，提供了很多有用的功能(神经网络层，损失函数，优化器等)。\n", "4. 神经网络训练: 以CIFAR-10分类为例演示了神经网络的训练流程，包括数据加载、网络搭建、训练及测试。\n", "\n", "通过本节的学习，相信读者可以体会出PyTorch具有接口简单、使用灵活等特点。从下一章开始，本书将深入系统地讲解PyTorch的各部分知识。" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.2" } }, "nbformat": 4, "nbformat_minor": 2 }