{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## 第四章 神经网络工具箱nn\n", "上一章中提到,使用autograd可实现深度学习模型,但其抽象程度较低,如果用其来实现深度学习模型,则需要编写的代码量极大。在这种情况下,torch.nn应运而生,其是专门为深度学习而设计的模块。torch.nn的核心数据结构是`Module`,它是一个抽象概念,既可以表示神经网络中的某个层(layer),也可以表示一个包含很多层的神经网络。在实际使用中,最常见的做法是继承`nn.Module`,撰写自己的网络/层。下面先来看看如何用nn.Module实现自己的全连接层。全连接层,又名仿射层,输出$\\textbf{y}$和输入$\\textbf{x}$满足$\\textbf{y=Wx+b}$,$\\textbf{W}$和$\\textbf{b}$是可学习的参数。" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import torch as t\n", "from torch import nn\n", "from torch.autograd import Variable as V" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "class Linear(nn.Module): # 继承nn.Module\n", " def __init__(self, in_features, out_features):\n", " super(Linear, self).__init__() # 等价于nn.Module.__init__(self)\n", " self.w = nn.Parameter(t.randn(in_features, out_features))\n", " self.b = nn.Parameter(t.randn(out_features))\n", " \n", " def forward(self, x):\n", " x = x.mm(self.w) # x.@(self.w)\n", " return x + self.b.expand_as(x)" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Variable containing:\n", " 0.6614 2.4618 1.6848\n", " 1.7110 2.8197 -1.7891\n", "[torch.FloatTensor of size 2x3]" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "layer = Linear(4,3)\n", "input = V(t.randn(2,4))\n", "output = layer(input)\n", "output" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "w Parameter containing:\n", " 0.7730 0.1062 -1.4568\n", "-0.0182 0.3505 1.9311\n", "-0.6398 -1.5122 0.5403\n", " 0.1200 -0.3439 0.3741\n", "[torch.FloatTensor of size 4x3]\n", "\n", "b Parameter containing:\n", " 0.4206\n", " 1.5090\n", " 1.1140\n", "[torch.FloatTensor of size 3]\n", "\n" ] } ], "source": [ "for name, parameter in layer.named_parameters():\n", " print(name, parameter) # w and b " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "可见,全连接层的实现非常简单,其代码量不超过10行,但需注意以下几点:\n", "- 自定义层`Linear`必须继承`nn.Module`,并且在其构造函数中需调用`nn.Module`的构造函数,即`super(Linear, self).__init__()` 或`nn.Module.__init__(self)`,推荐使用第一种用法,尽管第二种写法更直观。\n", "- 在构造函数`__init__`中必须自己定义可学习的参数,并封装成`Parameter`,如在本例中我们把`w`和`b`封装成`parameter`。`parameter`是一种特殊的`Variable`,但其默认需要求导(requires_grad = True),感兴趣的读者可以通过`nn.Parameter??`,查看`Parameter`类的源代码。\n", "- `forward`函数实现前向传播过程,其输入可以是一个或多个variable,对x的任何操作也必须是variable支持的操作。\n", "- 无需写反向传播函数,因其前向传播都是对variable进行操作,nn.Module能够利用autograd自动实现反向传播,这点比Function简单许多。\n", "- 使用时,直观上可将layer看成数学概念中的函数,调用layer(input)即可得到input对应的结果。它等价于`layers.__call__(input)`,在`__call__`函数中,主要调用的是 `layer.forward(x)`,另外还对钩子做了一些处理。所以在实际使用中应尽量使用`layer(x)`而不是使用`layer.forward(x)`,关于钩子技术将在下文讲解。\n", "- `Module`中的可学习参数可以通过`named_parameters()`或者`parameters()`返回迭代器,前者会给每个parameter都附上名字,使其更具有辨识度。\n", "\n", "可见利用Module实现的全连接层,比利用`Function`实现的更为简单,因其不再需要写反向传播函数。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Module能够自动检测到自己的`Parameter`,并将其作为学习参数。除了`parameter`之外,Module还包含子`Module`,主Module能够递归查找子`Module`中的`parameter`。下面再来看看稍微复杂一点的网络,多层感知机。\n", "\n", "多层感知机的网络结构如图4-1所示,它由两个全连接层组成,采用$sigmoid$函数作为激活函数,图中没有画出。\n", "![图4-1;多层感知机](imgs/multi_perceptron.png)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "class Perceptron(nn.Module):\n", " def __init__(self, in_features, hidden_features, out_features):\n", " nn.Module.__init__(self)\n", " self.layer1 = Linear(in_features, hidden_features) # 此处的Linear是前面自定义的全连接层\n", " self.layer2 = Linear(hidden_features, out_features)\n", " def forward(self,x):\n", " x = self.layer1(x)\n", " x = t.sigmoid(x)\n", " return self.layer2(x)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "layer1.w torch.Size([3, 4])\n", "layer1.b torch.Size([4])\n", "layer2.w torch.Size([4, 1])\n", "layer2.b torch.Size([1])\n" ] } ], "source": [ "perceptron = Perceptron(3,4,1)\n", "for name, param in perceptron.named_parameters():\n", " print(name, param.size())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "可见,即使是稍复杂的多层感知机,其实现依旧很简单。这里新增两个知识点:\n", "\n", "- 构造函数`__init__`中,可利用前面自定义的Linear层(module),作为当前module对象的一个子module,它的可学习参数,也会成为当前module的可学习参数。\n", "- 在前向传播函数中,我们有意识地将输出变量都命名成`x`,是为了能让Python回收一些中间层的输出,从而节省内存。但并不是所有都会被回收,有些variable虽然名字被覆盖,但其在反向传播仍需要用到,此时Python的内存回收模块将通过检查引用计数,不会回收这一部分内存。\n", "\n", "module中parameter的命名规范:\n", "- 对于类似`self.param_name = nn.Parameter(t.randn(3, 4))`,命名为`param_name`\n", "- 对于子Module中的parameter,会其名字之前加上当前Module的名字。如对于`self.sub_module = SubModel()`,SubModel中有个parameter的名字叫做param_name,那么二者拼接而成的parameter name 就是`sub_module.param_name`。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "为方便用户使用,PyTorch实现了神经网络中绝大多数的layer,这些layer都继承于nn.Module,封装了可学习参数`parameter`,并实现了forward函数,且很多都专门针对GPU运算进行了CuDNN优化,其速度和性能都十分优异。本书不准备对nn.Module中的所有层进行详细介绍,具体内容读者可参照官方文档[^1]或在IPython/Jupyter中使用nn.layer?来查看。阅读文档时应主要关注以下几点:\n", "\n", "- 构造函数的参数,如nn.Linear(in_features, out_features, bias),需关注这三个参数的作用。\n", "- 属性,可学习参数,子module。如nn.Linear中有`weight`和`bias`两个可学习参数,不包含子module。\n", "- 输入输出的形状,如nn.linear的输入形状是(N, input_features),输出为(N,output_features),N是batch_size。\n", "\n", "这些自定义layer对输入形状都有假设:输入的不是单个数据,而是一个batch。若想输入一个数据,则必须调用`unsqueeze(0)`函数将数据伪装成batch_size=1的batch\n", "\n", "[^1]: http://pytorch.org/docs/nn.html" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "下面将从应用层面出发,对一些常用的layer做简单介绍,更详细的用法请查看文档,这里只作概览参考。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4.1 常用神经网络层\n", "#### 4.1.1 图像相关层\n", "\n", "图像相关层主要包括卷积层(Conv)、池化层(Pool)等,这些层在实际使用中可分为一维(1D)、二维(2D)、三维(3D),池化方式又分为平均池化(AvgPool)、最大值池化(MaxPool)、自适应池化(AdaptiveAvgPool)等。而卷积层除了常用的前向卷积之外,还有逆卷积(TransposeConv)。下面举例说明一些基础的使用。" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAMgAAADICAAAAACIM/FCAABaCUlEQVR4nGX9+dMlWZIdhh13vzci\n3vJtuWfWXl29THfPTnAGhiFpBAmSv0lmNOnvk2SizARBIo0gODAAJAAS2wCD6Zneqpfaq3Jfvu1t\nEfe6H/1w42uMTFXWbVkvX+b3XkRcX46fc1z+b4Q6QysI0WzakbHfXr68KF0yMVdQAddsqoYUWYhE\nRyiQAiIAQBpDRQgKPCAQIagC8UhpcgNcBK4UEuIenHzsVkfDkLLGASIK0QqCsFANk8pbby9TwuGL\nf/vSug4JISpVQRWRCJiGsAT7949fKwAQIPDXfkWQ7T8J3vwHfvMWEIC0/5ebl29ex///2+e/6T/8\nDv/a/37zJs7vkr/+OQAAqn/tvQBF8Nd/hIB1TEGQDgkh1BUKIFgLyIpkTgBKAcIgAkcIEAAURABC\nyvxXkAChUBIQcQqBgEQVCQjIaB9Iov0hcS+dawgsEEpRYbtCDDAQu11vlHxy93UNigsEFIgI2s8j\nBIYsX5w/TlUgAKABhKi2T+UAQ8iqFEEAEISAImRA5D/cthABIRQhQkCGQoKCAOevhri5T4RDKUFo\nBcTqIWUzV1VQDAQMQSGUAEP2rwdT1eP3Xj33mlTIUKGAFEAgAkhO/vrTV1UFAapCBAJTIUmG0wkg\n2oNPBCEASaoACIoCArKCQZFo34sUiJAi858QFSCCYgKIqApGUFQAFRErh1I9ov2WJlWBiChIkkRc\nXhanDPffG9w9CAgDEYEAKBKgZXn56+dFVKAkBIRIyiZgeIVQRYQmqiqAMDhfWSJIantIBcIQBaEQ\nhqpAhRARaQ+JqoooQCoEJAUhEAUSwKScxogIWDJTtZRERNohCAIyvdk5oet3HygjIoJQiCAEKgKo\n2fj4kyuKKtGeA4CaVILe3h9gUAJkQNoTQpkvsghBiAg1iQhFlYQIBRRRUUFA2/lUEQjkJnpQlxDQ\nlBAzUR+niHAkE1FARNQAUMEAweuLKYh8+8Mj92ihQBRQAVQt5Tx++fHLHRhKiEBUINrCJ52E5qwQ\nDWE7QSKYQ14EICJAu3ZBRnsMCBAipLPFGWmfvT21FqCogqRIkBBVcUjUqQQBigEqUEBM20cSitRX\nm0rK4uEjiXAQAm1PqahatsOXn1yZKaAIQCAQAiKMYIQIIICAkhntngmCBCER83GIYDv0ZASChEQ4\nJeik0ANksN2MCECiBohgiDAoCYBoTI52oNrfrHQSaHEtgM2bQw3Y6QenQhJRwABpydTMDp9/uhGT\nUNP56EFUVSFkBCAa1YMgnKQoqHBSBC6miBa0IoI3QSwAUQToU7h7jSBB0quThLTDSYIS7aaSEoQo\n3NniJiJICKM9BO2GxpurQkp//0NzRwvQZLiHmOw/e3wASsBMoWCLzikJAA9AAK8eZY7IjDnACkXb\nPQgyKCoCMjxACsRMIiI85ntDtsshIsJ2d1r+NJIUUxFCNcIBEgSjvYkqIqIiIpDDm9EJO3r/bkS4\npnbIIyDYf/rLqygeFGPSFn2gkuzmxotSVLUFwzl9U4DQVnmIMCDzmTBSAIRoey0oIolAQKESyrl6\nkBbABaAoolIUmO+nQCUACFtGa185hJB4c9qbSnfno9cTWUxBCE1N9r/+dGuSYCLepRZEAbU5QLas\nrxBJYJjOFQp1zoqirUAIbXEA0p5sDSGgFSACQgEDMCGpjJv6AzpnUAABgiFwd4EAIUB7Simcb785\nds8GQdLVB29+5SIBhShEdfzq823SENMIbfkN0vJFy2gAI6IlT6FHUKL9RrSf0W6gMGJO9EHOtWB7\nMghQwxlsDz0EojIXTMKbjMoQiIUzIhiBdhnbAygkW+AWubp0iNjJ+0e1eDtRZunw+S+uwNRlBURr\ngsCkRcb27N98n6gwgiSEogBBQTCRAmr7oQFqtGexxTSKUERDEN4OA6gAJSiABBiqBIVBmjjpAUSE\ntOKtXY+5CJprBS0vT/pE7e5/cD6pBKlitv38sw3EoHARuqu12K4SXiNC2leSaDdFACAgDM7npd2f\nCAZFWr4QkgQEAYEoAZZaSCLoDIItw7ZbT7baj6SIaPt1eJAQIEA6naKtGiFJXr0ZnbSjDx4wangN\nlf0Xn27UxDpTDUaVRChFTd0DEiK8qaqkPecQtNdCqIAowFCJ+Tiy5e24KcYVrgiJEFKiVSot/yBM\nGSmEFIlAyx4iDEYIRL0VrAAp0WJjC+/wV2e9gfnWh69GA9T08PXnO9DM6A5G0BMogFqUSgBJNObQ\nB4gEdC55oRBtJWErOKRldrjonLuYEAxRUtOyo1efSq0a1u4a2pkDtH0vkQAixIPt47YLOPcbrqQz\nHAREtq9WJiqLd77+nBS1w+NPNhBTUyVDKy2SSECVpdYATFopI2I6F82iTkBETUIFc5UFaQ9Jaw6g\n7QorxXQA0nJtFkljvD7f7ScxFaWIQOe7Q8icVOceLQBo6By6QIG3Kg8gBP76tDNFPvvOm71qmr7+\nfAOltkjlAVASCKgxanGoyZxVWomIVsGzPdet6lORAEWp7Y2KokYKydT3SOpY9pROKV0Mx+9Mb15f\nHmqiRAKgflO9EaL1plEUkC4xh/NoPVOQLiSFwOHFKonp8u0PfqHZv/nsShGqBnHCXRRIDDGF1zqG\nJKAVhwRb9lYgKFQqaCqCCFOQKioEFRYUOgK0vuusloRhKa6SIg7FOhnefnvz+tkFVJ0a0rpnoLLl\nltBoBTklRLSFrQDAYLSoRSH9/Kg3RT799vl5PPvsGgKFIAoRSBKBRLEsXuo01pzmvksUgJNEKPwm\nMRtBaItNUGG7cKSBUMqyL2S4ZB9FtDA30IKWz87e+frLrdFAgQTnKspl7lhaVGknRFpean2PiN9E\nm93Tky4Rw8Nv/9XjzzZIKqoGMKp1EYAmiKqUMu33kZKqANpCKV0NrAoSkRHhEipsBYU42i8D0lmK\nw9hZ1XAMQj8kFXdxTYwkYKTVd+79+kkZWgVjcZNySReZQzd4UxDEb5CP1hA1wOHy2bpT6vr9X3x6\nTiEttYZXTYKhksSSllIOm11aa2sx5roIsLDWH0soENoKKhjbAQId+c64LIf9gXWXEDJEq3yM2Mug\nVU0pYPD20dkno0Os5e5WEUc4lQFI6x6DJhQp0lrOIERDQBWhv7ydRSUdl1eEaUML3GFamQBLahq1\nTturwwnQomvrv4Q+KZOEUiWgDCVAoc7PBsG82O9wMTpFQhkl+5gH8yyTiiZFb3Sq0AT9d+786rFr\nnsEezFk05hg4X3cXUVUndG5HfY53Yvunq5yFL35akwDJoIhCM3cHsqaUpdQ6Xl/GcasSMN9ddw1t\nBwJUaVUtAzegEhly7+p8ml8zRFBdorikQwxijN5ImDrEGLj1O8efbDwlZ6sFWpcSAcrcykDmGjkk\nRMG5cYNRlDx/eST+5h/+UimqSUI5hVp4CDJ3ydRLGXdXm67lutbOesNvFNB2C1wJhlChEtCGAuX8\npFYjVCqsqovDqqiOOaUsIuFqodEAJ/H+e7d//rgmzD11zGEDpioqTlICgqSt62kdAzHHR8SLO931\nP/4nbgIoQlGpWWrV1PFqlzRqrfuL87JAq1KoJIMRhBJJJVoxKCLqgKDMiI0k+7V0ZAggakYVTcW1\nZIUFFeHRVYE2wA4i6e2TH/9qZ5rUWsMqRCuQBYLwIFwoYhGtqWrtHKAUEavn+//tH22SQJSu4d6L\nBJMM3eU2JUSth93VJImikIZIYUYTG9ITJKgUhgjbhaIicFJHVS+SkHNI0cTkNUlWTiqTmqUM6bSh\nWySEXP9h/vHEnLKATp1bAxFRghI1AiGdzhVqq9YgDXFQ++bnf3athJmGBCfN7i6pW2633W8l1lr2\nFxtR2M3jL2x1okCkIZk3OKaBou1YiqS7nwtLaAcg1L0jiQFOqhahqQFi5FzYExypyz9+eP7rJ1kM\nJF09TGBJRaUS8LEqzazV1d6O+VwKqPov/+01SCZT8RLIwRDtht1FfPdvpVLKYXsR2dHqTiVdVCgi\nghDzdjfa4yMEQwFhGIbXW1ZIMndkKZ1UmNINNLdgTknVqlIVLgwPTcvc5bv43n//mIPQYd5QhohM\nwCB1PKiryU0jJPNFCxNTef7xtRCiFowa2isrJC3lOt76vQday2FzeQgxzEjlHNRDSSgrKIh2cxCg\nmKopQKR44w5NiaGGqvQaUVyqdzXUCFVDtEEDxrFgfet0mViLvvdHsj1UB73hDsG5bTCdrs8vdqOL\nMRytSQPmfHD+k6cANWUlCKYUE0OXq5CHv/8wJ6+H66tqN1D9PO2IANsxTU4ooQ15allASBPdbkM1\nm0Qyp6ubWTvWXtW0dxc3qFX4NOX1ojdHFSol/+CzPz/AIC5gbbARQYGYHPY765KqhgBUNuRDIOnw\nyecjVUUNEh6WfILYshvH299/tytpPFyf75MREvxrBwSWqjAULpTkDCPlBginKqRuqopKSEJI0Ltk\nMCBCICYWEgyYie/Rna671guLllA9+s+efxGdKcRbdarqrYmT8iYNq2RZ2Q5IQF1ATfHNzzcgNJu4\n1NJlFtpC+7w9+uC9vlDHzfkGppjjA+aUpzOewoAqQcAYIXMzAZhvA1ANJq/qNVSFKh6wjKQSwRhZ\n63R14WePbvdRSuuQzadDvfd37h7GGnTMnz9atSjJyvWbq32hWMOctRXLKuc/v1Cz1CdSImzQUpG7\nIfHog3fXHPe6u77yJC3I0oPzICPIEBFr5VzMwJ9AVBUIxG6CKUJzDU6TdkkLcqhZUjWrEyqN4S+u\nj98+TWWsLR+5V1Mgf/tv9dN4KO7z1WnVqplm7C+utvtKaWgIRFTN0u7n36xOk2ZziFf0UpnMQq1/\n+GilnDRtLg+SGRIxt8tz+qAIxEQj5sbWbW7IVGByKEgCEXUGXbUDu05pELEpaBopOa+ms0c9a9EO\nHgGlJaPQLf/h13/mKdRgrcckRBpGVMftfhg0m5M3oJrhya8OVcLEwVqi62s1YaS8uPVoLTEtjtLl\nVaRWj8y1oAYJFWu1tYFo9S4BKjwkVK3QIAgz0hkpKxXKAhO1MuXIZtTry/7BnY7wJO7uGpqDJQRe\n0/I/nX61DctSeYNnCcBwmG8O42Rq7dAKREzOf3VdoTlrIKaSu3oQkc4837l/mmuxdUqXxQRQUuEz\n4kFECIGawxquEqpQUCVEVKlp6yatUXUzQpJCkItWiyhGgzO9vrz7aKGFfVeICs2sJRxJFR7prf/T\nz//xkwJLNwNPp5hABbx6YZ2aSoN0RMzGT76sQusUoEcedB+dQdSO7h2rhBxnSRNMWks+V9Jt3tQq\nkdB5oCekCRkNvuvKFNppoJVEkC45UoLRJCbWRGG88g/uhTOzaERSLZMkms4le9TV31j/6S8lp4gZ\nPxWNCDGbJh2GLiVtMUBU48nHW9HUAB1Il8fIAEJuPzpWOtYdkMRUqCGqDdSVGaOFAhTS22w4rE1R\nFBFHunHXZBKCQEC6HJY7cTE6QgRap3H9/plZwQFKkyroUkUOiFRVhnjJv7X/5nzVBaKh+WRtZVns\nz4ehK52AiATRy4/fqIqZOFN4Z9Okpg49fe9eioplD4S2zkYhqq0NaXCjBJUyA6UMb9h4IKLKO9w6\n5kEOgmbhZknrFGIm4bXudt1772QKUaVGHErkzjyj3kR3hirrD/8Ypbj/ZpIkMFOqlovXF9vD1E6P\n6PTp11RVNTcLF8OExHCs334wsNS+F0BSan0n6WCbezagVMGEgCkDOo9eVIKIo9NPvGsoglKSmTpE\nUzmIkVE5ufDew9sBnwitWqDZQNFI+17aNIpAEu3+6LMvpuJkGwgEW7GnPp73ncQiIQzg818cBKpG\nMAKdVsnZD3723tvLqL4eLKiSFBQoSa9t3oM2wEEQdLsZwkrSVsY73sZWQHUNE0sEkTTpoYgi1CfW\n3fL92wMWY5lyKpMtxFyYVAuyKw8iSQkEVd7/r/+vu9pOSOvhIKYuGptngltvn8rT3fHuzc9ew6Ap\nVULDxGuyKCXdf/fYJ18uDaRLEsQNeDqXUa3mMhHCJMRAhpqydYp+/N43UyJFmVhURc3E3DUHIWXH\nuj97535H38NQNtINqZXD7eGsIEyV1QGk9Nvf+zde0XgX7We2Ybpf3f+d3z+J6QjLZz/+IsTC1KG5\nqriLYfThwbfOpNR+mRokmRrHQR2MeZbXarhgNbdiEgFoeKiSDi8PF2+qQCHZoSDV1FIx2yPpbusi\nb7+j2h32sRzKPvUWruJG9ySVktFXV6qGUop0P/yL4pznmARCQbCGfuv//IeymYaTTt67szMJmHry\nERCRnKZJTj56kErkVd/SDZOEQNFg8/hrAIcXBuIGdhQq6BB4fv9QrJpYuIhRqUQQ4y51st/Xw+m7\nR4tUD3vYYZ+OELUT1xDSpjazLMaaQqQKJu2+8+6zUhviJKCER3jU/g//2+9ebxani6Ta//6fXoNJ\nnSJO9TBx6vDWo4W7LjudK+fkgoZNa8OnlUG2uY57sjbJFBCgGpxHt6/b7C2UoLpl4TQVP+p8ci93\n3rmV0mHX91pK1qqKCBd1xKS50wiNmmLKACNhsrv/yd+fAmINpCei1MqTv/Pf3Hl6uHukUsX6j979\niTKq6hhMqDC4929/cMTCRQdHgwpTYiMDtAQyl8BAhJg4tY2foPMcQaa7y1ebEko3G2GeBqtThBzr\nIcZxeuvdLnNbGT6lEz+klVUUKVkOklPipFogRUELS2Cx/Puffh4VFoogGeN+4jv/h/9UPud7C5aK\nSLZ+96eURp2QKGoqwPH796xE1zeqDyhM0igMiQ2CafgD22A3KW0eSavAlEK9E8WLAWL7cRVdzzq6\nLBbjPte9Pnprmco+8tK9Q2VOQhtFbMO81lo1xiG5ppkzAdGDr/7mqwp6GyWUcZrsB//t71w/Pnq4\niOoSPtR02s6tOKVGEoma3no0lEi9zXNMNC6JWEA6qFBIhVDVREzF5jEPADWAwoln7qIGjWtZWjZ3\ncem7QO87PHwk3FfXNPWDjHXofVQYp6lbaMAmGotqoCsOZIEwCt79k09qhJJwH7ebW3/0X90/P79z\nx2qU0aRLAFw1aymukCRSRrn73nG4DBkzKtLQMMwUEd7MXWdOj6haSyk2D/sFGI7j+mAh7nJkIGNK\nXRiYpjfy/lq6cw6r7XbZo2SyCqrGdFgsMEmGuauUzjav0pGKEK5w5N+9+7GnAMLrYfroP/kenvrD\nlVTUKjATqaOoUkRBt6zcH87evyeT9501yI3i7Yu083FD5hPY/Ep7tR2fgEKEXK8O1y5TktwFXYex\nS9JxX20j75zZdueQq7zgoTOVXenJuuNKwzBSDbAwTud+xxrQVEPI9E58AgnGNC7/5IfH1/t0Zwh3\nOFVNheVCESGmJSwJYrJH7w6M1AJWowwxklK0Eb9MwaC06ksMAMJEQFQlAsrwadG/PGguYlZCpPeU\n1GIKbNJbZxGlLPTVyRKSIkJyrYZrnnZl6ozOEOTE8Vm6o25wIQln1HTv1QYRkT54+3R8XRfrVKOC\nVEkGcP9czGpVCYeRRW9/cCJVujw3AA0KTTPECkk21/8250VADCHaJhguAbif8tL6Qw8cxPMQUweM\nhwRf3urC92l9qad2GJJXj64f9lcx5MqujNGxLrJU22yOejBmWF9kUsnd2TXp63fuxHnVkwElglCF\nqYjg8lxMLG4g1rT86FHnrjZ3HnNySDIPVAmBWCPjhYkEAEObJrSiUYIVd0JNoGCuWaLkLkYZ5LC8\np9y74UKOTIJenB3c95vTfmIANe+TFj3sMxdWgJQkxUGhGkWnWFlZ3r7THUb261RrCKDKlgPk1SaJ\niLlXzYoun721cpfOGotCxQMgUptCzzQUbZBxS/WNeKQgIkRBF5Z827WzNNE0CZCNU+6x7W6Z+rbX\ni3ysZYHzIWUiXVY500kRBRodBx6u+myyS4PR3GsxK8Kaue/uHd5ebA/W5SwlAoAJGO20vtgJG/6e\nCMurR7etRtcwNAJs2E6KOaVry9cqNyEMjCmbwxAEQmB0Pz12tWwGqdFLMmEaYpPvGWvJ5TovXZcH\nWE1d2mzHfs2oKVGqdFxy8rUQuqRI1Ni3wZXUmkTumO6Y88AKEmECIKZOSKlfHjJcNQIilvo7D1cM\ny/obeqEIqZJSCVBCwAjMI90ZEmpko5msBaiDy36TrFumulehhPcmHOVOln0sy3axiFSvgHWne62H\nRZbCzsckElO/Zc6ufuiTVmJyL51FAfZ1fZTr9dClPscYM/UJ5DVyCOPqkxCA7jRNNW49OJWKZDpX\n6aCCKkiVsPaxYx6r8oYMLjd4vIRQ1QF0cA8dwnwEY7JFlX2+ZyjM0/nxmqrbabnQvvirxa10uOot\nulKT0veWwWoWVVWLi1RU1VonnpzgUochmdZSDW10DLk4zzlT9atvjJGlJ0VR0umdvkbOAvwGw4XS\nIoXS25lubRrlJnmomoAiEg3cp5BZAysH605YrRc77E7vUaun8eJ4qFmnqdei1baLtVTqOFhV33LB\n0Fo8q/STSxQ3W4y+66fD8m461L7rknhl6MxThexe77v1lZzgLy+yqsRYwZL74wdrhHbWxg3RAAZh\nICFm4rfMRNfGT2yEaGnFGkGRkCDhlehLz0Gv9ktZlbq6naZDWpxfnh0ZK2WdfCsyHh3j4GY6ap5C\n/ZB1K10Oj6EbC8LroDj4mO4c+yYvOwv3IqKN6SIi9fVW+1ebR4urH8XCqqJ6YqxOHt7Vwk4A+Pxw\nzfkj3cBgOp93AmyMpJm26LhJKxGRnWDXHfLywHQ4HsvqLkbCLq6PeoiPqZN9v9znpUwUq1llPy7I\nzK0iCbSQg4wi4Yd+OsjpmV/r0Cfx4hKSRG/o25dv5GT88gz6688x9YtpimS1W95+Z8WQZDcMirmh\nFSJRHcJQwmaOdWPVykxGCYE3TqWHx5JVZCQhnpA59fesjildHBbHqJysMtY20nq/ln4hI2gc+3HT\nZRhQs3NyoSuqH2J1Zte66PrkpTpERVVAhaiML+T48GR1AvvZ3iJKhOaQfrh9K03I1rpJaSM2CjmX\n723CB8xdCQTU9pw1OrCSiQQcK5ecd0ncFukQ0/HdVFxxUY/WzFNJpS6GupETO4iyBMcuqdVLlwVc\nfeoKWCbv7JCmms+OxjEveoudNyhTtUlDVPF6ezy9Psvjg+3PLSurmoCpO77be5gBM+O8cR/Ixg6y\ngCi0jWnYJlMixojWBQIkKkQZstIe3eALSO+Y1mcTXLo3dTGIBmQTR71s9YghY7fYcz9aFjVZoyLC\nbDRalDSKH/T4dt1zuUIdi2unKiKmggDU9Or5MO2Wu/3Dsz97IpogsbAaabh3gkA2kNHYor95sphA\nSU7VpG2OK2yc7CDdE93QKksiIsJSCluMk+bVYdTbedpnPZflSUy5xoHdglc8yTyMo6TefG9XT/Mf\nQqLCgomjliTObbaH3b4cdckONZDSzGhHG/BKvEBcb97s335o/37Xtal7qqYnd3ONlERmztBN20EC\nqZFMoCqtVqSQqoxGDjLlzIRQCZAiJlqlUy66bhl7lXRRl2tz1Ho9LGSssub+6oVinOxscXho9s2z\n5ZB0EynW8Dcv0gcssbjHy/5W1pgqEqAmM+IMiBrON/7yq4v+wQf3n/wkgpWhMYatHhzBJamgEYjm\nNqMdl9Q+vzDsZqTdIDpFWDsviZh5VIhq7DSNldC0ttGsvzislzrC625Q9Fe2Tnz1/Pj2Ybc/fDW+\ns+4Gvn7RxfFxjmdWhxf4+NUfxoP1dVkfDSwUA2jSmqJGyTPZPL349Bn7s299aP/0RW/S6G7UW3es\nQpM2acpNxGqgLlIrP4ywpK1gDLlhFgYCotoYW6KVKEzjMmoZh5RLWKTLw2qZPLCrnapfy+CyG9+x\n6/PYbMrmXrUiq7G+/vTYMO2uNlhd/PyHH9aXdnvFSjdtmMLNjFUgWl598cWTTdcdvf/B6sf/C1IC\nPYjJTx4eo0o2URLqrQhpJDUBUxPCRDKBzlS2G9j0hi4LtGdSs280NPebcRvDRUGn17ujVZpgW/aa\nZK9HOm0vjvvXL63GyUbs6RDb5y8rHvCzfD35pOf5+/f3V7dPevFG1poJdGAkiHH34rPHl7va5fW3\nvnV2/aebZIqQ3mKS07vJo8szkUhJBEWlsSAlhRprowQFVdjKFxUEVQAa6xyYNYRPiwky6y4O7Fh1\nv1hmB67rUuGQBWp9tegvr05kczd8fPPqrjzZXS3vb68l191k9uhPvrOp751pdW+P9w03HjDqxee/\nerEV025x561vnbz5J38pKQEG2iGWd9cM69LMUwu5IV8qGCJJldAqmVCyNVttkjDrY0KFUBcJknhV\ncgkzjuuFXsthHFYLSOxLhtKnhdZ6sVzuL/puf/j8auubo323snuc9ofoQzA++tvf+ubknaHc3OiW\nckkzqI6//PdvJkROeXn/nXsnv/h/f1EtM4hkgNy6lyOG1C51OyVzxwEImFL7+6KNpUIxD9xFlJwF\nVC1eByxdXJ0JUsXxkcTyep+HgcJtdKZR6zJxetUttI7lab3e7RTDWe/bssT6dd4sD7v+e//V0ce3\nP8iH0TqGOE2aOksYMj198rM3apGWt28d3bp1/OLvfp6ytUNgav1bJwjtcytgb6DdebINiaQgqRKz\ncgWgtxJZSQUkGmmZKiph22/u7TzyopauO2xTvxTWfR0ImabVMHGc7vLy1YvSd1GqDB8syhfPh/Xe\njp4cDgW/9d/Y14/etX1RcyJcZ02GiMbTXz+9czQWOzq5dTT0Q//4//llJ5pIhJgxHz8cwC7JTICc\nw5ZIkzgEkwQkRzCkac1k/rYKIGpuLza9BdTik9+V4PF2FxHbWB0nxnZcKKNOKRfRXvni8Ru7e3gy\nMeTe2e6bJ/4t7m599upWf+uD/2y6+tbbXqoYy41yh6JifPnLn+pKT1ZYHPVJIh+d/99/oWI5EAaI\nT3bvRDw6I+SG33oTe9txSNq4qdKY+CJo4kgC0eA5GueGXsXsyWYhsjiVq/7JdjVk8fPDsVHrPvJ5\nXZ3Qn3yxWa9fvOGw0+WD5xdf5+8++Gr5yVZXh7/xuxfjh29xcioqqVQqRETs+if/qt53390/VVEi\ndWe7/9fPRbUpc0y55/rBMiLn1k3NQ9s2ppVWzafQxiq9uT43z54CUCFnIq4YVWDd7vH3OaGu08Xk\nyyw+lpWhwHicuDm/HK6f4VRejHa0G9ePps8meevo+v6vx2/99O4Pbz2X753oWN0k6HB2AtAkvvoH\n5wt5eImJhInY8mj33/1FG5FBEDCa3j6Rit6g0bBpypxHMLMc0gwBULz6DJq2HivCm8AlZlmuKtDv\n//1HPa6nI+5e3slFpst1Rq1Wl8m68eKiinXddZke+TM5Sk8v0mrx+N7T8//48fD9O48X37vrhcpG\nxFSGqjDq5//sk5O7l1cne276IWtaLn/5px9HsiSicIggsLjT1dp12vQqwMyRm8VBJKGmqqYRTSiB\nOYc0FggpgInUaDQo0dw9/onlRa6rtFprHrddNkClz1fPX4/77bRb2uuL6+XbF55OpnNmfP16fHJ8\n9+qH+OXp79wmPSBUrSXMAIaXMumj04vl65p4WZ151f+r/8vPIaoBM1YQIt3xMfc1pyY28SY2Upln\nBwDYivkQgUclIigioDb+SRP4SKMdqwjEMv/sada7y3p1r1S4DibZtOoEi3xn4Yvl5etL3LO9dV68\nw26DVO7s/vC9L9bfWdOjVnfCK5Oq+XSIvHj4XuYadr1IUm19dvgHf++VStd11sR/YqqL07Uf2Nus\nLpxx0Jkg38CeBGGADW8JKABqC2wMtqIXSklNJ0X6+V/dNaY3dXWlKEvDJFOXNkybJ2PG8Xo7ZZf+\nuizvvhasjliFZyfl/uU7H+US5uEhrHWWdFnKORYP3rzaPrx+vTgu0eHn/8snNUmkJFAiwtQ8dWe5\nRp/niKoz2tNocmw6igShhEgwAoJI87dtoulZoTKP2UXMLdfPD0dX6G9dW9lnkXEni+Spf/P4Wscl\nYyuah+v+VtrtVFhsHC66zXa5eNvqqOZTNXUVoHruVUQlre6/vLq4fnD4/AeZuyc//2Q0agbEBO6U\nFJFPTlhTbwRbWzE/KMKYvwZFZ0Z0E1X5zN8Qoc76NtWmcZcmtlDTacwLXXBKQM+xDGcr2b2aPGTa\nvNjvok9peSkf2rkwyv78+ird90lWVA+ph+KgaSCmtBxMVYK2PFnmiy+G/XOkshsFQeRMMWFFUMKG\nWwt6Tg2zhvA3lgO/GXsoUtPJNqkWbwSAgIIiYjOhEI0MPLNRIH15/jSLHqUNj1bwst/sXvb5aoxs\nsVo9wXh4b/eUSkeRW+9cLe6/fbYS7eRQPDFhXzs5WonOagJZLUxeLLuL+7VL21CqUBCqFapk2NGZ\n1bTIrgLe4KRsLXn7BRuooE186IQg5jcz1NgwuoYJN+CLaoYKmT7fnAR0qqsh4oDxs+CDqz10f8fq\nEkxPH71/eVGKV7z7e/v8fXYr06TTNFFMp4PaoleIBiURfZ8NfPqoVE310lqyVk0OD00e3dlaOfTO\nbBCJxlxpeXCm7kubUYTfqNBxIwBsQ2nOIlwFxUTgFNXkFypPznVfYipHi/35OFatu7P9NYF6nqZx\nKf3063Tv0Wkt/vA/W+sfPIpdck0YJ9L62NuQTRgUzcly16/WXfLDftot+nHn82wAjEII2J3czm4r\nq9NhdALzxEZuQOz2lCXX/0BBby+KQERNbzqq30h8bKjqlstXP+AL9MSkp6zlefZNZDl9PKWevJJh\nq4vDapzefIVa7v7tvPv9W7vFfiN5KlUMuVzqIiczo4JiCebro2Ghfnn68qMMGjq2XrUGJYn3t061\n9gslvSYx+Q/S3/ZQtUvfBI1AzLepvUEZM59Z1MxEwaj29nFQxLrn17FN3eB1aVrfHF68fnrtne11\nOaTA5rIe7Pb9w/P1uI/T//zo4lt3Jul6uKm7dd3hFTuzdFO8KiGpP151AzeH11dYLbOGqKgZg5Iy\nws46l0WnKkCt4ziVCCJ8pjHMQppk80kndZbHsvEBBWpmqd0qUmVdzo0Klc2vfr9Tl7SU0q0OGi8O\n2+lWGdPyQHZ1FD/c083lu/3V0Z88Ov/wLrraFQxJA4zxkAaVnCAQ2sxC6Ja9Za+7xTVSv+1qE6ZF\nAKDb6bGWbpGqImZJcoXYLEhv/wZC2zGnhzFmBQQEFEHK1jhiToqqlQvCzDTh17s7KaZ1ThrnmxeP\n62mNfRwvrxEmWNVxef5ykW4v0h9++OrWfe+GIfOAzqcou7HrTRWtpZKms0x5MSx6kymuNJYsZjfw\nvym9f3gCdkmsy6lBofRpGvdTrc2egiQpiSCVHig3Zg7a0gwjTOEGyIw3docmDu22vzIbV3012b1+\nfB3vXdU0Hs6edUevrGg3YVv7J/Xu2cMfXB7fPXSSxPcnQ706wKO3XhURsxYZAFXysB6Ww1gP52WR\nHA2GhgtEC1Z3uiJ9QlMYBwm6hjOm2RKBoiBTSOPBRosGmAmNTdgzS7KFEFGflBVA5OEXuQyrJFN9\n9fVOr6b9Wndf3x9ubYqWoxjq8/Xtuv3lb727T2+FG2hLsXS5mURMk0J1rrd1Jr7YsBpWw+S1bgfv\nD00MWScmZenfWofnTqIx3JxsslhRDwqSaGON36ifm7xtdvS4cRD4DYOraZ9cFRE1hv3jPbLXsU5X\n17e9pKEvvezftSsXX+3rWZc6xXa4mD6QiCkkFvcXm22xvjNTVUi4A+5OujPIYZWTupT9drGmWeO0\nBonA8nZfIjfisRCWcjZtGBwYXqYxwmMeAjdDBrY8CswVJRlzNmxsbGUg5a5fnAx3P3y/VHcvL7uT\nEd3JZeWDd+v5pEh5Jyf16HNd/0F3/e3eBgmI6Im+eb4nNCdTUzJqjaZ/igBhXZ4c8N2mWyz6ZRJh\nFEmmnk7X6uz0NzwMVTVVM5nHB+HTWGowga2YVxIebPVv61hUSaoghBRtAiPw/tn5Zy8/f3dFP+Cb\n7f3DfrV6VpadP52uPbHbTUMkcf/++49/+9Z+NVWDme9/8tnyZLUYFktVZcBFE1itEWLqftqfT8Ja\nr9NKpVJS3gdDyf72MqLLmL0W5rgqphFNtC5BBIokgopZQD63Vb/5RzDXmRpChgkE+tHLf/OL/fLu\n78WPXu++Pj3+Wm7d/gwsG8/o1Ff9Wuvw+s2j33n+8NG0WB58SCoXP356hDevDsPJ/TunFsICCzho\nYvDp4skXzw9OQi70BN6PglqaHHaxzhH9PFcjJFQEUkUU2hSmgiAdSefm/WbWQ2Ez8oj2B1MjApJK\nhErqXm8PSY/vbF6+fsm0qherW7vs9ThNaX0oNW7f455Xqz/y9XemfoW9rTKffpp+gNifP37y9Sf3\nPnhnoQFvKgWX2Jw/+frNVI6iUvYbXy+ufQiMVR1KW57oaL01qLox+G8+GhLgGkGp4UggFN6UIwz+\nRjIpYm1qJSRDRcEpYXGqr/t+aXX6+pPww/HwJd4v39z6ouuvsDr9ZpLxyfv6Tdn/jbfOf2Ac+vPp\n6Eg//czW2Nr9vn9xufvm/PKjNU08FIBPrz/5Zop872Qn9VAjqqzgKQzRuG5HQ/gqs+noyVnMqwyB\nUqBI9BBKJDdRSoR7mym0g66i6g0E9tbj0wHpTo+qHP/xZXxdBn9zNdzS7vjsR+m31qH7cXj0jMl2\nn9yKVw9/8PJby2m9iGl5Kn/+acaVnV//wcudP9zRt6+QUnRirLE//+KVLq5jezn6cgorzgElN/IC\nPI5u55BOZ0efNr6RuRUJgSghuYYok4LePKva8NMoZDRfLFcLl5lJAD0+Giy0462VvvjReFit7p9+\nvFqWQ7pcbjrE9f7tncOmx133Bzy+e+iW3T4v8U9/gVoO3NerSz9+/3RZUDarYCgMcXh2fvl8t7dO\nMouHBKUfpsRaKK4x3TmzyX6j7wWhQGAOp0LQg0JNzkRpJkSM6oEmgAl4c/9inXVQ4KQPvrP+QqrJ\n+M0XJ19+GR/eOdw5pL7Pmq6kz5DtF7/XfXKdhNfff2f37ZL6XH1R//GP6zSKR+SdXW+ufu80hfjY\nZyCZvfnlXz4/0EqlFNsLWErGskREhahFDCtg0c1MZBGRkHa9Z9V4BKNZLKWiAQRqKbXyBn8kiahK\nwGFBStTyzt+49eRgKtf19ZOvz+Pb7365enpxGyf72k/jermIevmr73745LzWk9+7/LDb9cmK1X/+\ns+k6FFG7W3+z+3efXv2kexgMrwHr7fJf/ctrWd966/Nvxj4OUEr1XJBtqhQguLjdwQbjDJ4E7MaS\nRQMMDY8GZAWTtijmDG/Rl2g+WBC1ZCpBkfDg6qg+F9Hy/GQnr+zte7+u61/07yqem2/r9OBkb/L6\nr+L4+Dx9b+EPDoN1ovyzX+MwLW5d5ZfHi/c/eu8ff3r76tQHdwfzUH/8r6+rPXz43dXluRoAWKmq\nQ1EAqphw606qKZOzzQAQFG/jkea34oymdkBqUp2YByIziKdCACZmMuvocPX5R5vbxzse4urTK763\n+oWcvra+366Keeh0effNVmU/Hoq99e7l749Tv+iDP/t6vRddvdVr33tef4f1yDeqpkm6nB5/PBjK\n/s3y7q1NrRpJ8hQL6WsJFyHQna0ZXSO9NapJS2xEtH8j6F4R0CFpSDN2iWaoheZEBklZRSRJaGi/\nvXy9efr8d753+OrPP67X+nDxPG4d0uliVY7Es0m5OGZfKTgI7l/f98tl10f9+NdncnTxeeS3vzL8\n3oeqJ+9eLkuuTrXc+xN8/xuN6e2BySrIscv72iG6yWGSCnGyCOTZP65VHLNDA7w5dzi8BNJitUz8\na/3iLDWe0UZTUxNTcH94nk4eHN97eHf7p19dV7n/zgUEq5QMOznuA9Ddl+wqGLX78L3dd6fT9VLH\nX/56SIuzM7mM0+9v3v695Ps3t6EdE9yl7w+XZfHRC3nrh3VfpaOUnAR7y8yoASSwP5GwrLMeD6q1\nAVitp40gIxzDYjkMw8wTaEyBuWAHRMxEVSFJmacfjae/ff/ktE//4h/+4uGD8dHdePw03YvusB60\nu/d6v6nVdhbJnOy+e/jeyWXf2/jjXx91YkP+9vlYVmen56NGfxRjmQQOQHj8oB5/186unn913uc9\nmY0+5fUrFp0EQH93Bc8WIQwJCDrU9nQxPCQqhMyro6FPKSWloZK1yZUbxE21hlmbqkjnh0c//M64\n3d766b+4jP33p1ff9M9jsbxMm0Xcjf2jFxfIDDL76PhoOT7YLPqu/tXPjvpODHrU8XJ/cXg96qPV\nMV54p0CUArut12W3fX355WFY1H1U6yMmOYrkbhDF+v7gyBZtYkhy8ibeDpdwMii2PFoNnQokkVKD\njviNoVLT/kmbxlHicPxHD9bj4Zs3+Wo/8dXz35pubR++qpPkVM+P6t3rnyZoUAB6ffDDq/cmzR3/\n6t8dDSknsfCFIg7Z8oKbcJOO4SU8UkeUap6WeeC1RLiQLkdaKumrqOk41dTNjbkS4b+ZVEUgQtif\nHC06E2kAnbt70IvHDU5EoWizA2Rwyr+38IsvPysn5bqsx8cn357s7PVhk7YfbJ5D7dXJbgdQWSu7\nH6bhERfL/sf/sl+opU6ZQF9iqSl1KSI2XYpABUtanIilute7k9QxT6LujEgyUpR2qP0CTBpkm5Vz\nBnHDmxorZH37uE/N209SeGWt4aVMjoZk4wY3mWv4y6G+vHo83Yo3vjgrd/Rn48mznR3Ong/vTq+u\n02IsQjKMpm+/9+q9Pi3Wn/0TW5uRopoFrALLKkFeo7OA0GtIGg5VxyrHB47ZTNUQFZ0ipHIq6bij\nZhB6Y24jbGRqEvSiJ3dPBuVM9UvVnR4MlxYdGlyNG2sRQNReyoD3Vq9+NeaTk3xvvDj2c++6h93T\n9Vntv/XpQUzc3ZSL3y7DQxv6l39/87BLUIZZgvRRYRahUdzMKYzCEENfqmZE5zWBovRp4kIPU4YG\nF6fWZJuz69o8iSPhZHU9fXA0GJr4TyQVkwqp/hvLxeYt0LAtVukXi4FHQ7e7flIWPEl9PbynPzsI\nNy9vH9eX7558NX3/6uXWAjXqhw+efrTquul//PJ+JwJPACOLmEmQGVNtg5bwUlwSkSQgiZKSwFTh\nxbUfa8dEOTq1yKmZkalL+yrNwJYROL53PFibjohCUgjnQw1rhdYMNkRIxLA+W6jj8s+/XPDSkPua\nxnuHj6+HGvtvpvXtxSeP+beOv7pokv3uO9Nwrx+6//Vf3+qTNWTcKCbqpvOPSM3RtRbSpDAZSNUE\nipm57B2ngQqpOFkC3exFGE37SYBsE5Dl3eOuiakacJ1qQaVGqyFlngmx+fjChu76cjEd3pwv+wv1\n5d3ro7N0/uKVRaL4a7w4ev7yLWTJAQrrB/cu3llb+vhPkftkCCIsyCYlhsA9VNisxzwkSbioeUBR\nS2hdOad6OIkqmZGOO6SsTUMFidkiEQQQ3t867mXmb4iqaQonwwnOxnWNwtF8EVTGYjnO38SJ7I9e\nrd7ZXTzqn3+6yPtIhB4usVMg+NYzOq3kb0/5vYVe/A8Xq2xCA6takWRKMS9keAOYyMTqsDafVAF8\nAuEWPvp0lGCLAxbHitw11VJzwG3aOPEI5/HpYK2ZVZoaRD1qsFaPkGgGDO7RNHCgOxm7i+MPz3Yn\nq9sfbL6oF7UuNozOVFynaatp8Ne3vp28TvWtW9d3l7n7579etElrFqLSx3FyVgD08FKqh5PiB7cb\n+jgYY1R3kJPlfs1QseNjcLGYDbFnS9iGkbpzebZMM+SmXU6qojUajdQjmjmRewNTGTGs3AP5vXd3\nj4/T9Ojl01V+VoaTfVokcfcp3d3WznX3zXf+5gBZfLS3d4bV5/+09kr31lurwj3cI1rvwDo1J8qy\nc9Ua0aSOZRpFsnjsJ8Gt5E69NZCpmQoSjfwXLaRGpLOjGXuEppSarrOVlQxHmyeS0pzuoi6Pqt1+\n79H4z3908u3x7OnnJ7f7+8fD2aJ2xlKc5F07vnXnvU9/8dv3w8/u7+/cG7b/4KUOQq8VgDcHoupR\nahMbl+oejChOlw4EaxCY9kWzqaLsvK4thdtZRrYmBZ2l6cGmuCRXp0NrmSTllEwYVIiaoFWSLSrM\nOccZZXn0ztnVm8+v3nnrWX75cnl8dHx/L/5AJKKW0t97a82799/6aPXzr+oU31F92Pk//AtBZy0l\nNfPnYByKuzOaJ4YH6cE66hJtdICyn4LaQ6XuU17VKHV9bMgSsx33bDeLBk92p+vmd6Y5t+gooqJs\nPVZMjfqnwjnK8GJ8MH69KXn98PzpJi3uyQDZnhyGwyYg2vdu9fb97kU+e/WTFzh7tD19lH/0D12k\nbxBnI/G4e/ihVC9k1LmkC9bDZIM28npM+50IxcPLLklnmOx0SSSQXhu1dwagKaCsT7rW71lKAlJU\nNYkEIeLu0boXRZvdqkh9fDJ0CwfflIXvjpHHusnLaw0PGVy0e5K/h28+++KL8fF4/KHVd4fP/4eL\nHLY4SES4AozaRfHZJbiUqAfttRFWJySrUEawXE4KUxerG2J1fOHHp71oQhPTU1v2plDCPZ+umgIx\n5dS8/ZSiEmoGJyN8Bh9ETSlSubi1LLv9+fXJdjrdZ98cfzXaXvq4TVksfIzD0dG61+XTr30c/uZ3\nr1y/+NlrUDXZbIFLULxhHR6gR+wPmtKiMzhIw+gREeVwRSMNIeUSeXUC7291lmx2eg0PmHAm9WG5\n7pUQWEr4zXRKLQsYagpV/f+Bffvb76y3o/oGNd0rUbYLXmK5q8uru/1xHBaM4d75P381fPGrAw6/\n+ycfvHdrj2cbTSo2hJrOHo9RAwIGPUBUZLN+KYcS3JeqrBFRX20aWSQSr2GykuiPFTbzdluomge1\nDFsPSgBq1iKaCMCkRSDmltKQ5q63+Z0vpHszSWdWXRf5UF+tvvVaN47dg8/f/nobw+Jq9/r55en2\nel+tph+s0x2r//bfj6YVkNT0142+BkSkmaSgne5f24p+uN5vDnuQgrh61ty5a6R6XXofkNYLYScN\n0BIA0czNg2S/zkJCkiFk5p4JUmFQFJKGxZB1HiJCoHJZh35ArYdFqstaNu8uf3nrclpNjzKH67Ph\noIeXx3d+5+h/mvpa33vQL7T+i3+0VbOg9E1RaYA3ckvjR6raYorzn/d30xKX1+s3ENLpz7cGlQyI\n1c3YxUKP73aiM7kPITPSBgAhfW+AiJqBaJNzCaRGDVLRvOqzNfE6RIRTFVOoVs0yPt9cHr//tN65\nYtrWu1x9ZfVbJ59Mb76Vf8IT+slH11ZXX/6Lq6Uy1DJLSrgRbczaFBWIpMU0Ln83yWbqyrQrJjUC\n21cqVcQEIrbZHdvCjm4ZZ0NymSuTYDOftkU2QjSl1soqgQBS89kMZBs6VaG30n9m3FM7r8/vyLOL\n8Q8vn613PXy8ePdwizF9lC+/6FbPnwxvHa1OD9+8+k75ey/ySX8xUhP2qtLot67aSGAigKE/LqU3\nTen+naNSQ+BRn10DKtZpEWDaaKTFcAwmnTtWSggJYQDB1ETT1rC3xg0gNIEu5hbo+xaUwbkAEkFE\n1JMn52ud/AfyU8vlQUb31Qf68GySi/fOvjnbvHp7fXL3g6uPcfbof3486CTaTdRmwE0EQkJaLwEK\nTEW6MfqTKu+vSqEHg9vnIEWtK6aLmC4ZZXkyQIw+j2njpv2OCOTOhDMzown2JRoAx8SAh2adB26q\nzYscEOHqdNGLL945/dl+Wby62Xb04w+/GV6V3X39+q3v59N7+c/3b//g43/Z51AmmhbLIqbSHuS2\nU4ZQgCGKXtaP4kJbWU/Gi41FSA/bq4jxkonHdyxEG1NZwHmc2TzeuiSY3bE4g3YKQaqOSRglhTY3\nVja5RUDCyfXxq3GF9M7hF5eIIldjn+3FEe5d3fEXY43hd99Op+ufnOP4zd+7TlTkLmotS8NNWCSa\n9yLgKlEFYnl17KOqIYLU/ddVCLFgQGuNi5L1+EjDGvMNpMzBt+X2lBRipjcSknnErQG4u6vedIez\npZnAg7649boU7d6SH10fLc1tGg/TxWHhR/l+/Wp7OPvesSzPNl/i9OTvP7WOXqHOMobUmL2+AqBE\nLR7uAbpo7vq+T0htqMvnF4TAmGaos4yVy0WbbszkcBHMO0MikJJAzf4D2N7Kx8RmIC4pamnsskZ2\nEYYi37s8LMbhtHtSNGtU6aZ6wIs75G+t/81m8dsL28vKf3SVv/PnH/drVxF0ubCETI1C2CAZJ4op\nvdHXzZKoqaqUChy+qaoh5p1rcajUaqZdmEHABksLtXk8s3oyFTFr00FSdPbhVx9LU0gzbgibpu2u\nhDysbzBBXu1f51SKLRe9yPXx7mPK/S+fy7d/+PsfPnq4+smz/O3Lfxfre3QwrA/WylJmuE+aeSHq\n6HVy91Dl9vn1EZ+d+3ioePGa2onQFOy6HL6bUs4zx6oV78obnhxUoAKzZvM+sxsEECZ3ACKOpi29\nGU5LUHi2+CqSjK/vWzL1CcPpYXEY/f2vfvJwOH5wdH9855bIX/7SH5393Z0gVTIliggKpmkglC6U\npISUMbIlEjGhet+v9jvji77ff1FMC0xS7ZPuRWUae1nkuU+/kbXNHQkYyUTNbr4HblZDSNpD6ox4\nikJtXp1ARr398Pn+1t3XY9aPLWk3bvPprubY5vfq7gm/+5FGXugvfiqr7/zrZx02V4Anlg5k0Vrd\ns1Y4jZKol19iseiFGMNpi8Wb55tjwbG9eAloEUnEpBy7YV+3x8OiaV0bGZNsIF1zEtOk1k5I21mB\nmcyUAioS1NQ16GX2BQvi6Nu7V/39YcfT/evlEZEVK3uVby0vjz76+KXfqbx1Olz8dNf/4OJHOdU4\ndFXFS29edECpCG9eOp4l5LTst6nLuXoRYfAoL/q03H9yEGOodONSwLq067h+e9kpVJrCQkmwuTE3\nNytLZtoOj9xEJ0EkKsI1Us4z56GtbwAW741f8Gx4degd2a/6nIY4Hq+WeZ3rp123ePdk92Z7+ovX\n/XdO/x9TR6AKLEmtvdMn8VqF1Ggrc+S4n6oHcqd9n21Y65vn2iE+f6qiAU1eM0WSinJvKTVjACoh\nGjMfsUGgmlJjKTa795ZeKJxrOgQQPgtHAajq29tf1PXtzWXkeOuzSyyGw2pcPX95di/vpue37t06\nNo4X+58u3n/nHz/rBMSYtOa22wxWuawz+qrqKj2jSRnT0dFSuSQ15fr6V0Uy3BM3uT/YQSSJuKW2\n+KSJqWltR1PTfFNTSvPpnxlYDRNN6gpITSIRc6lNQTqZnv1sv7yzfFwu/OFpAU6ls9tI692b/OAS\n7usFx9t3/ln61t1/9RfwrKkGc4lEJqmJU65FRKgQFjWjZlYX47Qt1zZBt6nD9MWb5vIJCxiT9aEJ\njm4eBc3Rkwr/zTogSUn1N/7jIo0lACR3mVdR1LnZEoj6fvvsoEdvnx+63ebs+pBt9WrQ+4frD785\nfsK3j/synW7LsOe33/3pn8Ng0kkiewigCRYs+148JQWTBMNETSQcvp2SFUffYXrxJAzmXSjVooZa\nNY2AEdqK2rY2gGhWWQyGmFlLdwoQDVUBJAldLSiZ1b3mG0bB4RCI2/Eqb67z0YvAQjbbB3fs5cP3\n0uUz/d7kb5bMA9578OwfbUUECGpFFmoVHbts02GriQRTwNvQm6bhtcDA/qSf+OaTHaoWSaiVKSWR\nEDSWHnlTfAhSOHR2b/XazNdaxmNE+HztU4VGuAwLGL3GPDmlmoT224oX0+nhSn3YYru7XU7Pv3U4\nSv22ny5WdViVk83/+MoYRokI6tRFiFidaJXTvk8UYTQnb0KF0sNDrevzVOPzcxfRJAYPYW5HIrEq\nGnX5ZqQpcz0viMYumWsoRrSmECJI2jamLBeRNIIMoQZUVBWHjJ2su6cXWtKoA2R7+kYWq9uvvn4v\nv3mgtrh//d9/LgiKhog6fEpkssMYXT1AudCYt8C5wgiR3lIKAu72+ovMMIFGW/chgQSQU3TVE2al\nEdEMk9lgstl1hpBGtJkXBhGp0Ra7fgZZm14/grRULxaL0U8X17Vqvx/73auL3fr6rB/3fnHrbXl1\n94B/9hdtLq9gEOoTM4Vwqxi1rhWCZO1qKixppEW2GCNo8eXVaVEVzVLU0RkUlphJsaDMFPmY46zf\nRCezxpNjtPq9DRXbdkqBqqiwrXyJmVgTlOm6urtADRYFw2ZzsS2Hxeoudvn06OLq61/9SwIekI4h\nSld6jQIjK8KuXl5uDuOhCINQs9Qtl11KkK5LSPvHJlSkzCgDJKuZOqYeRDSGIG/KEgUQwaCKDJ0C\nwYjm1tGgTNIToBUqMTXjMNxQigQU1HI17c7A4VCZvA9//tWDD892x7br0mq1++Zx75PBxaTtxMvO\nmhEmZCrI1+XYeyZLafY0yuYSIZnF7cVFF2YB2hQRpkmEnbpJlRvPTr9RrDdLTAQpi6xSI9p6OsaM\nuUZNIhYED11emNxoMmS2GLDz1xpdit5Hj8NhwX578QKyWL7envhyevGNjRCE0JVqArEKN0oYlKWz\nba1HzEZCqkjUcRBVCqLi8Ma1dBJgOsikmgJ0E9EkApWo/6HXAOntu7inwaRpWVsnHqCHT9evkkZh\nyLRbDVmgMwUYCrCG4KLKyb5KWiJCLuL4ztVifzng7N51uXvns6OzJ5SAimUqBBqqEtVUPQfZr6/L\nRXXf5lXfJU+gxCDwkFrrYRKKwABTq6bGRJNUTZTVa7EkUHgbMLdJOUF2q7YlSjhXhOHj5Ysvv36a\niigYPnrsVwttaDwYUb3c+t0X79U3q822c5sK9VBO7jzP593JxX5djtPddf/69YVBjLk7lC5LqFGi\nQkwSrfRLjIzrXR6Ww2KVB++kovepwHd1v1O3edcKyY5iojCL3FvU3db6pOnGyxpCDac7V0uFB4Qz\nsl92V09+9asXUyQVcbqzTnW9nknxSvdpeue/2P3q/c/K5uiQXTfOyu3th5vL8uad7vF7E2L/5sGH\n27+8ciW6PDKCqCpURiSpgjEfrbr+OOVFn43ZLKWsEil32C9U16sizUROiwRcYBIR1Xr1/fUUU6dm\nzCaaWwCnu+No0HBv3FnGeLj6+uNfvzzE0KWspFNjgvo8lSfIGu/80fm/un5QbbzVR180F1Vc9bde\np1e/Wp++9esvjhevfv7Dv3Hnf32ZTB3ui7YqMEB6BbWWfbdYn6z63jSbw0QTEJOL6fEw6Adf/+rL\nc1di9JpEGsVVUtSOvj9oZTFRMbXUKYgQiaAdd/B6M+icthePf/TxNWw4sZQNRPU60mo4eePG/OCD\nL/+3bf/kspbUbXPtzqrL9vTq+J0f7/fv3tlv9clwq/z07Q/+9v96gawWKSfRCCMgiOSguuqwXHRS\nlSoCE2/C4NQn2LB++w8e/+Uv34zwICR1YFW6ptrBS0pWWBuYlVJSFUGQtEHpzeOfXqfD+Wdf7JCX\nR4tId8oUqkVin7eHRZAMjdA7Jz/+3638zmcjxyKTVj2ZLuMLeevZybfvnN4++tHm5PF9Tnevdt3f\n+l9GZVSGmrB2uSTQs9IEYTnN5DxRiQgpoIhZUpCy/O77T37608dFUAxGhIdE2CA1loZSo7I6IMim\nMJHw2jXyhwgB3++e/fKnr3Q9rJYS6aPN5fXuoN1U7OL0aDamGYby2S989cHwGFpeHfW+B09K7H58\nZ/Gk3D86/uLVka93tvzg/U9/8bs/+HcleYWpa8ATvGnkqKVQ1SOBLgmhMSdfVWmWZug+eO8P//2f\nPT0w12lyAbyW0oVbgvSHqF6kumPU5FDVMom7xkwyicPX/+aTbV6mYRB06ez4+PXmcsp7j3FsdTyt\n370+19MPPvifL0E977/9q30eT7f9i5d/9Qf3v3r2Rj+5lQOD3x6vr7c/+vDheUu1rVBL7ppcu1I4\neRQkl1kwAhfQNCVTE6ckAvruW7/9T/780hFkoMIFyTmkKtpFiuRjserTFExLmc5jF6SoBLxsn/3F\nT3mSTXqDReoj58u0Z0weQWGIMjZjyOLu2fVjEUHSHnVaLPT4Rffii+O3N6uLfXpbduOD6+6be9+8\nfPd3/ndMHiKmEsEuRjrNNIxA81eXCDORQJhYSiYRCJUwReTvPnj/T1+U66OoQsA5sFoWSnJl8lRr\nnaao3q+kxAiBiFFievXNv/8YZ2tQcwoghebcdZejX9cm/gaVdJdv68uvJmGN9XtHHwy37n9d33j/\nrfqzd9+6+NymF28d6TLixZMPXvz8j995QslJTUAlUg0k8dVBISEKaNA0KQCfd6yCDKGLiND19L++\n+/95UkevAolJFo6sRpq0PUSWLLvUxQDR9VJnw9vdr//NJ7i/pFuftIyWSp8X2exymw6uKmYENOBn\n3/t3z64WexHdfL4+/r5+vh3L0QdHf2W3v3wRclIvbq+nbLf+Sfcn//Txd5+WmHlTpARAMx2GMQhB\nsvCu/433nWaDFxMVhM3aye4/Xvzdz3fTpDlYkrW1yAJRVoWoaHIbs0GHh8dJFKRPb379Ge7dYmE3\nmENTCqcNAm5ym2/pDBDePd7i0rIyFvw0vb07uB6tp1+8+e7xX3A9be/tqhoW9+998kd/+Ozts8eN\nZQgTKEK097zA5TiJweIwZGtWAapmIgArVGEMihKk/R7+u714LCDshhDO3pGwgAlQs2om8sn9ZdvE\nUK5fPJ1unwhMFwMrUq/bbanarW+f9rNeWkSEUHkzfbS58tQPS13afuMre/OrX4394eJkxOD79eHS\nePX6d+988x19+iE0Zc1ZQYVlE9M0rBdRLNs49UsFTDVEZl1jePVa287dAEn93f+jbnaTjxJdHhtY\nFWwwm5qYWKfQ/uR2rwAQ4/mzzfHtnikNq4UKTfWz17tJF6frtc1fQlRVRa4fP//pKDiMB25TfXmd\nH2at9XT9zWcPji6rXGA5Idxu333y2btPP0zl4MGYSnHCx8OBtlj27s6prhZtw703W/dg1CBqePWY\niRZk/pv/uUy1HPaHwXxWrmHeJKoqYrnrLJ2cJoUIY//yGW6tiEjL5DWsz/rVi+tJuy6vVvDmKt2e\nz475GUDn0O8ncx4P5aSrfhfXm1sLZLxQO7+I9a13tn/18OTWyeTFG37DKhKMoesWiLLDuv9rw2U0\n0k5UJ0Ghx40ocvFf/q7GdPBYwxGOZvgjQm1uaGJqeTUkFTDq9smbYS3o8mLQw8QuQa9evp5MU+4X\nKNTG0gKBLYd7DghOxlI12SIfSopX/dlhvL6fjk8Or3A+1V13tvv63551pxEeIhER2dDpFINojsm1\nN7ZFXqLajOGaypwRcwtIMkjc/S9PiniJZY0bG9L23DXAncF8fJabM1A9f1oWVizlHrudp2Si9fzZ\nSNOuT7GvnJV8qnb+5vYVEmzwS3J0tUQudfPl3ePY7Y/99vu43PZd2r/0/vm9o7MkCssQk6BXRp9L\nNEcfcao4G0qu0mhu4cRcwM/s5OBv/cfY1j0Xsx8A5qawwQzuXN5992FHSITvXmz7npLzoPtdRTIP\nRT1UiqTs1WeksV29N1NRpdomlBGyxoEJaXt+tkqXe9JW4353ela3T07ldBg9kncZgPhU0clgdd91\nBp2LvGYGJdpoem3Ticus9RABydWfnExl0mXD0We/dDaCkYee3b17a50VAKfXL6TP0plpKa5JI0Sh\nZVdgyjHa9vbZwjQPg/bUzl0oWYd7Xq+7tMi7x18ftOg4FgS9lLvr/Y77V0h9ajsJlV7ZrXVMvSoj\nqSjUQOfs9NHISq3OnlkAbajz7R+WWnNHAUwhzaYX7XvL6cN7x0NOIkLfPL009dSlJKUgZaltf3mp\nhVGN3kR9ApjQ3rq60Nqn0d1lqmernUyHtMji59OI0RcDdxpvUL8jxT8pGZ34QUBxS7QkvjxKsASl\nWCMHxbyVua2D9gg45WZPPcO5+uMVo+9KxrzXPQgPegTD7j08G+Z5yPT65SRksmwxhSQjDBplN3nA\nsgrrjewtCExPJvTdodTQsu++7dX82gxTzdiNu82Ay11Ji/z40R/8YPHq79wl3Ktk4zT2XVqtj1ei\n0qlAk7BtSuZMw229dNtN0SAbJcCQ33o4+tLcbu5ZAAa6QzjcO150SQFh2by4VDTH7OJhWaAcVeiH\nogLtrC3QbSpXie15dIv9BIhMy9/S5288yhCl7mTwsn15OIqjoa665UX6rsjofd2Papos3HN/dtIJ\nqvZiEW1bb4M8m4+J3CxxFrZE4u1WnX1PuGyfoRmpz8p6r1yuOxNLAsZ0/nqytg99PKDt7oopURGE\ninRp1ivPNco2r9OmiILsPrS/+iD6yZZjVSI7re4W38VQ8fHR2fLBX/7j3ck6XJ0eFoFhue4k4hCJ\nCjBsFqeAwhCEtk0mGS4JEmJsBLP8w/9pXDb6TKNVt8kUCV1mhIcKom7PD1lgmpLXiJQkvJZJhU2M\nLX3fWhb3aSolFCcYiwhUj966/FmNISmkGe6prXcffzq8tSjl8qvX/vrRH//gYQcQq0QNWu6TQHQS\nS2rKtgcA7pxjUAM/w73NpWYeKQMf3mMnAm1pBiCkba7rekNtk/vp/JxJRNubRBHh7p4ITg6oZaH7\njFSKMNaDbqEheerHN4qtqHYXltkJ8kng9Xjn0Vj86JufPTzulh/s39uMU9kewomcFx1g7rnLOp8N\nkQiBwhhJ2JaCOhJdCMYsjIpbHz0eANJqzI5oTRSq1mtbhUnfvt6gmX95GV1U3VG1S0KOI6G5GaOQ\nqhCadodLFxOI81IWZZp2zHZYplJjwNEzLFK92p6dL+98/fVbn50Wfbe+J5tJDdEN/SJB1afOrBMR\nEZc5WYBAKD2RSiCqEDGbLRLRf/fPlgK6CjA7lwEiBrFAaw2n68vJKeKCcig+JHgFzJIC+6mEiKaG\nNrcJIxhl1LSnEbJwz7sRskXSbawOUrO/43qdNhHv1b8aunp99HJzUdZ3d3uPbug7FZEyLvKgaPNZ\nY20yIZFmyqBgaBUilNFWTSHShycqYm1J7ewqHiJKORxyhApj2uwYbaYz7aTvEaGhSKoaXmoErUu8\n+QcSebHI8HDhcLsv9Cl6QsYOlv1kmx5e3+YrpO2XFx++F8MuD4dwvv5kpGvuegPFr6c+J0gSoc6L\nQ9Ecu9re6TlHNsVtMADGWx/InCWJZsYP0CzJ+cuLTRWhl+1Uw2uIlMtdWpqPE5EUKUWMh4BIWhxu\nCIQIRyzT8fUukI6OdxeT9IxRlj6ulu6rvJVHr57/DYSky1/88PtP16++DEcwJnf0C+uM8HqNIZkk\nBWb1UNs2K5yNt0WEVc1buKciJI5/51IP3s8cjIA4xFRMDt9shtOVREyHUr2MkTnW1KcShWHUgwpQ\n9+6BbpUwTxwhgppslSnL28OrlwVmCvT9YLuljffizWZ69PxHn1zQd+cv1z/8j1TLpHUkR0hKyz4r\nYn/R9YvUXKGS3pwRKFtyj1nkHD4zXkEB0ndWvHr95mK7n0hVFdOUUt9l3T55TVP4fh+opYqMeyxz\n1CkoEN+mtqi6MqCNeN5U4zaltqf53COMnSvyYhNJD7tOd2U40u2LZ/Kdjz4rL+x3v/o6Rx0YqMza\nW28U8HJzpx8UwpBQJaTtvpvnzE02FBqh8/ADFAl79PXli2nQoRu6bNbMiXNWp1C6JDFdbd1rQfKd\nDz0LQ1Kq3G0TA2CVjjdsyNn2lxeiiWAAZO5ZptVwvdNcorvYH6ve3Y37688Hs4e7X3xdh4JJokbq\num7RqQDlNftVL82suy3fFhHXmLnspAroUAuR0Jk1juPVz157J/3QdV2vnahaTpnT5Hm1ENb91Tht\ntlhgt+8WXXFGGiLK9pCC4lcYModFlPAm6xFVu4L1xYZaRNLq0FX6IZmPan45HN+u8uFzvzv+ejqW\nqb7SOqYAScurxbpTBl4/O14tGiNEBN5AhkZADGqISbSFZsGY4ReKSrc8r9jHYS8Ly9Z1lpdD7kqM\nMpwO6uP51X6zHRc27XQ11Apa0orD3hPBMgpF8xpos3cKQJ2ymFiMoPYx9Q4N9GOxheP4yS1e3/3j\nH10dHcuw/eLVIZdIo1JSTv1ykRA+fTmdHHcQMISiodIYn22teLtabMOatnPsxlzxTo6uutcYhZZS\nzsenEC6WJQ9Zp6uXb84vp5zKgas1C+EpO30qnpK7kyHaLdNv3F1UdP9msMq0R8hRvmYiGaYoghxP\nx08edZuz/+jnQ7/47EnZDbnk6gFR6/tFIsy/en56sjZtHnCMtjdbCRChICXQ8uR8SsJIMIy3z57m\n3muUOhYGwo5v31k/vOtv1kvxw+XrV2+urS8TlkdpU0Uh4l4mSuppqiFqfbZme08IVPcVm12CmyzS\nTlCqdSMCK59SX1e7x4sjwXevX13ferPvOEoqQqr0w6JnaLz6Uu+dDNL43m3rA9TbwoGgzrwF+Iyu\nWCM9AMTq/efSMTxV4TiWgovz7W89PN3sFh3q/vz5600stNTVqg+yWjKg7kb3tF5JDJOo9tY1nV8b\nDZd+cdhrpMHGi6QxduJwAdyLLcpq96vxdLUnbuX1m1SKFpJi61U/ZFG9/nT/8N5Rs02djXCl0dhB\nIbypgcGIZnzAmElMhH7wl1BlqIkmOopPp+/eSRe+SJguv35yEUPPUfPA/RRE18lUisDS21pl5RWK\nYdnQJIEIPHp0k2g/7DyM5qkfO6jvoWm3XOZ6+Hz48FY3HYa7z8KVDs/ojvJiSLDrX14cPThLKqF6\no8mmQBzaRk+ItuaYpDdvZLpQQgl9eHzZmRpm82qyyjLhkBaYLj/72YvapxqyXtl2V0Wk64tHFy7J\nVLtVYkhaLEqDZ9vF0jELohRH4xrDl3RLBbmflkyL/un04NHZy8vn5oToBLVlzkMWbH7++Pa7DzLB\ntuOriRuphAUaxa8RMxRkVbXwG9IiKcf3Xg4QWGjyoclZdlM6Wtq4/eWfPSld1smW67S/LkS/6KQ4\nUkhWR79YdUpoymh+RyJU3WvZw6I6BJIh3FuUw+rMpJweX2vCbb16+bOrt169AMxIJFus0jpbvP7R\nk9O37g2YFUUN69Hmed6gDROgaa1AJ7y432jQg937OhUKTC13eXF8fKJbt7P17slP//zrybIeZHWU\ny6E4LA95Koypdiltbi16C7JWJLTeU4UBSOzDjJB8EAYxnOpBFw7xfOx1XcrS39l/9qTvJgoiJKV1\nfzT4m2+e1kePThY3jENCZwY7oNH2cnjjlxOBUJQQVwlv26cgby+mKokCEwvrBEdJe79+/MXnT6e8\n6EIWiwWnXYV0q4HFEaOlKW2PFbt+qobQFJz5taKyQ3HJpm7CvLgyTUcbpuHoXAu4v1c3Q3/+4Ysv\n7+StWgmNGBbdtLm4qCf3T1d9206GEFqTC8Nws/SAbQENIsxVAYlwCb1ZPHN69k2TW1Mlm6lnYWyf\n/OSrqzENHbE+XeKw39c0dJ1UyLSrnSJtNl225UTTZEAEm++rDs980lY80iWbadEIWz04lOtH2zy9\n+6vdUvWjl+dCCemA7Of1ADs9O1p2Q2pU3Vn2CWuMHQ2GosIag0jAkKAK4gbpCpCLd74qShMjojPT\nItevt8//6ivvkBS+OF7rdjOypmGRAfh+yphKqle26kuh0xbbG2qaii4u925RTQxJ8iQSpUxMx/3V\ny2r3nr04vfdqWpyffvhxVQmD41B0OFktV4tFTqqCUASaGZQyoAiFUECN5hEbymZeDwEiXESpoKS3\nrEgnTAh2OUni60/8ybPK1MEnWw7cX2/rpOv1kFBiLFKkjCmN14n7qSai68QbpQBEniZBL2NKkkWW\nF4GFF5bDYrk/zu893b+8s5sm/enJwy9SjuJIq9WwWHTdIueGwwQDRog32HA+FkqR2gA7BSNEEFVB\nMEJmdty9o3NzCVOFdkbF/mvfyEJSqdX6wQ67Ec7hqFMtrIVOuKWU6s4PkYa87G2ekAikXlRbHFwg\nDMXudh6nflfgG33w+fX06K3P3pydfP36rZPXBUXdswyrk8ViocmS3tDVRWEgRKWpCWdqqLGRiyWa\n/bNFYaNYqRBUntx50bdE0JtqFbCEdF2KKDL0tj3sqclWQ5ejTLvdGKxBDdbdZp+OT84enlqjBkNU\nXnyxLyEWqqieajmOOE5Qsef33kuX063V+eXtxZdP1Z+KuvUpLU+O1qu+0+ZVLZy9j4Mq4XGz7EAE\nAtVmQiZNDdm4EhFtzzI9+gf0EAElZ6UI1CQtVoOqdENfttdjdVsdL3ot034/VQaR1H2qhYtVzqu7\nA9oIKZzd/dXoPSnJYWoXK8lnUOvyLj5Iz55gbZev3/XXVycPjmMqko9P14tOYQ2lDKeIiYdTGpz8\n10cdImrajAE4qyc4Q4Ntz9vbfY0oXkWlbZOWPBwNKtKtOhZHUulOjgaZpnG3m1wgwy1FSBQsJMDV\n0DiQJGlHOaIgdQbr+mWtx0MUy/3kX5/eLtcn+bTuL9bjXu9/e1VdFsd9VghEdV4MJkJl02mhCRsF\nArowtO0HgaoKVFijQcMes7/7vZMaLjUQwZRM0PWrRWaNZGU75UW2/uhklTgddpv9oda8vH2cGMI4\nOANztmq3//L5Pu/KMoQmOSNfrs+mXe5Yp8uX37rQ49WbgZNvRco7f/jjw+kqq2lKkDbVFZ2VhxBh\nqDc9jMisKVTOBlkGCQXobYUFyRBQjh9dwkcdwtlnp1jKuY61ooxjZBPI8njZRYzXl1eFKun4xDQ0\nvO4PZEQpCnFRgMpDiLMLj958752cvLV39cCufr78oMo+DhcY9KrsPh3fPz7qskLoKrPejREiomxa\nYffmp0ifrXh1tqk2BajSrCeBZrlP5kf3Ogl6LbnLybqUF1l8mup+M0qXTHPKiaz7y51Pjv5o3ff/\nX6lU7Q3We5CWAAAAAElFTkSuQmCC\n", "text/plain": [ "" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from PIL import Image\n", "from torchvision.transforms import ToTensor, ToPILImage\n", "to_tensor = ToTensor() # img -> tensor\n", "to_pil = ToPILImage()\n", "lena = Image.open('imgs/lena.png')\n", "lena" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAMYAAADGCAAAAACs8KCBAABTfElEQVR4nFX9Z7MlSZIliJ2jak7u\nvY8GS06Kdvc0memZHqwIRgRY4AsEfxkCrEAgK72zg9md5tXVRbOqMjMy6GOXuLupHnwwfzmNlJSq\nlKyoF+HuZqaqhxmfyMPSghBByAUL5hToIFgSoBg0CJalOiRHUBRKJQEAkCkNAJMMGQWKIgCll1kO\nJInwNImAwOTsxWUyaDEDACZApCc8aFG785qkLd9h60xXElY9YYIpYQxDIPPjsyIkKlMUWM0MoBBB\nIEgPCfIkkEaQgTQwQRjEJJkyAQoQAgTKJIhMAUTSVC0TJCSSAijJBFkiTRDhUJpgTIoUlICIuZoA\nG7b3o5FBrO+aUJASQDFH378xSARgApIkBEEwgwQEQXMRICSKBNRepwQBEg0CaQYIgKpEPX6MTArr\nXxKkEGEpWAIwLZlCggTMjJAbRMEBSCX3IjLsql/WFxEwgRIJgqTkQ5/ffmsQZQYSaF8bJBIhkFIS\nSEGggZJkJCnRAEIIIkWmJEAQYQaRkECABiBTdAKkGTlTNEIEUDIgQWj/b0N7pyYJgOjzbEBquMxZ\nEkhTIlNMIIlAWL+d3sWZwdpbFGBuJKCgyWmU04wkDJKAJCCE1NYNQFBJg+Cg0o0gUiDa4oGRgAFK\nE9t3DAqUChRORhohmpEAzYG2KgCBLDyKRNXZuAhQSjCQCLZFChvK/sbGYsgkEkLSgES2X1+VmUyk\nEhQgQAYSZuvKI5gspIlmKVh7RTQaEO3VtgXLtm8gSNyKSGeKbjBEEgo5BWYKMBOQBiUEr5MRGeUJ\nJ8EAUjDA2sawsin3r+c0Ga29MBoFERQA64qBnky0I4QwkilkEiTXF61IpdRWEASaFADMQaKteZCg\nJ0R3Sgmm2kKvopQJku30ISjRSYJGk7kd0oiIcTeFku0LA+1b03u/e89d57Bse5CpBJSCRGrd1eyR\nQLbtKkGwjLaWle0DE4DEVMKkoIMJUSGsu1dCCmDWBJCQGVIoIOkIWFsB7bQ0hSSkRCHZ1RMghV15\nBYBcoIRUioEc/P17GwqDJj6uRHJ9p4Ap1U5eIelIQ0Akgu6PR0tm++3NxCRIJamKVIZAEKAiIYFJ\ntpUgy0zAmIEQYMxs50VbKKCSAEQlAJZDJZW1fzItSToFQBkR8L7cTH1vNWXGtmQfT861OEWk6rrW\nFRAokqK19ZNCytwIKSPb2jFnO4ZpJiUgRbZ9zvb12P5yKWXFgBRNCBJqGzAFUrYepwAKjmZU6qKf\npLRibY0m2HW3b1EiU7BiUlvGRoHZikDQDGinC9A2NZAEAYlEO45FyludSDBJrTWwBCC5DNGqDlvZ\nWmsOKUSatX9GezdaS78JrchBkKycth2U0T2Zpw0qAQlpYF/e3BlVYEC1tjtbFWA7RQE6oCIF2xsn\nZCAMmbkeO/lY10jBTJQCMJFAVrXjK2nSulEECXStlRQUUpnGVPtRa9+TePw0JA2ed0tERl4+i2i9\nD41i3+/3ZVskUvFYErA+TPsvZbbjzRQpMAURyCRh64lnkMwA0bKVSfOUwLZ1XIlW601oB0/bd9ae\nwxwKwkrm+jqsHT8CuX43iiD7WvtiqLrAKaC28G3sbt+oSytOydxAN7L9JhKSkgSjIlq3IbYSDpFK\nW2suJCCjImsYzajMbMuCBjNJqfZ6CVjbr6ICJiWRkndEVrXmSJlQO1j4/fmNhFjsIQhlHa6nCQRg\nsJHv31tPi1bvZFQqtDYeas0RLdX6OxBImFLWKnPKBEmSSHohrT04jAkCTEk1a3up2R43gVxXYKq1\nlVICaE0C1VYQ2f59itZ+ESD4cjIiU1fbWQIirfO7WxuK2Jl5ClFgSbNWPpLMBKWULChv70UCEVBJ\n0IRWsEmIKUPrY9cdYwhaakkCDqMbPGEJU5qFPKn1/UNpJLC27khC7W9Q2bY4AdHKYTQhWK419Yj0\nwe/vrBCOCJkAlDQIrhSRMlq2mgDBkCIgowQTLdeDF49dMYSAt18FFUhVSASHklGzJrqubWesZ5HU\nXgTXpZXW1oDUzrLH/ZkuJJUEwf50OKMyeDaezmk+locbdjSQJsAXlOIJIkNrn9RWs6z9NiQstK7w\nMAKplGFdPeTaqGT7XlWGMbPvh7QsrjjcH6N0nYFJsL2J9iraNNJqhCllokix1T5jApS1DpN+HJ1U\n+nNGx667vUXnwdafZlpmEbTuYMjaTxdFrcMHsk1aYq6/BG0oaidnGha6YJnVyw7OOQdfDkU5D+m7\na33Qn6YymoUTeBwyubYErYNLU3v/wtpfAaDAaMc1S91fMpGxuTicjf3DrQYkYOlJVpmxBNeKAqxj\nZyuCUBs9BKYZIFvPXG//C0AZSsqUSCZHM9apROlChCOPi3XHsr2c9ze3Y28S20NAYGX7Pon0Vmv4\nr2pgW4Va/8NEn0+jlOlP+tIf3qN3QVBWEShUFtKVopRcBzjAAkwjKLbhNUF6hWBgwpLW9r2ktBQd\nGPxYLIMWc8gjXShgxFy6y+vny7J1EART1qavxQnCaOsrJNrEtG4iCrRsr1W+3I+WmTE8u3+4U+nW\nVg6o3icAA+AAUsnCtnHbiZ0htHFA1ZkZyHX7JRC1bexlnlB2Ow/3Wiyx6aAlM5eKpDOMyNN+uv5y\nvNuD5kyVJKGsFJDWJj3icaG39uex6eY6GrJb7osxaz7vX0axSBoyzc0ckYFCt0gqKryN+8G2ZGBd\ntO4NTLYZGJTkbRQms8o/uN8t81yBI7GoR9AYPckpe0YxhqWm0+6zF0vZWJraZmtVThKU3lohMmWU\nKYxqD0YyWx/k3WHrgHD17fGMtAomTVXOYLfIiBSRtfrjIUIj4IZcakiJJJVCKxGEPdZdVbvAQXe3\nJ5SiYgzDMVjSvaass+g8a02ldDo++dFwd0oa2kHeajXFx4kFDSEQzdpoCVqb9QwEet3D6Nv69VhM\n6SYzRNJTsagzN4CMJawVAgkQpUjC1gMLrdkgqWyzOwlEfaGHw2mWRBRKLCrMKptngBFdgVBMKebp\nXh8/nx4mFfyrURHIXH8cFVqnnXWgVDtpGrrC/rTv++34Tze7lNGZ5KK18yw0KiVEom0ZM0HrHx4G\nkbkOC1AbWA0NN0B2m1e04qV41FwQrOkJsKorhU7N6bAACVL749Mv+7tTDaH9+ZAg6PAVd2gfJWnf\nD84QW8uYZCl3cXb2219d94ak0hmCs1ZYX6aihOV8RIcAyYABSE8BlihAtupHh1eQWtpCEKz8qu/I\nAAQvTAO7JbW4maVZJsKrSM+kSNbD9ofXD5uxDYZoi15A4rG3lQUg2gps/fdflwBHef+LX52ftbHI\nENGDkscwHB+KRKGSJcsKyIFo9bBNTVRSMkBJtoeTTLlc6DB4hjxLCVaUKHUpNghLYSTAXuiRgrfj\nJg/9Z0/fbPutS0DCLCGVNvcDzIbQOPPxOZIrokUkz+s/fbPbQTSGARVlyQT6zfywLwI0zQ75ik+i\nHdpqCERamreTRGkSvb05wj/4bWFNdCLDonaSMKJSVsKQ7gmWbENGmsApdP7JT/X2dDlAWHt5yi1F\nC1ki2qhMwcg20PKxOy3b29fjSGWhUTXVBURZf9zfXxURmryblCYQLqS5kgSRxRchLQnIzCAoKALp\nuXk3DZUoVjPd5t5CosKlUgH0sKQtJhJZqFrBLX15sfvU768KFPQ0tb3tARkZ7U3BYkVV2lZnGMy2\n9d2yIWCWkmQdkJA63h83HxgUS6v4DSBocC3SJFgukGFtQhEQ3cwJSJ3fFwBmGV4sCjKkmhYxBEqR\nAGvP7N7V+bSMF2cdlv27V/bF3T4kQ0JJIteOVOa5LBWSWTsXk/THwXDMb+eNieYUyTTPBaF+E/P4\nUVdMOTdkWCuOBUmJhMhEqSlLUqAsqRVGcZZ5MVqBwq16Mmkg08BYIFqiLJ4sYTwdudkUzg3mnW6f\nffhuHpkWRtbS5mjJAC9LhnfIRkikt9ENEIbu7b5HNK5CKStRIfV+uF+eni8FMUXvbXxd63eC8NKm\ng6DorZKAucLMRhEnGqhEQTqYhazWTmKaeXgSyeJ5qNxtLGaIgi3Guf5Rf3PdCY+HOc3ysQ88+Vkr\nJGrTI60SYDfuX7MIdDAtaimqYF/NbvX8fJksp8WNCtJgBBI0yBuWhrZCIaC0KbedjyqcjDJGlqgl\na7S+e6nJkt7GoJlJHG+ns+fnmKfaaluX9XSrP+V9tN7OCJAhKgV4Yc4BGK1hx6aEIBv0KnrSOss0\niYMtAafjxMudlqXEYkUJT4hBtp0hAVkAjxV3bquNoFHJdJ5kRKZ7wGotJkylqzRRNC7whQVWXufZ\nBWMOloTLQumWOp7/4J9PmwZPpLfGWTBKnS9L31o2iAmSJtPgLw/Pe6/OhV5TRQuKuFB1c57QXJbs\nSoornNxmSeba/ckyV3AkaIKEoKXbnOYUjAEoyBLZu8mSKLPClMWNt4ftc801OSKV1dI6JoHYf/Ls\n7dOeslzRQYFEGqSlnS70hDwhSVk29+/6eQ5mkpnpQ40OoU4adqg2DWU20wptCAa4IJq8wgC5QNBj\nhdxYUyI86ITCDAike1aKWbMnPJZBDmLYv7enu5gRHTOVlhwqaiCtLt0Pru7GnQPpWAEjAKmgWAcx\n8f2xSdiQb0vncmcSNayrk0mdJh/7kjFjMLRjk44QIWQKmUihai2jihVvsTR6EbsqL5EkFN73lkYh\nC7xmRPTsENzcvNx8vFmm6mPWOgeHIedlmum9o96V//AX9WZaEesVnjDSgLLso002XPvUvnv30Jus\no6RI722RG6HKjS+I3NLM25Sdj42nNWx9hWxaH0KtuA+kjL5Gg8EbOZPhHhWWVDGrWCJk/u7+kw/n\nw+yeAkrp6ml267re2lh3e//03+0eAiu3hASdjWrNpbYmrNVz+nB6bU5vtYTwMqtLYZm7MwtFjqbi\nRFqCgK1VrtGTViETkm0GJyiDETlvuDdJngKkhDpf2DuSlmlwuNeH9I/6E2mzTG5V0XWzSgisNNSu\n3pxe/Oh4e2GwpLVVValGHlazFVCSAYN9l9siGKP2UTtbqhkjFr8cl8zoSsgyQThIQ0AKKUljK3pt\nHFcmiWQmhDp/XE+EohFqmcaoBjIW0YtB0vFBHzxbJgjhMJuXKJ3NBQsyQSeNpdTXT768mVfumSDS\n4EbRsSwhPRa+rn+33zgEi9IpQMzwRAZ2G8WylCKhGKU0SGtVUzuoLNUhRctsGCQbSWRzv93TJUsZ\ng2YsM4lSZzYaJjIOZ2fDAalgibKIPZCw7PajyaSUFifi/pPL044S7JEEzsYzY3HXHEp27uP8mi6S\niUygt2ApuUx5cYFa61gYaUaRxQAFIYrWeCekoGiwlxnh5sqk1enFciwQAwmWzhVh9G46uUOOpPaH\np+d5GrhM1uc898NYowoedUhyijB3Eqnj+OcPR7QWTSuk6iZ40f7d62n74tPuWLB8m70T5kgYDFnd\nMuZTf27TVIceAlWYRIj5r7YGADoNLNaEE6JbtIKy2Mcv1SuNWbQAbeyv8kFpRdXm+83VpkZOKhZ3\n6HqL8GptulZIdINqmqLef/TsNdJlCwDIEop0kPnG/uhPbFqe4Hz55buNWxirrKu0SBiWxNkzX5Za\nSivWlsiANwCU6yTZumOkFlGRFRlzNprzcOVvoSTQSwaFDF5U+lqtzPs6HS4/ipNjOtiIo29LXcQo\nzKoulzDbeM2mNWBM95+cZpBiI9YgJ4wxvSv/w0/rN3+4CR4udw/GlBFFU4LGUlRnXg91Tut9pf2b\n/ANYKa+1mCMrZVppY5BORAJRP76LTqRngMWNTMhKvVdfluAhXmyXIerkY72dL8ZY6G2E95gq3Tib\np8k8gcjT091NtG46gXbeKB5eX/2PH/7hd3F53hPzD1mpYpFWsB4pUm42WgL9OsLDJBoNjYhD035Y\nG5qyjXpubLiem5ZyfhgKAIQByUVuFtPxOGy4EMfN08E47Q99l7OVWumCWERMi409kphNCyKyCnGy\nPz4cSVqDj2Am1dubH/+n4Wcvzy48l7rMTy8PUi7spiWLR1Kx5PYS8xLFsiZIyoxI4v8PwlXDahla\nsdV2gAEs83k5HUNAlE7Jit4156RLn2ocbs9eKHAMp064HE6zOxgWpczV+y5nWIUthJzdUEzLwycv\nbhRwrFqWqNOb03/4y/3f33y0jbmq1hOeTCAJmTlmmhkDF/2yZCmtpYfMHis1c6UjyYZWi84kxPZt\nZJ5QvTwFk4D1p5PVMiprarycH5xTvbxQqYfFt6gd51oKVWqiHPfcDVqEUxRZ8c6dWWuiPsw/PO0b\nukJKyvtX43/64e/+wT/zpQKxZI3dAmXje2saDcuy22ZNFihTgqgCwFLWVTI9mzynISiyx4YNMKZE\nTNP5zGI7eT5oY44IJYvN2vG+Xl0cuqXCu0MZcFqGYZkMPZdj2SjkSxbUztOGeUl1BKG8O//RcTxv\nkDrj8Pr+x39x8cvX1y/qIi2u3gBUuXGZ5WQRY5780o6hjtkkQq30k/9aHJMrqUy0Qcoob3IIEvJt\nHpeSyoXnHUDU7Gm0Dd6dnnfzeH+0i7wvQ05drzCXYzn2G83BLAiWhUO8uTcv1gaw5fDJD+PUZAF6\neFX+z395/0/vXlzFokynm6ilGBM0R5W7L8e8GuY53LliWZYqK+u2EqcS5IxVXyYoV6UJLUmr43ia\nyKVkVwIzhmNv2WGO/iGebeJgpv6Ou3roitmeAwyH3CqMC0DCqzv2cVVCSEcm4xRnH303WFLTcXr2\nw4uvHuy5T62SiSB5eK9MOitpgJbjxUdZ01ySGhVjWoHyVqolwFpH4AAQMEcqBCgg5VS6fTUH3OtS\nMaJ3dqwqD/hwN1VN3fCGO2EglqlHlLKP8wGL+p5MBbs+X9ozVlBO0IGc7s+GCsR0uvyzH9z85ra/\n4JxJwWSUDDe9uSKkCCnnpbvkEiyeAolGtRQ+ao8IrIC31ZWvcQVoSoBN1zWf17uuXzrTycIHHYvb\ncuoxlV3MeRrO3+sqD6WLyCjDOB9yU5aln5foNQ+F5vvjVZ9qkzBAX1zzcHbT1UP38fnxVbWLLhcJ\naSYzgd183FGNUYMY6ZfXUQPIdsQ2mLk0lVKKlLmtbFOrRiXhkSaXrRqR+Xr2zHCor51yLkMcubV9\nuVxqxOA3OAfgjMSYyuVw1R+FanU4udcyHYt2PmWaw1lN5hacT2O9659e6N0J/ZZzaOVYlTIMD7Uf\nAMua1ild/RNNFY9aF2MmqUJYNpRTMokJp9VGByhoahCMJHLGxZFFHul0RhbmVAbecVfh+83wrpzl\nMupuYEkN+8prTAURcPUxatlvBuC+6y1KJQQPWPScdufzR3YzW1e86QLU0IYCavOaBYQx4QFkbvsa\nWRq8nBDTAJR0NpkiG/1tSksGIc20pDHVOg5o6TcPBDMLlxzMofQx7/BkoeqY78tm4dkBfZS+7E+1\n39aoXmBVBbuYcc4w3yWg5MRVrbmwHs697rPYkEtbI02f4RYq42tXiO0sdUW/Y6T5o0ig6SisIAE0\n2KusLAPY9F6rkGRFdyws5FOX7HrOxqSWzk2neCo75nncDf1SdFQdTVPguHHU7OvU0bJ2x+gsGEtP\nX9wWYOk807BMw7Ys7/riveei9RyW+QEOsuBVOJQRMJZl6nZlqVoldKsYQIQB/v152+QxJqjJQZva\nhZYULYj0jDqHVboInaoHHvI6stpO77c7eFn2Gnx0eztdb/OOXQwlrLjmLM6IwhCLqnmY5Iz70+7S\n3x/GoXRY5nQ2TQP7aarpwPbufpdB2xSjYT51I5ZwW0m2hhoQbo7gI3vZZFur5sVoQNJKos2yQBYT\ntmbFFjE59qW+t2sxcojXm+4EX+axRN3nYbPVwrLAo/jxWIjCeU9yWIVN3BQurHv/6Pxw328Gt6wJ\nJxroWeIY0cdiZ19hAJFTheYFZWc1WVaUVpCUSUTJlfDBqvcDZaJyrSiCGseQEIUIpc9dduU0bzTW\nZbhGLuXs7uFiTMvgWanHs3zY7eopS8nZupNKTJ0dVIbE0g+TaFF72JR3uNqdHspQmMrFYO0IBUq5\nXbqLw7vLYr8rIyaWFDO6wa/r3MTm4U0q1yb2Rpagaa3Yeq1V2KEUrIEmDUlCdEFm16WNi7pTmW/L\nk+UIH/f7M6/EgsIHnE15xnnxrqIM9TgUjv3hyN5RIhbvIkGExWkaX/jbh34ozDqL3wtJiXJ6KE/4\nXVcu3txup7rBUovJ6GeogjfZ0CrHk0AUj1Wv6graun5sxXCpNFaSAgNZN1GdJyQ5Fex04lVo6TYP\np3FTK5a+LvOZT1n65YBuUyaoaN4sd10PT4aJmumJMmu/DFd6q947qxHtIdbDBsVuhuvpu7OruP75\n4NSi9BJpMQ61ytlIC6IJWExiWbdU4lHOK7Jphq0pkmCWKikycjPBcRpj3vRdjdN4jjlLua/nw1Ri\nGeoybOJBFzqxyzwxB6PVu8QuKjX36akafT/1ccTFdj953zGnYJqteE4DO+/q9XLzgd+8sK/POymc\nGUH5JucwJpq6tak3TSKKTKIlYQ3DXHsTXyRLUyNmUEWXtM1CK+wXY+LU7R669OEmeg+X81g3vT3Y\nZvF+7scHmysGW8J2uWTKfSkqufQhPGhzfbhbhk0uEcne21pqCjTr5veXWj46vOw++sPxeW/JGKwK\n2vZTwq31hK6mMgBMKEnLMLOqaIx3U0IkkDSlA1FcTCYUBKkeS2CYDriknfpya2U31z7spH5bb/Os\nZwQWjeSE4xv/d/fItD7llr2VKcvB8gVvj7uemBPm3sizVTVAljfbbfWv32+/PP9fRm/VyxfO/TaX\nNGsUhSmtsTmSWKA0bwQoACQTrswAQLFkMxjA+D1usnRdUbdwVxeWzW0MA+S0h7GLabGd4vS2MMKv\nN7rg51/fsO94c+q1G/Jw558+7Jfd8+V9uSRiDvO2NKhHma53s51Pr/feP/9y/80uM5XMKWTnZY5G\nmjeUtJHeApMFj+Kv9agFGzGQ2Rh7lZStlImiz35hpkva+ATb3M+bXidDPWygzY1t3G7fnX2xfBT1\n+Op8W598dFyGvHrm+PaQV/sP//7JF++fjTeHceuxwJxIeyTrJBL9aPeblw+98eLjy/+vzp2gnFaj\ne1YXwJreYAUJ2oEFFaRBHmrQSHM5paQkkwmQclEyhKwOvgyhWsc+7okcD6extwV2zIHIBxuzTPHl\n0/v7bl7sQGjyF5++eP3bMxtwd5jsaffb/+MXh2/yqp9D6VxlwBLTmEDZjNP7m6O20tnF2Tf/eO0U\nMljqwvMyV3jDoyyTpvZMgKGQAsKaUQBCG8exqmda466kIPNyug6D14jArXLgcb8dOHE4cIDxwC3r\n6faif3831no+XD/fp9m7O+jj57/bTe6Jb28/un773eXGRK5iKmt0crpss/HDb+/TrERcXl1u/5f7\nbQ8KvWlZzl7Eku5NZEVSSHDFdVDkCiJNSmtAZ6sbKZMoz4p1ykp0bz52GhJLuUWv8NPYs5rvYysF\nbYTs3bi9Pzy5OnxW3uV0c853djx7cZq2171kx82/vfzu7YebOmU2KVhj5gDJec53355gPSy6qxfn\nw9+9fe4mUFHmnLY2pfn37aCtFbDJjAqTFugS3mo6GnvcoJ2mn4cFmZT6h//TMEfnmJ/2ZRqW41BK\nGKbs0hinnsL9bjPtL7cV35xSIPpn2s71pPxEHV7v/+r677oP7RiNtFxlFJIZhm33+rfIC/SZ2J0N\n12//a1x6lwk5lPOwzYxCaaW1GxDf9HxkaTUn/7tTrymLiaZ+NjatOST4+Ho2wOfYPakY6slLqcYD\neppi2RB6P4zFB/+OC7IsvNzZPI9x/rCtF7/B/dN/O/zn8ZP6kOyy7bf1dxWHcf/6W7tQVtturPTX\n9r/hA/dWpI2y86s50LUx4lHbAqA1S7TWikupVbMTq6oz5aRlqrUnpLPkm4uMYMExO9u7dQFMOUgW\n89jXWuuzPg633+L8fCgRH23szbfv4+H+2TwfHt5e/Pu7v7v6bH6ocLV5s9U82O4Jfv0P8XyYDjx/\nftWz7vJ/1vNCJxFpLviZMn31pmAlL1Z/C5CWQGdSGqyxedCjpilrMxymjGmUfPx6NNXRl0k4ZbeV\n+Tz1oBT9mMXHMhzfvOJnl+/vK5fLM7+5WT5VXr59vyM+/8uvf/HhZzGlF2ajyRortH2++fp/3z/d\njpeXH338pK/3y5Pxr++fGkoq3UCdTluvEaVV+YasrUYArG6VXBXhmdZ+rNYTt5CE5A3GkpHY3vxf\nh736sTtu3p+2BvB+OTd5HMPfa3eO7v5Nvbp8f1fOj8AHb+c3/efXv7n8Laby1Wd//pu3L673R5IM\nyNMaqlcu+2/+oft0sofdZcS0JLuPLv76zbNCKj1ohlONC6vhTUeeTcvO5qlgUsDquKPkXH2YlEBj\n8wJAzV5WjBB63lx0mcfuvEqjGTN3hrowzzYb3t4Evy7P+jdz/wQLPshXh+6D7v1nt8cf3oxffPyz\nt59tpjnDTAZGNFNpf1n/37/8+PzTsZs1n2oFx8+e/PWbZz2VATIDcmy6CHWQrRR0k1m1QigYLZuQ\n0DJyFTG26TAzVleJSCiSRvSbn4+bcjjRH16dcarL7bZTTcybUnaW+685bse0+Vl3j8HfVpydfde/\nef1Xb5cvNr88fLqpM92RjWJNMyPw7X85nH8yvDl3P9WAcPbp4f/17mlvRpgLRkbFFkvQhXy0EKOZ\ncBp0LpmZuaUEb9hzG+olmKFN6qzpzXoDOzt8fdn1PJ6V3UZDPQ2mhHPo7l7fLDV83nb3p0P58E3a\nRTz0Q3n5UL473/3qh4e/G37YHzJBykuGrNlRa2D75cV323dhemBWXnzy9V+fnhWpyg3RSPvS1zl8\ndU81vgKG5lDDOv01ZSoSzBTZ9NyrEas5Gr2ZTlLqzv7ltIvrfnn//JhF1hODcym1v9xefrSt/fZ+\nf8ITHMwD8Pl+7/3x+uWfPPvH/jNM0dSjzJCTnvMxvHvyZJp26I5njurnnz/92d/snpRSiikzE3DK\n+r5O6lY+7/vJ7xEbAVlWrxqUJYslkA4l2uwkWiUoFK21fPfqt3+s2N7N/d2QdSNGnYbxnrt6gxHb\nszk3yzzczWcv3ngZdx8v0/Orq8Plyyef1tNSZIBlVjOjAmbuNcc+jx/c3I5XiPOzNz+bL0dfOrbF\nA9CWxAZVg61wZ2ucMh8F+a13wmrRBteJfj3UmpQ+IOVq+QSlcv4dzkLl6UOPg2DL/t7cxif1nYbo\nqL0J4/7hYrOPzNO7+5vTa799c8cnp9MpM2POrOFkPWUp7kj6dsPjVyO/YZf1mz/MVxuoZMrMkGDJ\nQD8sixemVkdHoynMHxvd1n6wgZuJaAowAGFSsyGZGZMwNFsKi6p32FntpZF1GS63Nt9149Aj7lRR\nErsHfTK/68yo4/KAZ9Npxn0NuUIwdAWWE4ciSRHBzsfl1XO+73r69qKk5CVpULDZbUar9dE6BQpu\nj7PF6hw30+PJZWwDB0XA0RTEDTR0NAIwlSINXX9z4+JZtz/0O0awe/h22FblFDk8WVhPl+MpYCyY\nt88e6vOrpxddDh2lIvdlqrHZNPUcM5bi7B+Wy4NVbk9sdYs1mXSTopYxKvuWJJBtcG2ySazuB8ka\ntwcpVs8eBElhlius0OCelQOhMVjKa3QJm+o4pmbn6+/m4RA2xNhNXmN88/DppaLG8X7z+auHL8bJ\nFrlFZEZazrLR1/dilCHF/l030137vjgBmXdkVpYIjSVr5zXNmxsttUJozESz+JgxQ4TRDGiVY5Uk\nUdmIfEPCSNQE4LYfhnfTeArO026cj103nA/LB/NkgO7GOm1y4K/K9UfPlA/Dn55ufrB5uOurFasB\n+cbm0lMRNYFiMKN3MEg+7fp5Us11P6q2dqnb5IIeNZZKc9JtVcM1Dp+NF88mXFpFZk3/ju+3T3t2\nIzAtniaW8u043AyjWa3nFn54WF4+jP3VMVSsO81nJ+/ms8D7X7zNO/3F+29/MN4M82HRPMnMet1E\nIWnm1vzEhlJUwON4WzqloS9OiQgIjshxyMk7kah1WZprt1kwGtOExuSLQAC5ZhkQNEWDDRM0cxLS\ngk/OIyFtH7I/ltIv80gt77TPCVb85GNn6acDFuwuDt89y5jzz06//WB4X0vPcFd6P9YbGwCGmpOa\nkSl0jo6zHU4YB0MVRHMA9IK6bDSra8YSISKTpDWJ2tpgks19TJPsEaJo/ToFmpd1dBexm954UrbN\nr88Llmq7PJXLJx9fz3Y72XK0cVb2qNRx++TJm2e2n/5o+N1H2wNp1GidDJoeyqimsENjR4mEo1pk\n7SeHL72UZs3ZIdUcSizeNcUxjYCihmgrX08BAWuka6prs9EjME95aVpKRROXLLdGI2Xjt/iAeRxF\n8qHevqxPUKrOh/eKQp3V5fLuzdX4ArdfXP5qc/lgpTiO2StM0zIUAErjWrKaaA5u8OiO/Tzm0j7E\nqpYNXPQ1HGJxawkDUGYs+T11gdXpY0hxydWWY1RLAAgzhFZNSYIbo9HcNvj6wpfB4BuvN/vyBU4l\n5qd7u6hlRpHVPb773Xf98w+/6q/uIeNytI1OmYGBBVQ2BbaafA1kJ3dT7rVhFWSKkEzyWrHRotLy\nYMxptsqe6xyNsUh5Ux+jtYHNvLjadJUNzjVrwgV6zKaaEXN59vWb7Dfsoh7eabOf7rajfz3zagrO\nwzL0r+JJ6X752b/5dn4+zWJiqBgOs0IUV8C7UVqrqcEcnUv04zB1EkQvuaC4Fl3bEsXRXAJuTTwi\nWUdEXQRjM+CZQXJ+P1iRq7WWzQ+IFgkDphsil6U7vTwFlzrDY/4govTd1OvuE3uALeeH6boMo+M9\nfn/z8WlRFRe/GiaVfigNy6EyjVq9y1KUDmSa8rTtF0HZNkcogyPncEbTyCdgRtAefeUZC9n6qJQU\nCcFMa24CmvfWVkgxDZJlyjuXzq9+8pd/NFWwdHfD5Ul2cVvri891jyLvjzjbX/4S+uN4/WlEbyBU\nL88turHrOmeT1zRO5XGstq6vIhnHzkvZOJFRrXOv6j1rFrT8kPYAXMH0751iEkvrqGAmNNXwCgmv\nnh0SYUiYmvD2eP0E394NuxLi5u3p491+6N7FgHgZi5Uol6dh9oH7T778b5+dv95qkdgN9uvvrs7G\n3p3xqLnxRAPsizk7HJyQH7yvOQnez1BQiV2ZsnjTR6+ILVvQDFYPKpioJWVAyyVqPC7Wfdc+P0CD\nJdVOF8UXN39/OL7XT/HLpb66+ug77v79r13LUjfWd7EpGy1nw6vui386++Ktj/fR08f51/sLLlw2\n2/PB6slVW/ZFklYGz+P+IRJAX323ZHeER4WBGSiKGFcFpNbYCYY1kK55WMRGSram9ntWsHknAkow\niiSDZxoskepemV3H2cfHm1Pdjmd5Uy7u+mU+51j6zJg/fqJDf/fwJ3fxk/fY6Ii+Gx6+O3/2RX58\nvLlLP7s+72qbQylL78v89t1pQG9pqMf/y9gvF2NwzhLwVPlICwv0aONqNa3hIgY8ZmKpSMaWCiPJ\nLB7N3HAjYU3+ajBiYfg2Xj+fjr4/nr55Ns6769+cflC/+SC9f8/h/Dt5vvyIL8u7L3/w2x8sy9Dt\nT+cX21evz85sbx88/PB+f7zP62eN8hVgtMOru8HG5/HRzTGTFjmc0qK3SAMMxU6192a91trZCoY0\nPu6VBGUFNFHIdD7ymjSBVq0AxmW1zyWW5Fk/ef9Hef77T54+me/9I/bdl7/s/t1m+XBZzj/4dvFN\n4Lr/dvjsZ8/10Hf1NDy5/M37K5/x8P6v7pUvrnTUcZBUSFXp9DB+efak5C2W8cRSFV3MTdlhysV3\nChRFc+K0D9GOBTVrI4AuQ1aoVLPoWwMLLZmGAJVo0mcBYK22KTmrnHb27M2v8njRXz392WU51v5u\nu+/YLYcXv5v7ccmy/OnRL95a77M/ufrn96PV2ZbjP70vu35jVrV4hjPoI05n8f7N5B2HrkZsQXin\nLjOAajjxo6iw70tltlcuNe5ZaMFAts5NSjek2khFAyNBAxGktaCUeW8ffjajouDbv/3N/e3N8xdn\nT0+d0cvmwRzV6jfPfpj77P3ti7N3n9+z2Fwvrv7+5Vj31eD9sa/7r2IoRgScop1tDt/+7vd3lf1Q\nErMDsXQxZghJeMHCrkaT2gKZsNaO0NQKDlZnYMJqZiZqtF/QJCZqyGcCEZmZUJy2P72+PST6uUqv\nJ/74y+/Kuz9czxeIotiNnZfpV0++uK7zyX78q2dxFJGbi5/dXiMcLh/+9E83p8Mv3vYN3BN8u/vt\nv0zDxZMfn51qsZMbLbOfwkzRWBXbqSXhoaWn5OprEJxEUq3mwWTeTEANTmjWy0ackGbu7aBO2PZs\n/hZy3GzLswNfPPnnevb7w9nIbwec4nbZVt+c/vl3m0udPp6WZ/cDDMP5L24uUHmZw8Mw9R/8+bPu\ng0OEQULYsPv6N32pz65efGBz0qn0JaiOa+c9z8Mu0zwkGI0gvweh/tXUISjCkkYDIW+pbEATvkim\nR0OqlRGvHr768OqY3Yvdu/v46MXP4uqmH/yBtc9acPeCgdF1c5tPn/3hRw+Tuo7nX99e1TnK9TA8\n55v377ufdmflEKFMOIeH31+dn3evfzP1o8WC7LzLHLL0oWjxV6PPYVpX0eOE1PrJhLSOsKjL0nS4\nJBQtBqDlwwnujd5AYbkYj3cvb35588EPnn79i7+Zlmt+a08OuekGP8/Fuq48FHWAd4m7zVd2eC0W\n23776vnl1ZN8OF1+9/7bT59Nx9jdDXMXopPFbvofwzd315qBVOiIktHHXDJZ2CPn0au8/YlXTAD/\n3YPWMqlQ61TL1prtoJXHx60EKEEZV9HF8fe3fNH/2YtPf/rq1M25eXocOw39Rb/czT1qoouvrUsJ\nOX/4ZP7ycHEx9OOr15fQdnvGh/LZ1U/+otry7dY5oCADpdftoT7dnz75N40cLnDCjma1R0oVqhpq\nWFmRToilrZEGUikzgVgq+t1YClbJLdeUHT7qXMAmoezGf+Dpp1svF09+/bfvr5bpwx/rzS0+3g/3\n1sGuxn5RdrXP0oH1+PFXn16FUMY/vL4cY3E+vZ7i7El5x9A2SFuMWQjU4fL19WdffHq6v3uw3WxW\nkFh81PXigURg9yyqo6ZnWsLUrVbvRoMhyawaehpQYJZAtZU1abXFAhKNolhw8+WHH7yb9ex3P+8/\nji8//CYvX3HzybsXD+f12bPX13HqOiBrFxH7j56e/vxlZ+Pu96+utxAWdePZ7c3+TPt4sd3M982W\nHhW5/fiZbh766f1mOGdlpSeW3M4uFIBZrjmrtL6CkLAkkCamWipUSNYXo6QCVmO2zkqP4qRG8Ta9\n5P7Jj5/nOxzuN/nUtP/gs/Li7MUSH/JDy5vt/Lz/mXWeTeB64Jf/9GyX3p9//duLc9ETOjn7cj7q\nrAwP9x2yZCpcYj1COfp47VtbJEQvS+0Q6aqbrP6RFpZs7LAJqLBEEpaSZbJy6NYBz5iN1MwGTTcr\nR8Jacp+EJX5wfWV372p/HsvoX//hj87OLlEP/cOX/u3E/s3VpgZEqzWmL/b5dO/js9e/eLJTC9p0\nHaEBvtt242YLJ8Fwq9ltx824Md+NJD1hkVClJtKtqzM9wtmwtVzdODQq1KwBk29HJ6AAisIkZCQu\nS7bAonYqGNWC5d5d+ru4nT/u79Vdjtv+56eP3y3lsPn6k89+ON2X69c9lUpLO108/Yfzp6V7/vAP\nTy7ax6dDCDM6kPQpO0gWj1K0AmVXjYQli1HoaLCqpXJk0FsqyRqy99+JMOQS/ab7Pj3SckUJ3duE\nvAIj8NU/HsnXX7s+/Y/X76v1tnta7q+2s3f9xx/8/rUdy5/odpEbkx7LD97oycnP+b+NF0Y9Ivhd\nDxCxpGpYyZbnKjAKQ4ZEJjPFjKgaOIXLlbaJivKvxiVkSwNDAhk5nHmmMpESzD1XV6webf+rigPA\ncpqxvXjy0z/7wdX+JTbdZsH97ecffSPD6fWzH8abn/yHl/dfXNhkQCx3F+dfP1Nur/73+dpDaNp2\nQ6R51vDCCss1xUT0lCsks9VO01qhfpbJld2odAop0lcDVMZKwmbtNvY9/EkUYuU+2CDHlqUBZiLT\nxzPX0h//y8unw7KZ0x7Ghw/8q2VwKHflavu7vzv9++27A2BgLJ++0VmMH/zid18O649PE0xcuubB\n8jCCNM9gGpbiLezOK2hmKhPjYhk2sLmefXCMrs2hrYo1aX0LXQvfeEvgowCgLBR9WXHzpsJo29wA\nH/122gSP8fE5u/Tnb/zS96f9hlRRnv/m+nj3NKd95iDT4fwn//zsMp7e/v3lsCFT1oJnKPMUyYY9\nsaVqiY4MJwEZItzmi2rJ/cVch5IRI8Ks6U7b4bTWA5EKbQrykSdb02sVtoKeAtspKzJpnO5rt13e\nTxcXy3iyT/hw1b/5qpzVSlqnZVvO3FSfYU5xPn36rr7Icfgv45lBaVAkGr/L4tnCQpsysqDNN7Y6\ncRFOS09VHLeCbTJ9iKDnGgG66jqt5VzGMvRYhRXN82RIIKL1I63paqeBqKgqzuP73Q8ub7a+/fz+\nd34PXR0ZHUl1NQ6d93h7/uLhFMv+yea7s8CLf9k/b5+0oxjUUqtQW4qlIiUlZFjkBHPtlCozg/Lo\nC4cliOyHmkOfLcy1acGQUiYzwwc+6hOaj9ZEmIN6hENbSyVC0W1qnSZ9+dPpt2Pcf/L69fnw7eJn\n96UnUqr+wf3EY5y++vLPZlCfvJmfxbPjzy82zbIQcLjz8fQLoalkm3NtElmldKMhFQZHcA5bLhSZ\nscVS7Xt+idZC6qzlx43eZAcr6Ka1JycgeNOK0KyJ2uow7uvZJ5/E//x3lz85vHjz+8vzeLoru+5U\nDCHF6eEyhq7/9Df/9GeX9dhdvNmdbc7+tmyGhCKaRVh0iowmdWRtuAACBegEoqbgqjIHzDTpOGaH\nqFtlsWkNPl0n0zU4YvGhYY5US09WGhpf0pARNKwUgIJYhu6jy/fvvl1+9OnX4/t3Z2NXnr7P/fP0\nlOpSrl9095vtxedXv3uLaf/54XSJq394dQ1vqUTrzxK4tN1JrrMbk6aZY0gkYRmgMMiNy8DNSXXp\n+4yC1gq2DOEW5yJk5ehKtRSJlXQyywhRRG1pGS0NAAJ5Xz85fHW07flH714vw/hkLtXunz1sT0cj\nWPyU9/2uu7n6PL8+RHf9cnh29eZnT7vwBrM2t7MEclHDJBNrHg1Uw7pGYBsQ1RiomZqGIDBr9KUF\n7NVVH7kGEJohS6fVE8CmyW2UYAvHZDuB6a2sG4u+fjVdPB2Q726vyv586WaeTud3B9W5dgU2/CF+\nMLz7h//8anNbcT0tn+7u/8vmjOiIzAywfdRYI0ZUw7A0ixihCvOgQzKeZHIL63hi7TZRbVS0dApX\n4ypaL0Jl1ejNdGLe5M4grNIdASlDq2Wuwb7zXK6vcJruDk8Oy/VhzLunX2s4+aY+nTEOy5SHs/OL\n7e5HZdpN0xeXXx358nU5bwxrooloZDJvZtT2QxcjOjeTk4VzI1s4l6J0hGlfrGxisZ1WDBlAhuBN\nAilkltL0UvYYNSTQOiiDxuQab7UqWyvPPj4/1M6mrvLZROxHHrqL47x5eZXn9WGUXf5g/s8PV999\n4927T78YzobjM5XOCQxhbo6GVNSkE49Zdui6cXdms6yrRMeQrOwrg0zLrhyLLUMsGCKtORtaWmfT\npJsh1VuucVFNG0ZCFiQdNBQ8ZjwAEIerzftvJnSDshbOfL/86O3wkH774f1Hdjt151oe/tu3tR66\nbliOH97W8x9/+eorJ1KZq2pm7YSQ9XuDcFdwex9d0ax4qAScXu86N0g13SZ1U7+kW0T3CHiyXXCA\nFh7ifTuIWstuXMV4ekRNbI24Wym1u9d7G1hPx3Gol1u7uxi/e6754vBsOfTL5eZ2mU/bj/7jD38X\nnrdn5xq63c//lqMZU8UtMhvD2xgMWXPv9NDdz3/93btFU+V9Z0Badx+dqALC/CQuQ+3Pmi/se6kd\n0ERizLRG4a5/0KZGUmG2pNJSmlcom6laFf0omKsvWG7r/fzFN3mh6Pen64fxu4fp0+dvNV3Y3/lm\nmerzDw98Mv/ez4hqLZQsLZuv5HE4gyiU6bj5t736o9WrGb2S6nUowwwhCXSvlsJ+2W0i2gZoenM1\n4k4E0DUthdl/zwYVCiyEkDlbZyCCiIZFWFrJfPu8v9vv/228f6pSw95/jrMZxw8/Hfd2ffMdx4ty\nPnxz/3n/P+HsrLuJpJX2QYWWualHvRDTNlh8MoxPP7teEqZkeb+4KDQrn6azoOdwSrN4JCya6Uot\nrtcZDUp7dLG39L2wUpxy+z6Ur80dJCLjuMNt2HL8KX8RfvrQyvnvNrzsZ70qm7iavn7xpPLT8e5w\n/udf1Sdlf1BBIh8LRKbUmBSDIHOUzmrttt3VJoMpoOh+ZAIoYRjc9p5H7y1hGS00ZE2Lb2lMMvs+\nwrYtG1GkkUXBXFq8mGDm9mjWMtThcjPaZD969rX3UmY/zDr68zx7/7vTs/638WLYPOHPl6u/+vYX\nH+w6l8t88UbsklYKRcK5RjxCSfPdWT0co4Hfwz57g3rZAtG6Q+nmfhtVtgawU60JXsMUSxtoVyqs\nPR9pVUsSVdm2FBRCag2IXsbzV8ftXD/rfnF0VT/eHrh9F6ezeO5Hrw/xw6HfnP1+Wq7i/7OxUwY6\nY4b7qhZCSsjMFFRDiGzDQBsDAGLg267l/0jpEbZHn0MXreS1maHdW9D87K1/bY0WHrdLa9YbMbDa\nvNaEcCKFubt+i6p8wX+sTwZGn0ZMuF66h0u9me/8E93nRX3bXX72v+psUFZhVixgINpCaqHRDdmg\nrfpqEOEhgervlwLKoogwyFVj9lLRrN2tSNs6ojZCbJXvgM02Qaxy9RRZMpZc3cyrh8tCz27rdhnP\nx3e9+dBP2S+1Dneoxx9dvF2Wj7vDt8dN/y8z//Tnr663QS/qCixRKo1YYzKZLW8pM9p76ovR+r5L\noPhNTwq+WLWlygpUYFbpa7AAqRYeIgARfLy+RSBaLiJINFOjlJCvS9Cb0JB1eRF3ZfHu3XKzKcti\nQ2Hy4cny24vy4hffLi+ev+i76ye/O/Y/WX55vnuyBLKiy5bevAIu1rQIzjVYEN0w8IRPzg+589C4\nPxbrgXRDdl2XmqIYHhXzAhK+StHV1HntRpb2LO01qQWmJVdtHleJqMAElt3mZRarb7vzjSPqsmx0\nNkz7H88/e5ivLgf+tv/RF0+/ecknL/7XYchSCS8JmAUzm4tKEtt+d+/GUoaOOd29/u723d3h/W0O\n5U1XOMPlFcUrYBmoPZuPoG2ORxYDjcdrIENDoxqUQ6IAYZbpCZloK9YIxL7/4tUyflTnzfaXfVU/\nHfHDvfXLw8VnJ935F5/tbg9l+93Xm/iTfzp+wOmFoTrmzcNstemMWmi6aPLDq34cugxMTKD44d3x\nEvXF9uG4dZ8pF2diPuuBk5XSIMxcZefrdMf2xh/V3aspqhXAIrolzFDav6Ma3Tvbl4cbf9Ld6jJu\nz86W9HLsxofNxfXd9U/+6eXUfXa/vZT9YV7+ePOrJ13lMsqQKkTUgeEuMWnMcCaurC5WrKRmmZvt\ncud5Yd8t5hI47AcIi3XG/XXv68izigVzrRQS5aAa69RgkPW0KoaUBVe3DdRyzLjgj/W1zp++rcPJ\nRt9jg81p9/5whe5av94NuLqa3/zxB/98LJ/96P+hDWBnLi8xL92BiK7Be4Y00JTjyOfKS/f0zyn7\nYXe8Ka7y5nDhlrSScXlyK6J48tX2IpkAz1x9SlwXWGsK1nZcJiFXBRKbYhjrHShA1k/0W/jlw1S6\n+mFZTtbPvvTx7jT6NH1zuHjKmvP0+jfDD37yN8sllJzSlpCHIRJa1owPSMiExTQt0DxVH3a70tlx\nrijLqwEti36PIf0EuSz9+80LgGHrXRCP0tV2Vul7XQVEorRRUbR47F4A+XY4fhN+Pr4eH+qLzanP\ny8XHi+rn+zN9eHsV9szq8dlf/k/6/JO/+eo8jVbTuik9srCSDR1UNjUTkPDIYMl63MMmdFPXWfdm\nuQCjZc94luidHqHSQL/HS39aKtqjZp5Maj1Twe8vOUqijRjxKAsCLY7TjXfDx2/zfLm7eD91ZXu/\nW57sb77g+d2ry+uPLqZnnMv96bMvXv/+6ehmpXSKPkGwoBezCgGasXNkBEE31ETSvePU9fTTfuzM\nUdwE1hr0xZp0k2YNkyLafVMtOHpNM3tMqOEaOxDKFmElWYZWqwaSXKKU5aK+P1um/mry6pzu6nW+\niufj5rvf7yrubk/DZv7w0/3fjB0oRVqi9CoBzoFeOdOUiqYfbXIPo5bj/uHmbtnusiyvrSw5B0su\ncDciKDKglbJsBJ43hBmQmNHu6vn+NGquEzeDI5baFdj6+5EmeMfUeKC9rdt6GOp23037J/XJy91c\nnm2momm5m/p35/jPZdeM/ZnijIyoXSqKOrabXPBYjxJQlrEvZXN12U85vJWbWXGaieFo+Rke/ijk\nWkn6xtM3W+4aP7nO2hLoZqSbpcTsi5dmAUIqG5duC23ybfdq6ms3la0N+ydFxT7Ilxzfle0wfvKD\nn5/OLZVImoMKp6xkVKE2P2SDUREtBty8bK7ONwVL3S7vziyhQKy2nZBbKud01hZgloCyqmEd68Vp\n1tw+JoSwXpbWrO9UKTUaLQUCzJTM7B7nrOdXewbGunSnu4fj5Z3j5qiH4SO8R17//ddXvTlbGiuc\nS7QbTSxtwUyuXXMzEVopDDNhqRFl8yZ9cYKlyKSCpDhEKU0/vyoKE2iygxVBbd7NNstQTeetVDZ+\nBO2uH2ulUyQyZXkEWCmzdFWczajezWN3Fffst3e5v//FtauG0GXQFAalKkwI0zDtI4F2X1kTCbpD\niDB3DHx/bmK6Z84byUHjotOgZDTYR+vxlIbV5mBE522YlLjCiBRBg9UUUR8fcKU4jHCHney2Vw5e\n2ccQvP3dDa5YnsDUnWF5+dmZmbNJFIlwQ5q1Nk/c1EOV2jVoMMJgg6vW9C5juD9uKx2ZFu3OIDB7\nq6bKZvfGo0pnRdTAhLJr8+tqYG9XGrEMxmKk1Ug3sN2EwzWQI8b5kDGUSj+l6rK52JS4e1jszN+d\nP/Tl+M9lFhnKuUVFw4R83AlzV+o0Sd4VMxOpmJN0c1MW7b1MHQ1RwiaHV+RCCQ7SnTBvUOeq/29y\ng8o2kK3QrwQzls1uNC0Sc2bXt7qZZFuPcVzm94qLY9I2IMt97XZ5jjlP3Yd1GjeveXVgyTRYKZRk\nYTSEuVcGVEbXaYr5dGpnB4iJNEQETehhJjM4BxksPZ3uJmQVIzLSmyqyddwCjMk+0a5/aZ2Jl2E3\n5qtflKaQDHA2e9SRSci42fz4u8/eTB8fT6PZkiw1fEi7Pb/6bjmru7z6o0+Op/ttQVnMYul7JH0x\nJWnqVJbSpUddqli89CikVZVcqnFx7I9Ls0tHhuQjDJRZtQcDo3qhoFVASMCRVEZXEhJXvyjp5tPb\nt6euEAlF16J92nFMBud3/B+3b/4PL/f7zbSr3cEUPPZPXjiPX0y//mLpdH/afPTpN8etR2xKIkNW\nC+UJWFazZRlLDGOiM+jcJGPpkm4bnOjzw+F2NwACu7SFYTIoQuzAmK3SJIsCoERSSiYyto+w7Rqm\nnqf3t8GzoRgYZKeIdsmpIUkuD/xP0389pjbLZV/Gin7fD/5QNm/7/ZM4/+zu5R9d/eL9T/78T//x\n9twxl4gCdZQJYoYSqQnWD0ZfTSKZrtUosjnvh7+8/+5Gu9E0K4xstJ11WZFY6mauNJlqUw/L2g0P\n2FD6fqTNVH1z51tiazK0K1vAdqMcacrT+f/w8q/n85dHRtctXu2c6upuv33xcLc///why/v85Itf\nfHf9F2VWZ12W4jC2m00oB+HyBKTlcDzs52VJZqSwTNM8Hw8P8fTf/ccPTjf7FEl6B0RBpcuFKN4D\nQiCixlxDsuag7tYgyebiHfu5bgaWbWe5JMxowdqI5kDWevHjX/71pv6byS3Sl6rYjNm/unv6zebz\nL843F1/V8e2U+2fvfz3/xTS7FKqgWUVRR4RZmkNmK3tmdJPAUIIszpz3b97Yn/zVx8vbe3iKRYYq\nl2xAxOB97wZmSLnUiKWG1j/8mjxHqvD4dir9MHRh20JVlLrU04ImV4r+fPuHbzb2Zf+qL3zflbrE\nPGrUV+PlV2+uL59+/XCeZyf6Jz/yf9j88H6qCnMLJSuyxnrblirMMkhlOpIrG4kW2UzMN2/yx//D\nFw8v7+fIpbarBeLYRcAiO7q7OVRVq2KeTss86/HuVcDLGF99tYwDwKXauOt7o1VwzU9w+BC3x37z\n9E/+2+JZ7o4/ygWHDc6H+qsnn8RXv7755uMROdTN7fsT/ubqYnZas2oSdCS7lDNRlVWWK2yfaNcQ\nmJHOzCSW++/mn/6np/cPUWw9ah2lqi+NtHcrZCGzZoZ7Hg/z9zIjG+zNu+5q57RMs7DBvbCw5X9n\nStovBXk1nt7tSqHPy/72xnuca3vz8vCDD36S5bDZTXfndvb6iye39Y8PWTMAs8LI0rVLKkrj4jNg\nxszvMTOaEe1il4Tq3Sv++38z3y0nMkUzxZhLm+kEuXVd6ZxUtaEfWclVF1By/vrt2eXo67WvkV4G\nZ8kQW0CogYzDi6vjvxRXXfofn330/IsXb/DuFD/uf/XtxfTVLl91Oyud3fzix/m322fHufSFBsAk\nM0PB1Aex7pCs7VJBBNbcOD7ejmea3t18/lf29iAjaFi0qehWk8cq5rauMytuUfuuwc5Wyv7XL+26\nW6YwM84GevEO1TLNm5nfjFP50fvbepUS73/32/6PPvzdYsfNT599O3z63csSF7jvL4/WffCPb/7T\n61/9ZK7talZAiSoFme4hAOlFYT2ygX0JN0WVoHZ1K5j7N5d/ZTenXBYlojLU0mYetQki0RnE0EWR\nDClY3u/Pn26b4V0NZ0UxQwsYb4iCEJfb2/5hLF3n5+PLn3/9i/tZmyf5i/tnT177BR6upzdzRHn2\nwd9/8Gffld0+GVUpc5lXsEvari6VpqIDHIhonVALPpC0Rt8B0Ont+V/6becY6eqGMD16KdkE6HQW\nRoVvFWhY8HLktj+FGkdmJlYFx6HJb9eTRPB7+8vTIawrG396vXHf2e1Xv+d53D05YafTde5H3n71\nxxe/+ol+/YOTl966ziiXdcUKgK5TlM6nqRsluXs2WQfbqdkM8KAIzG/P/92rmyWwdFH8RDU4AliF\n0IhkBlDGVe2Feqy7HcNhfUcErCrpAz3i+1h+M9jx1d3PVHyaDrrv8v3RnrPrypPdd7//dPNu8Xs/\nj94tuvNvf/ujn38+LXNmxlxrCDHPU8rNqoRl2fSVICNaTyoo2n0OuSa+g5hvPv7JzQmI5dizdtYo\nsgAaxwTQSlGUPtbpdnnwM59zVqcayc5q8y+VPrPhJoIAO+su3/WENIxL9NDWDlelzs/9dLzsauGb\nYbw/cDN8lL/+o+1mnJiiFXOgmsGyB3sxj9qWyNZotwM/E0AmWyb644VZx9ufPj8oq+UuExFtkmvc\nJwW1C1ztkUXjwzQMNYt1nddQgS0ziyB0yHanplGSnbB5IQhxOUulQ/HDZHg3XhyXuxc5bo9vxwPq\nHa/z5r9eHDeTMoHIzM4x2Bx9ouQi71FpQTaitw20TXDQLDwNioEOyx+fDp2pjlO7sU9skiW1yVqx\nVPYhgbBS7zFqIlAwz3KazXeAydw0JUJr3kD/cPvinYo45J1hbgT0tiy/+2Cbp+PF4fxTHpddZ6cb\ne/7qAz8zc5m3RLtYUDtfUsiCZMoZjW80w6obX6+xfiSRgHpz+fn9gVOOuWbqrVqfloeSdeZ4da4Q\nJeMhh1Ll7DjNgls1g5NpJdYNt4Ip/q7OxoAfrEARY87wGObb55eb+ykXbaPuL67j9M1zXZVTLD4X\nl8SoYpeFceyKicwINexGzY3+qK9C2JrZRgKYDz9djggO0cD+Fd9Y+/KIYei8jYJm077vLYvAENyZ\nsqJZLNSMYo833ojycedjoE8UWQ8+mbEfOXTzq280qJ+PE8lYlheb/YPHnVtn5jBmgWpyg6OVREYx\nmmwFAyORLXINaM4ktnuZKeD+8qP3Ge7RwhKamKgJ54noNh1Xl7tpv/RczGCMpBlW41AoalFmEilK\njtRn9w9l7suCTJsPY7nvc7audMOJUVK7jS893vvDZ7d3+Jc5a4c8UWlZPKDl2G8YcDFk1lZPY7ge\n8QCJgdWXIUE5n368KLty6tolYVhR7IQUdbtpGAzoPh/NlQZrES5MFNOcELyAjWLjiqbotaH0U6a8\nPtQvKkc8OFU1+qSYtro9zmXsf/3shz8YXv7fzxcIQTctU+9ZunEIqAiwgnwUBjbF44r6N3SGIOEi\nkfcfDrfLiOrf20wIb+H6id0q9SKQx6WHDNYum3bAYGQNh+SMRkI1CQOXPcrmVI20o/2E7+4S81hD\nJ4zCcjufHYZuutjuvnr1nNdeuzrV4nRXVmrsFQj2aRF6TOhsfkJbs88b598+RaM2T/hwv/TRFjXX\naxfTjFlradoqALT5qJI1U1gWM4dUrMBgsEhvIpvkCuLue7/YpxNZ9UX3Tz+2kdxMOQBd1TCyfDn5\n7D9/tvnhFz/7LxcczmE1pILMXh2UmNLqsB6ZjS3G9xEzJNPahRrCaleIw+fTuKntAaWkLMVgEjki\nMtMMspw1ukmlpZgboKqCJFJppWDtASqt0mOXm3TKtPlw/750GwKmHFhS2sXv+Xz3sPTL8PR6+jf/\nt3sexuPgu6i+ZPNIs8yk0VrKSoDRyNRENjX86hD5noxUHp71e0/Cl/VeZTUxMugOtBgis9OpmJII\nb4lhEoACBkwGS0S0Iw60jKswGBPdPMz3xZ44Ndx0nfeCnS/9cf/0430uT/JvPhj7Jz8+/sfb0yzU\nSIkqnnJkZ1LYarJtSk1PlTZMi9nybx+DdKiTv/hVF5A8m5gFsGBAREGzXcLyFCOiXcVdm77Q5lJg\nNQpgIefqIKdJXdzC3ICKW5156IiuO+5MkZFnXwuzjnfP31365mH3sxfH+PzwQ+1v0pnudIiuJVIl\nYWJw1cuuHXRLmbL1AiKsXCTq9OH/sxdUTVijhBqSnSqybAdcnSkQUahQ7RwZbRarSqFRIY/kJzIj\nZutrIoENw5dK7OnDHMOBs5+en5dpeyr46MfLxdPL4Xq5/Ze//e75Tktac3a55gGd1kXTLoY0ZpPT\ntquTEHqEvtmOoGc5C03c0sKlwISMXtc7QYjaLhpWUhV9J6CstCzQnHhNGqAUmD50lqHQgquhQpUb\nyI59sCxXD/78zeX0jbrl5bsPr97qfbmI7e5y+SoYTeWexuPSod0QuiZxtBfU0kzEBkAxHVRmu432\nsLs+0MzB75PkgDTQj5PWC/uWNknQtJ98YC4VchoRlWZAiyJfVcasQ2x8Vmr3oW4XDNI+hzJh8Knv\nTscn96//7E/Oyk6/v/9sev50kQzeuQDvgkiKJytNwQFThkCt92UbHwMmEdluvKSJSMz5xcligpVS\n1gzvBJFWdHe/r9ZON2VWdZYohdkIzc5gOUOCD+vx0X5uLRossXk6vn3fMoTQlcEfdn74IB+O+w9f\n/ePL6LXU++2XfzpbXUos5ouJ2bkTlvuBXSNWUNgoCbVs6ta5RrarWFaLLkTm4QNZXeqSKaebtWgg\nN+91f3AnGekOpXmdNFhmhdr1gIRb1JqrFq0dHWUJLknWd2/DO5aFsP4wDWU+mZ9kw9mLh/e/evfT\nnu9/s3v5hzHVkxYpWZgq2E2nDawRyplGmhvM7TGUA1z7RTbWCxAY01V3mmpdalUFihOGpBm9c++M\nxFzNTdZhVu8ZBhQXF2uX9pWiMFulWoBgd04XaH1BWLE69yOmJSq2d6fNctpsh52/eTv4F/r262Ej\nzoGa9E5OkV4ezFrLC1qT7AGsyEc6rAUQrld0YXUxLP321RJLXXJZllxEmhUSsST7kiAWs/mwuM+L\nulIjgx2kxQA+2EArXa3rLAaaDSdDP2HDELg9mRCHYqg03W3PvjzOL/b1efnVjbG7XrYx9a0WqGNH\nSN3p/c69CjI3WrSQvqaBUEIyZn4vbFG7b0CW2Lw3X5Z5OUWdlzkj6X3Pepjq1iDO8kx1XZwwdEsS\nZkyPMDDDU5ljrKOGIGVfRzk8F0Z2WT1YpM6CY+LZN6Xc55/ETXfx4Qd58zXGBV2SNBPMQZi/1bZj\nEopmBmucO7WyYEo9jn7rw7SxY76Sb4fBgGU5LfNU5wqo+uVohFmdoi619DlrGDOMYaUSKVOtQkLs\nsM6VgKHMt9sS6o6WuesO7CGFd0VSZ9/a15U3+GF3Njx798/dfhiyLJkJrPB8P76/3XX+GGoOWQud\nedT+Y30erYkVq6FbiXkznFBK527KmKf94XA6vY2zYelLShm1TlHiULsBpxabJkBm6jokCV9VGk3c\nU+a0/dQjlUN3LFwqhsglNrbEyKu89z5vP+jf/e7JzopN1smRkaR5Devmt37ulqtwpM3V1l540zSb\nrfHKq5KhPRJR8fwYpHvpShPmnu7e63o0OgRqqWGDp7rB0zzTTMRcZduxt5PRPK0NTu3CgaXfLFOp\n6HZxC8Pcl4ogkJrrGDu9fD/sloe4OvO5j4oKCegLjPLRX9WL0ZeWxPioRFlVNW1lZa7quhVjwGpU\nyuOLmusQUgY3A+f66YanqaMXP81hY48p0GnORDQQ0mQD+7KEWL3XYzwVoBxyqOSmNc6etjkVc57k\n/XEZxpHvv9bTD/o7PO8yC1JpwhDuwNa+PQ3bLsBswT/tAzdnGZPtDuT1Ou51k3x/rdl8WY7RJOjG\nUsw9OSge0NPL8V2lWz1pHLxWAbIO0tCZLc0KkiylJYG3riHLaSCz3i80AbEs0XklQzZOqByv99++\nLNd+/6arAj0ElEgjN+Xl282Tba2QoSknVxpe/z1hpgG0xOON6S0PEtJUzvdtXE+EeUkVHJZlPHe3\n+28W887mKINNJ6Gic1Sx1LSk9d6UxdHGTUDmpxITPdsd9R2cR9b5OJyLp934nlyedsv+F9NP728K\n3AW4vNTBxh1+8+78bMj62FpDmXLTYzLDmiezmo6TrjUEliQVy4t5aaMDiGBnfe4Tu53i/vXcu+Oo\nYWANkoT7nNSy9MVKItt9go/1lQiy2EJ4Av0REUB/DqFEHSufjf02TmfdD1/84en1TgQUsBy0G4Bv\n3t8/ueoVDWMCYE0V1WQfIIAQHRQYJckF66WhTc95eopa2oToTNrYL8+ts3o67CtL7zU6FKmSsL5k\nqGAuVqyrR0sBlZ5Yffukna6k6m4zobI5GPPigfKucq46fjLmwK9/cPH+h0MttkTRstkSh9u7OL96\norU6o3XZSQneoKfmTs8EGshM0Jp4EC3AaNqNpydq9g2T3AArlsc3kxlHj6UfHTUDLKCCJaccvFTU\npU83ekuCaU4CbTCruGk2S8roNltGlnPw/qNvNH3+L7615bP+/VaW7NEb7k97Ha52xRdbAT+0TBO5\nrWFUCGO7Ht2Stspbm+2+9aVC1bOXYesJRqf5MoeOr+at1Q0V3necFisTBzBhmnNktXmhUxaJfq1M\npBO7wx29LkkLg6WYS1T4sNsg9HR6G08xnb/bfb4sVDVmHO7vdPHiw0tnI+ASUOTjy2+7HaLa5MdW\nch1Qg45XPsOk5elcA5I7gjR6f7x98+1UMjvDrM5znmosGDqnAVWlRrWYqciadaEhhMb+YaP0HPua\nhV60ycSZkHFguXN9OMf7s75Cf9d9sJS+z7lmd3Z1vnMJqkttTss0ga1CrPtO66ELQEascrCGU2Pt\n16fzbubj/cmkbMDD0caxGw0VVjwWlEzvMxBGukjYwFColdUkmx/K/eT9qCVAVeND53nq7yNxOj7F\n7XH3bP+g8+P+/MObYI2lFi/uZsVcrYeVmtPIRLpB1m6HYpM4tXMkH0VTtUbVmuoizN3ZoU0L6U6k\naPRipaNqluLTUs2NDnRMLEsQNQ2oy2LjMFz08YhDW3nz+3lJeJohasn5fFnOjMbx26vndlyuNg/7\nZ7tv3g/5B1Bdj0AxMyFSxCqchTXAM6PdwfU45rPpz9YxFu1BgdV0mDUvTwlrCmElICuAuaXSvKvL\nFFHTO7rFKhxWmoJA32XgjKsOKxPjR2dzjpmyKsLf76ouK1DGef44336XvT+8/YHvjxdXwzQvqW5w\nQ9bmWo0107Vmgg1Ue7x+qMUNklzZSWRjC9C0dRKJ+VkuQF0DB5prgL0J8KEgwc4SfTEtQsZ69Rqc\nyTGr5t6j6cEA+FkRKqwz0W1Tlw3n2a2byx/Oz+f9ha6w3FxS5erDfJgr+3YbX4sk4+qRa/os0dsE\nDkDREl/Xj26QU/UxiG1lDY5n3UkqanGUpIJwZ0aWEqdaBhNL54CAUGQ131iCaJGIwPeeCLt/PfWa\nCxBk8a689SfHqRTJltdfeL0YHtiHTrfHt/anO216BemFIE1BFj4ylGx+Laxa1CbQbjePZW0EBR6n\n8VUOWcvFkrlUIsMcgHs3KJUl5zlASKU3gqjzLPPMrvz/AL8n+AMGG/NIAAAAAElFTkSuQmCC\n", "text/plain": [ "" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 输入是一个batch,batch_size=1\n", "input = to_tensor(lena).unsqueeze(0) \n", "\n", "# 锐化卷积核\n", "kernel = t.ones(3, 3)/-9.\n", "kernel[1][1] = 1\n", "conv = nn.Conv2d(1, 1, (3, 3), 1, bias=False)\n", "conv.weight.data = kernel.view(1, 1, 3, 3)\n", "\n", "out = conv(V(input))\n", "to_pil(out.data.squeeze(0))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "除了上述的使用,图像的卷积操作还有各种变体,具体可以参照此处动图[^2]介绍。\n", "[^2]: https://github.com/vdumoulin/conv_arithmetic/blob/master/README.md" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "池化层可以看作是一种特殊的卷积层,用来下采样。但池化层没有可学习参数,其weight是固定的。" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "[]" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pool = nn.AvgPool2d(2,2)\n", "list(pool.parameters())" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAGQAAABkCAAAAABVicqIAAAZt0lEQVR4nEV6Sa9lWXbW96299znn\nNq9/8eK96DIiMiOzsquiqih32FbJNjAwMp5YDJAQA5gwYMIY8Q8QI4QsMfCkQJjGSBjbYKrswmVX\n48rKqsiK7CIjMzKjyXgRr73NafZeazG4YXMHV1dXV3efs9c63/qazd+hmrnUlfRPPmyb6KSHOkYJ\nhCGAhIkL3IsQEPeQaTQ6SumGCzvj0FFE4UIyvT6xb789rkg4hRpd8952LDQ3GAgdvBQxguJ0E4Bu\nQndXKpywoIRR3d0AOBzeZQ3B4AQc8O68CdfuqIoQhNEZ5eP/LTCQjMFVzeEW6Eo3pxlIVwNAd4cT\nBvD58iwghXG+zB5iTFFIV9UHHa69nIu7uxkIxsd/+UTcCUgFLYV0Ak44ACfhpACkgySFBBhIgrBE\nCCXYIpeYggQJIBznh1p9eVrUGSgMVfrwRz3EhUIR10IhXGy1isEdbu5mZg5XLcXVzZ5/62YwBuNQ\n3Al3GAE6Hsx994tubu4IKXx4xxBFQISKXiBW1I0AHCDgZgDc3DzQclZTs1KccCfdQRdhKPZ8a0gR\non2cw+u7xYXu8Hd+ko0iJKUKdAfIQDzfLJiv3kAXV4O7GszcHauqA4CCsFLMnQ6nUISfner6L4Sc\nPQR5/x0PiRARiWJFCTMtjmIA4OrmMHNzBWHmAlgpZv5X9wkDGGmmpZibwwnAkD8ruHazL8rw7k8s\n1cQgAGPuB2B1eU7CHSThBqGbwBxOCVKPa+SianCKgAZzumkp6mYGdS1FcXhk1VeanP3Ddy1Fc/VI\nD9qWMHGKCAxiXD2CIEgQEHNvQorjtRh9dvy0j4xiBqdT3RwFoBsBteKGcn9jevFvvJU+umMhktkl\nepB2kddIAOZugAACQFzECHf1OGlymFZMqpsXvnD0wcID4W5mcJA0JWEGV1PQnz66WX/l8Tt3LET0\nJm7CShfHbRSSsOKqVuhm7mqmbjpaH6nYktL3Q6uBiFd++WbOpZhbMTUA4l7MSynmDnfaRwvbvHG7\nMImbEEEqdiePlSRI6qBqMFMzdbhbiuXs3HVou0E9u9TBLcc3fyEscimrX0KEbsVZ1AARhtDd0/yd\nXhKZDYUiKc8fn8FBqJqRgTA3e35VcrToYVB61/aMLE4Cvv1r1+bzvpg5nRKCl76jAAQgkOrp4Te+\nFYUoknJqTqO2x08DSHeHggI6VkAlxrXT0EWJFqPTy9COEqMIneFn198/HkNpihDE2zZILMEBoVDC\nf/y2S+2DN7kaH31JutkjE7oLV93kTkJWkMXSZqHCtMDc1VOguufM8cbX/+nmbMjFCCejHz+ZQ8xJ\nAowP/0IZvViTsT5c+jVZPjkVAQCYklSBrOCREHlmVQwciguAwd1d2Lb11lrQNvxGbrWompsztPee\ndgICAKT907lX0bUaOEm7X09y9FAC4e5cbakJnAYQkHkWUtUDA2NMpiUvziYHI1VV7a7++rwr5k7C\nQ3z62bnTAYD8/nlsAkrtNpqs/czmXB71CWbucDoYAszoJN0l5ySiRauYWNWelXaEa+MM6xXU/LWv\nLOeDEoCQ5elxK3CA8cHtIYp3Il7JxmtbeV2OAoTmK4A0dcIFDoCxje6Oqo6Mbjm7leP9XZXS5oCS\nfci/9Q+sVQdIQTi7f5IJY+z+LLMWQ9WH0Nzc9kktCGAQ+F/Du0AdTht3fQkwYxUnElBKWcqNDe3m\nAxJDCKq5fPXvzAZzdzcJ5eGDucMpbz2NyV3rPkJe3S+xgUTA1QG4rHrLhUJovroQFEqMSNq753m5\ncZ2tZaMIDKpa3L/+8lIdBBCku/upX6iqB7dDim7Bg7QvvqjV2CzC3N2wGigr8Aows9FahkQTiZUK\nbV62Xlpf1oOOaijEjBAUn/76vzWYQAh4++Wf1+B/lFOCGDUtrn3JwzoU0Shubi50z6pOhxmsvLB0\nklECS65k1t46sCVmo9oLkZkiPUez5a2rnQvdNeeL/+xa2Ujpq+8DyLSo06+O8oYoXEIIMThEBHC6\niASBBLkyz0DxkPo26qx/Y7/vB29ywJB7iZ7V3IHBfrOYCLVdfulfbaXpMOteA0xtgPibl/KEBY4Y\nGGzFSUIIIQkoCOK+cVe8b2LoMJbWX2nmIg206dmHZjV3hiqItFe//qmKLvt/9IvdXmpzTfEgnYcw\n339Nm8rcTWIkSMAhIhSQpAJo4jms9t7q2GfexJlMKg3ehflIvE2s7FlT04uWW3JY8sVfqO2CtRYT\nngZjzMHwlVFuHA5oJJwxEMJAQuA0V+S9ZaxtYGPWYrRjg05cBmvkdGreSq77oZEhGGTe7nxev7je\nVql3CUJ8EoMVIN26lms6aO7RxUhIEAbCYeJG47CPJngIEi3KLpc6jn46TUuMVHqJaR7qXPmQgyvT\njY3B15FdJAO85y4ews4blAR3AJRVC7sI3OkON7i7bVq1NSIy6Ttl6JswW45tHpKxYOj7yL4u/aIt\ny/PMYJNmKArmZ6onH6k5U76+YzWdpDBEA9zwHCJDoBgd7rGMB+37SZF9xfl2MJS5r4k6tIvdEOrY\n21APZ9wWroUBdAl8tMy7d5ZJMmz/utVhxYDEBWYGEZiDWJE0QEuwECduJewN+WQqfWZYxq6YLdT6\nbFh2XTssumYD44m2mUIJZ3P73L5lUSHTL048Pcd2IAJwrgaKCAADYdARYzVMtd4puthOfW7CMM1h\n2YdUMDIbMHeWZttHsYNIiEDEY7RVfj+pRN+7bBVdIeZ0RAQLFR0EaWpOC07VRkbZh/PNnu2W52UK\n3drJPI/RPHh51Fr081TZfkgj7z2JRELi4/nH7a+8PZ+6hHAzWsKqwHCPhAiFpK9IIyMImFW1+2i6\nqBfbyF0Vl9PTpcw+ib8aP+05OZ63p7+1m9eawoAQIE4++cH78rP7/7pyrcrBQRmJwelwh0cGZRCh\nr+iiwwkynI/XHh55jic7krtKuumJ8lTH7U/LXYyPj5b1L1bVtmW6B7rTP/7Jw3byxmv/+agi2L8Y\nWIEQd9AdMdCxamenkwBocD68ao7JcL7R5LZiHs365mg4PGIt69XD2KW/d2F7VFYFZAjPvvlA1196\n+cq/eye51rhwYGti9NUyQIwWXOFGBwS++hCqD3/OpmepH0+GRc1uzeejD878tH01/vhKvxxP/v7a\njaqrYSJCefb9ki7v7k42f+entRQP/kqDaiWnnE5oJFalAAA+ZxMu6XAZd8oT27TjqffVUSg/HvJZ\nmeCnlzbvLa//avMF9KF3VuDwR+9eDa9MQ9r43e/VwS3m8bUyJVasAnSnQOCqIEnC3GHmCCn95Thv\nnm2U0zXkCt2dDy2fH7TbH9ni2YWtV5o3ksEHNfHhm2+txXRqo/jb366iwMkrwRvQSQGFhAtAz8VW\n9JPqBCkMox893YqjPNSBkhfHqbYn07K2QN2Wza/pFzBkU42hWP3iVjqrS/Po39xJDX1gGF9aVgG2\nQi2u/s8dRd2dpJg7aABjCnfXGilLWXgoTx4/Y197XR/1i2U6Tde75bzVIrlpmnjl4OH5o/Tw956w\nkugabHvDR6u5T4qDQDTQzZ9rLAmBhIOCVJrj4zSe7/jJe6VdxGo9VzvLvLy+d+HF7TQsK5lN1mu6\nVLu4f0VtISKIA51XpYGt1K07CK56YIUqTllpGglgqI54Nw9lenR4cshzOVPWn+6Nlld/7o0X1Nla\nnFejSKma6e54ftqzDQHimb61z/V+MSjcV8V3xOdiF6S4CBnopru5VCcnfZObMHt8JKVJ/XG8VC8P\nL/7yzv6isHg8aeokZJDqwkY8fnRx1IYQe1T5YkrNoB2CiAhgDkSaqa/AHohR3MnQeUS5vTHb5Ozz\nuS+3BZK6Nuver0z2xv6sPu+HuhIBhQjrm+OzY6laETdBvOmTyGzmvTMKHHBRd/pKgtLMAQqliJZ4\n99laHD4405RO9vqSJs/KO39rcqFjM1qcaaoDFKrqmOxs+LknSvAcyuUpa0McjaoE7xftYCvShRXZ\nptMNgMGJOLl4Wnl3No8NuT0bwrDWnH0p3BgVVuGHRzmmaEPRUooitT4fNkYSMoVXvKK7e4ixChTP\nXTcIKEJ3CCBOPrc9KHuPv/Pu1c/e3z6/1PGsjzb1buP6a+MU0tGH8vhP/uTjbH3RIfcP/u+dTJ1t\nmlgBmt0ycTXN2Uxi3VT0vIgQFDOHUSCkUJ3McmBbX2refhz7gwcb7eLq/TIft796c7r2+e6d++P5\nfp795fx6rdFP3/l4GO31pVsrSU2420h0d3cXuAvpVjxCXYsDwkJRNQNM93clxpe+e7Z76eHupxfq\n46MXP5odf/XC1pqsf+etrhvGy1dv2dl0Mw0//uGztokqQ53YO7p9GeGvjB+H0dSDSclD3/cuAMyM\nrqp99Utus/vfPLyK97Yjzprp6fyirL10TRr84M8fd+36P3zj3sz7IpPv/Pfjr+0vFy6tjFjo9UWZ\nrNwrN4eV0ucua5Tn0oESwBiTQIZP3n6h+7OPFtcfjXzcIT/bap9mvLGVN+Wdp5vdK4drr974b7mt\nMT7/5JLvlwcF9VBPjyHlwlpY2XzidDOzknNYjwLj6mGkhCqm6B/Wr+1t/8F7wxufhcmmbI/bpSYf\nbly7tDF6+7MLVz6uD26V+Y1ZKRg/2j4Y7d0elVK1o1BUeJE1zCzEImZqxYpc3JxGYS6ucEpY0fl6\n82+W9/7Hh5cO8mwzdKN0cFeyZHn9cpr88N5eHV/uZw8fHGwuo+ey+dLR/NvnFx+59BueEdIVrc0d\n5q4oWhTbu5OIqHnQbjAAJM1M9JfO3rLDapHSeFZdPmtKVMDeXN/bvPPWpTSOLFuxlqFin/Noq9Cn\nZ4dZS6UObcaB7kYoYGZFru5FN8S+5FzM3Vdz3sLoWG6+55f2lt2z6Xa7OU6jZZHJrUtrh7//QnBU\nVaXitqwzhzytm36MgRFt0sKyLw1tZd+alcIbF2N2SBxU1QCsTFYbb6TZu/l8I47C+6n7+ODz+38X\nd5nfbK7w309qumkI0X1gMil91CBMwYJ0ZZyTHkizwlt392JXL4iTIcZetbj+tVfZmS42h5g63svR\n20fBhhvvaX312vQ/Pd0W0TIMManmYq6pD/Qg6NRkOWx5hc2qHoDVvmjZ2A9GSZFSDJodZmqOsp4n\nm0PTcNmnUZ1KHpW1gK+Ul3nzs2+NVAHzoR9US26z6mI1r+dDRFc2abtNWk0mN9MSrtTurBpxEYFp\nWTE903AQn85jN9u/FLaK9WF/49YXPjrxm5ftt3Ws5qamXTsUy6VY7svI3MupV8j9BP1+qNQcQsLN\ndjeBMKqEjKLQXIwUF+IsxYHzfmqdLwvH0r4of3i//xr3/svjagoz10KziL5vp46+1AO0PWIwnU/H\nw34IK7YAQsNBUtbRXCgxIIRIOMiwvbnQszBZhlMs0uVY9/bO4s78hd945fjtIDUFBjM3Vc8hS2kX\nCyvl81agPK+m9Vak+8pNdlufOFMwBynFJEw3K8DBcH5oGaJnBz7YxY1uI9298nloXhvv/oejxLpx\nw8pYRxjz84fHz54eQ/P9EIKHZ5hcraK5O9yKuW9GxIqMgYju4Np6glLgEFRnk/eufpImJV18e/Pm\nN+flZ1548uQHT3YW3nSBZhqc4jJhNfKXmlPPj+eMI4tHGO8h+mrw0SxMIUkcdGdUh3otDoEQLhvJ\nbj57utPw2cEtf7LV/e3LH8jZW2Ote4+BJIskA9jUV2VjyFruiddgPOH0gothZbO7pwohAjQhpZRs\nNCcdhNr2mV98OpvY2cmhXpl+uvvqHu6v/25US6W3Qrq5dkOfi4zWpxGDHZ4giJt1w2QUVvGNO1BS\nkooQJ4HYe4IbVpGBXzgZD+0MMmUbPnj5k6u/sjf6X1e/W3aO2HgfBgvuMLQxINq9zT4fV+8jMCCF\nTseBgDudbu5BquCEiBhiF2hBJES66w4wzQ+YlmuT+fn2R9M3dl76r/X8ETon2MfOUChV+zFGacij\n9aPR2uFhyBPz3CyXG2klpUAjLKVqlWuBiO6G+rkkmsqnrx01dnxhujbH7lE8aGbv9y98o+4KWeoe\nbe+BUo2vt5amu83avVK++dwVrTmsizgQFE53r+tgJEl3jwYHi7sANe5c70/X8tHENssz3cV+Xr/7\nxW+UlDTCUyFaqyhDrG3w7lQeI75/FCEamBCtdkDcxRVummpxhJXvLaqZKQ8GsJ/p9snTyaxqZ7vl\nVldtPpylV/7wSEiKqPS2OGv7Yrkg1qGcnDEevueeaktCRpUVewugmxVEShDkghCjq21Mq+JGMMoi\nh3ZedViOm6tPriy4+cfvBY0O62NcSjmlJINJasIk5dLd7hliseRCKVUWN3cC7mpB4FATMXhUQXye\nEsJ4NgwTicPoyafcGOzgo+99ly7RgQxK9uX5gaYoMU5iQpr1jxgTiLoUrW1VCwfdzDGVohBR9ZUV\n1fVR6CRR/HG3E8UWG90DuTisjX80CBgYNGgQZ5h/0mFtSg6JinyoqYYMIZTARNeOTndTc5UpDUIt\nunz0bjTqIi/pRmh/dX+4fxJtUHnx0+Vivb5664+fhroyybTciOfdzk9N1WKLUnIfCDoE0cJQ94cY\nVSGKA6WMpmpQL7MP/uynOY5sWc7rAhjzC/hztjtDHreznc8/fFMe8s5v/v7DldUGL8G69ZsXmhiZ\nxBZhfOmNr/zFu52WBgayXvbaUUJT0a3sVYPSyuz27x1Od+OLz8RmYaFuYfvsT6/9qC95uPn4J1++\nsHfw9nhjsnhz5oWRFrwkJ8ZBjVEHQzUNzauvPf7Wd/vkRel13kDfd+UkBpEexYRY3P6D053tKK+9\n+cJOPmvdGbsnr2CG8wuzyd7RvZTfG9fVXvfuGzY44JG0EQfNRnMUDXWSnO3yP/4X13KXc9YQJuPJ\nxtak6s+O/eTYIDZ75/e6g61KRC6/ernJIFGGW5MPWJqtn+kX23gwPJg16xvv7Z9d7EMIZEJskjto\nEsUQk5eSi9oX/uXrx+1gfZIU63q0vr65NlmvdkISe+cbabder1zacOHqhoNRiKvDrF6z248+Xdgn\nG58EHHeyV23vDwSDSKimG22UXCVzpkjt+5yL2cY/v7woLoEEQ6iayfoat2+E4Cc/ibvV+hoauX02\nPthxUoLIk/vLbCpPLobDnXf3hiXbxSXIhmT3bOjL2lYuOg2gC91yybmYmq3/k9jO4io9EYYQm2pn\nR6D3n1ySeqOwkbuP02hzcILkbAJMFirrOc7Gi63dxdEQ5YdXNsoqEdOmztbQnUKaWlY3czO7+hv9\nslFx54pG+ORGQ+8+CKGZdotaZDjxSgZzgvHMY8zFZNo2C+SFTM9SOMVOTrES7zk2VpQiMMOKqKkX\nB81+bbqYiJCAmaluv3y9oj0+HFU1lpEmed7DzQgKtp5K3Wrejb1NZs/aeT337mDz8XZyDSqy1kyj\nhEjY6uXFDOpwXf/FYbpip2ZZ5fWDStB+kL2ue68B8aVKTSXgXM5GS+fowmOItnX7VDYD49cflLhk\ngK5dGpc6mAhBNxhUV6cC3PTnMQGeRyQ63kmRevgkMea+EluKF/WmUtgwFK63CKPRR1ZTg9R867o8\nu/tELl/ZmjplfSQ5peAUFjM1MyvqbmauN/Yah6k7xTkJg7P7RELCoFGHHNl1UgU3JobQimEQPQl9\n0+a1tjmDjn94ht0tn7cbGzG0kxEDVla1azDVnMzEvf5iQ0AKAMZQEO34qdGHeQyd1ZL6VsPIzOHe\nNMhpV72lIAi26iiPN65vyfGDjwefjmGn0yoF+f/muxVdlUd/Pg4ZTCII1fLusejpcsg400lXAoWl\nLXFi5gb6JKTR8ZAYG60mZyW++8A+uHXNtM9DvTbivNuIpJDigLuWbJrNQPjr9SefHrUmkupp+Pgk\n5aO81DxMy0CcRw1DaoKaC2QuQealif30LGtMa1e/9/7W+r33XW2YbK6LP2w2CIJ0WRkC9JxKcAc3\nl59xNJ42TTNtTrHN88PT2fR0HYM058eRuQ+jaTEHuajJISVVbXKVy4CffX/r5MfI1DDamsrZJ19e\n+SYAXElQNRR1DQLuNjIsHyNMb1za7zbKxx8/XluMxi3q0vayfREyHtFhDPMyWBWHQvdaHffvb7+G\nqh3ENK5vQG/vXgiEm4is4lq3XLwUNXV/Ja5vraXlw9vf276yWZ/d+RTFtvrBYp5bvInG0ubCKWIh\n5BS6yioVM4nxXvXK1INaqbY2Jrjtr1R4Hh2YuCghtCGYqSgOxmiE9JNh09e6H3zf6nJxOJdJyqkW\n3Zo6xoQLWZaiQwjW18MFGdfV+vFn3QMPLtPNtdn3+9enz8WaQ1Zqqrjr0Gd1eLrWeYjTzYOd5Zb9\nn//Z17YTTrWedpareLo960OgEc68LA0hEevtxixXEm3rrpbA0MyP/GB/ugrYAIqYKE3MSvBStAT3\nl36aYgrc5nD0p+/mTVyYHM0m69QZLJ5+jn40CgoRyUdBK0kopR9d/+kLH758PBw8jaD0zeXtSYo0\nBTU4QVAIh7karAQVv+a91FrX5Xb/sKm78ejkzHYm2g+5j+Gsks3dJyTQPwvihI6fpb7diqPp07WH\nEzWm7YOtKiXh6riB0whBMBe4uWaREIKtb8zAOoHzfjOcjMZHC9najOenfTEp7XJ77WrtbmaTPohI\nZfUkVecv3dteVJ/bFcXW/kZKFDcJukq8HCRDogQHTM2sQG7kkoMEidOdkCZ9CaPLo+XJaQl7Yrq0\nAiGI/txNQtXkF2e1HY5Hm/1MH9ubO9tVCBJAcSdWh8ugcEEg6aZqpup2qxmGITTV5ELs8snQVDsb\n/uzzBXYP/h8CwMEc4WwTQAAAAABJRU5ErkJggg==\n", "text/plain": [ "" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "out = pool(V(input))\n", "to_pil(out.data.squeeze(0))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "除了卷积层和池化层,深度学习中还将常用到以下几个层:\n", "- Linear:全连接层。\n", "- BatchNorm:批规范化层,分为1D、2D和3D。除了标准的BatchNorm之外,还有在风格迁移中常用到的InstanceNorm层。\n", "- Dropout:dropout层,用来防止过拟合,同样分为1D、2D和3D。\n", "下面通过例子来说明它们的使用。" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "Variable containing:\n", " 0.0529 0.4152 0.6688 0.4281\n", " 0.4504 -0.3291 0.4206 1.4391\n", "[torch.FloatTensor of size 2x4]" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 输入 batch_size=2,维度3\n", "input = V(t.randn(2, 3))\n", "linear = nn.Linear(3, 4)\n", "h = linear(input)\n", "h" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "(Variable containing:\n", " 1.00000e-07 *\n", " 0.0000\n", " 0.0000\n", " 0.0000\n", " 2.3842\n", " [torch.FloatTensor of size 4], Variable containing:\n", " 15.9960\n", " 15.9988\n", " 15.9896\n", " 15.9994\n", " [torch.FloatTensor of size 4])" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 4 channel,初始化标准差为4,均值为0\n", "bn = nn.BatchNorm1d(4)\n", "bn.weight.data = t.ones(4) * 4\n", "bn.bias.data = t.zeros(4)\n", "\n", "bn_out = bn(h)\n", "# 注意输出的均值和方差\n", "# 方差是标准差的平方,计算无偏方差分母会减1\n", "# 使用unbiased=False 分母不减1\n", "bn_out.mean(0), bn_out.var(0, unbiased=False)" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Variable containing:\n", "-7.9990 0.0000 7.9974 -7.9998\n", " 0.0000 -0.0000 -0.0000 7.9998\n", "[torch.FloatTensor of size 2x4]" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 每个元素以0.5的概率舍弃\n", "dropout = nn.Dropout(0.5)\n", "o = dropout(bn_out)\n", "o # 有一半左右的数变为0" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "以上很多例子中都对module的属性直接操作,其大多数是可学习参数,一般会随着学习的进行而不断改变。实际使用中除非需要使用特殊的初始化,应尽量不要直接修改这些参数。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 4.1.2 激活函数\n", "PyTorch实现了常见的激活函数,其具体的接口信息可参见官方文档[^3],这些激活函数可作为独立的layer使用。这里将介绍最常用的激活函数ReLU,其数学表达式为:\n", "$$ReLU(x)=max(0,x)$$\n", "[^3]: http://pytorch.org/docs/nn.html#non-linear-activations" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Variable containing:\n", "-0.6869 0.7347 -0.2196\n", " 0.9445 -0.9042 -1.2652\n", "[torch.FloatTensor of size 2x3]\n", "\n", "Variable containing:\n", " 0.0000 0.7347 0.0000\n", " 0.9445 0.0000 0.0000\n", "[torch.FloatTensor of size 2x3]\n", "\n" ] } ], "source": [ "relu = nn.ReLU(inplace=True)\n", "input = V(t.randn(2, 3))\n", "print(input)\n", "output = relu(input)\n", "print(output) # 小于0的都被截断为0\n", "# 等价于input.clamp(min=0)" ] }, { "cell_type": "raw", "metadata": {}, "source": [ "ReLU函数有个inplace参数,如果设为True,它会把输出直接覆盖到输入中,这样可以节省内存/显存。之所以可以覆盖是因为在计算ReLU的反向传播时,只需根据输出就能够推算出反向传播的梯度。但是只有少数的autograd操作支持inplace操作(如variable.sigmoid_()),除非你明确地知道自己在做什么,否则一般不要使用inplace操作。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "在以上的例子中,基本上都是将每一层的输出直接作为下一层的输入,这种网络称为前馈传播网络(feedforward neural network)。对于此类网络如果每次都写复杂的forward函数会有些麻烦,在此就有两种简化方式,ModuleList和Sequential。其中Sequential是一个特殊的module,它包含几个子Module,前向传播时会将输入一层接一层的传递下去。ModuleList也是一个特殊的module,可以包含几个子module,可以像用list一样使用它,但不能直接把输入传给ModuleList。下面举例说明。" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "scrolled": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "net1: Sequential(\n", " (conv): Conv2d (3, 3, kernel_size=(3, 3), stride=(1, 1))\n", " (batchnorm): BatchNorm2d(3, eps=1e-05, momentum=0.1, affine=True)\n", " (activation_layer): ReLU()\n", ")\n", "net2: Sequential(\n", " (0): Conv2d (3, 3, kernel_size=(3, 3), stride=(1, 1))\n", " (1): BatchNorm2d(3, eps=1e-05, momentum=0.1, affine=True)\n", " (2): ReLU()\n", ")\n", "net3: Sequential(\n", " (conv1): Conv2d (3, 3, kernel_size=(3, 3), stride=(1, 1))\n", " (bn1): BatchNorm2d(3, eps=1e-05, momentum=0.1, affine=True)\n", " (relu1): ReLU()\n", ")\n" ] } ], "source": [ "# Sequential的三种写法\n", "net1 = nn.Sequential()\n", "net1.add_module('conv', nn.Conv2d(3, 3, 3))\n", "net1.add_module('batchnorm', nn.BatchNorm2d(3))\n", "net1.add_module('activation_layer', nn.ReLU())\n", "\n", "net2 = nn.Sequential(\n", " nn.Conv2d(3, 3, 3),\n", " nn.BatchNorm2d(3),\n", " nn.ReLU()\n", " )\n", "\n", "from collections import OrderedDict\n", "net3= nn.Sequential(OrderedDict([\n", " ('conv1', nn.Conv2d(3, 3, 3)),\n", " ('bn1', nn.BatchNorm2d(3)),\n", " ('relu1', nn.ReLU())\n", " ]))\n", "print('net1:', net1)\n", "print('net2:', net2)\n", "print('net3:', net3)" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(Conv2d (3, 3, kernel_size=(3, 3), stride=(1, 1)),\n", " Conv2d (3, 3, kernel_size=(3, 3), stride=(1, 1)),\n", " Conv2d (3, 3, kernel_size=(3, 3), stride=(1, 1)))" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 可根据名字或序号取出子module\n", "net1.conv, net2[0], net3.conv1" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": [ "input = V(t.rand(1, 3, 4, 4))\n", "output = net1(input)\n", "output = net2(input)\n", "output = net3(input)\n", "output = net3.relu1(net1.batchnorm(net1.conv(input)))" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "modellist = nn.ModuleList([nn.Linear(3,4), nn.ReLU(), nn.Linear(4,2)])\n", "input = V(t.randn(1, 3))\n", "for model in modellist:\n", " input = model(input)\n", "# 下面会报错,因为modellist没有实现forward方法\n", "# output = modelist(input)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "看到这里,读者可能会问,为何不直接使用Python中自带的list,而非要多此一举呢?这是因为`ModuleList`是`Module`的子类,当在`Module`中使用它的时候,就能自动识别为子module。\n", "\n", "下面举例说明。" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "MyModule(\n", " (module_list): ModuleList(\n", " (0): Conv2d (3, 3, kernel_size=(3, 3), stride=(1, 1))\n", " (1): ReLU()\n", " )\n", ")" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "class MyModule(nn.Module):\n", " def __init__(self):\n", " super(MyModule, self).__init__()\n", " self.list = [nn.Linear(3, 4), nn.ReLU()]\n", " self.module_list = nn.ModuleList([nn.Conv2d(3, 3, 3), nn.ReLU()])\n", " def forward(self):\n", " pass\n", "model = MyModule()\n", "model" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "module_list.0.weight torch.Size([3, 3, 3, 3])\n", "module_list.0.bias torch.Size([3])\n" ] } ], "source": [ "for name, param in model.named_parameters():\n", " print(name, param.size())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "可见,list中的子module并不能被主module所识别,而ModuleList中的子module能够被主module所识别。这意味着如果用list保存子module,将无法调整其参数,因其未加入到主module的参数中。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "除ModuleList之外还有ParameterList,其是一个可以包含多个parameter的类list对象。在实际应用中,使用方式与ModuleList类似。如果在构造函数`__init__`中用到list、tuple、dict等对象时,一定要思考是否应该用ModuleList或ParameterList代替。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 4.1.3 循环神经网络层(RNN)\n", "近些年随着深度学习和自然语言处理的结合加深,RNN的使用也越来越多,关于RNN的基础知识,推荐阅读colah的文章[^4]入门。PyTorch中实现了如今最常用的三种RNN:RNN(vanilla RNN)、LSTM和GRU。此外还有对应的三种RNNCell。\n", "\n", "RNN和RNNCell层的区别在于前者一次能够处理整个序列,而后者一次只处理序列中一个时间点的数据,前者封装更完备更易于使用,后者更具灵活性。实际上RNN层的一种后端实现方式就是调用RNNCell来实现的。\n", "[^4]: http://colah.github.io/posts/2015-08-Understanding-LSTMs/" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "Variable containing:\n", "(0 ,.,.) = \n", " 0.0545 -0.0061 0.5615\n", " -0.1251 0.4490 0.2640\n", " 0.1405 -0.1624 0.0303\n", "\n", "(1 ,.,.) = \n", " 0.0168 0.1562 0.5002\n", " 0.0824 0.1454 0.4007\n", " 0.0180 -0.0267 0.0094\n", "[torch.FloatTensor of size 2x3x3]" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "t.manual_seed(1000)\n", "# 输入:batch_size=3,序列长度都为2,序列中每个元素占4维\n", "input = V(t.randn(2, 3, 4))\n", "# lstm输入向量4维,隐藏元3,1层\n", "lstm = nn.LSTM(4, 3, 1)\n", "# 初始状态:1层,batch_size=3,3个隐藏元\n", "h0 = V(t.randn(1, 3, 3))\n", "c0 = V(t.randn(1, 3, 3))\n", "out, hn = lstm(input, (h0, c0))\n", "out" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "Variable containing:\n", "(0 ,.,.) = \n", " 0.0545 -0.0061 0.5615\n", " -0.1251 0.4490 0.2640\n", " 0.1405 -0.1624 0.0303\n", "\n", "(1 ,.,.) = \n", " 0.0168 0.1562 0.5002\n", " 0.0824 0.1454 0.4007\n", " 0.0180 -0.0267 0.0094\n", "[torch.FloatTensor of size 2x3x3]" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "t.manual_seed(1000)\n", "input = V(t.randn(2, 3, 4))\n", "# 一个LSTMCell对应的层数只能是一层\n", "lstm = nn.LSTMCell(4, 3)\n", "hx = V(t.randn(3, 3))\n", "cx = V(t.randn(3, 3))\n", "out = []\n", "for i_ in input:\n", " hx, cx=lstm(i_, (hx, cx))\n", " out.append(hx)\n", "t.stack(out)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "词向量在自然语言中应用十分普及,PyTorch同样提供了Embedding层。" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [], "source": [ "# 有4个词,每个词用5维的向量表示\n", "embedding = nn.Embedding(4, 5)\n", "# 可以用预训练好的词向量初始化embedding\n", "embedding.weight.data = t.arange(0,20).view(4,5)" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "scrolled": false }, "outputs": [ { "data": { "text/plain": [ "Variable containing:\n", " 15 16 17 18 19\n", " 10 11 12 13 14\n", " 5 6 7 8 9\n", "[torch.FloatTensor of size 3x5]" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "input = V(t.arange(3, 0, -1)).long()\n", "output = embedding(input)\n", "output" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 4.1.4 损失函数\n", "在深度学习中要用到各种各样的损失函数(loss function),这些损失函数可看作是一种特殊的layer,PyTorch也将这些损失函数实现为`nn.Module`的子类。然而在实际使用中通常将这些loss function专门提取出来,和主模型互相独立。详细的loss使用请参照文档[^5],这里以分类中最常用的交叉熵损失CrossEntropyloss为例说明。\n", "[^5]: http://pytorch.org/docs/nn.html#loss-functions" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Variable containing:\n", " 1.5544\n", "[torch.FloatTensor of size 1]" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# batch_size=3,计算对应每个类别的分数(只有两个类别)\n", "score = V(t.randn(3, 2))\n", "# 三个样本分别属于1,0,1类,label必须是LongTensor\n", "label = V(t.Tensor([1, 0, 1])).long()\n", "\n", "# loss与普通的layer无差异\n", "criterion = nn.CrossEntropyLoss()\n", "loss = criterion(score, label)\n", "loss" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4.2 优化器\n", "\n", "PyTorch将深度学习中常用的优化方法全部封装在`torch.optim`中,其设计十分灵活,能够很方便的扩展成自定义的优化方法。\n", "\n", "所有的优化方法都是继承基类`optim.Optimizer`,并实现了自己的优化步骤。下面就以最基本的优化方法——随机梯度下降法(SGD)举例说明。这里需重点掌握:\n", "\n", "- 优化方法的基本使用方法\n", "- 如何对模型的不同部分设置不同的学习率\n", "- 如何调整学习率" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [], "source": [ "# 首先定义一个LeNet网络\n", "class Net(nn.Module):\n", " def __init__(self):\n", " super(Net, self).__init__()\n", " self.features = nn.Sequential(\n", " nn.Conv2d(3, 6, 5),\n", " nn.ReLU(),\n", " nn.MaxPool2d(2,2),\n", " nn.Conv2d(6, 16, 5),\n", " nn.ReLU(),\n", " nn.MaxPool2d(2,2)\n", " )\n", " self.classifier = nn.Sequential(\n", " nn.Linear(16 * 5 * 5, 120),\n", " nn.ReLU(),\n", " nn.Linear(120, 84),\n", " nn.ReLU(),\n", " nn.Linear(84, 10)\n", " )\n", "\n", " def forward(self, x):\n", " x = self.features(x)\n", " x = x.view(-1, 16 * 5 * 5)\n", " x = self.classifier(x)\n", " return x\n", "\n", "net = Net()" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "scrolled": true }, "outputs": [], "source": [ "from torch import optim\n", "optimizer = optim.SGD(params=net.parameters(), lr=1)\n", "optimizer.zero_grad() # 梯度清零,等价于net.zero_grad()\n", "\n", "input = V(t.randn(1, 3, 32, 32))\n", "output = net(input)\n", "output.backward(output) # fake backward\n", "\n", "optimizer.step() # 执行优化" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [], "source": [ "# 为不同子网络设置不同的学习率,在finetune中经常用到\n", "# 如果对某个参数不指定学习率,就使用最外层的默认学习率\n", "optimizer =optim.SGD([\n", " {'params': net.features.parameters()}, # 学习率为1e-5\n", " {'params': net.classifier.parameters(), 'lr': 1e-2}\n", " ], lr=1e-5)" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [], "source": [ "# 只为两个全连接层设置较大的学习率,其余层的学习率较小\n", "special_layers = nn.ModuleList([net.classifier[0], net.classifier[2]])\n", "special_layers_params = list(map(id, special_layers.parameters()))\n", "base_params = filter(lambda p: id(p) not in special_layers_params,\n", " net.parameters())\n", "\n", "optimizer = t.optim.SGD([\n", " {'params': base_params},\n", " {'params': special_layers.parameters(), 'lr': 0.01}\n", " ], lr=0.001 )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "对于如何调整学习率,主要有两种做法。一种是修改optimizer.param_groups中对应的学习率,另一种是更简单也是较为推荐的做法——新建优化器,由于optimizer十分轻量级,构建开销很小,故而可以构建新的optimizer。但是后者对于使用动量的优化器(如Adam),会丢失动量等状态信息,可能会造成损失函数的收敛出现震荡等情况。" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [], "source": [ "# 调整学习率,新建一个optimizer\n", "old_lr = 0.1\n", "optimizer =optim.SGD([\n", " {'params': net.features.parameters()},\n", " {'params': net.classifier.parameters(), 'lr': old_lr*0.1}\n", " ], lr=1e-5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4.3 nn.functional\n", "\n", "nn中还有一个很常用的模块:`nn.functional`,nn中的大多数layer,在`functional`中都有一个与之相对应的函数。`nn.functional`中的函数和`nn.Module`的主要区别在于,用nn.Module实现的layers是一个特殊的类,都是由`class layer(nn.Module)`定义,会自动提取可学习的参数。而`nn.functional`中的函数更像是纯函数,由`def function(input)`定义。下面举例说明functional的使用,并指出二者的不同之处。" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Variable containing:\n", " 1 1 1 1\n", " 1 1 1 1\n", "[torch.ByteTensor of size 2x4]" ] }, "execution_count": 31, "metadata": {}, "output_type": "execute_result" } ], "source": [ "input = V(t.randn(2, 3))\n", "model = nn.Linear(3, 4)\n", "output1 = model(input)\n", "output2 = nn.functional.linear(input, model.weight, model.bias)\n", "output1 == output2" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Variable containing:\n", " 1 1 1\n", " 1 1 1\n", "[torch.ByteTensor of size 2x3]" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "b = nn.functional.relu(input)\n", "b2 = nn.ReLU()(input)\n", "b == b2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "此时读者可能会问,应该什么时候使用nn.Module,什么时候使用nn.functional呢?答案很简单,如果模型有可学习的参数,最好用nn.Module,否则既可以使用nn.functional也可以使用nn.Module,二者在性能上没有太大差异,具体的使用取决于个人的喜好。如激活函数(ReLU、sigmoid、tanh),池化(MaxPool)等层由于没有可学习参数,则可以使用对应的functional函数代替,而对于卷积、全连接等具有可学习参数的网络建议使用nn.Module。下面举例说明,如何在模型中搭配使用nn.Module和nn.functional。另外虽然dropout操作也没有可学习操作,但建议还是使用`nn.Dropout`而不是`nn.functional.dropout`,因为dropout在训练和测试两个阶段的行为有所差别,使用`nn.Module`对象能够通过`model.eval`操作加以区分。" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [], "source": [ "from torch.nn import functional as F\n", "class Net(nn.Module):\n", " def __init__(self):\n", " super(Net, self).__init__()\n", " self.conv1 = nn.Conv2d(3, 6, 5)\n", " self.conv2 = nn.Conv2d(6, 16, 5)\n", " self.fc1 = nn.Linear(16 * 5 * 5, 120)\n", " self.fc2 = nn.Linear(120, 84)\n", " self.fc3 = nn.Linear(84, 10)\n", "\n", " def forward(self, x):\n", " x = F.pool(F.relu(self.conv1(x)), 2)\n", " x = F.pool(F.relu(self.conv2(x)), 2)\n", " x = x.view(-1, 16 * 5 * 5)\n", " x = F.relu(self.fc1(x))\n", " x = F.relu(self.fc2(x))\n", " x = self.fc3(x)\n", " return x" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "对于不具备可学习参数的层(激活层、池化层等),将它们用函数代替,这样则可以不用放置在构造函数`__init__`中。对于有可学习参数的模块,也可以用functional来代替,只不过实现起来较为繁琐,需要手动定义参数parameter,如前面实现自定义的全连接层,就可将weight和bias两个参数单独拿出来,在构造函数中初始化为parameter。" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [], "source": [ "class MyLinear(nn.Module):\n", " def __init__(self):\n", " super(MyLinear, self).__init__()\n", " self.weight = nn.Parameter(t.randn(3, 4))\n", " self.bias = nn.Parameter(t.zeros(3))\n", " def forward(self,input):\n", " return F.linear(input, self.weight, self.bias)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "关于nn.functional的设计初衷,以及它和nn.Module更多的比较说明,可参看论坛的讨论和作者说明[^6]。\n", "[^6]: https://discuss.pytorch.org/search?q=nn.functional" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4.4 初始化策略\n", "在深度学习中参数的初始化十分重要,良好的初始化能让模型更快收敛,并达到更高水平,而糟糕的初始化则可能使得模型迅速瘫痪。PyTorch中nn.Module的模块参数都采取了较为合理的初始化策略,因此一般不用我们考虑,当然我们也可以用自定义初始化去代替系统的默认初始化。而当我们在使用Parameter时,自定义初始化则尤为重要,因t.Tensor()返回的是内存中的随机数,很可能会有极大值,这在实际训练网络中会造成溢出或者梯度消失。PyTorch中`nn.init`模块就是专门为初始化而设计,如果某种初始化策略`nn.init`不提供,用户也可以自己直接初始化。" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Parameter containing:\n", " 0.3535 0.1427 0.0330\n", " 0.3321 -0.2416 -0.0888\n", "-0.8140 0.2040 -0.5493\n", "-0.3010 -0.4769 -0.0311\n", "[torch.FloatTensor of size 4x3]" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 利用nn.init初始化\n", "from torch.nn import init\n", "linear = nn.Linear(3, 4)\n", "\n", "t.manual_seed(1)\n", "# 等价于 linear.weight.data.normal_(0, std)\n", "init.xavier_normal(linear.weight)" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "\n", " 0.3535 0.1427 0.0330\n", " 0.3321 -0.2416 -0.0888\n", "-0.8140 0.2040 -0.5493\n", "-0.3010 -0.4769 -0.0311\n", "[torch.FloatTensor of size 4x3]" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 直接初始化\n", "import math\n", "t.manual_seed(1)\n", "\n", "# xavier初始化的计算公式\n", "std = math.sqrt(2)/math.sqrt(7.)\n", "linear.weight.data.normal_(0,std)" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [], "source": [ "# 对模型的所有参数进行初始化\n", "for name, params in net.named_parameters():\n", " if name.find('linear') != -1:\n", " # init linear\n", " params[0] # weight\n", " params[1] # bias\n", " elif name.find('conv') != -1:\n", " pass\n", " elif name.find('norm') != -1:\n", " pass" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4.5 nn.Module深入分析\n", "\n", "如果想要更深入地理解nn.Module,究其原理是很有必要的。首先来看看nn.Module基类的构造函数:\n", "```python\n", "def __init__(self):\n", " self._parameters = OrderedDict()\n", " self._modules = OrderedDict()\n", " self._buffers = OrderedDict()\n", " self._backward_hooks = OrderedDict()\n", " self._forward_hooks = OrderedDict()\n", " self.training = True\n", "```\n", "其中每个属性的解释如下:\n", "\n", "- `_parameters`:字典,保存用户直接设置的parameter,`self.param1 = nn.Parameter(t.randn(3, 3))`会被检测到,在字典中加入一个key为'param',value为对应parameter的item。而self.submodule = nn.Linear(3, 4)中的parameter则不会存于此。\n", "- `_modules`:子module,通过`self.submodel = nn.Linear(3, 4)`指定的子module会保存于此。\n", "- `_buffers`:缓存。如batchnorm使用momentum机制,每次前向传播需用到上一次前向传播的结果。\n", "- `_backward_hooks`与`_forward_hooks`:钩子技术,用来提取中间变量,类似variable的hook。\n", "- `training`:BatchNorm与Dropout层在训练阶段和测试阶段中采取的策略不同,通过判断training值来决定前向传播策略。\n", "\n", "上述几个属性中,`_parameters`、`_modules`和`_buffers`这三个字典中的键值,都可以通过`self.key`方式获得,效果等价于`self._parameters['key']`.\n", "\n", "下面举例说明。" ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Net(\n", " (submodel1): Linear(in_features=3, out_features=4)\n", ")" ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "class Net(nn.Module):\n", " def __init__(self):\n", " super(Net, self).__init__()\n", " # 等价与self.register_parameter('param1' ,nn.Parameter(t.randn(3, 3)))\n", " self.param1 = nn.Parameter(t.rand(3, 3))\n", " self.submodel1 = nn.Linear(3, 4) \n", " def forward(self, input):\n", " x = self.param1.mm(input)\n", " x = self.submodel1(x)\n", " return x\n", "net = Net()\n", "net" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "OrderedDict([('submodel1', Linear(in_features=3, out_features=4))])" ] }, "execution_count": 39, "metadata": {}, "output_type": "execute_result" } ], "source": [ "net._modules" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "OrderedDict([('param1', Parameter containing:\n", " 0.3398 0.5239 0.7981\n", " 0.7718 0.0112 0.8100\n", " 0.6397 0.9743 0.8300\n", " [torch.FloatTensor of size 3x3])])" ] }, "execution_count": 40, "metadata": {}, "output_type": "execute_result" } ], "source": [ "net._parameters" ] }, { "cell_type": "code", "execution_count": 41, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "Parameter containing:\n", " 0.3398 0.5239 0.7981\n", " 0.7718 0.0112 0.8100\n", " 0.6397 0.9743 0.8300\n", "[torch.FloatTensor of size 3x3]" ] }, "execution_count": 41, "metadata": {}, "output_type": "execute_result" } ], "source": [ "net.param1 # 等价于net._parameters['param1']" ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "param1 torch.Size([3, 3])\n", "submodel1.weight torch.Size([4, 3])\n", "submodel1.bias torch.Size([4])\n" ] } ], "source": [ "for name, param in net.named_parameters():\n", " print(name, param.size())" ] }, { "cell_type": "code", "execution_count": 43, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " Net(\n", " (submodel1): Linear(in_features=3, out_features=4)\n", ")\n", "submodel1 Linear(in_features=3, out_features=4)\n" ] } ], "source": [ "for name, submodel in net.named_modules():\n", " print(name, submodel)" ] }, { "cell_type": "code", "execution_count": 44, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "OrderedDict([('running_mean', \n", " 1.00000e-02 *\n", " 5.1362\n", " 7.4864\n", " [torch.FloatTensor of size 2]), ('running_var', \n", " 0.9116\n", " 0.9068\n", " [torch.FloatTensor of size 2])])" ] }, "execution_count": 44, "metadata": {}, "output_type": "execute_result" } ], "source": [ "bn = nn.BatchNorm1d(2)\n", "input = V(t.rand(3, 2), requires_grad=True)\n", "output = bn(input)\n", "bn._buffers" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "nn.Module在实际使用中可能层层嵌套,一个module包含若干个子module,每一个子module又包含了更多的子module。为方便用户访问各个子module,nn.Module实现了很多方法,如函数`children`可以查看直接子module,函数`module`可以查看所有的子module(包括当前module)。与之相对应的还有函数`named_childen`和`named_modules`,其能够在返回module列表的同时返回它们的名字。" ] }, { "cell_type": "code", "execution_count": 45, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Variable containing:\n", " 0 2 0 0\n", " 8 0 12 14\n", " 16 0 0 22\n", "[torch.FloatTensor of size 3x4]" ] }, "execution_count": 45, "metadata": {}, "output_type": "execute_result" } ], "source": [ "input = V(t.arange(0, 12).view(3, 4))\n", "model = nn.Dropout()\n", "# 在训练阶段,会有一半左右的数被随机置为0\n", "model(input)" ] }, { "cell_type": "code", "execution_count": 46, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Variable containing:\n", " 0 1 2 3\n", " 4 5 6 7\n", " 8 9 10 11\n", "[torch.FloatTensor of size 3x4]" ] }, "execution_count": 46, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.training = False\n", "# 在测试阶段,dropout什么都不做\n", "model(input)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "对于batchnorm、dropout、instancenorm等在训练和测试阶段行为差距巨大的层,如果在测试时不将其training值设为True,则可能会有很大影响,这在实际使用中要千万注意。虽然可通过直接设置`training`属性,来将子module设为train和eval模式,但这种方式较为繁琐,因如果一个模型具有多个dropout层,就需要为每个dropout层指定training属性。更为推荐的做法是调用`model.train()`函数,它会将当前module及其子module中的所有training属性都设为True,相应的,`model.eval()`函数会把training属性都设为False。" ] }, { "cell_type": "code", "execution_count": 47, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "True True\n" ] }, { "data": { "text/plain": [ "(False, False)" ] }, "execution_count": 47, "metadata": {}, "output_type": "execute_result" } ], "source": [ "print(net.training, net.submodel1.training)\n", "net.eval()\n", "net.training, net.submodel1.training" ] }, { "cell_type": "code", "execution_count": 48, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[('', Net(\n", " (submodel1): Linear(in_features=3, out_features=4)\n", " )), ('submodel1', Linear(in_features=3, out_features=4))]" ] }, "execution_count": 48, "metadata": {}, "output_type": "execute_result" } ], "source": [ "list(net.named_modules())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`register_forward_hook`与`register_backward_hook`,这两个函数的功能类似于variable函数的`register_hook`,可在module前向传播或反向传播时注册钩子。每次前向传播执行结束后会执行钩子函数(hook)。前向传播的钩子函数具有如下形式:`hook(module, input, output) -> None`,而反向传播则具有如下形式:`hook(module, grad_input, grad_output) -> Tensor or None`。钩子函数不应修改输入和输出,并且在使用后应及时删除,以避免每次都运行钩子增加运行负载。钩子函数主要用在获取某些中间结果的情景,如中间某一层的输出或某一层的梯度。这些结果本应写在forward函数中,但如果在forward函数中专门加上这些处理,可能会使处理逻辑比较复杂,这时候使用钩子技术就更合适一些。下面考虑一种场景,有一个预训练好的模型,需要提取模型的某一层(不是最后一层)的输出作为特征进行分类,但又不希望修改其原有的模型定义文件,这时就可以利用钩子函数。下面给出实现的伪代码。\n", "```python\n", "model = VGG()\n", "features = t.Tensor()\n", "def hook(module, input, output):\n", " '''把这层的输出拷贝到features中'''\n", " features.copy_(output.data)\n", " \n", "handle = model.layer8.register_forward_hook(hook)\n", "_ = model(input)\n", "# 用完hook后删除\n", "handle.remove()\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`nn.Module`对象在构造函数中的行为看起来有些怪异,如果想要真正掌握其原理,就需要看两个魔法方法`__getattr__`和`__setattr__`。在Python中有两个常用的buildin方法`getattr`和`setattr`,`getattr(obj, 'attr1')`等价于`obj.attr`,如果`getattr`函数无法找到所需属性,Python会转而调用`obj.__getattr__('attr1')`方法,即`getattr`函数无法找到的交给`__getattr__`函数处理,没有实现`__getattr__`或者`__getattr__`也无法处理的就会raise AttributeError。`setattr(obj, 'name', value)`等价于`obj.name=value`,如果obj对象实现了`__setattr__`方法,setattr会直接调用`obj.__setattr__('name', value)`,否则调用buildin方法。总结一下:\n", "- result = obj.name会调用buildin函数`getattr(obj, 'name')`,如果该属性找不到,会调用`obj.__getattr__('name')`\n", "- obj.name = value会调用buildin函数`setattr(obj, 'name', value)`,如果obj对象实现了`__setattr__`方法,`setattr`会直接调用`obj.__setattr__('name', value')`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "nn.Module实现了自定义的`__setattr__`函数,当执行`module.name=value`时,会在`__setattr__`中判断value是否为`Parameter`或`nn.Module`对象,如果是则将这些对象加到`_parameters`和`_modules`两个字典中,而如果是其它类型的对象,如`Variable`、`list`、`dict`等,则调用默认的操作,将这个值保存在`__dict__`中。" ] }, { "cell_type": "code", "execution_count": 49, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "OrderedDict([('param', Parameter containing:\n", " 1 1\n", " 1 1\n", " [torch.FloatTensor of size 2x2])])" ] }, "execution_count": 49, "metadata": {}, "output_type": "execute_result" } ], "source": [ "module = nn.Module()\n", "module.param = nn.Parameter(t.ones(2, 2))\n", "module._parameters" ] }, { "cell_type": "code", "execution_count": 50, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "_modules: OrderedDict()\n", "__dict__['submodules']: [Linear(in_features=2, out_features=2), Linear(in_features=2, out_features=2)]\n" ] } ], "source": [ "submodule1 = nn.Linear(2, 2)\n", "submodule2 = nn.Linear(2, 2)\n", "module_list = [submodule1, submodule2]\n", "# 对于list对象,调用buildin函数,保存在__dict__中\n", "module.submodules = module_list\n", "print('_modules: ', module._modules)\n", "print(\"__dict__['submodules']:\",module.__dict__.get('submodules'))" ] }, { "cell_type": "code", "execution_count": 51, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "ModuleList is instance of nn.Module: True\n", "_modules: OrderedDict([('submodules', ModuleList(\n", " (0): Linear(in_features=2, out_features=2)\n", " (1): Linear(in_features=2, out_features=2)\n", "))])\n", "__dict__['submodules']: None\n" ] } ], "source": [ "module_list = nn.ModuleList(module_list)\n", "module.submodules = module_list\n", "print('ModuleList is instance of nn.Module: ', isinstance(module_list, nn.Module))\n", "print('_modules: ', module._modules)\n", "print(\"__dict__['submodules']:\", module.__dict__.get('submodules'))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "因`_modules`和`_parameters`中的item未保存在`__dict__`中,所以默认的getattr方法无法获取它,因而`nn.Module`实现了自定义的`__getattr__`方法,如果默认的`getattr`无法处理,就调用自定义的`__getattr__`方法,尝试从`_modules`、`_parameters`和`_buffers`这三个字典中获取。" ] }, { "cell_type": "code", "execution_count": 52, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 52, "metadata": {}, "output_type": "execute_result" } ], "source": [ "getattr(module, 'training') # 等价于module.training\n", "# error\n", "# module.__getattr__('training')" ] }, { "cell_type": "code", "execution_count": 53, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "2" ] }, "execution_count": 53, "metadata": {}, "output_type": "execute_result" } ], "source": [ "module.attr1 = 2\n", "getattr(module, 'attr1')\n", "# 报错\n", "# module.__getattr__('attr1')" ] }, { "cell_type": "code", "execution_count": 54, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Parameter containing:\n", " 1 1\n", " 1 1\n", "[torch.FloatTensor of size 2x2]" ] }, "execution_count": 54, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 即module.param, 会调用module.__getattr__('param')\n", "getattr(module, 'param')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "在PyTorch中保存模型十分简单,所有的Module对象都具有state_dict()函数,返回当前Module所有的状态数据。将这些状态数据保存后,下次使用模型时即可利用`model.load_state_dict()`函数将状态加载进来。优化器(optimizer)也有类似的机制,不过一般并不需要保存优化器的运行状态。" ] }, { "cell_type": "code", "execution_count": 55, "metadata": {}, "outputs": [], "source": [ "# 保存模型\n", "t.save(net.state_dict(), 'net.pth')\n", "\n", "# 加载已保存的模型\n", "net2 = Net()\n", "net2.load_state_dict(t.load('net.pth'))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "实际上还有另外一种保存方法,但因其严重依赖模型定义方式及文件路径结构等,很容易出问题,因而不建议使用。" ] }, { "cell_type": "code", "execution_count": 56, "metadata": { "scrolled": false }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/usr/local/lib/python3.5/dist-packages/torch/serialization.py:158: UserWarning: Couldn't retrieve source code for container of type Net. It won't be checked for correctness upon loading.\n", " \"type \" + obj.__name__ + \". It won't be checked \"\n" ] }, { "data": { "text/plain": [ "Net(\n", " (submodel1): Linear(in_features=3, out_features=4)\n", ")" ] }, "execution_count": 56, "metadata": {}, "output_type": "execute_result" } ], "source": [ "t.save(net, 'net_all.pth')\n", "net2 = t.load('net_all.pth')\n", "net2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "将Module放在GPU上运行也十分简单,只需两步:\n", "- model = model.cuda():将模型的所有参数转存到GPU\n", "- input.cuda():将输入数据也放置到GPU上\n", "\n", "至于如何在多个GPU上并行计算,PyTorch也提供了两个函数,可实现简单高效的并行GPU计算\n", "- nn.parallel.data_parallel(module, inputs, device_ids=None, output_device=None, dim=0, module_kwargs=None)\n", "- class torch.nn.DataParallel(module, device_ids=None, output_device=None, dim=0)\n", "\n", "可见二者的参数十分相似,通过`device_ids`参数可以指定在哪些GPU上进行优化,output_device指定输出到哪个GPU上。唯一的不同就在于前者直接利用多GPU并行计算得出结果,而后者则返回一个新的module,能够自动在多GPU上进行并行加速。\n", "\n", "```\n", "# method 1\n", "new_net = nn.DataParallel(net, device_ids=[0, 1])\n", "output = new_net(input)\n", "\n", "# method 2\n", "output = nn.parallel.data_parallel(new_net, input, device_ids=[0, 1])\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "DataParallel并行的方式,是将输入一个batch的数据均分成多份,分别送到对应的GPU进行计算,各个GPU得到的梯度累加。与Module相关的所有数据也都会以浅复制的方式复制多份,在此需要注意,在module中属性应该是只读的。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4.6 nn和autograd的关系\n", "nn.Module利用的也是autograd技术,其主要工作是实现前向传播。在forward函数中,nn.Module对输入的Variable进行的各种操作,本质上都是用到了autograd技术。这里需要对比autograd.Function和nn.Module之间的区别:\n", "- autograd.Function利用了Tensor对autograd技术的扩展,为autograd实现了新的运算op,不仅要实现前向传播还要手动实现反向传播\n", "- nn.Module利用了autograd技术,对nn的功能进行扩展,实现了深度学习中更多的层。只需实现前向传播功能,autograd即会自动实现反向传播\n", "- nn.functional是一些autograd操作的集合,是经过封装的函数\n", "\n", "作为两大类扩充PyTorch接口的方法,我们在实际使用中应该如何选择呢?如果某一个操作,在autograd中尚未支持,那么只能实现Function接口对应的前向传播和反向传播。如果某些时候利用autograd接口比较复杂,则可以利用Function将多个操作聚合,实现优化,正如第三章所实现的`Sigmoid`一样,比直接利用autograd低级别的操作要快。而如果只是想在深度学习中增加某一层,使用nn.Module进行封装则更为简单高效。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4.7 小试牛刀:搭建ResNet\n", "Kaiming He的深度残差网络(ResNet)[^7]在深度学习的发展中起到了很重要的作用,ResNet不仅一举拿下了当年CV下多个比赛项目的冠军,更重要的是这一结构解决了训练极深网络时的梯度消失问题。\n", "\n", "首先来看看ResNet的网络结构,这里选取的是ResNet的一个变种:ResNet34。ResNet的网络结构如图4-2所示,可见除了最开始的卷积池化和最后的池化全连接之外,网络中有很多结构相似的单元,这些重复单元的共同点就是有个跨层直连的shortcut。ResNet中将一个跨层直连的单元称为Residual block,其结构如图4-3所示,左边部分是普通的卷积网络结构,右边是直连,但如果输入和输出的通道数不一致,或其步长不为1,那么就需要有一个专门的单元将二者转成一致,使其可以相加。\n", "\n", "另外我们可以发现Residual block的大小也是有规律的,在最开始的pool之后有连续的几个一模一样的Residual block单元,这些单元的通道数一样,在这里我们将这几个拥有多个Residual block单元的结构称之为layer,注意和之前讲的layer区分开来,这里的layer是几个层的集合。\n", "\n", "考虑到Residual block和layer出现了多次,我们可以把它们实现为一个子Module或函数。这里我们将Residual block实现为一个子moduke,而将layer实现为一个函数。下面是实现代码,规律总结如下:\n", "\n", "- 对于模型中的重复部分,实现为子module或用函数生成相应的module`make_layer`\n", "- nn.Module和nn.Functional结合使用\n", "- 尽量使用`nn.Seqential`\n", "\n", "![图4-2: ResNet34网络结构](imgs/resnet1.png)\n", "![图4-3: Residual block 结构图](imgs/residual.png)\n", " [^7]: He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016: 770-778." ] }, { "cell_type": "code", "execution_count": 57, "metadata": {}, "outputs": [], "source": [ "from torch import nn\n", "import torch as t\n", "from torch.nn import functional as F" ] }, { "cell_type": "code", "execution_count": 58, "metadata": {}, "outputs": [], "source": [ "class ResidualBlock(nn.Module):\n", " '''\n", " 实现子module: Residual Block\n", " '''\n", " def __init__(self, inchannel, outchannel, stride=1, shortcut=None):\n", " super(ResidualBlock, self).__init__()\n", " self.left = nn.Sequential(\n", " nn.Conv2d(inchannel,outchannel,3,stride, 1,bias=False),\n", " nn.BatchNorm2d(outchannel),\n", " nn.ReLU(inplace=True),\n", " nn.Conv2d(outchannel,outchannel,3,1,1,bias=False),\n", " nn.BatchNorm2d(outchannel) )\n", " self.right = shortcut\n", "\n", " def forward(self, x):\n", " out = self.left(x)\n", " residual = x if self.right is None else self.right(x)\n", " out += residual\n", " return F.relu(out)\n", "\n", "class ResNet(nn.Module):\n", " '''\n", " 实现主module:ResNet34\n", " ResNet34 包含多个layer,每个layer又包含多个residual block\n", " 用子module来实现residual block,用_make_layer函数来实现layer\n", " '''\n", " def __init__(self, num_classes=1000):\n", " super(ResNet, self).__init__()\n", " # 前几层图像转换\n", " self.pre = nn.Sequential(\n", " nn.Conv2d(3, 64, 7, 2, 3, bias=False),\n", " nn.BatchNorm2d(64),\n", " nn.ReLU(inplace=True),\n", " nn.MaxPool2d(3, 2, 1))\n", " \n", " # 重复的layer,分别有3,4,6,3个residual block\n", " self.layer1 = self._make_layer( 64, 64, 3)\n", " self.layer2 = self._make_layer( 64, 128, 4, stride=2)\n", " self.layer3 = self._make_layer( 128, 256, 6, stride=2)\n", " self.layer4 = self._make_layer( 256, 512, 3, stride=2)\n", "\n", " #分类用的全连接\n", " self.fc = nn.Linear(512, num_classes)\n", " \n", " def _make_layer(self, inchannel, outchannel, block_num, stride=1):\n", " '''\n", " 构建layer,包含多个residual block\n", " '''\n", " shortcut = nn.Sequential(\n", " nn.Conv2d(inchannel,outchannel,1,stride, bias=False),\n", " nn.BatchNorm2d(outchannel))\n", " \n", " layers = []\n", " layers.append(ResidualBlock(inchannel, outchannel, stride, shortcut))\n", " \n", " for i in range(1, block_num):\n", " layers.append(ResidualBlock(outchannel, outchannel))\n", " return nn.Sequential(*layers)\n", " \n", " def forward(self, x):\n", " x = self.pre(x)\n", " \n", " x = self.layer1(x)\n", " x = self.layer2(x)\n", " x = self.layer3(x)\n", " x = self.layer4(x)\n", "\n", " x = F.avg_pool2d(x, 7)\n", " x = x.view(x.size(0), -1)\n", " return self.fc(x)" ] }, { "cell_type": "code", "execution_count": 59, "metadata": {}, "outputs": [], "source": [ "model = ResNet()\n", "input = t.autograd.Variable(t.randn(1, 3, 224, 224))\n", "o = model(input)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "感兴趣的读者可以尝试实现Google的Inception网络结构或ResNet的其它变体,看看如何能够简洁明了地实现它,实现代码尽量控制在80行以内(本例去掉空行和注释总共不超过50行)。另外,与PyTorch配套的图像工具包`torchvision`已经实现了深度学习中大多数经典的模型,其中就包括ResNet34,读者可以通过下面两行代码使用:\n", "```python\n", "from torchvision import models\n", "model = models.resnet34()\n", "```\n", "本例中ResNet34的实现就是参考了torchvision中的实现并做了简化,感兴趣的读者可以阅读相应的源码,比较这里的实现和torchvision中实现的不同。" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.9" } }, "nbformat": 4, "nbformat_minor": 2 }