machinelearning_notebook/6_pytorch/4-logistic-regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 逻辑回归的PyTorch实现"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "逻辑回归是一种广义的回归模型，其与多元线性回归有着很多相似之处，模型的形式基本相同，虽然也被称为回归，但是其更多的情况使用在分类问题上。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 模型形式\n",
    "\n",
    "逻辑回归的模型形式和线性回归一样，都是 $y = wx + b$，其中 $x$ 可以是一个多维的特征，唯一不同的地方在于逻辑斯蒂回归会对 $y$ 作用一个 logistic 函数，将其变为一种概率的结果。 \n",
    "\n",
    "$$\n",
    "h_\\theta(x) = g(\\theta^T x) = \\frac{1}{1+e^{-\\theta^T x}}\n",
    "$$\n",
    "\n",
    "Logistic 函数作为 Logistic 回归的核心，也被称为 Sigmoid 函数。Sigmoid 函数非常简单，其公式如下\n",
    "\n",
    "$$\n",
    "f(x) = \\frac{1}{1 + e^{-x}}\n",
    "$$\n",
    "\n",
    "![logistic function](imgs/logistic_function.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "可以看到 Sigmoid 函数的范围是在 0 ~ 1 之间，所以任何一个值经过了 Sigmoid 函数的作用，都会变成 0 ~ 1 之间的一个值，这个值可以形象地理解为一个概率，比如对于二分类问题，这个值越小就表示属于第一类，这个值越大就表示属于第二类。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 损失函数\n",
    "\n",
    "Logistic 回归使用了 Sigmoid 函数将结果变到 $0 \\sim 1$ 之间，对于任意输入一个数据，经过 Sigmoid 之后的结果我们记为 $\\hat{y}$，表示这个数据点属于第二类的概率，那么其属于第一类的概率就是 $1-\\hat{y}$。\n",
    "* 如果这个数据点属于第二类，我们希望 $\\hat{y}$ 越大越好，也就是越靠近 1 越好\n",
    "* 如果这个数据属于第一类，那么我们希望 $1-\\hat{y}$ 越大越好，也就是 $\\hat{y}$ 越小越好，越靠近 0 越好\n",
    "\n",
    "所以我们可以这样设计我们的 loss 函数\n",
    "\n",
    "$$\n",
    "loss = - \\left[ y * log(\\hat{y}) + (1 - y) * log(1 - \\hat{y}) \\right]\n",
    "$$\n",
    "\n",
    "其中 $y$ 表示真实的 label，只能取 {0, 1} 这两个值，因为 $\\hat{y}$ 表示经过 Logistic 回归预测之后的结果，是一个 $0 \\sim 1$ 之间的小数。\n",
    "\n",
    "* 如果 $y$ 是 0，表示该数据属于第一类，我们希望 $\\hat{y}$ 越小越好，上面的 loss 函数变为\n",
    "$$\n",
    "loss = - \\left[ log(1 - \\hat{y}) \\right]\n",
    "$$\n",
    "在训练模型的时候我们希望最小化 loss 函数，根据 log 函数的单调性，也就是最小化 $\\hat{y}$，与我们的要求是一致的。\n",
    "\n",
    "* 而如果 $y$ 是 1，表示该数据属于第二类，我们希望 $\\hat{y}$ 越大越好，同时上面的 loss 函数变为\n",
    "$$\n",
    "loss = - \\left[ log(\\hat{y}) \\right]\n",
    "$$\n",
    "希望最小化 loss 函数也就是最大化 $\\hat{y}$，这也与要求一致。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 程序示例\n",
    "\n",
    "下面通过例子来学习如何使用PyTorch实现 Logistic 回归"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x7f844c1603b0>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "# 设定随机种子\n",
    "torch.manual_seed(2021)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从 `data.txt` 读入数据，读入数据点之后我们根据不同的 label 将数据点分为了红色和蓝色，并且画图展示出来了"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f8332c92820>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAgF0lEQVR4nO3dfYxd9X3n8ffXT3GnOAHbk7R48HiSmgZDoMJjt1ltukhZFOcJl5YoBid0SVXL2SXKP0ugJaErIWtTVSulCWmRN0oDjBWEtklKtgQ3ArWRVpvEY2qDwcAaA2YgEsOMlIYnEZPv/nHu4Dt37sO5956H3++cz0u6Gt9zj+/93nPO/Z7f+T0dc3dERCR+y8oOQEREsqGELiJSEUroIiIVoYQuIlIRSugiIhWxoqwPXr9+vW/atKmsjxcRidLhw4dfcvfRdq+VltA3bdrE9PR0WR8vIhIlM3u202uqchERqQgldBGRilBCFxGpCCV0EZGKUEIXEamIngndzL5pZi+a2bEOr5uZfdXMTpjZw2Z2afZhZujAAdi0CZYtS/4eOFB2RCIimUhTQv8WsKPL6x8GNjcee4C/HT6snBw4AHv2wLPPgnvyd88eJXURqYSeCd3dfwTMd1llJ3CnJ34MnG1mv5lVgJm6+WZ49dXFy159NVkuIhK5LOrQNwDPNT2faSxbwsz2mNm0mU3Pzs5m8NF9OnWqv+UiIhHJIqFbm2Vt75rh7vvdfdLdJ0dH245czdfGjf0tz4iq7etJ+12KlkVCnwHOa3o+BryQwftmb98+GBlZvGxkJFmeE1Xb15P2u5Qhi4R+L3Bto7fL7wE/d/efZfC+2du9G/bvh/FxMEv+7t+fLM+Jqu3LU2YJWftdypCm2+K3gf8L/LaZzZjZn5jZXjPb21jlPuAkcAL4n8B/zi3aLOzeDc88A7/6VfI3x2QOqrYvUnMCX78ePvOZ8krIMex3VQlVkLuX8ti6davXwfi4e5JSFj/Gx8uOrFqmptxHRtpv6zK2+6D7fWoqWccs+Ts1lU987bbXyEh+nyfZAaa9Q17VSNGclVBtX0vtqjjaKaqEPMh+L7LeXVVC1VTdhB7I9WQJ1fa1lDZR59yh6S2D7Pcik2wMVULSP0tK8MWbnJz03G5wsVDUaf51jIwok1bYpk1Jibab0A+BZcuSknkrs6TJJ0udttf4eNK0JOEys8PuPtnutWqW0HU9WTvtqjhWroR16+K5MipymISqAqupmgld15O1066K4+/+Dl56qbAOTUMrMsmqKrCaqlnloutJidSBA8mF5KlTScl83z4lWVmsflUuup6USBU8TEIqppoJXdeTIlJD1UzooKKOBCuQHrVSQSvKDkCkTlp71C4MHgKVOWR41S2hiwRIPWolT0roIgWKqUetqobio4QuUqCS7rHSN83nHicldJECxdKjVlVDcVJCFylQLD1qY6oakjPUy0WkYLt3h5fAW61dC3Nz7ZdLuFRCF6kANWAKqIQuEr08+rbPz/e3XMKgErpIhJpL5H/8x9k3YMbSG0cWU0KXTOnSP3+tXQrffLP9esM0YMbSG0cWS5XQzWyHmT1hZifM7KY2r59jZt81s4fN7KdmdlH2oUro1He5GGnvnzpMaTqW3jiyWM+EbmbLga8DHwa2AFeb2ZaW1f4cOOLuFwPXAn+ddaDSnzJKyuq7XIw0Je8sStNVmN+ubleMaUro24ET7n7S3d8A7gZ2tqyzBXgAwN0fBzaZ2bsyjVRSK6ukrL7LxehU8l6+XKXpZnW8YkyT0DcAzzU9n2ksa3YU+EMAM9sOjANjrW9kZnvMbNrMpmdnZweLWHoqq6SshrRidKrfvuOOuEvTWavjFWOahG5tlrXet+7LwDlmdgT4HPCvwOkl/8l9v7tPuvvk6Ohov7FKSmWVlNWQVoxu9dt1q2Jo1vrd292FEip+xejuXR/A+4GDTc//DPizLusb8Azw9m7vu3XrVpfE1JT7+Li7WfJ3amq49xsfd08uMhc/xseHj7WXrL+LpDc15T4ysnifj4zUYx+0++5m5f0O8gRMe6f82+mFt1ZIBh+dBCaAVSTVKxe2rHM2sKrx7z8F7uz1vkroiTx+hHX+YddZmSfysnX67q1JPc/fQVGFmaESevL/+QjwJPAUcHNj2V5gr58pxf8/4HHgO8A5vd5TCT2R149QJeX66VQiNSs7svx1+u4Lv6W8fwdFFqK6JXRLXi/e5OSkT09Pl/LZIVm2LNn9rcySBi7p7MCBpIHr1Kmk4XXfvno3BnaqNx4fTxpKq6zs717k55vZYXefbPeaRoqWTD1DBlPHLmmdLDQGPvtsUhBoVpdG6bIb5EPpsquEXrKyD8RYhdglrYweJs0nNkhObgtJvU790cse2RpMwaxTXUzeD9Whn6H67v6FVl9cVkN0nRtCQ6I6dNWhyxDKrjNtVWQ8zW0HnX6+aoMpXlFtOqpDl8oJraqqqDrU1raDTtQGU7wQ5r5RQpcolV1n2qqoOtQ0My2qDaa+lNAlWiGUiBYUdcXQrcQfwolNyqVb0IlkYCGB5l2HunFjWG0HEhaV0EUyUsQVQ2htBxIWJXSRiITWdiBhUZWLSGR271YCl/ZUQhcRqQgldBGRilBCFxGpCCV0qbU637JNqkeNolJbC8PoF0ZeLkzBC2p0lDiphC61FeIUvCLDUEKX2grlpgQiWVFCl9oK5qYEIhlJldDNbIeZPWFmJ8zspjavv8PMvm9mR83sUTO7LvtQRbKlYfRSNT0TupktB74OfBjYAlxtZltaVvsvwGPufglwGfA/zGxVxrFWk7pZlEbD6KVq0pTQtwMn3P2ku78B3A3sbFnHgTVmZsBZwDxwOtNIq0h3Oi5dCFPw6pwuWUmT0DcAzzU9n2ksa3YbcAHwAvAI8Hl31w2welE3i9rTOV2ylCahW5tlrTe/+hBwBDgX+B3gNjN7+5I3MttjZtNmNj07O9tnqDkou2ikbhalK/sQ0DldspQmoc8A5zU9HyMpiTe7DvhO46bUJ4Cngfe2vpG773f3SXefHB0dHTTmbIRQNFI3i1KFcAjonC5ZSpPQDwGbzWyi0dC5C7i3ZZ1TwAcBzOxdwG8DJ7MMNHMhFI3UzaJUIRwCOqeHo+yrtSz0TOjufhq4HjgIHAfucfdHzWyvme1trHYr8O/M7BHgAeBGd38pr6AzEULRSN0sShXCIaBzehhCuFrLhLuX8ti6dauXanzcPdl3ix/j4+XGJYUJ5RCYmko+0yz5OzVV7OcPKta42wnlWEgDmPYOebW+I0XzLBpV4dqtBkIpHYfQdbJflSnRNoRwtZaJTpk+70fpJXT3fIoYU1PuIyOLT/MjI9EUX0IsdeUZU4jfNwYxlWjTiOn70KWEXu+EnoeYjowWIZ6LQoypagY5qZm1P8zN8o42HzEdZ0rorfIslkV8pId4LgoxpioZNJFVcb/EcrXWLaFb8nrxJicnfXp6uvgPbr2rASQVp1n1Ltm0KalQbDU+nlSQBmzZsuRn2cosqd8tQ4gxVcmgh2vePyPpzMwOu/tku9fq1yiad+fjUFraBhBin+gQY6qSQRsD1eM2TPVL6Hk3Z0d8pId4LgoxpirpdGJctqx3J60Ye+dUXqe6mLwfpdWhV7HyL0Mh1iOGGFNVtKtDb32E2jhYV6gOvYkq/0QWOXAgqXE8dSoplb/55tJ1ImgCqg3VoTdLUyWigUHR0K4aXnPVSaeG5ugG2NRU/UrovagEH412u2rVKlizBubnk/rhffu02/oRcSet2lAJvR8hTMEnqbTbVW+8AXNz1RiOXgY1Qucr7ytKJfRWlZnUofrS7BKdi/sTcSet4BUx/42qXFrpmjManXZVKw1CkhBklVpU5dKPrK851WqXm3a7qh0NQpIQFHHxr4TeKstrzqrNMRqY1l21bh2sXLl4HdX/SiiKGPWsKpc8qfqmcM19qtXLRUKSVQc6VbmURQ2shdNw9KVU6xeGIhqcldDzpJmlSle1ZNbv96larV/s+zP3AkenOQHyflT2BhfNYpo1P4XY5lSp2OYf6PtUaeqiqu3PQTHsDS6AHcATwAngpjav3wAcaTyOAW8Ca7u9Zy0Sunt8WbCDGH9MVUpm7oN9n4jvt7JE1fbnoIZK6MBy4Cng3cAq4Ciwpcv6Hwce7PW+tUnoFRHjj6lKycx9sO/Tab8t7LuQT8itQt6fvcptWZbruiX0NHXo24ET7n7S3d8A7gZ2dln/auDbqet8JAoxtu9WrQljkO/Tra9+bPXpoe7PXu0UhbZjdMr0Cw/gKuAbTc8/DdzWYd0RYJ4O1S3AHmAamN64cePgpygpXIwl9BiriboZ9PsslA67ldRjEOr+7PXbyPq3w5BVLp9ok9C/1mHdTwLf7/WeriqX6IT6Y+qlIk0Ybxnm+4RcZZFWiPuz13bNert3S+grUhTiZ4Dzmp6PAS90WHcXqm6ppIXuVbEN2tm9O/wY+zHM99m4sf04t7KrLPoR4v7stV2L3O5p6tAPAZvNbMLMVpEk7XtbVzKzdwD/AfiHbEOUUGjQTtw0NW4+em3XIrd7z4Tu7qeB64GDwHHgHnd/1Mz2mtneplWvBP7J3V/JPkwRGZamxs1Hr+1a5HbXXC4x0UQlwdCukLJ0m8slTR26hKB1Zp+Fvk+gTFIw7QoJleZyiYVujRcM7YowxD6vSx6U0GMR48ieitKuKF9eg3ViP0kooWehiKMg1GFyNaRdUb48rpKqMDOlEvqwijoKIu9zFnvJp1nku6IS8rhKqkRVWqcRR3k/KjNStMgx8SEOk0sh1lGm3US6Kyojj59dLCNp6TJSVN0Wh7VsWbLfW+lW82/Rnfgka1ndzq1ZLMepbkGXJ1Wo9qRGRMlaHoN1qlCVpoQ+rCocBTnTOU/ykPVUFFUYSauEPqwqHAX9GKB1U+c8iUXs8xVppGgWQpwCLg8DDpGMdaZGkdioUVTSi6XVSKTC1Cgq2VDrpkjQlNAlPbVuigRNCV3SU+umSNCU0CW9uvXoaVGl6QukOEUeN+rlIv1p7bKyMNFFxZO65kCXQRR93NS7hK4iV/+qMCXdACoxcZMUrujjpr4JvaaJqa1+Tmw1zWzq4CODKPq4qW9Cr2liWqLfE1tOR2joF0vq4CODKPq4SZXQzWyHmT1hZifM7KYO61xmZkfM7FEz+5dsw8yBilyJfk9sORyhMVwsqYOPDKLw46bTvLoLD2A58BTwbmAVcBTY0rLO2cBjwMbG83f2et/S50Mvch7zkPU7CXQOk5vHsis0B7oMIuvjhi7zoacpoW8HTrj7SXd/A7gb2NmyzjXAd9z9VOMk8eKQ55n8qciV6LfEnUPXxVgulmKfuEnKUeRxkyahbwCea3o+01jW7HzgHDP7ZzM7bGbXtnsjM9tjZtNmNj07OztYxFmpeZ/qtwxyYsv4CFX9tEg20iR0a7OsdUavFcBW4KPAh4Avmdn5S/6T+353n3T3ydHR0b6DzZyKXEGc2HSxJJKNNAOLZoDzmp6PAS+0Wecld38FeMXMfgRcAjyZSZSSr5Kn/9X0uiLZSFNCPwRsNrMJM1sF7ALubVnnH4APmNkKMxsBfhc4nm2oUmW6WKqP0LuoxqxnCd3dT5vZ9cBBkh4v33T3R81sb+P12939uJndDzwM/Ar4hrsfyzNwEYmPplDIV6p+6O5+n7uf7+7vcfd9jWW3u/vtTev8lbtvcfeL3P0rOcUroCKOBK/TIarxfPnS5FyxURFHAtftEI2li2qsdAu62Og2cBK4boco6PAdlm5BVyUq4kjguh2i6qKaLyX02GgUjgSu2yEawLCHSlNCj42KOBK4XoeouqjmRwk9NiriSOB0iJZHjaIiIhFRo6iISA0ooYuIVIQSuohIRSihi+RAszNIGZTQJT81zWox3CNVqkkJva7yTrahZLUSTiqagErKom6LddQ6exIkIz+y7CwcwpwzRXzPNpYtS85hrcySwTQiw+jWbVEJvY6KSLYhZLWSTiohnMukutQPXRYrYoKvEOacKWkiM83OIGVRQq+jIpJtCFmtpJOKhr5LWZTQ66iIZFt2VjtwAF5+eenygk4qmoBKyqCEXkfdkm2WvULKymoLjaFzc4uXr1unorJUWqqEbmY7zOwJMzthZje1ef0yM/u5mR1pPG7JPlRq2685F+2SbShdDYfVrt8gwFlnKZlLpfXs5WJmy4EngcuBGeAQcLW7P9a0zmXAf3X3j6X94L57uZTUBa1WqtI9I4QeNiI5GbaXy3bghLufdPc3gLuBnVkGmIpGa+SvU++Pdkk+ZCH0sBEpQZqEvgF4run5TGNZq/eb2VEz+4GZXdjujcxsj5lNm9n07Oxsf5HqXpr565TwzOKqdgmhh02NqWa0PGkSurVZ1no9+xAw7u6XAF8Dvtfujdx9v7tPuvvk6OhoX4Gq1FWAffuS5N3KPa4robJ72NRYVZphYpUmoc8A5zU9HwNeaF7B3f/N3V9u/Ps+YKWZrc8sSlCpqwi7d7eve4b4roTUb7AUqhktV5qEfgjYbGYTZrYK2AXc27yCmf2GWVK0M7PtjfedW/JOw1Cpqxjj4+2X60ooSkVXf6hmtFw9E7q7nwauBw4Cx4F73P1RM9trZnsbq10FHDOzo8BXgV2exyQxeZa6VPGX0JVQZZRR/aGa0ZK5eymPrVu3ejCmptxHRtyT4z55jIwky+toasp9fNzdLPn72c8ufl7X7dJJ6/YKZPuMjy8+pBce4+P5faZ+SvkDpr1DXlVCdy/nyI+FfqHdlbx9up1LzNof1mblxSTD65bQNX0uaCBKN1UZbJSXErdPr7F22nXVpOlze1HFX3sHDnQeVKRWrkSJrYC9epSoOWRwsTapKaGDjvx2Fop/ndT9ZLegxMJAr3NJHTqG5ZF4o+5L36kuJu9HUHXo7qr4a9WpXUF16IuVWIde96afvDZ96NsVNYpK3zq1qIGSeauSCgN5nktiKN/klXjLakxOq1tCV6OotKcWtSgcOJDUmZ86ldTy7Ns3fJVKLBOb5tWXIfRDv1qNollVmsXa6lEUtStEIY+xdrEM38+r+SLqQ79T0T3vx0BVLlldY6pvdToxXHdL5rKucsjrMKprlROVqUPvt9Ks014JvdVDpERZ/jzyLjs1/8TXrUseISbhLFUnofdTdOh2JIXe6iHhC7kIN6Qsk3BRZac6XXRXJ6H3c3R0W1cl9OooI7HWIHuk2qwpViqq7FSnn3R1Eno/P6RuR1INfpC1UNZ+rFP26CTlti9qU9Xpors6Cd09fYms15FU4Uvm2igrsdYpe3SSctsXdc6t0zm2Wgk9LZXCq6+sxBpK9iizUNLHti8izDr93OuZ0N1VCq+6shJrCNmj7BhCOak1qcvPvb4JXaqt6KQWUh+5shNq2SeUGuuW0FeUNqJJZFgLwyKzHvveTut4+Lm5ZPjgXXeVMx6+7Jt3Lnznz38+2RYAv/ZrxXy2dBTf0H+RZnneZ7ZZaOPhQ5nD/7XXzvx7bi6ieWarKajJuX75y18yMzPD66+/XkpMWVu9ejVjY2OsXLmy7FBkWKHd1SqEGbRCn8WqorpNzpWqysXMdgB/DSwHvuHuX+6w3jbgx8An3f1/9RvozMwMa9asYdOmTZhZv/89KO7O3NwcMzMzTExMlB2ODGvjxvbJq6wbfRRZ3dRJ2dU+skTPKhczWw58HfgwsAW42sy2dFjvL4GDgwbz+uuvs27duuiTOYCZsW7duspcbWQuttkuQ5yCr6jqpk5CqfaRt6SpQ98OnHD3k+7+BnA3sLPNep8D/h54cZiAqpDMF1Tpu2Qqxnt81eF+bv0K8SSXVmwFipTSJPQNwHNNz2cay95iZhuAK4Hbu72Rme0xs2kzm56dne03VqmK0BoY0yq7RByaWE9yMRYoUkqT0NsVM1tbh74C3Ojub3Z7I3ff7+6T7j45OjqaMsQuKnqWrTzVvVZHjCe5QQoUkeSaNAl9Bjiv6fkY8ELLOpPA3Wb2DHAV8Ddm9gdZBNhRyWfZO+64g82bN7N582buuOOOQj6zMlT3KmXqt0ARU4m+04ijhQdJT5iTwASwCjgKXNhl/W8BV/V633YjRR977LH0w6VKHCk3NzfnExMTPjc35/Pz8z4xMeHz8/Nt1+3rO9WFRhlKmfrNHWWPym1Bl5GiPUvo7n4auJ6k98px4B53f9TM9prZ3jxOMqnkcNl+6NAhLr74Yl5//XVeeeUVLrzwQo4dO7ZkvYMHD3L55Zezdu1azjnnHC6//HLuv//+gT+3dmKte5Vq6LcxN6IqwlT90N39PuC+lmVtG0Dd/T8NH1YKOfQL3rZtG1dccQVf/OIXee211/jUpz7FRRddtGS9559/nvPOO1MLNTY2xvPPPz/w59bS7t1K4FKOfvvwhzYGoYt4h/7n1GXqlltu4Yc//CHT09N84QtfaLuOtxkxqC6KIhHppzE3ou6Z8Sb0nC7b5+fnefnll/nFL37RcVDQ2NgYzz13pifnzMwM55577lCfKyKBiqiKMKi5XI4fP84FF1xQSjwLrrjiCnbt2sXTTz/Nz372M2677bYl68zPz7N161YeeughAC699FIOHz7M2rVrl6wbwncSkeroNpdLvCX0HNx5552sWLGCa665hptuuolDhw7x4IMPLllv7dq1fOlLX2Lbtm1s27aNW265pW0ylyaR9OMViZlK6Dmr4nfqWwgzA4pUhEroUq5Yh/qLREZ3LOrikUce4dOf/vSiZW9729v4yU9+UlJEkYqoH69IzJTQu3jf+97HkSNHyg4jfhH14xWJmapcJH8R9eMViZkSuuQvon68IjFTlYsUQ0P9RXIXdQldXZtFRM6INqGXPUXxjh07OPvss/nYxz5WzAeKiPQQbUIvu2vzDTfcwF133VXMh4mIpBBtQs+ja3Pa+dABPvjBD7JmzZrBP0xEJGPRNorm0bU57XzoIiIhiraEnlfX5jTzoYuIhCjahJ5X1+Y086GLBENdvaRJtFUukE/X5j179nDrrbfy9NNPc+ONN7adD10kCK2zWC509QL1+a+pVCV0M9thZk+Y2Qkzu6nN6zvN7GEzO2Jm02b277MPNX9p50MH+MAHPsAnPvEJHnjgAcbGxjh48GDB0Urtld3VS4LTcz50M1sOPAlcDswAh4Cr3f2xpnXOAl5xdzezi4F73P293d5X86GLDGnZsmQQRiuz5F6ZUknDzoe+HTjh7ifd/Q3gbmBn8wru/rKfOTP8OlDOXTNE6qRTly7NYllbaerQNwDPNT2fAX63dSUzuxL478A7gY9mEl3JNB+6BG3fvvZ3gtIslrWVJqFbm2VLSuDu/l3gu2b2+8CtwH9c8kZme4A9ABs7lCLcHbN2H1m8YedDL+v2flITCw2fN9+cjKjbuDFJ5moQra00VS4zwHlNz8eAFzqt7O4/At5jZuvbvLbf3SfdfXJ0dHTJ/129ejVzc3OVSITuztzcHKtXry47FKmy3bvhmWeSOvNnnlEyr7k0JfRDwGYzmwCeB3YB1zSvYGa/BTzVaBS9FFgFzPUbzNjYGDMzM8zOzvb7X4O0evVqxsbGyg5DRGqiZ0J399Nmdj1wEFgOfNPdHzWzvY3Xbwf+CLjWzH4JvAZ80gcoZq9cuZKJiYl+/5uIiJCi22Je2nVbFBGR7obttigiIhFQQhcRqYjSqlzMbBZoMwFuKuuBlzIMJ08xxQqKN08xxQpxxRtTrDBcvOPuvrSbICUm9GGY2XSnOqTQxBQrKN48xRQrxBVvTLFCfvGqykVEpCKU0EVEKiLWhL6/7AD6EFOsoHjzFFOsEFe8McUKOcUbZR26iIgsFWsJXUREWiihi4hURNAJPaZb3/WKtWm9bWb2ppldVWR8beLotW0vM7OfN7btETO7pYw4G7H03LaNeI+Y2aNm9i9Fx9gSS69te0PTdj3WOB7WBhrrO8zs+2Z2tLFtrysjzqZ4esV7jpl9t5EXfmpmF5URZyOWb5rZi2Z2rMPrZmZfbXyXhxsTGw7H3YN8kEwE9hTwbpLZG48CW1rWOYsz7QAXA4+HGmvTeg8C9wFXBb5tLwP+dyTHwdnAY8DGxvN3hhxvy/ofBx4MNVbgz4G/bPx7FJgHVgUc718Bf9H493uBB0o8Fn4fuBQ41uH1jwA/ILnnxO8BPxn2M0Muocd067uesTZ8Dvh74MUig2sjbbwhSBPrNcB33P0UgLuXuX373bZXA98uJLKl0sTqwBpL7jpzFklCP11smG9JE+8W4AEAd38c2GRm7yo2zIQn94aY77LKTuBOT/wYONvMfnOYzww5obe79d2G1pXM7Eozexz4R+AzBcXWqmesZrYBuBK4vcC4Okm1bYH3Ny61f2BmFxYT2hJpYj0fOMfM/tnMDpvZtYVFt1TabYuZjQA7SE7yZUgT623ABSQ3tXkE+Ly7l3UH6jTxHgX+EMDMtgPjJDflCVHqYyWtkBN66lvfuft7gT8gufVdGdLE+hXgRnd/M/9wekoT70Mkc0ZcAnwN+F7eQXWQJtYVwFaSe9l+CPiSmZ2fd2AdpDpuGz4O/B9371aKy1OaWD8EHAHOBX4HuM3M3p5vWB2liffLJCf3IyRXxP9KeVcUvfRzrKSS5o5FZen71ndm9h4zW+/uRU/SkybWSeDuxv1S1wMfMbPT7v69QiJcrGe87v5vTf++z8z+JuBtOwO85O6vAK+Y2Y+AS4AniwlxSSxpj9tdlFfdAulivQ74cqNq84SZPU1SN/3TYkJcJO1xex0kjY7A041HiPrKcamU1WCQokFhBXASmOBMA8iFLev8FmcaRS8luUWehRhry/rfotxG0TTb9jeatu124FSo25akSuCBxrojwDHgolC3bWO9d5DUr/564MfB3wL/rfHvdzV+Y+sDjvdsGo22wJ+S1FGXsn0bMWyic6PoR1ncKPrTYT8v2BK6F3jru4JiDUbKeK8CPmtmp0m27a5Qt627Hzez+4GHgV8B33D3tl3FQoi3seqVwD95clVRipSx3gp8y8weIUk8N3rxV2n9xHsBcKeZvUnS8+lPyogVwMy+TdJbbL2ZzQB/AayEt2K9j6SnywngVRpXFkN9Zgm/URERyUHIjaIiItIHJXQRkYpQQhcRqQgldBGRilBCFxGpCCV0EZGKUEIXEamI/w+U6Wmm4uuHBQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 从 data.txt 中读入点\n",
    "with open('./data.txt', 'r') as f:\n",
    "    data_list = [i.split('\\n')[0].split(',') for i in f.readlines()]\n",
    "    data = [(float(i[0]), float(i[1]), float(i[2])) for i in data_list]\n",
    "\n",
    "# 标准化\n",
    "x0_max = max([i[0] for i in data])\n",
    "x1_max = max([i[1] for i in data])\n",
    "data = [(i[0]/x0_max, i[1]/x1_max, i[2]) for i in data]\n",
    "\n",
    "x0 = list(filter(lambda x: x[-1] == 0.0, data)) # 选择第一类的点\n",
    "x1 = list(filter(lambda x: x[-1] == 1.0, data)) # 选择第二类的点\n",
    "\n",
    "plot_x0 = [i[0] for i in x0]\n",
    "plot_y0 = [i[1] for i in x0]\n",
    "plot_x1 = [i[0] for i in x1]\n",
    "plot_y1 = [i[1] for i in x1]\n",
    "\n",
    "plt.plot(plot_x0, plot_y0, 'ro', label='x_0')\n",
    "plt.plot(plot_x1, plot_y1, 'bo', label='x_1')\n",
    "plt.legend(loc='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来将数据转换成 NumPy 的类型，并转换到 Tensor 为之后的训练做准备"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "np_data = np.array(data, dtype='float32') # 转换成 numpy array\n",
    "x_data = torch.from_numpy(np_data[:, 0:2]) # 转换成 Tensor, 大小是 [100, 2]\n",
    "y_data = torch.from_numpy(np_data[:, 2]).unsqueeze(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在 PyTorch 当中，不需要我们自己写 Sigmoid 的函数，PyTorch 已经用底层的 C++ 语言为我们写好了一些常用的函数，不仅方便我们使用，同时速度上比我们自己实现的更快，稳定性更好。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 定义 logistic 回归模型\n",
    "w = torch.randn((2, 1), dtype=torch.float, requires_grad=True) \n",
    "b = torch.zeros(1, dtype=torch.float, requires_grad=True)\n",
    "\n",
    "def logistic_regression(x):\n",
    "    return torch.sigmoid(torch.mm(x, w) + b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在更新之前，我们可以画出分类的效果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f8332bf5160>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAo0ElEQVR4nO3deXQUVd4+8OdmYQk7AWUJSXAAJSBB0iwqqLggIqJsA4gssoRuRwdlRnF+jL5z9OW8OjqKinYTULaguICiDIICAgOi0oEgOzKSQNgNIGvMdn9/dBKz9FLdXWvyfM7pA+l0qr5dXf30rVv3VgspJYiIyLoijC6AiIjCwyAnIrI4BjkRkcUxyImILI5BTkRkcVFGrLRZs2YyMTHRiFUTEVlWRkbGL1LK5pXvNyTIExMT4Xa7jVg1EZFlCSGyvd3PrhUiIotjkBMRWRyDnIjI4hjkREQWxyAnIrI4VYJcCPGeEOK0EGK3GsurMZYsARITgYgIz79LlhhdERFZkFot8gUA+qu0rJphyRIgNRXIzgak9Pybmqp6mPOzInSVt91jj+m3LYN93bR+nbkfmZyUUpUbgEQAu5U8NiUlRdY46elSJiRIKYTn39hYKT0RXvGWkKDqKmNiKi4+JsZzP/nnbdtVvmm1LYN93bR+nbkfmQcAt/SWv97uDOUWKMgBpAJwA3DHx8fr8qRNQ0kqlN6EUG21CQmaf1ZUW762nR7b0te6Y2MrtgVKg1Tr19nf8iu3Txju2jI8yMvfalyLXGkqqJwMQmj+WVFt+dp2emxLpesubRUrfZ3Lh25srOemJID91RMdXfVnhrl2fAU5R63o4cgRZY+LiQFmzlRttfHx/u9Xs99Trz5Uvdbja9uF+jgt1n3lCjBjRuDXGah6SiY313OTJadnHnkEaNbM+/b0tfyICKCgoOJ9BQXA1KnK6teakec4dOct3UO5gS1y34I9VlaJv75NNfs99epD1aMvuHyLtVYtZS1itZSuv7Q1rfSIQMl28XVKRslz8rV8f8sxmhHnOPToZoKWXSsAPgBwAkABgBwAE/09vsYFuYFni3ztXGr2q+rVF6/lery9RNHRFbsfHA7t3qje1l8a5krOjfsLkfR0ZSHub3t6W76Zg1zvcxx6vcU1DfJgbzUuyKU03VkhNfvP9eqL13I9anxIhPMSB1p/OEERzCmaYLanrw+X2Fjlz1srep/j0KsxwyA3O52DPiH2Ilvk5YT7IeFwVF1GMC0yJev3toso2W2Uhlqw2zM9vWr3U61ahrdRpJT6t8j1aswwyM1M7eOyQO/u9HSZHj1exuBSxVXWKtCtjzyUzy21+/XVGtbvb+SI0qAI5UNK6fYIpkUe7PZUsKsZciAaTh95KDWzRV7TpadLGRmp3l6g5N1dstelY5RMwGEpUCQTcFimxz4R1tNQuvOHE8hqBIO39deqVXUonRpdF0pbZKFsE6XhESjUyvfFB/PhG+icgdETiYKtN5ya2UdekylpNqjVNIiM/P3vDR5gbvREJX+DiMq3zGNjlb35/XVdaNnHHszLGOoY8vJ/H2wL1+jXORTh1KzH0QmD3IwCHfOG0unqL1VK/97gd5i/ANLjUDzQJlLykigZAVT6fLQSTIs8mKMlb4Hv66DR37r1ai+ouc+EWrOSEFejxc4gNyN/iRJqp2ugD4fSvczAY15/LWI9yvJ30KIkrCrX5mvooMPhv45wA0jJyxjMS62k1R3oVj7w9GgvqL0ra3GuQs3eUwa5GYWSKL6aBuVb2oE+IMo/PlCKaNBE9rXjh3rCMdgSfa0/lOAqPzxQjRpCCXN/6w0mmIIdphhouXq0F/zVrNaHY6APZX/bWMn5iWAwyM3I354ezDswmKZUsJ22Gr0TvQWQz8NaFKteorf1hxJkoXYT6NW7FUxXQbDDFCvffM0K1bKrLFDNoeyuwQ4l9beNlRwgB4NBblblE6S0JV56il1pQvnrpA1nr1Y7bQK8q32uDod9NonULDGUroVg1xPoA0Pt/mO1W+SRkfrMdFVKSc3BvkbB7lP+Hq/2hwyD3Mx8NSuVvlMCDZsI9Z2m5tkqBU3n9HQpY8Tlig/BJZmOUT7PHAYqMZQuD29D1tT4XFQ6SElNavaRm/Ea5Eq2abC7a7C7vb+3r69llR9EFgwGuZmF26zU6jhdyVDGcJdVqcZ0PFxxbDtG+X0+wfZPajnPKhAtWmhq1x3uMMVwawplGwc6ytG6Re6rbi1GMzHIzSzclq9Wfdn+mjvBLl/pcwyy30Gt0wx6CHTgZLbWrlb8tWDD2Y3VPIGsxnL8vd6hYpCbmRqJo9VZJTXGTgWzDH9nPSsPiSh5vumxT8iE2ItVnrrq45jD3MZKX2atTxAaLdjBWka8DdRYjhYNCQa5lvQYEGykcBJR6RnE2Njfn2+gYQMKt5eqb6Rwr4qlsGyz7wpqCHZ0jE4TjlWnxWvJINeKFgOCteycDEU4iRjMmL6IiN+ft79toLS/Xa03ktKjBIWLqvKZX+7OhMijqrfiAq5fwe/U4u/gTM1LDpmF2tuUQa6VcJt93oZJmK1JFk4ihjM42dc6gjhCUOWNpOV4wUrbVqBIs1apv5dRzfH4way//LrU2PXVDk6zdXMxyLUSzBkNJaGtUstPdaHu0cG0yJU+by3PYnp7noHOUoazrkrN0AQc1uyp+dtsoY7U8Bu8lbalr2vgl19XOGPTvdXj7do44SzP6DYVg1wrSscY+dvLlNyqU0dhuM9bz1E6/q4dEMo4svLDabxsh3SMqnqdeJXCQ6DY59MI5TSI3/D3si19HW2o9VzVHt5ptlFPUjLItaO0/1SLlqlVeDsSCfTNxoGet5oDkUv5eo28Xc1LyVWxvNWs4MRvOkbJhMij6h7Op6fLBJHtcxOHElp+w9/LAn0dbai1qytpH3hdto99xvclI4oM62dhkGvJ115TvjkTbCtUrWaKWaWnB/5q92AHEXtbXjDL8JdMWo5HC6fmINbttbUvLofcR+43/BUebQR62wT5FINftp8n7vP54bBh70sGuZaUNGf8dcFU3onMcBELvfhqpZYfjhjqMoJt4ml9LK3kwzzUudsK113lW6HwcNlDgv2s8hv+PrZl2fqF+qNUlBzwVFm2n9fc6/MrvWSE2vuGQgxyLYUzQLgmhbYv4bZ2AzXFjJ4hq7ROLVt4/rqNwuDzpVPQ5ajF5vZ3CsLrsgOcHCh7ft4uGRHMvqUSBrnWlISR2cYyVReBWrrBtJq0fI3UHlYR7Lq9nZeIjtZuvQomUWm9uQMuW+lRmEnOfDLIqfry19I12/kFIz/MQ/3mjnCYvfGi9LDAJGMRGeRUfanRz14TqH7xmWpC6YeNCT6UfAW58PxOXzabTbrdbt3XS9XYkiXAjBnAkSNAfDwwcyYwerTRVZlLYiKQnV31/oQEICtL72ooBEKIDCmlrfL9EUYUQ6S60aM9YVRc7PmXIV7VzJlATEzF+2JiPPeTpTHIiWqK0aOBtDRPC1wIz79pafzQqwaijC6AiHQ0ejSDuxpii5yIyOIY5EREFscgJyKyOAY5EZHFMciJiCyOQU5EZHGqBLkQor8Q4oAQ4pAQ4lk1lklERMqEHeRCiEgAbwO4D0ASgFFCiKRwl0tERMqo0SLvAeCQlPJnKWU+gKUAHlRhuUREpIAaQd4awNFyP+eU3FeBECJVCOEWQrjPnDmjwmqJiAhQJ8iFl/uqXFJRSpkmpbRJKW3NmzdXYbVERASoE+Q5ANqU+zkOwHEVlktERAqoEeTbALQXQrQVQtQCMBLA5yosl4iIFAj76odSykIhxOMA1gCIBPCelHJP2JUREZEiqlzGVkq5CsAqNZZFRETB4cxOIiKLY5ATEVkcg5yIyOIY5EREFscgJyKyOAY5EZHFMciJiCyOQU5EZHEMciIii2OQExFZHIOciMjiGORERBbHICcisjgGORGRxTHIiYgsjkFORGRxDHIiIotjkBMRWRyDnIjI4hjkREQWxyAnIrI4BjkRkcUxyImILI5BTkRkcZYK8re+fwuDPxyMr/77FYplsdHlEBGZgqWCXEJi85HNuDf9XnR4qwNe2fIKfrnyi9FlEREZylJB/ueef0bOUzlYMmQJWjZoiWfWPoPWr7XGI8sfwZYjWyClNLpEIiLdCSPCz2azSbfbHfZydp/eDZfbhUU7F+Fi/kXceM2NcNgceKTLI2hQu4EKlRIRmYcQIkNKaatyv5WDvNSl/Ev4YNcHcLqd2HFyB+rXqo/RN46Gw+ZAcotk1dZDRGSkah3kpaSU2HZ8G5xuJ5buXoq8wjzcHHcz7DY7hicNR93ouqqvk4hILzUiyMs7d/UcFu5cCKfbiYO5B9G0blOMTx4Pu82O9rHtNV03EZEWalyQl5JS4pusb+B0O/HZ/s9QWFyIu6+7Gw6bA4OuH4SoiChd6iAiCleNDfLyTlw8gXd3vIu0jDQcvXAUrRq0wqSbJmFyymTENYzTvR4iomAwyMspLC7Eqp9WweV2YfWh1YgQEXjg+gdgT7Hjnj/cgwhhqVGZRFRD+ArysBJLCDFcCLFHCFEshKiycLOKiojCoOsHYdXoVTj050N4+panseXIFvRf0p8TjYjIcsJteu4GMATAJhVqMcR1Ta7D/939fzj61FG8P+R9tGrQihONiMhSwgpyKeU+KeUBtYoxUu2o2hh14yhsenQTdjt2I7VbKr44+AV6z++NZFcynNucuPDbBaPLJCKqgp3BXnS6phPeGvAWjk07hrSBaYiKiMJjqx5D69daw77Sjp0ndxpdIhFRmYAnO4UQawG08PKrGVLKFSWP2QDgr1JKn2cwhRCpAFIBID4+PiU7OzvUmnVXOtHI5Xbhg90fIK8wD73iesFhc3CiERHpRtNRK0qCvDyjR62E49zVc1i0cxGcbicO5B7gRCMi0o0mo1ZqoiZ1m2Bqr6nY96d9WD92Pe5qexfe/OFNdJjdAfcsvgfL9y1HQVGB0WUSUQ0SVotcCDEYwFsAmgM4DyBTSnlvoL+zcovcmxMXT+C9He8hbXsajvx6hBONiEgTnBCkg6LiIqz6aRWcbidWH1oNIQQe6PAAHDYHJxoRUdgY5Do7fO4w0jLS8O6Od3Hmyhlc1+Q6TEmZgke7Porm9ZobXR4RWRCD3CC/Ff6GT/d/CqfbiU3Zm1ArshaGJQ2Dw+bArW1uhRDC6BKJyCIY5Caw5/QezMmYg4U7F+LCbxfQ+ZrOZd9o1LB2Q6PLIyKTY5CbyOX8y/hgt+cbjbaf2I560fU832jU3YGuLboaXR4RmRSD3ISklHAfd5d9o9HVwqvoFdcL9hQ7/tjpj5xoREQVMMhNrnSikSvDhf2/7EeTOk0wvqtnolGH2A5Gl0dEJsAgtwgpJTZkbYDT7cSn+z9FYXEh7mp7F+w2Ox68/kFER0YbXSIRGYRBbkEnL53Eu9vfLZto1LJ+S0zqNgmTu01Gm0ZtjC6PiHTGILewouIifHnoSzjdTnz505dlE43sNjv6/aEfJxoR1RAM8mri8LnDmLt9Lt7d8S5OXz7NiUZENQiDvJrJL8rHp/s8E402Zm8sm2hkT7Gjd3xvTjQiqoYY5NXY3jN74XK7Kkw0sqfYMSZ5DCcaEVUjDPIaoHSikcvtQsaJjLKJRnabHTe1vMno8ogoTAzyGsZ93A3nNic+2P0BrhZeRc/WPeGwOTjRiMjCGOQ11Lmr57D4x8Vwup0VJhpNSZmC65tdb3R5RBQEBnkNJ6XExuyNcLldWLZvGQqLC3Fn2zvhsDk40YjIIhjkVObkpZN4b8d7mJMxhxONiCyEQU5VlE40crldWPXTKgghMLDDQDhsDk40IjIhBjn5lXU+q+wbjU5fPo22jdtiSsoUTLhpAicaEZkEg5wUKZ1o5MpwYUPWBtSKrIWhHYfCYXNwohGRwRjkFLR9Z/aVTTT69bdf0al5J9htdozpMgaN6jQyujyiGodBTiG7nH8ZS3cvhdPtLJto9PCND8Nhc3CiEZGOGOSkCm8Tjew2O0Z0GsGJRkQaY5CTqs7nnceinYvKJho1rtMY45M932jEiUZE2mCQkyaklNiUvQlOtxPL9y1HQXEB+ib2hcPmwEM3PMSJRkQqYpCT5k5dOoV3d7yLtIw0ZP+ajRb1W2DSTZMwOWUy4hvFG10ekeUxyEk3RcVFWH1oNZxuZ9lEo/vb31820SgyItLoEoksiUFOhsg+n420jDTM2zEPpy+fRmLjxLKJRtfUu8bo8ogshUFOhqo80Sg6ItrzjUY2O/rE9+FEIyIFGORkGpxoRBQaBjmZzpWCK2UTjdzH3WUTjew2O7q17GZ0eUSmwyAnU3Mfd8PlduH9Xe/jauFV9GjdA/YUO0Z0HoGY6BijyyMyBQY5WULpRCOX24V9v+wrm2g0xTYFNzS7wejyiAzFICdL4UQjoqoY5GRZpy6dKvtGo9KJRhNvmojUlFRONKIaRZMgF0K8AuABAPkA/gvgUSnl+UB/xyCnUHCiEdV0WgV5PwDrpZSFQoiXAUBKOT3Q3zHIKVzZ57Mxd/tczNs+D6cun+JEI6oRNO9aEUIMBjBMSjk60GMZ5KSW/KJ8fLb/M7jcLnyT9Q2iI6IxNMnzjUacaETVjR5B/gWAD6WU6T5+nwogFQDi4+NTsrOzVVkvUan9v+wvm2h0Pu88kponwZ5ix9jksZxoRNVCyEEuhFgLoIWXX82QUq4oecwMADYAQ6SCTwa2yElLVwqu4MPdH8LpdmLb8W2IiY7Bw509E41SWqUYXR5RyDRrkQshxgGwA7hLSnlFyd8wyEkvlScadW/VHQ6bgxONyJK0OtnZH8BrAG6XUp5R+ncMctLb+bzzWLxzMZxuZ9lEo3HJ42C32TnRiCxDqyA/BKA2gNySu76TUtoD/R2DnIwipcR/jvwHTrcTy/YuK5toZLfZ8dAND6FWZC2jSyTyiROCiCo5dekU5mfOx5yMOcg6n4Vr611bNtEooXGC0eURVcEgJ/KhqLgIa/67Bk63E/8++G8IITCg/QDYU+zo364/JxqRaTDIiRSoPNEooVECpqRMwcRuEznRiAzHICcKgq+JRvYUO25LuI0TjcgQDHKiEO3/ZT/muOdgwc4FFSYajUkeg8Z1GhtdHtUgDHKiMJVONHJluPDDsR8QEx2DUZ1HwWFzcKIR6YJBTqSi7Se2w+V2YcmuJbhScAW2VjY4bA6M7DySE41IMwxyIg38mvcrFv/omWi098xeNKrdqGyiUcfmHY0uj6oZBjmRhkonGrncLnyy9xMUFBfgjsQ7yr7RiBONSA0MciKdnL58GvN3zIcrw8WJRqQqBjmRzoplMdYcKplo9NO/IaXEgPYD4LA5ONGIQsIgJzLQkV+PYG7GXMzbMQ8nL51EQqMEpKakYuJNE3Ft/WuNLo8sgkFOZAIFRQVYcWAFnG4n1h9ej+iIaAzpOAR2mx23J9zOiUbkF4OcyGQO/HIAczLmYEHmApzLO4eOzTrCbvN8oxEnGpE3DHIik7pacBUf7vF8o9EPx35A3ai6nolG3R2wtarynqUajEFOZAGVJxqltEwpm2hUr1Y9o8sjgzHIiSykdKKRy+3CnjN70Kh2I4xNHguHzcGJRjWY6YO8oKAAOTk5yMvL072emqhOnTqIi4tDdHS00aWQH1JKbD6y2fONRvuWIb8oH7cn3A6HzYHBHQdzolENY/ogP3z4MBo0aIDY2FieudeYlBK5ubm4ePEi2rZta3Q5pFDpRKM5GXNw+PxhXFPvmrKJRomNE40uj3TgK8gjjCjGm7y8PIa4ToQQiI2N5dGPxVxT7xpM7z0dh/58CF+O/hK94nrh5S0v47o3rsP979+PlQdXoqi4yOgyyQBRRhdQHkNcP9zW1hUhItC/XX/0b9cfR389irnb52Lu9rl44IMHEN8oHqndUjGx20S0qN/C6FJJJ6ZpkRNR8No0aoMX+r6AI08ewcfDP0a7pu3w92/+jjavt8GIT0ZgQ9YGGNF9SvpikIcoKysL77//ftnPmZmZWLVqVdnPn3/+OV566SVV1jV+/Hh88sknAIBJkyZh7969qiyXqo/oyGgMSxqGdWPXYf+f9uOJHk/g6/9+jb4L+yLpnSS88d0bOJ933ugySSMM8hAFCvJBgwbh2WefVX298+bNQ1JSkurLperj+mbX47V7X8Oxacew4MEFaFS7EZ5c8yRa/asVJqyYgG3HthldIqnMVH3kpZ5c/SQyT2aqusyuLbpiVv9Zfh+zaNEivPrqqxBCoEuXLli8eDHGjx+PgQMHYtiwYQCA+vXr49KlS3j22Wexb98+dO3aFaNGjcLbb7+Nq1evYvPmzfjb3/6Gq1evwu12Y/bs2Rg/fjwaNmwIt9uNkydP4p///CeGDRuG4uJiPP7449i4cSPatm2L4uJiTJgwoWxd3txxxx149dVXYbPZUL9+fUydOhUrV65E3bp1sWLFClx77bU4c+YM7HY7jhw5AgCYNWsWbr31VtW2JVlD3ei6GNd1HMZ1HYcdJ3aUTTSanzkfKS1TYLfZMarzKE40qgbYIi+xZ88ezJw5E+vXr8fOnTvxxhtv+H38Sy+9hD59+iAzMxPTp0/HCy+8gBEjRiAzMxMjRoyo8vgTJ05g8+bNWLlyZVlLffny5cjKysKuXbswb948bN26NaiaL1++jF69emHnzp247bbbMHfuXADA1KlT8dRTT2Hbtm1YtmwZJk2aFNRyqfq5qeVNmPPAHBybdgyz75uNvMI8TP5iMlq91gpPrHoCe8+wu87KTNkiD9Ry1sL69esxbNgwNGvWDADQtGlTVZf/0EMPISIiAklJSTh16hQAYPPmzRg+fDgiIiLQokUL9O3bN6hl1qpVCwMHDgQApKSk4OuvvwYArF27tkI/+oULF3Dx4kU0aNBApWdDVtWoTiP8qcef8Fj3x7Dl6BY43U6kbU/D7G2zcVvCbXDYHBjScQgnGlmMKYPcCFJKr0PyoqKiUFxcXPaY/Pz8kJZfu3btCusq/2+ooqOjy2qOjIxEYWEhAKC4uBhbt25F3bp1w1o+VV9CCPSO743e8b0x695ZmJ85Hy63C6OWjeJEIwti10qJu+66Cx999BFyc3MBAGfPngUAJCYmIiMjAwCwYsUKFBQUAAAaNGiAixcvlv195Z+V6N27N5YtW4bi4mKcOnUKGzZsUOGZAP369cPs2bPLfs7MzFRluVQ9Na/XHM/c+kzZRKOb426uMNHoiwNfcKKRyTHIS3Tq1AkzZszA7bffjuTkZEybNg0AMHnyZGzcuBE9evTA999/j3r1PCeGunTpgqioKCQnJ+P1119H3759sXfvXnTt2hUffvihonUOHToUcXFx6Ny5M6ZMmYKePXuiUaNGYT+XN998E263G126dEFSUhJcLlfYy6Tqr3Si0WcjP0PW1Cw8d9tz2HFiBwYtHYTr3rwOMzfNxMlLJ40uk7wwzbVW9u3bh44da95V3S5duoT69esjNzcXPXr0wJYtW9CihT4z8mrqNiflCooK8PmBz+F0O7Hu8DpERURh8A2D4bA5cEfiHZwhrDNf11phH7nBBg4ciPPnzyM/Px/PPfecbiFOpER0ZDSGJg3F0KShOJh7EC63CwsyF+DjvR/j+tjrYbfZMS55HJrUbWJ0qTUaW+Q1GLc5heJqwVV8tOcjuDJc+C7nO9SNqouRnUfCYfN8oxFb6dox/dUPicgaSicabZ24FTum7MDY5LH4aM9H6DGvB2xzbZi3fR4u5182uswahUFORCHr2qIrXANdOP6X43h7wNvIL8rnRCMDMMiJKGwNazfEY90fw4/2H/GfR/+DgR0GIm17Gjq90wm3L7gdS3cvRX5RaHMwKLCwglwI8aIQ4kchRKYQ4ishRCu1CiMi6ymdaLRkyBLkPJWDl+9+GTkXcjBq2Si0eb0N/rb2b8g6n2V0mdVOuC3yV6SUXaSUXQGsBPB8+CUptGQJkJgIRER4/l2yRLdVE1FgpRONfnriJ6wevRo3x92Mf377T0400kBYQS6lvFDux3oA9BkCs2QJkJoKZGcDUnr+TU3VLcwXLlyI9u3bo3379li4cKEu6ySyqggRgXvb3YvPRn6G7CezK0w0avtGW/zvpv/lRKMwhT38UAgxE8BYAL8C6CulPOPjcakAUgEgPj4+JTs7u8LvgxoKl5joCe/KEhKArCzFtYfi7NmzsNlscLvdEEIgJSUFGRkZaNLEeuNoOfyQjFJQVIAvDn4Bp9uJtT+v5UQjhUIefiiEWCuE2O3l9iAASClnSCnbAFgC4HFfy5FSpkkpbVJKW/PmzcN5LkDJdbYV36/Atm3b0KVLF+Tl5eHy5cvo1KkTdu/eXeVxa9aswT333IOmTZuiSZMmuOeee7B69eqQ10tUE0VHRmNIxyH4eszXOPj4QUztORXrDq/DnYvuRMe3O2LWd7Nw7uo5o8u0jIBBLqW8W0rZ2cttRaWHvg9gqDZlVhIfH9z9CnTv3h2DBg3C3//+dzzzzDN45JFH0Llz5yqPO3bsGNq0aVP2c1xcHI4dOxbyeolquvax7fFqv1eR81QOFj60EE3qNsFTa55C69daY8KKCfjh2A/83tEAwh210r7cj4MA7A+vHIVmzgRiYireFxPjuT8Mzz//PL7++mu43W4888wzXh/jbYfiYSBR+OpG18XY5LFVJhr1nNcTtrk2zM2Yy4lGPoQ7auWlkm6WHwH0AzBVhZoCGz0aSEvz9IkL4fk3Lc1zfxjOnj2LS5cu4eLFi8jLy/P6mLi4OBw9erTs55ycHLRqxVGXRGoqP9HonQHvoKCoAKkrU8smGu05vcfoEk2F11opZ9CgQRg5ciQOHz6MEydOVLimd6mzZ88iJSUF27dvBwB069YNGRkZqn+jkB7MsM2JlJBS4tuj38LpduLjvR8jvygffeL7lH2jUe2o2oEXUg3w6ocBLFq0CFFRUXj44YdRVFSEW265BevXr8edd95Z4XFNmzbFc889h+7duwPwdMdYMcSJrEQIgVvjb8Wt8bdiVv9ZmL9jPlwZLjy8/GE0j2mOCTdNwJSUKWjbpK3RpRqCLfIajNucrKxYFmPtz2vhdDvx+YHPIaVE/3b94bA5MKD9AERGRBpdourYIieiaiVCRKDfH/qh3x/6IedCDuZtn4e52+di0NJBaNOwDVJTUjHxpolo2aCl0aVqjhfN8mHXrl3o2rVrhVvPnj2NLouIvIhrGId/3PEPZE3NwvI/LscNzW7Ac988h/hZ8Rj+8XCs+3ldtR7CyBa5DzfeeCO/tJjIYqIjozG442AM7jgYP+X+hDkZczA/cz4+2fsJOsR2gD3FjnFdx6Fp3ep1XostciKqlipPNIqtG4tpX01D69daY/xn4/F9zvfVppXOICeiaq10otG3E79F5pRMjEseh2X7lqHXu72QkpaCuRlzcSn/ktFlhoVBTkQ1RnKLZLgGunBs2jG8M+AdFBYXInVlKlq/1hqPr3ocu09Xvb6SFVg2yHk5ciIKVcPaDeHo7sBO+05smbAFg64fhLnb5+JG543oM78P3t/1Pn4r/M3oMhWzZJAbfDly9O/fH40bN8bAgQP1WSERaUIIgVva3ILFgxfj2LRjeOWeV3Di4gmMXj4aca/HYfrX0/HzuZ+NLjMgSwb5jBnAlSsV77tyxXO/Hp5++mksXrxYn5URkS6axTTDX2/5Kw4+cRBrHlmDPvF98K+t/0K7N9vhviX34fMDn5v2G40sGeQaXI5c8fXIAeCuu+5CgwYNQl8ZEZlW6USj5SOWI+vJLDx/+/P48dSPeHDpg2j7Rlu8uPFFnLh4wugyK7BkkGtwOXLF1yMnopqjdKJR9pPZWP7H5ejYvCOe3/C86SYaWTLINbocuaLrkRNRzRMVEYXBHQdjzSNr8NMTP+HJnk9i/eH1uHvx3bjh7Rvw2tbXcPbqWcPqs2SQa3Q5ckXXIyeimq1d03Z4pd8rODbtGBY9tAjNYprhL1/9xdCJRpYMcsAT2llZQHGx599wQxwAUlNT8eKLL2L06NGYPn16+AskomqrTlQdjEkegy0TtiBzSibGJ4+vMNEoLSNNt4lGlg1ytZW/Hvmzzz6Lbdu2Yf369V4f26dPHwwfPhzr1q1DXFwc1qxZo3O1RGQmyS2S4RzoxPFpx+G834kiWYQpK6eg1b9aYY57jubr5/XIazBucyJtSCmxNWcrXG4XRnUehfva36fKcnk9ciIinZRONLqlzS26rI9B7sOuXbswZsyYCvfVrl0b33//vUEVERF5Z6ogl1JCCGF0GQCq//XIzTD2lYjUYZqTnXXq1EFubi4DRgdSSuTm5qJOnTpGl0JEKjBNizwuLg45OTk4c+aM0aXUCHXq1EFcXJzRZRCRCkwT5NHR0Wjbtq3RZRARWY5pulaIiCg0DHIiIotjkBMRWZwhMzuFEGcAZIf4580A/KJiOWphXcFhXcFhXcExa11AeLUlSCmbV77TkCAPhxDC7W2KqtFYV3BYV3BYV3DMWhegTW3sWiEisjgGORGRxVkxyNOMLsAH1hUc1hUc1hUcs9YFaFCb5frIiYioIiu2yImIqBwGORGRxZk2yIUQ/YUQB4QQh4QQz3r5/WghxI8lt2+FEMkmqevBkpoyhRBuIURvM9RV7nHdhRBFQohhZqhLCHGHEOLXku2VKYR43gx1lastUwixRwix0Qx1CSGeLretdpe8lk1NUFcjIcQXQoidJdvrUa1rUlhXEyHEpyXvyR+EEJ11qus9IcRpIcRuH78XQog3S+r+UQjRLawVSilNdwMQCeC/AK4DUAvATgBJlR5zC4AmJf+/D8D3JqmrPn4/99AFwH4z1FXucesBrAIwzAx1AbgDwEoT7l+NAewFEF/y8zVmqKvS4x8AsN4MdQH4fwBeLvl/cwBnAdQyQV2vAPifkv/fAGCdTvvYbQC6Adjt4/cDAHwJQADoFW5+mbVF3gPAISnlz1LKfABLATxY/gFSym+llOdKfvwOgB7XZFVS1yVZ8koBqAdAj7PJAesq8QSAZQBO61BTMHXpTUldDwNYLqU8AgBSSj22WbDbaxSAD0xSlwTQQHi+GaY+PEFeaIK6kgCsAwAp5X4AiUKIazWuC1LKTfBsA18eBLBIenwHoLEQomWo6zNrkLcGcLTczzkl9/kyEZ5PN60pqksIMVgIsR/AvwFMMENdQojWAAYDcOlQj+K6Stxcckj+pRCik0nq6gCgiRBigxAiQwgx1iR1AQCEEDEA+sPzwWyGumYD6AjgOIBdAKZKKYtNUNdOAEMAQAjRA0AC9Gn0BRJsxvll1iD39n1vXlu2Qoi+8AT5dE0rKlmdl/uq1CWl/FRKeQOAhwC8qHVRUFbXLADTpZRF2pdTRkld2+G5fkQygLcAfKZ1UVBWVxSAFAD3A7gXwHNCiA4mqKvUAwC2SCn9tfrUoqSuewFkAmgFoCuA2UKIhtqWpaiul+D5QM6E54h0B7Q/UlAimNc6INN8sUQlOQDalPs5Dp5P+gqEEF0AzANwn5Qy1yx1lZJSbhJC/EEI0UxKqeUFfJTUZQOwtOQ7UZsBGCCEKJRSfmZkXVLKC+X+v0oI8Y5JtlcOgF+klJcBXBZCbAKQDOCgwXWVGgl9ulUAZXU9CuClkm7FQ0KIw/D0Sf9gZF0l+9ejgOcEI4DDJTejBZUlAenR8R/CiYIoAD8DaIvfT2J0qvSYeACHANxisrra4feTnd0AHCv92ci6Kj1+AfQ52alke7Uot716ADhihu0FTzfBupLHxgDYDaCz0XWVPK4RPP2v9bR+DYPYXk4A/yj5/7Ul+30zE9TVGCUnXQFMhqdfWvNtVrK+RPg+2Xk/Kp7s/CGcdZmyRS6lLBRCPA5gDTxnpt+TUu4RQthLfu8C8DyAWADvlLQyC6XGVztTWNdQAGOFEAUArgIYIUteOYPr0p3CuoYBcAghCuHZXiPNsL2klPuEEKsB/AigGMA8KaXXoWR61lXy0MEAvpKeowXNKazrRQALhBC74Amn6VLboyqldXUEsEgIUQTPKKSJWtZUSgjxATwjspoJIXIA/A+A6HJ1rYJn5MohAFdQctQQ8vo0fs8QEZHGzHqyk4iIFGKQExFZHIOciMjiGORERBbHICcisjgGORGRxTHIiYgs7v8Dp8llgTYBYR0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 画出参数更新之前的结果 \n",
    "w0 = w[0].data[0]\n",
    "w1 = w[1].data[0]\n",
    "b0 = b.data[0]\n",
    "\n",
    "plot_x = np.arange(0.2, 1, 0.01)\n",
    "plot_y = (-w0.numpy() * plot_x - b0.numpy()) / w1.numpy()\n",
    "\n",
    "plt.plot(plot_x, plot_y, 'g', label='cutting line')\n",
    "plt.plot(plot_x0, plot_y0, 'ro', label='x_0')\n",
    "plt.plot(plot_x1, plot_y1, 'bo', label='x_1')\n",
    "plt.legend(loc='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到分类效果不好，计算 loss，公式如下\n",
    "\n",
    "$$\n",
    "loss = -\\{ y * log(\\hat{y}) + (1 - y) * log(1 - \\hat{y}) \\}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 计算loss, 使用clamp的目的是防止数据过小而对结果产生较大影响。\n",
    "def binary_loss(y_pred, y):\n",
    "    logits = ( y * y_pred.clamp(1e-12).log() + \\\n",
    "              (1 - y) * (1 - y_pred).clamp(1e-12).log() ).mean()\n",
    "    return -logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意到其中使用 `.clamp`，可以查看[函数使用说明文档](https://pytorch.org/docs/stable/generated/torch.clamp.html?highlight=clamp#torch.clamp)，并且思考一下这里是否一定要使用这个函数，如果不使用会出现什么样的结果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.7655, grad_fn=<NegBackward0>)\n"
     ]
    }
   ],
   "source": [
    "y_pred = logistic_regression(x_data)\n",
    "loss = binary_loss(y_pred, y_data)\n",
    "loss.backward()\n",
    "print(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "得到 loss 之后，使用梯度下降法更新参数，这里可以使用自动求导来直接得到参数的导数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 自动求导并更新参数\n",
    "for i in range(1000):\n",
    "    # 算出一次更新之后的loss\n",
    "    y_pred = logistic_regression(x_data)\n",
    "    loss = binary_loss(y_pred, y_data)\n",
    "    \n",
    "    # calc grad & update w,b\n",
    "    loss.backward()\n",
    "    w.data = w.data - 0.1 * w.grad.data\n",
    "    b.data = b.data - 0.1 * b.grad.data\n",
    "\n",
    "    # clear w,b grad\n",
    "    w.grad.data.zero_()\n",
    "    b.grad.data.zero_()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f8332b61850>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAzyUlEQVR4nO3de5xN9f748ddnbpgoGaJMLvVT58xojAypEPVViugURSIhtzqnTqcLOdVESqVOIiFHx7UShUQqSkUuIzOGkZJbIwlTuRtmv39/7FFjzN6z9+zLWmvP+/l47MfM3nvt9Xnvtdd6r8/6rM/6LCMiKKWUcr4oqwNQSikVHJrQlVIqQmhCV0qpCKEJXSmlIoQmdKWUihAxVhVcvXp1qVevnlXFK6WUI61du3afiNQo6T3LEnq9evXIyMiwqnillHIkY8wOT+9pk4tSSkUITehKKRUhNKErpVSE0ISulFIRQhO6UkpFCE3oSikVITShK6VUhHBcQt97eC///Oif/H7sd6tDUUopW3FcQl+ybQmvrn6V5HHJLPx+odXhKKWUbTguoXdt2JWv+3xN1YpVaT+zPXfPvZu8o3lWh6WUUpZzXEIHaFa7GWv7reWJVk8wM3smyeOSmfvtXKvDUkopSzkyoQNUiKnAsDbDWHPvGmpVrsXf3vkbXWd3Ze/hvVaHppRSlnBsQj8ltVYqq/uu5pk2z/DepvdIGpfEOxveQe+VqpQqbxyf0AFio2MZ2moo6/qvo37V+nSd05XbZt3Gz4d+tjo0pZQKm4hI6Kckn5fMij4reLHtiyzasoik15KYmjVVa+tKqXIhohI6QExUDA9f9TBZA7JIPi+Zu+feTYe3OpB7INfq0JRSKqQiLqGfcknCJSzrtYzR7Ubz+fbPSR6XzBtr39DaulIqYkVsQgeIMlH844p/kD0wmybnN6Hfgn60ndaWbb9uszo0pZQKuohO6KdcdO5FfNrzU8a3H8/qXau57PXLGLt6LC5xWR2aUkoFTblI6OCurfdP68+GQRtoUacFf1/0d6753zV8v/97q0NTSqmgKDcJ/ZQ659RhUfdFvNnpTTb8soGU8Sm8tOIlClwFVoemlFIBKXcJHcAYQ6/UXmwctJHrL76ehz95mKsnX03O3hyrQ1NKqTIrlwn9lAuqXMDcO+Yy89aZbMnbQuMJjXn2y2c5UXDC6tCUUspv5Tqhg7u23u2ybmwctJFOl3Zi6NKhNP9vc7J+zrI6NKWU8ku5T+in1Kxck1ldZjG7y2xyD+SS9kYa6Z+nk1+Qb3VoSinlk1ITujFmsjHmF2PMBg/vG2PMq8aYLcaY9caYy4MfZvjclnQbOYNyuCP5Dp5e9jRpE9PI+CkjtIXOmAH16kFUlPvvjBmhLU8pFZF8qaH/D2jn5f0bgQaFj37A64GHZa2E+ASm3zqd+V3ns//ofppPas6QT4dw7OSx4Bc2Ywb06wc7doCI+2+/fprUlVJ+KzWhi8gXgLdbAnUCporbSqCqMeb8YAVopZsvvZmNgzbSK7UXI5ePJHV8Kit+XBHcQoYOhSNHTn/tyBH360GkBwHOpL+b8kcw2tBrAz8WeZ5b+NoZjDH9jDEZxpiMvXudcSOKqhWrMqnjJBbftZijJ4/SYnILHlr8EEdOHCn9w77YudO/18tADwKCJ5wJ1m6/m+5cHEBESn0A9YANHt77EGhR5PkSoElp82zSpIk4zYFjB2TQgkFCOnLx6Ivl822fBz7TunVF3Nvr6Y+6dQOfd/iKiFjTp7uXkzEiCQkicXGnL8P4ePc0oeDv71Y01rp1gxvX9Onu7xqu7648AzLEU6729MZpE3lP6BOAbkWebwbOL22eTkzop3y27TO5aPRFQjoyaMEgOXDsgG8fLGmLC8OWYkzJicGYoBURkUr6acK5Y/Tndwv1aqSVAvsIdUJvDywCDNAcWO3LPJ2c0EVEDh0/JA8uelBMupG6/6krH2/52PsHvG1xoaxaiW6MZeVpuYVrx+jP7xbq31grBfYRUEIH3gJ2Aydwt4/3AQYAAwrfN8BrwA9ANpBW2jwlAhL6Kct3LpdLx1wqpCN95/WV347+VvKEFmZVPVwuG09JLFw/oT+/W6gTrlYK7CPgGnooHpGS0EVEjp44KoM/GSxRT0dJ7Zdqy4LNC86cyOIqTogPAiKSLzX0UO8Yff3dQp1wtVJgH5rQw2TNrjXScFxDIR256727ZN/hfX++qVUcxykpicXGuk+O2m3HGI6Eq5UCe9CEHkbHTx6XJ5c+KTHDYqTmizVlTs4c9xtaxXEkOyYxTzHZMVYVfJrQLZC5O1Maj28spCNdZnWRPYf26BanAmbHeoGu1uHlLaEb9/vhl5aWJhkZIR4jxWInCk4wasUo0pelUyWuCmNuHEPXhl0xxlgdmnKoevXcFxgVV7cubN8e7mj+vPip6MXO8fEwcSJ07x7+eMoDY8xaEUkr8T1N6KGXszeH3vN6s2rXKjpd2olx7cdxQZULrA5LOVBUlLteXpwx4LLgFrmedjAJCbBvX9jDKRe8JXQdPjcMkmoksbz3cka1HcXiHxaTPC6Z/2X+D6t2psq56tTx73VvgnEpv6cRKvbv16EBrKAJPUyio6L511X/Yv2A9Vx23mXcM+8ebpp5Ezt/D96YLSryjRjhbtIoKj7e/bovTiVxY6BHj8DHifG2Iwny+HLKB5rQw6xBQgM+7/U5Y24cw5c7vqThuIZMyJigtXU/lOdBorp3d7dP163rTsp16/reXl10sC84s+mmLIN8etuRBHF8OeUrT2dLQ/2I9F4uvtiat1WunXKtkI5cO+Va+SHvB6tDCkg4ejvYsZeHU/hyoVRZrnNLSCh5Xna8xCISeuSg3Rbty+VyycSMiVLl2SoSPyJeXl35qhS4CqwOy2/hSrR6fVbZ+TKUQVmWo1N2sk6JszSa0B1g52875cbpNwrpyNX/vVo279sclnKDVWMJV6LVQaLKzpcaekJC2dYBu9Z8i8YVHR0ZlQFN6A7hcrlkSuYUqTqyqlR8pqK88NULcqLgRMjKC2aNJVyJVmvoZefrcMBOrLWWxNfvG8x1NBw7Nk3oDvPTgZ/klrdvEdKRphObyoY9G0JSTjCTY7gSrY5ZEpji381J7d/+8nX4Y6cNYKYJ3YFcLpe8nf22VH+husQOi5Xhy4ZL/sn8oJYRzFp1ONsnI/nOPOHcmUyfHp5aq1V8OWfgxJuAaEJ3sF8O/SK3v3u7kI6kjk+VdbvXBW3ewV4BI6Fma2WTTqh3JsVvpxcbG/paq5U8/ZbR0aFZR8PV7KgJPQK8l/Oe1HyxpsQMi5F/L/m3HDtxLOB5Wl0btSMrT7qGcmfia3tyJK0D4V6/tYau/LL/yH7p+X5PIR1Jfi1ZVuWuCniekVCrDiYra+ih3Jn42p4MkbUOhLsJS9vQld8+/O5Dqf1SbYl6Okoe+fgROZJ/xOqQIoaVRy2h3JlYfTu98sLqXi566b8D3dTgJjYO2kifxn14ccWLpE5IZfnO5VaHFRECubQ+UIGO0+KNL4N3Baus8qx7d/cwxi6X+2+4hxDWhO5Q51Q8h4k3T+STHp+QX5BPyzdb8uBHD3I4/7DVoTmeVRtlKHcmJe0s4uLcw9yGe8elQkfHQ48Ah/IPMeTTIYxdM5aLzr2ISTdPok39NlaHpWxmxgz34Fs7d7pr7CNGaAJ3Ih0PPcJVjqvMmJvGsKzXMqJMFNdOvZaBCwZy4PgBq0Mrd+w8EqTVzQEq9DShR5BWdVuRNSCLf135LyZ+M5GG4xqyeMtiq8MqN4oOTytS9jHGlSorTegRJj42nlHXj2J57+VUjqtMuxnt6D2vN78e/dXq0CLe0KGn31sTyjbGuFJl5VNCN8a0M8ZsNsZsMcYMLuH9c4wxHxhjsowxG40x9wQ/VOWP5onN+ab/Nzze4nGmZk0leVwy8zfPP3NCO7cROIynGzrojR5UuJSa0I0x0cBrwI1AEtDNGJNUbLL7gBwRaQS0Bl4yxsQFOdbIF+TkWjGmIiOuG8GqvquocVYNOr3die7vdWffkX1/lqdtBEFTrVrJr5flfp/+0H2y+oOnDuqnHsCVwOIiz4cAQ4pNMwQYBxigPrAFiPI2X72wqJgQX9Fy/ORxSf8sXWKGxch5L54n7258V8eiDaLp00Xi4s5clLGxkXF1orIPArlSFOgMTCryvAcwttg0VYDPgN3AIaC9h3n1AzKAjDp16oRvCThBmJJr1s9Z0mRCEyEdKTAeLheMhKH2wszTz5eQYE25uk8OjJ2HxPCW0H1pQzclVeyLPb8ByAQuAFKBscaYs0s4GpgoImkiklajRg0fii5HwtQAm1IzhZV9V/Lcdc/x4xm/UKFQtxFEIE8/U16eNeXaqd3eaU1CTm6J9CWh5wIXFnmeCPxUbJp7gPcKdyBbgG3AX4ITYjnhKYmWNbl62YpiomIY3GIwMSNf4GhcsVUgzNd/W72xB6v8YP98di/XV05Mjo7ureSp6n7qAcQAW3G3jccBWUBysWleB9IL/68J7AKqe5uvtqEXM3DgmSMolbUx1I+G1YJpU+X3mudKAciOqkY+e6avuFyuIHyhoIZp+/Kt+i5WlOtPc4QTm4Tsft9aAh1tEbgJ+A74ARha+NoAYEDh/xcAHwPZwAbgrtLmqQm9iJK2SmPcSb4syrAVfb//e2n1ZishHbl+2vWy/dftZSs7tGHaunyr2l3tPESs3ZNjSaxeL0sTcEIPxUMTehHBXoPKuBUVuApk7KqxctaIs6Tys5Vl3OpxUuAqKFsMoQszYsp3Im/jqpe0M7F7ciyJ1UeOpfGW0PVKUTsI9pmtMjasRpko7mt2HxsGbeDKxCsZtHAQ1065li15W8oWRymsbv+1unwn8rZKltQ+HsohgUPFyiGUA+Yp04f6oTX0IrxVY8pyPB2EKobL5ZJJayfJOc+dI5WeqSQvr3hZThacLOMXDFmYISu/+P03ExLs2YUt3Hy581Hx2reduwA6EdrkYnOeMsvAgWXPeEHainJ/z5X2M9oL6ciVk66UTXs3lWk+IQ4zqOWXdv9NOx1+h5sv9ybVJivPgrG+a0J3gpJ+aZs0QLpcLpmWNU3OHXmuVBheQUZ+OVJOFJwIawzhVJZaaHnibdUs78vGm2AdkWpCd6pgnbULUjV498Hdcus7twrpSNrENFn/8/oyzcfufLn/ptZCrW8yc5pg1c+8JXQ9KWpnwThrF8QrO2pVrsXsLrOZ1XkWO37bQZOJTRi2bBgnCk74PS8782XxRvqJU18uuHL0yUMLhOWqXk+ZPtQPraH7IBhVoBA12+w9vFe6ze4mpCONXm8ka39aG9D87KS0duK4OOedJPXnIM2JNW+rz8X4Ihw1dE3odhfomhriztZzN82V80edL9FPR8vQJUPl2IljQZmv1Ruop14uCQnuERSdluz8SdA2OXXjM6fsgLQNXQUuDFtn3pE86TW3l5CO/HXsX2XljysDmp+dN1CnJTsR/2P2dg7BDr9BcXb5TbxVQoqeSI6O/jM+7eWi/BPG7Ljo+0Vy4csXStTTUfKvxf+SI/lHyjQfu2ygJXHi1aX+xuytB4tddqxF2eE3Ke2ahmBugprQy7swtl/8fux36f9BfyEdafBqA/li+xd+z8MOG6gndt7ZeOJvzKWdQ7Dbd7XDb+IthmDHpwldhd2SrUuk/iv1xaQb+fvCv8vB4wd9/qwdNlBP7Nwc5ElZYp4+3XNCt8OOtSg7/CbeKiHBrqBoQleBK0Mt/9DxQ/KPhf8Qk26k/iv1ZcnWJT4XZfUG6k0wD3jCdfBUlnLsvGMtzuqT6FpD14TuHAFm2C93fCmXjLlESEf6ze8nvx/73aci7d4NLVBO2HHZOT5PgrXuBKurp7ahq7IJVRYMQhXjSP4ReeTjRyTq6ShJfDlRFn2/KDixFXLiDsAJNWCnLddgJc+yNlOV1sslGMtRE3p5EMrqVBAbAVflrpKk15KEdOTu9++WvCN5AYfn1JqknU/+OlWwdpJ23tl6S+jG/X74paWlSUZGhiVlR6R69dyX9RdXty5s326reR8/eZxnvniG5756jhpn1WB8+/F0+ksnu4QXNk6N286iotyptzhjwOUK/3xCwRizVkTSSnpPx3KJFKEcKCLIdymoEFOB4dcOZ829a6hVuRa3vHML3eZ0Y+/hvX9O5Mfdm8MyRkYIOPHmD3YXrJuWOPbmJ56q7qF+aJNLkIX6GDFEjan5J/Nl+LLhEjssVmq8UEPe2fCOuPxsQ7Hz4XFpnNZGbXdWtqGHC9qGXg7YeQ30QfaebEmbmCakI3uqV/IrQzv8q6sgs6KXSzhpQi8v7LoG+uhEwQl5/qvnpaCkZF7K2UKHf3WlfOYtoWsbeiTp3t19Ns3lcv912MDUMVExPHr1oxQkXlDyBNWqefys0766H6cIVAQI1++tCd1OdCsHIHbkC0hs7Bmvy8GDEbFMgnjPEeUAYf29PVXdiz6AdsBmYAsw2MM0rYFMYCOwrLR5apNLMeWlIdjXtpGEhBKbXU4k1g5ZkeHi5JO4yn+2uvQfiAZ+AC4C4oAsIKnYNFWBHKBO4fPzSpuvJvRiysNW7s9Oy8NVNwUgY1eNlQJXQdCLDBe9oKh8CefgXL40uTQDtojIVhHJB94Gil8FcifwnojsLKz1/+L/sUI559TO1P4YOhSOHDn9tSNH3K8X56HD796Eity/6H7aTGnDlrwtQS0yXBzbx1mVSTh/b18Sem3gxyLPcwtfK+oS4FxjzOfGmLXGmJ7BCrDcKA9buT87LQ9X3Zw3ehKTO04m6+csUl5P4eWvX6bAVRCUIsNFLygqX8L6e3uqup96AF2ASUWe9wDGFJtmLLASOAuoDnwPXFLCvPoBGUBGnTp1yna8Eans2DYQbGW504KHxu/c33Olw8wOQjrSfFJzyfklJyhFhovd2vXtJBKXjW0G5wKuBBYXeT4EGFJsmsFAepHn/wW6eJuvtqGXIBLX5KKCvNNyuVwyY/0MqfZ8NakwvII8+8WzcqLgRCiLVCGmv1fpAk3oMcBWoD5/nhRNLjbNX4ElhdPGAxuAht7mqwndgYKxwwnBTuvngz9L51mdhXSkyYQmkvVzVqiLVEFQ0u9i1yMqOwkoobs/z03Ad7h7uwwtfG0AMKDINI/g7umyAXiwtHlqQncYB1Sd3t34rpz34nkSMyxGnvrsKTl+8rjVISkPPK1OZbhIuNzxltB1+FzlG4eM9brvyD4e+OgBZmbP5LLzLuPNTm/S5IImVoelivG0OkVHQ0EJ57httppZSofPVYGzY3eRElSPr86MW2cwv+t89h/dzxWTruDxJY9z7OQxq0NTRXhabQoKtAdQIDShK984rFvlzZfezMZBG+nZqCfPffUcjSc05usfv7Y6LFXI02pTty5MnOj+a8yfz+0+No9daEJXvnFg5+mqFasyudNkPur+EYfzD3P15Kv51+J/ceTEkdI/rELK2+rktIHW7EQTuvJN9+72rDr5MKDZDf/vBjYM2kD/Jv15eeXLpLyewrLty8Ieqr8ieaw2u65OjufpbGmoH9rLpZwJRd/BMvS8Wbp1qVw0+iIhHRm0YJAcOHbA/zLD0AfSAZ2KlEXQG1woS4UqO5Wx0/Kh44fkgUUPiEk3Uvc/deXjLR9b+z1KoP2xlSfeErp2W1ShF6oujwHemn35zuX0nt+b7/Z/R9/GfRl1/SjOqXiO5w+Eseumne86r6yl3RaVtTz1UduxI7BG4gB73lxd52oy+2fy6FWPMjlzMsnjkvnwuw89f6CkZA4h6brpsE5FyiY0oQdLJJ/BCpSnLGRMYLdxCULPm0qxlXi+7fOs7LOSqhWr0uGtDvR8vyd5R/NOn3DGDHe8JQlBlnVgpyJlB57aYkL9iKg2dD2D5V1Jy8fTqP8JCf7PO0gnKY+dOCZPLn1SYobFSM0Xa8qcnDl/vumpUduYkJ4Y1TFoVHHoSdEQ0zNYpSuenTwN2gGWZ651u9dJ4/GNhXTk9ndvlz2H9njeAYGlsTqV7qzKzltC1yaXYHDIZfGWKn61SN26nqe18nZCQGqtVFb1XcUzbZ5h7rdzSXoticO1Ekqe2Nv3cKhQtx7qTbJDRxN6MITiDFakt8l7awzescPy7x0bHcvQVkP5pt83XFztYu69ch/H4qJPnygCG7XDkWzteFvAiOGp6h7qR0Q1uQS7Db28tMknJHhverHJ9z5ZcFJeXP6i9OwcIzuqGnEZxFWnTnDjCmMbhLeiwtF6qDfJDgzahh4Gwdwgy0ubfEk7Lht/7837NkuLyS2EdKTd9Hay87edwZlxGHfgpRUVjmTrtNXbbu39mtCdpjxVYaZPL72mbqPvXeAqkDGrxshZI86SKs9WkQkZE8TlcgU20zBmuNKKCkco4dh/BSsJ2/FgWRO60zitChOo0nq92PB7b83bKtdOuVZIR66dcq1szdta9pmFcQdeWlFlSWBlSZ6hrPUGMwnbcVPUhB4ugaylRT+bkCASF2evakEoeesSaOPv7XK5ZELGBKnybBWJHxEvr658VQpcBf7PyEY1dBH/VmM71mCDuTjteLCsCT0cfF2zS9paSvpsbKw7sdul4S6UPG2B0dGO+N47f9spN06/UUhHWkxuIZv3bfZvBjZqQ/dXMJJnsGvr3uoH/s5ba+jlNaH7WvUpaWvy1IZsw6aGkLBjNc9PLpdLpmROkaojq0rFZyrKi8tflJMFJ0v/YNFb3UdH//m7W9TLxV8+1WC9FBiKn95bC56/87bjqqkJPRx8WbNLayu203FduNmtK0EZ4/npwE/S6a1OQjrS7I1msmHPBu9l2C1b+KnUekwp3zEUNeDSOk/5O2+7rZqa0MPBlzXT27Fgea6h202Aidblcslb2W9JwvMJEjc8Tp5Z9ozkn8w/c0IrjueDnJ1KXVSlfMdQtVFPnx659SRN6OHgSxLwtHInJDi+phZRgpRo9xzaI7e/e7uQjjQe31jW7V7nfqNoM0s4M06Ijgi87iNKydih3KfZsf07GDShh0tptR9vG5TdjuvKsyBXG+fkzJGaL9aUmGExMuvxW8RV2sVUoco4VmS4UsoMZatTBLRolSjghA60AzYDW4DBXqZrChQAnUubZ0QmdF9o4ra/ECS+/Uf2S4/3esi2c7wk8lBnHCv64A0cWHKZAwf+MUmo+6RH2uYWUEIHooEfgIuAOCALSPIw3VJgoSZ0ZblArwkIUdXO5e08Sqgzjg1r6Mp/3hK6L6MtNgO2iMhWEckH3gY6lTDd34E5wC8+jwymVCgEOmRg9+4wcaJ7aFxj3H8nTnS/HiDjaQTOU/clDUIZHllxGySnDC0dIaOb+pLQawM/FnmeW/jaH4wxtYG/AeODF5pSZRSM8VmLj98erERbQlI9HAtTb/8Lh/MPB6cMT0K4o/LICTdH9bcCYOfk76nqfuoBdAEmFXneAxhTbJp3geaF//8PD00uQD8gA8ioU6dOmA5QVLljx+u1iyrSHFRQ50J58+G2Qjpy0eiL5LNtn1kdXXA54cykP81CNvg+BNiGfiWwuMjzIcCQYtNsA7YXPg7hbna5xdt8tQ1dhYwD222XbV8mF4++WEhHBnwwQA4cO2B1SMFj9zOT/lQAbLBueUvovjS5rAEaGGPqG2PigK7A/GK1/PoiUk9E6gGzgUEiMte/YwWlgsSKtuIAtarbivUD1/NQ84eYsHYCDV9vyOIti60OKzhC1XwVLP40C9n8nECpCV1ETgL3A4uBTcAsEdlojBlgjBkQ6gCV8tmpts0ePaBSJUhICF9bcRDEx8bz0g0vsbz3cuJj42k3ox295/Xm16O/Wh1aZPOnAmD3cwKequ6hfmiTiwoqG7RtBtPRE0dlyKdDJPrpaDl/1Pky/9v5VocU2XxtFrLBeoaXJhfjfj/80tLSJCMjw5KyVQSqV8/dO6G4U90BHWrtT2u5Z949ZP+SzZ2X3cmr7V4lIT7B6rDKtxkz3D2mdu5018xHjAjr0Z8xZq2IpJX4niZ0FRGiotz1peKMcbfdOlh+QT7Pffkcz3z5DNUqVeO1m16jc1Jnq8NSFvGW0H05KaqU/dm9bTMAcdFxPNX6Kdb2W0vi2Yl0ebcLXd7twp5De6wOTdmMJnQVGRzYs8VfKTVTWNV3Fc9e+yzzN88neVwyM7NnYtVRtrIfTegqMlhxFaQFYqJiGNJyCJn9M2mQ0IDu73Wn09ud2HVgl9WhKRvQNnSlHKrAVcDoVaMZunQoFaIr8PINL3NP6j0YY6wOTYWQtqErFYGio6J56MqHWD9gPY1qNaLP/D60m9GOnb+XcpGLncciUQHRhK6UwzVIaMBnd3/Gaze9xvKdy0kel8z4jPG4pITePYGORKlsTZtclIog23/bzr0f3MunWz+ldb3WTLp5EhdXu/jPCSK0v355ok0uSpUT9arW4+O7PuaNm9/gm93fkDI+hdErR1PgKnBPYPOxSFRgNKErFWGMMfS9vC8bB22kdb3WPLj4QVr9rxWb922O6P76ShO6UhEr8exEFnRbwNRbprJp7yYajW/EB3c3RyK8v355pgldqQhmjKFHox7k3JfDTQ1uomPUOzzRrSb5iedHdH/98koTulLlQK3KtZhz+xze6fwOEy49SOV++xj+2dOc+OF7TeYRRBO6UuWEMYbbk28nZ1AOtyXdxpOfP0nTN5qybvc6q0NTQaIJXalypsZZNXjrtrd4/4732XN4D03faMq/l/6b4yePWx2aCpAmdKXKqVv+cgs5g3Lo0agHI74cweUTL2dV7iqrw1IB0ISuVDl2bqVzebPTmyy8cyEHjh/gqslX8cjHj3D0xFGrQ1NloAldKcWNDW5kw8AN9G3cl1FfjyJ1Qipf7fzK6rCUnzShK6UAOKfiOUy4eQKf9viU/IJ8Wr3Zin8s+geH8w9bHZrykSZ0pdRprrvoOrIHZnN/s/sZs3oMl71+GUu3LbU6LOUDTehKqTNUjqvMqze+yhe9viAmKobrpl5H/w/6c+D4AatDU15oQldKedSybksyB2Ty8JUPM2ndJJLHJfPRlo+sDkt5oAldKeVVfGw8L17/Iit6r6BKXBVunHEj98y7h1+P/mp1aKoYnxK6MaadMWazMWaLMWZwCe93N8asL3ysMMY0Cn6oSikrXZF4Bev6r+PxFo8zLWsayeOSmb95vtVhqSJKTejGmGjgNeBGIAnoZoxJKjbZNuAaEUkBhgMTgx2oUsp6FWIqMOK6Eay+dzU1zqpBp7c7ceecO9l3ZJ/VoSl8q6E3A7aIyFYRyQfeBjoVnUBEVojIqeOvlUBicMNUStnJ5edfzpp71zCs9TBm58wm6bUk3t34LlbdAU25+ZLQawM/FnmeW/iaJ32ARSW9YYzpZ4zJMMZk7N271/colVK2ExcdxxPXPMHafmupW7Uut8++nc7vdubnQz9bHVq55UtCNyW8VuJu2BjTBndCf6yk90VkooikiUhajRo1fI9SKWVbl9W8jK/7fM3I60by4XcfkvRaEtOypmlt3QIxPkyTC1xY5Hki8FPxiYwxKcAk4EYR2V+WYE6cOEFubi7Hjh0ry8eVnypWrEhiYiKxsbFWh6IcLiYqhsdaPEanv3Si97ze9Jzbk1k5sxjffjy1z/Z2QK+CyZS2FzXGxADfAdcBu4A1wJ0isrHINHWApUBPEVnhS8FpaWmSkZFx2mvbtm2jSpUqJCQkYExJBwYqWESE/fv3c/DgQerXr291OCqCFLgKGLN6DI8veZzY6Fhevv5lejfurdt0kBhj1opIWknvldrkIiIngfuBxcAmYJaIbDTGDDDGDCic7EkgARhnjMk0xmR4mJ1Xx44d02QeJsYYEhIS9GhIBV10VDQPNn+Q7IHZXH7+5fT9oC83TL+BHb/tsDq0iOdTP3QRWSgil4jIxSIyovC18SIyvvD/viJyroikFj5K3Hv4QpN5+OiyVqF0cbWLWdJzCa+3f52vc7+m4esNGbdmHC5xWR1axNIrRZVSIRNlohiQNoANAzdw1YVXcd/C+2gzpQ1b8rZYHVpE0oQegO3btzNz5sw/nmdmZrJw4cI/ns+fP5+RI0cGpaxevXoxe/ZsAPr27UtOTk5Q5qtUONStWpePun/Efzv+l6yfs0h5PYX/fP0fClwFVocWUTShB6C0hN6xY0cGDz5jpISATZo0iaSk4hfrKmVvxhh6N+7NxkEbue6i63jo44do+WZLvt33rdWhRQxfui1a4sGPHiTz58ygzjO1ViqvtHvF6zRTp05l1KhRGGNISUlh2rRp9OrViw4dOtC5c2cAKleuzKFDhxg8eDCbNm0iNTWVbt268dprr3H06FG++uorhgwZwtGjR8nIyGDs2LH06tWLs88+m4yMDH7++WdeeOEFOnfujMvl4v7772fZsmXUr18fl8tF7969/yirJK1bt2bUqFGkpaVRuXJlHnjgARYsWEClSpWYN28eNWvWZO/evQwYMICdO3cC8Morr3D11VcHbVkqVVa1z67N/K7zmZk9k3989A9Sx6eS3jqdh696mJgo26YkR9AaehEbN25kxIgRLF26lKysLEaPHu11+pEjR9KyZUsyMzN57LHHGDZsGHfccQeZmZnccccdZ0y/e/duvvrqKxYsWPBHzf29995j+/btZGdnM2nSJL7++mu/Yj58+DDNmzcnKyuLVq1a8cYbbwDwwAMP8M9//pM1a9YwZ84c+vbt69d8lQolYwzdU7qTMyiHDpd0YMiSITSf1JzsPdlWh+Zott0dllaTDoWlS5fSuXNnqlevDkC1atWCOv9bbrmFqKgokpKS2LNnDwBfffUVXbp0ISoqilq1atGmTRu/5hkXF0eHDh0AaNKkCZ988gkAn3766Wnt7AcOHODgwYNUqVIlSN9GqcDVrFyT2bfPZnbObAZ9OIgmE5swtOVQhrQcQlx0nNXhOY5tE7oVRKTErnwxMTG4XK4/psnPzy/T/CtUqHBaWUX/llVsbOwfMUdHR3Py5EkAXC4XX3/9NZUqVQpo/kqFQ+ekzrSu15oHPnqA9GXpvPfte0zuOJkmFzSxOjRH0SaXIq677jpmzZrF/v3ukQvy8vIAqFevHmvXrgVg3rx5nDhxAoAqVapw8ODBPz5f/LkvWrRowZw5c3C5XOzZs4fPP/88CN8Err/+esaOHfvH88zMzKDMV6lQqR5fnRm3zmBe13nsPbyXKyZdweNLHufYSb34zVea0ItITk5m6NChXHPNNTRq1IiHHnoIgHvvvZdly5bRrFkzVq1axVlnnQVASkoKMTExNGrUiP/85z+0adOGnJwcUlNTeeedd3wq87bbbiMxMZGGDRvSv39/rrjiCs4555yAv8urr75KRkYGKSkpJCUlMX78+IDnqVQ4dLy0IxsHbaRno54899VzXD7hclbmrrQ6LEcodSyXUClpLJdNmzbx17/+1ZJ4rHTo0CEqV67M/v37adasGcuXL6dWrVphKbu8LnPlDIu3LObeD+4l90Au/2z+T4ZfO5z42Hirw7JUQGO5qNDr0KEDqamptGzZkieeeCJsyVwpu7vh/93AhkEbGJA2gJdXvkzK6yks277M6rBsS0+K2kCw2s2VikRnVzibce3H0SWpC30/6EvrKa25r+l9jPy/kVSOq2x1eLaiNXSllCO0qd+G9QPW88AVDzBuzTgajmvIp1s/tTosW9GErpRyjLPizuKVdq/w5T1fUiGmAm2nteXe+ffy+7HfrQ7NFjShK6Uc5+o6V5PZP5NHr3qUyZmTSR6XzMLvF5b+wQinCV0p5UiVYivxfNvnWdlnJVUrVqX9zPbcPfdu8o7mWR2aZZyd0GfMgHr1ICrK/XfGDKsjUkqFWdPaTVnbby1PtHqCmdkzSXotifc3vW91WJZwbkKfMQP69YMdO0DE/bdfv7Al9SlTptCgQQMaNGjAlClTwlKmUqpkFWIqMKzNMNbcu4bzq5zPrbNu5Y7Zd7D38F6rQwsr5yb0oUPhyJHTXztyxP16iOXl5fH000+zatUqVq9ezdNPP82vv/4a8nKVUt6l1kpldd/VPNPmGd7f9D5J45J4e8PbAY+Z5BTOTeiF43z7/LoP1qxZQ0pKCseOHePw4cMkJyezYcOGM6ZbvHgxbdu2pVq1apx77rm0bduWjz76qMzlKqWCJzY6lqGthrKu/zrqV61PtznduHXWrew+uNvq0ELOuQm9Th3/XvdB06ZN6dixI//+97959NFHueuuu2jYsOEZ0+3atYsLL7zwj+eJiYns2rWrzOUqpYIv+bxkVvRZwQv/9wKLvl9E8rhkpmROiejaunMT+ogREF9sTIf4ePfrAXjyySf55JNPyMjI4NFHHy1xmpJWiJKG3VVKWSsmKoZHrn6ErAFZJNVIote8XrSf2Z4ff//R6tBCwrkJvXt3mDgR6tYFY9x/J050vx6AvLw8Dh06xMGDBzl2rORhOxMTE/nxxz9XiNzcXC644IKAylVKhc6l1S/li3u+YHS70SzbsYzkcclMXDsx4mrrOtpiMR07dqRr165s27aN3bt3nzam+Cl5eXk0adKEb775BoDLL7+ctWvXBv0OR+Fgh2WuVDht/XUrfef35bPtn3Ft/WuZdPMk6p9b3+qwfBbwaIvGmHbGmM3GmC3GmDNuY2/cXi18f70x5vJAg7bC1KlTiYmJ4c4772Tw4MGsWbOGpUuXnjFdtWrVeOKJJ2jatClNmzblySefdGQyV6o8uujci1jScwkTOkxgza41NHy9IWNWjcElLqtDC1ipNXRjTDTwHdAWyAXWAN1EJKfINDcBfwduAq4ARovIFd7ma9caenmjy1yVZzt/30m/D/qx+IfFtKzTkv92/C8NEhpYHZZXgdbQmwFbRGSriOQDbwOdik3TCZgqbiuBqsaY8wOKWimlQqzOOXVY1H0Rb3Z6k+xfskkZn8JLK16iwFVgdWhl4ktCrw0UPSWcW/iav9NgjOlnjMkwxmTs3Wv/K7iys7NJTU097XHFFV4PPJRSDmOMoVdqLzYO2sj1F1/Pw588zFWTryJnb07pH7YZX25wUVJ/vOLtNL5Mg4hMBCaCu8nFh7Itddlll+nNlZUqJy6ocgFz75jL2xve5u+L/k7jCY156pqneOSqR4iNjrU6PJ/4UkPPBS4s8jwR+KkM0yillK0ZY+h2WTdy7suh06WdGLp0KFdMuoKsn7OsDs0nviT0NUADY0x9Y0wc0BWYX2ya+UDPwt4uzYHfRSTyr7NVSkWk8846j1ldZjG7y2x2HdxF2htpPPXZU+QX5FsdmlelJnQROQncDywGNgGzRGSjMWaAMWZA4WQLga3AFuANYFCI4lVKqbC5Lek2cgbl0LVhV4Z9MYwmE5uwZtcaq8PyyKd+6CKyUEQuEZGLRWRE4WvjRWR84f8iIvcVvn+ZiGR4n2Nw6HDoSqlQS4hPYNrfpvFBtw/IO5pH8/8257FPHuPoiaNWh3YGx176b/Fw6LRr146qVavSoUOH8BSolLJUh0s6sHHQRnqn9uaFFS/QeEJjVvy4wuqwTuPYhG7hcOgAPPLII0ybNi08hSmlbKFqxaq80fENPr7rY46dPEaLyS148KMHOZx/2OrQAAcn9BAMh+7zeOgA1113HVWqVCl7YUopx2p7cVuyB2YzqOkgRq8aTcr4FD7f/rnVYTk3oYdgOHSfx0NXSqkqFaow9qaxfH735xgMbaa0YdCHgzh4/KBlMTk2oYdoOHSfxkNXSqlTrql3DesHrueh5g8xPmM8DV9vyMc/fGxJLI5N6CEaDt2n8dCVUqqo+Nh4XrrhJZb3Xk58bDw3TL+BPvP68Nux38Iah2MTOriT9/bt4HK5/waazAH69evH8OHD6d69O4899ljgM1RKlRtXXngl6/qvY/DVg5mSNYXkccks+G5B2Mp3dEIPNl/HQwdo2bIlXbp0YcmSJSQmJrJ48eIwR6uUsqOKMRV57v+eY2XflSRUSuDmt26mz7w+Ybk7kt6xqJzTZa5U6OQX5PPsl89SIboCQ1oOCco8vY2H7stoi0oppcogLjqO9NbpYStPE7oX2dnZ9OjR47TXKlSowKpVqyyKSCmlPLNdQhcRjClpePXwi/Tx0CPtjudKlXe2OilasWJF9u/fr4kmDESE/fv3U7FiRatDUUoFia1q6ImJieTm5uKE29NFgooVK5KYmGh1GEqpILFVQo+NjaV+/fpWh6GUUo5kqyYXpZRSZacJXSmlIoQmdKWUihCWXSlqjNkL7Cjjx6sD+4IYTrDYNS6wb2wal380Lv9EYlx1RaRGSW9YltADYYzJ8HTpq5XsGhfYNzaNyz8al3/KW1za5KKUUhFCE7pSSkUIpyb0iVYH4IFd4wL7xqZx+Ufj8k+5isuRbehKKaXO5NQaulJKqWI0oSulVISwdUI3xrQzxmw2xmwxxgwu4f3uxpj1hY8VxphGNomrU2FMmcaYDGNMCzvEVWS6psaYAmNMZzvEZYxpbYz5vXB5ZRpjnrRDXEViyzTGbDTGLLNDXMaYR4osqw2Fv2U1G8R1jjHmA2NMVuHyuifUMfkY17nGmPcLt8nVxpiGYYprsjHmF2PMBg/vG2PMq4VxrzfGXB5woSJiywcQDfwAXATEAVlAUrFprgLOLfz/RmCVTeKqzJ/nJ1KAb+0QV5HplgILgc52iAtoDSyw4fpVFcgB6hQ+P88OcRWb/mZgqR3iAh4Hni/8vwaQB8TZIK4XgacK//8LsCRM61gr4HJgg4f3bwIWAQZoHoz8ZecaejNgi4hsFZF84G2gU9EJRGSFiPxa+HQlEI6xYH2J65AU/mLAWUA4zjyXGlehvwNzgF/CEJM/cYWbL3HdCbwnIjsBRCQcy8zf5dUNeMsmcQlQxbjvUFMZd0I/aYO4koAlACLyLVDPGFMzxHEhIl/gXgaedAKmittKoKox5vxAyrRzQq8N/FjkeW7ha570wb23CzWf4jLG/M0Y8y3wIdDbDnEZY2oDfwPGhyEen+MqdGXhofoiY0yyTeK6BDjXGPO5MWatMaanTeICwBgTD7TDvYO2Q1xjgb8CPwHZwAMi4rJBXFnArQDGmGZAXcJT+SuNvzmuVHZO6CXdh67Emq4xpg3uhP5YSCMqLK6E186IS0TeF5G/ALcAw0MdFL7F9QrwmIgUhD6cP/gS1ze4x6doBIwB5oY6KHyLKwZoArQHbgCeMMZcYoO4TrkZWC4i3mqBweJLXDcAmcAFQCow1hhzdmjD8imukbh3zJm4j1DXEfojB1/481v7xFY3uCgmF7iwyPNE3Hv+0xhjUoBJwI0ist8ucZ0iIl8YYy42xlQXkVAOEuRLXGnA24X3bK0O3GSMOSkic62MS0QOFPl/oTFmnE2WVy6wT0QOA4eNMV8AjYDvLI7rlK6Ep7kFfIvrHmBkYXPjFmPMNtxt1qutjKtw/boH3CcigW2FD6v5lUt8Eo6TA2U8oRADbAXq8+fJjuRi09QBtgBX2Syu/8efJ0UvB3adem5lXMWm/x/hOSnqy/KqVWR5NQN22mF54W4+WFI4bTywAWhodVyF052Du332rFD/hn4sr9eB9ML/axau99VtEFdVCk/OAvfibrcO+TIrLK8enk+Ktuf0k6KrAy3PtjV0ETlpjLkfWIz7TPZkEdlojBlQ+P544EkgARhXWOs8KSEeWc3HuG4DehpjTgBHgTuk8Be0OK6w8zGuzsBAY8xJ3Murqx2Wl4hsMsZ8BKwHXMAkESmxC1o44yqc9G/Ax+I+egg5H+MaDvzPGJONO0k9JqE9yvI1rr8CU40xBbh7LfUJZUynGGPewt2Dq7oxJhd4CogtEtdC3D1dtgBHKDyKCKjMEG83SimlwsTOJ0WVUkr5QRO6UkpFCE3oSikVITShK6VUhNCErpRSEUITulJKRQhN6EopFSH+P/tNVMVfRWTpAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 画出参数更新之前的结果\n",
    "w0 = w[0].data[0]\n",
    "w1 = w[1].data[0]\n",
    "b0 = b.data[0]\n",
    "\n",
    "plot_x = np.arange(0.2, 1, 0.01)\n",
    "plot_y = (-w0.numpy() * plot_x - b0.numpy()) / w1.numpy()\n",
    "\n",
    "plt.plot(plot_x, plot_y, 'g', label='cutting line')\n",
    "plt.plot(plot_x0, plot_y0, 'ro', label='x_0')\n",
    "plt.plot(plot_x1, plot_y1, 'bo', label='x_1')\n",
    "plt.legend(loc='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. torch.optim\n",
    "\n",
    "上面的参数更新方式较为繁琐、重复，如果模型参数很多，比如有 100 个，那么需要写 100 行来更新参数。为了方便，可以写成一个函数来更新，其实 PyTorch 已经封装了一个函数来做这件事，这就是 PyTorch 中的优化器 `torch.optim`\n",
    "\n",
    "使用 `torch.optim` 需要另外一个数据类型，就是 `nn.Parameter`，默认是要求梯度的，而 tensor 默认是不求梯度的\n",
    "\n",
    "使用 `torch.optim.SGD` 可以使用梯度下降法来更新参数，PyTorch 中的优化器有更多的优化算法，后面的课程会更加详细的介绍几种常见的优化器。\n",
    "\n",
    "将参数 $w$ 和 $b$ 放到 `torch.optim.SGD` 中之后，声明一下学习率的大小，就可以使用 `optimizer.step()` 来更新参数了"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 200, Loss: 0.58470, Acc: 0.62000\n",
      "epoch: 400, Loss: 0.54856, Acc: 0.66000\n",
      "epoch: 600, Loss: 0.51801, Acc: 0.75000\n",
      "epoch: 800, Loss: 0.49200, Acc: 0.78000\n",
      "epoch: 1000, Loss: 0.46968, Acc: 0.84000\n",
      "\n",
      "During Time: 0.480 s\n"
     ]
    }
   ],
   "source": [
    "# 使用 torch.optim 更新参数\n",
    "from torch import nn\n",
    "import time\n",
    "\n",
    "# 定义优化参数\n",
    "w = nn.Parameter(torch.randn(2, 1))\n",
    "b = nn.Parameter(torch.zeros(1))\n",
    "\n",
    "# Logistic函数\n",
    "def logistic_regression(x):\n",
    "    return torch.sigmoid(torch.mm(x, w) + b)\n",
    "\n",
    "# 计算loss, 使用clamp的目的是防止数据过小而对结果产生较大影响。\n",
    "def binary_loss(y_pred, y):\n",
    "    logits = (y * y_pred.clamp(1e-12).log() + \\\n",
    "              (1 - y) * (1 - y_pred).clamp(1e-12).log()).mean()\n",
    "    return -logits\n",
    "\n",
    "# 优化器\n",
    "optimizer = torch.optim.SGD([w, b], lr=0.1)\n",
    "\n",
    "# 进行 1000 次更新\n",
    "start = time.time()\n",
    "for e in range(1000):\n",
    "    # 前向传播\n",
    "    y_pred = logistic_regression(x_data)\n",
    "    loss = binary_loss(y_pred, y_data) # 计算 loss\n",
    "    \n",
    "    # 反向传播\n",
    "    optimizer.zero_grad() # 使用优化器将梯度归 0\n",
    "    loss.backward()\n",
    "    optimizer.step() # 使用优化器来更新参数\n",
    "    \n",
    "    # 计算正确率\n",
    "    mask = y_pred.ge(0.5).float()\n",
    "    acc = (mask == y_data).sum().item() / y_data.shape[0]\n",
    "    if (e + 1) % 200 == 0:\n",
    "        print('epoch: {}, Loss: {:.5f}, Acc: {:.5f}'.format(e+1, loss.item(), acc))\n",
    "\n",
    "        during = time.time() - start\n",
    "\n",
    "print('\\nDuring Time: {:.3f} s'.format(during))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到使用优化器之后更新参数非常简单，只需要在自动求导之前使用**`optimizer.zero_grad()`** 来归 0 梯度，然后使用 **`optimizer.step()`** 来更新参数就可以了，非常方便"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面画出更新之后的结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f83301f5370>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAz+klEQVR4nO3dd3gU1frA8e9JSCjSA4oYmj/BS0ITQlEExQYoYAEFRLzSi96LelVQRANYUFFBIUAICkgUFUQQKSIIgiAQJLSIihQJImBQWihJ9vz+2IAh7Ca7m5md2d338zz7JNmdzLw75Z0zZ+aco7TWCCGECHxhVgcghBDCGJLQhRAiSEhCF0KIICEJXQghgoQkdCGECBLFrFpwpUqVdM2aNa1avBBCBKRNmzb9qbWu7OozyxJ6zZo1SUlJsWrxQggRkJRS+9x9JlUuQggRJCShCyFEkJCELoQQQUISuhBCBAlJ6EIIESQKTehKqfeUUoeVUtvdfK6UUu8opXYppbYqpRobH2aQS06GmjUhLMz5MznZ6oiEEAHIkxL6dKBdAZ+3B2rnvvoDk4oeVghJTob+/WHfPtDa+bN/f0nqQgivFZrQtdbfAkcLmORuYKZ2+h4or5S60qgAg97w4ZCZefF7mZnO94UQwgtG1KFfBezP83d67nuXUEr1V0qlKKVSjhw5YsCig8Bvv3n3vo+kVicwyXYT3jAioSsX77kcNUNrnai1jtNax1Wu7LLlaqEOnTzEY4se46/Tf/n0/7ZTvbp37/tAanUCk2w34S0jEno6UC3P39HA7wbM16WVe1cyOWUyMQkxzPtxnlmL8Z+XX4ZSpS5+r1Qp5/sGkVod3+UvIQ8e7L8Ss922m1wtBACtdaEvoCaw3c1ndwGLcZbUWwAbPJlnkyZNtK9++P0H3WhyI008ussnXfTBEwd9npctzJqldY0aWivl/DlrlqGzV0prZxnv4pdShi4m6MyapXWpUq7X3flXqVKGb64LvN1uZu5GrtaFmd9duAekaHe52t0HFyaAj4CDQBbO0ngfYCAwMPdzBUwEfgW2AXGFzVMXMaFrrfW57HP65W9f1pGjI3WFMRX0jNQZ2uFwFGmepjM5cbtTo4brxFCjhl8WH7DcrTd/rUdvtpvZCVf2IfsoUkI361XUhH5e2uE0fcO0GzTx6Paz2ut9f+8zZL6Gs7CII6Ur37grIfvrSseb7WZ2wpWrPPsI6oSutdbZOdl6/PfjdamXS+nSr5TWCRsSdI4jx7D5G8LiIo5FFwcBzeoSutaebzezE66U0O0j6BP6ebuP7ta3zbxNE49u/X5r/fOfPxu+DJ9JESfgWF2H7g2zE65c5dlHQQk9qPpyqVWhFl899BXTOk1jyx9baDC5AW989wbZjmyrQ/PL44nCWD16QGIi1KgBSjl/Dhp08d+Jic7prGb2w1Ku1oVdvrvIw12mN/tlRgk9rwPHD+h7Zt+jiUfHJcbpLX9sMXV5hZIijjCIu2oYqVYLDYRKlUt+DodDf7z9Y1359cq62Khi+oUVL+iz2WdNX65bcsSJIpJygSgooSvn5/4XFxen/TWmaEZmBo8vfZxZW2cRWzmWaZ2m0Ty6uV+WLYSRatZ0thjNr0YN2LvX39E4JSc7Gzv99puzBvHll6UqxkxKqU1a6zhXnwVVHbo7UaWi+ODeD/jywS85dvYYN7x3A/9b+j8yszIL/2chbMRPXf94TLonsJeQSOjn3Vn7TnYM3kH/xv156/u3qD+pPt/s+cbqsITwmN3urbvrnmDIEGviCXUhldAByhYvy6QOk/jm39+gUNwy8xYGfDGAY2eOWR2aEIUy8mkWI/pmcXdlkJEhpXQrhFxCP+/mmjezddBWnrr+KZI2JxGbEMvCnxdaHZYQBTLq8UGjqkoKujKQzt/8LyRuihZmw4EN9FnQh+2Ht9O9XnfGtxtP5ct8695XCDs7fwPT1Y1V8P7manIyPPSQ68+UAofD6xBFIUL+pmhhml3VjE39N/HiTS8yJ20OMQkxzN4+G6tOdqJg0o2rb/KWyt3x9uZqjx4QFeX6M2kz53+S0HNFhkcSf3M8Pwz4gVrla9F9bnfu+fgeDhw/YHVoAcMfiVaeqvCdqxuY+fmShMePN71Lf8MEfWHA3QPqZr/80bDIV1k5WfqN797QJV4qocu9Wk5P3TTV/l3zWsxfDV6kkyjfFdZ7ZFG2VyC0mQuWRlmEesMiX/2S8Qv9vujHqn2ruKXWLUztOJWrK1xtdVi25K8GL2FhzkMxP6mvLZy7bQTO7RSMDYLyNnoKC4OcnEunsbJRli+kDt1HtaNqs+LfK5h812Q2HthI/Un1Gff9OHIcLvaKAGXUJai/GrzY7TnsQOLukcdBg5y/9+wZXNUQ+avnXCVzsK5RlincFd3Nftm5ysWV/cf267uS79LEo1sktdA7Du+wOqQiM/IS1F9VIcFy2WyV/FUjgwYF7/q0oj97f1Q9EaqdcxnN4XDoWVtm6ajXonTk6Eg9etVofS77nNVh+czIJOzPRGv2QRMI9cFGCeZ7Ep6MOGXkPuqvY0ASusEOnTykH/j0AU08uuGkhnrT75usDsknRo+5EQyJ0OorAH+uw1mz3Ce6YBh3xd3JKjzcnPXrr5OjJHSTzPtxnr5y7JU6fGS4HrpsqM48l2l1SF4J5tKZr6xcJ/6+yiloNKZg2Af8fXL216BkktBNdDTzqO79eW9NPLrOu3X06n2rrQ7JY1aXRu3IypEC/TGM3PnSf3i4+2QeTPuAP694pIQeRJb9ukzXHFdTq3ilH/vyMX38zHGrQ/JIMFSTGMnKErqZJxNPxkc9/wr1fcBXUoceZE6cPaH/u+i/WsUrXf3t6nrprqVWhyS8ZOVVi5knEyue+AhFVj/lIs+hG6h0ZGnGtx/P6l6rKRVRiraz2tJrfi/+Ov2X1aEJD1k5GLKZAz178qy1XZvrB5IePZyNlBwO509/N9SShG6CltVbsnnAZp678Tk+2PIBMQkxfPbjZ1aHJTxk1UFp5snEXcOr8HD/n7iEeaTpv8k2H9xM7wW9Sf0jlc51OzPhzglUKV3F6rBEiDnfajJv51ylSkkSD0TS9N9C1115HRv6buCVW15h4c8LiZkYw8wtM7HqRCpCk5VVScJ/pITuRzv/3EmfBX1Yu38t7a5px5QOU6heTjohEUJ4TkroNvGvSv9ida/VvNPuHVbvW01sQiwJGxNwaOkmMFgEfX/bwtYkoftZmArjP83/w/bB27k++noeXfQoN0+/mZ8zfrY6NFFEMviGsJokdIvULF+TpQ8t5f2732fb4W00nNyQ1797nWxHttWhCR+5GhEoM1MGSxb+41FCV0q1U0r9pJTapZQa5uLzckqpL5RSW5RSO5RSvYwPNfgopXik0SOkDU6j/TXtGfr1UFoktWDroa1WhyZ84K8+4YVwp9CErpQKByYC7YEYoLtSKibfZI8CaVrrhsDNwJtKqUiDYw1aV5a5krkPzOWTLp+w//h+miQ24cVvXuRs9lnzFy6VvoaRwTeE1TwpoTcDdmmtd2utzwGzgbvzTaOBMkopBZQGjgJSd+AFpRT3x95P2uA0utfrzqhvR9EksQnr09ebt1Cp9DWUmS09CyLnZHGeJwn9KmB/nr/Tc9/LawJQF/gd2AYM0frSRzeUUv2VUilKqZQjR474GHIQS04mKqYJMzvPIjPxctqsPcj1067nyaVPkplVyHDtvpBKX8OVLPnP71FR5j/rLedkkZcnCV25eC//w+ttgVSgKtAImKCUKnvJP2mdqLWO01rHVa5c2ctQg1y+I7Pk74d5Z95pkk7cwtvfv039SfX5Zs83xi5TKn0Nc37zZWT8897p0+YvV87J5gjUqx5PEno6UC3P39E4S+J59QI+y+0MbBewB/iXMSGGCBdHpso8Te85u1j1yCrCVBi3zLyFAV8M4NiZY8YsUyp9DWNVYpVzsvEC+arHk4S+EaitlKqVe6OzG7Ag3zS/AbcCKKWuAK4FdhsZaNAr4MhsXaM1Wwdu5ekbniZpcxKxCbEs/Hlh0ZdpVaVvELIqsQbCOTnQSrsBfdXjrl/dvC/gTuBn4FdgeO57A4GBub9XBb7CWX++HXiosHkGY3/oReJhZ9gb0jfoegn1NPHo7nO668MnD7uen6cdM1s8woXVA2wYtXyrBsaw+6hTdo/PFStHrfIEMsBFABg06NI9yc2efzb7rB65cqSOGBWhK71eSX+07SPtcDj+mSBAjiKrwzRy+VZ+F3+fFL1ZXiCOW2v3mCWh252rbKCUM8kXYNuhbbrZ1GaaeHTHDzvq9GPpzg/svkfmsjpMo5dv9dWGP3h74rJ7adcVqwsahZGEbndFyCzZOdl67HdjdcmXSuqyr5bViSmJ2hEgR5HVYVq9/EBU0FB2rk5iVp+0fWXnk3NBCV36crGDItxRCw8L5383/I+tg7bS+MrG9F/Yn0MV3TTStdOdMqy/oWf18gNRQbukq6dBAvW+u9VDyflKErodGJBZrql4DcsfXs6UDlN47hY4FZFvAhseRVYf7FYvPxAVtkvmfxpEBtbwM3dFd7NfUuWSR0GVdj5c++0/tl+/PriR3lMOnQP6bPSV9rpmzMPqS1t3y8/7flSU82XHy29/c7WrSpWVfyF16AHAVWYpwt0Zh8Ohk7cm66jXonTk6Eg9auUofS77nOlfIxgUlrTsdIPMCud31YLq0oVrRhRgJKEHKgPuKB06eUh3/bSrJh7dYFIDnXIgxbRwg0VByUqS1j/s/jSI3Ri1vgpK6FKHbmcGND+8/LLLmd1lNvO7zefIqSM0T2rO0GVDOZ3lh45GApQnq1ea1kv9uLf80QJVErqdGfUYRnIyndr+lwNP/8EfE0qwf/LrNJrSiNX7Vhc9xiDkyeqVJ2GcAvVpECv4o3sISeh2ZsRjGHl6GlJaU+nIKT5YXJz264/SenprHlv0GCfOnjA27gDnarXnFQpPwgRa/yuBwC+PybqrizH7JXXoHirqXRQ3FcI51avpIYuHaBWvdPW3q+slvywxIfjAVdBTLoMG2bfRiTve7EaBWDdu9dNSnvBHHbok9GBXSHPItb+t1XUn1NXEo/897986IzPD4oCd7HqABmqy8ybmQGvdGUjbRJ5yEUXjwdF5Ouu0Hr58uA4fGa6veOMKPTdtrmXham3vAzTQkp3W3sfsrgwA9tgG+dllmxSUrI0soEhCD2VeZMfNBzfrxlMaa+LRnT/urA+eOGhBwPY5QF0JxP5fvI25oMc27XJizcsO26SwtoFGFlAkoYc6L4oHWTlZ+tXVr+rio4vrCmMq6Ombp1/cNa8f2OEAdcfOJxt3vI25sIZVdvuudtgmBcVgdHyS0IXXfjzyo245raUmHt32g7Z639/7/LZsOxyg7ti5OsgdX2KeNct9QrfDiTUvO2yTggohRhdQJKELn+Q4cvQ737+jL3v5Ml36ldJ6wvoJOseRY/py7XCAFsSuN2wL4kvMdj6x5mf1NpESuiT0gLHnrz36lYH1LnT2dS66qulHjNUHqL/Y+Xva/cRqJ1KHLgk9cMyapR359shzxSN09gczrY4soAVCwrTzCcdu5CkXERjcXDMeqBihUw+mWh1dwAqkKo1AYlTytOvJrKCELk3/g4lZ7bXddDZR5WgWcVPjGLFiBGezzxqzLB8FYlN1f/TtEWry9HSB1q5HUfLnfPzOXaY3+yUldIOZef3upiiZXS1a9/yspyYeXXdCXb1u/7qiL8sHgVB14YqU0I1n1Dq187ZBqlxCgJl7YCEZc9HPi3S1t6ppFa/044sf1yfPniz6Mr1g54OvIIF6IrIzox4RtHNbiIISulS5BAszr98L6fi6fe32bB+8nYFxAxm3fhz1J9Vn+e7lRV+uhwK16kL6EzeeUT0aBuoA4pLQg4XZe2AhHV+XLV6WhLsSWPXIKsLDwrntg9vot6Aff5/527fleVEpHqgHH0h/4kYzauDvgB1A3F3R3eyXVLkYzEbX75nnMvUzXz2jw0aG6apvVtXzd873bgZefhcbfXVhA6H8lIsk9GBisz1w44GNun5CfU08utucbvrwycOe/aMPleI2++pCmKaghK6cn/tfXFycTklJsWTZwn/O5ZxjzJoxvPTtS5QrUY532r1Dt3rdUEq5/6ewMGcKz08pZ92EECFMKbVJax3n6jOpQxemigyP5IWbXmDzgM1cXeFqHvzsQe6efTcHjh9w/0+BXCnuoUB8bl74zl/bWxK6nQTxUR57eSxre6/lrTve4uvdXxOTEMPUTVNxeYXo6o6UUnDnnf4J1mQB22hF+MSv29tdXYzZL6lDzyeE7uztytil20xvo4lHt5neRu/K2HXpRIMGXfowcJCsj0B9bl74xp+9LUoJ3S6GD4fMzIvfy8x0vh9MkpP5v8a3srzXSo5PrkitReuoP6k+b697mxxHzj/TLVp0aT26j+vDbhc+gfrcvPCNP7e3RwldKdVOKfWTUmqXUmqYm2luVkqlKqV2KKVWGRtmCAiFozzPtafSmjJ/HGXqF4qRv1/Lk189Scv3WrLj8A7ntAatDztWb4TALQKRhz+3d6EJXSkVDkwE2gMxQHelVEy+acoDCUAnrXUscL/xoQa5UDjKXVyFhJ0+zVMLjzLr3lnsOrqL66Zcx+hVo9HVqrmeh5frw44XPgHbaEX4xK/b211dzPkXcD2wNM/fzwLP5ptmMPBSYfPK+5I69HxCoQ69kA4yDp08pLvN6aaJRw/tVU1nlyxR5PVh1z455Ln50GKb/tCBLkBSnr97AhPyTTMOZyl+JbAJeNjNvPoDKUBK9erVff9GwSrYj3IP7w7N3zlfXzn2Sv3gfeijl5fVjiKsD7kBKYJNQQndkzp0Vy1A8j9rVgxoAtwFtAVGKKXquLgaSNRax2mt4ypXruzBokNMsHfs4eG1Z6drO5H2aBol/92HioOPc+34a/h21Uyf1odUbwQeu93EDiSeJPR0IG+FZjTwu4tplmitT2mt/wS+BRoaE6IIGl50L1i+RHmSOiXxdc+vyXZkc9P0m3j0y0c5cfaEWYsUNmDHm9gBxV3RXf9TTVIM2A3UAiKBLUBsvmnqAstzpy0FbAfqFTRfqUMPQBZVCZ08e1IPWTxEq3ilq71VTS/+ZbFflivM5Wp3kiqywlHUzrmAO4GfgV+B4bnvDQQG5pnmaSAtN5k/Xtg8JaEHGBvctF3721pdd0JdTTz64XkP6z9P/em3ZQtjududXCVzO9zEtpOCErp0ziU8U7Om8/o3vxo1nPX9fnI2+ywvffsSY74bQ8WSFZl450S6xHTx2/KFMdztTuHhkJNz6ft+3s1sTTrnEkVnk4ZPxYsVZ/Qto0npl0J02Wju//R+On/SmYMnDvo1DlE07nabnBy5iV0UktCFZ2zW8KlhlYas77ueMbeO4cufvyQmIYbpqdOx6opTeMfdbnP+prXcxPaNJHThGRs+/1csrBhDbxzKloFbqHd5PXrN70X75Pbs+9vFtbywlYJ2p2B/etdMktCFZ2z8/N+1la5l1SOrmNB+Amt+W0NsQiwTNkzAoWUwDLuy8e4U0OSmqAgq+/7ex4CFA1j661JurH4j0zpNo07UJW3cAkJysrPPmd9+c1ZRnC+9itAmN0VF8MrXrLDGl2tY3GMx0++ezo7DO2gwqQGvrXmNbEe2Kcszq8WLNLARPnH3PKPZL3kOPcSY0SipkGfjD544qO/7+D5NPLrxlMY69WCqqcszkjSwEe4gz6ELS50vbubtx7ZUqaJXmnr4bPzctLk8uuhRMk5nMKzlMJ5v/TzFixU3bXlGkHGyhTsFVblIQhfmMysRepH1jp4+yhNLn2DmlpnUrVSX9+5+jxbRLUxbXlHZpB2XsCGpQxfWMqtRkhfPxlcsWZEZ98xgcY/FnDx3khum3cATS57g1LlTni+vYkXv4igCGz4lKgKAJHSjSJ+f7rlLeBUrFm2d+ZD12l3Tjh2DdzAobhDj1o+j/qT6LN+9vPBlJSfD8eOXvh8ZaUqWlcf6hE/cVa6b/Qqqm6I26LjK1lytn8hIrSMiir7OinCzddXeVbr2O7U18ei+8/vqv07/5X5id3cpo6K8i1eIIkJuippMKjwLl/+h6pMnISPj0umiouDPP/0W1ums08SvjGfsurFUKV2FSXdNotO1nS6dUO5SCpuQOnSz2aTjKlvL35776FHX02Vk+LW6qmRESV67/TXW911PpVKVuHv23XSf250jp45cPKHN+rIJdFJDaQ5J6EaQg917Ba2b4cP9F0euuKpxbOy3kVE3j2Ju2lzqTqzLh9s+/KezL7lLaRhpNGUeSehGkIPdewWtG4uubCLDIxlx0wg2D9jMNRWvocdnPeg0uxPpx9ND6i6l2aXn4cMvbpIAzr8tOI8HH3eV62a/guqmqNbGt4S0aLg3v4qKcn2jMTzc8u+dnZOt31r7li75Ukld9tWyekrKFO1wOMxboE22tz/u7yvlerPLqESeoahD0JnxCrqEbqRQeWrG1ffM/7L4e+/K2KVvmXGLJh7dZnobvStjl/EL8fP2Lujc4Y8uB6Rbg6KRhB5oQmmPL2hkYJt8b4fDoRNTEnXZV8vqki+V1G+ufVNn52QbtwA/bu/Czh3+KD0HWnnFJhdPF0hCDzShdk1aWEndJt87/Vi67vhhR008uvnU5nr7oe3GzNiP27uwc4e/zi1mJ0mj5m/Hk48k9EATSiV0rW1fQs/L4XDoj7Z9pCu9XklHjIrQo1aO0mezzxZtpn7c3oWdO3xJYHYrwRqZhO14KEpCDwR5j4qoKGdLSjsVC8zkLsvY+HsfPnlYd5/TXROPbjCpgd54YKPvM7NZt7zeJGg7lmCNTMJ2vFiWhO4vvhZVXB0VERHOxG6XYo+Z3B2B4eG2/94Ldi7QVd+sqsNGhulnvnpGZ57L9G1GfirmGp2A7ViCNTIJ2/H7SUL3B0+PFFcHrh33Gn+yYzHPC3+f/lv3W9BPE4+u/U5tvWrvKqtDKpCR5w4jkqfR5zIjywd23DUlofuDp9eyrvYOd9UNNrkZ6Bd2q4j1wfLdy/XV46/WxKMHLxysj585Xvg/Bfj39qgsUsB3NCNhFnSP3c/9v5lCEro/eFJUKajoEMoldDvy8Sg+efakfmLJE1rFK13trWp68S+LC16G3Yp/Xir0KxQygVkXp7NmBe9hJQndHzzZMwu7+RfAB3ZQMSDRrtu/TtedUFcTj+75WU/956k/L50oSKraCjz3FfIdzbzpaMcbmkaQhO4PniSBgnZuu13XhTKDEu2ZrDP6+eXP62KjiunL37hcf7rj04snsCLj+Hs/K+Q7mnlOC5Lz5SUkoftLYQdLEFxihwSDE23qwVTdZEoTTTz6vo/v00eTJhT87L1ZGceK/a+QrGpmSMF6uElCtxMpidufCUW7rJws/dqa1/TDXcL1qQg3idzsjGNFkXXWrEvbVERGXnJj1KxDIhgPt4ISuoxYJER+5zvsztvHa6lShnSXm1XtKiLSf3f9YY0azm6FzeqS14pRl5KToVcvyMr6572ICHj//aDsetgfijxikVKqnVLqJ6XULqXUsAKma6qUylFKdfE1WCEMUZROvU3s+zziwEGX72ulnCM5mZnkrBiIZfjwi5M5OP+Wzs9NUWhCV0qFAxOB9kAM0F0pFeNmuteApUYHKYRXjBgSJ/+QeUYlWjfJ81DFSH768ydjluGOFQOxBMrwjEEyJp4nJfRmwC6t9W6t9TlgNnC3i+n+A8wFDhsYnxDes/OQOC6SanaJSEbcFk7DyQ0Zs2YMWTlZbv65iKwYdSkQhmcMojHxPEnoVwH78/ydnvveBUqpq4B7gckFzUgp1V8plaKUSjly5EhBkwrhOzuXCl0k1WJJ7zE66Vc61OnAs8ufpXlSc1L/SDVv+WZcebgTCMMzelsAsHNp3t3d0vMv4H4gKc/fPYF3803zKdAi9/fpQJfC5huyT7kI8wXwA8hzdszRV7xxhQ4fGa6f+/o5fTrrtNUhFZ3dHzXx5jFVGzwLSQFPuXhSQk8HquX5OxrIf5s+DpitlNoLdAESlFL3+HaKEaKIAqFU6EbnmM6kPZpGz4Y9eWXNK1w35TrW7V9ndVhF4++rAm95Uy1k5+o8PKty2QjUVkrVUkpFAt2ABXkn0FrX0lrX1FrXBOYAg7XWnxsdrBAesaKu2EAVS1bk/bvfZ0mPJWRmZdLyvZY8vuRxTp07ZXVowcmbAoCdq/PwIKFrrbOBx3A+vfIj8InWeodSaqBSaqDZAQrhE7uXCj3Q9pq2bB+0ncFNBzN+/XjqTarH8t3LrQ4r+HhTALD7TV53dTFmv6QOXRjO7nW1RfDt3m91nXfraOLRfeb30X+d/svqkEJTENShC2F/QfTomSutarQidUAqQ1sOZXrqdGImxjB/53yrwwo9Nq/Ok6b/IjjUrOlM4vnVqOGscgkim37fRO8Fvdl6aCtdY7vybvt3qXxZZavDEn5S5Kb/QtiezW9WGalJ1Sak9EthdJvRzNs5j7oT6/Lhtg+xqnAm7EMSuggOdr9ZZbCI8Aieb/08mwds5pqK19Djsx50/Kgj6cfTrQ5NWEgSuggOAfzseVHEVI7hu97f8dYdb7FizwpiE2JJ3JSIQ5vUe6KwNUnoIjjY/GaVmcLDwnni+ifYPng7cVXjGLBwALfOvJVdR3dZHZrwM7kpKkQQ0VqT9EMSTy17iqycLEa3Gc3jLR4nPCzc6tCEQeSmqBAhQilFvyb9SBucxm1X38ZTy57ihvduYPvh7VaHJvxAEroQQeiqslcxv9t8PrzvQ3b/tZvGUxozcuVIzuWcszo0YSJJ6EIEKaUU3et3J21wGl1iuhC/Kp64xDh+nTDavt2/iiKRhC5EkKt8WWU+7PwhC7otoNWa/VR58oWgbVEb6iShCxEiOl7bkXdXl+Gy/AMi2aj7V1E0ktCFCCFh6a4bHukgbFEbiiShCxFK3LScPVA+jEW/LPJzMMJoktCFCCUuWtTmlCzB+E5XcNeHd/HwvIfJyMywKDhRVJLQhQglLlrUhk9N4qWk3YxoPYKPtn9ETEIMn+74VDr7CkDSUlQIccHWQ1vpPb83mw5u4t5/3cvEOydyZZkrrQ5L5CEtRYUQHmlwRQO+7/s9r932Got3LSYmIYb3N78vpfUAIQldCHGRYmHFeKblM2wZuIX6l9en94LetJ3Vlr1/77U6NFEISehCCJfqRNVh5SMrmXjnRNalr6NeQj3eXf+udM1rY5LQhRBuhakwBjcdzI7BO2hVoxX/XfJfWr/fmp1/7rQ6NOGCJHQhRKGql6vOogcXMeOeGaQdSaPR5Ea8uvpVsnLyNzsVVpKELoTwiFKKhxs+TNqjaXSo04HnVjxH86TmbD642erQRC5J6EIIr1QpXYU5D8xhzv1z+P3E7zSd2pThy4dzJvuM1aGFPEnoQgifdI7pTNqjaTzU4CFeWfMK1025jrX711odVkiThC6E8FnFkhWZfs90lvRYQmZWJje+dyP/XfxfTp47aXVoIclWLUWzsrJIT0/nzBm5dPOHEiVKEB0dTUREhNWhiCBw4uwJnlv+HBM2TqBm+Zokdkjk9v+73eqwgk5BLUVtldD37NlDmTJliIqKQillSVyhQmtNRkYGJ06coFatWlaHI4LI6n2r6ftFX37O+JnejXrzZts3KV+ivNVhBY2Aafp/5swZSeZ+opQiKipKroaE4VrVaEXqgFSGtRzGjC0ziJkYw+c7P7c6rJBgq4QOSDL3I1nXwiwlI0ry6m2vsr7vei6/7HLu/fheHvj0AQ6dPGR1aEHNdgldCBE8mlRtwsZ+G3mpzUvM/2k+MQkxzNo6Szr7MolHCV0p1U4p9ZNSapdSapiLz3sopbbmvtYqpRoaH6r97N27lw8//PDC36mpqSxa9M+oLwsWLGDMmDGGLOuRRx5hzpw5APTt25e0tDRD5iuE2SLCIxjeejibB2ymTlQdes7rSYePOrD/2H6rQws6hSZ0pVQ4MBFoD8QA3ZVSMfkm2wPcpLVuAIwGEo0O1I4KS+idOnVi2LBLzn9FlpSURExM/k0ghL3FVI5hTa81jGs7jpV7VxKbEMuUlCnS2ZeBinkwTTNgl9Z6N4BSajZwN3ChiKi1ztua4HsguqiBPb7kcVL/SC3qbC7SqEojxrUbV+A0M2fOZOzYsSilaNCgAR988AGPPPIIHTp0oEuXLgCULl2akydPMmzYMH788UcaNWpE9+7dmThxIqdPn2bNmjU8++yznD59mpSUFCZMmMAjjzxC2bJlSUlJ4Y8//uD111+nS5cuOBwOHnvsMVatWkWtWrVwOBz07t37wrJcufnmmxk7dixxcXGULl2aIUOGsHDhQkqWLMn8+fO54oorOHLkCAMHDuS33MF/x40bR8uWLQ1bl0L4IjwsnCEthtDx2o70/6I/A78cyOwds5nacSrXVLzG6vACnidVLlcBea+N0nPfc6cPsNjVB0qp/kqpFKVUypEjRzyP0k927NjByy+/zIoVK9iyZQvjx48vcPoxY8bQqlUrUlNTGTp0KKNGjaJr166kpqbStWvXS6Y/ePAga9asYeHChRdK7p999hl79+5l27ZtJCUlsW7dOq9iPnXqFC1atGDLli20bt2aqVOnAjBkyBCeeOIJNm7cyNy5c+nbt69X8xXCTFdXuJplPZeR1DGJzQc302BSA95c+yY5jhyrQwtonpTQXT0K4fKOhlKqDc6EfqOrz7XWieRWx8TFxRV4V6SwkrQZVqxYQZcuXahUqRIAFStWNHT+99xzD2FhYcTExHDokPNu/5o1a7j//vsJCwujSpUqtGnTxqt5RkZG0qFDBwCaNGnCsmXLAPj6668vqmc/fvw4J06coEyZMgZ9GyGKRilFn8Z9aF+7PYO+HMRTy57ik7RPmNZpGvUur2d1eAHJkxJ6OlAtz9/RwO/5J1JKNQCSgLu11gE5bLjW2uWjfMWKFcPhcFyY5ty5cz7Nv3jx4hctK+9PX0VERFyIOTw8nOzsbAAcDgfr1q0jNTWV1NRUDhw4IMlc2FLVMlX5vOvnzO48mz1/7aHxlMaMXDmSczm+HWehzJOEvhGorZSqpZSKBLoBC/JOoJSqDnwG9NRa/2x8mP5x66238sknn5CR4TwfHT16FICaNWuyadMmAObPn09WlrMP6DJlynDixIkL/5//b0/ceOONzJ07F4fDwaFDh1i5cqUB3wTuuOMOJkyYcOHv1NRUQ+YrhBmUUnSt15W0R9O4P/Z+4lfFE5cYx8YDG60OLaAUmtC11tnAY8BS4EfgE631DqXUQKXUwNzJXgCigASlVKpSKsXN7GwtNjaW4cOHc9NNN9GwYUOefPJJAPr168eqVato1qwZ69ev57LLLgOgQYMGFCtWjIYNG/L222/Tpk0b0tLSaNSoER9//LFHy+zcuTPR0dHUq1ePAQMG0Lx5c8qVK1fk7/LOO++QkpJCgwYNiImJYfLkyUWepxBmq1SqEsn3JfNF9y84evooLaa14OmvniYzK9Pq0AKCrfpy+fHHH6lbt64l8Vjp5MmTlC5dmoyMDJo1a8Z3331HlSpV/LLsUF3nwv6OnTnGM8ueIfGHRK6peA1JHZO4qeZNVodluYDpyyVUdejQgUaNGtGqVStGjBjht2QuhJ2VK1GOKR2nsOLhFTi0g5tn3MyghYM4fva41aHZlidPuQiTGVVvLkQwalOrDdsGbWPEihGMWz+Ohb8sZEqHKdxZ+06rQ7MdKaELIWyvVEQp3mz7Jmt7r6Vs8bLc9eFd9JzXk4zMgHygzjSS0IUQAaN5dHN+6P8DL7R+gdnbZ1N3Yl0+2fGJdPaVSxK6ECKgFC9WnJFtRrKp/yZqlK9B1zldue+T+zh44qDVoVlOEroQIiA1uKIB6/qs4/XbXmfJriXUnViX9za/F9Kl9cBO6MnJULMmhIU5fyYnWx2REMKPioUV4+mWT7Nl4BYaVmlInwV9aDurLXv/3mt1aJYI3ISenAz9+8O+faC182f//n5L6jNmzKB27drUrl2bGTNm+GWZQgjX6kTV4Zt/f0PCnQmsS19HvYR6vLP+nZDr7CtwE/rw4ZCZr/VYZqbzfZMdPXqUkSNHsn79ejZs2MDIkSP566+/TF+uEMK9MBXGoKaD2DF4B61qtGLIkiG0nt6anX/utDo0vwnchJ7bz7fH73tg48aNNGjQgDNnznDq1CliY2PZvn37JdMtXbqU22+/nYoVK1KhQgVuv/12lixZ4vNyhRDGqV6uOoseXMTMe2ay88+dNJrciFdXv0pWTpbVoZkucBN69ereve+Bpk2b0qlTJ55//nmeeeYZHnroIerVu7QbzwMHDlCt2j8dUEZHR3PgwAGflyuEMJZSip4Ne5I2OI1O13biuRXP0TypOZsPbrY6NFMFbkJ/+WUoVeri90qVcr5fBC+88ALLli0jJSWFZ555xuU0ru6iu+p2VwhhrStKX8En93/CZw98xsGTB2k6tSnDlw/nTPYZq0MzReAm9B49IDERatQApZw/ExOd7xfB0aNHOXnyJCdOnODMGdcbPTo6mv37/xnEKT09napVqxZpuUII89xb917SBqfxcMOHeWXNK1w35TrW7l9b+D8GGOltMZ9OnTrRrVs39uzZw8GDBy/qU/y8o0eP0qRJE3744QcAGjduzKZNmwwf4cgf7LDOhfCnr379iv5f9Oe3Y7/xn2b/4eVbX6Z0ZGmrw/KY9LbooZkzZ1KsWDEefPBBhg0bxsaNG1mxYsUl01WsWJERI0bQtGlTmjZtygsvvBCQyVyIUHTH/93B9sHbeazZY7y74V3qT6rPsl+XWR2WIaSEHuJknYtQtua3NfRd0JefMn6id6PejL1jLBVKVrA6rAJJCV0IIVy4sfqNpA5MZVjLYczYMoPYhFg+3/m51WH5TBJ6AbZt20ajRo0uejVv3tzqsIQQBipRrASv3vYqG/pt4PLLLufej+/lgU8f4NDJQ1aH5jUZ4KIA9evXl8GVhQgRja9szMZ+G3lj7RuMXDWS5XuWM77deHrU7xEwjyVLCV0IIXJFhEfwXKvnSB2QyrVR19JzXk86fNSB/cf2F/7PNiAJXQgh8qlbuS6re61mXNtxrNy7ktiEWCanTMahHVaHViBJ6EII4UJ4WDhDWgxh+6DtNLuqGYO+HMQtM27hl4xfrA7NrYBO6NIduhDCbLUq1GJZz2VM6zSN1D9SaTC5AWPXjiXbkW11aJcI2IRucXfotGvXjvLly9OhQwf/LFAIYRmlFL2v603ao2m0/b+2PL3saW6YdgPbDm2zOrSLBGxCt7A7dACefvppPvjgA/8sTAhhC1XLVGVe13nM7jybvX/vpXFiY1785kXO5ZyzOjQggBO6Cd2he9wfOsCtt95KmTJlfF+YECIgKaXoWq8raY+m0TW2K6O+HUXjKY3ZcGCD1aEFbkI3oTt0j/tDF0KISqUqMeu+WXzR/Qv+PvM310+7nqe+eorMrMzC/9kkAZvQTeoO3aP+0IUQ4rwOdTqwY/AO+jXux5vr3qTh5Ias3LvSklgCNqGb1B26R/2hCyFEXuVKlGNyh8mseHgFWmvazGjDoIWDOH72uF/jCNiEDs7kvXcvOBzOn0VN5gD9+/dn9OjR9OjRg6FDhxZ9hkKIkNGmVhu2DtrK/67/H4k/JBKbEMuXP3/pt+UHdEI3mqf9oQO0atWK+++/n+XLlxMdHc3SpUv9HK0Qwo5KRZRi7B1jWddnHeWKl6PDRx3oPb+3y6ErjeZRf+hKqXbAeCAcSNJaj8n3ucr9/E4gE3hEa/1DQfOU/tDtQda5EOY5l3OOV1a/QvHw4jzb6llD5llQf+iF9raolAoHJgK3A+nARqXUAq11Wp7J2gO1c1/NgUm5P4UQImRFhkcSf3O835bnSfe5zYBdWuvdAEqp2cDdQN6EfjcwUzuL+98rpcorpa7UWh80PGI/2rZtGz179rzoveLFi7N+/XqLIhJCCPc8SehXAXn7jkzn0tK3q2muAi5K6Eqp/kB/gOpuHhjXWtum7+Fg7w/dquEHhRDm8OSmqKvsmj8TeDINWutErXWc1jqucuXKl/xDiRIlyMjIkETjB1prMjIyKFGihNWhCCEM4kkJPR2olufvaOB3H6YpVHR0NOnp6Rw5csTbfxU+KFGiBNHR0VaHIYQwiCcJfSNQWylVCzgAdAMezDfNAuCx3Pr15sAxX+rPIyIiqFWrlrf/JoQQAg8SutY6Wyn1GLAU52OL72mtdyilBuZ+PhlYhPORxV04H1vsZV7IQgghXPFokGit9SKcSTvve5Pz/K6BR40NTQghhDekpagQQgQJj1qKmrJgpY4A+3z890rAnwaGYxS7xgX2jU3i8o7E5Z1gjKuG1vrSxwSxMKEXhVIqxV3TVyvZNS6wb2wSl3ckLu+EWlxS5SKEEEFCEroQQgSJQE3oiVYH4IZd4wL7xiZxeUfi8k5IxRWQdehCCCEuFagldCGEEPlIQhdCiCBh64SulGqnlPpJKbVLKTXMxec9lFJbc19rlVINbRLX3bkxpSqlUpRSN9ohrjzTNVVK5SilutghLqXUzUqpY7nrK1Up9YId4soTW6pSaodSapUd4lJKPZ1nXW3P3ZYVbRBXOaXUF0qpLbnryy9dgHgQVwWl1LzcY3KDUqqen+J6Tyl1WCm13c3nSin1Tm7cW5VSjYu8UK21LV84+435FbgaiAS2ADH5prkBqJD7e3tgvU3iKs0/9ycaADvtEFee6Vbg7Mqhix3iAm4GFtpw/yqPcyCX6rl/X26HuPJN3xFYYYe4gOeA13J/rwwcBSJtENcbwIu5v/8LWO6nfaw10BjY7ubzO4HFOLsfb2FE/rJzCf3CSEla63PA+ZGSLtBar9Va/5X75/c4u+21Q1wnde4WAy7DRd/wVsSV6z/AXOCwH2LyJi5/8ySuB4HPtNa/AWit/bHOvF1f3YGPbBKXBsrkjjFcGmdCz7ZBXDHAcgCt9U6gplLqCpPjQmv9Lc514M6Fkd601t8D5ZVSVxZlmXZO6O5GQXKnD86zndk8ikspda9SaifwJdDbDnEppa4C7gUm4z+ebsfrcy/VFyulYm0SVx2gglJqpVJqk1LqYZvEBYBSqhTQDucJ2g5xTQDq4hwLYRswRGvtsEFcW4D7AJRSzYAa+KfwVxhvc1yh7JzQPRoFCUAp1QZnQh9qakS5i3PxnqvRmeZprf8F3AOMNjsoPItrHDBUa51jfjgXeBLXDzj7p2gIvAt8bnZQeBZXMaAJcBfQFhihlKpjg7jO6wh8p7UuqBRoFE/iagukAlWBRsAEpVRZc8PyKK4xOE/MqTivUDdj/pWDJ7zZ1h7xqPtci3g0CpJSqgGQBLTXWmfYJa7ztNbfKqX+TylVSWttZidBnsQVB8zOHbO1EnCnUipba/25lXFprY/n+X2RUirBJusrHfhTa30KOKWU+hZoCPxscVzndcM/1S3gWVy9gDG51Y27lFJ7cNZZb7Ayrtz9qxc4b0QCe3JfVjNkpLeL+OPmgI83FIoBu4Fa/HOzIzbfNNVxDqpxg83iuoZ/boo2xjnSk7I6rnzTT8c/N0U9WV9V8qyvZsBvdlhfOKsPludOWwrYDtSzOq7c6crhrJ+9zOxt6MX6mgTE5/5+Re5+X8kGcZUn9+Ys0A9nvbXp6yx3eTVxf1P0Li6+KbqhqMuzbQldezZS0gtAFJCQW+rM1ib3rOZhXJ2Bh5VSWcBpoKvO3YIWx+V3HsbVBRiklMrGub662WF9aa1/VEotAbYCDiBJa+3yETR/xpU76b3AV9p59WA6D+MaDUxXSm3DmaSGanOvsjyNqy4wUymVg/OppT5mxnSeUuojnE9wVVJKpQMvAhF54jJ8pDdp+i+EEEHCzjdFhRBCeEESuhBCBAlJ6EIIESQkoQshRJCQhC6EEEFCEroQQgQJSehCCBEk/h/dYJ5K1qagJAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 画出更新之后的结果\n",
    "w0 = w[0].data[0]\n",
    "w1 = w[1].data[0]\n",
    "b0 = b.data[0]\n",
    "\n",
    "plot_x = np.arange(0.2, 1, 0.01)\n",
    "plot_y = (-w0.numpy() * plot_x - b0.numpy()) / w1.numpy()\n",
    "\n",
    "plt.plot(plot_x, plot_y, 'g', label='cutting line')\n",
    "plt.plot(plot_x0, plot_y0, 'ro', label='x_0')\n",
    "plt.plot(plot_x1, plot_y1, 'bo', label='x_1')\n",
    "plt.legend(loc='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到更新之后模型已经能够基本将这两类点分开了"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. PyTorch的Loss函数\n",
    "\n",
    "前面使用了自己写的 loss函数，其实 PyTorch 已经提供了一些常见的 loss函数，比如线性回归里面的 loss 是 `nn.MSE()`，而 Logistic 回归的二分类 loss 在 PyTorch 中是 `nn.BCEWithLogitsLoss()`，关于更多的 loss，可以查看[文档](https://pytorch.org/docs/stable/nn.html#loss-functions)\n",
    "\n",
    "PyTorch 实现的 loss函数有两个好处：第一是方便使用，不需要重复造轮子；第二就是其实现是在底层 C++ 语言上的，所以速度上和稳定性上都要比自己实现的要好。\n",
    "\n",
    "另外，PyTorch 出于稳定性考虑，将模型的 Sigmoid 操作和最后的 loss 都合在了 `nn.BCEWithLogitsLoss()`，所以我们使用 PyTorch 自带的 loss 就不需要再加上 Sigmoid 操作了"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.9880)\n"
     ]
    }
   ],
   "source": [
    "# 使用自带的loss\n",
    "criterion = nn.BCEWithLogitsLoss() # 将 sigmoid 和 loss 写在一层，有更快的速度、更好的稳定性\n",
    "\n",
    "w = nn.Parameter(torch.randn(2, 1))\n",
    "b = nn.Parameter(torch.zeros(1))\n",
    "\n",
    "def logistic_reg(x):\n",
    "    return torch.mm(x, w) + b\n",
    "\n",
    "optimizer = torch.optim.SGD([w, b], 1.)\n",
    "\n",
    "y_pred = logistic_reg(x_data)\n",
    "loss = criterion(y_pred, y_data)\n",
    "print(loss.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 200, Loss: 0.40936, Acc: 0.86000\n",
      "epoch: 400, Loss: 0.32933, Acc: 0.87000\n",
      "epoch: 600, Loss: 0.29321, Acc: 0.87000\n",
      "epoch: 800, Loss: 0.27238, Acc: 0.87000\n",
      "epoch: 1000, Loss: 0.25875, Acc: 0.87000\n",
      "\n",
      "During Time: 0.313 s\n"
     ]
    }
   ],
   "source": [
    "# 同样进行 1000 次更新\n",
    "\n",
    "start = time.time()\n",
    "for e in range(1000):\n",
    "    # 前向传播\n",
    "    y_pred = logistic_reg(x_data)\n",
    "    loss = criterion(y_pred, y_data)\n",
    "    \n",
    "    # 反向传播\n",
    "    optimizer.zero_grad()\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    # 计算正确率 0.5以上的判断为正确\n",
    "    mask = y_pred.ge(0.5).float()        \n",
    "    acc = (mask == y_data).sum().item() / y_data.shape[0]\n",
    "    if (e + 1) % 200 == 0:\n",
    "        print('epoch: {}, Loss: {:.5f}, Acc: {:.5f}'.format(e+1, loss.item(), acc))\n",
    "\n",
    "during = time.time() - start\n",
    "print()\n",
    "print('During Time: {:.3f} s'.format(during))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到，使用了 PyTorch 自带的 loss 之后，速度有了一定的上升，虽然看上去速度的提升并不多，但是这只是一个小网络，对于大网络，使用自带的 loss 不管对于稳定性还是速度而言，都有质的飞跃，同时也避免了重复造轮子的困扰"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}