{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 逻辑斯蒂回归模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上一节课我们学习了简单的线性回归模型，这一节我们会学习第二个模型：逻辑斯蒂回归模型(Logistic Regression)。\n",
    "\n",
    "逻辑斯蒂回归是一种广义的回归模型，其与多元线性回归有着很多相似之处，模型的形式基本相同，虽然也被称为回归，但是其更多的情况使用在分类问题上，同时又以二分类更为常用。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 模型形式\n",
    "\n",
    "逻辑斯蒂回归的模型形式和线性回归一样，都是 $y = wx + b$，其中 $x$ 可以是一个多维的特征，唯一不同的地方在于逻辑斯蒂回归会对 $y$ 作用一个 logistic 函数，将其变为一种概率的结果。 \n",
    "\n",
    "$$\n",
    "h_\\theta(x) = g(\\theta^T x) = \\frac{1}{1+e^{-\\theta^T x}}\n",
    "$$\n",
    "\n",
    "Logistic 函数作为 Logistic 回归的核心，我们下面讲一讲 Logistic 函数，也被称为 Sigmoid 函数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Sigmoid 函数\n",
    "Sigmoid 函数非常简单，其公式如下\n",
    "\n",
    "$$\n",
    "f(x) = \\frac{1}{1 + e^{-x}}\n",
    "$$\n",
    "\n",
    "Sigmoid 函数的图像如下"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEICAYAAAC3Y/QeAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAip0lEQVR4nO3deZQU5dn+8e8tyCLIoqCiLIqCSlwBt6hvQIiCxt0YPMpPjUg0IZpETVATQ9CTRD2Jr764oLgvwARhHA2iYAYToiiCSlgEUXFBlsgqiAww9++Pp5Bm7JlpZrq7erk+59SZrq6nq68pmrtrnqp6ytwdEREpLrvEHUBERLJPxV9EpAip+IuIFCEVfxGRIqTiLyJShFT8RUSKkIq/5AUze8DMfleH13U0s/Vm1qAOr73NzL4ws2U7+9r6qOvvKrIzTOf5S7qZ2WJgkLtPydf3NrOOwAKgk7uvSEe2at7nMkLekzL1HiLJaM9fJLmOwMpMFn6ROKn4S9aYWWMz+18z+zya/tfMGics/7WZLY2WDTIzN7ODomWPmdlt0eM2ZvaCma0xs1Vm9i8z28XMniQU7eejrp5fm9n+0XoaRq/dw8wejd5jtZmVJsnZF5gM7But5zEz62Vmn1Vptzhqi5kNM7MSM3vCzL40s7lm1jOhbQczG29m/zWzlWY2wswOBR4ATojeZ03V3zWav9LMFkW/a5mZ7ZuwzM3sKjN7P9oe95qZ1fffSgqfir9k083A8cBRwJHAscBvAcysH/AroC9wENCrhvVcB3wGtAX2Bm4C3N0HAp8AZ7p7c3e/I8lrnwR2A74D7AXcVbVB1GXUH/g8Ws9lKf5+ZwFjgFZAGTAi+t0aAC8AHwP7A/sBY9x9PnAV8Hr0Pq2qrtDMTgH+BFwItIvWMaZKsx8AxwBHRO1OSzGvFDEVf8mmi4Hh7r7C3f8L/AEYGC27EHjU3ee6+1fAsBrWs5lQCDu5+2Z3/5encPDKzNoRivpV7r46eu2r9fmFqpjm7hPdfSvhS+bI6PljgX2BG9x9g7t/7e7TUlznxcAj7j7L3TcBNxL+Utg/oc2f3X2Nu38ClBO+XEVqpOIv2bQvYc91m4+j57Yt+zRhWeLjqu4EFgEvm9mHZjY0xffvAKxy99Uptt9ZiWcFfQU0ibqbOgAfu/uWOqxzh23m7uuBlYS/Hqp73+Z1eB8pMir+kk2fA50S5jtGzwEsBdonLOtQ3Urc/Ut3v87dOxO6Wn5lZn22La7h/T8F9jCzVjsbHNhA6C4CvunKaZviaz8FOm477lBFbX+x7LDNzKwZsCewJMX3FklKxV8yZVcza5IwNQRGA781s7Zm1ga4BXgqal8CXG5mh5rZbkC157mb2Q/M7KDowOZaYCtQGS1eDnRO9jp3Xwq8CNxnZq3NbFcz+58Uf5+FhD35M8xsV8Kxisa1vGabNwlfbn82s2bR9jgxIW97M2tUzWtHE7bLUdHB8T8Cb7j74hTfWyQpFX/JlInAxoRpGHAb8BYwG/gPMCt6Dnd/EbiH0Ge9CJgerWdTknV3AaYA64HXgfvcvTxa9ifCF8waM7s+yWsHEo4ZvAesAH6Ryi/j7muBnwKjCHvdGwgHnVN57VbgTMKB7E+i1/0oWvwPYC6wzMy+SPLaKYQvwmcJXyAHAgNSeV+RmugiL8lJ0WmQc4DGdewrF5EaaM9fcoaZnRtdC9AauB14XoVfJDNqLf5m9oiZrTCzOdUsNzO7J7oIZbaZdU9/TCkSPyF0xXxA6Me/Ot44IoWr1m6f6IDYeuAJdz8syfLTgZ8DpwPHAXe7+3EZyCoiImlS656/u/8TWFVDk7MJXwzu7tOBVtHFNCIikqOSnXe8s/ZjxwtyPoueW1q1oZkNBgYDNGnSpEfHjh3T8PaZVVlZyS675P6hEeVMn3zICPmV02wXtm7dhS1bjMpKqKy0hAm2bt3+uLLScAf38PPb8+FxsTFzunRZz8KFC79w91SvMalWOop/ytz9QeBBgIMPPtgXLFiQzbevk6lTp9KrV6+4Y9RKOdMnHzJCbuR0h+XL4eOPw7R4cfi5bFl4fsUKWLp0Cxs21FxqGjeGFi3C1KwZNGkCTZvu+LPq48aNoWHD1KYGDXac32WXMJlt//nOO2/To8fRmO34fE2Pa1uWOMRe1eH2dnaZGXTsCGb2MWmQjuK/hB2vxmyPrj4UKSju8NFH8J//wPz5MG9emObPh6++2rFtq1aw336w117QvTscfvgyundvz157hedatdpe6LdNjVO9XC6j1nLyyXFnyJ50FP8yYIiZjSEc8F0bXUkpInnqyy9hxgyYPh1efz38/CLhErT27aFbN7jySujSBTp12j61aLHjuqZOXUSvXu2R3FJr8Tez0YThddtE45n/HtgVwN0fIFzJeTrhqsyvgMszFVZEMsMd3n4bJk0K0+uvw5boCotDDoEf/ACOPx6OPjrMVy3wkn9qLf7uflEtyx34WdoSiUhWuMNbb0FJSZg++SQ8f/TRcP310KsXHHsstG4da0zJkKwe8BWR+K1aBY8/DiNHwoIFsOuucOqp8Ic/QP/+sPfecSeUbFDxFykS8+fDHXfA6NGwaRN897vw8MNw7rnauy9GKv4iBW72bLjtNhg3LpwqefnlcNVVcOSRtb9WCpeKv0iBWroUfvMbePJJ2H13GDoUfvlLaFvvy4OkEKj4ixSYigq4+24YPjw8/s1v4Ne/hj32iDuZ5BIVf5ECMns2XHwxzJkTTs+86y446KC4U0kuyv2BQUSkVpWV8Je/wDHHhIuxysrg+edV+KV62vMXyXP//S9cdBG88gqccw48+KD69aV2Kv4ieWzuXDjzzHBw96GH4Iorvj1ImEgy6vYRyVNvvLEHJ5wAGzfCq6/CoEEq/JI6FX+RPPTww3DTTYdz4IFhALZjj407keQbFX+RPDNyZNjL79lzFdOmhRE2RXaWir9IHrn//nB17hlnwK23zqFZs7gTSb5S8RfJE/ffDz/9aTjA++yz0KiRxx1J8pjO9hHJA2Vl8LOfhQu3xo2DRo3iTiT5Tnv+Ijlu5sxwHn/PnjB2rAq/pIeKv0gO+/TT0M3Tpk3Y+99tt7gTSaFQt49IjtqwIRT+DRvg3/+GffaJO5EUEhV/kRx17bVhoLaJE+Gww+JOI4VG3T4iOWj06HAh1403Qr9+caeRQqTiL5JjPvgAfvKTcJvFYcPiTiOFSsVfJIdUVIQzexo0gGeeCTdXF8kE9fmL5JBbbw1j9YwbB506xZ1GCpn2/EVyxJw58Oc/wyWXwPnnx51GCp2Kv0gOqKyEwYOhZUv461/jTiPFQN0+Ijlg5Eh4/XV4/HHdhUuyQ3v+IjFbsgSGDoU+fWDgwLjTSLFQ8ReJ2S9+Ec7yeeAB3YlLskfFXyRG//pXOLPnppvgoIPiTiPFRMVfJCbucMMNsO++cN11caeRYqMDviIxGTcO3ngjDOOg0Tol27TnLxKDioowbs/hh8Oll8adRoqR9vxFYnD//WEMnxdfDEM5iGSb9vxFsmztWhg+HPr2hdNOizuNFCsVf5EsGzECVq2C22/XqZ0Sn5SKv5n1M7MFZrbIzIYmWd7RzMrN7G0zm21mp6c/qkj+W78e7ror3Ii9e/e400gxq7X4m1kD4F6gP9ANuMjMulVp9lugxN2PBgYA96U7qEghGDkSVq6Em2+OO4kUu1T2/I8FFrn7h+5eAYwBzq7SxoEW0eOWwOfpiyhSGDZuhDvvDH39xx8fdxopdubuNTcwuwDo5+6DovmBwHHuPiShTTvgZaA10Azo6+4zk6xrMDAYoG3btj1KSkrS9XtkzPr162nevHncMWqlnOmTqYwTJuzHPfd04a673uaoo9bWe335sC1BOdOtd+/eM929Z71X5O41TsAFwKiE+YHAiCptfgVcFz0+AZgH7FLTert27er5oLy8PO4IKVHO9MlExk2b3Nu3dz/pJPfKyvSsMx+2pbtyphvwltdSt1OZUjnPfwnQIWG+ffRcoiuAftGXyetm1gRoA6yo21eSSGF58kn47DMYNUpn+EhuSKXPfwbQxcwOMLNGhAO6ZVXafAL0ATCzQ4EmwH/TGVQkX7mHM3yOOgpOPTXuNCJBrXv+7r7FzIYALwENgEfcfa6ZDSf8+VEGXAc8ZGa/JBz8vSz680Sk6L3yCsydC48+qr1+yR0pDe/g7hOBiVWeuyXh8TzgxPRGEykM99wT7s41YEDcSUS20xW+Ihn0wQfwwgtw1VXQpEncaUS2U/EXyaD/+z9o2BCuvjruJCI7UvEXyZB16+CRR+DCC6Fdu7jTiOxIxV8kQx57DL78Eq69Nu4kIt+m4i+SAe5w771hGIdjjok7jci36WYuIhnwz3/CwoXwxBNxJxFJTnv+IhkwahS0bAnnnx93EpHkVPxF0mz16nBz9osv1o3ZJXep+Iuk2dNPw9dfw6BBcScRqZ6Kv0gaucNDD0GPHnD00XGnEameir9IGs2cCbNna69fcp+Kv0gajRoFTZvCRRfFnUSkZir+ImmyYQM880y4ordly7jTiNRMxV8kTSZMCFf0XnFF3ElEaqfiL5ImTz4J++8PJ50UdxKR2qn4i6TB0qUwZQpccolu2CL5QcVfJA3GjIHKynBhl0g+UPEXSYOnnoKePeGQQ+JOIpIaFX+Repo3D2bNCl0+IvlCxV+knp56Cho00D16Jb+o+IvUQ2VlGMvn+9+HvfeOO41I6lT8Reph2jT45BMYODDuJCI7R8VfpB6efhqaNYOzz447icjOUfEXqaMtW+DZZ+HMM8MXgEg+UfEXqaPycli5En70o7iTiOw8FX+ROho7Fpo3h3794k4isvNU/EXqYPNmGD8+9PU3aRJ3GpGdp+IvUgevvBLu1XvhhXEnEakbFX+ROigpgRYt4LTT4k4iUjcq/iI7qaIijN1/zjnQuHHcaUTqRsVfZCdNmQJr1qjLR/Kbir/ITiopgVatwpAOIvlKxV9kJ1RUQGlp6PJp1CjuNCJ1p+IvshNefRXWroXzzos7iUj9pFT8zayfmS0ws0VmNrSaNhea2Twzm2tmz6Q3pkhuKC2F3XaDvn3jTiJSPw1ra2BmDYB7ge8DnwEzzKzM3ecltOkC3Aic6O6rzWyvTAUWiUtlJTz3XLiit2nTuNOI1E8qe/7HAovc/UN3rwDGAFXHMLwSuNfdVwO4+4r0xhSJ38yZsGRJ6O8XyXfm7jU3MLsA6Ofug6L5gcBx7j4koU0psBA4EWgADHP3SUnWNRgYDNC2bdseJSUlafo1Mmf9+vU0b9487hi1Us70qS7jqFEHMHp0RyZM+DctWmyJIdmO8mFbgnKmW+/evWe6e896r8jda5yAC4BRCfMDgRFV2rwATAB2BQ4APgVa1bTerl27ej4oLy+PO0JKlDN9qsvYrZv7KadkN0tN8mFbuitnugFveS11O5UplW6fJUCHhPn20XOJPgPK3H2zu39E+CugS12/kERyzcKF4Ubt6vKRQpFK8Z8BdDGzA8ysETAAKKvSphToBWBmbYCuwIfpiykSr9LS8FN37JJCUWvxd/ctwBDgJWA+UOLuc81suJmdFTV7CVhpZvOAcuAGd1+ZqdAi2VZaCt27Q8eOcScRSY9aT/UEcPeJwMQqz92S8NiBX0WTSEFZuhSmT4c//CHuJCLpoyt8RWrx/PPgrv5+KSwq/iK1KC2FAw+Eww6LO4lI+qj4i9Rg3bpw165zzgGzuNOIpI+Kv0gNJk0KI3mqy0cKjYq/SA1KS6FtWzjhhLiTiKSXir9INSoq4O9/h7POggYN4k4jkl4q/iLVmDo19Pmry0cKkYq/SDVKS6FZM+jTJ+4kIumn4i+SRGVlKP4au18KlYq/SBIzZoQre9XlI4VKxV8kidLScJD3jDPiTiKSGSr+IkmUlkKvXtC6ddxJRDJDxV+kik8+2Y333lOXjxQ2FX+RKv797z0Bjd0vhU3FX6SKadPa0KMHdOhQe1uRfKXiL5Jg6VKYN6+lunyk4Kn4iyQoi25Qeu658eYQyTQVf5EEpaWw335f0a1b3ElEMkvFXySybez+k076QmP3S8FT8ReJTJwImzfDiSd+EXcUkYxT8ReJlJbCXntBt27r4o4iknEq/iLApk1hz19j90uxUPEXAcrL4csvdVWvFA8VfxE0dr8UHxV/KXqVlfDcc9C/PzRpEncakexQ8Zei9+absGyZunykuKj4S9ErLYWGDeH00+NOIpI9Kv5S1Nxh/HiN3S/FR8Vfitq8efD++3DeeXEnEckuFX8pahMmhJ8au1+KjYq/FLXx4+GEE2DffeNOIpJdKv5StBYvhrff1vDNUpxU/KVobevyUfGXYqTiL0VrwgQ4/HA46KC4k4hkn4q/FKXly2HaNJ3lI8UrpeJvZv3MbIGZLTKzoTW0O9/M3Mx6pi+iSPqVlYVz/NXlI8Wq1uJvZg2Ae4H+QDfgIjP71k3uzGx34FrgjXSHFEm3CROgc2c44oi4k4jEI5U9/2OBRe7+obtXAGOAZGdF3wrcDnydxnwiabd2LUyZEvb6dbtGKVYNU2izH/BpwvxnwHGJDcysO9DB3f9uZjdUtyIzGwwMBmjbti1Tp07d6cDZtn79euVMo1zI+core7F5czc6dZrF1KnfvmtXLmRMhXKmV77kTBt3r3ECLgBGJcwPBEYkzO8CTAX2j+anAj1rW2/Xrl09H5SXl8cdISXKmboLLnDfZx/3rVuTL8+FjKlQzvTKl5zAW15LfU1lSqXbZwnQIWG+ffTcNrsDhwFTzWwxcDxQpoO+kos2boQXXwzDOeyic92kiKXy8Z8BdDGzA8ysETAAKNu20N3Xunsbd9/f3fcHpgNnuftbGUksUg+TJ8OGDTrFU6TW4u/uW4AhwEvAfKDE3eea2XAzOyvTAUXSafx4aNkyDOEsUsxSOeCLu08EJlZ57pZq2vaqfyyR9KuoCDduOeccaNQo7jQi8VKvpxSNyZPDaZ4XXhh3EpH4qfhL0Rg7Flq1gr59404iEj8VfykKX38Nzz0XLuxSl4+Iir8UiZdfhnXr1OUjso2KvxSFkhLYYw/o0yfuJCK5QcVfCt7GjaHL57zzYNdd404jkhtU/KXgTZoE69ery0ckkYq/FLySEthzT+jdO+4kIrlDxV8K2oYN8PzzocunYUqXNIoUBxV/KWjPPRe+AC6+OO4kIrlFxV8K2lNPQYcOcPLJcScRyS0q/lKwli8P5/dffLGGbxapSv8lpGCNHQtbt8LAgXEnEck9Kv5SsJ56Co4+Grp1izuJSO5R8ZeCtGABzJgBl1wSdxKR3KTiLwXp6adDP/9FF8WdRCQ3qfhLwXEPXT59+0K7dnGnEclNKv5ScF57DT76SF0+IjVR8ZeC88gj0Lx5GLtfRJJT8ZeCsm4djBkT+vqbN487jUjuUvGXgjJmDHz1FQwaFHcSkdym4i8FZdQoOPxwOOaYuJOI5DYVfykY774bzu0fNAjM4k4jkttU/KVgjBoFjRvrLB+RVKj4S0HYuDGc23/++eFevSJSMxV/KQjjx8OaNTrQK5IqFX8pCA88AAceCN/7XtxJRPKDir/kvVmzYNo0+NnPNG6/SKr0X0Xy3t13Q7Nm8OMfx51EJH+o+EteW748XNh12WXQsmXcaUTyh4q/5LWRI6GiAn7+87iTiOQXFX/JWxUVcP/90L8/HHxw3GlE8ouKv+StkhJYtgyuvTbuJCL5R8Vf8pJ7ONB7yCFw6qlxpxHJPykVfzPrZ2YLzGyRmQ1NsvxXZjbPzGab2Stm1in9UUW2mzoV3noLrrlG4/iI1EWtxd/MGgD3Av2BbsBFZtatSrO3gZ7ufgQwDrgj3UFFEt12G+yzD1x+edxJRPJTKnv+xwKL3P1Dd68AxgBnJzZw93J3/yqanQ60T29Mke1eew3+8Q+44QZo0iTuNCL5ydy95gZmFwD93H1QND8QOM7dh1TTfgSwzN1vS7JsMDAYoG3btj1KSkrqGT/z1q9fT/M8uCVUMeUcOvRw3ntvd0aPnk7TppVpSrZdMW3LbFDO9Ordu/dMd+9Z7xW5e40TcAEwKmF+IDCimraXEPb8G9e23q5du3o+KC8vjztCSool58yZ7uD+xz+mJ08yxbIts0U50wt4y2upr6lMDVP4flgCdEiYbx89twMz6wvcDHzP3TfV4/tIpFq33QatWoVxfESk7lLp858BdDGzA8ysETAAKEtsYGZHAyOBs9x9RfpjisCcOTBhQjjDp0WLuNOI5Ldai7+7bwGGAC8B84ESd59rZsPN7Kyo2Z1Ac+BvZvaOmZVVszqROrvpplD0r7km7iQi+S+Vbh/cfSIwscpztyQ87pvmXCI7ePVVeP55+NOfYM89404jkv90ha/kvMpKuP566NBBQzmIpEtKe/4icRo7NlzN+/jj0LRp3GlECoP2/CWnbdoU+vqPOgouuSTuNCKFQ3v+ktNGjIDFi2HyZN2iUSSd9N9Jctann8KwYWG8/r46pUAkrVT8JSe5w5AhsHVr2PsXkfRSt4/kpAkToKwM7rgDOneOO41I4dGev+SctWvDXv9RR8Evfxl3GpHCpD1/yTk33gjLl8Nzz0FDfUJFMkJ7/pJTJk8ON2UfMgSOOSbuNCKFS8Vfcsby5TBwIBx6aBjGQUQyR39US06orIRLLw39/ZMnw267xZ1IpLCp+EtO+Otf4aWXQpfP4YfHnUak8KnbR2I3fXo4yHv++fCTn8SdRqQ4qPhLrBYvhrPPho4d4aGHwCzuRCLFQcVfYrN2LZxxBlRUwN//Dq1bx51IpHioz19isXkz/PCHsHBh6Os/5JC4E4kUFxV/ybrKytC3P3kyPPIInHJK3IlEio+6fSSrKith8GB49FH4/e/h8svjTiRSnLTnL1lTWQl33nkwkybB734Xir+IxEN7/pIVW7bAFVfApEnt+P3vYfhwndkjEicVf8m4NWvCWT2PPQaXXfYRw4bFHEhE1O0jmbVoEZx5Zvg5ahQceODHwAFxxxIpetrzl4x5+WU47jhYsQKmTAndPiKSG1T8Je02boRrr4XTToN27eDNN+F734s7lYgkUvGXtHrnnTAO/z33wDXXwIwZcOCBcacSkapU/CUtVq2Cn/8cevQIjydNgrvvhqZN404mIsmo+Eu9bN4MI0dC165w331w9dUwZ07o8hGR3KXiL3VSURFG4Tz4YLjqKvjOd2DWLBgxAvbYI+50IlIbFX/ZKStXwl/+AgcdFIZpaNMGyspg6lQ48si404lIqnSev9SqshKmTQt7+n/7G2zaBCefHOZPPVVX6orkIxV/SWrr1nCHrZISGDcOPv8cWrSAK68MI3IedljcCUWkPlT8BQD3cFet8vIwvv7kybB6NTRuDKefDhdeGK7UbdYs7qQikg4q/kVqxQqYPRvefRdefx1eew2WLg3L2rULt1bs1w/69w97/CJSWFT8C9imTfDxx/DBB/Dhh+HnnDmh6C9fvr3d/vtD795w4omhL/+ww9SPL1LoUir+ZtYPuBtoAIxy9z9XWd4YeALoAawEfuTui9MbVdxh/fpw79s1a8L0xRdhj3369P155hlYtizML10a+undt7++aVM49NDQjXPEEdunNm3i+o1EJC61Fn8zawDcC3wf+AyYYWZl7j4vodkVwGp3P8jMBgC3Az9KZ9BtRcx9+1R1PpU2O/uaVasafdMdUlkZDoRu2bLjlOy56p6vqAhj3yROX39d/XPr1m0v9GvXhnUmY9aJtm1Dl80++4S9906doHPnMLxC587hee3Riwiktud/LLDI3T8EMLMxwNlAYvE/GxgWPR4HjDAzc0/c79zR++/vTpMmqRXgeH034+/QpEnYK982Jc63axf21lu1ClPLltsft2oVLqhq1w7mz/8nffpo9DQRSU0qxX8/4NOE+c+A46pr4+5bzGwtsCfwRWIjMxsMDI5mN23aZHPqEjrL2lDl90i3r78O0+rV9VpNxnOmST7kzIeMoJzpli85D07HSrJ6wNfdHwQeBDCzt9y9Zzbfvy6UM73yIWc+ZATlTLd8ypmO9aQyvMMSoEPCfPvouaRtzKwh0JJw4FdERHJQKsV/BtDFzA4ws0bAAKCsSpsy4NLo8QXAP2rq7xcRkXjV2u0T9eEPAV4inOr5iLvPNbPhwFvuXgY8DDxpZouAVYQviNo8WI/c2aSc6ZUPOfMhIyhnuhVVTtMOuohI8dGQziIiRUjFX0SkCGW0+JvZD81srplVmlnPKstuNLNFZrbAzJLe9C86yPxG1G5sdMA5o6L3eSeaFpvZO9W0W2xm/4napeXUq51hZsPMbElC1tOradcv2saLzGxoDDnvNLP3zGy2mU0ws1bVtMv69qxt25hZ4+jzsCj6HO6fjVxVMnQws3Izmxf9X7o2SZteZrY24bNwS7ZzRjlq/De04J5oe842s+4xZDw4YTu9Y2brzOwXVdrEsj3N7BEzW2G2/fonM9vDzCab2fvRz9bVvPbSqM37ZnZpsjbf4u4Zm4BDCRckTAV6JjzfDXgXaAwcAHwANEjy+hJgQPT4AeDqTOZN8v5/AW6pZtlioE0281R5/2HA9bW0aRBt285Ao2ibd8tyzlOBhtHj24Hbc2F7prJtgJ8CD0SPBwBjY/h3bgd0jx7vDixMkrMX8EK2s+3svyFwOvAiYMDxwBsx520ALAM65cL2BP4H6A7MSXjuDmBo9Hhosv8/wB7Ah9HP1tHj1rW9X0b3/N19vrsvSLLobGCMu29y94+ARYRhJL5hZgacQhguAuBx4JwMxt1B9P4XAqOz9Z4Z8M3QHO5eAWwbmiNr3P1ld98SzU4nXCeSC1LZNmcTPncQPod9os9F1rj7UnefFT3+EphPuKI+H50NPOHBdKCVmbWLMU8f4AN3/zjGDN9w938SzpZMlPgZrK4GngZMdvdV7r4amAz0q+394urzTzZkRNUP9J7AmoTCkaxNJp0MLHf396tZ7sDLZjYzGrYiDkOiP58fqebPwVS2czb9mLDnl0y2t2cq22aHYUuAbcOWxCLqdjoaeCPJ4hPM7F0ze9HMvpPdZN+o7d8w1z6PA6h+5y4XtifA3u4eDS3JMmDvJG3qtF3rPbyDmU0B9kmy6GZ3f66+68+EFDNfRM17/Se5+xIz2wuYbGbvRd/cWckJ3A/cSvgPdyuhi+rH6Xz/VKWyPc3sZmAL8HQ1q8n49sxnZtYceBb4hbuvq7J4FqHrYn107KcU6JLliJBH/4bR8cOzgBuTLM6V7bkDd3czS9u5+fUu/u7etw4vS2XIiJWEPwsbRntdydrUSW2ZLQxRcR7h/gTVrWNJ9HOFmU0gdCOk9YOe6rY1s4eAF5IsSmU711sK2/My4AdAH486KZOsI+Pbs4qdGbbkM4tx2BIz25VQ+J929/FVlyd+Gbj7RDO7z8zauHtWBylL4d8wK5/HFPUHZrn78qoLcmV7RpabWTt3Xxp1ka1I0mYJ4TjFNu0Jx1lrFFe3TxkwIDqb4gDCt+qbiQ2iIlFOGC4CwvAR2fpLoi/wnrt/lmyhmTUzs923PSYc1MzqCKVV+krPreb9UxmaI6Ms3Ajo18BZ7v5VNW3i2J55MWxJdIzhYWC+u/+1mjb7bDsWYWbHEv5fZ/VLKsV/wzLg/0Vn/RwPrE3o0si2av+yz4XtmSDxM1hdDXwJONXMWkfdv6dGz9Usw0evzyX0P20ClgMvJSy7mXC2xQKgf8LzE4F9o8edCV8Ki4C/AY0zmTchw2PAVVWe2xeYmJDr3WiaS+jeyPaZAU8C/wFmRx+QdlVzRvOnE84Q+SCmnIsI/ZHvRNMDVXPGtT2TbRtgOOGLCqBJ9LlbFH0OO8ew/U4idO3NTtiGpwNXbfuMAkOi7fYu4aD6d2PImfTfsEpOI9wY6oPos9sz2zmjHM0IxbxlwnOxb0/Cl9FSYHNUN68gHGN6BXgfmALsEbXtSbir4rbX/jj6nC4CLk/l/TS8g4hIEdIVviIiRUjFX0SkCKn4i4gUIRV/EZEipOIvIlKEVPxFRIqQir+ISBH6//1zJnK5PI8iAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "plt.figure()\n",
    "plt.axis([-10,10,0,1])\n",
    "plt.grid(True)\n",
    "X=np.arange(-10,10,0.1)\n",
    "y=1/(1+np.e**(-X))\n",
    "plt.plot(X,y,'b-')\n",
    "plt.title(\"Logistic function\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "可以看到 Sigmoid 函数的范围是在 0 ~ 1 之间，所以任何一个值经过了 Sigmoid 函数的作用，都会变成 0 ~ 1 之间的一个值，这个值可以形象地理解为一个概率，比如对于二分类问题，这个值越小就表示属于第一类，这个值越大就表示属于第二类。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "另外一个 Logistic 回归的前提是确保你的数据具有非常良好的线性可分性，也就是说，你的数据集能够在一定的维度上被分为两个部分，比如\n",
    "\n",
    "![linear_sep](imgs/linear_sep.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到，上面绿色的点和蓝色的点能够几乎被一个黑色的平面分割开来"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 损失函数\n",
    "前一节对于回归问题，我们有一个 loss 去衡量误差，那么对于分类问题，我们如何去衡量这个误差，并设计 loss 函数呢？\n",
    "\n",
    "Logistic 回归使用了 Sigmoid 函数将结果变到 0 ~ 1 之间，对于任意输入一个数据，经过 Sigmoid 之后的结果我们记为 $\\hat{y}$，表示这个数据点属于第二类的概率，那么其属于第一类的概率就是 $1-\\hat{y}$。如果这个数据点属于第二类，我们希望 $\\hat{y}$ 越大越好，也就是越靠近 1 越好，如果这个数据属于第一类，那么我们希望 $1-\\hat{y}$ 越大越好，也就是 $\\hat{y}$ 越小越好，越靠近 0 越好，所以我们可以这样设计我们的 loss 函数\n",
    "\n",
    "$$\n",
    "loss = -(y * log(\\hat{y}) + (1 - y) * log(1 - \\hat{y}))\n",
    "$$\n",
    "\n",
    "其中 y 表示真实的 label，只能取 {0, 1} 这两个值，因为 $\\hat{y}$ 表示经过 Logistic 回归预测之后的结果，是一个 0 ~ 1 之间的小数。如果 y 是 0，表示该数据属于第一类，我们希望 $\\hat{y}$ 越小越好，上面的 loss 函数变为\n",
    "\n",
    "$$\n",
    "loss = - (log(1 - \\hat{y}))\n",
    "$$\n",
    "\n",
    "在训练模型的时候我们希望最小化 loss 函数，根据 log 函数的单调性，也就是最小化 $\\hat{y}$，与我们的要求是一致的。\n",
    "\n",
    "而如果 y 是 1，表示该数据属于第二类，我们希望 $\\hat{y}$ 越大越好，同时上面的 loss 函数变为\n",
    "\n",
    "$$\n",
    "loss = -(log(\\hat{y}))\n",
    "$$\n",
    "\n",
    "我们希望最小化 loss 函数也就是最大化 $\\hat{y}$，这也与我们的要求一致。\n",
    "\n",
    "所以通过上面的论述，说明了这么构建 loss 函数是合理的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 程序示例\n",
    "\n",
    "下面我们通过例子来具体学习 Logistic 回归"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x7f51cb512370>"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "from torch.autograd import Variable\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "# 设定随机种子\n",
    "torch.manual_seed(2021)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们从 `data.txt` 读入数据。读入数据点之后我们根据不同的 label 将数据点分为了红色和蓝色，并且画图展示出来了"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f51c805e590>"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAfqElEQVR4nO3dfYxd9X3n8fd3jI01jRPAnkqRxx5PKrPBEKTiMU3+yCZqltahWbN5qmwMDWpShzQklZomgCAWMhqlrVat2oVdyYlowOOC2PyxclU2FiJB0WYD9RAeDYIYY8w4SJmMm92Q4OWh3/3j3IE7d+7Dufeeh9/vnM9LuvLce4/v+d1zzv2e3/n+Ho65OyIiEr+RsgsgIiLZUEAXEakIBXQRkYpQQBcRqQgFdBGRijirrBWvW7fON23aVNbqRUSi9Mgjj/zc3cfavVdaQN+0aROzs7NlrV5EJEpm9mKn95RyERGpCAV0EZGKUEAXEakIBXQRkYpQQBcRqYieAd3M7jCzn5nZUx3eNzP7ezM7ZmZPmNkl2RczQwcPwqZNMDKS/HvwYNklEhHJRJoa+reB7V3e/yiwufHYA/y34YuVk4MHYc8eePFFcE/+3bNHQV1EKqFnQHf3HwCnuyxyBXCXJx4CzjGzd2dVwEzddBP8+tdLX/v1r5PXRUQil0UOfT3wUtPzucZry5jZHjObNbPZ+fn5DFbdp5Mn+3tdRCQihTaKuvt+d59y96mxsbYjV/O1cWN/r2dEaft60n6XomUR0E8BG5qejzdeC8/0NIyOLn1tdDR5PSdK29eT9ruUIYuAfgj4o0Zvl/cD/8fdX87gc7O3ezfs3w8TE2CW/Lt/f/J6TpS2L0+ZNWTtdylDmm6LdwM/Av6dmc2Z2WfN7Fozu7axyH3AceAY8E3gT3MrbRZ274YTJ+Df/i35N8dgDkrbF6k5gK9bB3/8x+XVkGPY70oJVZC7l/LYunWr18HEhHsSUpY+JibKLlm1zMy4j46239ZlbPdB9/vMTLKMWfLvzEw+5Wu3vUZH81ufZAeY9Q5xVSNFc1ZC2r6W2qU42imqhjzIfi8y766UUDVVN6AHcj1ZQtq+ltIG6pw7NL1lkP1eZJCNISUk/bOkBl+8qakpz+0GF4tVneZfx+ioImmFbdqU1Gi7Cf0QGBlJauatzJImnyx12l4TE0nTkoTLzB5x96l271Wzhq7rydppl+JYuRLWro3nyqjIYRJKBVZTNQO6ridrp12K4x/+AX7+88I6NA2tyCCrVGA1VTPloutJidTBg8mF5MmTSc18elpBVpaqX8pF15MSqYKHSUjFVDOg63pSRGqomgEdVNWRIAXSm1Yq6qyyCyBSF629aRcHDoHqG5KN6tbQRQKj3rSSNwV0kYLE1JtWqaE4KaCLFKSk+6v0TXO5x0sBXaQgsfSmVWooXgroIgWJpTdtpxRQr7lypHwK6CIFiqE3bacUkJnSLqFTQBeJXNYNmNPTSfBu5a60S+gU0EUilkcD5u7d7afxhTB75MjbFNBFIrRYK7/qqnwaMCcm2r8eWo8cWUoBXTKl/sv5a66VdzJsTTqWHjmyVKqAbmbbzexZMztmZje0eX/CzB4wsyfM7EEzG8++qBI69V8uRpr7pw5bk46lR44s1TOgm9kK4Hbgo8AWYJeZbWlZ7D8Dd7n7xcA+4BtZF1T6U0ZNWf2Xi9Gr9p1VTTqGHjm91O2KMU0N/VLgmLsfd/fXgHuAK1qW2QJ8r/H399u8LwUqq6Yc09D2mHWrfasm/bY6XjGmCejrgZeans81Xmv2OPCJxt8fB9aY2drWDzKzPWY2a2az8/Pzg5RXUiirphzL0PbYdcpvz8zEW5POQx2vGLNqFP0L4ENm9ijwIeAU8GbrQu6+392n3H1qbGwso1VLq7JqympIK0a3/HbdUgzNWr97p0bjSl8xunvXB/AB4HDT8xuBG7ss/w5grtfnbt261SUxM+M+MeFulvw7MzPc501MuCcXmUsfExPDl7WXrL+LpDcz4z46unSfj47WYx+0++5m5f0O8gTMeqf42+mNtxZIboJxHJgEVpGkVy5sWWYdMNL4exrY1+tzFdATefwI6/zDrrMyT+Rl6/TdW4N6nr+DoiozQwX05P9zOfAc8DxwU+O1fcCOxt+fAn7SWOZbwNm9PlMBPZHXj1A15frpVCM1K7tk+ev03Rd/S3n/DoqsRHUL6Ja8X7ypqSmfnZ0tZd0hGRlpP8zaLOkuJp0dPJg0cJ08mTS8Tk/Xu0GwU954YiJpLK2ysr97kes3s0fcfardexopWjL1DBlMHbukdbLYGPjii8sn1apLo3TZDfKhdNlVQC9Z2QdirELsklZGD5PWaQDc3w7qdeqTXvbI1mAqZp1yMXk/lEN/m/Ld/QstX1xWQ3SdG0JDohy6cugyhLJzpq2KLE9z20Gnn6/aYIpXVJuOcuhSOaGlqorKoba2HXSiNpjihTD3jQK6RKnsnGmronKoaWZaVBtMfSmgS7RCqBEtKuqKoVuNP4QTm5TrrLILIFIFiwE07xzqxo1htR1IWFRDF8lIEVcMobUdSFgU0EUiElrbgYRFKReRyOzerQAu7amGLiJSEQroIiIVoYAuIlIRCuhSa3W+ZZtUjxpFpbYWh9EvjrxcnIIX1OgocVINXWorxCl4RYahgC61FcpNCUSyooAutRXMTQlEMpIqoJvZdjN71syOmdkNbd7faGbfN7NHzewJM7s8+6KKZEvD6KVqegZ0M1sB3A58FNgC7DKzLS2L3Qzc6+6/DewE/mvWBa0sdbMojYbRS9WkqaFfChxz9+Pu/hpwD3BFyzIOvLPx97uAn2ZXxArTnY5LF8IUvDqnS1bSBPT1wEtNz+carzW7BbjKzOaA+4AvZVK6qlM3i9rTOV2ylFWj6C7g2+4+DlwOHDCzZZ9tZnvMbNbMZufn5zNa9RDKrhqpm0Xpyj4EdE6XLKUJ6KeADU3PxxuvNfsscC+Au/8IWA2sa/0gd9/v7lPuPjU2NjZYibMSQtVI3SxKFcIhoHO6ZClNQD8CbDazSTNbRdLoeahlmZPARwDM7AKSgB5AFbyLEKpG6mZRqhAOAZ3Tw1H21VoWegZ0d38DuA44DDxD0pvlqJntM7MdjcW+AvyJmT0O3A1c497tnuQBCKFqpG4WpQrhENA5PQwhXK1lwt1LeWzdutVLNTHhnuy7pY+JiXLLJYUJ5RCYmUnWaZb8OzNT7PoHFWu52wnlWEgDmPUOcbW+I0XzrBpV4dqtBkKpHYfQdbJflanRNoRwtZaJTpE+70fpNXT3fKoYMzPuo6NLT/Ojo9FUX0KsdeVZphC/bwxiqtGmEdP3oUsNvd4BPQ8xHRktQjwXhVimqhnkpGbW/jA3y7u0+YjpOFNAb5VntSziIz3Ec1GIZaqSQQNZFfdLLFdr3QK6Je8Xb2pqymdnZ4tfcetdDSBJnGbVu2TTpiSh2GpiIkmQBmxkJPlZtjJL8rtlCLFMVTLo4Zr3z0g6M7NH3H2q3Xv1axTNu/NxKC1tAwixT3SIZaqSQRsD1eM2TPUL6Hk3Z0d8pId4LgqxTFXS6cQ4MtK7k1aMvXMqr1MuJu9HaTn0Kib/MhRiHjHEMlVFuxx66yPUxsG6Qjn0Jkr+iSxx8GCScTx5MqmVv/nm8mUiaAKqDeXQm6VJiWhgUDS0q4bXnDrp1NAc3QCbmqpfDb0X1eCj0W5XrVoFa9bA6dNJfnh6WrutHxF30qoN1dD7EcIUfJJKu1312muwsFCN4ehlUCN0vvK+olRAb1WZSR2qL80u0bm4PxF30gpeEfPfKOXSStec0ei0q1ppEJKEIKvQopRLP7K+5lSrXW7a7ap2NAhJQlDExb8CeqssrzmrNsdoYFp31dq1sHLl0mWU/5VQFDHqWSmXPCl9U7jmPtXq5SIhyaoDnVIuZVEDa+E0HH05Zf3CUESDswJ6njSzVOmqFsz6/T5Vy/rFvj9zr3B0mhMg70dlb3DRLKZZ81OIbU6Vim3+gb5PlaYuqtr+HBTD3uAC2A48CxwDbmjz/t8CjzUezwG/6PWZtQjo7vFFwQ5i/DFVKZi5D/Z9Ir7fyjJV25+DGiqgAyuA54H3AKuAx4EtXZb/EnBHr8+tTUCviBh/TFUKZu6DfZ9O+21x34V8Qm4V8v7sVm/Luk7XLaCnyaFfChxz9+Pu/hpwD3BFl+V3AXenT/pIDGJs361aE8Yg36dbX/3Y8umh7s9u7RSFt2F0ivSLD+BTwLeanl8N3NZh2QngZWBFh/f3ALPA7MaNG4c7TUmhYqyhx5gm6mbQ77NYQ+xWU49BqPuz228jj98NQ6Zc+gno1wP/pddnulIu0Qn1x9RLRZow3jLM9wk5ZZFWiPuz23bNY5t3C+g9BxaZ2QeAW9z99xvPb2zU7L/RZtlHgS+6+//udWVQi4FFFaNBO3HTOLd8dNuukP02H3Zg0RFgs5lNmtkqYCdwqM1K3gucC/xosGJK6DRoJ26aGjcf3bZr0du8Z0B39zeA64DDwDPAve5+1Mz2mdmOpkV3Avd4ryq/iJRCU+Pmo9t2LXqbay6XmCjnEQztCilLt5TLWUUXRgbUOrPPYv8nUCQpmHaFhEpzucRCt8YLhnZFGGKf1yUPCuixiHFkT0VpV5QvrwE7sZ8kFNCzUMRREOowuRrSrihfHldJVZiZUgF9WEUdBZH3OYu95tMs8l1RCXlcJVUildZpxFHej8qMFC1yTHyIw+RSiHWUaTeR7orKyONnF8tIWoYZKZqXynRbHBlJ9nsr3Wr+LRqhKFnL6nZuzWI5TnULujwpodqTGhEla3kM2KlCKk0BfVhVOApypnOe5CHrqSiqMJJWAX1YVTgK0hqwZVPnPIlF7PMVaaRoFhYnbaiyIYZHLr6tofIi+VKjqKQTS4uRSMWpUVSGp5ZNkeApoEs6atkUCZ4CuqSjlk2R4CmgSzp16s3TQZWmL5DiFHncKKBLeot9ug4cSJ5ffXVtIlsVJm6S4hV93NQ7oKvK1b+aRrZKTNwkhSv6uKlvQK9pYFqm35NaTSObOvnIIIo+buob0GsamJYY5KSW0xEa+sWSOvnIIIo+blIFdDPbbmbPmtkxM7uhwzJ/aGZPm9lRM/vHbIuZA1W5Bjup5XCExnCxpE4+MojCj5tO8+ouPoAVwPPAe4BVwOPAlpZlNgOPAuc2nv9mr88tfT70IucxD9UgE0DnMLl5LLtCc6DLILI+bugyH3qagP4B4HDT8xuBG1uW+Wvgc70+q/lRekCv4l0X+jVoJM34CI3lxgIiIegW0NOkXNYDLzU9n2u81ux84Hwz+6GZPWRm29t9kJntMbNZM5udn59PseocqV/14NeDGU9Jp/y0SDayahQ9iyTt8mFgF/BNMzundSF33+/uU+4+NTY2ltGqhxD7XJnDCuSkpvy0SDbSTJ97CtjQ9Hy88VqzOeBhd38deMHMniMJ8EcyKaXkJ4CpfzW9rkg20tTQjwCbzWzSzFYBO4FDLcv8D5LaOWa2jiQFczy7YkrV1f1iqU5C76Ias541dHd/w8yuAw6T9Hi5w92Pmtk+kuT8ocZ7v2dmTwNvAl9194U8Cy4i8RniPimSQqocurvf5+7nu/tvuft047W9jWBOo/H1z919i7u/z93vybPQtacqjgSu0yGq8Xz50i3oYqMqjgSu2yGq8Xz50i3oYqNbwUnguh2ioMN3WLoFXZWoiiOB63aIqotqvhTQY6NROBK4bodoIEMfKksBPTaq4kjgeh2i6qKaHwX02KiKI4HTIVoeNYqKiEREjaIiIjWggC4iUhEK6CIiFaGALpIxzcwgZVFAl/zUMLLFcH9UqS4F9LrKO9iGEtkKPqlo8ikpk7ot1lHr7EmQjPzIsrNwCHPOFPE9W4yMJOevVmbJQBqRYXXrtqiAXkdFBNsQIlsJJ5UQzmNSbeqHLksVMcFXCHPOlDCRmWZmkDIpoNdREcE2hMhWwklFw96lTArodVREsC07sh08CK+8svz1Ak4qmnxKyqKAXkfdgm2WvULKimyLjaELLbe1XbtW1WWptFQB3cy2m9mzZnbMzG5o8/41ZjZvZo81Hp/LvqjUsl9zbtoF21C6Gg6rXd9BgHe8Q8FcKq1nLxczWwE8B1wGzAFHgF3u/nTTMtcAU+5+XdoV993LpYQuaLVTlS4aIfSwEcnJsL1cLgWOuftxd38NuAe4IssCpqIRG/nr1PujXZAPWQg9bERKkCagrwdeano+13it1SfN7Akz+46ZbWj3QWa2x8xmzWx2fn6+v5LqXpr56xTwzOJKu4TQw6amlBUtV1aNov8EbHL3i4H7gTvbLeTu+919yt2nxsbG+luDal35m55Ogncr97iuhMruYVNTVWmCiVmaHPoHgFvc/fcbz28EcPdvdFh+BXDa3d/V7XOVQw9Uu4C++Lryz9JFVZpgQjdsDv0IsNnMJs1sFbATONSygnc3Pd0BPDNoYTtSrasYExPtX9eVUJSKTIEoK1q+ngHd3d8ArgMOkwTqe939qJntM7MdjcW+bGZHzexx4MvANbmUNs9+zUr+JZR/royiUyDKigbA3Ut5bN261YMxM+M+OuqeHPfJY3Q0eb2OZmbcJybczZJ/v/CFpc/rul06ad1egWyfiYmlh/TiY2Iin/XpZ1QMYNY7xFUFdPfij/yY6FfaXYnbp9d5xKz9YW1WXplkeN0CuqbPBQ1E6UYtXd2VtH3S9BHQrqsmTZ/bi5J/7R082HlQkVq6EiW1BKYZZ6fmkMHF2qSmgA468ttZrAJ2UveT3aKSKgNpziN16BiWR+CNuj99p1xM3o+gcujuSv616tSuoBz6UiXl0NXsk9+mD33bokZR6VunFjVQMG9VQmUg7/NIDPWbvAJvGY3J/egW0NUoKu2pRS14Bw8mOfOTJ5MMz/R0NumUWAZl59WXIfRDv1qNolkkzWJt8SiS2hWCl9c4u1gmNs2r+SLqQ79T1T3vx0AplyyuM9WvOr0Yrrslc1mmHPI8hPL8KYd86FOZHHq/SbN2eyX0Fg+RkmX1Eymi7tT8E1+7NnmEGISzVJ2A3k/VodPR1KmhL5QWD4lDyFW4IWUViIusO9Xpwrs6Ab2fI6TTsitWqIZeJWUE1hpEj1SbtcdCRfYWqdOFd3UCej8/pG7d7ir+Y6yNsgJrnaJHJym2fZGbKfSuhlmqTkB3T18j63Y0VfhyuVbKCqx1ih6dpNj2RZ5v63SOrVZAT6sGl8W1V1ZgDSF6lF0pSbntiypmnX7u9Qzo7uUf9JKvsgJr2dGj7PW7h3FSa1GXn3t9A7pUW9GBLZQ+ciEE0xBOKjXVLaDHN1JUZFGR0wm2TsG3sACvvgoHDmR/O8ReQrh55+7d8JnPwIoVyfMVK5LnIc0NUEOay0UkjZAm+AihLLFM+FJB3eZyCSqgv/7668zNzXHmzJlSypS11atXMz4+zsqVK8suigwrpLtahRBMQzip1FS3gH5Wyg/YDvwdsAL4lrv/ZYflPgl8B9jm7n1Xv+fm5lizZg2bNm3CzPr970FxdxYWFpibm2NycrLs4siwNm5sH8DKuNHHYtDOY6rFtEJI+8gyPXPoZrYCuB34KLAF2GVmW9ostwb4M+DhQQtz5swZ1q5dG30wBzAz1q5dW5mrjczFNuNlaFPw5TXVYlq6bWOQ0jSKXgocc/fj7v4acA9wRZvlbgX+ChgqglUhmC+q0nfJVIz3+KrD/dz6EdoJrl+xVShSShPQ1wMvNT2fa7z2FjO7BNjg7v/c7YPMbI+ZzZrZ7Pz8fN+FlYqIZcLtVmXXikMS8wkuxgpFSkN3WzSzEeBvgK/0Wtbd97v7lLtPjY2NDbvqyp5lK0/512qI9QQ3SIUikliTJqCfAjY0PR9vvLZoDXAR8KCZnQDeDxwys7atsJkp+Sx75513snnzZjZv3sydd95ZyDorQ/lXKVO/FYqYavSdRhwtPkh6whwHJoFVwOPAhV2WfxCY6vW57UaKPv300+mHS5U4Wm5hYcEnJyd9YWHBT58+7ZOTk3769Om2y/b1nepCowylTP3GjhBG5jZhmJGi7v4GcB1wGHgGuNfdj5rZPjPbkcdJJpUcLtuPHDnCxRdfzJkzZ/jVr37FhRdeyFNPPbVsucOHD3PZZZdx3nnnce6553LZZZfx3e9+d+D11k7M+VeJX78NuhGlCFP1Q3f3+4D7Wl7b22HZDw9frBRy6Be8bds2duzYwc0338yrr77KVVddxUUXXbRsuVOnTrFhw9tZqPHxcU6dOrVsOeli924FcClHv/34QxqD0EO8c7nk1G1q79693H///czOzvK1r31tqM8SkUD106AbURfNeAN6TpftCwsLvPLKK/zyl7/sOCho/fr1vPTS2z055+bmWL9+fdtlRSRyEaUIg5rL5ZlnnuGCCy4opTyLduzYwc6dO3nhhRd4+eWXue2225Ytc/r0abZu3cqPf/xjAC655BIeeeQRzjvvvGXLhvCdRKQ6us3lEm8NPQd33XUXK1eu5Morr+SGG27gyJEjfO9731u23HnnncfXv/51tm3bxrZt29i7d2/bYC5NIunHKxIz1dBzVsXv1LcQZgcUqQjV0KVcsQ71F4lMqm6LdfXkk09y9dVXL3nt7LPP5uGHB55Qsp4i6scrEjMF9C7e97738dhjj5VdjPhF1I9XJGZKuUj+IurHKxIzBXTJX0T9eEVippSLFEND/UVyF3UNXV2bRUTeFm1AL3uK4u3bt3POOefwsY99rJgVioj0EG1AL7tr81e/+lUOHDhQzMpERFKINqDn0bU57XzoAB/5yEdYs2bN4CsTEclYtI2ieXRtTjsfuohIiKKtoefVtVnzoYtIrKIN6Hl1bU4zH7pIENTNS1pEm3KBfLo2f/7zn+fWW2/lhRde4Prrr287H7pI6VpnsFzs5gXq719jqWroZrbdzJ41s2NmdkOb9681syfN7DEz+19mtiX7ouYv7XzoAB/84Af59Kc/zQMPPMD4+DiHDx8uuLRSa2V385Ig9ZwP3cxWAM8BlwFzwBFgl7s/3bTMO939/zb+3gH8qbtv7/a5mg9dZAgjI8kAjFZmyX0ypbKGnQ/9UuCYux9399eAe4ArmhdYDOYNvwGUc9cMkbro1J1LM1jWWpoc+nrgpabnc8DvtC5kZl8E/hxYBfxuJqUrmeZDl2BNT7e/C5RmsKy1zBpF3f124HYzuxK4GfhM6zJmtgfYA7CxQ03C3TGzrIo1lGHnQy/r9n5SA4sNnzfdlIym27gxCeZqEK21NCmXU8CGpufjjdc6uQf4T+3ecPf97j7l7lNjY2PL3l+9ejULCwuVCITuzsLCAqtXry67KFJVu3fDiRNJzvzECQVzSVVDPwJsNrNJkkC+E7iyeQEz2+zuP2k8/QPgJwxgfHycubk55ufnB/nvwVm9ejXj4+NlF0NEaqJnQHf3N8zsOuAwsAK4w92Pmtk+YNbdDwHXmdl/AF4H/pU26ZY0Vq5cyeTk5CD/VUSk9lLl0N39PuC+ltf2Nv39ZxmXS0RE+hTt0H8REVlKAV1EpCJ6jhTNbcVm80CbCXBTWQf8PMPi5CmmskJc5Y2prKDy5immssJw5Z1w9+XdBCkxoA/DzGY7DX0NTUxlhbjKG1NZQeXNU0xlhfzKq5SLiEhFKKCLiFRErAF9f9kF6ENMZYW4yhtTWUHlzVNMZYWcyhtlDl1ERJaLtYYuIiItFNBFRCoi6IAe063vepW1ablPmpmbWaldrFJs22vMbL6xbR8zs8+VUc5GWXpuWzP7QzN72syOmtk/Fl3GlrL02rZ/27RdnzOzX5RQzMWy9CrrRjP7vpk9amZPmNnlZZSzqTy9yjthZg80yvqgmZU2O56Z3WFmPzOzpzq8b2b2943v8oSZXTL0St09yAfJRGDPA+8huWnG48CWlmXe2fT3DuC7oZa1sdwa4AfAQ8BU4Nv2GuC2SI6DzcCjwLmN578Zcnlblv8SyYR3QZaVpPHuC42/twAnQt62wH8HPtP4+3eBAyWW998DlwBPdXj/cuB/Aga8H3h42HWGXEOP6dZ3PcvacCvwV8CZIgvXRtryhiBNWf8EuN3d/xXA3X9WcBmb9bttdwF3F1Ky5dKU1YF3Nv5+F/DTAsvXKk15twCLd3b/fpv3C+PuPwBOd1nkCuAuTzwEnGNm7x5mnSEH9Ha3vlvfupCZfdHMngf+GvhyQWVr1bOsjcupDe7+z0UWrINU2xb4ZONS8DtmtqHN+0VIU9bzgfPN7Idm9pCZdb1Bec7SblvMbAKY5O0AVLQ0Zb0FuMrM5khmXP1SMUVrK015Hwc+0fj748AaM1tbQNkGkfpYSSvkgJ6Ku9/u7r8FXE9y67vgmNkI8DfAV8ouSx/+Cdjk7hcD9wN3llyebs4iSbt8mKTG+00zO6fMAqW0E/iOu79ZdkG62AV8293HSVIEBxrHc6j+AviQmT0KfIjkpjwhb99MhbxjMrv1XQF6lXUNcBHwoJmdIMmXHSqxYbTntnX3BXf/f42n3wK2FlS2VmmOgzngkLu/7u4vAM+RBPgy9HPc7qS8dAukK+tngXsB3P1HwGqSiaXKkOa4/am7f8Ldfxu4qfHaLworYX/6jXG9ldVgkKJB4SzgOMkl6WIDyIUty2xu+vs/ktxBKciytiz/IOU2iqbZtu9u+vvjwEMBl3U7cGfj73Ukl7FrQy1vY7n3AidoDO4LtawkjXbXNP6+gCSHXkqZU5Z3HTDS+Hsa2FfW9m2UYROdG0X/gKWNov8y9PrK/LIpNsblJLWt54GbGq/tA3Y0/v474CjwGEkDSMcgWnZZW5YtNaCn3LbfaGzbxxvb9r0Bl9VIUlpPA08CO0Peto3ntwB/WWY5U27bLcAPG8fBY8DvBV7eT5Hc0/g5kivLs0ss693AyyS35pwjudq5Fri28b4Btze+y5NZxAQN/RcRqYiQc+giItIHBXQRkYpQQBcRqQgFdBGRilBAFxGpCAV0EZGKUEAXEamI/w9XgLB6OEkCCAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 从 data.txt 中读入点\n",
    "with open('./data.txt', 'r') as f:\n",
    "    data_list = [i.split('\\n')[0].split(',') for i in f.readlines()]\n",
    "    data = [(float(i[0]), float(i[1]), float(i[2])) for i in data_list]\n",
    "\n",
    "# 标准化\n",
    "x0_max = max([i[0] for i in data])\n",
    "x1_max = max([i[1] for i in data])\n",
    "data = [(i[0]/x0_max, i[1]/x1_max, i[2]) for i in data]\n",
    "\n",
    "x0 = list(filter(lambda x: x[-1] == 0.0, data)) # 选择第一类的点\n",
    "x1 = list(filter(lambda x: x[-1] == 1.0, data)) # 选择第二类的点\n",
    "\n",
    "plot_x0 = [i[0] for i in x0]\n",
    "plot_y0 = [i[1] for i in x0]\n",
    "plot_x1 = [i[0] for i in x1]\n",
    "plot_y1 = [i[1] for i in x1]\n",
    "\n",
    "plt.plot(plot_x0, plot_y0, 'ro', label='x_0')\n",
    "plt.plot(plot_x1, plot_y1, 'bo', label='x_1')\n",
    "plt.legend(loc='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来我们将数据转换成 NumPy 的类型，接着转换到 Tensor 为之后的训练做准备"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "np_data = np.array(data, dtype='float32') # 转换成 numpy array\n",
    "x_data = torch.from_numpy(np_data[:, 0:2]) # 转换成 Tensor, 大小是 [100, 2]\n",
    "y_data = torch.from_numpy(np_data[:, 2]).unsqueeze(1)\n",
    "\n",
    "x_data = Variable(x_data)\n",
    "y_data = Variable(y_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在 PyTorch 当中，不需要我们自己写 Sigmoid 的函数，PyTorch 已经用底层的 C++ 语言为我们写好了一些常用的函数，不仅方便我们使用，同时速度上比我们自己实现的更快，稳定性更好。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 定义 logistic 回归模型\n",
    "w = Variable(torch.randn(2, 1), requires_grad=True) \n",
    "b = Variable(torch.zeros(1), requires_grad=True)\n",
    "\n",
    "def logistic_regression(x):\n",
    "    return torch.sigmoid(torch.mm(x, w) + b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在更新之前，我们可以画出分类的效果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f51c35893d0>"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAxTklEQVR4nO3deXgUVfbw8e9NSIgssio4QBKcwVdwDAiICzjigiAquOAoi4IoCCMjjM4ITkYniPwIiMrmCqIo4IaMGyrjhhoWIQgiAiooSxjEGHZIIKTP+0d3MIR00p3U1t3n8zz9pJdK1emuqlO37r11y4gISimlIl+c2wEopZSyhiZ0pZSKEprQlVIqSmhCV0qpKKEJXSmlokQ1txbcsGFDSU1NdWvxSikVkVauXPmriJxS1meuJfTU1FSys7PdWrxSSkUkY8yWYJ9plYtSSkUJTehKKRUlNKErpVSU0ISulFJRQhO6UkpFiQoTujFmpjHmF2PM2iCfG2PMFGPMRmPMGmNMW+vDjGJz5kBqKsTF+f/OmeN2REqpCBVKCf0FoFs5n18JtAg8BgNPVT2sGDFnDgweDFu2gIj/7+DBmtSVUpVSYUIXkc+BXeVM0hN4UfyWAXWNMadZFWBUS0+HQ4eOf+/QIf/7SikVJivq0JsA20q8zgm8dwJjzGBjTLYxJjs3N9eCRUe4rVvDe7+StFYncum6U+FwtFFURJ4VkfYi0v6UU8q8cjW2JCeH934laK1O5NJ1p8JlRULfDjQr8bpp4D1VkbFjoUaN49+rUcP/vkW0VsdaTpaYvbbu9GwhAohIhQ8gFVgb5LOrgPcBA5wPLA9lnu3atRMlIrNni6SkiBjj/zt7tqWzN0bEX747/mGMpYuJWiVXT4MGIomJx/+ONWpYvsqOCXfd2bkpzZ7t/65OfXcVHJAtwXJ1sA+OTQAvAzuAQvz147cDQ4Ahgc8N8ASwCfgGaF/RPEUTumNSUspOCikpbkfmfWUlMSd/y3DWnd0JV7cj76hSQrfrEZMJ3ebSeLBFasmqcoIlMafOdsJZd3YnXD3T847yErpeKeoUl1q4+vaFZ5+FlBQwxv/32Wf976vyhdrZyMI27OOEs+7s7jDlQPu9soDxJ3zntW/fXmJqPPTUVH8SLy0lBTZvdjoaFYJgq6ykGjW8cYC0e/MqLo+UbKT1ynePNcaYlSLSvqzPtITuFIf6nCvrlNUJKSEBGjTw3tmO3R2m9EwvMrh2x6KYk5xcdhFKz1k9qzhZpaf7j7vJyf4E6cUk5kSsfft687ur32iVi1P0nFUpZQGtcvECPWdVFtELfFQwWuXiJD1nVVVU+kSvuLMU6KaltISuVETx2nAAoGcMXqIldKUiiNc6S+kZg7doCV2pCOK1C3y8eMYQyzShKxVBHBigMyxeO2OIdZrQlYogVnaWsqLuu3798N5X9tI6dKUijBWdpbTuOzppCV2pGFJcKu/Xz5q6711B7jYc7H1lL03oKuJoN7nKKTngZzDh1n17rZE21mlCVxFF77NZeWX1SCkt3ETstUbaikR7YUATurKMEzuLdpOrvIpK35VJxJE0okUsFAZ0cC5lCafGHouL8++MpRkDPp91y4lG5Y3vnpLi3ZEkrRIttyTQwbmU7ZwqOWudbeUFqx4ZOtT//JZboq8aouRZY7CDWTT1mdeEHuOsqiZx6gKTSKuz9ZKyqkf694dZs6KzGqJ0FUswUVUYCHazUbsfMXmTaI+x8gbSTt4V3oV7bUctJ9eb00K5ybfVN0x3YtuknJtEa0KPYVbuzFYeHNwWSwcMY8reBoxxO7KqC/bdir+f1evWqX1AE7oqk9U7czQkwmg6MFVk9myR+PjYK6Hb9d2cWl55CV3r0GOY1Q2Mffv6ewv4fP6/kdhjwu1ukU71ky6uXy4qOvGzaGmTcLq9xQsDlWlCj2HawHgiN3dKu/tJlzxY9O9f9kVG8fHe7UceLqf7yHuiB1aworvdD61y8YZoqCaxkpuNhHYuu6yqpGitO3eLF+rQtYQe46KhmsRKbp612Hl2EMpl/xBlXfgc5oWrZjWhK1WCmzulnafsoRwUYr26zQpuF5A0oStVils7pZ1nB8EOCvHx3h+DRYVOE7pSHmHn2UGwg8WsWVrdFk30jkVKeYgVdyMKNl/w16Vv3eovsUf7YFyxSBO6UjHCroOF8g6tclFKqSihCV0ppaKEJnSlLBTttzhT3qZ16EpZpPRdm4ov3Qetu1bO0BK6UhZxe2AvpTShK2URL4y2p2JbSAndGNPNGPOdMWajMWZUGZ8nG2M+NcasMsasMcZ0tz5UpbzNE6PtqZhWYUI3xsQDTwBXAq2A3saYVqUm+xfwmoicA9wMPGl1oMom2opnGbcG9tJVqIqFUkLvAGwUkR9F5AjwCtCz1DQCnBx4Xgf4n3UhxhCn90y7B+COQSed9NvzBg3sHx9FV6EqKZSE3gTYVuJ1TuC9kjKAfsaYHOA94K9lzcgYM9gYk22Myc7Nza1EuFHMjT1TW/EsU7z68vJ+ey8/3/7l6ipUJRn/eOnlTGBML6CbiNwReH0LcJ6IDCsxzT2BeT1qjLkAeA74o4j4gs23ffv2kp2dbcV3iA6pqf4kXlpKin/kJDvExfkPHqUZ4x+xSYXMjdUHugpjkTFmpYi0L+uzUEro24FmJV43DbxX0u3AawAishRIAhqGH2oMc6OLhLbiWcatHi66Cu0Rqe0SoST0FUALY0xzY0wi/kbPt0tNsxW4DMAY0xJ/Qtc6lXC4sWfqTUUt41ZijYRVGGnJMaLbJYLdm67kA+gOfA9sAtID7z0E9Ag8bwUsBr4GVgNXVDRPvadoKVbfkDDUm4XqTUUt4dT9JIMt26ur0M3fpbLcvK9sKCjnnqJ6k2gvsWrPjKC9yM1kZPWyvZxY3eL15FgWY7x9A21N6F5ndSaIkL3I7VJthBzzPCXcTdXrybEsXt99NKF7mR2ZJUL2Ijd3HK/vtF5UmU01En9nrx/sNaF7mR1bfITsRW4edyLkmOcpldmsvJ4cg7HqpHl3/m4Z+/lYmb9uvmWxlZfQdXAut9nR3y0Suj7gbpc77e4XvmCb5JYtwXuw2Hnjazv17eu/fqCyN9D++cDPjPxwJMmPJ5P+STqLNi+yIcoyBMv0dj+0hB5gV2k6AlrotA49sgTbVPU3/M3GvI1y5zt3SvUx1SVudJzc9PpNsmrHKkuXgVa5eFiMZxav9nKJgOOh48raVCOgZs8Rq3eslt7zekvc6DhJHJMog94eJD/k/WDLsjShe51mFk8pK3EV17nH+ioouTkGS+ix1A7xxZYvpPuc7kIGUuv/asnfF/5dtu/bbusyNaFHqhgvvbuloqoFXQV+EdL2bjmfzycLvl8gnWZ2EjKQhhMaykOLHpJdh3ZV+L9WlM80oUeqWN1jXFZe6VNXwW9irbxRWFQoc9fMlbSn0oQMJPnxZJmybIocOHwgpP+36vcqL6FXONqiXXS0xRDoUHquCDZyYkm6CvzmzPEP1bt1q7+H0Nix3u/BEq6CowXMWj2LCUsm8OPuH2nZsCUjO46kz9l9SIhPCHk+Vo3IWdXRFpVbrOpbF2mjI7msrF6fpWn3Rr+qdu/zsn2H9zFh8QSaT27OkAVDaFijIf+56T+s/cta+rfpH1YyB2dG5NSE7mVW9CeP6KHj3FGy7zT4S+MlebBLv6Vi/fj/y8FfSP84neTHkxn50UjOPvVsPr71Y5bdvoxrz7yWOFO5tOnItQ/B6mLsfmgdeoiq2oqi9fBVFksdjWKtXrykzbs3y7AFwyTp4SQxGUZ6vdZLVmxfYdn8nahD14Qe7fQad8tFYoIPNeZIPf5XZZ2s3blWbpl/i8SPjpeEhxJk4JsDZUPuBs/FWUwTeiyL0D3Uq0kzEkuw4cQcicf/yq6TZduWSc+XewoZSI2xNWTE+yNk656tzgRdBZrQY1kEZiAvhxyJx8dwYi6vD76XDqwlhfP9fD6fLNy4UDq/0FnIQOpl1pMHP3lQcg/mVjkOp64P1IQe67xa3A3Cy0kzEkuw4cRc0eX9XjmwlhTK9ztadFReW/uatH2mrZCB/O7R38mjSx6V/Yf3WxJDeYUQqwso5SV07YeuPMfL3e+t6kvspHBjLu5bHqwvvte+a3nf77uNh5m9ZjYTlkzg+7zvaVG/BSM7jqRfWj+qV6vuSAxg7Taj/dBVRPHy0LYRMjLxccKNubhveenumsWs7DdthbK+30k1hI4DFvD7Kb/njnfuoFZiLV7r9Rrr71rP7W1vtzSZQ/l9zJ3of15ME7ryHC8nzUgc37uyMXv5wFrS8d9PqNNoD3HX3MlcczUtGrRgYb+FZA/K5sazbiQ+Lt6WGMr7rRz9HYPVxdj90Dp0VZ4Iq/avNC9/Ty83Tpe2be82GfH+CKkxtoaQgfR8uacs3bbUseV7pQ5dE7qqmJezTgSLhITp9VW/IXeD3PbmbZLwUILEj46XW+bfImt3rnUlFu3lorwvErJOhPJybx6vy96eLTe8eoOYDCNJDyfJsAXDZPPuzSLi/YNQVWlCV5WnWcc2kdgF0k0+n08+/vFjufzFy4UMpM64OpL+cbrsPLDz2DRWlj+8emDQhB4r7NgCIyTreHXnK48eK0NT5CuS+evmS4fpHYQMpPHExjI+a7zsLdh7wrRW/aZePjHVhB4L7NoCIyDreHnnK0+kxu2UI0ePyAurXpCW01oKGcjpk0+Xp1c8LfmF+UH/x6ryh5c3e03oscCuLTACso6Xd76KROKZhd0OHjkok5dNlmaPNRMykLSn0mTumrlSWFRY4f9atS14+cRUE3ossHML9HjW8fLOp0K369AuGfPZGGk4oaGQgXSa2Une/e5d8fl8Ic/DqvKHlwsJ5SX0ajZ0bVduSE4u+/piK65e6NvX01fO2PnVlf127N/BY0sf4+mVT3PgyAG6t+jO/Z3up1Nyp7DnVbyZVvW2eGPH+u8Dc+jQb+955eK2cgXL9HY/tIRusQioGglLGGcF0fbVY8UPeT/I4LcHS+KYRIkbHSe95/WWr3/+2u2wjvHqiSla5RIjvLoFhqsSGTpavnosWLVjldz0+k0SNzpOqo+pLne+c6dszNvodlgRo7yErqMtKu+JxCENVblEhC+2fsG4rHF8sPEDaifWZmj7oYw4fwSn1T7N7fAiSnmjLWoduvIeJ4enU7byiY8F3y8gc3EmS7Yt4ZQap/DwJQ9zV4e7qJtU1+3woo4mdOU92soZ8Y76jvLq2lfJXJzJ2l/WklwnmalXTmXgOQOpkVCj4hmoStHhc5X3lDV+rjHQvbs78dhgzhx/zVJcnP/vnDluR2SN/MJ8nlrxFGdMPYN+/+mHT3zMunYWG/+6kWEdhsVsMndsfQerXLf7oY2iZdCWvd8MHXpiB/Mo6boSjb1y9uTvkXFfjJNGjzQSMpDzpp8nb65/U4p8RW6H5jq9BV0smjOn7I6vXr97gl2iuGE0mr7azgM7mbRsEk9mP8m+w/vo+vuujOo0iotTLsYEu+VRjLF6fest6CJBevrxyRz8r9PT3YnHLqGee1rYMOq16o1oaPP9afdP3LXgLlInpzJ+8Xi6/r4rKwev5IN+H9A5tbMm8xIcXd/Biu4lH0A34DtgIzAqyDR/BtYB3wJzK5qnVrmUEgvXr4dz7mnRtdderN7w8mXlFflm5zfS942+Ej86XhIeSpA73rpDvv/1e7fD8jSr1zdVubAIiAc2AacDicDXQKtS07QAVgH1Aq9PrWi+mtBLieS9PFThfEeLMrEXf1YvHmQqsnjrYrl67tVCBlJzbE2554N7JGdvjtthRQRP3YIOuABYWOL1/cD9paaZANxR0bxKPjShlxKJe3m4wj0LsaCR2KsnPpHQ/u3z+eT9H96Xi5+/WMhAGoxvIKMXjZZfD/7qdmgRx6lb0FXYKGqM6QV0E5E7Aq9vAc4TkWElpnkT+B7oGCjRZ4jIB2XMazAwGCA5ObndlrJaCmLZnDlVH1XIy1xoDYymBkinFPmKmLduHpmLM1n982qantyUey+4l0FtB1Ezsabb4cU8JxpFq+GvdukM9AamG2Pqlp5IRJ4VkfYi0v6UU06xaNFRpG9ff5bx+fx/oymZQ9n9y20ews6FRUasw0cPM33ldM584kxufuNm8gvzmdljJpvu3sSI80doMo8AoST07UCzEq+bBt4rKQd4W0QKReQn/KX1FtaEqKJG377+bpgpKf4LhVJSbO+W6cIiI87+w/uZuGQizSc3Z/C7g6mbVJd5N87j2798y23n3EZifKKj8XitV1IkCaXKpRr+BH0Z/kS+AugjIt+WmKYb0FtE+htjGuJvIG0jInnB5qv90JVy16+HfmXKl1OYtnwauwt2c1nzyxjVaRSXNb/MtW6HejlGxapU5SIiR4FhwEJgPfCaiHxrjHnIGNMjMNlCIM8Ysw74FPhHeclcRSgtOkWFrXu3Mvz94SQ/nsyYz8fQObUzX97xJR/d+hGXn365Y8m8rM0pVi7HsIteKapCo0WniLc+dz0Tlkxg9prZAPRL68d9F95Hy1NaOh5LsM2pdDIvZoy/aUmVX0LXhK5Co91FItaK7SsYlzWONze8SVK1JAa1HcS9F95Lch33Rq8MtjnFx0NR0Ynv62b2Gx0PXVVdNFyvHkNEhI9/+pjMrEw+/ulj6ibVJf2idO4+725Oqel+D7Ngm01R0Yklde2VFDody0WFJthY5DpGuaf4xMcb696gw4wOdHmpC+ty1/FIl0fYOmIrYy4d44lkDsE3m+JeSNorqXI0oavQaIduTztSdITnVz1Pqyda0ev1Xuwp2MOzVz/LT8N/4u8X/p3a1Wu7HeJxytucov1yDDtplYsKTfFeFc1Xskagg0cOMv2r6Ty69FFy9uXQpnEbXu31Kje0vIH4uHi3wwtKNyd7aKOoUhFoV/4upn45lSnLp7Arfxd/SvkT93e6n66/76pD10Y5HQ9dqQgTrMv/9n3buXfhvSQ/nkzGZxl0bNaRxQMX89mAz+j2h26azGOcVrmoyBWlg5mV7qO9ZQvcMcjHjK+eY3Hdu/CJj5v/eDMjO47k7EZnuxus8hQtoStnWH2VaXHW27LFPyLuli3+13ZfverA1bJlXS1ZkB/HoueuYFDbQfzw1x+Yff1sTebqBFqHruxnx1Wmblzo5NDVsnFxgsiJVSfGCD6fVqnEOq1DV+6yY4AONy50snmgEZ/4eGvDWyTU+7nMz5OTNZmr8mlCV/azI/m6caGTTQeRwqJCXvr6JdKeSuPaV6/l5O7jSUwqPG4a7fKvQqEJXdnPjuTrxoVO9euX/X4lv8ehwkNMWz6NFlNbcOubtxJn4ph93Wx2zJrIzBkJerWkCpsmdGW/YMm3e/fKNzA6feeKOXNg374T309MDPsgsqdgD2M/H0vqpFT++v5faXJyE97p/Q5fD/mavml9qRZXTa+WVJUT7Gajdj+i7ibRkXDXXzeV/n2GDo2sm2KnpJR9t+kGDUKexY79O+S+/94ntf+vtpCBXDn7Svls82fi8/nsi1tFHapyk2i7RFUvFx0rPHzBeqk0aAC//up4OBWKi/On8NJCGKj7x90/8sjiR3h+9fMU+gq5sdWNjOo0ijaN29gTq4pq2svFbnqblfAFa0jMy/PmnZAq0Q6wZuca+rzRhxZTWzBz9Uz6t+7Pd8O+45Ver2gyV7bQhG4FHSs8fOU1JHrxQBhGI2zW1iyunns1rZ9uzTvfv8O9F9zL5uGbeeaaZ/hD/T84FLB36Z0MbRSsLsbuR1TVoQerX01JcTsy75o9u+zfDPz17F5UTjuJz+eTBd8vkE4zOwkZSMMJDWXMZ2Nk16FdroVbWXY2B82eHVlNJ15EOXXomtCtoFtp5TRoEPEHwsKiQpm7Zq6kPZUmZCDNHmsmk5dNloNHDrodWqXYvSlr2afqykvoWuViBTu60MXCeenkySdWYyQkwIEDnv/eBUcLeCb7Gc6cdiZ95vehsKiQF3q+wKa7N3H3eXdTI6FGxTPxILubg7R20mbBMr3dj6gqoVstlkr8Jc/vGzQQSUz09PfeW7BXxmeNl8YTGwsZSIfpHWT+uvlS5Cuq2owd7PZa3qKMsbcWTEvoVYdWuUSYWN3qPfy9dx7YKekfp0vdzLpCBtLlxS7yyY+fWNOH3MEDeEWLsnsVxFJZxS6a0CON3cUkL/JoI+nm3Ztl2IJhctLDJ4nJMNLrtV6yYvsKaxfi4IGsokU5kXAj7Ro8r8WrCT3SeLikaouysojL33vtzrVyy/xbJH50vCQ8lCAD3xwoG3I32LMwBw/goSzKawmsMqz6Dl48o9CEHmm8uBXZKdgBzIXvvXTbUunxcg8hA6kxtoaMeH+EbNu7zd6FeqiEXhleOwBYuft4sWylCT0SlDXWiZf2EjsFKzaCI9/b5/PJwo0LpfMLnYUMpF5mPfn3p/+W3IO5ti9bRDxVh+72/KxgZRL2Yu2nJnSnVLao4sW9wkkuFYOOFh2V1799Xdo9007IQH736O/k0SWPyv7D+21dbpk80sslXF4swVqZhL34/TShOyHUpFzW3uTFrcZJDh/QDh89LDNWzpAzpp4hZCAtprSQ6SunS0FhgS3Li2ZWJE+rj2XBdqf4+PDn7cWyliZ0J4SSlINtHcGqG6K5V0tpDpRQ9x/eL48ueVSaPNpEyEDOefoceW3ta3K06Kjly4oVVS2L2JEwy2tjr8y8vdZGoAndCaEUVcorOsRyCd1mvx78VR785EGpl1lPyEAueeESWbhxYfl9yJ3ai72WLcIUUkIu5zvadXI6e3b07laa0J0QypZZXuOf187rosDWPVtlxPsjpMbYGkIG0vPlnrJ029KK/9Gp82wvns9XQrnHpAq+o52Njl5s0LSCJnQnhLJzlpf0I7yk5iUbcjfIbW/eJgkPJUj86Hi5Zf4tsnbn2tBn4FSbRiy0nVTwHe38CaL159WE7pSKknKUlMi8Knt7ttzw6g1iMowkPZwkwxYMk827N4c/I6eKdm4UIZ0uOFTwHe3cJaJ1d9OE7iVaEreUz+eTj3/8WLq82EXIQOqMqyP//OifsvPAzsrP1IkBTcq7mMquIqQbGS6E39Lu8dejbXfThK6iTpGvSOavmy8dpncQMpBGjzSS8VnjZW/B3qrP3Olio1MJ1o06iNmzTxxBMzExOjKrS8pL6NVcGLFXqUorLCpk7jdzGb94POt/Xc/p9U7n6auepn+b/iRVS7JmIcXj2Ken+wfqTk7232rOiht+lzXgeLGUFOuWUxa3BiMXKf+1soyREH5cY0w3YDIQD8wQkcwg090AzAPOFZHs8ubZvn17yc4udxKljjlUeIgZX81g4pKJbNu3jbRGaYzqOIobz7qRanERVC6Jiys7oRkDPp+9y05NhS1bTnw/JQU2b46eZUY5Y8xKEWlf1mcV3rHIGBMPPAFcCbQCehtjWpUxXW1gOPBl1cJV6je783fz8OcPkzIpheEfDCelbgoL+ixg9Z2r6X127+DJ3Kt3fAp2c+zybpptlTBudG0ZvUWRo0K5BV0HYKOI/CgiR4BXgJ5lTDcGGA8UWBifilH/2/8//vHff5A8KZkHPn2A85qcxxe3fcEXt31B9xbdMcYE/+c5c2DwYH/JUMT/d/BgbyR1N5JqMTtulVgRNw9g4fBqASBcwSrXix9AL/zVLMWvbwGmlZqmLfBG4PkioH2QeQ0GsoHs5ORkB5oPVKT5Ie8HGfT2IEkckyhxo+Ok97ze8vXPX4c3E693QI7GrhfBRELfwXBjdHn9UZVeLhUldPyl/EVAqlSQ0Es+tJeLKmn1jtVy0+s3SdzoOKk+proMeWeIbMzbWLmZReslgpHK6wewcAoAHjhAlZfQK2wUNcZcAGSISNfA6/sDJftxgdd1gE3AgcC/NAZ2AT2knIZRbRRVIsIXW78gMyuT9ze+T+3E2gxtP5QR54/gtNqnVX7G2hCnwhFOQ7UHtq3yGkVD6R6wAmhhjGkObAduBvoUfygie4GGJRa2CPh7eclcxTaf+Hjvh/cYlzWOJduWcEqNUxh76Vj+cu5fqJtUt+oLGDvWX2desnugU/XUKvIkJ5edpMuq5/d4I2+FjaIichQYBiwE1gOvici3xpiHjDE97A5QRY+jvqPMWTOH1k+35pqXr2H7vu1Mu3Iam0ds5p8X/dOaZA7uNP6pyBVOQ7XHG3lD6oduB61yiR35hfk8v/p5HlnyCJv3bKbVKa0Y1XEUN//xZhLiE9wOTyl/r5ZQLiQr7kFV+uzPwQJDVatclKqUvQV7eXLFk0z6chK/HPyF85qcx6Suk7jm/11DnAmlx6xSDunbN7SEbOdVxBbQvUpZbueBndz/0f0kT0rmn5/8kzaN2/Bp/09ZevtSep7Z075kHi19iZW39e3rbwD1+fx/PZLMQUvoykI/7f6JiUsmMnP1TA4fPUyvVr0Y1WkUbU9ra//CS58KF19MBJ7a4ZSyk9ahqypb+8taMrMyeWXtK8SZOPq37s8/Ov6DMxqc4VwQHuhOppQTtA5d2WLJtiWMyxrHu9+/S82Emow4fwR/O/9vNDm5ifPBeLw7mVJO0ISuwiIifLDxAzIXZ/L5ls9pcFIDRncezbAOw6h/Un33AgunL7FSUUoTugpJka+Ieevmkbk4k9U/r6bpyU2Z1HUSd7S9g5qJNd0OTy8mUgpN6KoCh48eZtbXs5iweAKbdm/ijAZnMLPHTPqm9SUxPrHc/y0sLCQnJ4eCAgcG4GzbFj77DHbvhqIiiI+HevWgZk1Yv97+5XtAUlISTZs2JSFB+/bHKk3oqkz7D+/nmZXP8NjSx9hxYAftTmvHvBvnce2Z1xIfFx/SPHJycqhduzapqanlD3erqkxEyMvLIycnh+bNm7sdjnKJJnR1nNyDuUxdPpVpy6exu2A3lza/lBeve5HLml8WdlIuKCjQZO4QYwwNGjQgNzfX7VCUizShKwC27t3Ko0seZfpX08k/ms91Z17HqE6j6NCkQ5Xmq8ncOfpbK03oMW597nrGLx7PnG/8V1X2S+vHfRfeR8tTWrocmVIqXHrpf4xavn051716Ha2ebMXr617nrnPvYtPdm3i+5/Mxm8w3b97M3Llzj71evXo177333rHXb7/9NpmZZd4fPWwDBgxg3rx5ANxxxx2sW7fOkvmq2KYJPYaICB9u+pDLXryM82acx6LNi3jgTw+wZcQWJnWbRHKd2O6zXVFC79GjB6NGjbJ8uTNmzKBVqxPuu24fHfMmammVSwzwiY//rP8P47LGsXLHSk6rdRqPdHmEO9vdSe3qtR2JYcQHI1j982pL59mmcRsmdZtU7jQvvvgiEydOxBhDWloaL730EgMGDODqq6+mV69eANSqVYsDBw4watQo1q9fT5s2bejduzdPPPEE+fn5ZGVlcf/995Ofn092djbTpk1jwIABnHzyyWRnZ/Pzzz8zYcIEevXqhc/nY9iwYXzyySc0a9aMhIQEBg4ceGxZZencuTMTJ06kffv21KpVi+HDh/Puu+9y0kkn8dZbb9GoUSNyc3MZMmQIWwNXvk6aNImOHTuG/6PpmDdRTRN6FDtSdIQ5a+YwfvF4vsv7jj/U/wPPXv0st7a+lerVqrsdnu2+/fZbHn74YZYsWULDhg3ZtWtXudNnZmYyceJE3n33XQAaNWp0LIEDvPDCC8dNv2PHDrKystiwYQM9evSgV69ezJ8/n82bN7Nu3Tp++eUXWrZsycCBA0OO+eDBg5x//vmMHTuW++67j+nTp/Ovf/2L4cOH87e//Y1OnTqxdetWunbtyvrK9K9PTz/+4ivwv05P14QeBTShR6GDRw4y/avpPLr0UXL25dCmcRteueEVerXqFXIfcqtVVJK2wyeffMKNN95Iw4b+OyTWr2/t0ATXXnstcXFxtGrVip07dwKQlZXFjTfeSFxcHI0bN+aSSy4Ja56JiYlcffXVALRr144PP/wQgI8++ui4evZ9+/Zx4MABatWqFV7QOuZNVNOEHkV25e9i2vJpTPlyCnn5eVyccjHTr5lO19931S5tJVSrVg1f4Oa/Pp+PI0eOVGo+1av/dpZj1ailCQkJx9ZVfHw8R48eBfxxLlu2jKSkpKotQMe8iWraKBoFtu/bzr0L7yX58WT+vejfXNjsQpYMXMKiAYvo9oduMZvML730Ul5//XXy8vIAjlW5pKamsnLlSsDfc6WwsBCA2rVrs3///mP/X/p1KDp27Mgbb7yBz+dj586dLFq0yIJvAldccQVTp0499nr16tWVm1E4989UEUcTegT7Pu97Br09iOaTmzP5y8lc1/I6vhn6DW/3fpsLml3gdniuO+uss0hPT+fiiy+mdevW3HPPPQAMGjSIzz77jNatW7N06VJq1vQPLpaWlkZ8fDytW7fm8ccf55JLLmHdunW0adOGV199NaRl3nDDDTRt2pRWrVrRr18/2rZtS506dar8XaZMmUJ2djZpaWm0atWKp59+unIz0htoRzW9wUUE+mrHV2RmZTJv3TyqV6vO7efczr0X3Evzet4aw2P9+vW0bBl7fdqL67bz8vLo0KEDixcvpnHjxo4sO1Z/81iiN7iIAiLCZ1s+Y1zWOP676b+cXP1kRnUaxfDzhtOoViO3w1MlXH311ezZs4cjR47wwAMPOJbMldKE7nE+8fHOd++QuTiTZTnLOLXmqYy7bBxD2w+lTlLVT+WV9ayqN1cqXJrQPaqwqJBX1r7C+MXj+Tb3W5rXbc6T3Z9kQJsBnJRwktvhKaU8SBO6x+QX5jNz1UweWfIIW/Zu4exTz2bO9XP481l/plqcri6lVHCaITxiT8EenlzxJJOWTSL3UC4XNruQad2ncVWLq2K226FSKjya0F3284GfeXzp4zyV/RT7j+znyj9cyf2d7ueilIvcDk0pFWG0H7pLNu3axJB3h5A6KZWJSyfSvUV3Vt25ivf6vhe7yVxHAVSqSjShO2zNzjX0eaMPZ0w7g+dXP0//1v35bth3vNLrFdo0buN2eO4pHgVwyxYQ+W0UQIeS+qxZs2jRogUtWrRg1qxZjixTKatplYtDsrZmkZmVyYIfFlArsRb3XnAvI84fwe9q/87t0LzBxVEAd+3axejRo8nOzsYYQ7t27ejRowf16tWzdblKWU1L6DYSEd774T0uev4iLnr+Ir7c/iUPdX6IrSO2MqHLBE3mJdkwCuCKFStIS0ujoKCAgwcPctZZZ7F27doTplu4cCFdunShfv361KtXjy5duvDBBx9UerlKuUVL6DY46jvK69++TubiTNbsXENynWSmdJvCwHMGUjOxptvheZMNowCee+659OjRg3/961/k5+fTr18//vjHP54w3fbt22nWrNmx102bNmX79u2VXq5SbtGEbqGCowW8sPoFHlnyCD/u/pGWDVvyQs8X6HN2HxLiE9wOz9vGjj3+TjpgySiADz74IOeeey5JSUlMmTKlikEq5W1a5WKBfYf3MWHxBJpPbs7QBUNpWKMh/7npP6z9y1r6t+mvyTwUNo0CmJeXx4EDB9i/fz8FBQVlTtOkSRO2bdt27HVOTg5NmjSp0nKVcoOOtlgFvxz8hcnLJvPEiifYe3gvl59+Ofd3up9LUi/Ri4Hwxsh/PXr04Oabb+ann35ix44dx24nV9KuXbto164dX331FQBt27Zl5cqVlt/hyAle+M2VvXS0RYtt2bOFiUsm8tyq5yg4WsD1La9nZMeRnNvkXLdDUyW8+OKLJCQk0KdPH4qKirjwwgv55JNPuPTSS4+brn79+jzwwAOce65//T344IMRmcyV0hJ6GNblriMzK5O538wlzsTRL60f93W8jzMbnul2aJ6kpUXn6W8e/bSEXkXLcpaRmZXJW9+9RY2EGvy1w1+554J7aFanWcX/rJRSDgkpoRtjugGTgXhghohklvr8HuAO4CiQCwwUkTL6oEUOEeHDHz9kXNY4Fm1eRL2kevz74n8zrMMwGtZo6HZ4qhK++eYbbrnlluPeq169Ol9++aVLESllrQoTujEmHngC6ALkACuMMW+LyLoSk60C2ovIIWPMUGACcJMdAdutyFfE/PXzyVycyVc7vqJJ7SY8dsVjDGo3iFqJtdwOT1XB2WefXfmbKysVAUIpoXcANorIjwDGmFeAnsCxhC4in5aYfhnQz8ognXD46GFeWvMSExZP4IddP9CifgtmXDODfmn9qF6tutvhKaVUhUJJ6E2AbSVe5wDnlTP97cD7ZX1gjBkMDAZIrsIVgFY6cOQAz658lseWPsb2/ds5p/E5vNbrNa5veT3xcfFuh6eUUiGztFHUGNMPaA9cXNbnIvIs8Cz4e7lYuexw5R3KY+ryqUxdPpVd+bvonNqZmT1n0uX0LtqHXCkVkUK5UnQ7ULI7R9PAe8cxxlwOpAM9ROSwNeFZL2dfDn/74G8kT0pm9Gej6ZTciaW3L+XT/p9yxe+v0GTuIh0OXamqCaWEvgJoYYxpjj+R3wz0KTmBMeYc4Bmgm4j8YnmUFvju1++YsHgCL615CZ/46HN2H0Z2HMlZp57ldmiK34ZDLx7KpXg4dLB99FwAunXrxrJly+jUqRPvvvuu/QtUygYVJnQROWqMGQYsxN9tcaaIfGuMeQjIFpG3gUeAWsDrgRLuVhHpYWPcIcv+XzaZWZnMXz+f6tWqc2e7O/n7hX8npW6K26GpElwcDh2Af/zjHxw6dIhnnnnG/oUpZZOQ6tBF5D3gvVLvPVji+eUWx1UlIsKnmz9lXNY4PvrxI+pUr8M/L/ond593N6fWPNXt8FQZbBgOnRUrVnD77bezfPlyioqK6NChA6+++mqZQ+hedtllLFq0qPILU8oDoupKUZ/4eGvDW2QuzmT59uU0rtWY8ZePZ0j7IZxc/WS3w1PlsGE49JDHQ1cqWkRFQi8sKmTuN3MZv3g8639dz+n1Tufpq56mf5v+JFVLcjs8FQKbhkPX8dBVTInohH6o8BDPffUcE5dOZOveraQ1SmPu9XO58awbqRYX0V8t5hTXk6en+6tZkpP9ybyq9efF46EXFhZSUFBAzZp6xygVvSIy6+3O380TK55g8peT+fXQr3RK7sRTVz3FlX+4UrsdRrC+fa1vAL3zzjsZM2YMP/30EyNHjixzPHSlokXEJfSZq2Yy/IPhHDhygKtaXMWoTqPolNzJ7bCUB4U6HjrARRddxIYNGzhw4ABNmzblueeeo2vXri5ErVTlRVxCT62bSo//14ORHUeS1ijN7XCUh916663ceuutAMTHx5c7quIXX3zhVFhK2SbiEvqlzS/l0uYnlrCUUirWRVxCV6qydDx0Fe00oStbiYhnGqqjfTx0t24nqbwjlMG5lKqUpKQk8vLyNNE4QETIy8sjKUmvu4hlWkJXtmnatCk5OTnk5ua6HUpMSEpKomnTpm6HoVykCV3ZJiEhgebNm7sdhlIxQ6tclFIqSmhCV0qpKKEJXSmlooRxqweCMSYXKGPA1JA0BH61MByraFzh0bjC59XYNK7wVCWuFBE5pawPXEvoVWGMyRaR9m7HUZrGFR6NK3xejU3jCo9dcWmVi1JKRQlN6EopFSUiNaE/63YAQWhc4dG4wufV2DSu8NgSV0TWoSullDpRpJbQlVJKlaIJXSmlooSnE7oxppsx5jtjzEZjzKgyPr/HGLPOGLPGGPOxMSbFI3ENMcZ8Y4xZbYzJMsa08kJcJaa7wRgjxhhHunOF8HsNMMbkBn6v1caYO7wQV2CaPwe2sW+NMXO9EJcx5vESv9X3xpg9Hokr2RjzqTFmVWCf7O6RuFIC+WGNMWaRMcaREcyMMTONMb8YY9YG+dwYY6YE4l5jjGlb5YWKiCcfQDywCTgdSAS+BlqVmuYSoEbg+VDgVY/EdXKJ5z2AD7wQV2C62sDnwDKgvRfiAgYA0zy4fbUAVgH1Aq9P9UJcpab/KzDTC3Hhb+gbGnjeCtjskbheB/oHnl8KvOTQNvYnoC2wNsjn3YH3AQOcD3xZ1WV6uYTeAdgoIj+KyBHgFaBnyQlE5FMRORR4uQxw4sgbSlz7SrysCTjR8lxhXAFjgPFAgQMxhROX00KJaxDwhIjsBhCRXzwSV0m9gZc9EpcAJwee1wH+55G4WgGfBJ5/WsbnthCRz4Fd5UzSE3hR/JYBdY0xp1VlmV5O6E2AbSVe5wTeC+Z2/Ec7u4UUlzHmLmPMJmACcLcX4gqc0jUTkQUOxBNyXAE3BE475xljmnkkrjOAM4wxi40xy4wx3TwSF+CvSgCa81uycjuuDKCfMSYHeA//2YMX4voauD7w/DqgtjGmgQOxVSTcHFchLyf0kBlj+gHtgUfcjqWYiDwhIr8HRgL/cjseY0wc8Bhwr9uxlOEdIFVE0oAPgVkux1OsGv5ql874S8LTjTF13QyolJuBeSJS5HYgAb2BF0SkKf7qhJcC253b/g5cbIxZBVwMbAe88ptZygs/djDbgZIltaaB945jjLkcSAd6iMhhr8RVwivAtXYGFFBRXLWBPwKLjDGb8dfZve1Aw2iFv5eI5JVYdzOAdjbHFFJc+EtMb4tIoYj8BHyPP8G7HVexm3GmugVCi+t24DUAEVkKJOEfhMrVuETkfyJyvYicgz9XICJ7bI4rFOHmkoo50ThQyQaFasCP+E8pixs7zio1zTn4G0RaeCyuFiWeXwNkeyGuUtMvwplG0VB+r9NKPL8OWOaRuLoBswLPG+I/PW7gdlyB6c4ENhO4ONAjv9f7wIDA85b469BtjS/EuBoCcYHnY4GHnPjNAstLJXij6FUc3yi6vMrLc+qLVfLH6I6/VLQJSA+89xD+0jjAR8BOYHXg8bZH4poMfBuI6dPyEquTcZWa1pGEHuLvNS7we30d+L3O9EhcBn811TrgG+BmL8QVeJ0BZDoRTxi/VytgcWA9rgau8EhcvYAfAtPMAKo7FNfLwA6gEP/Z3u3AEGBIie3riUDc31ixP+ql/0opFSW8XIeulFIqDJrQlVIqSmhCV0qpKKEJXSmlooQmdKWUihKa0JVSKkpoQldKqSjx/wGzAjrwjE8m/wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 画出参数更新之前的结果 \n",
    "w0 = w[0].data[0]\n",
    "w1 = w[1].data[0]\n",
    "b0 = b.data[0]\n",
    "\n",
    "plot_x = np.arange(0.2, 1, 0.01)\n",
    "plot_y = (-w0.numpy() * plot_x - b0.numpy()) / w1.numpy()\n",
    "\n",
    "plt.plot(plot_x, plot_y, 'g', label='cutting line')\n",
    "plt.plot(plot_x0, plot_y0, 'ro', label='x_0')\n",
    "plt.plot(plot_x1, plot_y1, 'bo', label='x_1')\n",
    "plt.legend(loc='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到分类效果基本是混乱的，我们来计算一下 loss，公式如下\n",
    "\n",
    "$$\n",
    "loss = -\\{ y * log(\\hat{y}) + (1 - y) * log(1 - \\hat{y}) \\}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 计算loss, 使用clamp的目的是防止数据过小而对结果产生较大影响。\n",
    "def binary_loss(y_pred, y):\n",
    "    logits = (y * y_pred.clamp(1e-12).log() + \\\n",
    "              (1 - y) * (1 - y_pred).clamp(1e-12).log()).mean()\n",
    "    return -logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意到其中使用 `.clamp`，这是[文档](http://pytorch.org/docs/0.3.0/torch.html?highlight=clamp#torch.clamp)的内容，查看一下，并且思考一下这里是否一定要使用这个函数，如果不使用会出现什么样的结果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.6824, grad_fn=<NegBackward>)\n"
     ]
    }
   ],
   "source": [
    "y_pred = logistic_regression(x_data)\n",
    "loss = binary_loss(y_pred, y_data)\n",
    "loss.backward()\n",
    "print(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "得到 loss 之后，我们还是使用梯度下降法更新参数，这里可以使用自动求导来直接得到参数的导数，感兴趣的同学可以去手动推导一下导数的公式"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 自动求导并更新参数\n",
    "for i in range(100):\n",
    "    # 算出一次更新之后的loss\n",
    "    y_pred = logistic_regression(x_data)\n",
    "    loss = binary_loss(y_pred, y_data)\n",
    "    \n",
    "    # calc grad & update w,b\n",
    "    loss.backward()\n",
    "    w.data = w.data - 0.1 * w.grad.data\n",
    "    b.data = b.data - 0.1 * b.grad.data\n",
    "\n",
    "    # clear w,b grad\n",
    "    w.grad.data.zero_()\n",
    "    b.grad.data.zero_()\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f51c31f1310>"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAvgUlEQVR4nO3deXxU1d348c9JSEjYBEIQIZCADSaAqBDcwBVBNoMM2gcFH1ER8ZFH69bSF+iDtvy0Sm1RaZVWZFVrzbCIICKIiooQLCiQgKwBZDOgsgQJyfn9MUMcQmYyk9x15vt+veZFZuYy9zt3+c6555x7jtJaI4QQwv3i7A5ACCGEMSShCyFElJCELoQQUUISuhBCRAlJ6EIIESXq2LXiZs2a6YyMDLtWL4QQrrRmzZrvtdapVb1nW0LPyMggPz/frtULIYQrKaV2BntPqlyEECJKSEIXQogoIQldCCGihCR0IYSIEpLQhRAiSlSb0JVSU5VSB5RS64O8r5RSLyqltiilvlZKdTE+zCg2ezZkZEBcnO/f2bPtjkgI4VLhlNCnAX1CvN8XyPQ/RgJ/r31YMWL2bBg5EnbuBK19/44cKUldCFEj1SZ0rfUnwKEQiwwEZmiflUBjpdR5RgUY1caOhePHz3zt+HHf60IIESEj6tBbAbsCnu/2v3YWpdRIpVS+Uir/4MGDBqza5YqKInu9hqRWx71k34lIWNooqrWeorXO0VrnpKZWeedqbGnTJrLXa0BqddxL9p2IlBEJfQ/QOuB5mv81UZ0JE6BevTNfq1fP97pBpFbHWFaWmGXfiUgZkdDnA//t7+1yOfCj1nqvAZ8b/YYOhSlTID0dlPL9O2WK73WDWFSrE7UCE3izZnD33daVmJ2276T6xwW01iEfwJvAXqAUX/34PcAoYJT/fQVMBrYC3wA51X2m1pquXbtqYb70dK196efMR3q63ZE536xZWterV/X2s2JbRrrvZs3yvaeU799Zs4yLpaptUa+esesQ4QHydbB8HewNsx8xmdDNPONCrFJOxJoJllArP5QyZ/2R7Duz97MUDJxDEroT2JhZbfgdiQpKhZfQzUxq4e47sxNusG1h1o+ZCC5UQle+962Xk5OjY2o89IwMX6VrZenpsGOH1dGIMATbZYHq1TO82aNG4uJ8KbYypaC8vPafL4evcyil1mitc6p6T8ZysYrTWrhEtarqhJSQACkpprVh15jZPWAt6JAlDCAJ3SoW9DkXxqqqE9Lrr8P33/tKvTt2OCOZg/kJ14IOWcIAUuVildN3iQR2LHbK9bqICrNn+/qoFxX5ygkTJsihFY1CVbnYNqdozDl9ZskZJ0wydKgcTrFOqlysNHSo7zrdadfrwlXkBh8RjJTQhXCRyjV3p+9WBSkfCCmhC+EqThzfRa4YnENK6EK4iNN6v8oVg7NICV0IF3Fa71cnXjHEMknoQriI027wcdoVQ6yThC6Eixh5g48Rdd9Nm0b2ujCX1KEL4TJG9DeXuu/oJCV0IWLI6VL5sGHG1H0fCjJ9fLDXhbkkoQvXkW5yNRM4R2kwkdZ9O62RNtZJQheuIhMn11xVPVIqizQRO62RtjrRXhiQhC4MY8XJIt3kaq660ndNErGbRmGMhcKAjLYoDGHVYJJmT+QQzUJN2JGeHv1jxUXLJB0ywYUwnVUlZ6mzrblg1SP33+/7+447oq8aIvCqMdiPWTT1mZeEHuOMqiax6gYTt9XZOklV1SN33gnTp0dnNUTlKpZgoqowEGyyUbMfMTdJtAMZOW+1lbPCy6TXxrFyv1kt2Hczc552K45NQkwSLQk9hhl5Mhv542C3WPrBUKrqY0ApuyOrvWDf7fT3M3rfWnUOSEIXVTL6ZI6GRBhNP0zVmTVL6/j42Cuhm/XdrFpfqIQudegxzOgGxmiYkMnubpFW9ZM+Xb9cVnb2e9HSJmF1e4sTBiqThB7DpIHxbHaelGb3kw78sbjzzqpvMoqPd24/8khZ3UfeET2wghXdzX5IlYszREM1iZHsbCQ0c91VVSVFa925XZxQhy4l9BgXDdUkRrLzqsXMq4NwbvuHKOvCZzEn3DUrCV2IAHaelGZesofzoxDr1W1GsLuAJAldiErsOinNvDoI9qMQH+/8MVhE+CShC+EQZl4dBPuxmD5dqtuiicxYJISDGDEbUbDPBV9delGRr8Qe7YNxxSJJ6ELECLN+LIRzSJWLEEJECUnoQggRJSShC2GgaJ/iTDib1KELYZDKszadvnUfpO5aWENK6EIYxO6BvYSQhC6EQZww2p6IbWEldKVUH6XUJqXUFqXUmCreb6OU+kgp9R+l1NdKqX7GhyqEszlitD0R06pN6EqpeGAy0BfoANymlOpQabFxwNta60uAIcDfjA5UmERa8QwjwxELu4VTQr8U2KK13qa1Pgm8BQystIwGGvn/Pgf4zrgQhWnMHoA7xtg1sJf8JovTwknorYBdAc93+18LNB4YppTaDSwE/reqD1JKjVRK5Sul8g8ePFiDcKOc1WemtOIZavZs62+tl99kEcioRtHbgGla6zSgHzBTKXXWZ2utp2itc7TWOampqQatOkrYcWZKK55h7Eqs8pssAoWT0PcArQOep/lfC3QP8DaA1voLIAloZkSAMcOOM1Na8QxjV2KV32RzuLUaK5yEvhrIVEq1VUol4mv0nF9pmSKgJ4BSKhtfQpc6lUjYcWZKK55h7Eqs8ptsPDdXY1Wb0LXWp4DRwGKgAF9vlg1KqaeVUrn+xR4F7lVKrQPeBIb7574T4bLjzHTCnFlRwq7E6obfZLeVdl1djRVsslGzHzJJdCVGzzDrktmf7QzTyHVbNUFwsHU7dVfbuV1qSilnT6BNiEmiJaE7iVFnpkvOIruToNHrdnJitUt6etXJMT3d7siCc3rMktCdzuhM4PQj0s/OMF2yiRwn0kPV6aXdqji9PCQJ3cnMOHpcchbZGaZLNpGj1ORQdesPp5OvtkIldBmcy25mtMC4pOuDnWG6ZBM5SrBD9c47gzd4uqHRtipDh/omznbbBNqS0O1mRn83l5xFdobpkk3kKMEOybKy4N37pCOVxYIV3c1+SJWLn1nXpE6+ZgwQLb1cYkGwQ9Vt1SluR4gqF+V733o5OTk6Pz/flnU7SuVpbsBXVJRijK3sGJfF6ao6VKuilK+qQphDKbVGa51T1XtS5WK36q5J3XZXRhSo6k7BO+7w7Z5Y3gWVD9X4+KqXk3YIGwUrupv9kCqXMDi9/1SUqq5qQXaBjxyekTOimg+pcnGpjAxf8bCy9HRf07swRVycLz2FIrvAR6qmwmdU7WqoKhdJ6E4WLLNIJaWpgv2OBpJdICJlVPlM6tDdyqjO0lIPH5GqujRWJvXEIlJWjMgpCd3JjOgs7eaxQG0S2PgHvtJ4IOmvLmrCipvZJKE7mRF3Zbh6LFD7nL5TUGuYOTO2boyRCzpzWHIzW7DWUrMf0svFIjJoieHceENSuDG7teeKW/aJ2b1cJKFHO5eOjuTUE9SNCS+SmN14uLhxn9SGJPRY5sKj3ckhuzHhRRJzsAu608s7YR9U5pR9YlUhRBJ6rHNqcTcIp5ygVXFjDVYkMbvxpion7JPqCiFGnoKS0IWrOOEEDcbJPzbBRBJzVYnJ6d/VCfskVAxGX3GGSujSy0U4jpPHKnfjsLuRxFy5y2ZVjOw3bQQn7JNQfcwt7WgWLNOb/ZASugjGyXXoWruuBktrXbOYnVDyDZfd+yTUtjL6ihOpchFuY/cJKpz/w+okobaV0T+MoRK6VLmI6tlwp4lbpwCLlJNv4pHZhsIXaltZWiUULNOb/ZASuktIMc00smljh1W9XKSELkKToQNMI5vWHEZd9Rh59WTVFWcdcz5WRA0rhoiLUbJpjVd5zPHTY9FBZEnUqM+xmpTQo4kZFbJO7kMYwMl10cG4ZNO6ilFXPW69epKEHi3MGibXCZ18q+HWEYJdsGldx6irHtdePQWrXDf7IY2iBjOz07DD+xC6qb90ZQ7ftK5j1LHg5GMKmVM0BsTwdHUx/NVFJUbN22nU55hBpqCLBdFWIRtBpXi0fXVRc0b1nXdtH/xgRXezH1LlYrBo6tQc4XeJpq8uRHWQfugxwLVFiipE2MUgmr66ELUhdejCeaRSXIigpA5duItUigtRI5LQhfNIB20hakQSunCe05XiKSm/vJacbF88JnDjna2i5qza35LQnUTO8jOVlPzyd3GxO27/DINb72wVNWPp/g7W/cXsh3RbrCRW+t6Fe2ukk2/Vq6Uo/mqiCjLBRSxy62hAkYikqGLgYBpOu/Bx7Tghokas3N9hJXSlVB+l1Cal1Bal1Jggy/xaKbVRKbVBKfWGsWHGgFg4yyP50TKop4sTqzekE09ssXJ/V5vQlVLxwGSgL9ABuE0p1aHSMpnA74HuWuuOwG+MDzXKxcJZHsmPlkE9XZx44SOdeGKLlfs7nBL6pcAWrfU2rfVJ4C1gYKVl7gUma60PA2itDxgbZgyIhbM8kh8tg27/dOKFj9zZGlus3N/V3imqlLoF6KO1HuF/fgdwmdZ6dMAyc4HNQHcgHhivtX6/is8aCYwEaNOmTdedO3ca9DWixOzZvqJjUZEvyU2YEF1nuQ1D2GVk+KpZKktP900FJoTbWHGnaB0gE7gWuA34h1KqceWFtNZTtNY5Wuuc1NRUg1YdRaJ9qnsbiqaxcOEjxGnhJPQ9QOuA52n+1wLtBuZrrUu11tvxldYzjQlRRBWLf7SkesN9nNYryU3CSeirgUylVFulVCIwBJhfaZm5+ErnKKWaAe2BbcaF+Ytth7ex6NtFnCw7acbHiygU7Rc+0cSJvZLcpNqErrU+BYwGFgMFwNta6w1KqaeVUrn+xRYDxUqpjcBHwONa62IzAp62dhr93uhH6vOpDPMOw1vg5djJY2asSlQmRSdhoKoOJyf2SnIT1w2fe+LUCZZuW4q3wMu8TfMoLikmuU4yfTP74sny0L99fxonNTY+4Fjn5Dm5hOsEO5wqJ/PTZOTkX4RqFHVdQg90qvwUn+z8hLyNeczdNJfvjnxHQlwCPdv1xJPlYWDWQJrXb25QxDFOuosIAwU7nOLjoazs7NflMPtF1Cb0QOW6nFV7VpG3MQ9voZdth7cRp+K4qs1VeLI9DMoaROtzWlf/QaJqMumEMFCwwwnOLqnLheCZYiKhB9Ja8/X+r/EWePEWell/YD0A3Vp2w5PtwZPtoX1Ke1PWHbWkhC4MFOpwmjAhum/HqK2YS+iVbS7ezJyCOXgLvazaswqATs074cnyJffO53ZGKWVJLK4ldejCQHI41VzMJ/RAu37cxZzCOeQV5LGiaAXlupx2TdpVJPfL0i4jTskglFWK9jtZhaXkcKoZSehBHDh2gHmF85hTOIcPt31IaXkpLRu2ZFDWIDzZHq5Ov5o6cXVsjVEIIQJJQg/Djyd+ZMHmBXgLvSz6dhElp0pISU4h94JcPNkeerXrRd06de0OUwgR4yShR+h46XEWb1lMXkEe725+l59+/omGiQ3p374/niwPfTP70iCxgd1hiigm1REiGCsG54oq9RLqMSh7ELM8szj4+EEWDV3EkE5DWLptKb9+59ekPp/KzW/dzIx1MzhcctjucGOXHXeuWrBOuf1d1FiwuenMfrhxTtFTZaf08u3L9YMLH9RpL6RpxqPrPF1H95rRS/999d/13iN77Q7RucKdSzSSz7N6DlaL1ilzjopQCDGnqFS51JDWmtXfrWZOga/HzLeHvkWhuLL1lQzOHsyg7EFkNM6wO0xnMKOPmh394i1ap9zDJUKROnSTaa3ZcHADeRvzmFM4h3X71wHQ5bwueLI8DO4wmKxmWTZHaSMzEqEdWc+idco9XCIUSegW23poa8Vdqit3rwQgu1l2xV2ql7S4JLZuZDIjEUZxCV1uuhGhSKOoxc5vej6Pd3+cL+75gt0P7+blvi/TokELnl3xLF2ndKXtpLY8svgRVhStoKy8ipGIoo0ZE2DbMRVRv36+HyGT1ymTcogaC1a5bvbDjY2iIYXR6Hfw2EE99aupesAbA3TiHxI149EtJrbQo94dpT/Y8oE+eeqk5WFbIlhj4v33166h1OiG1urWVfk7KOX7DkJYCGkUNVkNrpF/+vknFn67EG+Bl4XfLuRY6TGaJDXhpgtuwpPloff5vUlOSLboC1igcsfqfv1g+nT31CtIxbZwCKlDN1stT/aS0hI+2PoBcwrnMH/TfA6fOEz9hPr0zezL4OzB9MvsR6O6jQwP21bBtllKCnz/veXhVEu6ngiHkDp0sxUVRfZ6JckJyQzMGsi0m6ex/7H9fDDsA4Z1HsanOz/ltrzbSH0+lQFvDGDqf6by/XEHJruaCLZtioudeQeNGe0AMUpmMjSPlNCNYNLleFl5GSt3rySvIA9vgZedP+4kXsVzTcY1eLI83Jx1M60atarx59sq2DYDZ1ZjSNcTQ8hmrL1QJXRpFDWCBXcQlpeX6zXfrdFjl47VWS9nacajGY++/J+X6+dWPKe3FG8xbF2WmDWr6tshTzc2OpGVjbA2MvNryl2wtYc0ilrA4tGUCg4WVIzr/tXerwC46NyLKvq6d0zt6Py+7s2a+apYKnNiCT1GmF2ClqaI2pMSuhtFUEzafni7fuHzF3T317prNV5pxqMzX8zUY5aM0at2r9Ll5eWWhR2Rqq5sEhK0TkmJ+lLwWRxS+je7BC0l9NojRAldEroT1aIK57ufvtN/W/U33WtGL13n6Tqa8ejWL7TWDy58UC/fvlyfKjtlwReIQGAiS0nROjHR1KorR7J4oLFQvx1KmVsLZseYatFGErrbGFSMKT5erKevna5z38zVSX9M0oxHpz6XqkfMG6EXbl6ofz71synh11isFt8s/N7VJVQrQnHIxYhrhUroUofuRCZUNB49eZT3t7xPXkEeCzYv4OjJozSq24ib2t+EJ9vDjeffSP3E+rUMvBZmz4Zhw6p+L9orWC2sWK6uQ5b0Qjmb0yYbkTp0tzG5mFRSWqIXbFqg7557t075U4pmPDr5j8l60FuD9Mx1M/XhksOGrCdsVRUbpYRuyvcOp0olGkrQRn0HJ1YRIVUuLmPhUVRaVqqXbVumR783Wrf8c0vNeHTC0wm6z6w+ekr+FL3/6H7D13mWYAnNCWePFSzc32b8djjtB8DIzenEWkBJ6G5kw1lSVl6mPy/6XD/+weO63aR2mvHouKfi9NWvX60nrZyki34oMmfFwYqNYH92sIpF+9vo3w4nlmCNTMJmNxLXhCR0q9TmpHRYMae8vFyv3btWP7HsCd3pb50qbmTqNqWbfubTZ/Sm7zcZtzInFoOimJGHmhN3nZFJ2InfTxK6FWpTVHFiMaeSTd9v0s9++qzuNqVbRXLvOLmjfmLZE3rt3rW16+vugu8vqmZE8jS6LGNkEnbioSkJ3QrhHkVVHb1OLAaEUPRDkZ60cpK++vWrddxTcZrx6HaT2unHFj+mPy/6XJeVl0X+oQ67QhHhqe2ha0bCDNbGnpJSs8912qEpCd0K4XYfqOroDVZ/7NQxTQLsP7pfT8mfovvM6qMTnk7QjEe3/HNL/cB7D+il25bq0rJSu0N0NqdliwiFlZBDfEezyjKzZvkSeOXPtbt0bQRJ6FYI58gMtkx8vKtK6MEcLjmsZ62bpT3/8ujkPyZrxqNT/pSi75p7l16waYEuKS2xO8TwWZFonXg9XwMhN1U139HMRkeXXfiGTRK6FcI5OUP15oiCEzvQsZPHdN7GPD00b6hu9EwjzXh0g//XQP/Xv/9Lv73+bX3k5yN2hxicVYk2WjNOoGq+o5mbwIk9VIwgCd0q1ZXqQh29Lr/0DuXnUz/rRd8u0iPmjdCpz6VqxqPr/qGuzn0zV0/7zzRdfLzY7hDPZFWitSPjWH2cVfMdzfztjNbfS0noThEll9i1carslP54x8f6oUUP6dYvtNaMR9d5uo6+YcYN+m+r/qb3Htlrd4jWjFAV6mYqszKOHcdfGFnVrN+YaD3dJKE7SRSXxCNVXl6uV+1epccsGaMzX8zUjEer8Up3f627/vPnf9bbD2+3JzAzi3bVDXNgZsaxo8g6a9bZI2gmJlp23Efj6RYqocvgXMIRtNZsOLgBb4EXb4GXdfvXAdDlvC54snyTdmSnZlsTjJkjVFU39Z6ZIz/ZMbvE7Nlw111QWvrLawkJ8PrrsTvaVy2FGpxLErpwpK2HtjKncA7eAi9f7P4CgKxmWQzOHown28MlLS4xd0Yms4bYs3PKHpPmvnXcOqNcrRO6UqoPMAmIB/6ptX42yHKDgXeAblrrkNlaEroI156f9jC3cC7eQi8f7/iYMl1G+jnpFdPtXZF2BfFx8XaHGR47E5wdY+PKnHOGC5XQ48L4z/HAZKAv0AG4TSnVoYrlGgIPAV/WLlwhztSqUSseuPQBlv73UvY9to+puVPp1LwTk1dP5qrXryLtL2ncv+B+lmxdQmmZ/9J+9mxf8oyL8/07e7adX+EXEyb4kmigevV8r5tt6FBf8k5P9yXU9HTzBzpv0yay10XtBKtcP/0ArgAWBzz/PfD7Kpb7K9AfWA7kVPe5MdsoKgzz44kf9ZvfvKlvfftWXX9Cfc14dJNnm+i//aaHLk2q69zuDdHYUheMW7qauGifUJteLsAt+KpZTj+/A3i50jJdgDz/30ETOjASyAfy27RpY9kGENHv+Mnjel7hPH3nnDv1zsZx1vfmEME5PVlG+qNj8/cJldCrrUNXSt0C9NFaj/A/vwO4TGs92v88DlgGDNda71BKLQce01KHLmyi4+JQVRzX5cC0r14j94JcmtVrZn1gwpkiaddwwBx9tapDB/YArQOep/lfO60h0AlYrpTaAVwOzFdKVT3nnRAmU0HqZ79rEs898++hxcQW9JzRk8mrJvPdke8sjk44TlFR+K+PHXtmMgff87FjjY+rBsJJ6KuBTKVUW6VUIjAEmH/6Ta31j1rrZlrrDK11BrASyK2uhC6EaYI0PLZ6aTprRq7hd91/x3dHvmP0otG0eqEVV752JRM/n8jWQ1vtiVfYK5KG20iSvw2qTeha61PAaGAxUAC8rbXeoJR6WimVa3aAQkQsSG8ONXQoXc7rwoSeEyh4oICN/7ORCddP4Oeyn3l8yeP86qVfcfErF/P0x0+z/sB6qquOFFEikp5HDu+1IzcWCQHs+GEHcwrmkFeQx+e7PkejyWyaiSfbw+DsweS0zDH3RiZhr3BvJHN4HbokdCEq2Xd0H/MK5+Et9LJs+zJOlZ+idaPWDMoahCfbQ482PdxzI5Mwnll3EYdJErqIDSacaIdLDvPu5nfxFnhZvHUxJ06dILVeKgMvGIgn20PPdj1JjE806AsIUT3XJPTS0lJ2797NiRMnbIkp1iQlJZGWlkZCQoLdodSeBZfCR08eZdG3i/AWelmweQFHTx6lUd1G3NT+JjzZHvr8qg/1EupV/0FC1IJrEvr27dtp2LAhKSkpUl9pMq01xcXFHDlyhLZt29odTu1ZPEbKiVMnWLptKXkFeczfNJ/ikmKS6yTTN7MvniwP/dv3p3FSY8PXK0SohF7H6mBCOXHiBBkZGZLMLaCUIiUlhYMHD9odijEs7k6WVCeJ/u370799f06Vn+LTnZ/6hv4t9A3/mxCXQM92PfFkeRiYNZDm9ZubEocQgcLph24pSebWiaptbWN3sjpxdbiu7XW81O8ldj28iy/u+YLfXP4bNhdvZuSCkZz35/O4Zto1vPjli+z6cZfp8YjY5biELkSN2DmKYYA4FcflaZfzXK/n2PK/W1h731rGXTWOQyWHeOj9h2jz1zZc+o9L+dOKP/Ft8beWxiainyT0WtixYwdvvPFGxfO1a9eycOHCiufz58/n2WerHDo+YsOHD+edd94BYMSIEWzcuNGQz40adgwNWw2lFBe1uIinrnuKb+7/hk2jN/FMz2cAGLN0DO1fbs+Ff7+Q//vo/1i3b53cyCRqzVGNogUFBWRnWzTNmAGWL1/OxIkTWbBgAQDTpk0jPz+fl19+2fB1DR8+nAEDBnDLLbcY+rlu2+bRoujHIt+kHQVePi36lHJdTrsm7Sqm27ss7TLilJS3xNlc0yga6Dfv/4a1+9Ya+pkXt7iYv/b5a8hlZsyYwcSJE1FK0blzZ2bOnHlWMm3QoAFHjx5lzJgxFBQUcPHFF3PbbbcxefJkSkpKWLFiBb///e8pKSmpSPDDhw+nUaNG5Ofns2/fPp577jluueUWysvLGT16NMuWLaN169YkJCRw9913h0zc1157LRMnTiQnJ4cGDRrw0EMPsWDBApKTk5k3bx7nnnsuBw8eZNSoURT5GwX/+te/0r17d8O2paidNue04cHLHuTByx7kwLEDzN80n7yCPCZ9OYmJX0ykZcOWFTcyXZ1+NXXiHHuqCgeRoyTAhg0b+OMf/8jnn39Os2bNOHToUMjln3322TNK6Oeee+4ZJfRp06adsfzevXtZsWIFhYWF5Obmcsstt+D1etmxYwcbN27kwIEDZGdnc/fdd4cd87Fjx7j88suZMGECv/3tb/nHP/7BuHHjeOihh3j44Yfp0aMHRUVF3HjjjRQUFES2QYQlmtdvzoguIxjRZQQ/nviR9759D2+Bl9fXvs7k1ZNJSU4h94JcPNkebmh3A0l1kuwOWTiUYxN6dSVpMyxbtoxbb72VZs18Y2U3bdrU0M+/+eabiYuLo0OHDuzfvx+AFStWcOuttxIXF0eLFi247rrrIvrMxMREBgwYAEDXrl1ZsmQJAB9++OEZ9ew//fQTR48epUGDBgZ9G2GGc5LO4fYLb+f2C2/neOlxFm9ZXNEV8vW1r9MgsQH9M/szOHswfTP70iBR9qf4hWMTupPUqVOHcv+EtuXl5Zw8ebJGn1O3bt2Kv41qu0hISKjofhgfH8+pU6cAX5wrV64kKUlKc25VL6Eeg7IHMSh7ECfLTrJs+zK8BV7mFs7lXxv+Rd34utz4qxvxZHnIvSCXJslN7A5Z2ExaXQJcf/31/Pvf/6a4uBigosolIyODNWvWAL6eK6WlvomIGzZsyJEjRyr+f+Xn4ejevTt5eXmUl5ezf/9+li9fbsA3gd69e/PSSy9VPF+7dq0hnyvskRifSJ9f9WHKTVPY++hePh7+Mfd1vY+v9n7F8HnDaT6xOb1n9uaV/FfYd3Rf6A9z6gTaotYkoQfo2LEjY8eO5ZprruGiiy7ikUceAeDee+/l448/5qKLLuKLL76gfv36AHTu3Jn4+Hguuugi/vKXv3DdddexceNGLr74Yv71r3+Ftc7BgweTlpZGhw4dGDZsGF26dOGcc86p9Xd58cUXyc/Pp3PnznTo0IFXXnml1p8pnCE+Lp6r069mUt9JFP2miFUjVvHoFY+y44cd3P/e/bT8c0t6TO3BC1+8wI4fdpz5n0+PebNzp2/2zJ07fc8lqUcF6bboAKfrtouLi7n00kv57LPPaNGihSXrjtVtHo201mw4uAFvgZe8gjy+3v81AF3O61LRHTK7W19Lx7wRxnNlt8VYMmDAAH744QdOnjzJE088YVkyF9FFKUWn5p3o1LwTT17zJFsPba0YX2bcR+MY99E4ynYGuSx3yBRqonYkoTuAUfXmQgQ6v+n5PN79cR7v/jh7ftrD3MK5HHjpUVoU/3zWsrp1a6JoZJ+YJXXoQsSAVo1a8cClD9Bi0mvo5OQz3juWAP/T/QdGLRjFkq1LKC0rtSlKUVtSQhcilgwd6iuJ+2d2Km+dxrr/yeX7dvtZ9PUsXl3zKo2TGvtuZMry0Pv83iQnJFf3qcIhJKELEWuGDq0YtCwOuNL/KCkt4YOtH+At9DJ/03xmrJtB/YT69M3sy+DswfTL7Eejuo3sjFxUQxK6EAKA5IRkBmYNZGDWQErLSlm+Yzl5BXnMLZzLOxvfITE+kV7teuHJ9t3I1KxeM7tDFpW4uw5dbpAQwhQJ8Qn0Or8Xrwx4hT2P7OHTuz7lgW4PsP7Aeu6Zfw8tJrag54yeTF41mT0/7bE7XOHn3oRu8w0S06dPJzMzk8zMTKZPn27JOoWwQ3xcPD3a9OCFG19g+0Pbyb83nzE9xrD3yF5GLxpN2l/SuOK1K3j+s+fZemir3eHGNPfeWGTxpMCBDh06RE5ODvn5+Sil6Nq1K2vWrKFJE/eNpSE3FonaKDhYUNHX/au9XwHQ+dzOeLI8DO4wmI6pHaNrqkMHCHVjkXtL6CZMCrx69Wo6d+7MiRMnOHbsGB07dmT9+vVnLbd48WJ69epF06ZNadKkCb169eL999+v8XqFcKvs1GzGXj2WNSPXsP2h7bzQ+wUaJjbkqY+f4sK/X8gFL1/AmA/HsHrPapmRyQLubRRt06bqEnotJgXu1q0bubm5jBs3jpKSEoYNG0anTp3OWm7Pnj20bt264nlaWhp79kg9oohtGY0zePiKh3n4iofZd3QfcwvnkleQx8TPJ/Knz/5E60atKybt6NGmB/Fx8XaHHHXcm9AnTPDVmR8//strBkwK/OSTT9KtWzeSkpJ48cUXaxmkELGpRYMWjMoZxaicURwqOcS7m97FW+jl1TWv8uKqF0mtl8rNWTfjyfZwfdvrSYxPtDvkqODeKheTJgUuLi7m6NGjHDlyhBMnTlS5TKtWrdi1a1fF8927d9OqVatarVeIaNU0uSl3Xnwn84bM4/vffs/bt7xNz3Y9eXP9m/Sd3ZfU51MZ5h2Gt8DL8dLj1X+gCMq9jaImyc3NZciQIWzfvp29e/dWOeHzoUOH6Nq1K1995WsE6tKlC2vWrDF8hiMrOGGbi9h04tQJPtz2Id4CL/M2zeNQySGS6yTTN7MvniwPA9oP4Jyk2g8lHW1ktMUwzZgxg4SEBG6//XbKysq48sorWbZsGddff/0ZyzVt2pQnnniCbt26Ab5qGjcmcyHslFQniQHtBzCg/QBOlZ/ik52fkLcxj7mb5uIt8JIQl8AN7W6ouJGpef3mdofseFJCj3GyzYXTlOtyVu1ZVTGu+7bD24hTcVzV5io82R4GZQ2i9Tmtq/+gKBWqhC4JPcbJNhdOprXm6/1fVyT3DQc3ANCtZTc82R4GZw8mMyXT5iitJQm9hr755hvuuOOOM16rW7cuX375pU0RGc9p21yIUDYXb2ZOwRy8hV5W7VkFQKfmnSpmZOp8bueov5FJEroISra5cKtdP+6q6Ov+adGnlOty2jVpV5HcL0u7jDjl3o58wUhCF0HJNhfR4MCxA8wrnIe30MvSbUspLS+lZcOWFTcyXZ1+NXXioqMPiPRyEUJEteb1m3Nv13u5t+u9/HDiB97b/B55BXlM/c9UJq+eTEpyCgMvGIgn28MN7W6gbp26dodsCknoQoio0jipMUM7D2Vo56EcLz3O4i2L8Rb6GlWnrp1Kw8SG9G/fH0+Wh76ZfWmQ2MDukA3j6gomGQ5dCBFKvYR6DMoexMxBMznw+AEWDV3EkE5DWLptKb9+59c0e64ZA98ayPS10zlcctjucGstrISulOqjlNqklNqilBpTxfuPKKU2KqW+VkotVUqlGx/qmWweDp0+ffrQuHFjBgwYYM0KhRC1khifSJ9f9WHKTVPY++hePh7+Mfd1vY+v9n7F8HnDaT6xOb1n9uaV/FfYd3Sf3eHWSLWNokqpeGAz0AvYDawGbtNabwxY5jrgS631caXU/cC1Wuv/CvW5tW0UtXE4dACWLl3K8ePHefXVV1mwYIH5KzSJNIqKWKe1ZvV3q5lTMIe8gjy+PfQtCkX3Nt3xZHkYlD2IjMYZdodZobbjoV8KbNFab9NanwTeAgYGLqC1/khrfXpUnZVAWm0CDocJw6GHPR46QM+ePWnYsGHNVyaEcASlFJe2upRnbniGTaM3sf7+9Yy/djxHfj7CIx88QttJbek6pSsTPplAwcECu8MNKZxG0VbAroDnu4HLQix/D7CoqjeUUiOBkQBtajFuue//Gz4cetjjoQshopNSio7NO9KxeUeevOZJth7aypxCX8l93EfjGPfROLKaZTE4ezCebA+XtLjEUTcyGdooqpQaBuQAz1f1vtZ6itY6R2udk5qaWqt1TZjgG/48kAHDofPkk0+yZMkS8vPz+e1vf1u7DxNCuNr5Tc/nsSsf44t7vmD3w7t5ue/LtGzYkmdWPEPXKV1pO6ktjyx+hM+KPqNcl9sdblgJfQ8QOBJOmv+1MyilbgDGArla65+NCS84k4ZDD2s8dCFE7GnVqBUPXPoAS/97Kfsf28/U3KlceO6FTF49mR6v96DVC624f8H9LNm6hNKyUltiDKfKZTWQqZRqiy+RDwFuD1xAKXUJ8CrQR2t9wPAogxg6tPYJvLL77ruPP/zhD2zfvp3f/e53VY6HLoSIbc3qNeOuS+7irkvu4qeff2LhtwvxFniZ+fVMXlnzCo2TGpN7QS6eLA+9z+9NckKyJXFVm9C11qeUUqOBxUA8MFVrvUEp9TSQr7Wej6+KpQHwb399UpHWOtfEuE0R7njoAFdddRWFhYUcPXqUtLQ0XnvtNW688UYbohZC2KlR3UYM6TSEIZ2GUFJawgdbP8Bb6GX+pvnMWDeD+gn1efSKR3nquqdMj0XGcolxss2FMEdpWSnLdyzHW+Cla8uujOgywpDPlbFchBDCYgnxCfQ6vxe9zu9l2ToloYcQC+OhCyGih+MSutbaMf06L7zwQtauXWt3GKaxq7pNCGEORw3OlZSURHFxsSQaC2itKS4uJikpye5QhBAGcVQJPS0tjd27d3Pw4EG7Q4kJSUlJpKWZPkqDEMIijkroCQkJtG3b1u4whBDClRxV5SKEEKLmJKELIUSUkIQuhBBRwrY7RZVSB4EqBsANSzPgewPDMYrEFRmJK3JOjU3iikxt4krXWlc5XK1tCb02lFL5wW59tZPEFRmJK3JOjU3iioxZcUmVixBCRAlJ6EIIESXcmtCn2B1AEBJXZCSuyDk1NokrMqbE5co6dCGEEGdzawldCCFEJZLQhRAiSjg6oSul+iilNimltiilxlTx/iNKqY1Kqa+VUkuVUukOiWuUUuobpdRapdQKpVQHJ8QVsNxgpZRWSlnSnSuM7TVcKXXQv73WKqWMmdqllnH5l/m1/xjboJR6wwlxKaX+ErCtNiulfnBIXG2UUh8ppf7jPyf7OSSudH9++FoptVwpZcmIdEqpqUqpA0qp9UHeV0qpF/1xf62U6lLrlWqtHfnAN3/pVqAdkAisAzpUWuY6oJ7/7/uBfzkkrkYBf+cC7zshLv9yDYFPgJVAjhPiAoYDLzvw+MoE/gM08T9v7oS4Ki3/v/jm+bU9LnwNfff7/+4A7HBIXP8G7vT/fT0w06Jj7GqgC7A+yPv9gEWAAi4HvqztOp1cQr8U2KK13qa1Pgm8BQwMXEBr/ZHW+rj/6UrAil/ecOL6KeBpfcCKludq4/L7A/An4IQFMUUSl9XCieteYLLW+jCA1vqAQ+IKdBvwpkPi0kAj/9/nAN85JK4OwDL/3x9V8b4ptNafAIdCLDIQmKF9VgKNlVLn1WadTk7orYBdAc93+18L5h58v3ZmCysupdQDSqmtwHPAg06Iy39J11pr/Z4F8YQdl99g/2XnO0qp1g6Jqz3QXin1mVJqpVKqj0PiAnxVCUBbfklWdsc1HhimlNoNLMR39eCEuNYBHv/fg4CGSqkUC2KrTqQ5rlpOTuhhU0oNA3KA5+2O5TSt9WSt9fnA74BxdsejlIoDXgAetTuWKrwLZGitOwNLgOk2x3NaHXzVLtfiKwn/QynV2M6AKhkCvKO1LrM7EL/bgGla6zR81Qkz/ced3R4DrlFK/Qe4BtgDOGWbGcoJGzuYPUBgSS3N/9oZlFI3AGOBXK31z06JK8BbwM1mBuRXXVwNgU7AcqXUDnx1dvMtaBitdntprYsD9t0/ga4mxxRWXPhKTPO11qVa6+3AZnwJ3u64ThuCNdUtEF5c9wBvA2itvwCS8A1CZWtcWuvvtNYerfUl+HIFWusfTI4rHJHmkupZ0ThQwwaFOsA2fJeUpxs7OlZa5hJ8DSKZDosrM+Dvm4B8J8RVafnlWNMoGs72Oi/g70HASofE1QeY7v+7Gb7L4xS74/IvlwXswH9zoEO21yJguP/vbHx16KbGF2ZczYA4/98TgKet2Gb+9WUQvFG0P2c2iq6q9fqs+mI13Bj98JWKtgJj/a89ja80DvAhsB9Y63/Md0hck4AN/pg+CpVYrYyr0rKWJPQwt9cz/u21zr+9shwSl8JXTbUR+AYY4oS4/M/HA89aEU8E26sD8Jl/P64FejskrluAb/3L/BOoa1FcbwJ7gVJ8V3v3AKOAUQHH12R/3N8YcT7Krf9CCBElnFyHLoQQIgKS0IUQIkpIQhdCiCghCV0IIaKEJHQhhIgSktCFECJKSEIXQogo8f8BnXtj/dYQn3QAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 画出参数更新之前的结果\n",
    "w0 = w[0].data[0]\n",
    "w1 = w[1].data[0]\n",
    "b0 = b.data[0]\n",
    "\n",
    "plot_x = np.arange(0.2, 1, 0.01)\n",
    "plot_y = (-w0.numpy() * plot_x - b0.numpy()) / w1.numpy()\n",
    "\n",
    "plt.plot(plot_x, plot_y, 'g', label='cutting line')\n",
    "plt.plot(plot_x0, plot_y0, 'ro', label='x_0')\n",
    "plt.plot(plot_x1, plot_y1, 'bo', label='x_1')\n",
    "plt.legend(loc='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4 torch.optim\n",
    "上面的参数更新方式其实是繁琐的重复操作，如果我们的参数很多，比如有 100 个，那么我们需要写 100 行来更新参数，为了方便，我们可以写成一个函数来更新，其实 PyTorch 已经为我们封装了一个函数来做这件事，这就是 PyTorch 中的优化器 `torch.optim`\n",
    "\n",
    "使用 `torch.optim` 需要另外一个数据类型，就是 `nn.Parameter`，这个本质上和 Variable 是一样的，只不过 `nn.Parameter` 默认是要求梯度的，而 Variable 默认是不求梯度的\n",
    "\n",
    "使用 `torch.optim.SGD` 可以使用梯度下降法来更新参数，PyTorch 中的优化器有更多的优化算法，在本章后面的课程我们会更加详细的介绍\n",
    "\n",
    "将参数 w 和 b 放到 `torch.optim.SGD` 中之后，说明一下学习率的大小，就可以使用 `optimizer.step()` 来更新参数了，比如下面我们将参数传入优化器，学习率设置为 1.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 使用 torch.optim 更新参数\n",
    "from torch import nn\n",
    "\n",
    "w = nn.Parameter(torch.randn(2, 1))\n",
    "b = nn.Parameter(torch.zeros(1))\n",
    "\n",
    "def logistic_regression(x):\n",
    "    return torch.sigmoid(torch.mm(x, w) + b)\n",
    "\n",
    "optimizer = torch.optim.SGD([w, b], lr=1.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 200, Loss: 0.42650, Acc: 0.90000\n",
      "epoch: 400, Loss: 0.33615, Acc: 0.92000\n",
      "epoch: 600, Loss: 0.29681, Acc: 0.91000\n",
      "epoch: 800, Loss: 0.27461, Acc: 0.91000\n",
      "epoch: 1000, Loss: 0.26027, Acc: 0.90000\n",
      "\n",
      "During Time: 0.348 s\n"
     ]
    }
   ],
   "source": [
    "# 进行 1000 次更新\n",
    "import time\n",
    "\n",
    "start = time.time()\n",
    "for e in range(1000):\n",
    "    # 前向传播\n",
    "    y_pred = logistic_regression(x_data)\n",
    "    loss = binary_loss(y_pred, y_data) # 计算 loss\n",
    "    \n",
    "    # 反向传播\n",
    "    optimizer.zero_grad() # 使用优化器将梯度归 0\n",
    "    loss.backward()\n",
    "    optimizer.step() # 使用优化器来更新参数\n",
    "    \n",
    "    # 计算正确率\n",
    "    mask = y_pred.ge(0.5).float()\n",
    "    acc = (mask == y_data).sum().item() / y_data.shape[0]\n",
    "    if (e + 1) % 200 == 0:\n",
    "        print('epoch: {}, Loss: {:.5f}, Acc: {:.5f}'.format(e+1, loss.item(), acc))\n",
    "during = time.time() - start\n",
    "print()\n",
    "print('During Time: {:.3f} s'.format(during))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到使用优化器之后更新参数非常简单，只需要在自动求导之前使用**`optimizer.zero_grad()`** 来归 0 梯度，然后使用 **`optimizer.step()`**来更新参数就可以了，非常简便\n",
    "\n",
    "同时经过了 1000 次更新，loss 也降得比较低了"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面我们画出更新之后的结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f51c3110f50>"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAzFElEQVR4nO3dd3wU1fr48c9JSOgIBBSvoYhfvJogNdgQURRF4WIDlQsKIl1Frz9REK83UhRQ77XQlN4UARsqitIEpEiA0IINBaUICIpUU/b5/bGJhpBNdpOZnZnN83699pXs7uycZ2dnnz1z5sw5RkRQSinlfVFOB6CUUsoamtCVUipCaEJXSqkIoQldKaUihCZ0pZSKEKWcKrhatWpSp04dp4pXSilPWr9+/S8iUj2/5xxL6HXq1CElJcWp4pVSypOMMbsCPadNLkopFSE0oSulVITQhK6UUhFCE7pSSkUITehKKRUhNKErpVSE0ISulFIRwnMJ/atfvuKpJU/xR+YfToeilFKu4rmE/sHXHzB8xXAav9aYtbvXOh2OUkq5hucS+oDmA/i488ccSz/GlZOv5LFPH+NExgmnw1JKKcd5LqEDtPm/Nmztt5WeTXry4uoXaTi+ISt2rXA6LKWUcpQnEzpApdKVGN9uPIvvXUyWL4urp17NQwse4lj6MadDU0opR3g2oedodX4rNvfdTP9L+zNm3RguGXcJi75f5HRYSikVdp5P6AAVYivw8k0vs/y+5cRGx9J6Rmt6fdCLI6eOOB2aUkqFTUQk9BxX1bqK1N6pDLhyAJM2TqL+uPp8/O3HToellFJhEVEJHaBsTFlGtR7F6vtXU6l0JW5+42a6vteVwycPOx2aUkrZKuISeo5Lz7uUDb02MLjFYGZtnkXi2ETe++o9p8NSSinbRGxCByhdqjTDWg1jXc91nFP+HG576zbunnc3B48fdDo0pZSyXEQn9ByNz23Mup7rGHbtMN7Z/g4JYxN4a+tbiIjToSmllGVKREIHiImOYfDVg9nYeyPnVz6fu9++mzvm3MHPx352OjSllLJEoQndGDPZGHPAGLM1wPPGGPOKMeY7Y8xmY0wT68O0TuLZiay6fxUjrx/Jgm8XkDAmgRmbZjhXW581C+rUgago/99Zs5yJQynlecHU0KcCbQp4/iagXvatFzCu+GHZq1RUKR5v/jib+mzi4uoXc+9799LuzXbs/n13eAOZNQt69YJdu0DE/7dXL03qSqkiKTShi8hyoKA+f7cA08VvDVDZGHOuVQHa6e/V/s7ybst5uc3LLNu5jMSxiUxYPyF8tfXBg+FEnoHFTpzwP66UUiGyog39POCnXPd3Zz92BmNML2NMijEm5eBBd/Q0iY6Kpv9l/dncZzNNzm1Crw970XpGa3b+ttP+wn/8MbTHVYmjLXIqFGE9KSoir4tIkogkVa9ePZxFF+qCqhew+N7FjGs7jrV71lJ/bH3GfDkGn/jsK7RWrdAeLyJNCt6kLXIqVFYk9D1AzVz347Mf85woE0WfpD5s67eN5rWa8+DHD3LN1Gv49tC39hQ4fDiUK3f6Y+XK+R+3iCYFa4Xzx1Fb5FTIRKTQG1AH2BrgubbAx4ABLge+DGadTZs2FTfz+XwyacMkOeu5s6TssLLy4qoXJTMr0/qCZs4UqV1bxBj/35kzLV197doi/lR++q12bUuLiVi5P564OJHY2NO3Y7lyln9kfzIm/8/OGHvKK4zNu6oKEpAigXJ1oCf+XADeBPYBGfjbx+8H+gB9sp83wBhgB7AFSCpsneKBhJ5jz+975B9v/ENIRi6bcJlsO7DN6ZBC4rak4CUzZ/oTdn7bLxw/jqH+GNuZcPPbFnb+mKnAipXQ7bp5JaGL+Gvrb2x+Q+JGxkns0Fh5dvmzkpGVEfqKHKjiaA296AJtu3D9OIaSRO1OuLofuYcmdIvsP7ZfOs7pKCQjTV9rKpt+3hT8ix2q4mjNqugCHd2EM6kFWwewO+HqkZ57aEK32Lxt8+Ts58+WmCExkrw0Wf7I/KPwFzlYxdG2z6IJpobulh9HuxOu1tDdo6CEXmLGcrHSHQl3kNYvjTsT7yT582SSXk9i/d71Bb/IwT7nnTvDzp3g8/n/du5se5ERIb9OSDExEBcHxkDt2vD66+7Ynnb3gA1DhyxlAU3oRRRXLo6Zt89k/t3zOXTyEJdNvIwnFz/JqcxT+b8gTH3OlXU6d/Yn7Nq1/0rgU6bAL7+478fR7oSb37Zwy4+ZyiVQ1d3um5ebXPL69eSv0v297kIyctHoi2TVj6vOXEgbs5XNtGmtZECbXOxVuUxlJt0yiYVdFnIi4wTNJzfn0YWPciIj11UhWsVRFgl0cZM2rSnjT/jhl5SUJCkpKY6UbaejfxzliUVPMC5lHBdUuYBJ7SfRsk5Lp8NSESLnyt/cV5CWK6d1g5LEGLNeRJLye05r6BarWLoiY9uOZWnXpQjCNdOu4YGPHuBY+jGnQ1MRwI3DAehYQe6hCd0m19S5hs19NvPIZY8wLmUc9cfW57MdnzkdlvI4tw3QqWMFuYsmdBuVjy3P/9r8j5XdV1KmVBlumHkDPef35MipI06HpjzKbZ2l3HjEUJJpQg+DK2teSWqfVJ5o/gSTUyeTODaRj775yOmwlAe5rT+4244YSjpN6GFSplQZRlw/gjX3r6FK2Sq0e7Md97x7D4dPFjQZlFKns7KzlBVt31Wrhva4spf2cnFAelY6w5cP59mVzxJXNo6xbcdy+8W3Ox2WKkGs6i1TrRocOnTm43Fx/guwlPW0l4vLxEbH8sy1z7Cu5zr+VvFv3DHnDu6adxcHjh9wOjQV4XJq5V26WNP2fTjAAWagx5W9NKE7qFGNRqztsZbhrYbz3lfvkTg2kdlbZ+PUUZNXaDe5osndIyWQUNu+3XaStqTThO6wmOgYnmzxJBt6baBulbp0ersTt711G3uP7nU6NFfSbnJFl1+PlLxCTcRuO0lbmIivDAQaE8DuWySN5WKVzKxMeeGLF6TMsDJSeURlmbJxivh8PqfDClo4xhLRYVyLrrDx3Ys6tJBXxpCJlOGU0PHQveXrX76WFpNbCMlIm5ltZNdvu5wOqVDh+rLoRAtFV9D47m5OxFaJlMpAQQldm1xc6MK4C1nWbRmv3vQqK3atoP7Y+ryW8pqr29bDdYGJttkWXaDmkb59/f/fc0/kNUPkbmIJdO4govrMB8r0dt+0hh6c7w9/L62mtRKSkVbTWsmOwzssXb9Vh8vhqjlHymGzU/J+3n37Ru72dHqSb7ugTS7e5vP55PWU16XisxWl3PBy8vKalyXLl1Xs9VqZHMN5OOuVNlsviJRmiPw4MYVgOPZNTegR4sfffpSbZt4kJCPNJzWXr3/5uljrs/LLHEk155L0gxHJ5yQKOglsx2cbru+AJvQI4vP5ZOrGqVJ5RGUpM6yMjFo5SjKzMou0Lqu/zJGQCJ3+YQrnNpw5UyQ6uuTV0O16b+EqTxN6BNr7+1655c1bhGTk0gmXytb9W0NeRyQfbheVk9sknD8mBbUve/XIKq9w/ziH62hHE3qE8vl8MnvLbKk2qprEDo2VYZ8Pk/TM9KBf73Rt1I2cbIKw+8ckd+0/UM08OjqyPv9wHvFoDV1Z4sCxA3LX3LuEZKTx+Maycd/GoF8bCc0kVnKyhm7nj0mwPT4ioe3cKW5oQ9d+6BGgevnqzO4wm3fufIe9R/fSbEIznl76NH9k/lHoa3Vi4dM5eSm7nX3sg7ns36qySipXzAMfKNPbfdMauj0OnTgk9757r5CMJI5JlC93f+l0SJ7j1FGLnTW8wi771+Y270Br6CVH1bJVmXbrND7650f8duo3Lp90OU989gQnM046HZpnOHXUYmcNL1DNOzrawdqkspxOcBHBjpw6wmOfPsbEjRO5MO5CJrefTPNazZ0OSznAqgktlPN0gosS6qwyZzGh/QQ+u+cz/sj8gxZTWvDIJ49wPP2406GpMHNF+66yndbQS4hj6ccYuGggY9aNoW6Vukz8x0SuPf9ap8NSSoVIa+iKCrEVGH3zaJZ1XYbB0Gp6K/p+2Jejfxx1OjSllEU0oZcwLeu0ZHPfzTx6+aO8tv416o+rz8LvFjodVsSI+BlxlKtpQi+BysWU48UbX+SL7l9QLqYcbWa1ofv73fn15K9Oh+ZpOj2ecpom9BLsippXsLH3RgZdNYjpm6aTODaRD77+wOmwPCtck3woFYgm9BKuTKkyPHvds6ztsZZq5arRfnZ7Or/TmUMnDjkdmucEmvkmombEUa6mCV0B0PRvTUnplUJyy2TmbJtDwtgE5qXNczosT9Hp8ZTTgkroxpg2xpivjTHfGWMG5vN8LWPMUmPMRmPMZmPMzdaHqmyR6yxe7AUX8p89/8f6XuupWakmHed2pOPcjuw/tt/pKD3ByXFglIIgEroxJhoYA9wEJACdjDEJeRZ7CpgjIo2Bu4GxVgeqbBDgLF6DRVtY02MNz133HPO/nk/C2ARmbZ6FU9cseIVTF+9ozxqVI5ga+qXAdyLyvYikA7OBW/IsI0Cl7P/PAvZaF2IJEu5vZgFn8UpFlWLgVQNJ7Z3KhXEX0uXdLtwy+xb2/L7H3pg8bNYs/yb98Ud/M8vw4eFJ5tqzRv0p0KhdOTegAzAx1/17gNF5ljkX2ALsBn4FmgZYVy8gBUipVauW/cOSeYkTs00EOQB3ZlamvLjqRSk7rKyc9dxZMmnDJPH5fPbF5UFOTRais06VPIRhtMVOwFQRiQduBmYYY85Yt4i8LiJJIpJUvXp1i4qOEE70eQvyLF50VDSPXvEom/psomGNhtw//37azGrDrt922RebxzjVZVF71tjDq81YwST0PUDNXPfjsx/L7X5gDoCIrAbKANWsCLDEcOKbGeJZvHpx9VjadSljbh7DFz9+Qf1x9Rm3bhw+8dkXo0c4lVi1Z431vNyMFUxCXwfUM8acb4yJxX/Sc36eZX4ErgMwxlyMP6EftDLQiOfEN7MIZ/GiTBT9mvVja7+tXBF/Bf0W9OO66dex4/AO++L0AKcSqxd61nittuvpC8QCtcXkvuFvRvkG2AEMzn5sCNA++/8E4AtgE5AK3FDYOnXGojysboQNw7Q7Pp9PJq6fKJWeqyTlhpeTl1a/JJlZmW4LMyxlOznhtpvnhfXiROROThQeDHSSaI+w6psZ5m/RT0d+kptn3SwkI1dOulK2H9zuxjBtL9vNidUpXjxp6/aYNaG7ndWZwIE90ufzyfTU6VJlRBUpPbS0jFgxQjKyMtwWpivK9rJQd1W313bz4/ajCk3obmbH3uPgt2jf0X1y+1u3C8lI0utJsmX/FjeG6clE47Si7Kpe/eF089GWJnQ3s2OPd/hb5PP5ZM7WOVJ9VHWJGRIjQ5YNkfTMdFeF6dVE46RA2yw6OnDic3tt14s0obuZHVVFl3yLDhw7IJ3mdRKSkYbjGsqGvRtcE6ZLNpGnBNpVC9uGbq7tepEmdDezq6room/Re9vfkxov1JDoZ6Jl8OLBcirjlCvCdNEm8oRAu6oe5YSXJnQ3K6yqGCFZ5/CJw9L13a5CMpIwJkHW7l7rdEgFyr3Z4+L8N49/BMWW366q5yHCTxO62wVK2hHYLrDgmwUS/994iXomSgZ8OkBOpJ9wOqQzFJa4PP4RFEvuXTU6WmvoobKifqYJ3asi9MzdkVNHpPcHvYVkpN4r9WTFrhVOh3SaYJoWPP4RWCIC6xu2smp7FZTQdcYiN4vQkZcqla7E+HbjWXTPIjJ8GVw95Wr6f9yf4+nHnQ4NCG7zevwjsIRT4797VTiGFNCE7mZWDRDi0sE0rqt7HVv6buGhSx9i9JejuWTcJSz5YYnTYQW1eXXwK7/OnWHnTvD5/H81mQcWjvqZJnQ3s2LkJZcPHVchtgIv3/Qyy+9bTqmoUlw3/Tp6f9Cb3//43bGY8tvsublt8CvlDWEZwC1QW4zdN21DD1Jxz6J4qB3+RPoJeWzhYxL1TJTE/zdePv72Y8diKcm9XCKkY5XrhKMNXRN6pPPgNe5rflojCWMShGSk23vd5PCJw06HdBovJrxgY/bqiU6vfCbay0UVj4dq6LlNnZ4ulc4+LJAlUZV/lH+NWud0SCLizYQXSsxe3F28+JkUhyb0ksyDe3u+/cBjjsnl/V+Rg8cPOhqbFxNeKDEXdHm/W2u+XvxMiqOghK4nRSOdB/uW5de9i4zyrJnWnoQxCczZNsdfG3GAF3uShhJzQSfoXHY+/U9u+UwK6kwWto5mgTK93TetoatAAjf7+yTp9SQhGbn9rdtl39F9YY/Ni7XBUGIO5vJ+t71XN3wmBR0IW32QjDa5KC8p6AuakZUhI1eOlNJDS0uVEVVkeup08fl8YYvNgy1YIcecc+IuUEJ32/l0N3wmBe2zVv/gaEJXnhLMF/Srg1/JlZOuFJKRtrPayk9HfgprfF7oUZFbUWJ2Q803WE5/JgV1JrO6o5kmdOU5wXxBM7My5aXVL0nZYWWl0nOVZML6CWGtrUc6N9R8vUJr6JrQvcPp6k8hvjv0nVwz9RohGbl++vXyw68/OB1S0Fy+aV0fn1toG7omdG/wSDUty5cl49aNkwrPVpDyw8vL6LWjJcuX5XRYBfLIplVBKujHz8ofRk3oqui81JAqIjt/3Sk3zLhBSEaunnK1fHvoW6dDCshjm9YzrEqebj06KSihG//z4ZeUlCQpKSmOlK1CEBXlzzN5GeMfYs+FRISpqVP518J/kZ6VzvBWw+l/WX+io6KdDu00Hty0rpczFl3u6xjKlQv90gur1mMHY8x6EUnK7zm9sCiS2HH1QliGiCu+3G/9/PMNsWn3kfZAGtfXvZ5HP32Uq6ZcxfaD250O8zQe2bSeYtWY4+EYu9wWgarudt+0ycVidjXIeqCht6AQfT6fzNo8S6qOrCqxQ2Pl2eXPSkZWhtMhi4gnNq3nWNVF0M1j2qFt6CWAnQ2ybm1MzBbMW//56M/SYU4HIRlp+lpT2fTzJqfCPY3LN63nWPU1cPP5jYISurahR4oS3CAbyluflzaPfh/147dTvzG4xWAGtRhEbHRseAJVttM2dBUZIq1BNoTzAaG89Q4JHUh7II2OiR1J/jyZpNeTWL93vSUhK+dZNRadB8e08wtUdbf7pk0uFoukBtkQ30tR3/r7X70v575wrkQ/Ey2DFg2SkxknbXgzSlkLbUMvISKlQbYIDZhFfeu/nvxV7nvvPiEZuWj0RbLqx1UWvAGl7FNQQtc2dOU+DpwPWPjdQnp+0JPdv+/mX5f/i6GthlIupoCZopVyiLahK29x4HzAjf93I1v7baVPUh/+u+a/NBjXgM93fm5beUrZQRO6cp/hw/1dCnIrV87/uI0qla7E2LZjWXLvEgThmmnX8OCCBzmWfszyssI2g41yhXB93prQlfvkdDGIi/vrsbJlw1b8tedfy+Y+m3n4socZu24s9cfWZ9H3iyxbf06XuF27/C1Lbp3aTVkjnJ+3JnQ30Wrb6U6e/Ov/Q4fCmvXKx5bnpTYvseK+FZQuVZrWM1rTc35Pjpw6Uux1e/ayclUkYf28A50ttfumvVzyiKRuhwUJtjuKiy7VO5F+Qh7/9HGJeiZKznvxPPnw6w+LtT43X1aurBfOGYuCqqEbY9oYY742xnxnjBkYYJk7jTFpxphtxpg3LP3VKQlKQrUtlGNPC6dyL+6BT9mYsoxsPZI196+hcpnKtHuzHfe+ey+HTx4OORaIvGvAVMHC+nkHyvQ5NyAa2AHUBWKBTUBCnmXqARuBKtn3zy5svVpDz6MkVNtCqXVbVEO3+sDnVMYpeXrJ01JqSCk55/lz5J20d0JeR0k5GFN+rpqxCLgCWJjr/iBgUJ5lRgE9CltX7psm9Dxc1MRgm1B+tCz6Fti1WTfu2yiNxzcWkpE7594p+4/tD+n1kXINmAqOa2YsAjoAE3PdvwcYnWeZ97KT+hfAGqBNgHX1AlKAlFq1ahX9HUWiklBtCzW7WvAtsPPAJz0zXYZ9Pkxih8ZKtVHV5M0tb+ok1cp2BSV0q3q5lMpudrkG6ARMMMZUzruQiLwuIkkiklS9enWLio4Qnh0NKASh9i/v3Bl27vRfHbpzZ5G2hZ3tlzHRMQy+ejAbem2gbpW6dHq7E7e9dRv7ju4r/sqVKoJgEvoeoGau+/HZj+W2G5gvIhki8gPwDf4Er0JhQQJzNQd+tMJxjVLi2Yl80f0Lnm/9PAt3LCRhbALTUqflHJWqEGnv3WIIVHWXv5pJSgHfA+fz10nRxDzLtAGmZf9fDfgJiCtovdqGrsIlnO3VX//ytVw1+SohGblp5k3y428/2ldYBCoJLY/FRXGaXEQkE3gQWAhsB+aIyDZjzBBjTPvsxRYCh4wxacBSYICIHLLsV0e5g0erTuE88Lkw7kI+7/Y5r7R5hc93fU7i2EReX/+61tbzkd/uVBJ679pJR1tUwXHzFC4u9f2v39Pzg54s+WEJrc5vxYR/TKBulbpOh+UKgXanvMk8RwmYeCtoBY22qAldBadOHf+FQHnVru2v9qp8iQgTNkzgsU8fI0uyGHHdCB649AGiTMkedSPQ7hQdDVlZZz6uu9lfdPhcVXwWXrlZkhhj6NW0F9v6bePq2lfT/5P+tJzakm8OfeN0aI4KtNtkZTky0GbE0ISugqPXqxdLzbNqsuCfC5h6y1S2HthKw/ENeWHVC2T58qmOlgCBdpucjk+R3HvXTprQVXAcGqM8khhj6NqoK2n90rjxghsZ8NkArpx8JWkH05wOLewK2p0ivfeunTShq+CUhAufwuTciufy7l3v8uYdb7Lj8A4av9aY4cuHk5GV4XRoYaO7kz30pKhSDjpw/AAPffwQc7bNoXGNxky5ZQoNazR0OizlYnpSVCmXOrv82bzV4S3evvNt9h7dS9KEJP6z9D9Mm5HpxS7/ymGlnA5AKQW3X3w7LWu35JGFjzBk9LeYDzOQdP/XM2fYeNAmCVUwraGr8LDjKlMnrly1scy4cnHMuG0G1ddMRNJPn0NVr5ZUwdAaurJf3ssCrahy2rFOl5T5y75y+T6uXf5VYfSkqLKfHVeZOnHlapjKDFRMxbMPs++n0pSPLW9ZWcp79KSocpYdV5k6ceVqmMrMr492qdJ/cPSqB2kwvgHLdi6ztDwVOTShK/vZcZWpE1euVq0aljLz66M9dVJplr3YG4Ph2mnX0u+jfhz946il5Srv04Su7BfossCbby76CcZwX7k6axb8/vuZj8fG2lJmfldLtqzTks19N/Po5Y8yPmU89cfV59Mdn1petvKwQAOl232LuAkudNbfguXdPn37Fn8mg3Bu80DzocbF2VdmAVb9uEouGn2RkIx0f6+7/HryV0fiUOFHARNc6ElRK+hY4aHz2nC8UVH+FJ6XgwN1n8o8xTPLnuH5Vc9zToVzGN92PP/4+z8ciUWFj54UtZtOsxK6QCcS80vybuDC0SbLlCrDc9c/x5oea4grG0f72e3p8k4XDp3QycJKKk3oVtCxwkMXKBEa487r3F082mTS35JI6ZVCcstk3tr2FgljE3g77W2nwwrIozMZeoImdCu4sPbmesOH+5N3XiLuPLJx+fCAsdGx/Oea/7C+13riK8XTYW4HOs7tyIHjB5wO7TQ5rZO7dvk/6pxrszSpWyRQ47rdt4g6KapTlRdNficZwX+SUxVZRlaGPLv8WYkdGitxI+Nk1uZZ4vP5gn69neeaA51brl3bujIiHQWcFNUauhXsqL2VhOPS2rXzfzwqKrLft81KRZViUItBpPZOpV5cPTq/05lb37qVvUf3Fvpau2vQ2jpps0CZ3u5bRNXQrVZSavz5vc+8t0h83/mxqVqcmZUpL656UcoMKyNnPXeWTN4wWWbO9AUsyu4atNbQi48Cauia0N2oJO31uRNZdHTJed+5heEH/JtfvpGrp1wt3N5JomJPBizKGHtbwUpKXcVOmtC9xu5vlRvNnBm4lh7J71skbD/gWb4sqVLjSIFFhSMUr12D57Z4C0ro2obuRiWt10xOw20gkfq+c4SpYTnKRPHb/koFFhWO3pnhmATaqlNQnuuVEyjT233TGnoBStpxaaBqYaS/7xxhbGILVFSVGkckMytTREKvkbqtBmvl18eNrZ9ok4sH5DfWiZu+JXYK1MQEkf2+c4TxBzy/oqJiTwq3d5IrJ10pXx38yq2hB83KJOzG1k9N6OFS1KqKG78V4eTGalC4hbGae2ZRPpmeOl2qjKgipYeWlpErR0pGVkZQ63LjR2dlEnbj+9OEHg7FScpu3GvCqaT/oLnE3t/3yq2zbxWSkWavN5Mt+7cU+horkqfVv2WBvk7R0aGv2427pib0cAg2Kee397rxuC7c3NYQW0L5fD6ZvWW2VBtVTWKGxMjQz4dKemZ6wOWLWxexI2EWdHlDUdbttl1TE3o4BJOUA+29cXElu4buRuH6FrstW2Q7cOyA3DX3LiEZaTS+kWzYuyHf5YJKyAW8R7sOTmfOjNzLGjShh0Mwe2ZBkyS47biuJAvXcbYbj+fzeHf7u1LjhRoS/Uy0DF48WE5lnDpjmQJ/kwp5j3YenEbqga8m9HAI5stZ0B7m0ppaiRSucxoeOXdy6MQh6fpuVyEZSRiTIGt3rw3+xYW8Rzs3gUc2b8g0oYdLYUk5UvewSBOuqp0TVchiVBw++uYjif9vvEQ9EyUDPh0gJ9JPFP6iQt6jnQcpHjgAKhJN6G4RqXtYpLH7hzcnqeZXhp0/8Bbsf7+d/E16zu8pJCMXvnqhrNy1suAXBLEt7Tw4jcQDX03obhKJe1ikCXe1MVw/8Bb+UH224zOp81IdMclG+i/oL8f+OJb/gjNnisTGnl5ebKzu98VQUELXsVzCLRwDWajisXN2ovzmn81h9yxIFo4Zc33d69nSdwsPXvogr3z5CpeMu4QlPyzJf2GRgu8ryxgJYuMaY9oALwPRwEQRGRFguTuAeUAzEUkpaJ1JSUmSklLgIkpFnqio/BOaMf4feTvVqZP/JNy1a/srF0W0YtcKus/vzneHv6N3096Maj2KSqUr2VpmSWaMWS8iSfk9V2gN3RgTDYwBbgISgE7GmIR8lqsIPAysLV64SlnArTM+OTmSpk1DKbao3YJNfTbx/674f0zYMIH6Y+vzyXef+J/UKYrCqlQQy1wKfCci3wMYY2YDtwBpeZYbCowEBhQ1mIyMDHbv3s2pU6eKugoVgjJlyhAfH09MTIzToVgrZ8zTnKaNnDFPwfkmruHDT48NrB+fNpCc9z54sD+h1qrlL9eCbVIuphwv3PACHRI60P397tw06ya6NerGxPjziP5p95kvcNuQyLNm2bJdwi5Q43rODeiAv5kl5/49wOg8yzQB3s7+fxmQFGBdvYAUIKVWrVpnNPZ///33cvDgwZAmtFVF4/P55ODBg/L99987HYr13N49NMJPjJ/KOCVPLnpSop+Jlr6dzpKMMqXd3bMr1JPgDn9+FKeXS2EJHX+zzTKgjhSS0HPf8uvlkpaWpsk8jHw+n6SlpTkdhvUi9RJBj1m/d700GNdAOt2OHKhWTnxu/QELpQLggq7HBSX0YHq57AFq5rofn/1YjopAfWCZMWYncDkw3xiTb6N9YYwxRXmZKoKI3dYlbcYnl2pybhPW9VzHxf2HcF7/DM4ZWY25Hz2P/POfTod2ulDa+fPrpXTihP9xFwgmoa8D6hljzjfGxAJ3A/NznhSRIyJSTUTqiEgdYA3QXgrp5aKUbcIxj5oKSmx0LP9u+W/W91pP7cq1uXPenXSY24Gfj/3sdGh/CaUC4PKTvIUmdBHJBB4EFgLbgTkiss0YM8QY097uAN1s586dvPHGG3/eT01NZcGCBX/enz9/PiNG5NvDM2TdunVj3rx5APTo0YO0tLznpNWf7OxHrorkknMuYfX9qxlx3Qg++uYjEscmMnPzzJxmW2eFUgFw+9FfoLYYu2+B2tC9ZOnSpdK2bds/70+ZMkUeeOABW8rq2rWrzJ071/L1em2bK+/bfnC7XDHxCiEZafdGO9l9ZLfTIQV/otPlbejBdFt0xCOfPELqz6mWrrNRjUa81OalApeZPn06L7zwAsYYGjRowIwZM+jWrRvt2rWjQ4cOAFSoUIFjx44xcOBAtm/fTqNGjejUqRNjxozh5MmTrFy5kkGDBnHy5ElSUlIYPXo03bp1o1KlSqSkpPDzzz8zatQoOnTogM/n48EHH2TJkiXUrFmTmJgYunfv/mdZ+bnmmmt44YUXSEpKokKFCjz88MN8+OGHlC1blvfff59zzjmHgwcP0qdPH37MPhR86aWXaN68uWXbUqmiuqjaRay4bwWvfvkqTy5+koSxCfz3hv/SvXF3587rdO4c3BGcjV0/raCX/ueybds2hg0bxpIlS9i0aRMvv/xygcuPGDGCFi1akJqayhNPPMGQIUO46667SE1N5a677jpj+X379rFy5Uo+/PBDBg4cCMA777zDzp07SUtLY8aMGaxevTqkmI8fP87ll1/Opk2buPrqq5kwYQIADz/8MP/6179Yt24db7/9Nj169AhpvZ7k1ouJ1Bmio6J55PJH2NJ3C41rNKbHBz24ceaN7Potn6tK3cbFw3e4toZeWE3aDkuWLKFjx45Uq1YNgKpVq1q6/ltvvZWoqCgSEhLYv38/ACtXrqRjx45ERUVRo0YNrr322pDWGRsbS7t27QBo2rQpn332GQCLFi06rZ39999/59ixY1SoUMGid+Mybr6YSAV0QdULWNJ1Ca+lvMbjix6n/rj6jLx+JH2S+hBltL4ZKt1iQShVqhS+7HE2fD4f6enpRVpP6dKl//xfLDoZFBMT8+dhanR0NJmZmYA/zjVr1pCamkpqaip79uyJ3GQOru9OpgKLMlH0bdaXrX23ckX8FTyw4AFaTWvFd4e/czo0z9GEnkurVq2YO3cuhw4dAuDw4cMA1KlTh/Xr1wP+nisZGRkAVKxYkaNHj/75+rz3g9G8eXPefvttfD4f+/fvZ9myZRa8E7jhhht49dVX/7yfmppqyXpdy+XdyVThaleuzcIuC5nUfhKpP6fSYFwD/rf6f2T5spwOzTM0oeeSmJjI4MGDadmyJQ0bNuTRRx8FoGfPnnz++ec0bNiQ1atXU758eQAaNGhAdHQ0DRs25H//+x/XXnstaWlpNGrUiLfeeiuoMu+44w7i4+NJSEigS5cuNGnShLPOOqvY7+WVV14hJSWFBg0akJCQwPjx44u9Tldze3cyFRRjDN0bd2dbv220Or8Vj376KC2mtOCrX75yOjRvCNT9xe5bJHRbtMrRo0dFROSXX36RunXryr59+8JWdsRscxd0J1PW8vl8MnPTTKk6sqqUHlpanlvxnGRkZTgdluPQCS7crV27djRq1IgWLVrw73//mxo1ajgdkvfoxUQRxxhD5wadSeuXRrsL2zFo8SAun3g5m/dvdjo01wpqggs75DfBxfbt27n44osdiaek0m2uvGLutrk8sOABfjv1G4NbDGZQi0HERsc6HVbYFWuCC6WUcoOOiR1JeyCNjokdSf48mWYTmrFh3wanw3IVTehKKc+oVq4as26fxft3v8/B4we5dMKlPLn4SU5l6qQ4oAldKeVB7f/enm39tnFvw3t5buVzNHmtCWt2r3E6LMdpQleqpImQIRKqlK3C5Fsm80nnTziWfozmk5vz2KePcSLjROEvjlDeTugRsmMqFTY5QyTs2uXv3JkzRIKHvzs3/t+NbO23lV5NevHi6hdpOL4hy3ctdzosR3g3oTu8Y06bNo169epRr149pk2bFpYylSq2CB0ioVLpSoxrN47F9y4my5dFy6kteXDBgxxLP+Z0aGHl3W6Lder4k3hetWv7R0Cz0eHDh0lKSiIlJQVjDE2bNmX9+vVUqVLF1nLtoN0WS5ioKH8FKC9j/KMHRoDj6cd5cvGTvPrlq9SuXJsJ/5jA9XWvdzosy0Rmt0Ubxu5Yt24dDRo04NSpUxw/fpzExES2bt16xnILFy6kdevWVK1alSpVqtC6dWs++eSTIperVNiUgCESyseW5+WbXmbFfSuIjY6l9YzW9PqgF0dOHXE6NNt5N6HbsGM2a9aM9u3b89RTT/H444/TpUsX6tevf8Zye/bsoWbNv+bNjo+PZ8+ePWcsp5TrlKD5VpvXak5q71QGXDmASRsnUX9cfRZ8u6DwF3qYdxO6TTvm008/zWeffUZKSgqPP/54sdallOuUsCESysaUZVTrUay+fzWVSlei7Rtt6fpeVw6fPOx0aLbwbkK3acc8dOgQx44d4+jRo5w6lf/FCueddx4//fTTn/d3797NeeedV6xylQobF8+4Y5dLz7uUDb028FSLp5i1eRYJYxJ4d/u7TodlOe+eFLVJ+/btufvuu/nhhx/Yt28fo0ePPmOZw4cP07RpUzZs8F923KRJE9avX2/5DEfh4IZtrlQ4bdy3ke7zu5P6cyp3Jt7J6JtGU718dafDClpknhS1wfTp04mJieGf//wnAwcOZN26dSxZsuSM5apWrcq///1vmjVrRrNmzXj66ac9mcyVKokan9uYL3t8ybBrh/Hu9ndJGJvA7K2zLZtFzElaQy/hdJurkmzbgW3c9/59rNu7jlsvupWxN4/l3IrnOh1WgbSGrpRS+Ug8O5FV969i1PWj+Pjbj0kcm8j0TdM9W1vXhF6ALVu20KhRo9Nul112mdNhKaUsVCqqFAOaD2BTn00kVE+g63tdaftGW3468lPhL3aZUk4H4GaXXHJJ5E+urJQC4O/V/s7y+5Yz+svRDFo8iMSxibx4w4v0aNIDY4zT4QVFa+hKKZUtykTR/7L+bOm7haS/JdHrw160ntGaH379wenQgqIJXSml8qhbpS6L7l3E+Lbj+XLPl1wy7hJGfzkan7h7vBtN6EoplY8oE0XvpN5s7beVq2pdxUMfP0TLqS359tC3TocWkKcTug6HrpSyW62zavFx54+ZcssUth7YSoPxDXhh1Qtk+bKcDu0Mnk3oTo/T36ZNGypXrky7du3CU6BSyjHGGLo16sa2ftu44YIbGPDZAK6cfCVpB9OcDu00nk3oTo/TP2DAAGbMmBGewpRSrvC3in/jvbve443b32DH4R00fq0xz654loysDKdDAzyc0G0YDj3o8dABrrvuOipWrFj0wpRSnmSModMlnUh7II1b/n4Lg5cM5rKJl7Hp501Oh+bdhG7HOP3BjoeulFJnlz+bOR3nMK/jPPYc3UPShCT+s/Q/pGelOxaTZxO6XeP063joSqlQ3JFwB2n90uhUvxNDlg+h6etNSdmbUvgLbeDZhG7XOP3BjIeulFK5xZWLY/pt0/mg0wccPnmYyyZexsBFAzmVGd4c4tmEDvaM09+7d2+GDh1K586deeKJJ4q/QqVUidHuwnZs67eN+xrdx8gvRtJofCNW/bQqbOV7OqFbLdjx0AFatGhBx44dWbx4MfHx8SxcuDDM0Sql3KhymcpMbD+RT7t8ysnMk1w1+SqSlyWHpeygxkM3xrQBXgaigYkiMiLP848CPYBM4CDQXUR2FbROHQ/dHXSbK2Wfo38cZeCigVwefzn3NLzHknUWNB56oaMtGmOigTFAa2A3sM4YM19Ecveo3wgkicgJY0xfYBRwV/FDV0op76pYuiJj2o4JW3nBDJ97KfCdiHwPYIyZDdwC/JnQRWRpruXXAF2sDNIpW7Zs4Z57Tv9VLV26NGvXrnUoIqWUCiyYhH4ekHuk991AQbM83A98nN8TxpheQC+AWgE6jIuIa8YejvTx0L06K4tSKn+WnhQ1xnQBkoDn83teRF4XkSQRSape/cxZtsuUKcOhQ4c00YSBiHDo0CHKlCnjdChKKYsEU0PfA9TMdT8++7HTGGOuBwYDLUXkj6IEEx8fz+7duzl48GBRXq5CVKZMGeLj450OQyllkWAS+jqgnjHmfPyJ/G7gn7kXMMY0Bl4D2ojIgaIGExMTw/nnn1/UlyulVIlWaJOLiGQCDwILge3AHBHZZowZYoxpn73Y80AFYK4xJtUYM9+2iJVSSuUrqEmiRWQBsCDPY0/n+v96i+NSSikVIr1SVCmlIkRQV4raUrAxB4ECryYtQDXgFwvDsYrGFRqNK3RujU3jCk1x4qotImd2E8TBhF4cxpiUQJe+OknjCo3GFTq3xqZxhcauuLTJRSmlIoQmdKWUihBeTeivOx1AABpXaDSu0Lk1No0rNLbE5ck2dKWUUmfyag1dKaVUHprQlVIqQrg6oRtj2hhjvjbGfGeMGZjP848aY9KMMZuNMYuNMbVdElcfY8yW7GEQVhpjEtwQV67l7jDGiDEmLN25gthe3YwxB7O3V6oxpocb4spe5s7sfWybMeYNN8RljPlfrm31jTHmN5fEVcsYs9QYszH7O3mzS+KqnZ0fNhtjlhljwjIinTFmsjHmgDFma4DnjTHmley4NxtjmhS7UBFx5Q3/dHc7gLpALLAJSMizzLVAuez/+wJvuSSuSrn+bw984oa4sperCCzHPxFJkhviAroBo124f9XDPxtXlez7Z7shrjzLPwRMdkNc+E/09c3+PwHY6ZK45gJds/9vBcwI0z52NdAE2Brg+Zvxzx1hgMuBtcUt08019D9nShKRdCBnpqQ/ichSETmRfXcN/qF93RDX77nulgfCcea50LiyDQVGAqfCEFMocYVbMHH1BMaIyK8AUoyRRC2OK7dOwJsuiUuAStn/nwXsdUlcCUDObO9L83neFiKyHDhcwCK3ANPFbw1Q2RhzbnHKdHNCz2+mpPMKWD7gTEkWCyouY8wDxpgd+OdX7e+GuLIP6WqKyEdhiCfouLLdkX3YOc8YUzOf552I60LgQmPMF8aYNdmTpbshLsDflACcz1/Jyum4koEuxpjd+Afze8glcW0Cbs/+/zagojEmLgyxFSbUHFcoNyf0oBU2U5ITRGSMiFwAPAE85XQ8xpgo4L/A/3M6lnx8ANQRkQbAZ8A0h+PJUQp/s8s1+GvCE4wxlZ0MKI+7gXkikuV0INk6AVNFJB5/c8KM7P3OaY8BLY0xG4GW+Od1cMs2s5QbNnYgoc6U1F6KOFOSHXHlMhu41c6AshUWV0WgPrDMGLMTf5vd/DCcGC10e4nIoVyf3USgqc0xBRUX/hrTfBHJEJEfgG/wJ3in48pxN+FpboHg4rofmAMgIquBMvgHoXI0LhHZKyK3i0hj/LkCEfnN5riCEWouKVw4Tg4U8YRCKeB7/IeUOSc7EvMs0xj/CZF6LourXq7//wGkuCGuPMsvIzwnRYPZXufm+v82YI1L4moDTMv+vxr+w+M4p+PKXu4iYCfZFwe6ZHt9DHTL/v9i/G3otsYXZFzVgKjs/4cDQ8KxzbLLq0Pgk6JtOf2k6JfFLi9cb6yIG+Nm/LWiHcDg7MeG4K+NAywC9gOp2bf5LonrZWBbdkxLC0qs4Ywrz7JhSehBbq/nsrfXpuztdZFL4jL4m6nSgC3A3W6IK/t+MjAiHPGEsL0SgC+yP8dU4AaXxNUB+DZ7mYlA6TDF9SawD8jAf7R3P9AH6JNr/xqTHfcWK76Peum/UkpFCDe3oSullAqBJnSllIoQmtCVUipCaEJXSqkIoQldKaUihCZ0pZSKEJrQlVIqQvx/vHDJPHFjoEsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 画出更新之后的结果\n",
    "w0 = w[0].data[0]\n",
    "w1 = w[1].data[0]\n",
    "b0 = b.data[0]\n",
    "\n",
    "plot_x = np.arange(0.2, 1, 0.01)\n",
    "plot_y = (-w0.numpy() * plot_x - b0.numpy()) / w1.numpy()\n",
    "\n",
    "plt.plot(plot_x, plot_y, 'g', label='cutting line')\n",
    "plt.plot(plot_x0, plot_y0, 'ro', label='x_0')\n",
    "plt.plot(plot_x1, plot_y1, 'bo', label='x_1')\n",
    "plt.legend(loc='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到更新之后模型已经能够基本将这两类点分开了"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. 5 PyTorch的Loss函数\n",
    "前面我们使用了自己写的 loss，其实 PyTorch 已经为我们写好了一些常见的 loss，比如线性回归里面的 loss 是 `nn.MSE()`，而 Logistic 回归的二分类 loss 在 PyTorch 中是 `nn.BCEWithLogitsLoss()`，关于更多的 loss，可以查看[文档](http://pytorch.org/docs/0.3.0/nn.html#loss-functions)\n",
    "\n",
    "PyTorch 为我们实现的 loss 函数有两个好处，第一是方便我们使用，不需要重复造轮子，第二就是其实现是在底层 C++ 语言上的，所以速度上和稳定性上都要比我们自己实现的要好\n",
    "\n",
    "另外，PyTorch 出于稳定性考虑，将模型的 Sigmoid 操作和最后的 loss 都合在了 `nn.BCEWithLogitsLoss()`，所以我们使用 PyTorch 自带的 loss 就不需要再加上 Sigmoid 操作了"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 使用自带的loss\n",
    "criterion = nn.BCEWithLogitsLoss() # 将 sigmoid 和 loss 写在一层，有更快的速度、更好的稳定性\n",
    "\n",
    "w = nn.Parameter(torch.randn(2, 1))\n",
    "b = nn.Parameter(torch.zeros(1))\n",
    "\n",
    "def logistic_reg(x):\n",
    "    return torch.mm(x, w) + b\n",
    "\n",
    "optimizer = torch.optim.SGD([w, b], 1.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.6314)\n"
     ]
    }
   ],
   "source": [
    "y_pred = logistic_reg(x_data)\n",
    "loss = criterion(y_pred, y_data)\n",
    "print(loss.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 200, Loss: 0.39446, Acc: 0.88000\n",
      "epoch: 400, Loss: 0.32373, Acc: 0.87000\n",
      "epoch: 600, Loss: 0.29020, Acc: 0.87000\n",
      "epoch: 800, Loss: 0.27049, Acc: 0.87000\n",
      "epoch: 1000, Loss: 0.25745, Acc: 0.88000\n",
      "\n",
      "During Time: 0.232 s\n"
     ]
    }
   ],
   "source": [
    "# 同样进行 1000 次更新\n",
    "\n",
    "start = time.time()\n",
    "for e in range(1000):\n",
    "    # 前向传播\n",
    "    y_pred = logistic_reg(x_data)\n",
    "    loss = criterion(y_pred, y_data)\n",
    "    # 反向传播\n",
    "    optimizer.zero_grad()\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    # 计算正确率 0.5以上的判断为正确\n",
    "    mask = y_pred.ge(0.5).float()        \n",
    "    acc = (mask == y_data).sum().item() / y_data.shape[0]\n",
    "    if (e + 1) % 200 == 0:\n",
    "        print('epoch: {}, Loss: {:.5f}, Acc: {:.5f}'.format(e+1, loss.item(), acc))\n",
    "\n",
    "during = time.time() - start\n",
    "print()\n",
    "print('During Time: {:.3f} s'.format(during))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到，使用了 PyTorch 自带的 loss 之后，速度有了一定的上升，虽然看上去速度的提升并不多，但是这只是一个小网络，对于大网络，使用自带的 loss 不管对于稳定性还是速度而言，都有质的飞跃，同时也避免了重复造轮子的困扰"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}