machinelearning_notebook/2_knn/knn_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# kNN 分类算法\n",
    "\n",
    "\n",
    "K最近邻(k-Nearest Neighbor，kNN)分类算法，是一个理论上比较成熟的方法，也是最简单的机器学习算法之一。该方法的思路是：***如果一个样本在特征空间中的k个最相似（即特征空间中最邻近）的样本中的大多数属于某一个类别，则该样本也属于这个类别***。\n",
    "\n",
    "kNN方法虽然从原理上也依赖于[极限定理](https://baike.baidu.com/item/%E6%9E%81%E9%99%90%E5%AE%9A%E7%90%86/13672616)，但在类别决策时，只与极少量的相邻样本有关。由于kNN方法主要靠周围有限的邻近的样本，而不是靠判别类域的方法来确定所属类别的，因此对于类域的交叉或重叠较多的待分样本集来说，kNN方法较其他方法更为适合。\n",
    "\n",
    "kNN算法不仅可以用于分类，还可以用于回归。通过找出一个样本的`k`个最近邻居，将这些邻居的属性的平均值赋给该样本，就可以得到该样本的属性。更有用的方法是将不同距离的邻居对该样本产生的影响给予不同的权值(weight)，如权值与距离成正比（组合函数）。\n",
    "\n",
    "kNN可以说是一种最直接的用来分类未知数据的方法。基本通过下面这张图跟文字说明就可以明白kNN是干什么的\n",
    "![knn](images/knn.png)\n",
    "\n",
    "简单来说，kNN可以看成：**有那么一堆你已经知道分类的数据，然后当一个新数据进入的时候，就开始跟训练数据里的每个点求距离，然后挑选这个训练数据最近的K个点，看看这几个点属于什么类型，然后用少数服从多数的原则，给新数据归类**。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "该算法存在的问题：\n",
    "1. 当样本不平衡时，如一个类的样本数量很大，而其他类样本数量很小时，有可能导致当输入一个新样本时，该样本的K个邻居中大数量类的样本占多数。在这种情况下可能会产生误判的结果。因此我们需要减少数量对运行结果的影响。可以采用权值的方法（和该样本距离小的邻居权值大）来改进。\n",
    "2. 计算量较大，因为对每一个待分类的数据都要计算它到全体已知样本的距离，才能求得它的K个最近邻点。目前常用的解决方法是事先对已知样本点进行剪辑，事先去除对分类作用不大的样本。该算法比较适用于样本容量比较大的类域的自动分类，而那些样本容量较小的类域采用这种算法比较容易产生误分。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 算法步骤：\n",
    "\n",
    "输入：\n",
    "* 训练数据： $T=\\{(x_1,y_1),(x_2,y_2), ..., (x_N,y_N)\\}$, 其中$x_i \\in X=R^n$，$y_i \\in Y = {0, 1, ..., K-1}$，i=1,2...N\n",
    "* 用户输入数据：$x_u$\n",
    "\n",
    "输出：预测的最优类别$y_{pred}$\n",
    "\n",
    "\n",
    "1. 准备数据，对数据进行预处理;\n",
    "2. 计算测试数据与各个训练数据之间的**距离**；\n",
    "3. 按照距离的递增关系进行排序；\n",
    "4. 选取距离最小的`k`个点；\n",
    "5. 确定前`k`个点所在类别的出现频率；\n",
    "6. 返回前`k`个点中出现频率最高的类别作为测试数据的预测分类。\n",
    "\n",
    "\n",
    "\n",
    "**深入思考：**\n",
    "* 上述的处理过程，难点有哪些？\n",
    "* 每个处理步骤如何用程序语言来描述？\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 距离计算\n",
    "\n",
    "要度量空间中点距离的话，有好几种度量方式，比如常见的曼哈顿距离计算、欧式距离计算等等。不过通常 kNN 算法中使用的是欧式距离。这里只是简单说一下，拿二维平面为例，二维空间两个点的欧式距离计算公式如下：\n",
    "$$\n",
    "d = \\sqrt{(x_2 - x_1)^2 + (y_2 - y_1)^2}\n",
    "$$\n",
    "\n",
    "在二维空间其实就是计算 $(x_1,y_1)$ 和 $(x_2, y_2)$ 的距离。拓展到多维空间，则公式变成：\n",
    "$$\n",
    "d(p, q) = \\sqrt{ (p_1-q_1)^2 + (p_1-q_1)^2 + ... + (p_n-q_n)^2 } = \\sqrt{ \\sum_{i=1,n} (p_i-q_i)^2}\n",
    "$$\n",
    "\n",
    "kNN 算法最简单粗暴的就是将 `预测点` 与 `所有点` 距离进行计算，然后保存并排序，选出前面 k 个值看看哪些类别比较多。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## 2. 机器学习的思维模型\n",
    "\n",
    "针对kNN方法从原理、算法、到实现，可以得出机器学习的思维模型，在给定问题的情况下，是如何思考并解决机器学习问题。\n",
    "\n",
    "![machine learning - methodology](images/ml_methodology.png)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上图是机器学习的经典的流程\n",
    "* 问题：我们需要解决的问题是什么？\n",
    "* 核心思想： 通过什么手段解决问题？\n",
    "* 数学理论： 如何构建数学模型，使用什么数学方法？\n",
    "* 算法： 如何将数学理论、处理流程转化成计算机可以实现的算法？\n",
    "* 编程： 如何把算法变成可以计算机执行的程序？\n",
    "* 测试: 如何使用训练、测试数据来验证算法\n",
    "* 深入思考：所采用的方法能够取得什么效果，存在什么问题，如何改进？\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 生成数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAEICAYAAABGaK+TAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAA5cElEQVR4nO3dd5hU5dnH8e89bTvLAgsoXVEUkaJrwUrsBWuMscSumFixxFheu7H32IIi1hgjorGhEkWxIAqIFRtN6W0pW6ec+/3jDEtddnfa2dm9P9fF5c7MmXN+g3DzzHOeIqqKMcaY7OPzOoAxxpjEWAE3xpgsZQXcGGOylBVwY4zJUlbAjTEmS1kBN8aYLGUF3LRYIvKYiFybonPdICLPpeJcxqSKFXDTLInIbBE5IJlzqOqfVfXmVGVqLBF5SkRuyfR1TetjBdxkJREJeJ3BGK9ZATfNjog8C3QHXheRChG5QkR6ioiKyFki8ivwfvzYl0RkoYisFJEJIrLDOuepawmLyBARmSsil4nIYhFZICJnbCZDLxH5UERWi8g4oMMGr2/yuiIyDDgZuCKe/fX481eKyIz4+b4XkWNS+7tmWiMr4KbZUdVTgF+BI1S1UFXvXOflfYHtgYPjj8cC2wAdganA85s5dWegGOgCnAU8LCIl9Rz7L2AKbuG+GThtg9c3eV1VHRH/+c549iPix88A9o5f/0bgORHZYjNZjWmQFXCTbW5Q1UpVrQZQ1SdVdbWq1gI3AANEpLie90aAm1Q1oqpvARVAnw0PEpHuwC7Atapaq6oTgNfXPaaJ10VVX1LV+arqqOqLwM/Ark387Masxwq4yTa/rflBRPwicnu8a2IVMDv+UodNvhOWqWp0ncdVQOEmjtsSKFfVynWem5PEdRGRU0VkmoisEJEVQL/NHW9MY1gBN81Vfctkrvv8ScBRwAG4XRM9489LktdeAJSISME6z3VvwnXXyy4iPYDHgQuA9qraFvg2BTlNK2cF3DRXi4CtGjimCKgFlgH5wK2puLCqzgEmAzeKSEhE9gKOWOeQhq67YfYC3KK+BCB+87RfKrKa1s0KuGmubgP+L97lcHk9xzyD27UxD/ge+CyF1z8J2A1YDlwfv1ZjrzsS6BvP/qqqfg/cA0zELe47Ap+kMKtppcQ2dDDGmOxkLXBjjMlSVsCNMSZLWQE3xpgsZQXcGGOyVEYXBOrQoYP27Nkzk5c0xpisN2XKlKWqWrrh8xkt4D179mTy5MmZvKQxxmQ9EZmzqeetC8UYY7KUFXBjjMlSVsCNMSZLWQE3xpgsZQXcGGOylBVwY4zJUlbAjcliqhGcVX/HWXocTtVLXscxGWY7e5smU1WoHQ9aAbkHIZLrdaRWSytHQtWLQA2s+hkNbIOEBnody2SIFXDTZLr6Fqh+GVCofArav4yIbS7jidgcoMb9WQRi84CBHgYymWRdKKbpat4CrQKthuiPoOVeJ2q1JP8UkAL3l68d5OzjdSSTQdYCN00XHAi1HwExt2jUvxm7STMJ9oXS9yD6GwT7WHdWK2MF3DSZtL0XrRwFziqk4HRE/F5HatXE1w5C7byOYTxgBdw0mUgeUnie1zGMafWsD9wYY7KUFXBjjMlSVsCNaQE0/DnOistxKp9ENeZ1HJMh1gduTJbT6K/o8rOBGqgZh2oEKTzX61gmA6wFbky2i84AWdMWq4bINC/TmAyyFrgx2S60E0guoKAxJO94rxOZDLECbkw9NLYIrXgI8CGFFyD+jfaUbRbEVwwd3oTaiRDYCglu53UkkyFWwI2phy4/FWK/AoJGpiAd3vA6Ur3EVwJ5h3kdw2SYFXBjNkFV4wtFOe4T0Vme5jFmUxq8iSkiT4rIYhH5dhOvXSYiKiId0hPPmPTT2k9xyi/AqXgU1SiAu7pi7qEg+e6vvCOSv05sKRpblPR5jFmjMS3wp4CHgGfWfVJEugEHAb+mPpYxmaHR2Wj5X4BqqJ2A4iCF5wMgxfdA+BPAB6HBSV3HqXwaVt/lXrPgXHxFFyaZ3JhGtMBVdQKwfBMv3QdcAWiqQxmTMdFZIGv+GtRA5Ju6l0R8SM7eSM6eiCQ54rbiPiDs/qp8xCbbmJRI6E+liBwFzFPVrxpx7DARmSwik5csWZLI5YxJn1AZSKG7nja5SP5J6bnOukvuSiE2BcOkQpNvYopIPnA1bvdJg1R1BDACoKyszFrrplkRX5E7BC/8OQR6IoHe6blOyQh05VVADGlzk+1gZFIikVEoWwO9gK/ifwi7AlNFZFdVXZjKcMZkgvjaQO4B6b1GsA/SYUxar2FanyYXcFX9Bui45rGIzAbKVHVpCnMZY4xpQGOGEb4ATAT6iMhcETkr/bGMMcY0pMEWuKqe2MDrPVOWxhhjTKPZrXBjjMlSVsCNMSZLWQE3xpgsZQXcGGOylBVwY4zJUlbAjTEmS1kBN8aYLGUF3BhjspQVcGOMyVJWwI0xJktZATfGmCxlBdwYY7KUFXBjjMlSVsCNMSZLWQE3ppE0PBWt/i/qrEjuPDX/w1l1G1r7WWqCmfVo9Be0chQa/sLrKGmXyJZqxrQ6TtXLsOomQMBXCB3eRnyFTT6P1oxHV1wK1KBVL0D755Bg/5Tnba00+iu67DjQCOCHkgeRnCFex0oba4Eb0xjVo4FqoAq0CqLTEzqN2yqsWfMIwl+nKKABIDwZUCAC1KA173kcKL2sgBvTGKFdgNz4Awf8PRM6jeTuFz9PLogPcganJp9xhfq79RuAXCRnLy/TpJ11oZhWT7UanCrE377eY6TwYtTXDqIzkPwTEX9pQteSUBm0/zdEvoLQbkhgq0Rjb0RVoeZNcBZC7lDE3zll584WEugN7Z9Da8YjwR2R3N95HSmtrICbVk3Dn6Pl54BG0dxDkeK7EJGNjhPxIwWnp+SaEuwLwb4pOde6tOJBqHwSiELFE1D6HuIrSPl1mjsJ7ogEd/Q6RkZYF4pp1XTV7aDVQARq3oXYTK8jJa72Pdx++ghQC7E5Hgcy6dZgAReRJ0VksYh8u85zd4nIDyLytYi8IiJt05rSmHTxlbD2r4GCZHGLNWc/IA/3i3UO+Ht4HMikW2Na4E8Bh2zw3Dign6r2B34CrkpxLmMyQopvgeAA8HWGNjdmdb+xFF6MFN+CFA5HOrya1u4Tjc5Bq15AIzaKxksN9oGr6gQR6bnBc++u8/Az4LgU5zKtkEZnoCuvAQ0jba5HQgPSfk3xb4G0fzHt18kEEYG8I9J+HXes9VGgjvtEyaNIzp5pv67ZWCr6wM8Extb3oogME5HJIjJ5yZIlKbicaam0/M8Q+RKi36LlZ7qjKkzzE/4cVHHHs9egNe829A6TJkkVcBG5BogCz9d3jKqOUNUyVS0rLU1s6JVpJZxl1A3i1SrcP1qmOVANu1PUnUoI7sjawdZ5SGg3L6O1agkPIxSR04GhwP5qTaWsoloDtRPB38kd0tZcFA6H1XcCAvmnIhL0OpEB1KlAlx0LzmIggLT/D9JuFFozDgkNQnIP9jpiq5VQAReRQ4ArgH1VtSq1kUw6qUbctSJi80BjaJtr8eX/wetYAPgKTkVzDwKNIoGuXscxa9S+B7FFuEMUBa18Fl/x9UhoZ6+TtXqNGUb4AjAR6CMic0XkLOAhoAgYJyLTROSxNOc0qRKdAbG5oJVADVQ963Wi9Yi/sxXvemjkG7RyJBr+KrMX9nWEurlNIfBvkdnrm3o1ZhTKiZt4emQasphM8Hdm7d/GHAju4GUa00ga+QZddjLufYEAtHsKCe2UkWtLzmC04HyofgVCO6dsRqpJnk2lb2XE1xbaPY1WPAH+bkjRhV5HSjun6kVYfTf4SpCSR9z1MrJNeBJu8Y4CDoQ/g00UcKf6XagdC6G98eUfm7LL+wqHQeGwlJ3PpIYV8FZIgv2Rkge9jpER6iyHVTcDYYitRFdeibQf7XWspguW4f51dYBQfHXE9Wn4C1h5OVADNe+jvgK7wdjCWQE3LZuGN3hcs+njMkSjv0DtBAjuiGyiCNdHQgOh3ah4y3vXTb83Mh23wANUo5HvrIC3cFbATYsm/s5o/ulQNRIkF2lzw2aPV40h4k9LFo3OQZf9HjRKIrvFSGhn2NzIj5whUHEfEAQcK96tgBVw0+L52lyOFl0EBBDZ9MArjc5Fl58MzkI09zCk+J56j01YeEr8hwgQQWveS+l2XxLoDh3GQngqBPu5j1PEneoRQSSUsnOa5NlysibjVGOoszKjU+VFQpstyFpxHziLAIXa8e508RRQrcYpPxdn0W5o7YT4+iFCOnaLUacCrX4Fjc4AX5vUnTfyE7p4d3RRf5yV19gSB82ItcBNRrkt3T+CUw7BnaHdk81jxqUEWTu8UkFS81dDK5+G2k+AsDs6BI2vfHgtkntgSq5Rd63yYRD5GnDQ2neRDq+n5ryrbwUtdx9UvwH5Jye0IYVGvnXvQQR3Sv23m1bKfhdNRmnlY/E1T6IQ/QZqP/Q6EgBSeBkEtgVyIe849x+XVNAKILbmgfsfZwXilKfm/OuKfAOEcX9vf0I11tA7GimH9ddMb3o3irP6fnTZyWj52eiK4SnKZayAm8ySQmDNTUIFyfcyTR3xl+Lr8F98nb/G1+baTW6rltB5809xZzKu91dNUtbCX0/OECAfJM/dbzNFN2Ol+Drwb+1udlF4bmLj6KueBardRcpq33X3ITVJsy4Uk1FSeD4anQ6RHyDvWAi17F3Zxd8JSsejzhJYcYW7XG7OXpCb+LrdGv0FXXl1fN30G+vWTZe290HNWCAKuYen6BOA+LsgpW8mdxJ/D4jGhzlKCZCbimitnmTyhkRZWZlOnjw5Y9czLZ+Gv0Cr34DgQCTv6JS1nJszZ8kBEPsN9xtMEdJxckKfW2ML0FW3gNYiRX9DgtukPmzdtZagq+8ArUaKLkUCW9d/rGqr+P/YFCIyRVXLNnzeWuAma2n0F3T52UA11LyKEkWaycqKaeUsZ+N105t+I1jLh0H0Z0DR5d9Cx4lpK5ziL0Xa3t3gcU7lC7D676jkIiWPNmmyUzppbCG66mbQSqToimazDLP1gZvsFfmBupEjWg3hLzyNkzGFw4EQkAP5ZyU+iic2D3fmpoKuwB2f7h11qmD1LUAYdBW64gpP86xLy89zl9UNf4ouPzWFN4iTYy1wk71CZSB+IA9UkbyhXifKCHfd9IOBGOLfMvETFZwJFSNABHIP9X6Szoat/+bUjRKbS90yBVrpNhik0NNIYAXcZDHxd4YOb7i7CwX7IEkujauRH9auU5LTvG+uir9T0ufwFV6A5hwEhCHg/bLCInlom+th1S0geUjxnV5HWqvgHKj4ByCQewDi8754g93ENAYAjc5Elx0DWgv4oPBSfIVnex3LNCManeG2vAM7NOlegaqCrgYpTHgCU303Ma0P3GQt1TBO+fk4iwbhlP8F3WDlQY0tRKtfd1cAbEj4y/g0dweIQsU9aO0nacltspMEtkaC/ZpWvJ0KdNmR7lIESw92lzdOISvgJuuoRtHaj9HV90PtR26fZO0nUP3y2mNiC9Glh6Mrr0WXHuuulb05oYGsXYoVIIbWvJ+G9KapNLYIZ+U1OCv/D40t8jpO09S8BtE5QBRi89Gq/6T09NYHblB1IDYTpC3i7+B1nM1SVbT8XIhMAY2wtug666/1Xfsp7vA6d8afVr++2SFpEtgaLb4rviFCFHexqebdD95a6PLTITbb/Tk8BSkd62meJpF81q6x40NSPPPYCngrp+q4BTE8yX2i7QNI7u+8DbU5WgHhibhFFtwvkT4IbAV564wBD24Ha+7vSB4EBzV4al/eYah/S7R2PBIc2Lx/H5KgtZ+ilY9DoDdSdBki7qxIdVaiVf8CQkjBSUBu85hQE5tN3XoysTleJmm63KFuY6L2AwjtBvknpPT0DRZwEXkSGAosVtV+8efaAS8CPYHZwPGqmobVeUzaxWbGi7fbetWK+5p34ZJ8kOL46ng+CA5C2j2JSM76hwX7QskjaM0bENwZyTu6cacPDXR3v2mhNLYQLf8zUAPhyahGkOIb3NeWnwTR2YCglaNAl6L+bki7ZxAvd6LPHQq149yfcw7yLkcCRAJI2/SNpmlMH/hTwCEbPHcl8J6qbgO8F39sspG0XedBAJIZV5wBIn6k/QuQexjk/QEpeXij4l13bM5eSO6RUDserXwIVW8nqjQLsXnxsfMAtRD9EXBvCBP9BXcyTxh0MeBA7Df3XoOHpPgOpO3D7q/i2z3N0tw02AJX1Qki0nODp48ChsR/fhr4APhbKoOZzBB/B2j7gLuhgX9LpPhWryM1SAI9kbb3NnicRn9d29qs/djdLq1oeNrzNWvBHd1/pGPzQWNIwVmAu+GFBgfGZ7c6uIU8vvmEx70oIj7I2dPbEM1Uon3gnVR1QfznhUDyswqMZyT3d8272wRQpxxd8Te3P7TgL/jyj2n4TbE5bmtTAWog8m2aU3pDVd3dhHxt6/qz6yMSgvZj3G3X/FsigR5rX2v3NFS/hhJ0lyWoeRX8PZDCS5LIFnNnMfo6IL6ChM9jNi3pm5iqqiJS72wgERkGDAPo3j11e/SZ1FN14hMO2jSPm1fr0FU3Q/hjIAqrrkNDOze852NwkNtfDm5rM//ktOfMNNUYWn62W3AlBO2ebXBGqkgObGKEjUgu5B/vNrjzjwGS+zamGkaXnegumCUBaPdcs1kECkCdVWjlM4AgBaciviKvIzVZouPAF4nIFgDx/y6u70BVHaGqZapaVlpamuDlTLppbCG6ZAi6eDC6/I+o1nodaX2xxdSNPBG/uyVbA8RXiHR4Aym+F+nwSrP/lpGQyFR3jXHCoBXo6ge8TrRWeCLEZgA1braKf3qdaD26/HSofBQqH0HLz/Q6TkISLeCvAafFfz4N+G9q4hivaOVT4CxhzXZc1IzzOtJ6pGi4OxyQXAgOhGC/xr3PV+h2EW1m/emsJsXxGaTg3oRu52mc9fjar5MtBP6OnsZZl6pC9Hvcvv4IRL7zOlJCGjOM8AXcG5YdRGQucD1wO/AfETkLmAMcn86QJgOkAHers/h422ay1dkaEiqD0o/dlre/W7Pr4vGKBLdFiy6DyichsBVS1HzGEkiwH1p0ubudWmB7pHC415HqiAga2hPC8bWZcnb3NlCCbDErA7hrMeuKi9xdzXMPQ9pcb0UySao14KwAX6emrZ8Rnev+v3AWQ+El+PJ/n76QrZhqOL4F3ZrldBNcVz0DbEces1niy0faPeF1jBZDI9+iy091p/uHdoaSkY3eZFhXXRP/eu/AqhvQnL1SsnysWZ9ICPKO8jpGUmwxK2PSQFc/4E77pxYi09ybjY3lrGLtGi/iLmFqzCZYATcmHfztqPuCq84GM143T9pcHb8HEYC8oe6O7h5TpwJn1S04Ky5z18U2zYJ1oRiTBlJ0FRpbCtEZUHBWo3Z8V1Wo+S8anQPt/oP4OyG+4gykbZiuuBTCnwBRtHYCdPzE+y3YjBVw0/o4FSOg8nHwd3V3Pvd3Tvk1xNcWaTeySe/Ryseg4jGgFqqehtL/pTxXwqLTqdv0WKvdm7PNaFhga2VdKKZV0egvUPEQ6EqITkdX3eJ1pLVqJ+CuXx7v/25OXRX5J7rj8CXfXU/FZ5PymgNrgZvWRatZuzqT4+7m01zkHuaOPlEHCEKgj9eJ6vgKz0NDg92Wd86eNsS0mbACblosjUxHKx4FfylSeIm7k3igH+QeADVvuZvMNqOJL76CU9BAN4j+CrkHIb42abuW29/+OhqdheQdiQR6NfgeCTW8KYbJLCvgpkVSrUaXnxwfyhdCYwuQkkcQEaTtPahzM0hOo8dmZ4rkDIFNL2+eUlr5OFQ8DNSgVU9Dh3GIv336L2xSyvrATcsUWwa6Ztu1cN3GBWuIL7/ZFe+Mqutvj8/Ejv7sZRqTICvgpmXybwnB7eNrvORB/ileJ2pecg+NLw4WAgLuHqIm61gXimlWNDrbXZM8sIO7E0uCRHzQ7ll3v09fuwbXyG5JVB109T3uRrq5+7sbMkS+dFebzNkHkTx8BSejga4QnRPvb2/rdWyTACvgptlwKp+H1XcAAjl7ISUPJ3U+kRDk7J2acNmk5jWofs4dcVM5F40tgJp3QQT8XaH9q+5muzn7ZqS/3aSPdaGY5qPyMaAGqIbaD9DYMq8TNWuqYZxVt+EsOwmneuzaF2KL3EW0AIhC7UdANWgVxH5zt5rLQqqKOsttc+p1WAE3zYe/G+6a5Ljbg/kKPY3T3GnFo1D1AkQmw8q/oZH4jdq8o0DauP3/vrYQ2hW3r1uAAPiyb2VD1Shafjq6eG908V5odKbXkZoF60IxzYa0fcDd+1JXIIWXuXs3NpJGfobIFAjuhAS3TWPKZiQa364M3G3mYvMg2MddGqD0fYj9CoEegKAVj4GzAMk/yx0Pn23CEyHyFRABXYFWPIS0vdfrVJ6zAm6aDfGXIiUPNvl9GpmOLjuBuiFx7f+FNHLLtWwmBaei4Q8BH/jaxVva8dd8+eBbO7JEioZnPmAqSSF1/38JuN8wjBVw0wLUfoy70FIU8Ll9vq2hgIfKoMM7br92sB8ieV5HShsJDUILzoaqf0FgW6ToEq8jNQtWwE32Cw3E/aMcBUIQHOBtngwSf2dIw2qKa2hsoXvz09/L8/VPfIUXQuGFnmZobqyAm2bDqXweql+F0O5I0fBGz5SU0C5Q8gha+yGSsxeSs0d6g7YSTtV/YNXNgEDe4UjxbV5HMhuwAm6aBa2dCKvvBKoh+iPq74wUnNzo97uFe6/0BUwDjfzgrkPi74IUDGt+GyRUPADUuj9X/xcturLZbDBhXEkVcBG5BDgb9+7CN8AZqlqTimCmlYnNXedBDcRme5UkI9RZhS4/Kb7YVg4aW4QU3+x1rPX5OoGzDHDcYZ0tuI89WyU8DlxEugAXAWWq2g93AO8JqQpmWpnc/cFX5I5dlgIk/49eJ0qv2DzWblwc3/g4Q9SpwFl+Bs6i3XBW3e4uLbsJUvIPCA2G4I5IyRPN7xuCSboLJQDkiUgEyAfmJx/JtEbia+eOqIj+BIGeiK/E60jpFdg63sJdBKqQl7l/sLTyUQh/DkTciUA5e0POnhsdJ/4uSLtRGctlmi7hAq6q80TkbuBX3HUp31XVdzc8TkSGAcMAunfvnujlTCsgvgJoJZsGiISg/RgITwBf58xuluCsBmJrgjSvXYlMkyTThVICHAX0ArYECkTkTxsep6ojVLVMVctKS20fPWPWEF8Bkntoxne6kYJh4GsPBCCwA+QMyej1Teok04VyADBLVZcAiMgYYA/guVQEM8akhwS6QukE9waqtPF8fLdJXDKLWf0K7C4i+eL+CdgfmJ6aWMaYdBLxI75iK95ZLpk+8EkiMhqYijsF7ktgRKqCGWMyQ51VaMUDoBVIwXlIoIfXkUwjJTUKRVWvB65PURZjMkKjM9HVdwIhpM1ViH8LryN5SldcAOEpQBStnQClH7fu/UKziM3ENK2KqqLLTwFnKSDo8hlI6Ztex/JW5EfcxcAAZ2W8b9xmXGYD29DBtDKx+OxCBRxw5jb0hpYv73h3lqXkQ2gXW6o1i1gL3LQqIgE07xiofst9Im/tbvUaWwwSbPmTiDYgRZdC7j7gVLh7kdqNzaxhBdy0OtLmVsg7wS3Wwb4AOKvuhKpnANA2N+HLP9bLiBklIm7L22Qd60IxrY6IIKEBdcVbnSqoGgWE3V+r7/Q0X3OnTjnO6odxKp5AtdrrOK2atcCNkaC72p5WAwKtrAulKVQVXXZSfGd7HxqeiLQb6XWsVsta4KbVEwkibUeAf2sI9ENKHvI6UjMWhtgs3KkfYYhM9TpQq2YtcNPqaGw+1LwLgV5Izr4ASM5uSOlYj5M1fyI5aLB/fOghto6Kx6yAm1ZDtRaN/AjlZ4LWAH606Cp8BbaMfVNIu6eh+g2QHMg9zOs4rZoVcNMqaGwhuuxYcFbhTlqJb2JQOxasgDeJSB7k/8HrGAbrAzethFa9DE457kiTNTvQ5EG8CyXh86qDs/penKVH4VQ8XO/uNsakQ6tpgTuOQ/XqavLb5NtEhVZI/O1RgrgbGeRAzhAk92DIPTy5E9f8F6qedkewVM6GQG/IPTgFiY1pWKso4EvnLeOiPa5h+YIVbLdbb+783/WEcoJexzKZlHccRKZD+BPIOQgp+mtK/iHX6ALQcPxBDGILkj6nMY3VKrpQ/nP3ayybX04sGmPGtNl8+urnXkcyGSYSwFd8I77S/+Frc0XKvoVJ/lEg8c2YfUV2U89kVKtogecX5eEP+HBi7i7geYW5HicyLYX4u0Dp++7EFn8vxJfvdSTTirSKFvjxfz2KAfvuQFG7Qg48bQi7HraT15FMCyK+QiS4gxVvk3GtpgV+29v/53UMY4xJqVbRAjfGmJbICrgHYrGYjRc2xiTNCniG/ffhtxmafzJHFp3C5He/8jqOMSaLWQHPoHBNmMcufYpoJEZNVS13n/mw15GMMVksqQIuIm1FZLSI/CAi00VkcKqCtUgiiG/t+GN/0Hb+NsYkLtkW+APA26q6HTAAmJ58pJYrlBPk8lHnU1CcT7vObbn6+eFeRzLGZDFJ9GaaiBQD04CttJEnKSsr08mTJyd0vUR98faXvPKPsfQe1JNTrvsDwZBNoTfGZBcRmaKqZRs+n8w48F7AEmCUiAwApgAXq2rlBhceBgwD6N69exKXa7p5vyzgxuPuprYqzNcffAfAmbeclNEMJvM0thDCkyDQFwlu43UcY9ImmS6UALAT8KiqDgIqgSs3PEhVR6hqmaqWlZaWJnG5pps/YxE+v9vPXFsdZsa0ORm9vhdUlcnvfsWHL00kXBP2Ok7GaWw+uvQwdNX16LLfo+EvvI5kTNok0wKfC8xV1Unxx6PZRAH3Ur89+9CmfSEiEIs6HH3hoV5HSrsnrnyO1x55B4AeO3TjHxNvbV3L59Z+6q4KiLtbula/gYR28TaTMWmScAFX1YUi8puI9FHVH4H9ge9TF61xYrEYb/xzHAtmLuKwsw+g+3Zd6l7LK8zj8a/v4dtPfmTLrTvRpfcWmY7XaF+8/SU/T53F4CPL6NUv8a6mcc98SE1lLQAzv5rNisUrKenUNkUps0Bwe9bbsCE4yMs0xqRVsmuhXAg8LyIhYCZwRvKRmmbE5c/wxohxhGsivD3yfZ6d+TBFJYV1r+cV5rHLwQMzHatJPnxpIned8TCR2ggv3DaGf067my237pzQubbbbRsmvzONWCRGYdtC2rQvSnHa5k2CO0DJo2jNGxDcBck7yutIxqRNUgVcVacBG90ZzYRYLMaNx97NxNfXjmpRR5k/YxF9ygo3887m57M3p1Bb5baaRYQfJv2ccAG/6vmLefGOV1ldXsEfLjsSf6D1jTWXnD2RnD29jmFM2mXNaoQLZi3i1X+MpaRjMccOP5yvPvyeaeO/rXvd5/eRX5xHj75dPUyZmMFDd+aj0Z8RDUdRVbbffduEz5VXkMvpN9kmvca0BllRwMO1ES7c/WpWLVtNMBRg1re/ctg5B9QtCOUL+Oi353ZcP/pycvNzPE7bdPscN5j8Nvn8MnUmg48sY4utOnkdyRiTBbJiLZTyhSuoqahBHSVcE+Hbj3+g/z59Oezs/clvk0efsq257qXLsrq/t+ygAZxw5TH06NvN6yjGmCyRFS3wDl3b0XmrTiyavRiA/U/eGxHhL/edwV/uS/6+qaryyaufs2rpavY9fjAFxQVJn9MYY9It4an0iUhmKn11RTUfv/I5xR3asMshA1M6tvmff32GNx57F1WltGt7nvj2vlZ5888Y0zzVN5U+K7pQwB0OeOAp+7LroYNSPjFlwksTqamspbYqzJLflrF03vKUnr85efn+Nzi65DTO2O4i5v403+s4xpgkZE0BT6cBQ3YglBfC5/e5KwVu0dbrSGmx+LelPHn1v6hcWcW8nxdy77B/eh3JGJOErOgDT7dLRpxL7516sXLxKob++aAWu2JhLBKD+LcXVSVc3frWSjGmJWmRBbxyZSX3/XkEv02fx4lXHcM+fxjMx2MmUbmyin2P34P8orz1jg+Gghx70eEepc2cLbbqxBF/PohXHnyL/KJcLnzoLK8jGWOSkDU3MZvi7rMe4b3nPyIajpKTF2Lv3+/Ox69MQlXZcuvOPPblXfh8qek9chyHGdNmU9SukM49O6bknOkWjUTxB/yta5ErY7JY1t/EbIqFsxYTDUcBd4bmpLem1t2knPvTfFYuWVV37JPX/IvD8k7i1N4XMO+XBU26jqryf0Nv49J9r+OsHS7hf89PSOnnSJdAMGDF25gWoEUW8FOu+wO5+TnkFuTQq38PBu3Xj1BuEH/AR3Fpcd2En9nf/caYB94kUhth4ezFPHzxqCZdZ+GsxXz14ffUVNYSrg7z3E2jNzrGcRyWzltGNBKte6588Upqq2uT+5AptmrZal59aCwfvPgJjuN4HccY0wgtsg98wJAdeG72IyxfuILu23chFnV47ZG3qVheyRHnHVw3xlvXLVSq6z/ejFg0xpK5y8gtzK1ryfoD/o0WoKpaXc1Fe1zDghkLKSwp5B+f3crIq55nwujPCAQD3Db2avrttX1qPnQSopEo5+9yJcsXluPz+/jh81/48z2neR3LGNOAFlnAAYo7tKG4QxsA/H4/x11yxEbH9OzXncPOOYD/PvQ2JZ3bNmpWZ8WKSi7Y7SqWzltGQZt8rnjqfMY88CYdtmzHhQ+fvd6xE0Z/xsJZiwnXRFixeCXP3vQSn7z6OdFwlGg4yj8vf4Z/fHZbaj5wEpb8tozyxSsJ10QA+OSVz62AG5MFWmwBbwwR4bz7zuAv957e6D7hD178lKVzl1FbHSZSG2XmV3O4/6NbNnlsUUnBmlF77vkVnJjbyvf5fc1m7Zb2XdpRUJxPNBwhEAqw80H9vY5kjGmEFlPAv5/4I3ec+g+cmMPlo85nwL478PErk5g2/lv2OHIXdjqg/qK0YfH+Zdos7jvnMSKRKIeeuR9DTtiLko7FABR3KEJ87vGBUIC2nYrrPe/gI8vIyQtRU1lLLBrjnafG06FLOxzHoVP3Ui4ZcW4KPnnyQjlBHv7idt4ZNZ62pW045Mz9vI5kjGmEFjOM8A+dz2bF4pUABHODHHfJUF6+/03C1WFy8kPc/f4NbLdr43Yo/+OW57B84Qr3gUBh2wL+Oe1uOnbrgKry+N+eZcJLnzFwv34Mf2wYgeCm/x2srqjm6JLT61rdAHlFuVz9/HB2H7pzUp/XGNN6tPhhhDWVNXU/R2oivHDbK3UzDdVRfvxixkbvWV1ewVcffEd5vPCvsWp5xdoHCuHqMJ+9PgVwW+vD7jyV52Y9wuUjz6u3eAPkFuTSrc+WdS12cLtQWupUfbNp6lSiaiN7TOq1mAJ+/gNnIv6N+7Fz8nPw+X3sfOD6XShL5y3j9G0v4rqj7+D0bS7k1x/m1b126g3H4w+uXY1QfL56d/qJhCNMfH0yX0/4ng2/zYgI9398C2fcdAJ9dulNt+26cO7dp7Ltzlsn81FNllB1cMovQheXoYsHo9FfvI5kWpgW04UC8OYT47h/2Ii6x1v27swfrziaAUP6brQj/X8ffpsRf32GcE0EEeHEq4/hjJtPrHt9ydxlvDPqfX78YgZDTtiT/U/ae6PrqSqX7HMtM7+ag6ry+0uHcvqN6dvO7H/PfcgDf3mcYE6Q60dfzoAhOzTqfapqE3c8oOGpaPmZoFWAQM5++Eoe9TqWyUL1daEkfRNTRPzAZGCeqg5N9nzJOOT0/Zj42hQmvz2Njj068I+Jf6dN+zabPHbL3p3x+d0vIDl5Ibpuu+V6r5d2bc+frv3DZq+3YvFKfpo8g0itO0ln7OPvpbyA/zx1Jm898R5dt92Cx//2HLFIjJrKWm770wP8e+6Izb53+qSf+b+ht1K1qprTbzmRP/614R3av3z/G8b/+xN23Ht7DvjTPlb4kyG5UNd14gPJ9zSOaXlSMQrlYmA6sOlKmUH+gJ9bXruyUcf2Hbwtx//1SN5+cjyI2xWyOWNHvsdzN4+mc6+OXPPCcNp1LqGoXSH5RXmsClfgD/rpvVOvVHyMOkvnL+fSfa+nprKGnPzQejdDG/PN6d6zH2XVMrc//+nr/s3Bpw+hbWn9o2Z+mTaLa4+8ndqqMO//62NEhAP+tE/yH6SVkmBftHAYVD4N/h5IUeP+bBrTWEkVcBHpChwO/B24NCWJMuDrCd9zzeG3EglHcWIO6iiPXvwU3bfrSr89t9vo+AWzFvHQRU8Srg6zbN5yHjz/CS574i88dtnTdO3ThYK2+XTrsyWnXLf5FntT/fbDPHzxfv3aqjDdtuvCwlmLCYYC/O3pCxt8/4b3BBpqTf/y5ey6Y2qravnu0x+sgCfJV3gBFF7gdQzTQiXbAr8fuAKod0aKiAwDhgF07949oYv8+MUvrFiyikH770goJ/m1ukf93wvUVG6wFonAghmLNlnAK1dW4YuPJInFHFYuWcWdpz3E5He/IhqO4g/6OeDkvSlok9qvyNuWbU1ufo77j4zCSVcfw/4nuwW1MV0bl488j2uPuJ2KlVWcdetJdTNT67PT/v3w+X3kFuSgqgw5fs+UfA5jTHokXMBFZCiwWFWniMiQ+o5T1RHACHBvYjb1OmMeeIMnr/k3Pp/QbfsuPPjp3/H7k9uvsl3ntvgDfmLRGAC5hTkUtMlnt8N32uTxW/XvQdnBA5n05hQCwQBn3/4n7jrj4boVD2ORGHef9Sgg/O4Et+jFojEmjP4MdRz2Pm73hDaJKGiTz4iv7+Hzt76kyzad6Tu4T5Pev+3OW/Pi/McbfXzH7qU8/vU9TBv/Hb0H9WKr/j2aGtkYk0EJj0IRkduAU4AokIvbBz5GVf9U33sSGYVy2jYXMH/GIsC92Tji63s2WjSqKaZP+pkFMxby7jMfsmDmIo69+DC2HtCTrQb03Gijh3WpKuWLVpDfJp/c/BzGjnyPB8573N3lJu7g04dw+ZPnA3DDsXcxZdxXAOy4T19uffPqhDMbY1q3lI9CUdWrgKviJx8CXL654p2o3oN6sWTuciK17jodJZ3bJnyu/9z9Gs/e+B9EhB59uzLqhwcavbGDiNCuc0nd40PP2p/OvUq59og7iMUc/AE/exy9a93rk96cQjRe3Ce/M22zQ/m+/Xg67z79Advu0pvDzzkgJSM/opEo4ZrIZv9RMsZkt2a/FsrlT57H09f/h6Xzl3PSVceSV5Cb8Llef+Sdur7vmd/8ypLfltGpR2nC5xu0X38enXonX4ydRu+detF/n751r/XeaSt++XIWKPTYoWu9RXnuT/O58pC/U1tVy/gXPiEajnL0BYcmnAngm4+mc/XhtxKpiTD0zwdywYO2dZoxLVFKCriqfgB8kIpzbSivMC9lS5tuPagnyxYsJxqOEswJ0LZj8iMfu/XpQrc+XTZ6/va3r+Hl+99EHYdjh9c/PH7WN7/ij49Hr6mq5duPf0i6gD988ZPUVLhLC4x94j3m/riAmV/P5ncn7Mmfm7DyojGmeWv2LfBUuuKpC3jmhhdZvnAlJ151DDl5OU0+x1cffsedpz8EwN+evnC9Vve6CooLOPX64xs8X7+9tycQ8teN/DjwlOSH7eUX5SEiqCrRaIyvPvyOaDjKW0+8x6D9+9tCWsa0EC1qKv2GKldW8thlz7Bo9mJOvvY4BuzbuKnnm3NM+9OpKK8EoKhdIWOWjuKLd6bx0AUjyS/O48pnL6LH9mvXTXnz8XE8f/PLbLFVJ6759/D1+tHXWL6wnCnjvqZXv+70HpT8ZKB5vyzgxt/fTfnCFWw1oAfTxn+HE3PILchh+GPnsv/JGy8LYIxpvuq7idmiC/gtJ97Hp698TiQcJbcgh+dnP9qoTRTW7F+5qZUGD88/qW7nmlBeiFeWjeLY9mdQWx1GBHr07cbj39wLwPwZCzmn/2WEq8P4/T4GH7UL14++PIWfsGGLf1vKhbtfzerlq+mxQzfum3AzuflN/+ZhjPFOi19OdlN+nT6XSHjtZsJ1a3xvxjtPjeeIolM4ougUxj374Uavn/fAGQRCAQKhABc8eCaRcJRofDy5qrtE7RqbmgCUaR27deBfcx7l+TmP8cgXd1jxNqYFadEF/IS/HU1OXoi8wly26t+Dbttt2eB7Hjzv8bo9Kx/4y8aTYA4/50BeWf4UL84fwT7H7U5Bm3x+P3wogVCAYE6Qc9e54br1wJ6UHTSQYE6AvMJczr4j5aMsG8Uf8FPSsdhuXhrTwrSIm5iryyu495zHmPfzAk6+5vfse/weAOx34t702aU35YtWst2uvTc7g9NxHKa8+xWyzrjwYM6mf3umjf+Om467GycW49jhQxl25yn84fIjCOYE15tO7/P5uG70ZaxYvJL8NnkJ3TRNlcqVlVx58C38NGUmOx/YnxtfvSKh2aHGmOajRbTAH774ST57YwqzvvmVu854mIWzF9e91qX3FvTbc7vN7pwDcO85j3HT8feijoM/6Mfn95GTF+Kbj6avd1w0EuWm4+4mUhshFnUYc/+brFiykralxZtcC0VEKOnU1tPiDe76579Mm40Tc/jmo+l88O9PPc1jjElei2iBL56ztG5dEp/fx4rFK+ncsyOqyqxvfiWvMJcttuoEwOzvfuPzt6aybdnWDPxdv7pzjH/h47qbk2ssm1/ONYffyg57bUfvgT059Ybj+X7iT3U3OcFtuQdTsMBWJlgHijEtS9YX8OqKaqpWVwNu8e49qBfb7LQVALed/ACfvjYZdRzOvec0djqgPxfufhWRcJRA0M81L1zC4CPcG7s9dujGzK/ngKPEHAd0zflrmPz2NL758HsQYf+T9iYYCtQV+4G/65fyVQjT4agLDmXi61P4ecoMdtynL0NO2MPrSMaYJGV9AR/z4Fv8On0uAIGQn6HnHoQ/4GfV8tV8NGZSXcv8+VtGk5MXAtzVA2ORGJPenFJXwG8bew3P3TSaaDRKUUkRL9w+pq6IA9RWh5n51RzO+vtJ/OX+M3jp7tfouUM3/jrq/Cbl/fGLX3j5/jfpuk1nTrjq2JQsj9sYBW3y+cfEWzNyLWNMZmR9AY/WRnEct9L6fL61S8QW5BLKCRINR/H5fXTq0ZG+g7cFBZ9PCOYGKTt4YN15iju04fwHzwTgbwffvNF1QnlBjrnoMACGDjuQocMOrHttydxlVFfUuDvQb2akR/nilVy+343UVNYQyguxanmFrVNijElY1hfwYy46jAmjJ/LbD/PYZuet2Pf4wQCEcoLc/u61jPjrMxS2LeDiR8+hQ5f23PfxzUx60+0D32WdAr6uZfOWr9f6DoYCXPefyyg7aMBGx44d+R4PXTgSEWHfP+7BSVcfyzujxtO5Z0cOPvN36418WThrMRK/bRyuDvPDJNul3BiTuBYzEzMSjqRsWNykN6dww+/vIhqO4Qv42Kp/Dx767Db8gY2HIf6xyzCWLygHwB/0k5uXQ1VFNaHcIEedfwjn3HFK3bHhmjDn9L+M8kUrcGLKRY+czUGnDklJZmNMy5W2Xembi1SOad7t8J15cf7j/DxlJuIT+u21/SaLN7i715cvWoE6SiDgJ+a4e2zWVoWZMu5rzrlj7bGh3BCPTb2Tqf/7ho7dO9TdbDXGmES0iHHg6dCmXRE7HziAnfbvv9kbjde9dCk7HdCfPrtszY2vXkFBmzxCeSFyC3LqtldbV15hHnsevasVb2NM0lpMC9wrPr+PJb8tZf6MRXz08mc8OvUuPh4ziY7dO7DroYO8jmeMacGsgCdp5DUvMPenBTgxh/ee/4j9T96HI/58kNexjDGtgHWhJEljztoHIjiOU//BxhiTQlbAk3Tm30+kU89SxCfsfexu9e7QY4wxqWZdKEnq2L2UZ35+aLO7zhtjTDpYCzxFrHgbYzIt4QIuIt1EZLyIfC8i34nIxakMZowxZvOS6UKJApep6lQRKQKmiMg4Vf0+RdmMMcZsRsItcFVdoKpT4z+vBqYDXVIVzCQuEo4wfdLPLJ23zOsoxpg0SslNTBHpCQwCJm3itWHAMIDu3bun4nJpU11ZQzAUaHD3nuYsXBvhwt2vYsGMRTiOww1jrtjkIlzGmOyX9E1MESkEXgaGq+pG266r6ghVLVPVstLS0mQvlzaPDB/FMSWncWz7MzbaRi2b/Pj5LyyYuYjqihpqq8L8+45XvI5kjEmTpAq4iARxi/fzqjomNZEyb9GcJbw5YhyxqEN1RQ0PXTTS60gJ69C1HU7UnUwUzAnSrY/1ahnTUiUzCkWAkcB0Vb03dZEyL5QbZM2quiJCXmGet4GSsEWvTlz9wnD67tGHA0/Zh2F3ndLwm4wxWSnh9cBFZC/gI+AbYM388atV9a363pPO9cCTNXbke4y8+l+UdCzm+jF/pes2W3gdyRhjgPrXA28xGzoYY0xLVV8Bt5mYxhiTpayAG2NMlrICbowxWcoKuDHGZCkr4MYYk6WsgBtjTJayAm6MMVkqo+PARWQJMAfoACzN2IXTI9s/Q7bnh+z/DJbfe9nyGXqo6kaLSWW0gNddVGTypgalZ5Ns/wzZnh+y/zNYfu9l+2ewLhRjjMlSVsCNMSZLeVXAR3h03VTK9s+Q7fkh+z+D5fdeVn8GT/rAjTHGJM+6UIwxJktZATfGmCyV0QIuIt1EZLyIfC8i34nIxZm8fqqIiF9EvhSRN7zOkggRaSsio0XkBxGZLiKDvc7UFCJySfzPz7ci8oKI5HqdqSEi8qSILBaRb9d5rp2IjBORn+P/LfEy4+bUk/+u+J+hr0XkFRFp62HEBm3qM6zz2mUioiLSwYtsicp0CzwKXKaqfYHdgfNFpG+GM6TCxUD27nwMDwBvq+p2wACy6LOISBfgIqBMVfsBfuAEb1M1ylPAIRs8dyXwnqpuA7wXf9xcPcXG+ccB/VS1P/ATcFWmQzXRU2z8GRCRbsBBwK+ZDpSsjBZwVV2gqlPjP6/GLRxZteuuiHQFDgee8DpLIkSkGNgHdz9TVDWsqis8DdV0ASBPRAJAPjDf4zwNUtUJwPINnj4KeDr+89PA0ZnM1BSbyq+q76pqNP7wM6BrxoM1QT3/DwDuA64Asm5Eh2d94CLSExgETPIqQ4Lux/2f7TRwXHPVC1gCjIp3Az0hIgVeh2osVZ0H3I3bWloArFTVd71NlbBOqrog/vNCoJOXYZJ0JjDW6xBNJSJHAfNU9SuvsyTCkwIuIoXAy8BwVV3lRYZEiMhQYLGqTvE6SxICwE7Ao6o6CKikeX91X0+8n/go3H+ItgQKRORP3qZKnrrjebOuBQggItfgdo8+73WWphCRfOBq4DqvsyQq4wVcRIK4xft5VR2T6esnaU/gSBGZDfwb2E9EnvM2UpPNBeaq6ppvPqNxC3q2OACYpapLVDUCjAH28DhTohaJyBYA8f8u9jhPk4nI6cBQ4GTNvkklW+M2BL6K/53uCkwVkc6epmqCTI9CEdy+1+mqem8mr50KqnqVqnZV1Z64N87eV9Wsav2p6kLgNxHpE39qf+B7DyM11a/A7iKSH//ztD9ZdBN2A68Bp8V/Pg34r4dZmkxEDsHtTjxSVau8ztNUqvqNqnZU1Z7xv9NzgZ3if0eyQqZb4HsCp+C2XKfFfx2W4QwGLgSeF5GvgYHArd7Gabz4N4fRwFTgG9w/w81+OrSIvABMBPqIyFwROQu4HThQRH7G/WZxu5cZN6ee/A8BRcC4+N/lxzwN2YB6PkNWs6n0xhiTpWwmpjHGZCkr4MYYk6WsgBtjTJayAm6MMVnKCrgxxmQpK+DGGJOlrIAbY0yW+n9E16iMuDicTgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAEICAYAAABGaK+TAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAl3ElEQVR4nO3deXxcVf3/8ddnZrKW7g3doBRaKIWylbJWoEBBqEhRZBMUBKwI8kPZFwXli4rsKIiAIqAIKCDIKmWRskNbltKFlq0rpemeZk/u5/fHDKWEpJkkd3JnJu/n4zGPzNy599x30+STM+fee665OyIikntiUQcQEZH2UQEXEclRKuAiIjlKBVxEJEepgIuI5CgVcBGRHKUCLtICM/ulmf096hwiLVEBl6xmZp+Y2fgQ2jnJzF4KI1ML7d9pZldkqn2R5qiAi4jkKBVwyVpm9jdgCPComa0zs/NTy/c0s1fMbLWZvWNm4zbY5iQz+8jMKszsYzM73sxGAn8C9kq1s7qF/W1pZi+ktp0M9Gvy/r/MbKmZrTGzKWa2fWr5JOB44PxU+4+mll9oZh+m2ptlZt8K+3skXZy766FH1j6AT4DxG7weDKwAJpDsgByUel0GdAPWAiNS6w4Etk89Pwl4qZV9vQpcBxQB+wIVwN83eP9koHvq/RuAtzd4707giibtHQUMSuU8BqgEBkb9PdUjfx7qgUuuOQF4wt2fcPfA3ScDU0kWdIAAGGVmJe7+qbvPTKdRMxsC7Ab8wt1r3X0K8OiG67j7He5e4e61wC+BncysZ0ttuvu/3H1JKuf9wDxg9zb+e0VapAIuuWYL4KjU8Mnq1HDI10j2bCtJ9nRPAz41s8fNbNs02x0ErEq18bn5nz8xs7iZXZkaEllL8pMBNBlm2ZCZfd/M3t4g56iNrS/SVirgku2aTpe5EPibu/fa4NHN3a8EcPf/uvtBJIdP5gC3t9BOU58Cvc2s2wbLhmzw/LvARGA80BMYmlpuzbVvZluk9v0ToK+79wLe22B9kQ5TAZds9xmw1Qav/w5808y+nuoVF5vZODPbzMz6m9nEVBGuBdaRHFL5vJ3NzKywuZ24+3ySQzG/MrNCM/sa8M0NVumeanMFUAr8ppWc3UgW9XIAM/sByR64SGhUwCXb/Rb4eWoY4lx3X0iyJ3wxyeK4EDiP5M9yDDgbWAKsBPYDfpxq5zlgJrDUzJa3sK/vAnuktr0MuHuD9+4mOaSyGJgFvNZk278A26VyPuzus4BrSR4Y/QzYAXi5Xd8BkRaYu27oICKSi9QDFxHJUSrgIiI5SgVcRCRHqYCLiOSoRGfurF+/fj506NDO3KWISM6bNm3acncva7q8Uwv40KFDmTp1amfuUkQk55nZ/OaWawhFRCRHqYCLiOQoFXARkRylAi4ikqNUwEVEcpQKuIhIjlIBF+nCPFhLsPpcghXH4LUvRR1H2kgFXKQL87WXQs2TUP8Wvup0vLGlmXYlG6mAi3RlDZ8A9cnnZhCogOcSFXCRLsw2OQMoBiuFxPaQ2DrqSNIGnXopvYhkFys+CMr+m+x5J0ZiFo86krRBqz1wM7vDzJaZ2XvNvHeOmbmZ6U7bIjnK4gOxgh0wU38u16QzhHIncEjThWa2OXAwsCDkTCIikoZWC7i7TyF5k9emrgfOJ3nnbRER6WTtOohpZhOBxe7+ThrrTjKzqWY2tby8vD27ExGRZrS5gJtZKXAxcGk667v7be4+xt3HlJV9ZT5yERFpp/b0wIcBWwLvmNknwGbAdDMbEGYwERHZuDYfdnb3GcCmn79OFfEx7q4rAEREOlE6pxHeC7wKjDCzRWZ2SuZjiYhIa1rtgbv7ca28PzS0NCIikjZdSi8ikqNUwKXdvG4avu5PeF2rZ5OKSAbo2llpF697E195CsmZ7P4Ife/BCnaIOpZIl6IeuLSL174G1ACNyUfdmxEnEul6VMClXaxoT6AYiCcfhbtHnEik69EQirSLFe4Gfe6AuqlQuBdWMCrqSCJZyd3BK8G6YWahtq0CLu1mhWOgcEzUMUSylgeV+MrvQsNciA+Bvvdhsd6hta8hFBGRTKn5DzR8DDRC4yK86v5Qm1cBFxHJFCsBPh82iZGcCzA8KuAiIplSfBgUjwfrAUX7QOkxoTavMXARkQwxS2C9rs1Y++qBi4jkKBVwEZEcpQIuIpKjVMCly/PGpQTLv0nw2S4EFTdEHUckbSrg0uX52l9Dw7zk1XKVd+D1s6OOJJIWFXARrwU89cLA66JMI5I2FXDp8qzHBRDrA8Sg+FAo2DHqSCJp0Xng0uVZYhiUvQLUY1YYdRyRtKkHLgKYmYp3Ex5UEqy5iGDFUXjNM1HHkWakc1f6O8xsmZm9t8Gyq81sjpm9a2b/NrNeGU0pIp3OK34H1Y9C/Tv46rPxhoVRR5Im0umB3wkc0mTZZGCUu+8IzAUuCjmXSLt443IVmrA0fAykDuhaHILPIo0jX9VqAXf3KcDKJsuedveG1MvXgM0ykE2kTYKqB/Hycfjygwg+25Og+umoI+U02+Q0oBisFOJb6OBuFgrjIObJQIuT3JrZJGASwJAhQ0LYnUgL1l3H+h6jr4Q15+AFT2CJzSONlausaCyUTU72vBMjMSuIOpI00aGDmGZ2CdAA3NPSOu5+m7uPcfcxZWVlHdmdyMbF+jVdAMGySKLkC4v3xwp2VPHOUu0u4GZ2EnAYcLy7eyuri2Sc9foDJEaQnEC/CBLD9bFf8lq7hlDM7BDgfGA/d68KN5JI+1hiCNbvUbyxPPWxf1vMdKmD5K9Wf7rN7F5gHNDPzBYBl5E866QImJy6y/Jr7n5aBnOKpM3iZRDXcJ3kv1YLuLsf18ziv2Qgi4iItIGuxBQRyVEq4CIiOUoFXEQkR6mAi4jkKBVwyRj3aoKVJxN8tjPBqjNxr486Ut7yYA3BylMJlu1LUPnXqONIJ1EBl8ypug/q3gCvgtopUPNY1InylldcDXWvQrAUKq7H62dFHUk6gQq4ZIx7HV/cqszz5lZlXj8Xr7oPb/gg6ihfCFYCqU84FoNgTaRxpHOogEvGWOmxkBgKGBRsCyWHRx2pw7z+PXzFUfja3+DLj8Tr50QdCQDb5Eyw7kAhJHaEwt2ijiSdQNcZS8ZYrCfW73Hc6/Lnbje1U4BaIAASUPti8o9TxKxgJGz6CgSrINaf1BXSkufUA5eMy4bi7fUzCFb9hGDtr/Ggsv0NFewEfP7vSUDhziGkC4dZERYfoOLdhagHLnnPgwp85ffBK4FCPFiB9bquXW1Z0Vjo/Xu89iWsaD9MQxUSIRVwyX/BZ+BB6kUd1M/uUHNWNA4rGtfhWCIdpSEUyX/xoZDYCqwbUAylJ4bWtDd8QrD6HII1l+HBqtDaFUmHeuCS98wS0Pd+qHsdYv2SB/xC4N6IrzgOfBUQwxs+wPq2eHMqkdCpgEvGeeNiqHsHCnaI7P6UZoVQtE+4jXoV+BqSZ6QE0DAv3PZFWqECLhnlDR/iK44keZszhz7/xAq2iTpWOGwTKNwL6qeBO5QeG3Ui6WJUwCWzap9PXYHZAMSg9hnIkwJuZtD7Vqh7DawUKxwddSTpYlTAJbMSI4ECkgW8KPU6f5gloOhrUceQLkpnoUizgrXXECwdRVA+Hm9Y0O52rGgs9PwtFH8TevwfVrx/iClFujb1wOUrvH4uVN0N1EHjInzt/2F9bm93e7GSCVAyIbyAIgKk0QM3szvMbJmZvbfBsj5mNtnM5qW+9s5sTOlcAcmDjpCcTTDYyLoiEpV0hlDuBA5psuxC4Fl33xp4NvVa8kViBJQeBcQgtinW45KoE4lIM1ot4O4+BVjZZPFE4K7U87uAI8KNJVEyM2I9fo71n01s0xexxFZRRxKRZrT3IGZ/d/809Xwp0L+lFc1skplNNbOp5eXl7dydREGz2olktw6fheLuzhe3XWnu/dvcfYy7jykrK+vo7kREJKW9BfwzMxsIkPq6LLxIIiKSjvYW8P8An0/pdiLwSDhxREQkXemcRngv8CowwswWmdkpwJXAQWY2Dxifei0iIp2o1Qt53P24Ft46MOQsIiLSBrqUXkQkR6mAi+QZ9wCvn4c3rog6imSYCrhIHnFvxFf9AF/xHbx8f7x2Sgb24XiwCvfG0NuWtlEBF8knDe9D3dtANVCDV/w+1OY9qMJXfAtf9jW8/AC8UWcQR0kFXHKKByvxundwr406SnaK9eGLyccSEB8cbvs1T0HDx0A9BMvwqrvDbV/aRNPJSs7w+ln4yuOTL2J9oO8jWGyTaENlGYsPgF7X4ev+APHNsZ6/CncHsW4bvIiD9Qi3fWkTFXDJGV71N/DK5IvAoXaK5hlvhhUfhBUflJnGiw6CkiOg5r9QOBrr9v3M7EfSogIuuSO+OVAM1CRvIhz28EDI3Ovxyjuh8ROs9HisYLuoI3WYWSzZqw+7Zy/togIuOcO6nYoHK5IH6UqPxgp3ijrSRnnFNVB1L1CD1zwBZc9isT5Rx5I8ogIuOcOsEOvxi6hjpK9uGlDzxeuGBVCoAi7h0VkoIplSehRQApSC9YSCEVEnkjyjHrhIiDxYC1aAWQmx0mPwxNbQuASK9sWsJOp4kmfUAxcJSVBxLb5sL/yz3QmqJwNghaOxksOwmE63k/CpgIuEwIO1UHkHUA/UQsUVoe8jqPg9wdIdCMoPxBs+Cb19yT0q4CJhsAK+9OsU6x5q897wMVT+GaiFxkX42l+G2r7kJhVwkRCYlWC9boDYQEhsk3weqqYTR2kiKdFBTJHQWPGBWHFm7nNiieF46XFQdRfE+mDdc+h0SskYFXCRHBHrcRHe/QLM9MFZkvSTIJJDVLxlQ/ppEBHJUR0q4Gb2MzObaWbvmdm9ZlYcVjAREdm4dhdwMxsM/D9gjLuPAuLAsWEFExGRjevoEEoCKDGzBFAKLOl4JBERSUe7C7i7LwauARYAnwJr3P3ppuuZ2SQzm2pmU8vLy9ufVEREvqQjQyi9gYnAlsAgoJuZndB0PXe/zd3HuPuYsrKy9icVEZEv6cgQynjgY3cvd/d64CFg73BiiYhIazpSwBcAe5pZqZkZcCAwO5xYIvnH3aOOIHmmI2PgrwMPANOBGam2bgspl4TAg1V4wyIVjoi5O8Gai/HPRhKUj8cbF0cdSfJEh85CcffL3H1bdx/l7t9z99qwgknHBNVP48v2xZcfiq+5MOo4XVv9NKh5AgiSMwlWXBt1IskTuhIzX627CqhNPmoexxuXRp2oC2v6a2aRpJD8owKer2JlfPHfa2DdokzTtRXsAsWHAwUQH4p1PzfqRJInVMDzlPW6Fgp2hfgwrPdNWMg3GOiqvPZFvPJvbfpEY2bEel5ObMBMYmVPYfGBGUwoXYmmk81TFh+E9b0n6hh5Jaj8B1T8Dghg3R+gbDIW6xl1LOnC1AMXSVfNE0A1yWMLDVCf3lmz7tV47Ut4w0eZTCddkAq45J2g8i6Cz3ZJnrLX8GF4DRftC5SQPAjpkNi61U3ca/HlR+Crz0x+rZkcXh7p8jSEInnFG5dCxdVAHTRW4WsuwfreF0rb1u2HEO+PN3yElRyOxfu2vlH9DAiWgVcm81X+FSs+KJQ8Iirgkmca+eI0PQevC61lM4OSiW07CTC+GXiQelEEBSNDyyOiIRTJKxYfDKXfB+JgPbCev4w4zwCs961QOA5Kj8e6nx9pHskv6oFL3on1OA/v/lMgkew1R8yK9sSK9ow6huQhFXDJS2YFUUcQyTgNoYiI5CgVcBGRHKUCLiKSo1TARURylAq4iEiOUgEXEclRKuAiIjlKBTxkH8+Yz8cz5kcdQ0S6AF3IE6I/nXMXj936NACHn34Ik676XsSJRCSfdagHbma9zOwBM5tjZrPNbK+wguWih//wBLVVddRW1fHvG5+IOo6I5LmODqHcCDzl7tsCOwHpzXCfp/oM7I2ZYTGj3+A+UccRkTzX7iEUM+sJ7AucBODudUB4c3fmoKsmX8off3YnZsbpN5wUdRwRyXPm7u3b0Gxn4DZgFsne9zTgLPfUzPVfrDcJmAQwZMiQXefP1wE+EZG2MLNp7j6m6fKODKEkgNHALe6+C1AJXNh0JXe/zd3HuPuYsrKyDuxOREQ21JECvghY5O6vp14/QLKg5621Kyo4bfR5fL3gGH515DU0NjRGHUlEurB2F3B3XwosNLMRqUUHkhxOyVv/vPoRPpm5kKAxYOrT7/DKI29GHUlEurCOngd+JnCPmRUCHwE/6HgkERFJR4dOI3T3t1Pj2zu6+xHuviqsYNno6PMmMnT7zYknYow5eCf2nrhbWtstW1DO+1M/pLFRQy4iEh5didkGPfp250/Tr27TNi/861WuOukmYvEYI8YM46pnLiUW0wwGItJxqiRpCIKg3dvec8UD1FXXUbOuhjlvfMDC95eEmExEujIV8I1YMGcxxwyexCGFx/L7M26nPefMDxo+gERBPPXK6VXWI9yQItJlqYBvxC1n38mqpavwwHn6rhf48J1P2tzG2befxj7f2Yvtx47g8ocvoGc/FXARCYfGwDcikYhjZqmetxNPxFvd5nMzXpzNW8/NYKdx23PxPWdlLqSIdFnqgW/E6Tf+gEHDB1BUUsiRPz2MLUcNSWu7ma+8z0WHXMHfLv8Xl0z4DTNe7PgcX4/c/BSH9/wexw/9MR+9q+kIREQ98I0auGV//jrn923ebsaLs2mobwSH+tp63p0yix32GdnuHGuWr+XWc++mvrae6ooarj75Zm6ZelW72xOR/KAeeAbscsAoEgVxEoVxCooLGD1+xw61FzQGwBcHUJcvWsms1+Z2MKWI5DoV8DSsWb6WudM+pK62Pq31R+w2nOumXM7Jvz6ea/93OSP32LpD++/dvxfHXfRtYvHkf9fqZWs4f/yveH/qhx1qV0Rym4ZQWjHnjXmcN/5yzKDvoD78cervKOlW3Op22+w6jG12HRZaju9dehT1dQ3c+5uHgGSvfOZLcxgxJrx9NOXeAA1zIbYpFu+Xsf2ISPuoB96KB69/nJp1NVRX1LBi8UreemZGWtstfH8xz97zIssWlIeWZczBO1FUUki8IE4sHmPHcduF1nZT7vX4imPwld/Fyw/Ea1/N2L5EpH3UA2/FoOH9KSwppK66jiAI2HRI6z3ROW/M49wDfkUsZpgZt0y/ikHDBqS9z8aGRm468y+8/fx77HfM3pz4y2MwM3bcdzuueuZS3ntpDrscuAPDd96yI/+0jat/Fxo/BK8CwNfdghV16VueimSdvCng9XX1zJgym979e7LlDluE1u7xlxzJ2uUVzJ36IYefcQjDd2m9aL700OvUVtUCUFhcwJtPvc3EMw5Je5+P3fo0k+9+gdrqOh687jG22XUYex+enDhru71GsN1eI1ppIQSxMvDPpxAohER6p1CKSOfJiwLe2NjIz/b5BQtmLyYIAk6/4QdMOHV8KG0XFhdy1i2T2rTNiN2GU1RaRG1VLRaztIr+hlYsWUV96oCpu7Nq6eo2bR8GSwzBe14FlbdCYhjW/Ss3WxKRiOVFAV88bymfzFy0vtf74PWPhVbA22OfI/ektrqO6c++y75H7sX2e7etxzzhh+N5/PZnqKuuo1dZT/Y96qtDF7XVtSQKEm26OrStYiWHQEn6nxxEpHPlRQHvO7AXsZgBUFCUCHUIpb3Gn7Av40/Yt13bDhi6Kfd8cgvLFixn4FabUlBY8KX3/3zh33ngukcpLC7kiscuYsd9M3cwU0SyV16chdKtZzeueuZS9jp8DBNOHc/Zt58WdaQOKy4tYvMRg75SvJcvWclDNz5BY0NA9boafn/67RElFJGo5UUPHGDb3bfm8ocviDpGKBobG7nimOt5+eE3GLLtYK5+9jJ69+8FQEHhF/9lZlCcxjnpIpKf8qIHnglBEHD9j27lqAGncPnR16Z9FWYYXn98OtOefgcPnEVzl3Dvbx9a/17Pfj048+ZT6NG3O4O3GcR5d57RablEJLt0yQK+unwNs16bS03qoGdzpvzrVZ77x4usXraW1x+bxqO3/DcjWebPWsjz973Mik+/uJ1oLBbDU3OfOGBNbsF26MkH8mD5Hfx19o1sMXKzjOQSkezX4SEUM4sDU4HF7n5YxyOFq3pdNTNfmcugYf0ZNGwAH7z1MWfvdylmxiZ9unHb29fQrWe3r2xXtbZ6/R14GhsCKtdUhZ7tnRdmcsk3fkssZsQL4tz+7rX0G9yX3Sfswk7jRvH6Y9MIGgIq11Ti7phZ6BlEJHeF0QM/C+j4hNcZUFVRzQ93PIf/O/paJu10DtMmv8MjNz9J9boaqiqqqVixjjeefLvZbccdO5ZBwwaQKEzQd3BvvnnawaHne/aeF6mtqqV6XQ0N9Y289dx7QLIHbrC+YL/wz1eZN/2j0PcvIrmtQz1wM9sM+Abwa+DsUBKF6L2X5lCxYh1VFdUAPPyHJ9l+7LYUlRRSW11HEDj9h5Y1u21p9xJuffsa1q6oYJPe3YjHwz/feuQeW/P8P16ipqoWD5wtd/jiasdYPAYGOLizfiZCEZHPdXQI5QbgfKB7SyuY2SRgEsCQIZ17OfagYf1pbGwEoKikkGE7D+U7Zx/G6vI1zHz5fSaceiDb7blNi9ubWdr3sHzvpdm88MBrjBq7Lfs1c+FNcw45+QAaU7MKHnD8Pl+a2+S0605k/qxFLFuwnMNPPziz856ISE6y9txpHcDMDgMmuPvpZjYOOLe1MfAxY8b41KlT27W/9nr9iek8fNOTbD16S75/2dEkCsI/c/LjGfM5c6+Lqa2qo6i0iLNvP40Djvta6PvZmOp11axZXkH/Lco0Vi6SZ8xsmruPabq8I9VsLHC4mU0AioEeZvZ3dz+hA22Gbo8Jo9ljwujQ2w2CgFjq7JC50z5aXzRrq2qZMWVWpxbw2a/P44KDLqexMWC7vbbhyqd+ntFL7EUkO7R7YNXdL3L3zdx9KHAs8Fy2Fe9MWDTvU7475DQOKTyW6390K+7OTuO2JxaLUVRSSFFJIfscuWenZrr7svupXldDXXUd77/xAbNfn9ep+xeRaOTNlZid5bbz7mb54pW4O8/940UmnHogI3Ybzp/euprpz7zL8NFbpXWXnI/enc+LD77GVjtuwde+vUeHhj16D+xFoiBOQ30jQWNAj74tHpIQkTwSSgF39/8B/wujrWwXT8QxS54ZsuHZIQO36s83Jh2UVhtLP1nGWWMvoaaqluKSItatruTQUw5sd6bTrj2RNeUVLJyzmGMumMiQbQe3uy0RyR3qgbfRadcmzw75bH45h59+MFuP3qrNbcyb/nFy/NyhpqqWN596u0MFvEef7vz6sYvavb2I5KasL+BBEPC/+15m9bK1HHjCPmmf1pcp/bco445ZN7T4vrtz/1UP8+qj09j78DEcfd7ErwyPjNxjOFhy6ttYPN7pY+Yikh+yvoDfdu7dPH77MzQ2NPLgDY9x59zff2WK1Wwy5YHXuOeKB6mprOWjdz5h4Fb92fc7Xz4vvN/gvtwy7Spee3QaW+44hF0O2AFIThX70kOvM2jYAHY/dJco4otIDsn6Av7Kf6ZSU5mcdGrtigqWLVjO4OEDI07Vsk8/XLr+dmgNdQ18+tGyZtcbNGwA3/7pN9a/Xre6kh/tfC4162qIxWOc8tvjOeInh3ZKZhHJTVl/ffZuh+5MUWkR8YI43XqWUrZ563eFj9L+x32N0u4llPYooaR7CeOO2Tut7T6esYCGugbqauqpqazlhX++kuGkIpLrsr4HfvoNP2DrXbZi9bI1HHzSOAqLsnf4BJJj5Hd9cBPzZy1i6Pabs0mvr8502JwhIwdjGGZGYUkhY76+c2aDikjOa/el9O0RxaX0YVi2cDk3/vh2KtdU8aNrvs/IPbbOyH4WzV3CM3+fwuDhAxn/vX11SbyIAC1fSq8CnoYzdr+QD6Z/RBA4pT1KeLD8jozMqSIi0pyWCnjWj4Fng/KFywmC5B+62uq69QdVRUSipAKehuN/fiSFxQUUlRax/7Fj2aRXN9586i1+ts8vuPrkm9fPNy4i0pk0DpCGiWccyu4TRlOzroaho4awfMlKfvWda6itquP9qR/ggXP+nT+JOqaIdDEq4GkauGX/9c9XLFm1firZ+toGFr6/JJR9VK6t4p3/zWTQsAEM3X7zUNoUkfylAt4Ow3ceyhbbb878mQsJGgO+e/G3O9xmdWUNP9r5XCpWVNDYEHDRPWcx9ojdQ0grIvlKBbwd4ok410+5nHnTP6bvwF5sOqT5+2q2xftvfJC6f2cNAI/c/JQKuIhslAp4G1SuqWT+7MUM2XYwm/Tq1uL54B+89TGv/OdNRuw2PO27ASXv3xkAyft3brNr22c5FJGuRQU8TeWLVnDa6PNoqGsgnojzx6m/Y8DQTb+y3qK5S/jZvr+gpqqWopJCzrvjDPY7uvXL6TcdUsavH7+IR/7wJEN3GBLKsIyI5DcV8DT97/5XqFpTRUN9I/FEjOfvfZnjLvrWV9ab/fo8knd8gNqqOqb+9+20CjjATvttz077bb/RdRrqG3j5328Qi8fY+4jdiMd170uRrkoFPE0DttyURGGChvpGCooK6D+0+XHvUWO3BXcShQniiRh7Tdwt1ByXfesq3n1hFgB7fGM0P7/v7FDbF5HcoQKepq99a3cWzzuSl/79Bnsdtiv7Hzu22fUGbtWfm964kjeeeIttdt2KncZtvEfdFkEQ8OaTb/P59AevPPJmaG2LSO5RAU+TmXHsBd/i2Au+OmzS1BYjN2OLkZuFniEWi7HFdpuxcO4SzIxhO28Z+j5EJHe0u4Cb2ebA3UB/wIHb3P3GsIJJ8655/pf865r/EEvEOfrcw6OOIyIR6kgPvAE4x92nm1l3YJqZTXb3WSFlk2b07NeDU688IeoYIpIF2j2Zlbt/6u7TU88rgNnA4LCCSTTee3kOP9zxbM7Y/QI+mbkw6jgishGhzAduZkOBKcAod1/b5L1JwCSAIUOG7Dp//vwO708yIwgCvtX7pPWzKw4c1p+7590UcSoRydh84Ga2CfAg8NOmxRvA3W9z9zHuPqasrOOXnOezmqpaHrn5KR65+Slqqjp/zvGgMfjSftcur+j0DCKSvg6dhWJmBSSL9z3u/lA4kbquSyb8hjlvzAPgxQde5Zrnf9Wp+08UJDj2wiP41zX/AYxTrzy+U/cvIm3T7iEUS96w8S5gpbv/NJ1tcvWWap3l6wXHEKTmQ4knYjxVd38kOZYvWUmiIE6vsp6R7F9EviwTQyhjge8BB5jZ26nHhA601+WNGrsthcUFFJYUJq/ojEi/QX1UvEVyQLuHUNz9JUC3TQ/Rr5+4mKfueA6AQ04+IOI0IpLtdCVmFikuLeKInxwadQwRyRG6qbGISI5SARcRyVEq4CIiOUoFXEQkR6mAi4jkKBVwEZEcpQIuIpKjVMBFRHKUCniWc3cq11YRBEHUUUQky6iAZ7H6unrO2f+XHNnvBxw/9McsW7g86kgikkVUwLPYa49OY970j2hsCFj56Wru/93DUUcSkSyiAp7FirsVrX8ei8co6V4cYRoRyTYq4FlszNd35usnjWOT3t3YYZ+RHHfRt6OOJCJZJJR7YqZLN3QQEWm7jN0TU0REoqECLiKSo1TARURylAq4iEiOUgEXEclRKuAiIjlKBVxEJEd16nngZlYOzG/DJv2AXJkARFkzJ5fyKmtmdPWsW7h7WdOFnVrA28rMpjZ38no2UtbMyaW8ypoZyto8DaGIiOQoFXARkRyV7QX8tqgDtIGyZk4u5VXWzFDWZmT1GLiIiLQs23vgIiLSAhVwEZEclZUF3Mw2N7PnzWyWmc00s7OiztQaM4ub2Vtm9ljUWTbGzHqZ2QNmNsfMZpvZXlFnaomZ/Sz1//+emd1rZllzSyIzu8PMlpnZexss62Nmk81sXupr7ygzfq6FrFenfgbeNbN/m1mvCCN+SXN5N3jvHDNzM+sXRbamWspqZmemvr8zzeyqTO0/Kws40ACc4+7bAXsCZ5jZdhFnas1ZwOyoQ6ThRuApd98W2IkszWxmg4H/B4xx91FAHDg22lRfcidwSJNlFwLPuvvWwLOp19ngTr6adTIwyt13BOYCF3V2qI24k6/mxcw2Bw4GFnR2oI24kyZZzWx/YCKwk7tvD1yTqZ1nZQF390/dfXrqeQXJIjM42lQtM7PNgG8Af446y8aYWU9gX+AvAO5e5+6rIw21cQmgxMwSQCmwJOI867n7FGBlk8UTgbtSz+8CjujMTC1pLqu7P+3uDamXrwGbdXqwFrTwvQW4HjgfyJozL1rI+mPgSnevTa2zLFP7z8oCviEzGwrsArwecZSNuYHkD1YQcY7WbAmUA39NDff82cy6RR2qOe6+mGTPZQHwKbDG3Z+ONlWr+rv7p6nnS4H+UYZpg5OBJ6MOsTFmNhFY7O7vRJ0lDdsA+5jZ62b2gpntlqkdZXUBN7NNgAeBn7r72qjzNMfMDgOWufu0qLOkIQGMBm5x912ASrLnY/6XpMaPJ5L8ozMI6GZmJ0SbKn2ePD83a3qKLTGzS0gOWd4TdZaWmFkpcDFwadRZ0pQA+pAc/j0P+KeZWSZ2lLUF3MwKSBbve9z9oajzbMRY4HAz+wS4DzjAzP4ebaQWLQIWufvnn2YeIFnQs9F44GN3L3f3euAhYO+IM7XmMzMbCJD6mrGPzmEws5OAw4DjPbsvCBlG8g/5O6nfs82A6WY2INJULVsEPORJb5D8ZJ6Rg65ZWcBTf63+Asx29+uizrMx7n6Ru2/m7kNJHmR7zt2zsqfo7kuBhWY2IrXoQGBWhJE2ZgGwp5mVpn4eDiRLD7hu4D/AiannJwKPRJhlo8zsEJLDfoe7e1XUeTbG3We4+6buPjT1e7YIGJ36ec5GDwP7A5jZNkAhGZpJMSsLOMle7fdI9mbfTj0mRB0qT5wJ3GNm7wI7A7+JNk7zUp8SHgCmAzNI/qxmzeXUZnYv8CowwswWmdkpwJXAQWY2j+QniCujzPi5FrLeBHQHJqd+v/4UacgNtJA3K7WQ9Q5gq9SphfcBJ2bqE44upRcRyVHZ2gMXEZFWqICLiOQoFXARkRylAi4ikqNUwEVEcpQKuIhIjlIBFxHJUf8felrU3Cit+SQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# 生成模拟数据\n",
    "np.random.seed(314)\n",
    "\n",
    "data_size1 = 100\n",
    "x1 = np.random.randn(data_size1, 2) + np.array([4,4])\n",
    "y1 = [0 for _ in range(data_size1)]\n",
    "\n",
    "data_size2 = 100\n",
    "x2 = np.random.randn(data_size2, 2)*2 + np.array([10,10])\n",
    "y2 = [1 for _ in range(data_size2)]\n",
    "\n",
    "\n",
    "# 合并生成全部数据\n",
    "x = np.concatenate((x1, x2), axis=0)\n",
    "y = np.concatenate((y1, y2), axis=0)\n",
    "\n",
    "data_size_all = data_size1 + data_size2\n",
    "shuffled_index = np.random.permutation(data_size_all)\n",
    "x = x[shuffled_index]\n",
    "y = y[shuffled_index]\n",
    "\n",
    "# 分割训练与测试数据\n",
    "split_index = int(data_size_all*0.7)\n",
    "x_train = x[:split_index]\n",
    "y_train = y[:split_index]\n",
    "x_test = x[split_index:]\n",
    "y_test = y[split_index:]\n",
    "\n",
    "\n",
    "# 绘制结果\n",
    "plt.scatter(x_train[:,0], x_train[:,1], c=y_train, marker='.')\n",
    "plt.title(\"train data\")\n",
    "plt.savefig(\"knn_train_data.pdf\")\n",
    "plt.show()\n",
    "plt.scatter(x_test[:,0], x_test[:,1], c=y_test, marker='.')\n",
    "plt.title(\"test data\")\n",
    "plt.savefig(\"knn_test_data.pdf\")\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 最简单的程序实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 0, 0, 0, 1, 0, 1, 0, 0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 0, 0, 1, 0, 1, 1, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 1, 0, 1, 0, 0, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 0, 1, 1, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 0]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import operator\n",
    "\n",
    "def knn_distance(v1, v2):\n",
    "    \"\"\"计算两个多维向量的距离\"\"\"\n",
    "    return np.sum(np.square(v1-v2))\n",
    "\n",
    "def knn_vote(ys):\n",
    "    \"\"\"根据ys的类别，挑选类别最多一类作为输出\"\"\"\n",
    "    vote_dict = {}\n",
    "        for y in ys:\n",
    "            if y not in vote_dict.keys():\n",
    "                vote_dict[y] = 1\n",
    "            else:\n",
    "                vote_dict[y] += 1\n",
    "                \n",
    "    method = 1\n",
    "    \n",
    "    # 方法1 - 使用排序的方法\n",
    "    if method == 1:\n",
    "        sorted_vote_dict = sorted(vote_dict.items(), \\\n",
    "                                  #key=operator.itemgetter(1), \\\n",
    "                                  key=lambda x:x[1], \\\n",
    "                                  reverse=True)\n",
    "        return sorted_vote_dict[0][0]\n",
    "    \n",
    "    # 方法2 - 使用循环遍历找到类别最多的一类\n",
    "    if method == 2:\n",
    "        maxv = maxk = 0                     \n",
    "        for y in np.unique(ys):\n",
    "            if maxv < vote_dict[y]:\n",
    "                maxv = vote_dict[y]\n",
    "                maxk = y\n",
    "        return maxk\n",
    "    \n",
    "def knn_predict(x, train_x, train_y, k=3):\n",
    "    \"\"\"\n",
    "    针对给定的数据进行分类\n",
    "    参数\n",
    "    x - 输入的待分类样本\n",
    "    train_x - 训练数据的样本\n",
    "    train_y - 训练数据的标签\n",
    "    k - 最近邻的样本个数\n",
    "    \"\"\"\n",
    "    dist_arr = [knn_distance(x, train_x[j]) for j in range(len(train_x))]\n",
    "    sorted_index = np.argsort(dist_arr)\n",
    "    top_k_index = sorted_index[:k]\n",
    "    ys=train_y[top_k_index]\n",
    "    return knn_vote(ys)\n",
    "    \n",
    "\n",
    "# 对每个样本进行分类\n",
    "y_train_est = [knn_predict(x_train[i], x_train, y_train) for i in range(len(x_train))]\n",
    "print(y_train_est)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Accuracy: 100.000000%\n"
     ]
    }
   ],
   "source": [
    "# 计算训练数据的精度\n",
    "n_correct = 0\n",
    "for i in range(len(x_train)):\n",
    "    if y_train_est[i] == y_train[i]:\n",
    "        n_correct += 1\n",
    "accuracy = n_correct / len(x_train) * 100.0\n",
    "print(\"Train Accuracy: %f%%\" % accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Accuracy: 96.666667%\n",
      "58 60\n"
     ]
    }
   ],
   "source": [
    "# 计算测试数据的精度\n",
    "y_test_est = [knn_predict(x_test[i], x_train, y_train, 3) for i in range(len(x_test))]\n",
    "n_correct = 0\n",
    "for i in range(len(x_test)):\n",
    "    if y_test_est[i] == y_test[i]:\n",
    "        n_correct += 1\n",
    "accuracy = n_correct / len(x_test) * 100.0\n",
    "print(\"Test Accuracy: %f%%\" % accuracy)\n",
    "print(n_correct, len(x_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. 通过类实现kNN程序"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import operator\n",
    "\n",
    "class KNN(object):\n",
    "    def __init__(self, k=3):\n",
    "        \"\"\"对象构造函数，参数为：\n",
    "        k - 近邻个数\"\"\"\n",
    "        self.k = k\n",
    "\n",
    "    def fit(self, x, y):\n",
    "        \"\"\"拟合给定的数据，参数为：\n",
    "        x - 样本的特征；y - 样本的标签\"\"\"\n",
    "        self.x = x\n",
    "        self.y = y\n",
    "        return self\n",
    "\n",
    "    def _square_distance(self, v1, v2):\n",
    "        \"\"\"计算两个样本点的特征空间距离，参数为：\n",
    "        v1 - 样本点1；v2 - 样本点2\"\"\"\n",
    "        return np.sum(np.square(v1-v2))\n",
    "\n",
    "    def _vote(self, ys):\n",
    "        \"\"\"投票算法，参数为：\n",
    "        ys - k个近邻样本的类别\"\"\"\n",
    "        ys_unique = np.unique(ys)\n",
    "        vote_dict = {}\n",
    "        for y in ys:\n",
    "            if y not in vote_dict.keys():\n",
    "                vote_dict[y] = 1\n",
    "            else:\n",
    "                vote_dict[y] += 1\n",
    "        sorted_vote_dict = sorted(vote_dict.items(), key=operator.itemgetter(1), reverse=True)\n",
    "        return sorted_vote_dict[0][0]\n",
    "\n",
    "    def predict(self, x):\n",
    "        \n",
    "        y_pred = []\n",
    "        for i in range(len(x)):\n",
    "            dist_arr = [self._square_distance(x[i], self.x[j]) for j in range(len(self.x))]\n",
    "            sorted_index = np.argsort(dist_arr)\n",
    "            top_k_index = sorted_index[:self.k]\n",
    "            y_pred.append(self._vote(ys=self.y[top_k_index]))\n",
    "        return np.array(y_pred)\n",
    "\n",
    "    def score(self, y_true=None, y_pred=None):\n",
    "        if y_true is None and y_pred is None:\n",
    "            y_pred = self.predict(self.x)\n",
    "            y_true = self.y\n",
    "        score = 0.0\n",
    "        for i in range(len(y_true)):\n",
    "            if y_true[i] == y_pred[i]:\n",
    "                score += 1\n",
    "        score /= len(y_true)\n",
    "        return score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train accuracy: 100.000000 %\n",
      "test accuracy: 96.666667 %\n"
     ]
    }
   ],
   "source": [
    "# data preprocessing\n",
    "#x_train = (x_train - np.min(x_train, axis=0)) / (np.max(x_train, axis=0) - np.min(x_train, axis=0))\n",
    "#x_test = (x_test - np.min(x_test, axis=0)) / (np.max(x_test, axis=0) - np.min(x_test, axis=0))\n",
    "\n",
    "# knn classifier\n",
    "clf = KNN(k=3)\n",
    "train_acc = clf.fit(x_train, y_train).score() * 100.0\n",
    "\n",
    "y_test_pred = clf.predict(x_test)\n",
    "test_acc = clf.score(y_test, y_test_pred) * 100.0\n",
    "\n",
    "print('train accuracy: %f %%' % train_acc)\n",
    "print('test accuracy: %f %%' % test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. sklearn program"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature dimensions:  (1797, 64)\n",
      "Label dimensions:    (1797,)\n"
     ]
    }
   ],
   "source": [
    "#% matplotlib inline\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn import datasets, neighbors, linear_model\n",
    "\n",
    "# load data\n",
    "digits = datasets.load_digits()\n",
    "X_digits = digits.data\n",
    "y_digits = digits.target\n",
    "\n",
    "print(\"Feature dimensions: \", X_digits.shape)\n",
    "print(\"Label dimensions:   \", y_digits.shape)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAABLCAYAAABZX83EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAUfElEQVR4nO2deZwV1ZXHv6c3GrqhBRob2QTEFiFRVELUBHEZI8aZgJpPNJqYMSoJDE6Mmo0xH0liJBMT0bgQMUjc4pL5BJ24O1FQFJeOEAhKE1lkX5q19+2d+aNev6r7pJvmvdevKnK+n8/79L3v1rv161u3blWdOudeUVUMwzCM6JITtgDDMAyjY2ygNgzDiDg2UBuGYUQcG6gNwzAijg3UhmEYEccGasMwjIhjA7VhGEbEicRALSJ9RGSBiNSKyEciclkIGqaLSIWINIrI77O9/4CObiIyL94O1SKyTETOD0nLIyKyVUT2i8hqEbk6DB0BPceKSIOIPBLS/hfG918T/1SGoSOu5VIR+SB+zqwRkfFZ3n9N0qdVRO7KpoaAlqEi8pyI7BGRbSJyt4jkhaDjeBF5RUT2iciHInJhpuqOxEAN3AM0AWXA5cAcERmdZQ1bgFuAB7K832TygI3ABKAEuAl4UkSGhqBlFjBUVXsBXwJuEZFTQtDRxj3AuyHuH2C6qhbHP8eFIUBEzgX+G7gS6AmcAazNpoZAGxQD/YF64I/Z1BDgXmAHcBQwBu/cmZZNAfELw9PAM0AfYArwiIiUZ6L+0AdqESkCLgZ+rKo1qroY+F/g69nUoap/UtWngF3Z3O8BdNSq6kxVXa+qMVV9BlgHZH2AVNWVqtrYlo1/jsm2DvDuIIG9wF/C2H/E+AnwU1V9K95HNqvq5hD1XIw3UL4e0v6HAU+qaoOqbgNeALJ9ozcSGADMVtVWVX0FeIMMjWOhD9RAOdCiqqsD3/2N7Dd0JBGRMrw2WhnS/u8VkTpgFbAVeC4EDb2AnwLXZ3vfB2CWiFSJyBsicma2dy4iucBYoF/88XpT/FG/e7a1BPgG8JCGNx/FHcClItJDRAYC5+MN1mEjwKcyUVEUBupiYH/Sd/vwHukOa0QkH3gUeFBVV4WhQVWn4R2L8cCfgMaOf9El/AyYp6qbQth3kB8Aw4GBwFzgzyKS7SeMMiAf+DLeMRkDnIRnIss6InI0nqnhwTD2H+c1vBu7/cAmoAJ4KssaKvGeKr4nIvki8gW8dumRicqjMFDXAL2SvusFVIegJTKISA7wMJ7tfnqYWuKPcouBQcDUbO5bRMYA/wLMzuZ+D4Sqvq2q1araqKoP4j3afjHLMurjf+9S1a2qWgXcHoKONr4OLFbVdWHsPH6evIB3E1EElAK98Wz4WUNVm4HJwAXANuAG4Em8C0faRGGgXg3kicixge9OJKRH/SggIgLMw7t7ujjeCaJAHtm3UZ8JDAU2iMg24EbgYhF5L8s6DoTiPd5mb4eqe/BO/qCZIcwpMK8g3LvpPsAQ4O74BXQXMJ8QLlyqulxVJ6hqX1U9D+/p651M1B36QK2qtXhXw5+KSJGIfA6YhHc3mTVEJE9ECoFcIFdECsNw8YkzBzge+DdVrT/Yxl2BiBwZdwErFpFcETkP+CrZf5k3F+/iMCb++S3wLHBeNkWIyBEicl5bvxCRy/G8LcKwhc4Hro0fo97Ad/G8DbKKiJyOZwYKy9uD+BPFOmBq/LgcgWczX55tLSJyQrx/9BCRG/G8UH6fkcpVNfQP3lXxKaAW2ABcFoKGmfieDW2fmSHoODq+7wY8s1Db5/Is6+gHLMLztNgPrACuiUBfmQk8EsJ+++G5BlbH2+Qt4NyQ2iAfzyVtL95j9m+AwhB03Ac8HIE+MQZYCOwBqvBMDmUh6LgtrqEGeB4Ykam6Jb4DwzAMI6KEbvowDMMwOsYGasMwjIhjA7VhGEbE6dRALSITRaQyHgn1w64WZTpMh+kwHZ9UHalw0JeJ8ZDV1cC5eP6b7wJfVdX32/tNgXTTQooOWNZS6n7fv//uRHpz7RFOWeEm331YValp2U0PihFyqKOaQorIJZcGamnSxo/5s3ak42PbjvSvWd1yWpyyvdv9IElVpWH31i7TETvC327o4O1O2bZmPy5IVdlTuTdjOpoGut9/qu/ORHp3LNcp21Xpb9vVx0XyfA/J2HD3vkJWN/k6UGrZnzEdwf4AUNtckEjnr2loV2+mdXSkK7mfVr/vl2VaR9MA93sNdInSnm5s2lF5fvuoKisqmxg6LI+8PKhcqXTP7UWu5FHfWk1TrP6QdDQOdQP9Bhf748fGfX2dssKtfhCtqlLTmrl+quUFTj54LJpWxQ74m4PRng7wAhgOxjjgQ1VdCyAij+P5Obc7UBdSxGflnAOWVV18mpP/3g2PJ9I//uskp6z8+q2J9J6mbfxj12JOjs/muC4eUT1MRvK2Hti1tyMdyQx40B+Mj+2xwyl76vazE+maHevZ+eyTXaaj7uzPJtLz7rjdKZu1dWIivXPFDt6+uiJjOtZd6x6Xd74xJ5F+vLq3U/bwhHGJdFcfl9zSIxPp+nvd6SwKzv0okd6ru1jL+xnTEewPAO9sHpJID7q4/VisTOvoSFdyP110gt8+mdax4VunO/mmEn8wuuqcV52yGaX+7K9LKur5wW07+d0j3iD6pVFeOw4vPoUlVQd2v+5Ix+qbxzr5X473x48bnvmaU3bcL/wJBfc2beMfu9/IWHs03Xu0kx/a079gbDk1taDq9nRA50wfA/Gm3WxjU/w7BxGZIt58zhXNXTAdRGNrDYX4HbGQ7jTy8ViQrtbRXLcvEjrqdtRFQkdUjksj9aYjgjo2b2ul/wD/9rswt5iGWG3WdTTEaiPRHqmSsZeJqjpXVceq6th8umWqWtNhOkyH6TjsdCTTGdPHZmBwID8o/l1KBE0dAJf23JNI33FEjVP27HsvJtJLKuo5f2oRVRd4j+gN962iG+6jcDqsr+6TSM8f4k6re/8Z/uIZjQOKaH7VvxI3UJ+WjtiEk5z86/fcl0ivTprhY1LfpYl05YhaVpCejtVzfBPGrLPd4/KpO/151//+nXudsrvGD02km6ug4ZVlaenoiHVTRyTSTX93bX8j8E0f3ehOQ5rtESTY1pDUJ7a42z5VW5xIV75Xy6++kjkde/7dNUm9OMQ3SR3zxLedshG8lUhnuj2SKdjn3+M9f/OZTtnL00Ym0vvrt7B93SJmbfUi/uv2vUsB0Fq/A1XXxt4ZzhzV/qI6v/5Xd+Gfp0/zz62cFXlsvDq99sgd7a8T8eroJ9rfMKl/3Frlri8RNFF1ls7cUb8LHCsiw0SkALgUb2L/rPKZMYU07quicf8uYq0tbGcj/Tgq2zIoGDaIemqo11piGgtNx4gTekRCR3GfwZHQ0YvekdARleMSlfboeVx/qjfup3pLNa3NraHp6Ht8aSTaI1UOeketqi0iMh14EW/CogdUNesz2+XlCYM+fxFrn5uLqjKYQRRLSbZlILm5HMcYlvI6ijKAoaHoyM2TSOiQnGi0R47kcJyGryMqxyUq7SG5OYy78VT+7z9fQmNKGUPCaY+8nEgcl1Tp1Oxwqvocaazs0XK2v4rUpT2XOWXnT7w0kS5Z7s6N/5XF7hvXlknHUDbp+wAMm5re7IHJJof7yu8O5FyXnF4rXFecUjmK0gxdjddOdu1gwcekeX85yylbc8lvnfwcGZGWjpFz/PUaHv7JOKfspkWPJdLJXh/Ff3zbzWewPXLLjnTyX7/IfxP+xHy3PwQfRQHKOI4yzgSgdWV6686+X+++L59c5Ne3utl9GfZfyy938kf330kZJ3o6trueGYfK5Otfabds+FMdv+zKZD8dMvPNdss+nH2qk7+qzD2PF/+ynNPxlg5slfTaY+H77jF/p6R9b5y7PnInNrzqousZw2QAeixw+3BnaC5tfw2AKzf45tGghxDAz0942skvYgSHikUmGoZhRBwbqA3DMCKODdSGYRgRJysrmDT09Xdz045PO2WxJLt0kHdXZHbVpw0z/eiqp6+8zSkrz28/hHfgS7ucfGsGNQWjpwCe2ODbYZ+/ztV41srLnHxBwD0tFZy2P2GkUxZ0m/zKWtc2nNff7TYt29xQ93QIuuMB3FGyIJFeNNt1a/rgATdKLWefr2vEd9PT8fJ2tz2C0XbJfSW2wn0p1bo9c+/aR3V3PWGD7zByFi1N3jyj1F3oR8luOaP9Fceev+jXHdbzxGV+/+k/Oz0b9YgH3bPv5cceTaSvfGu8U/Z+U5mT77l6byKdyjmcv6p9r+Ttk/y+Oe7pDU7ZqILk88Ns1IZhGJ84bKA2DMOIONkxffT2rwePLnEjrco7WKQ3r6TJybfsK2hny84RdDG6bs6FTtlzS19q93fJbjnpXt2CLmiVPxzulF11TvsTs3T/mjs3QSZNMMkmqAtO9teOPemFpFCrpOVcl04ckEinYgYJRt99MMWNghy9ZEoiPShpYfp1E3/n5E+8bRqZIjjhE8D4C7+VSFed6M4mmKz5eHwdHbm1dYbkx+and/lupRtmumbEYX9MMtGl6aIYNBUMmebOGHhf+R/a/d1V113v5PsvSK8NgjT0aX8MSI4o/uK5lzj5dNsj6GqZHG0YHD+GvXC1U/ajo9wTJuhW2llNdkdtGIYRcWygNgzDiDidMn2IyHqgGu9pu0VVx3b8i65h84xZ5BR2gxxhjzZ0eg7fTLNYnyOXPARByDnsdSzc8RB5ko8goLHQdFT+7mfk5HdDcnL4SOtD07F29s/IKfB0bA1RR1T6h+lIn0OxUZ+lqlWp7KRwjz/r2Wc+vcYp2xcU0991p7lk1F+d/G0xGHjVVHKLihn2oyWpSEmJHScnzXa1EE5hAgWS2jSIH8zyQ0zXTfxtu9uNm3Gjk++9/eP/czo6OiJoaw7aoAF2PeBOqt9cspC+M64lt7iI8hRC+7vt8/tHcnj2ytN896tbl7t2wWRya1o4rXQyBTnd0w5VTiYYclzKZzvYEjRX6XfTNeT2LKL8mxVp7fd/9p3s5IN22Fsvcv/HGVNce2fR0AJOOmUaBQVFKbnyBe2nBee6ZeVbfBfFcTOmOmW9F2S2nwanewjOLgnuDIKFQ9wJ+y9/zG37RSfnM+6Yb1KQ1yNte3XyDHivTrgykS5f5O73vAe+4+SH3uGvnpTcru1hpg/DMIyI09k7agVeEhEF7lPVuckbiMgUYApAIe1PXpIWImyZNxdEyNcjGSTDD7BJFnQAS3kdFAYy3HQI7LjzfhChh5aFpkMEKnb/GUEYoINDbA9hx6/mhd4eiLBs+XwEYaD2P+z7qQAV6/8Q7x8DQm2PQ6WzA/XnVXWziBwJvCwiq1T1teAG8cF7LkAv6dPxirkpMvDb08krKaGlpppNt8ymSHvSW/o522RDx1jOolC606QNvMfrh72Oshunkde7hNb9NWz6/p2h6RjX50IKc4tpbK2jYueC8Npjxrf99vjOXaHpOOWka+jWrYSmphqWvXnvYd9Pxw2/gsL8XjS21FKxan5oOlKhs9Ocbo7/3SEiC/AWvH2t41/59Kr0LdE3D3rGKbtiiu9zmT95Jx1x7C/89XRzGMB+dtObfh38omsoFM8+VSCF9NND1xEMg711rGt3DYYqv3PrHKfsrMvdxX/rHx2QWMOj3/w1abVHcLUXgAGv+CHDQT94gIdGuYvuTt47FWgir6SAfikcl6D999oFn3PKgvbJex662ykL+lgDDKpaSSt15EFKOoIkr6wStKOP+EG76zoDMHRx22pBJWxNU8fDf3JfeAXt0Mlh7l8uec/Jb76kzV+/G/3eTE/H6qRw/dXNbyTSpc+7752S/fvTPV+CodvJ7zCCUzA0j3Snpp3xmGuHnje1bdrg3vS7LrPjR/AdQHJbvXjOnU4+6Gfe2WkgDmqjFpEiEenZlga+APy9U7VnkKa6FlrUW5uqVVvYzXaKyP7E37HmxkjoaK1vprXZC0JobW4MTUddXYxYvTcvcqyhKbzj0tAUieNSWxfzj0tLeO1RVxcj1uDpiDWG1z9aNRrnbW1E2iNVOnNHXQYsEJG27f+gqi90/JPMU7u7kQreBgVF6c9gSqV/tmXQUltDBQtD19G0p5bKZ707TNUYAzkqFB27q2JsmXm/p6M1xpCQdLTsrY3Ecdm+s5UVr3mRihqLMSCk9ti1M8bW39zjZWLhHZdGGljOkkgclyi0R6p0ZimutRBfriJFguHJl8y5wSm76QZ/JZE71riPee+OCYbq9uJU6aQvSydIXnnjrJW+WeHV0e6KDC2f9003OeRx6vz0dAQfkzpy82m5abdbFtQ1Gobd7rv9DEvTDSx/rxsWfe0tj7ezJUx+03XHOmPjMj/T/iRrKZFfVZdIJ89a1+eR4kCumOEZ7B87z3BXFk4OVw8yeom7wsvp+wOh7mm2x7A5H7r5IX54cvIj9bdWu7Mrji/3F2zO2Z7eTHvXjHXDs792s+86eiC30TZ6SDGnkt5xCZ6ryf/jq0v9cyLZLJI82+TZMX8KhnTdN5PNG8FFdyf0cNvqP66Y7uR7LDr01WXMPc8wDCPi2EBtGIYRcWygNgzDiDiimnlXQRHZCdQCKYWcJ1HaiXqOVtWP+dmYjkjr+KiTdZgO0/FJ0NEZLQfUAYCqdskHqIhCPaYjmjqsDqvjcKoj3XrM9GEYhhFxbKA2DMOIOF05UH9s4qaQ6jEdmf19JuuxOqyOw6WOtOrpkpeJhmEYRuYw04dhGEbEsYHaMAwj4nTJQC0iE0WkUkQ+FJEfplHPehFZISLLROSQJ7MwHabDdJiOf3YdQOb9qIFcYA0wHCgA/gaMSrGu9UCp6TAdpsN0HI462j5dcUc9DvhQVdeqahPwODDpIL/pCkyH6TAdpuOfXQfQNaaPgcDGQH5T/LtUaFur8a/xtcxMh+kwHabjcNIBdH7NxLA46FqNpsN0mA7T8UnX0RV31JuBwYH8oPh3h4wG1moE2tZqNB2mw3SYjsNFR6KSjH7w7tLXAsPwjfCjU6inCOgZSL8JTDQdpsN0mI7DRUfbJ+OmD1VtEZHpwIt4b04fUNWVB/nZgUhrrUbTYTpMh+n4Z9fRhoWQG4ZhRByLTDQMw4g4NlAbhmFEHBuoDcMwIo4N1IZhGBHHBmrDMIyIYwO1YRhGxLGB2jAMI+L8P6Rg4NXcREyMAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 10 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot sample images\n",
    "nplot = 10\n",
    "fig, axes = plt.subplots(nrows=1, ncols=nplot)\n",
    "\n",
    "for i in range(nplot):\n",
    "    img = X_digits[i].reshape(8, 8)\n",
    "    axes[i].imshow(img)\n",
    "    axes[i].set_title(y_digits[i])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# split train / test data\n",
    "n_samples = len(X_digits)\n",
    "n_train = int(0.4 * n_samples)\n",
    "\n",
    "X_train = X_digits[:n_train]\n",
    "y_train = y_digits[:n_train]\n",
    "X_test = X_digits[n_train:]\n",
    "y_test = y_digits[n_train:]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "KNN score: 0.953661\n",
      "LogisticRegression score: 0.927711\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/bushuhui/anaconda3/envs/dl/lib/python3.7/site-packages/sklearn/linear_model/_logistic.py:765: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n"
     ]
    }
   ],
   "source": [
    "# do KNN classification\n",
    "knn = neighbors.KNeighborsClassifier()\n",
    "logistic = linear_model.LogisticRegression()\n",
    "\n",
    "print('KNN score: %f' % knn.fit(X_train, y_train).score(X_test, y_test))\n",
    "print('LogisticRegression score: %f' % logistic.fit(X_train, y_train).score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. 深入思考\n",
    "\n",
    "* 如果输入的数据非常多，怎么快速进行距离计算？\n",
    "    - [kd-tree](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KDTree.html#sklearn.neighbors.KDTree) \n",
    "    - Fast Library for Approximate Nearest Neighbors （FLANN）\n",
    "    - [PyNNDescent for fast Approximate Nearest Neighbors](https://pynndescent.readthedocs.io/en/latest/)\n",
    "* 如何选择最好的`k`？\n",
    "    - https://zhuanlan.zhihu.com/p/143092725\n",
    "* kNN存在的问题？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 参考资料\n",
    "* [Digits Classification Exercise](http://scikit-learn.org/stable/auto_examples/exercises/plot_digits_classification_exercise.html)\n",
    "* [knn算法的原理与实现](https://zhuanlan.zhihu.com/p/36549000)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}