{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# kNN 分类算法\n",
    "\n",
    "\n",
    "K最近邻(k-Nearest Neighbor，kNN)分类算法，是一个理论上比较成熟的方法，也是最简单的机器学习算法之一。该方法的思路是：***如果一个样本在特征空间中的k个最相似（即特征空间中最邻近）的样本中的大多数属于某一个类别，则该样本也属于这个类别***。\n",
    "\n",
    "kNN方法虽然从原理上也依赖于[极限定理](https://baike.baidu.com/item/%E6%9E%81%E9%99%90%E5%AE%9A%E7%90%86/13672616)，但在类别决策时，只与极少量的相邻样本有关。由于kNN方法主要靠周围有限的邻近的样本，而不是靠判别类域的方法来确定所属类别的，因此对于类域的交叉或重叠较多的待分样本集来说，kNN方法较其他方法更为适合。\n",
    "\n",
    "kNN算法不仅可以用于分类，还可以用于回归。通过找出一个样本的`k`个最近邻居，将这些邻居的属性的平均值赋给该样本，就可以得到该样本的属性。更有用的方法是将不同距离的邻居对该样本产生的影响给予不同的权值(weight)，如权值与距离成正比（组合函数）。\n",
    "\n",
    "该算法存在的问题：\n",
    "1. 当样本不平衡时，如一个类的样本数量很大，而其他类样本数量很小时，有可能导致当输入一个新样本时，该样本的K个邻居中大数量类的样本占多数。在这种情况下可能会产生误判的结果。因此我们需要减少数量对运行结果的影响。可以采用权值的方法（和该样本距离小的邻居权值大）来改进。\n",
    "2. 计算量较大，因为对每一个待分类的数据都要计算它到全体已知样本的距离，才能求得它的K个最近邻点。目前常用的解决方法是事先对已知样本点进行剪辑，事先去除对分类作用不大的样本。该算法比较适用于样本容量比较大的类域的自动分类，而那些样本容量较小的类域采用这种算法比较容易产生误分。\n",
    "\n",
    "kNN可以说是一种最直接的用来分类未知数据的方法。基本通过下面这张图跟文字说明就可以明白kNN是干什么的\n",
    "![knn](images/knn.png)\n",
    "\n",
    "简单来说，kNN可以看成：**有那么一堆你已经知道分类的数据，然后当一个新数据进入的时候，就开始跟训练数据里的每个点求距离，然后挑选这个训练数据最近的K个点，看看这几个点属于什么类型，然后用少数服从多数的原则，给新数据归类**。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 算法步骤：\n",
    "\n",
    "输入：\n",
    "* 训练数据： $T=\\{(x_1,y_1),(x_2,y_2), ..., (x_N,y_N)\\}$, 其中$x_i \\in X=R^n$，$y_i \\in Y = {0, 1, ..., K-1}$，i=1,2...N\n",
    "* 用户输入数据：$x_u$\n",
    "\n",
    "输出：预测的最优类别$y_{pred}$\n",
    "\n",
    "\n",
    "1. 准备数据，对数据进行预处理;\n",
    "2. 计算测试数据与各个训练数据之间的**距离**；\n",
    "3. 按照距离的递增关系进行排序；\n",
    "4. 选取距离最小的`k`个点；\n",
    "5. 确定前`k`个点所在类别的出现频率；\n",
    "6. 返回前`k`个点中出现频率最高的类别作为测试数据的预测分类。\n",
    "\n",
    "\n",
    "\n",
    "**深入思考：**\n",
    "* 上述的处理过程，难点有哪些？\n",
    "* 每个处理步骤如何用程序语言来描述？\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 距离计算\n",
    "\n",
    "要度量空间中点距离的话，有好几种度量方式，比如常见的曼哈顿距离计算、欧式距离计算等等。不过通常 KNN 算法中使用的是欧式距离。这里只是简单说一下，拿二维平面为例，二维空间两个点的欧式距离计算公式如下：\n",
    "$$\n",
    "d = \\sqrt{(x_2 - x_1)^2 + (y_2 - y_1)^2}\n",
    "$$\n",
    "\n",
    "在二维空间其实就是计算 $(x_1,y_1)$ 和 $(x_2, y_2)$ 的距离。拓展到多维空间，则公式变成：\n",
    "$$\n",
    "d(p, q) = \\sqrt{ (p_1-q_1)^2 + (p_1-q_1)^2 + ... + (p_n-q_n)^2 } = \\sqrt{ \\sum_{i=1,n} (p_i-q_i)^2}\n",
    "$$\n",
    "\n",
    "这样我们就明白了如何计算距离。kNN 算法最简单粗暴的就是将 `预测点` 与 `所有点` 距离进行计算，然后保存并排序，选出前面 k 个值看看哪些类别比较多。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## 2. 机器学习的思维模型\n",
    "\n",
    "![machine learning - methodology](images/ml_methodology.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 生成数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEICAYAAABPgw/pAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd8VFX6x/HPMzU9hBAQ6TYUWLFgb9hdG7vqunax4aor\n9t527RXLiq4Nu6w/sbuKoiioCypgAcWuIEVKAqTOZMrz++MOoYYkM5O5yeR5v177MjNz555vsuHJ\nmXPPPUdUFWOMMe2fx+0Axhhj0sMKujHGZAkr6MYYkyWsoBtjTJawgm6MMVnCCroxxmQJK+gma4nI\nv0XkmjSd6x8i8kw6zmVMa7GCbtokEflVRPZL5Ryq+jdVvSFdmZpLRJ4QkRsz3a4xVtBNuyQiPrcz\nGNPWWEE3bY6IPA30Bl4XkWoRuVRE+oqIishpIjIXmJg49gUR+V1EVojIZBEZuNp5GnrKIjJUROaJ\nyEUislhEForIKRvI0E9EJolIlYhMALqs9fp62xWREcDxwKWJ7K8nnr9cRH5KnO8bEflzen9qxlhB\nN22Qqp4IzAUOU9UCVb19tZf3ArYCDkw8fgvYHOgKzACe3cCpNwKKgR7AacBoESlp5NjngOk4hfwG\n4OS1Xl9vu6r6cOLr2xPZD0sc/xOwR6L9fwLPiEj3DWQ1psWsoJv25h+qWqOqdQCqOkZVq1Q1DPwD\nGCwixY28NwJcr6oRVX0TqAb6r32QiPQGdgCuUdWwqk4GXl/9mBa2i6q+oKoLVDWuqs8DPwA7tvB7\nN2aDrKCb9ua3lV+IiFdEbk0MZVQCvyZe6rLed0K5qkZXe1wLFKznuI2BZapas9pzc1JoFxE5SUS+\nEJHlIrIcGLSh441JhhV001Y1tgzo6s8fBwwD9sMZyuibeF5SbHshUCIi+as917sF7a6RXUT6AI8A\nfwdKVbUTMCsNOY1ZgxV001YtAjZp4phCIAyUA3nAzeloWFXnANOAf4pIQER2Bw5b7ZCm2l07ez5O\nkV8CkLgYOygdWY1ZnRV001bdAlydGKK4uJFjnsIZCpkPfANMTWP7xwE7ARXAdYm2mtvuY8CARPZX\nVPUb4C5gCk6x/wPwcRqzGgOA2AYXxhiTHayHbowxWcIKujHGZAkr6MYYkyWsoBtjTJbI6AJHXbp0\n0b59+2aySWOMafemT5++VFXLmjouowW9b9++TJs2LZNNGmNMuycic5o+yoZcjDEma1hBN8aYLGEF\n3RhjsoQVdGOMyRJW0I0xJktYQTfGmCxhBd2Ydkw1Rrz6EeLLRqJhW8Cxo7Od001SNPINxJdDYAdE\n/G7H6bC0ZgxU/wsIoeEPoMvLiG9Tt2MZl1hBNy0Wr3kCqkaBeMA3CDo/jYhtvuOKyFdAyPlavBD9\nGaygd1g25GJaruYJIARaC5EvIf6724k6LMk7BsgByQfJhcAObkcyLrIeumk53xZQvwSIgPjBU+J2\nog5LgrtBl5ednnlgB8TTye1IxkVW0E2LSac70ao7IV6OFPwdkRy3I3Vo4tvUhlkMYAXdJEE8RUjx\n9W7HMMasxcbQjTEmS1hBN8aYLGFDLsZkAY18g4bGI/6BEDzAppF2UFbQjWnnNDoXrTgWtA4lF4pq\nkLwj3I5lXGBDLsa0d9GvWfVPuQ7qP3QzjXGR9dCNae/82ya+CABeCB7kZhrjIivoxjRC45VozdMg\nHiTvRMRT4Hak9RLvRlD6KoQ/AP+WSGBHtyMZl1hBN6YRWnESRH9wvg5PQkr/43KixomvN/hOcjuG\ncZkVdGPWQ1UhOhtQ54nITFfzGNMcTV4UFZExIrJYRGat57WLRERFpEvrxDOm9WlkFvHKm9DaF51C\nDs60v8AuzoJX5EJwz9Tb0TCqoZTPY0xjmtNDfwK4H3hq9SdFpBdwADA3/bGMyQyNzUcrjl815U+r\nkPzhAEjJwxD6L+CBnINTaide+xJUXgMoWnQtnrxjUs5uzNqa7KGr6mSgYj0v3Q1cSsNnUmPaocj3\ngDfxoA7qpza8JBJAcv+M5A5LfROPqn8CESAKlTc2fBIwJp2SmocuIsOA+ar6ZTOOHSEi00Rk2pIl\nS5JpzpjWExiMU9BznOGVnMNbqaHgqi8l2PhhxqSgxRdFRSQPuBJnuKVJqvow8DDAkCFDrFti2hTx\ndIYubzhT/nybI4HtWqedktHo8osBRYpvt1vzTatIZpbLpkA/4MvEL2VPYIaI7KiqtnWNaXfE2w3y\n/tq6bQR2QLpOatU2jGlxQVfVmUDXlY9F5FdgiKouTWMuY4wxLdScaYtjgSlAfxGZJyKntX4sY4wx\nLdVkD11Vj23i9b5pS2OMMSZpttqiMcZkCSvoxhiTJaygG2NMlrCCbowxWcIKujHGZAkr6MYYkyWs\noBtjTJawgm6MMVnCCroxxmQJK+jGGJMlrKAbY0yWsIJujDFZwgq6McZkCSvoxhiTJaygG9NMGvke\nDU1E4zWpnSf8P7T6QTTydZqSmdVp9Be09kU0+pPbUTIumS3ojOlw4nVvworLQbzgKYUubyCS0+Lz\naPgDdNlIoB6q/w2l4xD/5ukP3EFp5Hu04i+AgCqUjkX8A9yOlTHWQzemOWqfBkKgNRAvhyR71xr+\nyDkPcUAhMj2NIQ31k0EjoLVAGMLvu50oo6ygG9Mc/kFAMPEgDt6eSZ1GgnsCOYAXEAgMSU8+4/D9\ngVUDDwHwb+1mmoyzIRfT4anGgMgGh1Ck8BJU8iD6M5J/MuLtllRbEtwTSh6ByJcQ3APxbZZk6nWp\nqtMjjS+BnIMQT3Hazt1eSHAn6HQvGv4ACe6BBPdwO1JGiapmrLEhQ4botGnTMtaeMU3R+i/RZac6\nH9HzjsVTdK3bkZIWr7oXaseAAt5SpMt4RAJuxzJpICLTVbXJj3M25GI6NK26EbQKiEHtODT6q9uR\nkhf6L2gdUAfxCojNczuRybAmC7qIjBGRxSIya7Xn7hCRb0XkKxF5WUQ6tW5MY1qJ5LLqn4FCe+7R\nBnbGGZ/3AEHwdnc5kMm05vTQnwAOWuu5CcAgVd0a+B64Is25jMkIKboRvJuAFEPhJYh3Y7cjJU2K\nroHCSyD/dKT0RURyW60tjc5D615r359oslCTF0VVdbKI9F3ruXdWezgVOCq9sUxHpLH56Ip/AiGk\n8PKMzB8WX2+k7M1WbycTRPxI/omt3o5Gf0XL/+SM1aNQ+iziH9Tq7ZqmpWMM/VTgrcZeFJERIjJN\nRKYtWbIkDc2ZbKXLznLmEddPRStOQjXudiSzPuEPQKNALVCHht51OZBZKaWCLiJXAVHg2caOUdWH\nVXWIqg4pKytLpTmT7WILcG64wbmBh3o305jVqEbR2AJU68E/EGcePUCu9c7bkKTnoYvIcOBQYF/N\n5NxHkzLVKEQ+B09JWudBpyz/TKi+D/BA7sFJ3Vpv0k/jlWj5X5w/uJ5iZ3y+0yg0/A4E9kBy9nM7\noklIqqCLyEHApcBeqlqb3kimNanG0YrhEJ0FGkeLrsKT91e3YwHgKTgDzdkfNAS+/m7HMSuF3oTY\nQiAM8XK09nk8hSORnH3dTmbW0pxpi2OBKUB/EZknIqcB9wOFwAQR+UJE/t3KOU26xOZD5KvEWhch\nqHnE7URrEF9fxL8lIuJ2lDZHI9+itWPR6I+ZbdhTDKz8/8OPeDpntn3TbM2Z5XLsep5+rBWymEzw\nlIB4EjMU/NCWhlxMozTyFVp+QuKRQOnziH/LzDQePBDyZkDoHWeuexv5RGfWZXeKdjDiKUBKnoTA\nnpA7DCm+ze1IrS5e9zrxxbsTXzoMjc5xO05ywh8CYZyVGiNQ//F6D9PQ+8RXXIeG3ktb0yIePEVX\n4ek6CU+n2xDxp+3cJr1sca4OSAKDkc6Puh0jIzS+DFZcAdRDfAm64hKk9P/cjtVy/m1wVnsMAT7w\nD17nEA1PQZefB4TQupeh5EEkuFuGgxo3WUE32U3Dqz+AeJVrUQDnE0L9FPAPatF0PwnuBiX/QsNT\nnFUE17fsbmQmEEk8qHeulVhB71CsoJusJt6N0Ly/Qu1/AC9SdLVrWTQ6Fy0fBitvmCp5xFnutZkk\nuBcS3KvxA4J7Qs1oGnbr2dCxJitZQTdZz1N0DVowEiQHkeB6j9HY7850zthcyD0CKboh/TNt6qck\ninnIaTP0TosKelPEvyWUvgz1MyCwLeLbNG3nBme9dZt91LbZRVGTcarq3HGYQeIpbrSYA2jVXRD7\nFYhC6A2IpGfdftUQ8WUjiS/eA63/nMT0IiAHCe6YljYa2orXJG7LrwJP+lZa1Mhs4ot3RRcNIF51\nT9rOa9LPeugmozS2GK04FmLz0cAuSMnDbXTWRHpuftaaMRCeCNRD6BXnvJ6NnZUdcw5MSxsNbS07\nLbHXqUJoAlL6XHrOW3kdxJc6D2oeQ3OPRHy9Wn6eyLeAIv6t0pLLrMt66CajtOaRVWu2RD6HcPqm\n16VCCi8Eb2/ACzl/BP8O6TlxvAJnuSNo2BhalyGtsU5N5AucqY31zs82bdYqE9LyshGvvA0tPxot\nP4Z45c1pymXWZgXdZJYEWfPXrvFhkEwSb3c8Ze/g2Wg2nuJb0jZWLHnDnZu5GhazIl2d/3UFdsTZ\n4CIHAmkcmy+6ITGEE4CCsxFvj5afpPYpnGsHdVD7dNqymTXZkIvJKMkfgdbPgOhspyec5TMxxNcT\nyiah0bmw4nKIfg2BHSDnkKTPqdGf0Mp/gMaRousQ/xZOWyUPQ90rzkG5f05Deof4N0e6TkrtJJ5u\nEF+AM+SU3Abbpmm2SbRp1zQyE61705nTnXNwh5iFEV+892rFsQuerv9L6jwaW4BW3eH8YSi8xPnj\n00o0OgetvAlQpOgqxNe31drKRs3dJNp66Kbd0uicxPomdSi5oHVIXgfYPCu+lIZxm/iypKcTasUp\nEHOWQtDITKTrxDSGXJP4+iCdH27yuHjNE1B9N0gnpOShzK1X0wSNznX++OFBii5NbtgpA2wM3bRf\nka9Xu0BX58zz7gjyR+BcewhC3qnJfyqJzcO5UBuH+ALc3tZAY0uh6k7QOogvRFdc6Wqe1WnFyRCe\nAOG30YpT3Y7TKOuhm/YrsB3Osq5BQJCcP7ocKDM8heeiucOAeGpDF7lHQ91LINJGhqtiTTx2h6pC\nfCENu2nF5rmaZ0OsoJt2S7wbQemrUP8h+Pojge1TOp9Gf4LwR846Kymeq7WJr3fq5yi6FhJ/GPBv\nm3qoVPN4u6EFf4Pq0SAFSNH1bkcCQETQ3KOg7nWn/5Dbdof17KKoMawcjx8GGgEUCi/Hk3+S27E6\nJGdzcGkDnxhWUVWIfAkI+LduUTanxoZT2lKxuRdFbQzdtFuqUeLLLyK+aAfiy85dZzkBjS1FQ2+j\n0V+aPln9Z6AxnNUKo1B1Cxqe2iq5zYaJeNpUMQenly6BbZylp1tSzGPl6NID0EXbEC8/Fl1j9c/0\ns4Ju2h3VOFr/GVo9GkLvgq6A8CSoG7fqmNhidOkf0RWXo0uHofVNfDL0D2bNMdsYGnq3VfKbltHY\nAuIrriNeeSsar3Q7Toto7RPOto/EIfoNhMa3ans2hm6cj4SxeeApQDwlbsdpki4/19mxRyM0XKgi\nltgnNaH+Y9B6oM55T92r619DPEH8m6NFN0PllTi36ucgwTTd/m+Spqpo+TEQXwx40cjXSGk7utNU\ncljVb5bE49ZjBb2DU1V0xcXOfpGAFt+FJ/cAl1M1TuO1icWuVvamPYAPvD3XvFjl24JV99jngq/p\nzSQ8eX9CfT3Q0LtIYLu0L57VVmj4Y7T2KfBtiRT8vWFxNI0vg7oXQfIg9yhEAi4nxfkjHV9Cw/TK\n6Gy3E7WI5A1H6z9NbDZyAAT3a9X2mizoIjIGOBRYrKqDEs91Bp4H+gK/Aker6rLWi2laTfz3RDFP\njO1V3wFtuKAjOSDFoMsAL/gGIp0fAylcY2xT/AOh071o6FXwD0Hyjm7e6QM7IIHs7ZlrdC667Cwg\nBOEpKIoUXugMY5X/NTE84IXqh9H4EvBtjnQeg3g6u5JXPPloYMfEBUkSs3LaD/HkI52fzFh7zRlD\nfwI4aK3nLgfeU9XNgfcSj017JAWrPfCAp6trUZpDxOMsCxvcH3IPQ0oeRDxF671QJTl7I3nHQ2Q6\nWvMQqtH1nLGDic0FWblQWCix3C7OGuqxeTgXhUOJpQUiEP0erX7QnawJUvIoUnwH0mk0UujejlPt\nQZM9dFWdLCJ913p6GDA08fWTwAfAZWnMZTJEPIXOXpWVt4O3C1J8m9uRmiS+TZCS+5s8TqPz0IrT\nccbR30M1jBSe1+r52jT/tuDplFjJN4bkn+g8L0Xg7Qux33CGqmI0DGut3DLPJSJ+yNnf1QztRbJj\n6N1UdWHi698BWz6tHZPgUKRsqNsxNkjjleiKqyD6E+SfiSevGR+9Y786vVEFpzf6VSundIdzJ2MF\neIqa3CxEPPlQ+jpEpoO3N+Lr5zwvAqXPQ93LqORBaCLUvwfePkjBWSlkiznDep6ytjEmn+VSviiq\nqioijd6dJCIjgBEAvXunfnebaT2q6lyEkrw2Nw9Yq25OXAyNQOU1aGBw07e9+7cBycfZVCKG5B2T\ngaSZpRp1diqqn+Z8r6XPIb7NNvge8RSsd9li8RRC/kkIQN5RKe8hqvFatOKvEJ0DkgulL6TlDtd0\n0XgFWjPW+XnkHZsVf3CSnYe+SES6AyT+u7ixA1X1YVUdoqpDysrKkmzOtDaNV6BLD0QXD0HLj0K1\nzu1Ia4otxBnfxVmQK17e5FvEU4B0+S9SfCtS+gKSjR/b6z9JXDCMgK5Aq/6VtlOn/Ec9PDExhBMC\nXe5sx9dGOBeBj4Ka0WjVnejyi9yOlBbJFvTXgJMTX58MvJqeOMYtWjM2cVEsBtEfnXUr2hAp+LvT\nyyMXfFslbgRqxvs8RUjOgYi/f+sGdIsUQMPyHT5nfLyt8JSwauqoHzylbqZZk1ZC7Hecew7CUP+p\n24nSojnTFsfiXADtIiLzgOuAW4H/E5HTgDlA8+aEmbZrjRsgSBTPtkMCO0DZZGctcG8/JIl9LbOR\nBAajBWdC7bPOFMPCC92OtEpgV8g7DepehsA2SMEZbidaRYrBt6kzHIRAzj5uJ0oLW5zLAKAaQpeN\ndC6WBfdGim9DxNv0G02jVMMQrwJPacvW/4j+hi4/3/njVXgJntxDWzFlx6Xxagi94XzKyfljm/59\ntx2LTIuI5DRrRxnTPBqZhVac5Cw/ENwNOj3Y7E8VuuIyZ+9R4rDiCjS4q2s39mQz52Jodl0ot8+t\nxrQCrboLtBqoX+3CZXPfvIJVa9Tg7OBjTDNYQTemNXhKaPgArHHwFDb7rVJ4JZAD+CF3WJvYv1Lj\ny4lX3kB8xZVotO3u2NPR2ZCLMa1Aiq521kKJ/gr5I5qcGw6J+wBCrzoFs8uLiKe0zQy16LKzIfIF\nEEfDk6HswzZ3r4Kxgm46oHjNY1D9MHh7ISWjEW/6b3QWT2ekc8uWedWa+6H6USAMtU9C2Xtpz5W0\n6Pc4U/xw7gHQOmdVRtOm2JCL6VA0+hNU3eus1hj9Gq280e1Iq4Qm4aw7E8e5H+BnlwOtJvcIZyqr\n5EFgF8Rjxbwtsh666Vi0DmenX3A2xahyM82acg6A6h9wFsUKQDOGaTJFCq+A4D7Ozy+4h9txTCOs\noJuspZHv0JoHwdMFKTjfmabmGwg5+zpbgUkuUnip2zEbSP4Z4NvEWZM85yAnbytRjSfG6+cjucMQ\nX68NZxOB4M6tlsekhxV0k5VUQ2jFcYmpgz40tgApecDZ7LfTKDR+PUgOIm3nn4CIQE7r7mizklb/\nC2rHgIbR2iehbKKzOJdp12wM3WSneHliT1EFIhBZc+sy8RS0qWKeceFJieGnOBBtW+P1JmlW0E12\n8nQH/4DETIxcyD/B7URtS85BifV6As46Pm1ovN4krwN3UUxbpNG5zp2SvoEpLcAl4oHOT0P9FPCU\nIP6t05iybVONoVW3Q/3HkHMwkn8WRKZBfDkE90QkmBiv7wexBc46Jp58t2ObNLCCbtqMeO3zUHmT\ns955YOfE+ifJ37wiEljvRg5Zr+4FqB0LhKDmNzT6M4QmgAh4N3U2mhCPbeuWhWzIxbQd1f/G2Qyh\nFsIfQXyR24naNNUw8cobiZcfj4ZW3YSksYVAfeJB3Bkvp875uUa/b7c/V1V1NmLRmNtR2iwr6Kbt\n8PWi4UOj+MBT7Gqctk6r7oba5yHyGbr8AjT6CwCSexRIobMlnafE2RiaACDOeHkbWU6gJVRDaPmR\n6OI90CV7Jf5ombXZkItpM6T4brTyeoiXI4UXIi3YZEOjPzr7avq3zd7didYW/REIO1+L1xkP9/Vz\n5pSXve9s/+brBxpDax5wfq75ZyASdDV2UkITIPYzEIF4OVrzBFJ0hdup2hwr6KbNEG8pUnJvi9+n\nke/Q8qNp2O6s9JkOcRFU8k9F6z9zrjl4ukJgu1WveQrAs1XiAUjhJS6lTBNZ/SYrn/MJxKzDCrpp\n/+o/wtlAOgoIhCdDRyjowV2hbLyzF6x/6/bZ826u4FDIPQpC/wX/YCT/VLcTtUlW0E375x+M86sc\nBYKJMeOOQbzdwdu91c6vsfmgEcTXt9XaaA4RQYquhqKrXc3R1llBN21GvGYs1L0IgSFI4cXNvpNT\nAkOg5EE0PBkJ7oIEd2vlpB1DvOZxqBoFCJp3NB4rpm2eFXTTJmj9Z1B1K1AH0e9Rbzck/5Rmv1+C\nu7W7Qq6Rb9CaJ8HXO3GxMuB2pDVV30/DRdfaZ9DCS9teRrOGlAq6iFwAnI5zNWomcIqqhtIRzHQw\nsfmsWtY2BNE5bqZpdRpfhlYcD1oDBNHYUqT4OrdjrclTBrFqQBMXJf1uJzJNSHoeuoj0AEYCQ1R1\nEOAFsmsLbZM5wb2deeeSD5KP5B3rdqLWFVt9X85wyzaRTpHGlxEvP574oh2JV41q9DgpeQgCO4F/\nG6TzE7blXDuQ6pCLD8gVkQiQByxIPZLpiMRT7MzYiHwPvl5tZi/NVuPbAjylEFdQhbzM9YW06m6I\nfA5EoeZJNDgUWW3K40ri64N0fipjuUzqki7oqjpfRO4E5uLsm/WOqr6z9nEiMgIYAdC7d+9kmzMd\ngEguBAa7HSMjRIJQ+ooz5dLTHcnk963VOLsi4azvorWZa9u0qlSGXEqAYUA/YGMgX0TWWaNUVR9W\n1SGqOqSsrCz5pMZkGfEUIDkHZbaYA1IwEqQE8IF/CAR2yWj7pvWkMuSyH/CLqi4BEJGXgF2BZ9IR\nzBjTOsTXF7p+DFpruxRlmVQW55oL7CwieeJcLdkXmN3Ee4wxbYCI14p5FkplDP0TERkHzMC5Re9z\n4OF0BTPGZIbGytHqu0HrkYKRiK+n25FMklKa5aKq1wFtbPKsMRum0R/RyttA/EjRVYi3h9uRXKXL\nRkB0NhBH66dC2SSbothO2Z2ipkNRVbTiJGcTaQSt+AUpe8vtWO6K/YTzIRuIL8ZZ6MzuCG2PbIML\n08HEIF6Bc3NzHOJ26wS5RzmbaUseBPez2/vbMeuhmw5FxIfmHgWh1xM39AxveE1jC4AA4u3iWj43\nSOFVkHMgaL2zl6tpt6ygmw5Him6AvGMBX8PuRvHKmxMbKytadB2evL+4mjGTRAQCO7gdw6SBDbmY\nDkdEEP/AhmKu8VqofRpnZcF6qLrL1XxtncYWEa+6i3j1I6iG3Y5jVmM9dGPEDxJM3AIvzhorZr1U\nY2j5XyC+BPChka+Qkn+5HcskWA/ddHgifqTkEfD1d1YWLBntdqS2K748cVE5hrNK5HS3E5nVWA/d\ndDga/Q1C48HXD4L7OkMwgR2QLq+7Ha3t85SAb5PEevUCwQPdTmRWYwXddBiqdWj917D8TNA6wA+F\nFyP5J7odrd0Q8UDnsc4fRE8hBPdzO5JZjRV00yFobCG69M+JHYLqceahRyE0Aaygt4h48iHvSLdj\nmPWwMXTTIWjtS6ArcGayKM52d7mQs29q59UY8cqbiS/5I/Gqe1HVNKQ1Jjkdpocei8WoLK+muEsh\nHo/9HetoxNsVxY9zMS8IOQcgOQdA8IDUTlz3EtT+BwhBzRjwb+ncpGOMCzpEQa+sqOLcna9kyW9L\n6dq7jPum3ERRZ1s6tEPJPQKiP0D4Q8g5ECk4Ly0LUGlsEc4QDkAcYotTPqcxyeoQXdW3H/+AxXOX\nEglHWTx3CROenOR2JJNhIl48RVfiKXsLT+H5aVtNUPKOAOnkrIPi6QS5h6TlvMYko0P00POL8/D6\nPETrweP1kF+c53YkkyXEuzGUTYTYPPD1cfYKNcYlHaKHfuDwoexx5M506lrMnkftwv4n7eV2JJNF\nxJOH+LewYm5c1yF66F6fl8uePNftGMYY06o6RA/dGGM6AivoLojUR4jUR9yOYYzJMlbQM2z8mIkc\nXngihxedxISnbbaNMSZ9rKBnkKpy398fJRqJEa2Pct/Zj7gdyRiTRVIq6CLSSUTGici3IjJbRHZJ\nV7Bs5Q+sug7tD/pdTGKMyTap9tDvBcar6pbAYGB26pGyl4hw3biL6dy9hNKNO3PduIvdjmSMySKS\n7GJCIlIMfAFsos08yZAhQ3TatGlJtZesL96fxTM3jqN7v278bdTJ5BfZTUXGmPZFRKar6pCmjktl\nHno/YAnwuIgMBqYD56lqzVpBRgAjAHr37p1Ccy23fMkKrj7sVsK1Yb75+DtCtWGueu78jGYwmafR\nuVD/P/APRPx/cDuOMRmTypCLD9gOeFBVtwVqgMvXPkhVH1bVIao6pKysLIXmWq5i4XJWLtkRqY/y\n27fzM9q+G1SVqW9MZ/yYidSsqGn6DVlGo/PQ8mFo5c1o+fFo+GO3IxmTMakU9HnAPFX9JPF4HE6B\nbzP6DOxJvz/0Jrcgh0BugGMu+5PbkVrdE9c9z03H3s39I8dw9g6XE41E3Y6UWfWfAnEgBITQ0HiX\nAxmTOUkPuajq7yLym4j0V9XvgH2Bb9IXrXni8TivP/g2P0z/mYNO3YdBu2/V8JrX62XUpOv59tMf\n6bxRJzbedKNMx2u2T96cwawPZ7Pjwdvxhz22avoNjfhg7EeEasIAVCxcxu+/LqHn5t3TFbPt8w+C\nlZd0JBfN9utkAAAZKElEQVQCO7mbx5gMSnUtl3OBZ0UkAPwMnJJ6pJZ59uYX+c8tr1BfV88H/zeF\nh764gx6brSpgPr+PQbttmelYLfLJmzO44ehRhGvDvHzfm4yafD1bbL9pUucaPHQgSxcsIxqOEMwN\nUNazc5rTtm3i3wI6j0FD7yCBbZGcP7odyZiMSamgq+oXQJNXXluDqnLLCffxwX8+btj2y+vzMOfr\neWsU9Pbgq0lfE651etWqyuypPyRd0Ec+cAZ9B/WmfEEFh511IMHcjrcCoASGIAFXfi2NcVW7WW1x\nybxynr3xRfxBHyde+xeWzq/gf69+1lDMPR7BH/QzaPe23Rtfn50O2Z5XR49H44rH62GbfQYlfS6f\n38cR59kmC8Z0RO2moF+417UsnrsU8QizP/mBq547v6GYi0fYZJu+3PzmVRSVtr+t5bbecwB3fXA9\ns6d+z7b7DKLPVj3djmSMaYfaRUGPRqIs+nWJU8Bj8Ous3+i+STfOvONEnrnxRTbqU8a14y6ipGux\n21GT1n/IpvQfktwwizHGQDsp6D6/j50P254vJs5CFfY5bncADj/7IA4/+6CUz6+qTB43lcVzljD0\nmN0o61ma8jmNMSbTkr71Pxmp3Pofi8X47K0v8AV8bL//1mnb5Bfg2Zte5D+3vkw0EiOvKJenfrzf\nlggwxrQZzb31v90sn+v1etn50O0ZcsDgtBZzgP+98imhmjDR+ijRLL+jdNzdb3Bw7rH8ZaPT+e6z\nH92OY4xJo3ZT0FvTLocPISc/iNfnwef30WvLHm5HahXLl6xgzBXPEglHWb54BXed/qDbkYwxadQu\nxtBb2/FXH0WPzTdm0Zwl7HPsbjbcYoxpl7Kyh15XE+KGo+/ihE3O5rmbX0RVmTj2I/7vjldZuqBi\nneNFhL2P2Y1jLvsTXXtndgGxTOpUVsypNx+LL+CjU1kRFz16ltuRjDFp1G4uirbEmKueY9yoN4iE\nI+TkB9n9iJ356KWpROujFJQU8NRP95Obn9NwfCwaw+P1JDU2H4/H+frj78jJD7L5dpuk89swxhgg\nCy+KtsSy35cTrXdWGRQRvpr0tXPRMxIjXFfP/B8WNhz79PUvcHDucfy583C+mfJdi9v651F3ctUh\nN3PBntfy9A0vpO17MMaYlsrKgn70pcMo6JRHMDdAzy26M/SY3cjJD+Lzewnm+umRWH2w4vdljL3l\nJeKxODUrarn7zIda1E5NZS2fvDGDuuoQ4dowr9z31jrHxKIx5syeR01lLeDMeV/48yKWL1mR+jea\nRsuXrOC5W17itQfeJlIfcTuOMSYJWXlRtFf/Hoyd9xDLF1dS1qsUEWHTwX1Z8ls5+x6/e8Nwi8fr\nAVYNs/gCzftxROojzJ09ny49S8kvzqOqogqP10PP/msuChaqDTNylytZ+PMiPF4Pd0++gRfueo3J\n46aCKpc+eS57/cX9fbVjsRjn7nQlS+aX4/V5+fp/33LFM+e5HcsY00JZWdABgrlBuvVZdYFzn2N3\nX+eYTmXF/O2uk3j0imcpLCng0sfPafK8ddV1nL3D5ZTPr0BEuPjxc3jvmUnkFeVxxu0nrnHs9He+\n5PdfFjesT/7MDS8w9Y0ZRMJOD/ixK55tEwW9cmkV5QsqiEVixCIxPn9vptuRjDFJyNqC3lwtXT7g\ns/FfUD6/grrqEABTXvuMf7x06XqPLdmoE/F4HACPz/k0sPK6q8cjlHYvSSl7uhSXFbHRJt1Y9Oti\nPF4POx9mS88a0x5lTUH/fvpP/OPPd1BbVcdZo4Zz4Cl78+GLU5k8bgrb7b81B52yT7Nnsfwycw43\nHXcv1ctr2OfY3Tn49H3pucXGAHTpWYrGnZlBgZwAG/Xr2uh5Buy8BZ3Kilk0ZwnxaJyPXv6EgpJ8\nuvXtSunGJVzSjE8EmeDxePjXlJt495kPKeiUz9BjdnU7kjEmCVkzbfHUAec33LIvHuHQMw/g7Sfe\np76unpz8IBePOafZwxunbHke875f0PA4mBvg7g9vaJiW+NoD4/nvw++y5U6bc859pxII+hs914H+\nvxKPxRse5+QF+fv9p3Hg8L2T+TaNMR1Qh5u2uHKaIoDGldcffJv6unoAQjVhfvz8l3XeU1lexUcv\nf8Jv3625dkttYkbKSuG6ev736mcNjw8/+yAe+uJOLnjozA0Wc4Dt9vsDPr931RPCBnv1JruoKhr7\nHdWQ21FMB5A1Bf3CR/6GP7juCFJeYS45+cF1eucrllZy2sDzuWP4aM7a/jI+n7jqQuA59526RhHO\nyQuy+fbrv2moPhxh4nMfMumFKcRisXVe/+crlzHywTPY8eDt2HKnzTnzrpMZvNfAZL9NYtEYsei6\n7Zi2RzWKVpyELtkfXbwrGsn4Huqmg8maIReAWR/N5sKh1zWMcXfu3olz7j2VzbfbhO6bdFvj2Ekv\nTOGu0x+grsrpOe1z3O5rTNWrqwnxv1c/4/N3v2L7A7Zh72N2W2+bl+7/T2ZP/QGAnQ/dnqvGXtAa\n3xoAE56exKgz/o0IXPTY2ex73B5NvicWjVEfjqxxZ6zJDA1PQZefBZr4xBfcD0/JA+6GMu1Sc4dc\nUr4oKiJeYBowX1UPTfV8qRi0+1YcfckwXhs9ni49OnPHxOso7b7+Xe97b9WjYWw7mBdcZ1Pm3Pwc\n9j1ujw0WzVg01rDpBsCU19L/x+rbT3/ghTtfY6N+XXn5vjcbhpbuOfOhJgv6N1O/54oDbyRcF+bg\nM/Zj5Ogzmmzvk/9O581H32Ornbfg6EsOx+PJmg9xmecpBl15/cQPHts4xbSudMxyOQ+YDRSl4Vwp\nO/2W4zn9luObPK6kWzGn3nQcrz3wNjWVtVQsWo6qNjoT5q0x7/HIpc9QVFrAP166lL4De+H1eem1\nVU/m/7AQjwibp3kLucryKi7Z73pC1SECOX7i8VWfpnz+pv+ve+C8x6mtqgPg7Sc+4IjzD6Xn5t0b\nPf6XmXO44a93E64NM33Cl/gDPo68wNW/0e2a+AeghRdCzRPg2wwpvMTtSCbLpVTQRaQncAhwE3Bh\nWhJlwOxPfuDS/a4nHosRCUdRVV4bPZ5NB/dd7w1IleVV/OucR4mEo1RVVHP78Pu5fcK13HPmQ4jA\nLodtzyZb9+WI8w9Ja87Fc5c23MdaH4qw8abdqK0KIQKXN+NOzpyCIOIRZwhKlWBuYIPHz/12AR6v\n02K4tp7vp/+U6rfQ4Xnyh0P+cLdjmA4i1R76PcClQGFjB4jICGAEQO/evZNqZNZHs1nw0yJ2OmQ7\niruk/kFg3KjXCdWsOesgFolRvmDZeo+PRqJrPA7Xhhk9cgwfv/pZYoejBWyydV9yC9I7Tt1nYE+6\n9ilj8dwlxGNxjr3yCA46ZZ9mv/+iR87i2j/dTsXvyzj5n39tcq/UbfcZRE5eEEGIxWIcfMZ+qX4L\nxpgMSvqiqIgcChysqmeLyFDg4qbG0JO5KDr+8YmMPncMiDNjZcy396a8AcVjVz7HS/e8QX3IuQU/\nmBsgvziPB2fcTueN1n/35pirx/L8ba8QyPFz03+v5PGrxzLro28bXvfn+Dnh6iM57sojAeePwDtP\nTiJUHeKA4UMp6JSfVNZQbZgv359Fl56lbDq4b1LnaInKiiq+/vg7em/Vgx6bNT48Y4zJnOZeFE2l\noN8CnAhEgRycMfSXVPWExt6TTEG/cOi1zJw8G4C8olyuf/WylKb9ffnB1/z2/QJmT/meX2fN5eAR\n+zNgly3ovkk3cvKCG3xvfager9+L1+tlxrtfcfVhtxAJr+q9DzlwMLe8dTUANx57N1Nfn0Y8pmy8\naTcemTkq7XuhGmM6hlaf5aKqVwBXJBobitNDb7SYJ2vwXgP5ftpPhGvr0bjSq//GSZ/rtQfG88hl\nz4BCQUk+Y769t0XT+QI5q8agt9tvax6ZOYpzd76SSDiCKmvc/TljwleEa50bm+b9sJCaFbWN9tI/\nnziTl+97kz4De3HSdX/BH9jwzUrNUbOihqplNXTrU2Z/SIzpINr8Wi4nXHsUnboWMffb+fzxtH0b\nHRJpjglPT25Y+dDj8/DLzLkM2HmLpM/XY7PujJl9D5+N/4Je/Tdmyx03b3htu/23buihd9+kK/nF\n6x8mWjx3CdccfltiZslXROujnHnHSUlnAvhy0tdcdcgtaDzONvv8gRteu8ymHxrTAaSloKvqB8AH\n6TjX2rxeL8PO+WNazjV4rwH8MnOO03NWNjiFr7k6lRWz/4l7rfP85U+dyztPfEBddYgDT9m70V7y\nwp8X4/U5xba+rp4fZ6y7REFLjblqLOFa5w/Xlx98zchdr+LXWXMZcsA2XP38Bc2a8miMaX861L/s\nU246li49S5n/w0IOGbE/RaWNTs5p1FeTv+H6o+4kXBdh5AOnr7eYgzNPvDmzRPrvuBnFXYpQhXgs\nxp/PO7jFmdZW2r0Er89DLBonWh/lpy9/JRqOMn3Cl0x87iMOOHloym0YY9qerC7otVV13D3iIX6Z\nNYejLjyMg07Zhz/9PbXe/q0n3seKpVUAjDrj3+x19K58+f4s7jhlNB6Ph8ufGck2ew9qOP61B99m\nzJXPUdSlkBtevYw+A3qtcb6cvCAPfXEHMz/8lo36daX3lj1Sygcw8oHTqa2q4/efF7HZdv34+BVn\nYTFVJRqxdWCMyVZZtZbL2u4751HGj3mPSDhKMC/A6E9vXaegrk/VsmqikRglXYvXee3YXmeydH4F\nAD6/l1eWP8lfup3esOFFUWkhLy4ZA8CyRcs5vu/ZDTsU9d9xM+6feku6vr1mqV5ew0VDr+OXmXMY\nsEt/bptwDcHcDc/mMca0LR1u+dz1WTx3ScO0Qo/XQ8Xvy5t8z7vPTOKvG4/g+N5/47Ern13n9Use\nP4e8wlx8AR9/v/80AjmBNW48iqy2jO/aqyJGQpnffLmgUz4PfXEnb4X/wz0f3WjF3JgsltUF/bir\njiSnIIdgXpDeW/Zg0O5bNvmehy5+ikg4QqQ+ygt3vk4ocXFxpe3225pXlj/Jf+Y91DAffuToM/D5\nffiDPi54+MyGY7v0KOWoCw/F4/WQV5TLeQ82vThWa/H6vKgqNStqyOSnMmNM5mTFkEtNZS03HXMP\nP33xC388Y1+G//OYhteqllVTsXAZPftvjNfrbfQc8XicD8dNZfT5j7Ms0ZMP5gV5ZdkT68wKmfrm\nDK4/8g5EhG32HsQNr19OPBZHRPD61m0jGoni9XldnQ9evbyG83e/mnnfL6TnFt259+MbyS9O7u5V\nY0xmdaghl6evf4HPJ86k4vflvDjqDb6avGojgcKSAvoM6LXBYg5w/7mPcedpD1C9vAZvwIvH6wGU\niWM/WuO4WCzGTceMIhKOUh+KMOO9mcz5Zh4+v2+9xRycGS9u39zz9uPvs+CnRcSiMRb+vIi3H//A\n1TzGmPTLilkuVeXVxBKzN0SEmhXOhgKqyleTv8Hj8TBo9y0REX784hfefXoSfQf14cDhQxsK7ccv\nf9pw09FK4dp6Rp3+b1665w16bNad8/49gvL5FdTXrRoLj0aiSU1/zLSc/GDDSoriEXLybSzdmGzT\n7gt6uC7ML7PmOuPCAn0G9mLIgYMBuPO0B5j8whQA9jthT4676kgu2ONaQjUhgnlBKsurOPriwwH4\nw14DmPLqZ8SiscSccGdjglg0xk9fzGHO1/MQEc7512n4Ar6G/UoH7NKf0u7J372aKQcMH8rn781k\n+rtfsf3+W3PA8KFuRzLGpFm7L+jvPfsRc2fPA5wLf7scPgR/wI+qMuGpSQ3b0Y0f8z67DtthtfW+\nw3z+3syGgn7pE3/n9Qffpnp5DZ27l/Cvcx5teC9ANBJj4S+LKelazA2vXsbT17/ARn27cs59p7Yo\n76yPv+Wp656ndOPOnHXPcIo6Z6Z37w/4ufr5drNkvTEmCe2+oDubOTtF2uP1NCxsJSJs1K8ri35Z\nDCJsvFk3+u+wGT6/z9noQYR9j1+1mUUg6OfI853Vf288ZhSsdbHYH/Rz0j+OBpyZLtvtt3XDa3O+\n+Y1li1YwcLf+G1xYq6aylisOupFQTRif30vV8hpufO3ytPwcjDGm3Rf0fY7bnY9f+ZTp73zJVrts\nwWFnHdDw2p3vXcdjV47F4xVOu/k4ikoLefiru5j6+jR6bdmDrfccsN5z1qyoW6Oe+4N+Lnvq7+x0\n8HbrHPvWmPcYfe4YPD4PfQb04qJHz+L/7niVkm6dOOGaI8ktyG04dsWSyoZefzQSY953C9L0UzDG\nmCyZtphuP335KxfseQ11VSE8Xg+bDO7DvR/fRCC4bu/7tIHnM3f2fACCeQG8Pi91VSF8AS87HbI9\n1427uOFYVeWy/a/n209/JBaLM3L06WssuWuMMevT6uuhZ7NNB/fl5fInWPzbUuqqQvQZ0LPRKYl9\nB/Zi4c+LEnekCpF6Z4/SSDi6zp6cIsItb1/Nd5/9RFFpYVpWezTGmJWyYh56a/D6vHTv141Ntu7T\naDEHuPDRszhw+N4MOWAwt71zNf0G9Sa3IIec/CCHnnnAOsd7vV4G7LyFFXNjTNpZDz1FddUhPh3/\nOeXzKwjX1XPbhGv48v2vKe5SyKDdt3I7njGmA7GCnqJxd71O+bwKYrE4P8z4mWnjv2DoX3dzO5Yx\npgOyIZcUOXdgrvwxCsEmNpo2xpjWYgU9RUdfMow/7DmAgk757HvC7ux0yLpTG40xJhNsyCVFeYW5\n3PbONW7HMMYY66EbY0y2SLqgi0gvEXlfRL4Rka9F5Lx0BjPGGNMyqQy5RIGLVHWGiBQC00Vkgqp+\n09QbjTHGpF/SPXRVXaiqMxJfVwGzgdS3rDcpq6sJ8fErn/LdtJ+aPtgYkzXSclFURPoC2wKfrOe1\nEcAIgN69e6ejuVbz+6+LCeT46bxR21/fvDGR+gjnDLmMpQsqiMeUs+4+mUPO2N/tWMaYDEj5oqiI\nFAAvAuerauXar6vqw6o6RFWHlJWVpdpcq/nXuY9y2oDzOaHfObz12Htux0nanK/nsXR+BXVVIcK1\nYV5/8B23IxljMiSlgi4ifpxi/qyqvpSeSJlXW1XHfx96l/pQhEg4wpirxrodKWld+3Rp+DqQ46f/\nDpu5mMYYk0lJD7mIsxnnY8BsVR2VvkiZF8jx48/xE6uOISJ03qiT25GSVtS5kDvf/wcv3v0G3Tfp\nxnFXHuF2JGNMhiS9HrqI7A58CMwE4omnr1TVNxt7T1teD33mh7MZfd4Y8gpzuXjM2Wy86UZuRzLG\nGKD566HbBhfGGNPGNbeg252ixhiTJaygG2NMlrCCbowxWcIKujHGZAkr6MYYkyWsoBtjTJawgm6M\nMVkio/PQRWQJMCfxsAuwNGONp5/ld197/x4sv/vay/fQR1WbXAwrowV9jYZFpjVnonxbZfnd196/\nB8vvvmz4HlZnQy7GGJMlrKAbY0yWcLOgP+xi2+lg+d3X3r8Hy+++bPgeGrg2hm6MMSa9bMjFGGOy\nhBV0Y4zJEhkt6CLSS0TeF5FvRORrETkvk+2ni4h4ReRzEXnD7SzJEJFOIjJORL4VkdkisovbmVpC\nRC5I/P7MEpGxIpLjdqamiMgYEVksIrNWe66ziEwQkR8S/22zu5M3kv+OxO/QVyLysoi02a2+1pd/\ntdcuEhEVkS7re297kukeehS4SFUHADsD54jIgAxnSIfzgNluh0jBvcB4Vd0SGEw7+l5EpAcwEhii\nqoMAL3CMu6ma5QngoLWeuxx4T1U3B95LPG6rnmDd/BOAQaq6NfA9cEWmQ7XAE6ybHxHpBRwAzM10\noNaQ0YKuqgtVdUbi6yqcQtIjkxlSJSI9gUOAR93OkgwRKQb2xNkPFlWtV9Xl7qZqMR+QKyI+IA9Y\n4HKeJqnqZKBiraeHAU8mvn4S+FNGQ7XA+vKr6juqGk08nAr0zHiwZmrk5w9wN3ApkBWzQ1wbQxeR\nvsC2wCduZUjSPTi/APGmDmyj+gFLgMcTw0aPiki+26GaS1XnA3fi9KgWAitU9R13UyWtm6ouTHz9\nO9DNzTApOhV4y+0QLSEiw4D5qvql21nSxZWCLiIFwIvA+apa6UaGZIjIocBiVZ3udpYU+IDtgAdV\ndVughrb9UX8NiXHmYTh/mDYG8kXkBHdTpU6d+cPtspcoIlfhDKc+63aW5hKRPOBK4Fq3s6RTxgu6\niPhxivmzqvpSpttP0W7A4SLyK/AfYB8RecbdSC02D5inqis/GY3DKfDtxX7AL6q6RFUjwEvAri5n\nStYiEekOkPjvYpfztJiIDAcOBY7X9nVTy6Y4nYIvE/+eewIzRGQjV1OlKNOzXARn7Ha2qo7KZNvp\noKpXqGpPVe2LcyFuoqq2q96hqv4O/CYi/RNP7Qt842KklpoL7CwieYnfp31pRxd11/IacHLi65OB\nV13M0mIichDO8OPhqlrrdp6WUNWZqtpVVfsm/j3PA7ZL/PtotzLdQ98NOBGnZ/tF4n8HZziDgXOB\nZ0XkK2Ab4GaX8zRb4pPFOGAGMBPnd7jN374tImOBKUB/EZknIqcBtwL7i8gPOJ88bnUz44Y0kv9+\noBCYkPi3/G9XQ25AI/mzjt36b4wxWcLuFDXGmCxhBd0YY7KEFXRjjMkSVtCNMSZLWEE3xpgsYQXd\nGGOyhBV0Y4zJEv8PyAeps6WNomQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f26dc6f6da0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEICAYAAABPgw/pAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VOXZ//HPNZNksgABJIgKCIoiSFErKoobFVp3qP25\n1d22aBeXPlYfta1La63WpdrazSqPWq3Waltrq1VEhdq6AbLjjmyybwlZJsnM9ftjBkQkJJmZ5ExO\nvu/XKy9mOee+v4Rw5Z773Occc3dERKTjiwQdQEREckMFXUQkJFTQRURCQgVdRCQkVNBFREJCBV1E\nJCRU0EWaYGY3mNnDQecQaSkVdMlrZvaRmY3JQTvnm9krucjURPsPmNlNbdW+SEuooIuIhIQKuuQt\nM/sD0B942sw2mdlV6ddHmtl/zWyDmc0ys6O32ud8M/vQzKrMbKGZnWVmQ4DfAoem29nQRH8DzWxK\net9JQK9t3v+zma0ws41mNtXM9k2/PgE4C7gq3f7T6devNrMP0u3NN7Mv5/67JLIVd9eXvvL2C/gI\nGLPV892AtcDxpAYkY9PPK4AyoBIYnN52F2Df9OPzgVea6etV4E4gBhwJVAEPb/X+hUDX9Pt3ATO3\neu8B4KZt2jsV2DWd83SgGtgl6O+pvsL7pRG6dDRnA8+4+zPunnT3ScA0UgUeIAkMM7MSd1/u7vNa\n0qiZ9QcOAn7o7nF3nwo8vfU27j7R3avcPQ7cAOxnZuVNtenuf3b3j9M5/wS8Bxzcyr+vSIupoEtH\nsztwanq6ZUN6+uRwUiPfalIj4YuB5Wb2TzPbp4Xt7gqsT7ex2aLND8wsama3pKdQKkl9coBtpmW2\nZmbnmtnMrXIO29H2ItlSQZd8t+3lQJcAf3D37lt9lbn7LQDu/py7jyU13fI28Psm2tnWcqCHmZVt\n9Vr/rR5/FRgHjAHKgQHp12177ZvZ7um+vwPs5O7dgblbbS+Scyroku9WAnts9fxh4CQz+1J61Fxs\nZkebWV8z29nMxqWLchzYRGoKZnM7fc2saHuduPsiUlM3N5pZkZkdDpy01SZd022uBUqBm5vJWUaq\nyK8GMLMLSI3QRdqMCrrku58CP0hPW3zP3ZeQGilfS6pYLgGuJPWzHAH+B/gYWAccBXwz3c6LwDxg\nhZmtaaKvrwKHpPe9Hnhoq/ceIjUFswyYD7y2zb73A0PTOf/m7vOBO0gdaF0JfA74T0bfAZEWMnfd\n4EJEJAw0QhcRCQkVdBGRkFBBFxEJCRV0EZGQKGjPznr16uUDBgxozy5FRDq86dOnr3H3iua2a9eC\nPmDAAKZNm9aeXYqIdHhmtqj5rTTlIiISGiroIiIhoYIuIhISKugiIiGhgi4iEhIq6CIiIdGuyxZF\nJL94sgbf9EtIrsDKJmCFQ4KOJFlQQRfpxLzy+1A3CajH41Og4t9YpKzZ/SQ/acpFpDNrmAfUpx57\nApKrAo0j2VFBF+nMSs8CSsDKoGAgRPs3u4vkL025iHRikbLz8KIDILEWYodhFg06kmSh2RG6mU00\ns1VmNnc7711hZm5mupO5SAdlhcOx4tGYxYKOIllqyZTLA8Cx275oZv2ALwKLc5xJREQy0GxBd/ep\npG6au62fA1eRurO5iIgELKODomY2Dljm7rNasO0EM5tmZtNWr16dSXciItICrS7oZlYKXAtc15Lt\n3f1edx/h7iMqKpq9PruIiGQokxH6nsBAYJaZfQT0BWaYWZ9cBhMRkdZp9bJFd58D9N78PF3UR7j7\nmhzmEhGRVmrJssVHgVeBwWa21My+1vaxRESktZodobv7mc28PyBnaUREJGM69V9EJCRU0CVj3vgB\nXvNnvPH9oKOICLqWi2TIG97G157+yQs7PYoVDg0ukIhohC4Zik8lddnVWiAO8SkBBxIRFXTJTOFw\noDD9JJZ+LiJB0pSLZMRiI6H73Xh8ChY7CouNCjqSSKengi4Zs+LRWPHooGOI5C1PVuLrLoDG+RA7\nHOv+a8wKm98xQ5pyERFpI179EDS+DSQg/gbUPdum/amgi4iEhAq6iEgbsbJzoWAfIApFB0PxcW3a\nn+bQRUTaiEW6Yb2ebLf+NEIXEQkJFXQRkZBQQRcRCQkVdOn0PLmO5NqzSa46gmT1g0HHEcmYCrp0\nel55EzTMgORKqLpDV4+UDksFXSS5AWhMPbYIJKsCjSOSKRV06fSs6xVgXYBCKDoECvcLOpJIRrQO\nXTo9K9wXer+WGplHemJmQUcSyYhG6CKAWREW3UnFfCvutSQrf0Jy3UV4/fSg40gLNFvQzWyima0y\ns7lbvXabmb1tZrPN7K9m1r1tY4pIe/PKH0PNY1D/Er7+Qjy5LuhI0oyWjNAfAI7d5rVJwDB3Hw68\nC1yT41wiGfFkNZ7cEHSMcGhYAMTTTwwSy4NMIy3QbEF396nAum1ee97d08sCeA3o2wbZRFolWTsJ\nXzUSXzWS5KrRePw/QUfq2EovAIrByiDaDwr2DjqRNCMXB0UvBP7U1JtmNgGYANC/f/8cdCfShKqb\n2DKiTC7D118MFc9h0V0DjdVRRUpPxouGQmIlFI1o0xszSG5kdVDUzL5PagHvI01t4+73uvsIdx9R\nUVGRTXciOxbpts0LUUgsCyRKWFjBICw2CrNY0FGkBTIu6GZ2PnAicJa7e84SiWTIuv8cIptn/2IQ\n7aObV0unktGUi5kdC1wFHOXuNbmNJJIZKxiE9X4Rb1yUGpkXfV4jS+lUmi3oZvYocDTQy8yWAteT\nWtUSAyal1+2+5u4Xt2FOkRazgt2hYPegY4i0u2YLurufuZ2X72+DLCIikgWdKSoiEhIq6CIiIaGC\nLiISEiroIiIhoYIubca9geSGK0iuHElyw1V8crUIyTVPVpFc/x2Sq48nWfO3oONIQFTQpe3UPgl1\nk8DXQd1zUPtU0IlCy6tuhvhLkHgfKq/DGz8KOpIEQAVd2o5XA4n0kyT4piDT5Iw3LsZr/4E3Lg06\nyicSy4GG1GOLQHJtoHEkGCro0nZKvgLR3YDC1J8lXw46Uda8YQG+9iR84w/xtSfijR8EHQkA6/Id\nsBKgGAqG6DZ6nZRuQSdtxiLdode/wDeAdccsBOOH+GTwOsDBCyH+MhTsGXQqrGgEVEyF5BqIDgzH\n91paTQVd2pRZBKxn0DHwhvfwmokQ6YN1uQiz4swaKtgXKAZqgSgUDM1hyuxYpBwi5UHHkACpoEvo\neXITvu6M9Bx+EZ74GOt+a0ZtWfFovPzm1Mg8NgaLHZrTrCLZUEGX8EuuIHXZfgfi0DArq+YiJSdA\nyQm5SCaSUyroEn7RARDtm77ZhUPp6Tlr2huX4tW/h0hXrOxiLNIlZ22LtJYKuoSeWQH0/DPU/xsi\nvbCiA3PSrnsCX3d6eolgFG+Yj/WcmJO2RTKhgi5tzhNroGEOFA7Bon0CyWCRUij+Um4b9SpIrgeS\nqa+GebltX6SVVNClTXliGb7mZFLz10no+ThWGJK7x1t5ar134wJwD8U6e+nYVNClbdVNBo8D9YDh\ndc+EpqCbGfR8MHXKvXWBosOCjiSdnAq6tK2CQUA09diKsYK9Ao2Ta2ZFuZ/KEcmQTieT7UpW309y\n5YEkVx+Xuulyhix2GHS7EWKjocuVUHx8DlOKyNY0QpfP8MYlUHUXEIdENV55HdbzwYzbi5SOh9Lx\nuQsoItvV7AjdzCaa2Sozm7vVaz3NbJKZvZf+s0fbxpT21QBY+nEyPQcuIvmuJVMuDwDHbvPa1cBk\nd98LmJx+LiFhBXtA6alABKwb1u0HQUcSkRZotqC7+1Rg3TYvjwM2fwZ/ENDn6ZCJdPshtvNsrPeb\nWOGwoOOISAtkelB0Z3dfnn68Ati5qQ3NbIKZTTOzaatXr86wOwmCWVFqaZ6IdAhZr3Jxdyd11khT\n79/r7iPcfURFRUW23YmISBMyLegrzWwXgPSfq3IXSUREMpFpQf87cF768XmA7v4rIhKwlixbfBR4\nFRhsZkvN7GvALcBYM3sPGJN+LiIiAWr2xCJ3P7OJt47JcRYREcmCTv0XEQkJFXSRkHFP4o0f4cnK\noKNIO1NBFwmR1F2UzsfXnIyvPgKvf7MN+vDUjbe9ydXKEhAVdJEwaZgDjbOBOvBavOrunDbvyU34\n2nH4qoPwNcfhyY05bV+yo4IuHYonK/GG+bjXBx0lP0V6gifTTwoh17f8q/0bNC4EEpBYAjWP57Z9\nyYounysdhje8i687A0hCpBfs9Dcs0iXoWHnFCvrj3X4M1b+BggG5v7CalfLJODAKVpLb9iUrKujS\nYXjNH8CrAYfkGoi/DCUnBh0r70RKx0HpuLZpvOQkqH8F4lOgaCSUntY2/UhGVNCl44juBhQBccBz\nP52QY+6NqV9CjYuxsrOwgkFBR8qaWSHW/c6gY0gTVNClw7CyC/HEcmiYASWnYkUjgo60Q171M6h5\nDKjD656CipewSHnQsSTEVNClwzArwspvDDpGy9W/AdSlnzgkFkPkc0EmkpDTKheRtlJyClCS+rJy\nKNgr6EQSchqhi+SQJzeBFWIWI1J2Ll4wGJIfQ+wLmBUHHU9CTgVdJEeSVXdC9X1AFHr8EosdjcUO\nCTqWdCKachHJAU9ugOr7gUYgjlf+KOd9JKvuIrlyP5Krj8Ubl+a8fen4VNBFcqIQ2Or+q1aW09a9\n4T2ongheC4mP8Kqbctq+hIMKukgOWKQMym+HSG+IDsLKc71Wu4FPfmE46NIHsh2aQxfJkUjJsVBy\nbNs0XjAESsZB7eMQ6Yl1u7Zt+pEOTQVdpAMwM6z8R3i364AoZtbsPtL5qKCLdCBm+i8rTdMcuohI\nSGRV0M3su2Y2z8zmmtmjpjMnREQCk3FBN7PdgEuBEe4+DIgCZ+QqmIiItE62Uy4FQImlJvZKgY+z\njyQiIpnIuKC7+zLgdmAxsBzY6O7Pb7udmU0ws2lmNm316tWZJxURkR3KZsqlBzAOGAjsCpSZ2dnb\nbufu97r7CHcfUVFRkXlSERHZoWymXMYAC919tbs3AH8BDstNLBERaa1sCvpiYKSZlVrqLIdjgAW5\niSUSPu4edAQJuWzm0F8HngBmAHPSbd2bo1ySA57ciDcuUSEJmLuT3PC/+MohJFePTd1GT6QNZLXK\nxd2vd/d93H2Yu5/j7vFcBZPseHwqvuoIfM3x+IZLVNSDVP8GxJ8DkpBYglfdFXQiCSmdKRpSXnU7\nqftZxiE+FRILg47UeX3quiv2qavsiuSSCnpYRXqTOtcLwMG6BZmmcys8CIpPBAohOgDr8t2gE0lI\nqaCHlJX/FIpGQnQPKL8Di/YKOlIoeHwKXv0QnljR4n3MjEj5TUT6zCNS8SwW7dOGCaUz06XbQsqi\nFVjP/ws6Rqgkqx+GTbeBJ2DTPVAxCYuUBx1LZAuN0EVaqu7Z1C3gqAcaofGdFu3myZrUQerGD9s0\nnogKuoROsvpBkiv3J7nqC3jj+7lrOHYkUMKW/zYFg5rdxb0WXzsO33AZvmY8Xjc5d3lEtqEpFwkV\nT6yEqtuAevAafOO12E6P56RtK5sA0T5440KsZBwW6dn8TvWzIbkGvDqVr/oBrPiYnOQR2ZYKuoRM\nkk+tC/SGnLVsZlAyrnWrDgv6gifTT2JQODRneUS2pSkXCRWL7gJlFwBRsG5Y+Y0B59kN6/FbiI2G\n0vOwrlcEmkfCTSN0CZ1I1//Bu1xKvtxM2WKHYrFDg44hnYAKuoSSbqYsnZGmXEREQkIFXUQkJFTQ\nRURCQgVdRCQkVNBFREJCBV1EJCRU0EVEQkIFPcc+nL2IBa+/p1u+iUi709kXOfTQjY/z+G1PYWaM\nGn8wV//h0qAjiUgnktUI3cy6m9kTZva2mS0ws059fvNf7von8Zp66qrjvPTYf6ivqw86koh0ItlO\nudwN/Mvd9wH2AxZkH6nj6rNHbyLRCGbQtWcXCmOFQUcSkU4k4ykXMysHjgTOB3D3elK3cum0bvr7\n1fzmuw9QV1PPhJ+dnRcXhhKRzsMyPXhnZvsD9wLzSY3OpwOXuaev5P/JdhOACQD9+/c/cNGiRVkF\nFhHpbMxsuruPaG67bKZcCoDPA79x9wOAauDqbTdy93vdfYS7j6ioqMiiu/yRTCab30hEpJ1lU9CX\nAkvd/fX08ydIFfjQqq6s4dsHX82xhWdw+RE/JF4bDzqSiMgWGRd0d18BLDGzwemXjiE1/RJaz/x+\nMgvnLMLdeX/Ghzz/4JSgI4mIbJHtOvRLgEfMrAj4ELgg+0j5KxqNfHKg0yBaEA02kIjIVrJatuju\nM9Pz48Pdfby7r89VsHx0/IQxDD1sMIWxQvYfPYwx5xzZov1WfLSKOf9eQH08dzcsFhHZls4UbYXi\n0hi3vXB9q/Z5/Z/T+fHpdxKJRugzoDf3vHELRVqfLiJtQNdyaYa7Z3XG5+O3/Z14TT21VXWs+GgV\n7077IIfpREQ+oYK+A8sXruTMfhdxUpezuW78rSQSiVa3sfu+fSkqTo3Ik4kkvfvtlOuYIiKAplx2\n6MHrH2fdig140pn54lxmvjiXA8fu16o2Jtx2LmbGogVLOf3KcfTuH461+CKSf1TQdyBWUkQkEiGR\nTODuFBUXtXjfua8s4L9/f5Nho4ZwyT1fb8OUIiIpmnLZgQtuOoO9D9yD0m4lnDBhLMMO36dF+70/\ncyFXH3sTf779aW4+6y5e+evrze/UjH/87nmOL/0qp/S6gLmvdOproIlIEzRC34HuFeX84tWbW73f\nO2+8v+VxvKaeWVPmc/iXD8k4R211Hb+67P9orG+koa6BW8+7hz988KuM2xORcNIIvQ3sN3oYZhGK\niguJlRQxatxBOW2/at0m3nxuZk7bFJGOTwW9Bdav2sisKfOorqxp0fZ999qF30y/lYvvOI87pvyI\n/UcPy6r/krJivvOLCykoSn2gqt5Yw41fuY0ZL8zOql0RCRdNuTRj4ZxFXHb4DzAzYiUxfjfrdnr0\nLm92v75770rfvXfNWY4TJoylprKWid//I40NCeprG5j333f4/JjhOetjW+4N0DAHIhVYQb8260dE\nckMj9GY8O/FFaqvqqKmspbaqlteentai/Ra/vYxnfv8CH81bkrMsnx87nGhhAUXFhRQVFzLiS/vn\nrO1tuTfga8/A11+IrzkBr5vUZn2JSG5ohN6MfoN3I1YaI14TB4Nd9+zT7D4L5yzi0sO+T+rmIcYd\nL9/I4BF7trjPRCLBL799H2888xYHHXcAl/7660SjUfbcbwC/fO1mZk+Zz76jBjNo/4FZ/M2a0fgO\nJD4AT00z+abfYsVj264/EclaaAp6Y0Mj05+fRXGXYoYfOTRnt387YcIY1q/cwMwX5zLmnCPZ7+h9\nm93nzX/NpKG+kURDAosYbz77VqsK+gt/mMoLD/+beE2cyY/8m6GH7s2Xzh8NwMBh/Rk4rH/Gf58W\ni/QG33wjjyIoaHl+EQlGKAq6u3P1sTfx7psf4O6M/85xfO2nZ+Wk7UgkwrnXn8a515/W4n32OWQv\nCgqjJBoSFBUXMmTkXq3qs3LtJhKNqcsMJBoTVK7d1Kr9c8GivaHHb/BNv4GC/ljXa9s9g4i0Tsb3\nFM3EiBEjfNq0ls1Bt0bluipO3+UbNDakimB5r648sWpizvtpjTefm8kbz77FiLHDOeSEA1u178Y1\nlXz74KtZv3IjPXqX86s3b6G8V7dPbVO1fhMFRQWUlBXnMraI5KGW3lM0FCP0svJSuvTowsbVlUQL\no+yx34CgI3HQl/bnoAwPWpb36saD7/6StcvXs9MuPT5zI437r32EJ+54GotGuObhyzjilMxPWhKR\n8AjFCB3g4w9W8MefPElZeRnnXH8qXbqXtUk/7SleG6eouOhTxwOqN1bz/3p/bcunkYq+O/HHxb8N\nKqKItINONUKH1OqT7038dtAxciKZTPKjU+/g1afepHvvcn7+7x9vWV1TUFRAJBqBdEEvKy8NMqqI\n5BGtQ2+Cu3P3N+/l5G7ncPkRP6BqffsdmJz18jymT5pNMumsX7WRB3742Jb3YiUxfvj4Fey8ewUD\nh+/OD/703XbLJSL5LTQj9NZYt2I9C+csZtABAz9zsHGzac/P4oVH/k3dpjreefMDHrnpSS6+47yc\nZ/lw9iLefv09hh81dMuZpbGSIkhPhZlBrDT2qX1GnnggI09s3YFWEQm/rAu6mUWBacAydz8x+0i5\nVbuplumTZlPRrxeDR+zJogVLuXTktWAQiUb43Vu3bfemEw3xBjbPXCcTyaxuQ9eU+a++w1Vjfwyk\nCvc9b9zC7kP6MmTk3ow+83CevX8yyYSzZtlaEokE0Wi0mRZFpDPLxZTLZUBeXqC7Pt7ANw/8X352\n3j1ccfT1PDtxMi89+gq1m1Kn8sdr6nnlr29sd9+DjzuAfUcNxiJGr749OfOaU3Ke79V/TCdeEyde\nEyfRmGTGpNTFtsyMeE09lv6VMu8/7zDrpXk5719EwiWrEbqZ9QVOAH4C/E9OEuXQwjmLWbdiPbWb\n6gB4+tfP8/+uOIlYaRF11XGiBRH67bPbdvctKCzgp8/+gPq6egpjhTk783Rr+x46mFhpEfGaeiIR\nY/DBg7a8V1wWIxKNkGhM3S1p22kXEZFtZTvlchdwFdC1qQ3MbAIwAaB//3Y4ZX0rfQZUQHpVZlFJ\nIfscMojRZ4xi/coNvPnsWxx56qHNrhVv6W3nZk+dz7P3TWavA/dg/CXHEYk0/+Fn5IkHcu0jl/PW\nS3M59MQDGTpy7y3vXXDTGSycs5jFC5ZywoQxDD107x20JCKSxTp0MzsRON7dv2VmRwPfa24OvS3X\noTfl3ekf8Je7/smue+7MGdecQlGsMOd9LHt/ORftfyXxmjix0hjn3nAqp31vXM772ZHKtVWsXrqW\n3Yf2paCwUx7rFgmt9liHPgo42cyOB4qBbmb2sLufnUWbObf3gXty9R8uzWmbyWSS2k11lHYtwcxY\nNH8p0WhqRB6viTP/1Xdz2l9z3n7jPa4c8yMg9ankntd/SqxEUzQinU3GB0Xd/Rp37+vuA4AzgBfz\nrZi3heULV3Jm34v4Sq8LuObYm0g0Jhh2+D4UlRRS0rWYWGkRx17whXbN9PhtT1G3qY66TXWsXLSa\nmTqAKtIp6bN5Kz1y05OsX7URTzrzX32Xac/N5JATDuS+uT/nrclz6D+kL3sM373Zdt5/ayH/mjiZ\nfkP6ctLFX2zRnHtTdhm4M0XFhdTXNZBMJOm1W8+M2xKRjisnBd3dXwZezkVb+a6kSzHRgiiN9Y24\nO8Xpqx2W9+rG0aePalEbaz5ex3ePvI666jpipUVUrqninOtOzTjTOTecxsa1Vbw340O+fOnx7JkH\nFycTkfanEXornXvDaXw4exEfzl7EcV/7AsOPGtrqNpa8vYxINLUMMl5Tz6yX52VV0ItLY3zv/m9l\nvL+IhEPeF/RkMsmkh6aw7L3ljD33KPoN3v668fbStUcX7njpxibfd3f++JMnefGx/7D/6H351s8v\n+Mzlb/c+cA9iJTGSiSTu8KULRrd1bBHpBPK+oP/xJ0/y2K1PUV9bz1O/+hcPvX9Pk9dfyQdvPDOD\nx279G3XVcVYuXEX/fXZj3LeP+9Q2ZeVl3Dv7dt545i12HdSHYaP2AWDVkjVMemgKvXbrydhzj8pq\nXl1EOp+8L+jTnp+VukFz2pK3l1F+eP4W9LXLN7B5bX99vIFVi9dsd7vuFeV88byjtzyvra7jWyP+\nl03rN1FQVMjiBUv5xq3ntEdkEQmJvB8CHn36KIrLYhQVF1IYK2Tg59r3bNPWOvyUg+leUU5Jl2K6\ndC/jxIu+2KL9ln+wkvq6ehKNSeI1cd545q02TioiYZP3I/Tx3zmO3Qb14eMPVnL4KYdQVp7fdyLq\n1rMrE9++m4/fX8HOu/eipEtJi/bbdVAfSsqKaaxvJFoQZdQpB7dxUhEJm9Dcgq4trV66llvO+QXr\nlm/gG7eezWHjDmqTftatWM/Lf/ovvXbryRFfGdkmFwQTkY6npaf+q6C3wJXH3MDsqQtIJpIUFRfy\n6JLf0W2nJq9HJiKSUy0t6Hk/h54P1q3YSDKR3PK8urImwDQiItungt4CE352NrGSIoqKCxn15UPo\nM6A3rz8zg2987n+4auyPWLNsbdARRUQ05dJSVes3UVtVS0W/XmzaUM2ZfS8iXltPJBph+JFDuG3y\nDUFHFJGQao/L53YqXXt0oWuPLgDUVNZuWWueTCRZ8/H6nPRRtX4Trz09nd79e7Hf0fvmpE0R6TxU\n0DPQu38vDht3EP/9+zRw52s3fzXrNuO1cS4+4Eoq11YBcOHNX+XLlxyfdbsi0nmooGfAzLj2j5ez\n/MOVlJWX5uRSBIvmL6VqfTV11amzYp9/4GUVdBFpFR0UbYWNayp57R/TWbloNWbGrnv22W4xX/D6\ne9z9zXt5+rfPkUwmt9PSZ+2yx85sXnYeKyli+JFDchldRDoBjdBbaO3y9UwYfgWNDY0kE0luf/EG\nBh806DPbrfhoFVeNuZG66tT9Reuq45x6xcnNtt+1Rxd+8d+f8PRvJ7HLwArGa3QuIq2kEXoLvfns\nW9TX1lNTWUtddZxJD7283e0WzVtCZKv7i86eMr/Ffew+tB/f+cWFfOW7J33mkrub1ccb+Oe9k/j7\nr5+jtrqu1X8PEQkvjdBbqP/Qvmxe4BkrjbHH8AHb3W7IyL0pKCqguCyGO4w996ic5rjxK7cx6+V5\nuMMLD0/lF//9SU7bF5GOSwW9hYaO3JsrJ36LFx6eyvCjh3Lc14/Z7nbddurKfXPu5M1/zaT/kN3Y\n5+C9cppj5otzqa9rAODt198jkUgQjW5/NC8inYsKeiscddphHHXaYc1u12Pn7p+61nkufe6IIcx5\n5W086Qz8XH8VcxHZIuOCbmb9gIeAnQEH7nX3u3MVTLbvxr9dxbP3vUiiMdHkpwQR6ZyyGaE3Ale4\n+wwz6wpMN7NJ7t7yo4DSarGSGOMvOa75DUWk08l4lYu7L3f3GenHVcACINg7OEvWZkyew5n9LuKr\nu1/M7Kn63SzSkeTk4lxmNgCYCgxz98pt3psATADo37//gYsWLcq6P2kb7s74HudRU1kLpA7wPrl6\nYsCpRKRsGM+oAAAGOElEQVTdroduZl2AJ4HLty3mAO5+r7uPcPcRFRUV2XYXatWVNTx04+P8/uqH\nWb9yQyAZGuINWx7X19UHkkFEMpNVQTezQlLF/BF3/0tuInVe14//GY/99K88eefTXH7ED2nPSxtD\n6ho13/7FhRQUFVBQVMAlv/p6u/YvItnJZpWLAfcDC9z9ztxF6rzenfYBDfWNACz/cCUN9Y0UxQrb\nNcMJ3xjL2HOOArN271tEspPNCH0UcA7wBTObmf7SBUiyMOqUQyguK6a4LMbnjhgSWEEtKi5SMRfp\ngDIeobv7K4BuS59D37v/mxx28kE0xBs4/JRDgo4jIh2MzhTNI9FolCNUyEUkQ7raoohISKigi4iE\nhAq6iEhIqKCLiISECrqISEiooIuIhIQKuohISKigi4iEhAp6nkskEix5ZxnVlTVBRxGRPKczRfNY\nfbyByw//AYsXLCMajXD7Szew1+f3CDqWiOQpjdDz2MwX57L0nY+J18Spqarl0Z/+NehIIpLHVNDz\nWM8+3UkmkwAUxgqp6LtTwIlEJJ+poOexQQcMZMJt59J38K6MGn8w5//49KAjiUgey8k9RVtqxIgR\nPm3atHbrT0QkDNrtnqIiIpIfVNBFREJCBV1EJCRU0EVEQkIFXUQkJFTQRURCQgVdRCQk2nUdupmt\nBha1YpdewJo2ipNrHSkrdKy8yto2lLVttEXW3d29ormN2rWgt5aZTWvJYvp80JGyQsfKq6xtQ1nb\nRpBZNeUiIhISKugiIiGR7wX93qADtEJHygodK6+ytg1lbRuBZc3rOXQREWm5fB+hi4hIC6mgi4iE\nRF4WdDPrZ2Yvmdl8M5tnZpcFnak5ZhY1s7fM7B9BZ9kRM+tuZk+Y2dtmtsDMDg06U1PM7Lvpf/+5\nZvaomRUHnWkzM5toZqvMbO5Wr/U0s0lm9l76zx5BZtysiay3pX8GZpvZX82se5AZN9te1q3eu8LM\n3Mx6BZFte5rKa2aXpL+/88zsZ+2VJy8LOtAIXOHuQ4GRwLfNbGjAmZpzGbAg6BAtcDfwL3ffB9iP\nPM1sZrsBlwIj3H0YEAXOCDbVpzwAHLvNa1cDk919L2By+nk+eIDPZp0EDHP34cC7wDXtHaoJD/DZ\nrJhZP+CLwOL2DtSMB9gmr5mNBsYB+7n7vsDt7RUmLwu6uy939xnpx1Wkis5uwaZqmpn1BU4A7gs6\ny46YWTlwJHA/gLvXu/uGYFPtUAFQYmYFQCnwccB5tnD3qcC6bV4eBzyYfvwgML5dQzVhe1nd/Xl3\nb0w/fQ3o2+7BtqOJ7yvAz4GrgLxaxdFE3m8Ct7h7PL3NqvbKk5cFfWtmNgA4AHg92CQ7dBepH7Zk\n0EGaMRBYDfxfenroPjMrCzrU9rj7MlIjm8XAcmCjuz8fbKpm7ezuy9OPVwA7BxmmFS4Eng06RFPM\nbBywzN1nBZ2lhfYGjjCz181sipkd1F4d53VBN7MuwJPA5e5eGXSe7TGzE4FV7j496CwtUAB8HviN\nux8AVJM/0wKfkp5/Hkfql9CuQJmZnR1sqpbz1HrgvBpNbo+ZfZ/UFOcjQWfZHjMrBa4Frgs6SysU\nAD1JTRdfCTxuZtYeHedtQTezQlLF/BF3/0vQeXZgFHCymX0EPAZ8wcweDjZSk5YCS91986edJ0gV\n+Hw0Bljo7qvdvQH4C3BYwJmas9LMdgFI/9luH7UzYWbnAycCZ3n+npCyJ6lf6rPS/8f6AjPMrE+g\nqXZsKfAXT3mD1Cf3djmQm5cFPf3b7H5ggbvfGXSeHXH3a9y9r7sPIHXQ7kV3z8uRpLuvAJaY2eD0\nS8cA8wOMtCOLgZFmVpr+eTiGPD2Au5W/A+elH58HPBVglh0ys2NJTROe7O41QedpirvPcffe7j4g\n/X9sKfD59M9yvvobMBrAzPYGiminK0XmZUEnNeo9h9Rod2b66/igQ4XEJcAjZjYb2B+4OeA825X+\nFPEEMAOYQ+pnNW9O/zazR4FXgcFmttTMvgbcAow1s/dIfcK4JciMmzWR9R6gKzAp/f/rt4GGTGsi\na95qIu9EYI/0UsbHgPPa6xOQTv0XEQmJfB2hi4hIK6mgi4iEhAq6iEhIqKCLiISECrqISEiooIuI\nhIQKuohISPx/BpASGgrgEXQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f26b6e38cf8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# data generation\n",
    "np.random.seed(314)\n",
    "\n",
    "data_size1 = 100\n",
    "x1 = np.random.randn(data_size1, 2) + np.array([4,4])\n",
    "y1 = [0 for _ in range(data_size1)]\n",
    "\n",
    "data_size2 = 100\n",
    "x2 = np.random.randn(data_size2, 2)*2 + np.array([10,10])\n",
    "y2 = [1 for _ in range(data_size2)]\n",
    "\n",
    "\n",
    "# all sample data\n",
    "x = np.concatenate((x1, x2), axis=0)\n",
    "y = np.concatenate((y1, y2), axis=0)\n",
    "\n",
    "data_size_all = data_size1 + data_size2\n",
    "shuffled_index = np.random.permutation(data_size_all)\n",
    "x = x[shuffled_index]\n",
    "y = y[shuffled_index]\n",
    "\n",
    "# split train & test\n",
    "split_index = int(data_size_all*0.7)\n",
    "x_train = x[:split_index]\n",
    "y_train = y[:split_index]\n",
    "x_test = x[split_index:]\n",
    "y_test = y[split_index:]\n",
    "\n",
    "\n",
    "# plot data\n",
    "plt.scatter(x_train[:,0], x_train[:,1], c=y_train, marker='.')\n",
    "plt.title(\"train data\")\n",
    "plt.show()\n",
    "plt.scatter(x_test[:,0], x_test[:,1], c=y_test, marker='.')\n",
    "plt.title(\"test data\")\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 最简单的程序实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 0, 0, 0, 1, 0, 1, 0, 0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 0, 0, 1, 0, 1, 1, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 1, 0, 1, 0, 0, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 0, 1, 1, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 0]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import operator\n",
    "\n",
    "def knn_distance(v1, v2):\n",
    "    return np.sum(np.square(v1-v2))\n",
    "\n",
    "def knn_vote(ys):\n",
    "    vote_dict = {}\n",
    "    \n",
    "    maxv = 0\n",
    "    maxk = 0\n",
    "    for y in ys:\n",
    "        if y not in vote_dict.keys():\n",
    "            vote_dict[y] = 1\n",
    "        else:\n",
    "            vote_dict[y] += 1\n",
    "        if maxv < vote_dict[y]:\n",
    "            maxv = vote_dict[y]\n",
    "            maxk = y\n",
    "    return maxk\n",
    "\n",
    "    sorted_vote_dict = sorted(vote_dict.items(), \\\n",
    "                              #key=operator.itemgetter(1), \\\n",
    "                              key=lambda x:x[1], \\\n",
    "                              reverse=True)\n",
    "    return sorted_vote_dict[0][0]\n",
    "    \n",
    "def knn_predict(x, train_x, train_y, k=3):\n",
    "    dist_arr = [knn_distance(x, train_x[j]) for j in range(len(train_x))]\n",
    "    sorted_index = np.argsort(dist_arr)\n",
    "    top_k_index = sorted_index[:k]\n",
    "    ys=train_y[top_k_index]\n",
    "    return knn_vote(ys)\n",
    "    \n",
    "\n",
    "#a = knn_predict(x_train[0], x_train, y_train)\n",
    "\n",
    "y_train_est = [knn_predict(x_train[i], x_train, y_train) for i in range(len(x_train))]\n",
    "print(y_train_est)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Accuracy: 100.000000%\n"
     ]
    }
   ],
   "source": [
    "n_correct = 0\n",
    "for i in range(len(x_train)):\n",
    "    if y_train_est[i] == y_train[i]:\n",
    "        n_correct += 1\n",
    "accuracy = n_correct / len(x_train) * 100.0\n",
    "print(\"Train Accuracy: %f%%\" % accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Accuracy: 96.666667%\n"
     ]
    }
   ],
   "source": [
    "y_test_est = [knn_predict(x_test[i], x_train, y_train, 3) for i in range(len(x_test))]\n",
    "n_correct = 0\n",
    "for i in range(len(x_test)):\n",
    "    if y_test_est[i] == y_test[i]:\n",
    "        n_correct += 1\n",
    "accuracy = n_correct / len(x_test) * 100.0\n",
    "print(\"Test Accuracy: %f%%\" % accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. 通过类实现kNN程序"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import operator\n",
    "\n",
    "class KNN(object):\n",
    "\n",
    "    def __init__(self, k=3):\n",
    "        self.k = k\n",
    "\n",
    "    def fit(self, x, y):\n",
    "        self.x = x\n",
    "        self.y = y\n",
    "        return self\n",
    "\n",
    "    def _square_distance(self, v1, v2):\n",
    "        return np.sum(np.square(v1-v2))\n",
    "\n",
    "    def _vote(self, ys):\n",
    "        ys_unique = np.unique(ys)\n",
    "        vote_dict = {}\n",
    "        for y in ys:\n",
    "            if y not in vote_dict.keys():\n",
    "                vote_dict[y] = 1\n",
    "            else:\n",
    "                vote_dict[y] += 1\n",
    "        sorted_vote_dict = sorted(vote_dict.items(), key=operator.itemgetter(1), reverse=True)\n",
    "        return sorted_vote_dict[0][0]\n",
    "\n",
    "    def predict(self, x):\n",
    "        y_pred = []\n",
    "        for i in range(len(x)):\n",
    "            dist_arr = [self._square_distance(x[i], self.x[j]) for j in range(len(self.x))]\n",
    "            sorted_index = np.argsort(dist_arr)\n",
    "            top_k_index = sorted_index[:self.k]\n",
    "            y_pred.append(self._vote(ys=self.y[top_k_index]))\n",
    "        return np.array(y_pred)\n",
    "\n",
    "    def score(self, y_true=None, y_pred=None):\n",
    "        if y_true is None and y_pred is None:\n",
    "            y_pred = self.predict(self.x)\n",
    "            y_true = self.y\n",
    "        score = 0.0\n",
    "        for i in range(len(y_true)):\n",
    "            if y_true[i] == y_pred[i]:\n",
    "                score += 1\n",
    "        score /= len(y_true)\n",
    "        return score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train accuracy: 100.000000 %\n",
      "test accuracy: 96.666667 %\n"
     ]
    }
   ],
   "source": [
    "# data preprocessing\n",
    "#x_train = (x_train - np.min(x_train, axis=0)) / (np.max(x_train, axis=0) - np.min(x_train, axis=0))\n",
    "#x_test = (x_test - np.min(x_test, axis=0)) / (np.max(x_test, axis=0) - np.min(x_test, axis=0))\n",
    "\n",
    "# knn classifier\n",
    "clf = KNN(k=3)\n",
    "train_acc = clf.fit(x_train, y_train).score() * 100.0\n",
    "\n",
    "y_test_pred = clf.predict(x_test)\n",
    "test_acc = clf.score(y_test, y_test_pred) * 100.0\n",
    "\n",
    "print('train accuracy: %f %%' % train_acc)\n",
    "print('test accuracy: %f %%' % test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. sklearn program"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature dimensions:  (1797, 64)\n",
      "Label dimensions:    (1797,)\n"
     ]
    }
   ],
   "source": [
    "#% matplotlib inline\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn import datasets, neighbors, linear_model\n",
    "\n",
    "# load data\n",
    "digits = datasets.load_digits()\n",
    "X_digits = digits.data\n",
    "y_digits = digits.target\n",
    "\n",
    "print(\"Feature dimensions: \", X_digits.shape)\n",
    "print(\"Label dimensions:   \", y_digits.shape)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAABLCAYAAABQtG2+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFJBJREFUeJztnXmUFNW9xz+/noWBGRiBwWEXEEeERFGRqE+CyzNi8l5A\nzYlGE3OMSoSHL0bNZswJSYwm8SWQuBCIYtyiJu8EzTOoMVFQFJWJGhAFIovIzrDOvvXv/VE9XXXb\n6WGY7unqDr/POX3m3r7Vdb9z69avqn71u/eKqmIYhmHkDpGwBRiGYRiHhxluwzCMHMMMt2EYRo5h\nhtswDCPHMMNtGIaRY5jhNgzDyDHMcBuGYeQYWWG4RaSfiCwSkVoR+UBELg9BwywRqRSRRhH5babr\nD+joISL3x9qhWkTeFpELQ9LyiIjsEJGDIrJORK4JQ0dAz3Ei0iAij4RU/5JY/TWxz9owdMS0XCYi\n78XOmfUiMinD9dckfFpF5K5MaghoGSEii0VkX6y/3i0i+SHoOEFEXhCRAyLyvohc1F11ZYXhBu4B\nmoBy4ApgnoiMy7CGbcBtwMIM15tIPvAhMBkoBW4Ffi8iI0LQ8hNglKr2AT4L3CYip4ago417gBUh\n1g8wS1VLYp/jwxAgIucDPwWuAnoDnwQ2ZFJDoA1KgIFAPfCHTGoIcC+wGxgEjMc7d2ZmUkDsQvEU\n8DTQD5gOPCIiFd1RX+iGW0SKgUuA76lqjaouw2uAL2VSh6r+UVWfBPZkst52dNSq6mxV3aSqUVV9\nGtgIZNxgquo7qlrXlo19js20DvDuMIH9wN/CqD/L+AHwQ1V9LdZHtqrq1hD1XALsAl4Oqf6RwBOq\n2qCqO4BngUzf+I0BBgNzVLVVVV8AXqGb7FjohhuoAFpUdV3gu3+Q+YbPSkSkHK+NVodU/70iUges\nAbYDi0PQ0Af4IXBjputuhztEpEpEXhGRszNduYjkAROAAbHH8S0x10DPTGsJ8GXgIQ1v/oy5wKUi\n0ktEhgAX4hnvsBHgY92x42ww3CXAwYTvDuI9Ah7RiEgB8CjwoKquCUODqs7EOxaTgD8CjSHI+BFw\nv6puCaHuIN8CRgFDgAXA/4lIpp9AyoEC4HN4x2Q8cDKeSy3jiMgxeK6JB8OoP8ZLeAbyILAFqASe\nzLCGtXhPHd8QkQIR+RReu/TqjsqywXDXAH0SvisFqkPQkjWISAR4GM/3PytMLbFHv2XAUGBGJusW\nkfHAvwNzMllve6jq66paraqNqvog3qPwpzMsoz729y5V3a6qVcAvQtDRxpeAZaq6MYzKY+fJs3g3\nFcVAGdAX7x1AxlDVZmAa8BlgB3AT8Hu8C0nayQbDvQ7IF5HjAt+dREiugWxARAS4H+/u6pJYp8gG\n8sm8j/tsYASwWUR2ADcDl4jImxnW0R6K9zicuQpV9+EZg6BbIswpPq8k3LvtfsBw4O7YBXUP8AAh\nXMhUdaWqTlbV/qp6Ad7T2RvdUVfohltVa/Gulj8UkWIROQsvguHhTOoQkXwRKQLygDwRKQojpCjG\nPOAE4D9Vtf5QG3cHInJ0LOSsRETyROQC4Atk/uXgAryLxfjY59fAn4ELMilCRI4SkQva+oWIXIEX\nzRGGL/UB4PrYMeoLfB0vmiGjiMiZeG6jsKJJiD1xbASuix2Xo/B87iszrUVEToz1j14icjNelMtv\nu6UyVQ39g3fVfBKoBTYDl4egYTZ+5ETbZ3YIOo6J1d2A50Zq+1yRYR0DgKV4kRwHgVXAtVnQV2YD\nj4RQ7wC8UMTqWJu8BpwfUhsU4IXA7cd7LP8VUBSCjvnAw1nQJ8YDS4B9QBWei6I8BB13xjTUAM8A\no7urLolVaBiGYeQIobtKDMMwjMPDDLdhGEaOYYbbMAwjx+iU4RaRKSKyNjZS69vdLcp0mA7TYTr+\nVXWkg0O+nIwNsV0HnI8XP7oC+IKqvpvsN4XSQ4sobrespcz9fuDAvfH01tqjnLKiLX74sqpS07KX\nXpQgRKijmiKKySOPBmpp0saPxNN2pOMj247xr2E9Ii1O2f6d/iBOVaVh7/Zu0xE9yt9uxLCdTtmO\nZn+ckqqyb+3+tOloGuJ+/7H+u+PpvdE8p2zPWn/b7j4uku9HZEZHufcZsq7J14FSy8G06Qj2B4Da\n5sJ4umB9Q1K96dbRka7Eflr9rl+Wbh1Ng93vNdAlynq7Y+UG5fvto6qsWtvEiJH55OfD2tVKz7w+\n5Ek+9a3VNEXrD0tH4wh3IOKwEt9+fHigv1NWtN0f5Kuq1LSmr59qRaGTDx6LpjXRdn9zKJLpaI/O\nxClPBN5X1Q0AIvI4MBVIariLKOYTcl67ZVWXnOHkv3HT4/H09/4+1SmruHF7PL2vaQf/3LOMU2Kz\nV26MjQAfKWN4XdsPLe5IRyKDH/SN83G9djlli+acG0/X7NpE1dO/7zYdded+Ip6+f+4vnLI7tk+J\np3ev2sXr11SmTcfG693j8saX58XTj1f3dcoenjwxnu7u45JXdnQ8XX+vOx1H4fkfxNP7dQ8beDdt\nOoL9AeCNrcPj6aGXJB8blm4dHelK7KdLT/TbJ906Nn/1TCffVOobp6vPe9Epu6XMn+12eWU937pz\nN/c94hnVz4712nFUyaksr2o//LsjHeu+P8HJ/2ySbz9uevqLTtnxP/EnTNzftIN/7n0lbe3RdO8x\nTn5Eb/8Csu30rg36TqajPTrjKhmCN81oG1ti3zmIyHTx5rOubO6G6SwaW2sowu+YRfSkkY+OTelu\nHc21B7JCR92uuqzQkS3HpZF605GFOrbuaGXgYP/2vCivhIZobcZ1NERrs6I90kXaXk6q6gJVnaCq\nEwroka7dmg7TYTpMxxGn41B0xlWyFRgWyA+Nfdclgq4RgMt674un5x5V45T9+c3n4unllfVcOKOY\nqs94j/QN89fQg/TNZLmpul88/cBwd1rh30zyFxdpHFRM84v+lbqB+pR0RCef7ORfvmd+PL0uYYaS\nqf3fiqfXjq5lFanpWDfPd3ncca57XD72S38e+ne+dq9TdtekEfF0cxU0vPB2Sjo6YuOM0fF00zuu\n73A0vqukBz1pSLE9ggTbGhL6xDZ32ydrS+LptW/W8j+fT5+OvVe5LqznhvsurGOfuM4pG81r8XS6\n2yORwgP+Pd8z3z/bKXt+5ph4+mD9NnZuXMod270ZCuoOrKAQaK3fharro+8MZ49NvujQz//DXRjp\nqTP8cyuyKp8Pr0mtPfLG+etmvDjuieQbJvSP26vc9TaCLq2u0pk77hXAcSIyUkQKgcuAP6Vc82Fy\n2vgiGg9U0XhwD9HWFnbyIQMYlGkZFI4cSj011GstUY2GpmP0ib2yQkdJv2FZoaMPfbNCR7Ycl2xp\nj97HD6T6w4NUb6umtbk1NB39TyjLivZIF4e841bVFhGZBTyHNwHTQlXN+Mx9+fnC0LMuZsPiBagq\nwxhKiZRmWgaSl8fxjOctXkZRBjMiFB15+ZIVOiSSHe0RkQjHa/g6suW4ZEt7SF6EiTefzl//+y9o\nVClneDjtkR/JiuOSLjo1+52qLiaFlU9azvVX3bqs99tO2YVTLounS1e6awV8fpn7Rrdl6rGUT/0m\nACNnpDZbYqKLYn7F3YGcGwLUZ5Ub+lMmgyhL09V6wzTXjxZ8rLr/b+c4Zesv/bWTnyejU9IxZp6/\nfsXDP5jolN269LF4OjGqpOQPr7v5NLZHXvnRTv5LF/tv2p94wO0PwUdXgHKOp5yzAWhdndo6vu/W\nu+/fpxX7+1vX7L5c++7KK5z8MQN3U85Jno6dbuTH4XLR119IWjbqyY5fnqWznw6f/WrSsvfnnO7k\nry53z+NlP6vgTLylF1sltfZY8q57zN8oTR7tc9cH7sSNV198I+OZBkCvRW4f7gzNZcnXRLhqs+9O\nDUYgAfz4xKec/FJGkyo2ctIwDCPHMMNtGIaRY5jhNgzDyDEyssJLQ3+/mlt3fdwpi65MvgbuilXp\nXSVr82x/9NdTV93plFUUJB9yPOQve5x8axo1BUd3ATyx2ffjPnODq/Gc1Zc7+cJAOFxXcNr+xDFO\nWTBM8/MbXN9y/kC327TscIfmp0Iw/A9gbumieHrpHDeM6r2F7ii6yAFf1+ivp6bj+Z1uewRHAyb2\nlegq9yVX6870vbsf29ONvA2+A4ksfStx87RSd5E/infbJ5OPxH7m4p93uJ8nLvf7z8A5qfm4Rz/o\nnn3PP/ZoPH3Va5Ocsnebyp1873X74+munMMFa5JHQe+c6vfNiU9tdsrGFiaeH+bjNgzDOOIww20Y\nhpFjZMZV0te/Pjy63B0JVtHBIsj5pU1OvuVAYZItO0cwpOmGeRc5ZYvf+kvS3yWGAaV6tQuGvK39\n9iin7Orzkk800/OL7twK6XTZJLqsPnOKvxbvyc8mDAVLWB73rSmD4+muuE2CowPfm+6O0hy3fHo8\nPRTXBbFxyn1O/qQ7Z5IughNYAUy66KvxdNVJ7myJiZpPwNfRURhdZ0h8zH5qjx/Gunm263Yc+YcE\nl16KIZFB18Lwme6MiPMrfpf0d1ffcKOTH7gotTYI0tAvuQ1IHPH86fMvdfKptkcwtDNxNGTQfox8\n9hqn7DuD3BMmGMbaVU12x20YhpFjmOE2DMPIMTrlKhGRTUA13tN5i6pO6PgX3cPWW+4gUtQDIsI+\nbej0HMbpZpkuJo98BEGIHPE6lux6iHwpQBDQaGg61t73IyIFPZBIhA+0PjQdG+b8iEihp2N7iDqy\npX+YjvRzOD7uc1S1qiuVFO3zZ3U77ePrnbIDQTED3fCdS8f+3cnfGYUhV88gr7iEkd9Z3hUpXWLX\nKQmzeS2BU5lMoXRt2sf37vCHxG6c8uuk25323ZudfL+dH/2fU9HREUFfddCHDbBnobvIQHPpEvrf\ncj15JcVUdGEqgqL9fv9IHE6++gw/3Ov2la5fMZG8mhbOKJtGYaRnykOrEwkOkS7jEx1sCZqnDLj1\nWvJ6F1PxlcqU6v3fA6c4+aAf9/aL3f/xlumuv7R4RCEnnzqTwsLiLoUOBv2vhee7ZRXb/JDI0747\nwynrtyi9/TQ4PUVw9kxwZ0gsGu4uYHDFY27bLz2lgInHfoXC/F4p+7sTZ/h7cfJV8XTFUrfeCxZ+\nzcmPmOuvLpXYrp3FXCWGYRg5RmfvuBX4q4i0AvNVdUHiBiIyHZgOUETyyVhSQoRt982HSIQCPZqh\nMqqdTTKgA3iTlxAVhjDKdAjsmrsAIhF6aXloOkSgcu+fECIM1mEhtoew6877Qm8PRHj7HwsRiTBE\nBx7x/VSAyk2PxvrH4FDbI1U6a7jPUtWtInI08LyIrFHVl4IbxIz5AoA+0q/jFYi7yJDrZpFfWkpL\nTTVbbptDsfamrwxwtsmEjgmcQ5H0pEkbeJOXj3gd5TfPJL9vKa0Ha9jyzV+GpmNiv4soyiuhsbWO\nyt2LwmuPW67z2+Nrd4Wm49STr6VHj1Kammp4+9V7j/h+OnHUlRQV9KGxpZbKNQ+EpiMddHZa162x\nv7tEZBHeAsIvdfwrnz5rfU/294c+7ZRdOd2P+SyYtpuOOO4n/vrEEQZzkL30ZUAHv+geisTzbxVK\nEQP08HUEh+3ePsH12waHVq/48Tyn7JzL3cWU6343mLpYesDC9Sm1R3A1HIDBL/hDnINx+AAPjXUX\nMZ62fwbQRH5pIQO6cFyC/uPrF/2bUxb0b97z0N1OWTDGG2Bo1WpaqSMfuqQjSOLKM0E//OhvJV0n\nG4ARy9pWUyple4o6Hv6j+wIt6MdOHJb/udI3nfzWS9vGC/RgwKup6ViXML3AuuZX4ukBi933Vonj\nC1I9X4JDzRPfgQSnjGge407Fe8tjrh/7/hlt0yT3ZcAN6bUfwXcIiW313Hm/dPLBOPeuTltxSB+3\niBSLSO+2NPAp4J0u1ZYCTXUttKi3llertrCXnRST+YnQo82NWaGjtb6Z1mZvUERrc2NoOurqokTr\nvXmhow1N4R2XhqasOC61dVH/uLSE1x51dVGiDZ6OaGN4/aNVs+O8rc2S9kgXnbnjLgcWiUjb9r9T\n1Wc7/kn6qd3bSCWvg4KiDGQYZTIw0zJoqa2hkiWh62jaV8u6p707UNUoQxgUio69VVG2zf6Np6M1\nyvCQdLTsr82K47JzdyurXvJGUmo0yuCQ2mPP7ijbf3WPl4mGd1waaWAly7PiuGRDe6SLzixdtgFi\ny3l0keBw6kvn3eSU3XqTv9LK3PXuY+GK8cGhxX04XboYO9MOiSuTnLPad0O8OM5dsaLlLN/VEyGf\n0x9ITUfwsaqjsKKWW/e6ZUFd42DkL/wwo5Ephp0V7HeHcV9/2+NJtoRpr7rhX5/8MLCqUfJJ5LpE\nQVVdPJ04K1+/R0oCuRJGpbF/VE1yV2pOHF4fZNxydwWcMw8Ghuan2B4j573v5of7w6kTH8G/us6d\nPXJShb8AdmRnajMJXjvBHU5+xWw/VLW9MNU2ekkJp5PacQmeq4n/44tv+edEohslcTbNc6P+lBGp\nhosmukOCixhP7uW21X9dOcvJ91p6+KvvJGLhgIZhGDmGGW7DMIwcwwy3YRhGjiGq6Q9VFJHdQC3Q\npSHyCZR1Yj/HqOpH4npMR1br+KCT+zAdpuNfQUdntLSro11UtVs+QGU27Md0ZKcO24ft40jaRzr3\no6rmKjEMw8g1zHAbhmHkGN1puD8yEVVI+zEd6f19Ovdj+7B9HCn7SOd+uuflpGEYhtF9mKvEMAwj\nxzDDbRiGkWN0i+EWkSkislZE3heRb6ewn00iskpE3haRw56Mw3SYDtNhOnJdR7ukK64wEKuYB6wH\nRgGFwD+AsV3c1yagzHSYDtNhOo5EHck+3XHHPRF4X1U3qGoT8Dgw9RC/6Q5Mh+kwHaYj13W0S3cY\n7iHAh4H8lth3XaFtrcu/x9aCMx2mw3SYjiNJR7t0ds3JsDjkWpemw3SYDtNxpOnojjvurcCwQH5o\n7LvDRgNrXQJta12aDtNhOkzHkaIj6U7T+sG7i98AjMR36o/rwn6Kgd6B9KvAFNNhOkyH6ThSdCT7\npN1VoqotIjILeA7vzexCVV19iJ+1R0prXZoO02E6TEeu60iGDXk3DMPIMWzkpGEYRo5hhtswDCPH\nMMNtGIaRY5jhNgzDyDHMcBuGYeQYZrgNwzByDDPchmEYOcb/AzEEEMUJp2tjAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f26b6f12080>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot sample images\n",
    "nplot = 10\n",
    "fig, axes = plt.subplots(nrows=1, ncols=nplot)\n",
    "\n",
    "for i in range(nplot):\n",
    "    img = X_digits[i].reshape(8, 8)\n",
    "    axes[i].imshow(img)\n",
    "    axes[i].set_title(y_digits[i])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# split train / test data\n",
    "n_samples = len(X_digits)\n",
    "n_train = int(0.4 * n_samples)\n",
    "\n",
    "X_train = X_digits[:n_train]\n",
    "y_train = y_digits[:n_train]\n",
    "X_test = X_digits[n_train:]\n",
    "y_test = y_digits[n_train:]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "KNN score: 0.953661\n",
      "LogisticRegression score: 0.908248\n"
     ]
    }
   ],
   "source": [
    "# do KNN classification\n",
    "knn = neighbors.KNeighborsClassifier()\n",
    "logistic = linear_model.LogisticRegression()\n",
    "\n",
    "print('KNN score: %f' % knn.fit(X_train, y_train).score(X_test, y_test))\n",
    "print('LogisticRegression score: %f' % logistic.fit(X_train, y_train).score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. 深入思考\n",
    "\n",
    "* 如果输入的数据非常多，怎么快速进行距离计算？\n",
    "    - [kd-tree](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KDTree.html#sklearn.neighbors.KDTree) \n",
    "    - Fast Library for Approximate Nearest Neighbors （FLANN）\n",
    "    - [PyNNDescent for fast Approximate Nearest Neighbors](https://pynndescent.readthedocs.io/en/latest/)\n",
    "* 如何选择最好的`k`？\n",
    "    - https://zhuanlan.zhihu.com/p/143092725\n",
    "* kNN存在的问题？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 参考资料\n",
    "* [Digits Classification Exercise](http://scikit-learn.org/stable/auto_examples/exercises/plot_digits_classification_exercise.html)\n",
    "* [knn算法的原理与实现](https://zhuanlan.zhihu.com/p/36549000)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}