{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 感知机\n",
    "\n",
    "感知机（Perceptron）是二分类的线性分类模型，输入为实例的特征向量，输出为实例的类别（取`+1`或`-1`）。感知机对应于输入空间中将实例划分为两类的分离超平面，感知机旨在求出该超平面。为求得超平面导入了基于误分类的损失函数，利用梯度下降法对损失函数进行最优化。感知机的学习算法具有简单而易于实现的优点，感知机预测是用学习得到的感知机模型对新的实例进行预测的，因此属于判别模型。\n",
    "\n",
    "![perceptron](images/perceptron.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 生物学、心理学解释\n",
    "\n",
    "心理学家唐纳德·赫布（Donald Olding Hebb）于1949年提出`赫布理论`，该理论能够解释学习的过程中脑中的神经元所发生的变化。赫布理论描述了突触可塑性的基本原理，即突触前神经元向突触后神经元的持续重复的刺激，可以导致突触传递效能的增加：\n",
    "> 当细胞A的轴突足以接近以激发细胞B，并反复持续地对细胞B放电，一些生长过程或代谢变化将发生在某一个或这两个细胞内，以致A作为对B放电的细胞中的一个，导致突触传递效能的增加。\n",
    "\n",
    "![neuron_cell](images/neuron_cell_cn.png)\n",
    "\n",
    "\n",
    "心理学家弗兰克·罗森布拉特(Frank Rosenblatt)于1957年提出了`感知机`，它作为简化的数学模型解释大脑神经元如何工作：它取一组二进制输入值（附近的神经元），将每个输入值乘以一个连续值权重（每个附近神经元的突触强度），并设立一个阈值，如果这些加权输入值的和超过这个阈值，就输出1，否则输出0，这样的工作原理就相当于神经元是否放电。\n",
    "\n",
    "\n",
    "感知机并没有完全遵循赫布理论，**但通过调输入值的权重，可以有一个非常简单直观的学习方案：给定一个有输入输出实例的训练集，感知机应该「学习」一个函数：对每个例子，若感知机的输出值比实例低太多，则增加它的权重；否则若设比实例高太多，则减少它的权重。**\n",
    "\n",
    "\n",
    "感知机模仿的是生物神经系统内的神经元，它能够接受来自多个源的信号输入，然后将信号转化为便于传播的信号在进行输出(在生物体内表现为电信号)。\n",
    "\n",
    "![neuron](images/neuron.png)\n",
    "\n",
    "* dendrites - 树突\n",
    "* nucleus - 细胞核\n",
    "* axon - 轴突\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 感知机模型\n",
    "\n",
    "假设输入空间(特征向量)为$\\mathbf{X} \\subseteq \\mathbb{R}^D$，输出空间为$\\mathbf{Y} \\in \\{-1, +1\\}$。输入$x \\in \\mathbf{X}$ 表示实例的特征向量，对应于输入空间的点；输出$y \\in \\mathbf{Y}$表示示例的类别。由输入空间到输出空间的函数为\n",
    "\n",
    "$$\n",
    "f(x) = sign(w x + b)\n",
    "$$\n",
    "\n",
    "称为感知机。其中，参数$w$叫做权值向量，$b$称为偏置。$w·x$表示$w$和$x$的内积。$sign$为符号函数，即\n",
    "![sign_function](images/sign.png)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 几何解释    \n",
    "感知机模型是`线性分类模型`，感知机模型的假设空间是定义在特征空间中的所有线性分类模型，即函数集合$\\{ f | f(x)=w·x+b\\}$。线性方程 $w·x+b=0$对应于特征空间$\\mathbb{R}^D$中的一个超平面$S$，其中$w$是超平面的法向量，$b$是超平面的截距。这个超平面把特征空间划分为两部分，位于两侧的点分别为正负两类。超平面$S$称为分离超平面，如下图：\n",
    "![perceptron_geometry_def](images/perceptron_geometry_def.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 感知机学习策略\n",
    "\n",
    "假设训练数据集是 **线性可分**，感知机学习的目标是求得一个能够将训练数据的正负实例点完全分开的分离超平面，即最终求得参数$w, b$。这需要一个学习策略，即定义（经验）损失函数并将损失函数最小化。\n",
    "\n",
    "损失函数的一个自然的选择是误分类的点的总数，但是这样得到的损失函数不是参数$w，b$的连续可导函数，不宜优化。损失函数的另一个选择是误分类点到分类面的距离之和。\n",
    "\n",
    "首先，对于任意一点$x$到超平面的距离（[参考资料](https://www.cnblogs.com/graphics/archive/2010/07/10/1774809.html)）为\n",
    "$$\n",
    "\\frac{1}{||w||} | w \\cdot x + b |\n",
    "$$\n",
    "\n",
    "其次，对于误分类点$(x_i,y_i)$来说 $-y_i(w \\cdot x_i + b) > 0$\n",
    "\n",
    "这样，假设超平面$S$的总的误分类点集合为$\\mathbf{M}$，那么所有误分类点到$S$的距离之和为\n",
    "$$\n",
    "-\\frac{1}{||w||} \\sum_{x_i \\in \\mathbf{M}} y_i (w \\cdot x_i + b)\n",
    "$$\n",
    "不考虑$1/||w||$，就得到了感知机学习的损失函数\n",
    "$$\n",
    "L = - \\sum_{x_i \\in \\mathbf{M}} y_i (w \\cdot x_i + b)\n",
    "$$\n",
    "\n",
    "### 3.1 经验风险函数\n",
    "\n",
    "给定数据集$\\mathbf{T} = \\{(x_1,y_1), (x_2, y_2), ... (x_N, y_N)\\}$，其中\n",
    "* $x_i \\in \\mathbb{R}^D$\n",
    "* $y_i \\in \\{-1, +1\\}，i=1,2...N$\n",
    "\n",
    "感知机$sign(w·x+b)$学习的损失函数定义为\n",
    "$$\n",
    "L(w, b) = - \\sum_{x_i \\in \\mathbf{M}} y_i (w \\cdot x_i + b)\n",
    "$$\n",
    "其中$\\mathbf{M}$为误分类点的集合，这个损失函数就是感知机学习的[经验风险函数](https://blog.csdn.net/zhzhx1204/article/details/70163099)。\n",
    "\n",
    "显然，损失函数$L(w,b)$是非负的。\n",
    "* 如果没有误分类点，那么$L(w,b)$为0\n",
    "* 误分类点数越少，$L(w,b)$值越小\n",
    "\n",
    "该损失函数在误分类时是参数$w,b$的线性函数，在正确分类时该损失函数是0。因此，给定训练数据集$\\mathbf{T}$，损失函数$L(w,b)$是$w,b$的连续可导函数。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 感知机学习算法\n",
    "\n",
    "\n",
    "最优化问题：给定数据集$\\mathbf{T} = \\{(x_1,y_1), (x_2, y_2), ... (x_N, y_N)\\}$，其中\n",
    "* $x_i \\in \\mathbb{R}^D$\n",
    "* $y_i \\in \\{-1, +1\\}，i=1,2...N$\n",
    "\n",
    "求参数$w,b$，使其成为损失函数的解，其中$\\mathbf{M}$为误分类数据的集合：\n",
    "\n",
    "$$\n",
    "min_{w,b} L(w, b) =  - \\sum_{x_i \\in \\mathbf{M}} y_i (w \\cdot x_i + b)\n",
    "$$\n",
    "\n",
    "感知机学习是误分类驱动的，具体采用随机梯度下降法：\n",
    "* 首先，任意选定$w_0$、$b_0$，\n",
    "* 然后用梯度下降法不断极小化目标函数\n",
    "   - 极小化的过程不是一次性的把$\\mathbf{M}$中的所有误分类点梯度下降\n",
    "   - 而是一次随机选取一个误分类点使其梯度下降。\n",
    "\n",
    "假设误分类集合$\\mathbf{M}$是固定的，那么损失函数$L(w,b)$的梯度为\n",
    "$$\n",
    "\\triangledown_w L(w, b) = - \\sum_{x_i \\in \\mathbf{M}} y_i x_i \\\\\n",
    "\\triangledown_b L(w, b) = - \\sum_{x_i \\in \\mathbf{M}} y_i \\\\\n",
    "$$\n",
    "\n",
    "随机选取一个误分类点$(x_i,y_i)$,对$w,b$进行更新：\n",
    "\\begin{eqnarray}\n",
    "w & = & w + \\eta y_i x_i \\\\\n",
    "b & = & b + \\eta y_i\n",
    "\\end{eqnarray}\n",
    "\n",
    "式中$\\eta$（0 ≤ $ \\eta $ ≤ 1）是学习速率（步长）：\n",
    "* 步长越大，梯度下降的速度越快，更能接近极小点。如果步长过大，有可能导致跨过极小点，导致函数发散；\n",
    "* 如果步长过小，有可能会耗很长时间才能达到极小点。\n",
    "\n",
    "> 直观解释：当一个实例点被误分类时，调整$w,b$，使分离超平面向该误分类点的一侧移动，以减少该误分类点与超平面的距离，直至超越该点被正确分类。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 算法\n",
    "\n",
    "\n",
    "**输入：**\n",
    "* $\\mathbf{T}=\\{(x_1,y_1),(x_2,y_2), ..., (x_N,y_N)\\}$, 其中$x_i \\in \\mathbf{X}=\\mathbb{R}^D$，\n",
    "* $y_i \\in \\mathbf{Y} = {-1, +1}，i=1,2...N$，\n",
    "* 学习速率为η\n",
    "\n",
    "**输出：**\n",
    "* $w$, $b$; \n",
    "* 感知机模型$f(x)=sign(w·x+b)$\n",
    "\n",
    "**处理过程:**\n",
    "1. 初始化$w_0$,$b_0$\n",
    "2. 在训练数据集中选取$(x_i, y_i）$\n",
    "3. 如果$y_i(w * x_i+b)≤0$\n",
    "           \n",
    "    $w = w + η y_i x_i$\n",
    "    \n",
    "    $b = b + η y_i$\n",
    "\n",
    "4. 如果所有的样本都正确分类，或者迭代次数超过设定值，则终止\n",
    "5. 否则，跳转至（2）\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. 示例程序\n",
    "\n",
    "生成数据："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2 2 2 2 2 1 1 2 1 1 1 1 2 1 1 1 1 2 2 2]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAEICAYAAABGaK+TAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAha0lEQVR4nO3deXxddZ3/8dfnbkma7rQNbboDLd1YSsrSIltLB0oFFFBBEEcdRsefIuIo4CAOjg6OiqDy+Dn8UEHgByObgKy1pZRKBdKCdC8tTRe6pWvabHf7zB8JtaS5WW/vzc19Px+PPpKcc+4979um75x87znna+6OiIjknkC2A4iISMeowEVEcpQKXEQkR6nARURylApcRCRHqcBFRHKUClxEJEepwKXbMbMKM6s1s/1mttfMXjezL5tZq9/vZjbSzNzMQpnIKtIZKnDprj7u7r2AEcAdwHeA32Q3kkh6qcClW3P3fe7+DPBp4Fozm2hmF5nZ22ZWZWabzOz7hzxkQePHvWZ2wMzOMLNjzGyeme0ys51m9rCZ9c30axFpSgUuecHd3wQ2Ax8DqoHPAX2Bi4CvmNmljZue1fixr7v3dPdFgAH/CQwBxgHDgO9nKrtIKipwySdbgP7uPt/dl7p70t3fBR4Bzk71IHdf6+5z3L3e3SuBO1vaXiRT9EaN5JNSYLeZnUbDuPhEIAIUAI+lepCZlQB303D03ouGA589RzytSCt0BC55wcym0FDgC4H/DzwDDHP3PsCvaRgmAWju9pw/alw+yd17A1cfsr1I1qjApVszs95mNht4FHjI3ZfScBS9293rzOxU4KpDHlIJJIHRhyzrBRwA9plZKfCvmUkv0jLT/cCluzGzCqAEiNNQxiuAh4Bfu3vCzC4Hfgb0B14FKmh40/LqxsffDnwFCAMXAPuB3wNjgbXAg8AN7j40c69K5HAqcBGRHKUhFBGRHKUCFxHJUSpwEZEcpQIXEclRGb2QZ8CAAT5y5MhM7lJEJOctXrx4p7sPbLo8owU+cuRIysvLM7lLEZGcZ2YbmluuIRQRkRylAhcRyVEqcBGRHKW7EYqkibuzfcG7bPjjQhJ1UQZNncjIK84mVBjJdjTppjJ6KX1ZWZnrTUzpjmq27ealmf/KgYptxKvrwZ1QzyIsGGD6U7cz+JyTsh1RcpiZLXb3sqbLNYQi0knJRIIXzv4G+1ZtIn6gDhoPiuIHaontq+bPs7/LvtWbspxSuiMVuEgnbX7uDWq27sbjiWbXJ+qjvPvjRzKcSvKBClykk9Y+OIf4gdqU6z2RpOKJBSnXi3SUClykk2JV1a1uk6yPZSCJ5BsVuEgnDTh1HIGCcIvb9Bo9JENpJJ+owEU66fh/no1Z6ikyQ8WFTPr2pzOYSPKFClykk4qHDmTKT79MsEfBYeuCPQoo+dgkjrnm/Cwkk+5OF/KIpMG4f7mE3seWsuS2+9lVvhrMKBrUl4nf+hTj/s8nCASD2Y4o3ZAKXCRNSmeWUTqzjHhdFI/FGy7kaWFoRaSzVOAiaRYqjIAun5cM0Bi4iEiOUoGLiOQoFbiISI5SgYuI5CgVuIhIjlKBi4jkKBW4iEiOUoGLiOQoFbiISI5SgYuI5CgVuIhIjmq1wM3st2a2w8yWHbLsJ2a2yszeNbOnzKzvEU0pIiKHacsR+P3ABU2WzQEmuvsJwBrg5jTnEhGRVrRa4O6+ANjdZNnL7h5v/PKvwNAjkE1ERFqQjjHwLwAvpOF5RESkHTpV4Gb2XSAOPNzCNteZWbmZlVdWVnZmdyIicogOF7iZfR6YDXzW3T3Vdu5+r7uXuXvZwIEDO7o7ERFpokMz8pjZBcC3gbPdvSa9kUREpC3achrhI8AiYKyZbTazLwK/AnoBc8zsHTP79RHOKSIiTbR6BO7uVzaz+DdHIIuIiLSDrsQUEclRKnARkRylAhcRyVEqcBGRHKUCFxHJUSpwEZEcpQIXEclRKnARkRylAhcRyVEqcBGRHKUCFxHJUR26G6GIZE7dzn2sfXAO+1ZuoKikH6M/O4O+xw/PdizpAlTgIl3Yil88SflN/w/MSNTWY6Egy372GMNmn85ZD91CMBLOdkTJIg2hiHRR6/8wn/Jb7iNRFyVRWw+AxxMk6qJseu4NXr/uziwnlGxTgYt0Qe5O+c33kaipb3Z9orae9//nFWq27spwMulKVOAiXdD+dVuo3b67xW0sGGDTs4sylEi6IhW4SBcUr64jEAq2uI3HE8Sr6zKUSLoiFbhIF9Rz1NEko/EWtwlEwvSdMDIzgaRLUoGLdEGR3sWMuOxjBMKpTxQL9+rBkBmTM5hKuhoVuEgXddrPv0rR4P4Emp4qaEaouJBzH7sNC+i/cD7Tv75IF1U4oA+XLPlvxn31EkK9igiEQ1g4yIhLpzH7r/dQMnVCtiNKlpm7Z2xnZWVlXl5enrH9iXQXnkwS219DqEdhi8Mq0j2Z2WJ3L2u6XN8JIjnAAgEifXpmO4Z0MRpCERHJUa0WuJn91sx2mNmyQ5b1N7M5ZvZe48d+RzamiIg01ZYj8PuBC5osuwmY6+7HAXMbvxYRkQxqtcDdfQHQ9JreS4AHGj9/ALg0vbFERKQ1HR0DL3H3rY2fbwNKUm1oZteZWbmZlVdWVnZwdyIi0lSn38T0hvMQU56L6O73unuZu5cNHDiws7sTEZFGHS3w7WY2GKDx4470RRIRkbboaIE/A1zb+Pm1wNPpiSMiIm3VltMIHwEWAWPNbLOZfRG4AzjfzN4DZjR+LSIiGdTqlZjufmWKVdPTnEVERNpBV2KKiOQoFbiISI5SgYuI5CgVuIhIjlKBi4jkKBW4iEiOUoGLiOQozcgjInIEJBMJPnjhTVb88ikObNhOjyEDGPfVSxh+yTQCoWBa9qECFxFJs3hdlDkX3sTOxWuIH6gFoGrNZnaWr6bP8cO4cN6dhHsWdXo/GkIREUmzN2+4h8o3Vx4s7w/FD9Syd2kFC7/wX2nZjwpcRCSNolXVrH3gZRK10WbXJ+qjbHp2EbXbm86T034qcBGRNNr51moCkXCL2wQiYba/trTT+1KBi4ikkSeTYG3YLuU0OG2nAhcRSaMBp4whWd/88MmHktEYg84Y3+l9qcBFRNKooH9vRl5xDoHC5odRApEQg2dMpnho56eY1GmEItIhe1dUsPHp14nX1NH/xGMazm8Oq1IAzrjnevYur2Df6k3Eq+sOLg8VF1I8bBBn/f7mtOzHPB0DMW1UVlbm5eXlGdufiKRfbH8N8674d7a/tpRkLI7HE4R7FWGhIOc9/n0Gn3tytiN2CYlojIo/zGf5L56kZvNOikr6Me5rn2D0lecRKipo13OZ2WJ3LztsuQpcpHXJRILY/lrCPYvSdhVdLnJ3Xjj3m1S+sZJkfeyw9aEeBVy06Ff0nzQ6C+m6r1QFrjFwkRZUf1DJX677GQ/1ns2jR1/Gg70uYsHn/pP9FduyHS0rdr61ml2L1zRb3gDx2ijv3P77DKfKXxqwEklhf8U2np3yFaL7qvF44uDy9x+Zx8ZnFzH79V/Sd9yILCbMvHUP/5l4bX3qDdzZ+PTreDKJBXR8eKTpb1gkhdc+/2Oiew58pLwBPJEkVlXD/Cv/I0vJsqd+dxUkWx529WSSZCyeoUT5TQUu0owDG7az881VDRdlNMedqrUfsHvp+5kNlmX9TzyGYCtvwBUe1ZtgQSRDifKbClykGXtXbiBQ0Mrl0MEge5dXZCZQF3Hc5y9o8RLCYGGE8V//ZAYT5bdOFbiZ3WBmy81smZk9YmaF6Qomkk2h4sLWr3W2xu3ySOGAPpx611ebPQoPFoTpPWYoE264PAvJ8lOHC9zMSoGvA2XuPhEIAp9JVzCRbBp0+njMWr6hRTKeYPB5+XfO8/HXzebcx26j34mjCURChHoUEO7Vg3Ff+wQX/eUXhHrk1w+1bOrsWSghoMjMYkAPYEvnI4lkXyAc4sRbr+Ht2+7/yJV0Hwr1KGT8DZcRLu78Tflz0bBZpzFs1mnU7dxHvLaeHkf311WYWdDhI3B3/wD4KbAR2Arsc/eXm25nZteZWbmZlVdWVnY8qUiGTbjhcsZ97RMECyMHx8MDkTDBwgjHXDuTyf/++ewG7AIKB/Sh57BBKu8s6fCVmGbWD3gC+DSwF3gMeNzdH0r1GF2JKbmo+oNK1j44hwPvb6V42CCO/dxMeo4oyXYsySOprsTszI/NGcB6d69s3MGTwFQgZYGL5KLi0oGceNNV2Y4hcpjOnIWyETjdzHpYw7s904GV6YklIiKt6cwY+BvA48ASYGnjc92bplwiItKKTr3z4O63AbelKYuIiLSDrsQUEclRKnARkRylAhcRyVEqcBGRHKUCFxHJUSpwEZEcpQIXEclRKnARkRylAhcRyVG6B6R0SDIW58DGHQQiIYqHDmx18gMRST8VuLRLoj7KOz94kJX3PI3HE3giSVFJP076/rUcd+0/ZDueSF5RgUubJWNxXpzxr+xasoZEbfTg8gMbtvPXr97NvjWbKPvhl7KYUCS/aAxc2uy9+19i99trP1LeH4rX1LPi50+wb/WmLCQTyU8qcGmzZXc+Rrzm8PkhP5SMJ1h5zx8zF0gkz6nApc2qN+5ocb3HE+xZuj5DaUREBS5tFioubHWbgqN6ZSCJiIAKXNrhmKtnEIikft871LOI474wK4OJRPKbzkKRNpt44xW897sXSUbjh60LREL0Gj2Y0n84bOLsLql+z37WPvASO5e8R0Hfnoz6zHkMOmO8zmeXnKIClzYrLh3IrFd/zpzZ3yW69wDxmnoCwQAWCjKgbAzTn7qdQDCY7ZitWvvQHF7/5zvBjERNPQSM9373Iv0mjeL85++goG/PbEcUaRNz94ztrKyszMvLyzO2PzkyPJlky9wl7HxrNYFwiKEXnkq/iaOyHatNts5/hzkX3UKitv6wdYFImAGnjuWiBXdnIZlIama22N0P+/VWR+DSbhYIUHp+GaXn58ZwyaGW/Ntvmy1vgGQ0xu4la9m5eA0DThmT4WTtE6+tZ/0f5rPxqYUk4wlKZ5Zx7LUzifTRbw/5RAUueSNeU0flm6ta3qaung1PvtalC3z3u+t4cca3SNTFiB+oBWDb/L+x+N9+y/SnbmfI9MlZTiiZorNQJG8k6qJYsJVv+aQT21+TmUAdEN13gBfOu5H6nVUHyxsafjjFD9Qy99Jb2f/+liwmlEzqVIGbWV8ze9zMVpnZSjM7I13BRNIt0rdnq+eyh3oWcVQXPvpe+/uXSdbFUq5PROMsv+uJDCaSbOrsEfjdwIvufjxwIrCy85FEjgwLBBj/9U8SLIq0uN2oT52TmUAd8P4jr7R4OwOPxal4YkEGE0k2dbjAzawPcBbwGwB3j7r73jTlEjkiJn37M/SbOIpgUcFHV5gR7FHAOY/eSqjpui4kUXf4jcSaSsYSGUgiXUFnjsBHAZXA78zsbTO7z8yKm25kZteZWbmZlVdWVnZidyKdFyqMcOGrd3HS966hsKQfFghgwQDDLjqNWQvuYtis07IdsUUlZ07Ewi2fe9Bv4kje+cGDvHTBd5h72W1UPP4qydjhF19J7uvweeBmVgb8FZjm7m+Y2d1AlbvfmuoxOg9cupp4XZRgJIQFcuP9/Kq1H/DHE7/U7C19oeFcdnAsEDh4tB7qVURB/97MevXn9BxeksG0ki6pzgPvzHftZmCzu7/R+PXjgM5fkpwSKozkTHkD9D62lFN/9hVCPQ4f5gkWRkgmEiSj8Y8MtcT311KzqZIXzvkmyYSGV7qTDn/nuvs2YJOZjW1cNB1YkZZUIpLS8V++mJkv/RelF0whWBghEAkx8PRx9Bk3HBLJZh/jySR1u/ax+fk3ml0vuamzF/J8DXjYzCLA+8A/dj6SiLSmZNpEZj5/x8Gv3Z0HClqekzS+v5aKxxcw/ONTj3Q8yZBOFbi7vwPk3vXUIt2NO57i6PtQqW4jILkpdwb/RCQlCwToPaa0xW1CxYWUnHVChhJJJqjARbqJE75zZctXmjoc+7mZmQskR5wKXKSbOPZzMxk667TDStwCgYaLlP7wPSK9D7tUQ3KYClykm7BAgHMe+Tem/vc36TdpNIFIiFBxISM/fTazF/2qy1+kJO2n28mKdCMWCHDMVdM55qrp2Y4iGaAjcBGRHKUjcBHplNrtu9n+l+WQTDLwjPEUlw7MdqS8oQIXkQ6JHahl4Zd+wqanX2+4B4tBsj7GkJllfOz+71DQr1e2I3Z7GkIRkXZLxhO8cN6NbHp6EYn6GLH9NcSqakjUx/jgpbd4btrXieuioSNOBS4i7bbx6b+wb9UGEvWH3xUxGY1TvWkH7z8yLwvJ8osKXETabeU9TxM/kHpmoHh1HSt/9VQGE+UnFbiItFvNlp2tblO7fU8GkuQ3FbiItFtbJoboMWRABpLkNxW4iLTbuK9dSqhnUcr1oeJCJlz/yQwmyk8qcBFpt6GzTuOoyccRLIwcti5YEKHP2GGM/NQ5mQ+WZ1TgItJugWCQmS/+mGOuOZ9gYYRw7x6E+xQTLAgz4vKPceH8nxOMhLMds9vr8KTGHaFJjUW6n+i+A1S+uQqSzlFlYyg8qk+2I3U7qSY11pWYItIpkT49KT1fE3Nlg4ZQRERylApcRCRHqcBFRHKUClxEJEepwEVEclSnC9zMgmb2tpn9KR2BRESkbdJxBH49sDINzyMiIu3QqQI3s6HARcB96YkjIiJt1dkLee4Cvg1o7qQuLJFI8pf57/P8U8up3HaAcCTIqdNGcNEnJ1AyuHe244lIB3X4CNzMZgM73H1xK9tdZ2blZlZeWVnZ0d1JByUSSe78wTweuvdNtm6uIh5PUlsT47W567j1hudYs3JHtiOKSAd1ZghlGnCxmVUAjwLnmdlDTTdy93vdvczdywYO1GzVmfbSMytZs2IH9fWJjyxPJp36ujh3/fAV4rFEikeLSFfW4QJ395vdfai7jwQ+A8xz96vTlkw6zd158ekVRKOpCzoRT7L4jU0ZTCUi6aLzwLux6v1Rqg8cPunsoerq4qxdraEtkVyUlrsRuvt8YH46nkvSJxgO0JbbBYdDwQykEZF00xF4N1ZUFKZ0eN8WtykoDHHyaUMzE0hE0koF3s1ddtVJRAqaP8IOBI2Swb04dqzeXBbJRSrwbu6kKUO57KqTCIeDhEJ//+cuLAxRcnQvvnXbdMwsiwlFpKM0I08euOCS8ZSdMZx5L66hYu0uioojTDt3NCedUkogqJ/hIrlKBZ4jdlVW89yTy3j91fXU1cXp26+If5h9POfNGktBQev/jAMG9eRTn5ucgaQikikq8BywqWIPP7zlJaL1cRKJhrNK9uyq4clH/sbC+e9z6x0XUFikGcBF8o1+f+7i3J27fzSf2prYwfL+UDSaYNuWKv7ngSVZSici2dTlCzyZSLLh/d2sW1PZ6kUp3dHqFTuoqqpLuT4eS7LwlXXU18czmEpEuoIuO4Ti7sx5bjXP/OFdYtEEFjDi8SQnTi6lZEgvyhdtJBpNMHxkPy68dDzjJh2d7chHRMW6XSTiyRa3CQQC7Ni6n2Ej+2UolYh0BV22wB+9fzHzXnyPaJMjy/K/bvzI13t217Bq2XbOnH4M1/zTlIOnxNXWRNlUsRcLwIhR/Ym04Y2+lrg7a1bsYOnbW0gmnTHjB3HCyUOO+Fkc4VAQC7R8ml8ymSQc1tWUIvmmSxb4ti1VzH1hDbEWbsJ0kEN9fZyFc9dx3PEDmXzqUB667y0WLag4eN5zMunMmDWWyz57EsEOFO6uymp+evtcdlVWU1/X8ANl7gurKSwK863vTT+iR74nnDKER37X4h17KS6OUDJEt2QXyTddcgx8/ktrSCZaHjZoqr4+ztP/8y4/vOVlFr26nlg0QW1NjNqaGPV1ceY8t4pf/vjVNt0b5FDRaIL/uPlFtn1QdbC8Aepq4+zdXcuPvvsS+/bWtus522NgSS8mnjSYcLj5f6pIQZBLrzxRF+OI5KEuWeBbP6g67IyLtti2pYotm/cSix1e/tH6BCve3cbq5e2bwODNhRVUH4iSTDafJxZN8OfnV7c7a3t8+ZtnMuq4ARQUhqCxp4OhAOFIkJmzj+fsGcce0f2LSNfUJYdQ+vYvwgzaebCMO8SiqY/c6+vjzPnTKo6fWNLm51wwd+1HjrybisWS/GXeOi676qT2RG2XwqIwt/xwJmtW7GDB3LVU7aujdGgfzr1gLCWDNXQikq+6ZIGfNeNYXn91PdH69s0UEwhAsqWRF4cd2/a36zlrq2OtbpOJU/jMjLETShg7oe0/fESke+uSQyijjxvAmHGDCEfafmZFQUGIXr0LW92uT7+ig58nE0n2V9W1WMDDR/cj0MpZIEeX9mlzThGRdOmSR+BmxvW3nMtvf7WItxZtIBgM4EnH3YknnFAocPAMFQsY4XCA82ePpVefIp546O2UU4gVFIaYfuEY6mpjPPv4Uua+sIZ4LEEi6Rx3/EAuv/pkxowb9JHHzJw9jjcXbmjxOWddOj69fwEiIm3QJQscIBIJ8uVvnsln9pzC0iVbiMcTjBjdn5LBvVn4yjoWzltHtD7BiGP6c+El4xl17FHU1sZ4+dmV7NlVc9ibjuFwgCFDezN2/CBu//YL7Ni2/yNvdq5evoOf3PZn/vmbZ1J2+vCDy0eM7s/0WWOZ+8Lqw4Z0IgVBJp40mJNPHXZk/zJERJph7T2trjPKysq8vLz8iO5j755a7vnJAtav3UUgYBiQSCSZNLmU674xjWcfW8rLz65s9kwVaBiK+eUDl1NQ+PebQ7k7ixas54+PvsuuymowKO5ZwEWfGM/5Fx2vW7KKyBFlZovdveyw5d2twD+0bUsVa1dVYgFj3KSj6X9UD5KJJP9yzR+orUn9xmRBYYirvzSFs5o5Nc/dqd4fJZlM0rN3Yatj4yIi6ZCqwLvsEEpnHT2kN0cP6f2RZdXVUeKxls9sqa+Ls3H97mbXmRk9exekLaOISGfk1e/+kUiQRIoLcj4UCBhFxZEMJRIR6bi8KvCCwjDHtTKBbygU4LRpIzKUSESk4/KqwAEuv/pkIinOLw+HA4wZP4ihI3RbVhHp+jpc4GY2zMxeMbMVZrbczK5PZ7AjZcz4QVz3jWlECoIN9xahYdgkEgkydkIJX7/p7CwnFBFpm868iRkHbnT3JWbWC1hsZnPcfUWash0xU6aO4ITJQ/jraxVsqthDUXGEU6eO0IQIIpJTOlzg7r4V2Nr4+X4zWwmUAl2+wKFhPPzs84/LdgwRkQ5Lyxi4mY0ETgbeaGbddWZWbmbllZWV6didiIiQhgI3s57AE8A33L2q6Xp3v9fdy9y9bODAls8AERGRtutUgZtZmIbyftjdn0xPJBERaYvOnIViwG+Ale5+Z/oiiYhIW3TmCHwacA1wnpm90/hnVppyiYhIKzpzFspCDs7Q2P25O0ve2MSfnljGpoq9BILGCZOH8PHLJzFidP9sxxORPNRtb2aVTu7OA79+g9fnr//77D0xKF+0kb+Vf8A/XT+VU6eN7NBzJxJJ/lb+AetWVxKKBDlhcinHjBmQvvAi0m2pwNvg7Tc3f7S8G7lDNJrg3rtfZ+yEEvr0LUrxDM1bv3YXd/5gHtFonLraOGbw/FPLGTK0Dzfeeh692/l8IpJf8u5eKB3x7BPLWp24eP7L77XrOXfuOMAdt75M1b466mobntsdovUJNlXs4UfffZlEoqUZmkUk36nA22BTxZ4W18eiCVYt296u53zhj8sPzuvZVCLh7NlVw98Wf9Cu5xSR/JJXQyjuzrJ3tvLcE8vYsH43wWCAU04fzoWXjj9s8odDBQNG6jl8GoTCzd/hMJVFCypIJFLfm7yuLs5rc9cxWfNtikgKeXME7u48dN9b/PKO+axctp2a6hj7q+pZMHctt97wJ5a9syXlY084pRRr4XybgsIQp585sl15Uh19H6rmQLRdzyki+SVvCnzxG5t47c/rqG8ys3wy4UTrE/zijlepqW6+MD9++UTCqY6wrWEi5CntnARiQEnPFtcHg8bw0bo7ooikljcF/qfHW34j0t1ZOG9ds+uGj+rfcA/xSJDwIZNBFBaG6N27kJt/ODPlJBGpzLp0AgUFqR8TCAaYMWtsu55TRPJL3oyBp5qo+EPR+gTL/7aVmR8f1+z6KVNHMHZCCfNffo9VS7cRigQ57cyRTJk6ot3lDTD1nFG8/ur7rF1VSbTJcEqkIMjsyyZSMjj1uLyISN4UeMOtW1qZ0DjY8i8kvfsUcvEVk7j4ikmdzhMMBrjxe9N5/qnlvPTMSurr43jSGXh0Tz555Umcqnk5RaQVeVPg4yYdzdK3U79RWVgYYsrU4RlM1DCB8sVXTGL2JydQta+OYChAr96FGc0gIrkrb8bAL/7UpBaHOsIFQaZMzc5RbyAYoG//HipvEWmXvCnwMeMG8dkvTSEcCRIM/f1lFxSG6NmrgJt+MDP1mSYiIl1Q3gyhAJwz8zgmnDiYPz+/ijUrdhAKB5l69ijOOGsUhUXhbMcTEWmXvCpwgIElPbnyH8uyHUNEpNPyZghFRKS7UYGLiOQoFbiISI5SgYuI5Chzb/nqxLTuzKwS2JCxHabHAGBntkNkST6/dtDrz+fX39Ve+wh3H9h0YUYLPBeZWbm75+VpK/n82kGvP59ff668dg2hiIjkKBW4iEiOUoG37t5sB8iifH7toNefz68/J167xsBFRHKUjsBFRHKUClxEJEepwJthZsPM7BUzW2Fmy83s+mxnygYzC5rZ22b2p2xnyTQz62tmj5vZKjNbaWZnZDtTppjZDY3f98vM7BEz69Y3qjez35rZDjNbdsiy/mY2x8zea/zYJWcYV4E3Lw7c6O7jgdOBr5rZ+CxnyobrgZXZDpEldwMvuvvxwInkyd+DmZUCXwfK3H0iEAQ+k91UR9z9wAVNlt0EzHX344C5jV93OSrwZrj7Vndf0vj5fhr+85ZmN1VmmdlQ4CLgvmxnyTQz6wOcBfwGwN2j7r43q6EyKwQUmVkI6AGknouwG3D3BUDTWc8vAR5o/PwB4NJMZmorFXgrzGwkcDLwRpajZNpdwLeBZJZzZMMooBL4XeMQ0n1mVpztUJng7h8APwU2AluBfe7+cnZTZUWJu29t/HwbUJLNMKmowFtgZj2BJ4BvuHtVtvNkipnNBna4++JsZ8mSEDAZ+L/ufjJQTRf9FTrdGsd6L6Hhh9gQoNjMrs5uquzyhnOtu+T51irwFMwsTEN5P+zuT2Y7T4ZNAy42swrgUeA8M3sou5EyajOw2d0//K3rcRoKPR/MANa7e6W7x4AngalZzpQN281sMEDjxx1ZztMsFXgzzMxoGP9c6e53ZjtPprn7ze4+1N1H0vAG1jx3z5ujMHffBmwys7GNi6YDK7IYKZM2AqebWY/G/wfTyZM3cJt4Bri28fNrgaezmCUlFXjzpgHX0HDk+U7jn1nZDiUZ9TXgYTN7FzgJ+FF242RG428djwNLgKU0dEROXFbeUWb2CLAIGGtmm83si8AdwPlm9h4Nv5Xckc2MqehSehGRHKUjcBGRHKUCFxHJUSpwEZEcpQIXEclRKnARkRylAhcRyVEqcBGRHPW/W2JU8OyhZngAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# data generation\n",
    "np.random.seed(314)\n",
    "\n",
    "data_size1 = 10\n",
    "x1 = np.random.randn(data_size1, 2) + np.array([2,2])\n",
    "y1 = [-1 for _ in range(data_size1)]\n",
    "\n",
    "data_size2 = 10\n",
    "x2 = np.random.randn(data_size2, 2)*2 + np.array([8,8])\n",
    "y2 = [1 for _ in range(data_size2)]\n",
    "\n",
    "# all sample data\n",
    "x = np.concatenate((x1, x2), axis=0)\n",
    "y = np.concatenate((y1, y2), axis=0)\n",
    "\n",
    "shuffled_index = np.random.permutation(data_size1 + data_size2)\n",
    "x = x[shuffled_index]\n",
    "y = y[shuffled_index]\n",
    "\n",
    "train_data = np.concatenate((x, y[:, np.newaxis]), axis=1)\n",
    "\n",
    "label_y = y.copy()\n",
    "label_y[label_y==-1] = 2;\n",
    "print(label_y)\n",
    "# plot data\n",
    "plt.scatter(train_data[:,0], train_data[:,1], marker='.',  s = 300,\n",
    "            c=label_y, cmap=plt.cm.Spectral)\n",
    "plt.title(\"Data\")\n",
    "plt.savefig(\"fig-res_perceptron_sample_data.pdf\")\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "学习模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "lines_to_end_of_cell_marker": 2
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "update weight/bias:  3.662519607024163 3.038628576040485 0.5\n",
      "update weight/bias:  3.1283482021588505 1.595298136548129 0.0\n",
      "update weight/bias:  1.7022056555578562 0.9488336160189257 -0.5\n",
      "update weight/bias:  0.6064534385664728 0.25821553124912766 -1.0\n",
      "update weight/bias:  -0.0072588404447664345 -1.1732780618880432 -1.5\n",
      "update weight/bias:  3.56979581700504 4.157268901515593 -1.0\n",
      "update weight/bias:  2.7104258301108155 3.6232316160976543 -1.5\n",
      "update weight/bias:  1.6146736131194321 2.9326135313278563 -2.0\n",
      "update weight/bias:  1.0805022082541196 1.4892830918355 -2.5\n",
      "update weight/bias:  -0.3456403383468747 0.8428185713062968 -3.0\n",
      "update weight/bias:  3.4230241861565407 3.6955935657768997 -2.5\n",
      "update weight/bias:  2.5636541992623156 3.161556280358961 -3.0\n",
      "update weight/bias:  1.1375116526613214 2.5150917598297577 -3.5\n",
      "update weight/bias:  0.38678027777241897 1.1329290169473543 -4.0\n",
      "update weight/bias:  -1.446079265832824 -0.7361650837497964 -4.5\n",
      "update weight/bias:  1.7930043074867144 5.9278879714909145 -4.0\n",
      "update weight/bias:  1.2588329026214018 4.484557531998558 -4.5\n",
      "update weight/bias:  0.3697277316535954 3.2936957431536 -5.0\n",
      "update weight/bias:  -0.519377439314211 2.1028339543086423 -5.5\n",
      "update weight/bias:  -2.352236982919454 0.23373985361149163 -6.0\n",
      "update weight/bias:  1.8159337720901148 4.415105700242464 -5.5\n",
      "update weight/bias:  0.7328910527198487 3.024123458986641 -6.0\n",
      "update weight/bias:  -0.3501516666504174 1.6331412177308182 -6.5\n",
      "w =  [-0.3501516666504174, 1.6331412177308182]\n",
      "b =  -6.5\n",
      "\n",
      "\n",
      "\n",
      "ground_truth:  [-1. -1. -1. -1. -1.  1.  1. -1.  1.  1.  1.  1. -1.  1.  1.  1.  1. -1.\n",
      " -1. -1.]\n",
      "predicted:     [-1. -1. -1. -1. -1. -1.  1. -1.  1.  1.  1.  1. -1.  1.  1.  1.  1. -1.\n",
      " -1. -1.]\n",
      "accuracy:      0.95\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "def sign(v):\n",
    "    \"\"\"符号函数\"\"\"\n",
    "    if v > 0:  return 1\n",
    "    else:      return -1\n",
    "    \n",
    "def perceptron_train(train_data, eta=0.5, n_iter=100):\n",
    "    \"\"\"对感知机模型进行训练\"\"\"\n",
    "    weight = [0, 0]      # 权重\n",
    "    bias = 0             # 偏置量\n",
    "    learning_rate = eta  # 学习速率\n",
    "\n",
    "    train_num = n_iter  # 迭代次数\n",
    "\n",
    "    for i in range(train_num):\n",
    "        # select one data\n",
    "        ti = np.random.randint(len(train_data))\n",
    "        (x1, x2, y) = train_data[ti]\n",
    "        \n",
    "        y_pred = sign(weight[0] * x1 + weight[1] * x2 + bias) \n",
    "        \n",
    "        if y * y_pred <= 0:  # 判断误分类点\n",
    "            weight[0] = weight[0] + learning_rate * y * x1  # 更新权重\n",
    "            weight[1] = weight[1] + learning_rate * y * x2\n",
    "            bias      = bias      + learning_rate * y       # 更新偏置量\n",
    "            print(\"update weight/bias: \", weight[0], weight[1], bias)\n",
    "\n",
    "    return weight, bias\n",
    "\n",
    "def perceptron_pred(data, w, b):\n",
    "    \"\"\"输入数据，模型，对数据进行分类\"\"\"\n",
    "    y_pred = []\n",
    "    for d in data:\n",
    "        x1, x2, y = d\n",
    "        yi = sign(w[0]*x1 + w[1]*x2 + b)\n",
    "        y_pred.append(yi)\n",
    "        \n",
    "    return np.array(y_pred, dtype=float)\n",
    "\n",
    "\n",
    "# 训练感知机\n",
    "w, b = perceptron_train(train_data)\n",
    "print(\"w = \", w)\n",
    "print(\"b = \", b)\n",
    "\n",
    "# 预测 \n",
    "y_pred = perceptron_pred(train_data, w, b)\n",
    "\n",
    "# 计算分类精度\n",
    "c = y_pred == y\n",
    "cn = np.sum(c == True)\n",
    "acc = cn / len(y_pred)\n",
    "print()\n",
    "\n",
    "print(\"\\n\")\n",
    "print(\"ground_truth: \", train_data[:, 2])\n",
    "print(\"predicted:    \", y_pred)\n",
    "print(\"accuracy:     \", acc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAEICAYAAABGaK+TAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAhbUlEQVR4nO3de3xU9Z3/8ddnbkkIhItAhHBH5a6owQu43kBWgaqt2lartdt23Xb7a61tt6t2rV27dt3d1movj1/Xn2216k+33qrWK6KIVsQGtHJHkAgISLgGcpvMzGf/SKQYMrnNMJPJvJ+PB48k55yZ8x4I75x853vOMXdHRERyTyDbAUREpGtU4CIiOUoFLiKSo1TgIiI5SgUuIpKjVOAiIjlKBS4ikqNU4NLjmFmlmdWZ2X4z22tmr5vZV8ys3e93MxtlZm5moUxkFUmFClx6qk+4ex9gJHAb8M/Ar7MbSSS9VODSo7n7Pnd/EvgMcLWZTTazuWb2lplVm9lmM/vBIQ9Z1Pxxr5kdMLPTzWysmb1kZrvMbKeZPWBm/TL9WkRaUoFLXnD3N4EtwN8ANcDngX7AXOCrZnZx86ZnNn/s5+693X0xYMC/A0OBCcBw4AeZyi6SjApc8slWYIC7L3T35e6ecPd3gAeBs5I9yN3Xu/t8d29w9yrg9ra2F8kUvVEj+aQM2G1mp9I0Lj4ZiAAFwMPJHmRmpcCdNB2996HpwGfPEU8r0g4dgUteMLNpNBX4a8D/B54Ehrt7X+BXNA2TALR2ec4fNS+f4u4lwJWHbC+SNSpw6dHMrMTM5gEPAfe7+3KajqJ3u3u9mZ0CXHHIQ6qABDDmkGV9gAPAPjMrA/4pM+lF2ma6Hrj0NGZWCZQCMZrKeBVwP/Ard4+b2aXAT4ABwCtAJU1vWl7Z/PhbgK8CYeB8YD/wO2AcsB64D7jO3Ydl7lWJHE4FLiKSozSEIiKSo1TgIiI5SgUuIpKjVOAiIjkqoyfyDBw40EeNGpXJXYqI5LylS5fudPdBLZdntMBHjRpFRUVFJncpIpLzzOz91pZrCEVEJEepwEVEcpQKXEQkR+lqhCJp4u58uOgd3v/Da8TrowyePplRl51FqDCS7WjSQ2X0VPry8nLXm5jSE9Vu383zs/+JA5XbidU0gDuh3kVYMMDMx29hyNlTsx1RcpiZLXX38pbLNYQikqJEPM6zZ32TfWs2EztQD80HRbEDdTTuq+HFed9j39rNWU4pPZEKXCRFW55eQu223Xgs3ur6eEOUd/7jwQynknygAhdJ0fr75hM7UJd0vccTVD66KOl6ka5SgYukqLG6pt1tEg2NGUgi+UYFLpKigadMIFAQbnObPmOGZiiN5BMVuEiKxv/DPMyS3yIzVFzIlO9+JoOJJF+owEVSVDxsENN+/BWCvQoOWxfsVUDp30xh7FXnZSGZ9HQ6kUckDSb840WUHFPGspvvYVfFWjCjaHA/Jn/n00z4P58kEAxmO6L0QCpwkTQpm11O2exyYvVRvDHWdCJPG0MrIqlSgYukWagwAjp9XjJAY+AiIjlKBS4ikqNU4CIiOUoFLiKSo1TgIiI5SgUuIpKjVOAiIjlKBS4ikqNU4CIiOUoFLiKSo1TgIiI5qt0CN7PfmNkOM1txyLL/MrM1ZvaOmT1uZv2OaEoRETlMR47A7wHOb7FsPjDZ3Y8H1gE3pDmXiIi0o90Cd/dFwO4Wy15w91jzl28Aw45ANhERaUM6xsC/CDybhucREZFOSKnAzex7QAx4oI1trjGzCjOrqKqqSmV3IiJyiC4XuJl9AZgHfM7dPdl27n6Xu5e7e/mgQYO6ujsREWmhS3fkMbPzge8CZ7l7bXojiYhIR3RkGuGDwGJgnJltMbMvAb8A+gDzzextM/vVEc4pIiIttHsE7u6Xt7L410cgi4iIdILOxBQRyVEqcBGRHKUCFxHJUSpwEZEcpQIXEclRKnARkRylAhcRyVEqcBGRHKUCFxHJUSpwEZEcpQIXEclRXboaoYhkTv3Ofay/bz77Vr9PUWl/xnxuFv3Gj8h2LOkGVOAi3diqnz1GxfX/D8yI1zVgoSArfvIww+edxpn330gwEs52RMkiDaGIdFMbf7+QihvvJl4fJV7XAIDH4sTro2x+egmvX3N7lhNKtqnARbohd6fihruJ1za0uj5e18B7//Mytdt2ZTiZdCcqcJFuaP+GrdR9uLvNbSwYYPNTizOUSLojFbhINxSrqScQCra5jcfixGrqM5RIuiMVuEg31Hv00SSisTa3CUTC9Js0KjOBpFtSgYt0Q5GSYkZe8jcEwsknioX79GLorJMymEq6GxW4SDd16k+/RtGQAQRaThU0I1RcyDkP34wF9F84n+lfX6SbKhzYl4uW/TcTvnYRoT5FBMIhLBxk5MUzmPfGLymdPinbESXLzN0ztrPy8nKvqKjI2P5EegpPJGjcX0uoV2GbwyrSM5nZUncvb7lc3wkiOcACASJ9e2c7hnQzGkIREclR7Ra4mf3GzHaY2YpDlg0ws/lm9m7zx/5HNqaIiLTUkSPwe4DzWyy7Hljg7scCC5q/FhGRDGq3wN19EdDynN6LgHubP78XuDi9sUREpD1dHQMvdfdtzZ9vB0qTbWhm15hZhZlVVFVVdXF3IiLSUspvYnrTPMSkcxHd/S53L3f38kGDBqW6OxERadbVAv/QzIYANH/ckb5IIiLSEV0t8CeBq5s/vxp4Ij1xRESkozoyjfBBYDEwzsy2mNmXgNuA88zsXWBW89ciIpJB7Z6J6e6XJ1k1M81ZRESkE3QmpohIjlKBi4jkKBW4iEiOUoGLiOQoFbiISI5SgYuI5CgVuIhIjtIdeUREjoBEPM4Hz77Jqp8/zoH3P6TX0IFM+NpFjLhoBoFQMC37UIGLiKRZrD7K/AuuZ+fSdcQO1AFQvW4LOyvW0nf8cC546XbCvYtS3o+GUERE0uzN635J1ZurD5b3R2IH6ti7vJLXvvifadmPClxEJI2i1TWsv/cF4nXRVtfHG6JsfmoxdR+2vE9O56nARUTSaOef1xKIhNvcJhAJ8+Gry1PelwpcRCSNPJEA68B2SW+D03EqcBGRNBp48nEkGlofPvlIItrI4NMnprwvFbiISBoVDChh1GVnEyhsfRglEAkxZNZJFA9L/RaTmkYoIl2yd1Ulm554nVhtPQNOGNs0vzmsSgE4/ZfXsndlJfvWbiZWU39weai4kOLhgznzdzekZT/m6RiI6aDy8nKvqKjI2P5EJP0a99fy0mX/yoevLifRGMNjccJ9irBQkHMf+QFDzjkx2xG7hXi0kcrfL2Tlzx6jdstOikr7M+Hrn2TM5ecSKiro1HOZ2VJ3Lz9suQpcpH2JeJzG/XWEexel7Sy6XOTuPHvOt6hasppEQ+Nh60O9Cpi7+BcMmDImC+l6rmQFrjFwkTbUfFDFn675CfeXzOOhoy/hvj5zWfT5f2d/5fZsR8uKnX9ey66l61otb4BYXZS3b/ldhlPlLw1YiSSxv3I7T037KtF9NXgsfnD5ew++xKanFjPv9Z/Tb8LILCbMvA0PvEisriH5Bu5seuJ1PJHAAjo+PNL0NyySxKtf+A+iew58rLwBPJ6gsbqWhZf/W5aSZU/D7mpItD3s6okEicZYhhLlNxW4SCsOvP8hO99c03RSRmvcqV7/AbuXv5fZYFk24ISxBNt5A67wqBKCBZEMJcpvKnCRVuxd/T6BgnZOhw4G2buyMjOBuoljv3B+m6cQBgsjTPzGpzKYKL+lVOBmdp2ZrTSzFWb2oJkVpiuYSDaFigvbP9fZmrfLI4UD+3LKHV9r9Sg8WBCm5LhhTLru0iwky09dLnAzKwO+AZS7+2QgCHw2XcFEsmnwaRMxa/uCFolYnCHn5t+c5/HXzOOch2+m/wljCERChHoVEO7Tiwlf/yRz//QzQr3y64daNqU6CyUEFJlZI9AL2Jp6JJHsC4RDnHDTVbx18z0fO5PuI6FehUy87hLCxalflD8XDZ9zKsPnnEr9zn3E6hrodfQAnYWZBV0+Anf3D4AfA5uAbcA+d3+h5XZmdo2ZVZhZRVVVVdeTimTYpOsuZcLXP0mwMHJwPDwQCRMsjDD26tmc9K9fyG7AbqBwYF96Dx+s8s6SLp+JaWb9gUeBzwB7gYeBR9z9/mSP0ZmYkotqPqhi/X3zOfDeNoqHD+aYz8+m98jSbMeSPJLsTMxUfmzOAja6e1XzDh4DpgNJC1wkFxWXDeKE66/IdgyRw6QyC2UTcJqZ9bKmd3tmAqvTE0tERNqTyhj4EuARYBmwvPm57kpTLhERaUdK7zy4+83AzWnKIiIinaAzMUVEcpQKXEQkR6nARURylApcRCRHqcBFRHKUClxEJEepwEVEcpQKXEQkR6nARURylK4BKV2SaIxxYNMOApEQxcMGtXvzAxFJPxW4dEq8IcrbP7yP1b98Ao/F8XiCotL+TP3B1Rx79d9mO55IXlGBS4clGmM8N+uf2LVsHfG66MHlB97/kDe+dif71m2m/NYvZzGhSH7RGLh02Lv3PM/ut9Z/rLw/EqttYNVPH2Xf2s1ZSCaSn1Tg0mErbn+YWO3h94f8SCIWZ/Uv/5C5QCJ5TgUuHVazaUeb6z0WZ8/yjRlKIyIqcOmwUHFhu9sUHNUnA0lEBFTg0gljr5xFIJL8fe9Q7yKO/eKcDCYSyW+ahSIdNvnbl/Hub58jEY0dti4QCdFnzBDK/vawG2d3Sw179rP+3ufZuexdCvr1ZvRnz2Xw6RM1n11yigpcOqy4bBBzXvkp8+d9j+jeA8RqGwgEA1goyMDy45j5+C0EgsFsx2zX+vvn8/o/3A5mxGsbIGC8+9vn6D9lNOc9cxsF/XpnO6JIh5i7Z2xn5eXlXlFRkbH9yZHhiQRbFyxj55/XEgiHGHbBKfSfPDrbsTpk28K3mT/3RuJ1DYetC0TCDDxlHHMX3ZmFZCLJmdlSdz/s11sdgUunWSBA2XnllJ2XG8Mlh1r2L79ptbwBEtFGdi9bz86l6xh48nEZTtY5sboGNv5+IZsef41ELE7Z7HKOuXo2kb767SGfqMAlb8Rq66l6c03b29Q38P5jr3brAt/9zgaem/Ud4vWNxA7UAbB94V9Y+i+/YebjtzB05klZTiiZolkokjfi9VEs2M63fMJp3F+bmUBdEN13gGfP/TYNO6sPljc0/XCKHahjwcU3sf+9rVlMKJmUUoGbWT8ze8TM1pjZajM7PV3BRNIt0q93u3PZQ72LOKobH32v/90LJOobk66PR2OsvOPRDCaSbEr1CPxO4Dl3Hw+cAKxOPZLIkWGBABO/8SmCRZE2txv96bMzE6gL3nvw5TYvZ+CNMSofXZTBRJJNXS5wM+sLnAn8GsDdo+6+N025RI6IKd/9LP0njyZYVPDxFWYEexVw9kM3EWq5rhuJ1x9+IbGWEo3xDCSR7iCVI/DRQBXwWzN7y8zuNrPilhuZ2TVmVmFmFVVVVSnsTiR1ocIIF7xyB1O/fxWFpf2xQAALBhg+91TmLLqD4XNOzXbENpWeMRkLtz33oP/kUbz9w/t4/vx/ZsElN1P5yCskGg8/+UpyX5fngZtZOfAGMMPdl5jZnUC1u9+U7DGaBy7dTaw+SjASwgK58X5+9foP+MMJX271kr7QNJcdHAsEDh6th/oUUTCghDmv/JTeI0ozmFbSJdk88FS+a7cAW9x9SfPXjwCavyQ5JVQYyZnyBig5poxTfvJVQr0OH+YJFkZIxOMkorGPDbXE9tdRu7mKZ8/+Fom4hld6ki5/57r7dmCzmY1rXjQTWJWWVCKS1PivXMjs5/+TsvOnESyMEIiEGHTaBPpOGAHxRKuP8USC+l372PLMklbXS25K9USerwMPmFkEeA/4u9QjiUh7SmdMZvYztx382t25t6Dte5LG9tdR+cgiRnxi+pGOJxmSUoG7+9tA7p1PLdLTuONJjr4PlewyApKbcmfwT0SSskCAkuPK2twmVFxI6ZnHZyiRZIIKXKSHOP6fL2/7TFOHYz4/O3OB5IhTgYv0EMd8fjbD5px6WIlbINB0ktLvv0+k5LBTNSSHqcBFeggLBDj7wX9h+n9/i/5TxhCIhAgVFzLqM2cxb/Evuv1JStJ5upysSA9igQBjr5jJ2CtmZjuKZICOwEVEcpSOwEUkJfv21vHu6ircnbHjBjHgqF7ZjpQ3VOAi0iX1dY38+heLeevNzQRDQcCJxxJMmjqUa66dTnHv7ntVx55CQygi0mnxeILbbprPsjc309iYoL6ukfq6GI2NCVa8tZUfXv8c0QZdAfFIU4GLSKctW7KZrVv2EWs8/OzPWCzB7qpa3ni1MvPB8owKXEQ67cVn1tJQn/wIu6Ehxvyn276BtKROBS4inbZ3d12721TvTX7rN0kPFbiIdNpRg9qfadJvQFEGkuQ3FbiIdNp5c8dTUJh8EltBQYjZn5iQwUT5SQUuIp12wslljBo7gHAkeNi6UDjA0WUlnDpjZBaS5RcVuIh0WiAY4Ds3z2LG2WMIR4IU9QpT1CtMOBzglOkjufHW2YTCh5e7pFeXb2rcFbqpsUjPU1sT5b13d+IOo8ceRe8SncCTbsluaqwzMUUkJb2KI0yeOjTbMfKShlBERHKUClxEJEepwEVEcpQKXEQkR6nARURyVMoFbmZBM3vLzP6YjkAiItIx6TgCvxZYnYbnERGRTkipwM1sGDAXuDs9cUREpKNSPZHnDuC7QJ/Uo8iREo8n+NPC93jm8ZVUbT9AOBLklBkjmfupSZQOKcl2PBHpoi4fgZvZPGCHuy9tZ7trzKzCzCqqqqq6ujvpong8we0/fIn773qTbVuqicUS1NU28uqCDdx03dOsW70j2xFFpItSGUKZAVxoZpXAQ8C5ZnZ/y43c/S53L3f38kGDBqWwO+mK559czbpVO2hoiH9seSLhNNTHuOPWl4k1xpM8WkS6sy4XuLvf4O7D3H0U8FngJXe/Mm3JJGXuznNPrCIaTV7Q8ViCpUs2ZzCViKSL5oH3YDX7o9QciLa5TX19jPVrNbQlkovScjVCd18ILEzHc0n6BMMBOnK54HBI120WyUU6Au/BiorClI3o1+Y2BYUhTjx1WGYCiUhaqcB7uEuumEqkoPUj7EDQKB3Sh2PG6c1lkVykAu/hpk4bxiVXTCUcDhIK/fWfu7AwROnRffjOzTMxsywmFJGu0h158sD5F02k/PQRvPTcOirX76KoOMKMc8Yw9eQyAkH9DBfJVSrwHLGrqoanH1vB669spL4+Rr/+RfztvPGcO2ccBQXt/zMOHNybT3/+pAwkFZFMUYHngM2Ve7j1xueJNsSIx5tmlezZVctjD/6F1xa+x023nU9hUTjLKUUk0/T7czfn7tz5o4XU1TYeLO+PRKNxtm+t5n/uXZaldCKSTd2+wBPxBO+/t5sN66raPSmlJ1q7agfV1fVJ18caE7z28gYaGmIZTCUi3UG3HUJxd+Y/vZYnf/8OjdE4FjBisQQnnFRG6dA+VCzeRDQaZ8So/lxw8UQmTDk625GPiMoNu4jHEm1uEwgE2LFtP8NH9c9QKhHpDrptgT90z1Jeeu5doi2OLCve2PSxr/fsrmXNig85Y+ZYrvr7aQenxNXVRtlcuRcLwMjRA4h04I2+trg761btYPlbW0kknOMmDub4E4ce8Vkc4VAQC7Q9zS+RSBAO62xKkXzTLQt8+9ZqFjy7jsY2LsJ0kENDQ4zXFmzg2PGDOOmUYdx/959ZvKjy4LznRMKZNWccl3xuKsEuFO6uqhp+fMsCdlXV0FDf9ANlwbNrKSwK853vzzyiR77HnzyUB3/b5hV7KS6OUDpUl2QXyTfdcgx84fPrSMTbHjZoqaEhxhP/8w633vgCi1/ZSGM0Tl1tI3W1jTTUx5j/9Bp+/h+vdOjaIIeKRuP82w3Psf2D6oPlDVBfF2Pv7jp+9L3n2be3rlPP2RmDSvsweeoQwuHW/6kiBUEuvvwEnYwjkoe6ZYFv+6D6sBkXHbF9azVbt+ylsfHw8o82xFn1znbWruzcDQzefK2SmgNREonW8zRG47z4zNpOZ+2Mr3zrDEYfO5CCwhA093QwFCAcCTJ73njOmnXMEd2/iHRP3XIIpd+AIsygkwfLuENjNPmRe0NDjPl/XMP4yaUdfs5FC9Z/7Mi7pcbGBH96aQOXXDG1M1E7pbAozI23zmbdqh0sWrCe6n31lA3ryznnj6N0iIZORPJVtyzwM2cdw+uvbCTa0Lk7xQQCkGhr5MVhx/b9nXrOuprGdrfJxBQ+M2PcpFLGTer4Dx8R6dm65RDKmGMHctyEwYQjHZ9ZUVAQok9JYbvb9e1fdPDzRDzB/ur6Ngt4xJj+BNqZBXJ0Wd8O5xQRSZdueQRuZlx74zn85heL+fPi9wkGA3jCcXdicScUChycoWIBIxwOcN68cfTpW8Sj97+V9BZiBYUhZl5wHPV1jTz1yHIWPLuOWGOceMI5dvwgLr3yRI6bMPhjj5k9bwJvvvZ+m8855+KJ6f0LEBHpgG5Z4ACRSJCvfOsMPrvnZJYv20osFmfkmAGUDinhtZc38NpLG4g2xBk5dgAXXDSR0cccRV1dIy88tZo9u2oPe9MxHA4wdFgJ4yYO5pbvPsuO7fs/9mbn2pU7+K+bX+QfvnUG5aeNOLh85JgBzJwzjgXPrj1sSCdSEGTy1CGceMrwI/uXISLSCuvstLpUlJeXe0VFxRHdx949dfzyvxaxcf0uAgHDgHg8wZSTyrjmmzN46uHlvPDU6lZnqkDTUMzP772UgsK/XhzK3Vm8aCN/eOgddlXVgEFx7wLmfnIi580dr0uyisgRZWZL3b38sOU9rcA/sn1rNevXVGEBY8KUoxlwVC8S8QT/eNXvqatN/sZkQWGIK788jTNbmZrn7tTsj5JIJOhdUtju2LiISDokK/BuO4SSqqOHlnD00JKPLaupiRJrbHtmS0N9jE0bd7e6zszoXVKQtowiIqnIq9/9I5Eg8SQn5HwkEDCKiiMZSiQi0nV5VeAFhWGObecGvqFQgFNnjMxQIhGRrsurAge49MoTiSSZXx4OBzhu4mCGjdRlWUWk++tygZvZcDN72cxWmdlKM7s2ncGOlOMmDuaab84gUhBsurYITcMmkUiQcZNK+cb1Z2U5oYhIx6TyJmYM+La7LzOzPsBSM5vv7qvSlO2ImTZ9JMefNJQ3Xq1kc+UeioojnDJ9pG6IICI5pcsF7u7bgG3Nn+83s9VAGdDtCxyaxsPPOu/YbMcQEemytIyBm9ko4ERgSSvrrjGzCjOrqKqqSsfuRESENBS4mfUGHgW+6e7VLde7+13uXu7u5YMGtT0DREREOi6lAjezME3l/YC7P5aeSCIi0hGpzEIx4NfAane/PX2RRESkI1I5Ap8BXAWca2ZvN/+Zk6ZcIiLSjlRmobzGwTs09nzuzrIlm/njoyvYXLmXQNA4/qShfOLSKYwcMyDb8UQkD/XYi1mlk7tz76+W8PrCjX+9e08jVCzexF8qPuDvr53OKTNGdem54/EEf6n4gA1rqwhFghx/UhljjxuYvvAi0mOpwDvgrTe3fLy8m7lDNBrnrjtfZ9ykUvr2K0ryDK3buH4Xt//wJaLRGPV1MczgmcdXMnRYX75907mUdPL5RCS/5N21ULriqUdXtHvj4oUvvNup59y54wC33fQC1fvqqa9rem53iDbE2Vy5hx997wXi8bbu0Cwi+U4F3gGbK/e0ub4xGmfNig879ZzP/mHlwft6thSPO3t21fKXpR906jlFJL/k1RCKu7Pi7W08/egK3t+4m2AwwMmnjeCCiycedvOHQwUDRvJ7+DQJhVu/wmEyixdVEo8nvzZ5fX2MVxds4CTdb1NEksibI3B35/67/8zPb1vI6hUfUlvTyP7qBhYtWM9N1/2RFW9vTfrY408uw9qYb1NQGOK0M0Z1Kk+yo+9D1R6Iduo5RSS/5E2BL12ymVdf3EBDizvLJ+JOtCHOz257hdqa1gvzE5dOJpzsCNuaboQ8rZM3gRhY2rvN9cGgMWKMro4oIsnlTYH/8ZG234h0d157aUOr60aMHtB0DfFIkPAhN4MoLAxRUlLIDbfOTnqTiGTmXDyJgoLkjwkEA8yaM65Tzyki+SVvxsCT3aj4I9GGOCv/so3Zn5jQ6vpp00cyblIpC194lzXLtxOKBDn1jFFMmz6y0+UNMP3s0bz+ynusX1NFtMVwSqQgyLxLJlM6JPm4vIhI3hR406Vb2rmhcbDtX0hK+hZy4WVTuPCyKSnnCQYDfPv7M3nm8ZU8/+RqGhpieMIZdHRvPnX5VE7RfTlFpB15U+ATphzN8reSv1FZWBhi2vQRGUzUdAPlCy+bwrxPTaJ6Xz3BUIA+JYUZzSAiuStvxsAv/PSUNoc6wgVBpk3PzlFvIBig34BeKm8R6ZS8KfDjJgzmc1+eRjgSJBj668suKAzRu08B1/9wdvKZJiIi3VDeDKEAnD37WCadMIQXn1nDulU7CIWDTD9rNKefOZrConC244mIdEpeFTjAoNLeXP535dmOISKSsrwZQhER6WlU4CIiOUoFLiKSo1TgIiI5ytzbPjsxrTszqwLez9gO02MgsDPbIbIkn1876PXn8+vvbq99pLsParkwowWei8yswt3zctpKPr920OvP59efK69dQygiIjlKBS4ikqNU4O27K9sBsiifXzvo9efz68+J164xcBGRHKUjcBGRHKUCFxHJUSrwVpjZcDN72cxWmdlKM7s225mywcyCZvaWmf0x21kyzcz6mdkjZrbGzFab2enZzpQpZnZd8/f9CjN70Mx69IXqzew3ZrbDzFYcsmyAmc03s3ebP3bLO4yrwFsXA77t7hOB04CvmdnELGfKhmuB1dkOkSV3As+5+3jgBPLk78HMyoBvAOXuPhkIAp/Nbqoj7h7g/BbLrgcWuPuxwILmr7sdFXgr3H2buy9r/nw/Tf95y7KbKrPMbBgwF7g721kyzcz6AmcCvwZw96i7781qqMwKAUVmFgJ6AcnvRdgDuPsioOVdzy8C7m3+/F7g4kxm6igVeDvMbBRwIrAky1Ey7Q7gu0AiyzmyYTRQBfy2eQjpbjMrznaoTHD3D4AfA5uAbcA+d38hu6myotTdtzV/vh0ozWaYZFTgbTCz3sCjwDfdvTrbeTLFzOYBO9x9abazZEkIOAn4v+5+IlBDN/0VOt2ax3ovoumH2FCg2MyuzG6q7PKmudbdcr61CjwJMwvTVN4PuPtj2c6TYTOAC82sEngIONfM7s9upIzaAmxx949+63qEpkLPB7OAje5e5e6NwGPA9CxnyoYPzWwIQPPHHVnO0yoVeCvMzGga/1zt7rdnO0+mufsN7j7M3UfR9AbWS+6eN0dh7r4d2Gxm45oXzQRWZTFSJm0CTjOzXs3/D2aSJ2/gtvAkcHXz51cDT2QxS1Iq8NbNAK6i6cjz7eY/c7IdSjLq68ADZvYOMBX4UXbjZEbzbx2PAMuA5TR1RE6cVt5VZvYgsBgYZ2ZbzOxLwG3AeWb2Lk2/ldyWzYzJ6FR6EZEcpSNwEZEcpQIXEclRKnARkRylAhcRyVEqcBGRHKUCFxHJUSpwEZEc9b8AEVbAEWmLsQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "label_y = y_pred.copy()\n",
    "label_y[label_y==-1] = 2;\n",
    "\n",
    "# plot data\n",
    "plt.scatter(train_data[:,0], train_data[:,1], marker='.',  s = 300,\n",
    "            c=label_y, cmap=plt.cm.Spectral)\n",
    "plt.title(\"Data\")\n",
    "plt.savefig(\"fig-res_perceptron_res_data.pdf\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 参考资料\n",
    "* [感知机（Python实现）](http://www.cnblogs.com/kaituorensheng/p/3561091.html)\n",
    "* [Programming a Perceptron in Python](https://blog.dbrgn.ch/2013/3/26/perceptrons-in-python/)\n",
    "* [损失函数、风险函数、经验风险最小化、结构风险最小化](https://blog.csdn.net/zhzhx1204/article/details/70163099)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}