OpenI
/
machinelearning_notebook

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 感知机\n",
    "\n",
    "感知机（Perceptron）是二分类的线性分类模型，输入为实例的特征向量，输出为实例的类别（取`+1`或`-1`）。感知机对应于输入空间中将实例划分为两类的分离超平面，感知机旨在求出该超平面。为求得超平面导入了基于误分类的损失函数，利用梯度下降法对损失函数进行最优化。感知机的学习算法具有简单而易于实现的优点，感知机预测是用学习得到的感知机模型对新的实例进行预测的，因此属于判别模型。\n",
    "\n",
    "![perceptron](images/perceptron.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 生物学、心理学解释\n",
    "\n",
    "心理学家唐纳德·赫布（Donald Olding Hebb）于1949年提出`赫布理论`，该理论能够解释学习的过程中脑中的神经元所发生的变化。赫布理论描述了突触可塑性的基本原理，即突触前神经元向突触后神经元的持续重复的刺激，可以导致突触传递效能的增加：\n",
    "> 当细胞A的轴突足以接近以激发细胞B，并反复持续地对细胞B放电，一些生长过程或代谢变化将发生在某一个或这两个细胞内，以致A作为对B放电的细胞中的一个，导致突触传递效能的增加。\n",
    "\n",
    "![neuron_cell](images/neuron_cell_cn.png)\n",
    "\n",
    "\n",
    "心理学家弗兰克·罗森布拉特(Frank Rosenblatt)于1957年提出了`感知机`，它作为简化的数学模型解释大脑神经元如何工作：它取一组二进制输入值（附近的神经元），将每个输入值乘以一个连续值权重（每个附近神经元的突触强度），并设立一个阈值，如果这些加权输入值的和超过这个阈值，就输出1，否则输出0，这样的工作原理就相当于神经元是否放电。\n",
    "\n",
    "\n",
    "感知机并没有完全遵循赫布理论，**但通过调输入值的权重，可以有一个非常简单直观的学习方案：给定一个有输入输出实例的训练集，感知机应该「学习」一个函数：对每个例子，若感知机的输出值比实例低太多，则增加它的权重；否则若设比实例高太多，则减少它的权重。**\n",
    "\n",
    "\n",
    "感知机模仿的是生物神经系统内的神经元，它能够接受来自多个源的信号输入，然后将信号转化为便于传播的信号在进行输出(在生物体内表现为电信号)。\n",
    "\n",
    "![neuron](images/neuron.png)\n",
    "\n",
    "* dendrites - 树突\n",
    "* nucleus - 细胞核\n",
    "* axon - 轴突\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 感知机模型\n",
    "\n",
    "假设输入空间(特征向量)为$\\mathbf{X} \\subseteq \\mathbb{R}^D$，输出空间为$\\mathbf{Y} \\in \\{-1, +1\\}$。输入$x \\in \\mathbf{X}$ 表示实例的特征向量，对应于输入空间的点；输出$y \\in \\mathbf{Y}$表示示例的类别。由输入空间到输出空间的函数为\n",
    "\n",
    "$$\n",
    "f(x) = sign(w x + b)\n",
    "$$\n",
    "\n",
    "称为感知机。其中，参数$w$叫做权值向量，$b$称为偏置。$w·x$表示$w$和$x$的内积。$sign$为符号函数，即\n",
    "![sign_function](images/sign.png)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 几何解释    \n",
    "感知机模型是`线性分类模型`，感知机模型的假设空间是定义在特征空间中的所有线性分类模型，即函数集合$\\{ f | f(x)=w·x+b\\}$。线性方程 $w·x+b=0$对应于特征空间$\\mathbb{R}^D$中的一个超平面$S$，其中$w$是超平面的法向量，$b$是超平面的截距。这个超平面把特征空间划分为两部分，位于两侧的点分别为正负两类。超平面$S$称为分离超平面，如下图：\n",
    "![perceptron_geometry_def](images/perceptron_geometry_def.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 感知机学习策略\n",
    "\n",
    "假设训练数据集是 **线性可分**，感知机学习的目标是求得一个能够将训练数据的正负实例点完全分开的分离超平面，即最终求得参数$w, b$。这需要一个学习策略，即定义（经验）损失函数并将损失函数最小化。\n",
    "\n",
    "损失函数的一个自然的选择是误分类的点的总数，但是这样得到的损失函数不是参数$w，b$的连续可导函数，不宜优化。损失函数的另一个选择是误分类点到分类面的距离之和。\n",
    "\n",
    "首先，对于任意一点$x$到超平面的距离（[参考资料](https://www.cnblogs.com/graphics/archive/2010/07/10/1774809.html)）为\n",
    "$$\n",
    "\\frac{1}{||w||} | w \\cdot x + b |\n",
    "$$\n",
    "\n",
    "其次，对于误分类点$(x_i,y_i)$来说 $-y_i(w \\cdot x_i + b) > 0$\n",
    "\n",
    "这样，假设超平面$S$的总的误分类点集合为$\\mathbf{M}$，那么所有误分类点到$S$的距离之和为\n",
    "$$\n",
    "-\\frac{1}{||w||} \\sum_{x_i \\in \\mathbf{M}} y_i (w \\cdot x_i + b)\n",
    "$$\n",
    "不考虑$1/||w||$，就得到了感知机学习的损失函数\n",
    "$$\n",
    "L = - \\sum_{x_i \\in \\mathbf{M}} y_i (w \\cdot x_i + b)\n",
    "$$\n",
    "\n",
    "### 3.1 经验风险函数\n",
    "\n",
    "给定数据集$\\mathbf{T} = \\{(x_1,y_1), (x_2, y_2), ... (x_N, y_N)\\}$，其中\n",
    "* $x_i \\in \\mathbb{R}^D$\n",
    "* $y_i \\in \\{-1, +1\\}，i=1,2...N$\n",
    "\n",
    "感知机$sign(w·x+b)$学习的损失函数定义为\n",
    "$$\n",
    "L(w, b) = - \\sum_{x_i \\in \\mathbf{M}} y_i (w \\cdot x_i + b)\n",
    "$$\n",
    "其中$\\mathbf{M}$为误分类点的集合，这个损失函数就是感知机学习的[经验风险函数](https://blog.csdn.net/zhzhx1204/article/details/70163099)。\n",
    "\n",
    "显然，损失函数$L(w,b)$是非负的。\n",
    "* 如果没有误分类点，那么$L(w,b)$为0\n",
    "* 误分类点数越少，$L(w,b)$值越小\n",
    "\n",
    "该损失函数在误分类时是参数$w,b$的线性函数，在正确分类时该损失函数是0。因此，给定训练数据集$\\mathbf{T}$，损失函数$L(w,b)$是$w,b$的连续可导函数。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 感知机学习算法\n",
    "\n",
    "\n",
    "最优化问题：给定数据集$\\mathbf{T} = \\{(x_1,y_1), (x_2, y_2), ... (x_N, y_N)\\}$，其中\n",
    "* $x_i \\in \\mathbb{R}^D$\n",
    "* $y_i \\in \\{-1, +1\\}，i=1,2...N$\n",
    "\n",
    "求参数$w,b$，使其成为损失函数的解，其中$\\mathbf{M}$为误分类数据的集合：\n",
    "\n",
    "$$\n",
    "min_{w,b} L(w, b) =  - \\sum_{x_i \\in \\mathbf{M}} y_i (w \\cdot x_i + b)\n",
    "$$\n",
    "\n",
    "感知机学习是误分类驱动的，具体采用随机梯度下降法：\n",
    "* 首先，任意选定$w_0$、$b_0$，\n",
    "* 然后用梯度下降法不断极小化目标函数\n",
    "   - 极小化的过程不是一次性的把$\\mathbf{M}$中的所有误分类点梯度下降\n",
    "   - 而是一次随机选取一个误分类点使其梯度下降。\n",
    "\n",
    "假设误分类集合$\\mathbf{M}$是固定的，那么损失函数$L(w,b)$的梯度为\n",
    "$$\n",
    "\\triangledown_w L(w, b) = - \\sum_{x_i \\in \\mathbf{M}} y_i x_i \\\\\n",
    "\\triangledown_b L(w, b) = - \\sum_{x_i \\in \\mathbf{M}} y_i \\\\\n",
    "$$\n",
    "\n",
    "随机选取一个误分类点$(x_i,y_i)$,对$w,b$进行更新：\n",
    "\\begin{eqnarray}\n",
    "w & = & w + \\eta y_i x_i \\\\\n",
    "b & = & b + \\eta y_i\n",
    "\\end{eqnarray}\n",
    "\n",
    "式中$\\eta$（0 ≤ $ \\eta $ ≤ 1）是学习速率（步长）：\n",
    "* 步长越大，梯度下降的速度越快，更能接近极小点。如果步长过大，有可能导致跨过极小点，导致函数发散；\n",
    "* 如果步长过小，有可能会耗很长时间才能达到极小点。\n",
    "\n",
    "> 直观解释：当一个实例点被误分类时，调整$w,b$，使分离超平面向该误分类点的一侧移动，以减少该误分类点与超平面的距离，直至超越该点被正确分类。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 算法\n",
    "\n",
    "\n",
    "**输入：**\n",
    "* $\\mathbf{T}=\\{(x_1,y_1),(x_2,y_2), ..., (x_N,y_N)\\}$, 其中$x_i \\in \\mathbf{X}=\\mathbb{R}^D$，\n",
    "* $y_i \\in \\mathbf{Y} = {-1, +1}，i=1,2...N$，\n",
    "* 学习速率为η\n",
    "\n",
    "**输出：**\n",
    "* $w$, $b$; \n",
    "* 感知机模型$f(x)=sign(w·x+b)$\n",
    "\n",
    "**处理过程:**\n",
    "1. 初始化$w_0$,$b_0$\n",
    "2. 在训练数据集中选取$(x_i, y_i）$\n",
    "3. 如果$y_i(w * x_i+b)≤0$\n",
    "           \n",
    "    $w = w + η y_i x_i$\n",
    "    \n",
    "    $b = b + η y_i$\n",
    "\n",
    "4. 如果所有的样本都正确分类，或者迭代次数超过设定值，则终止\n",
    "5. 否则，跳转至（2）\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. 示例程序\n",
    "\n",
    "生成数据："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2 2 2 2 2 1 1 2 1 1 1 1 2 1 1 1 1 2 2 2]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAEICAYAAABGaK+TAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAha0lEQVR4nO3deXxddZ3/8dfnbkma7rQNbboDLd1YSsrSIltLB0oFFFBBEEcdRsefIuIo4CAOjg6OiqDy+Dn8UEHgByObgKy1pZRKBdKCdC8tTRe6pWvabHf7zB8JtaS5WW/vzc19Px+PPpKcc+4979um75x87znna+6OiIjknkC2A4iISMeowEVEcpQKXEQkR6nARURylApcRCRHqcBFRHKUClxEJEepwKXbMbMKM6s1s/1mttfMXjezL5tZq9/vZjbSzNzMQpnIKtIZKnDprj7u7r2AEcAdwHeA32Q3kkh6qcClW3P3fe7+DPBp4Fozm2hmF5nZ22ZWZWabzOz7hzxkQePHvWZ2wMzOMLNjzGyeme0ys51m9rCZ9c30axFpSgUuecHd3wQ2Ax8DqoHPAX2Bi4CvmNmljZue1fixr7v3dPdFgAH/CQwBxgHDgO9nKrtIKipwySdbgP7uPt/dl7p70t3fBR4Bzk71IHdf6+5z3L3e3SuBO1vaXiRT9EaN5JNSYLeZnUbDuPhEIAIUAI+lepCZlQB303D03ouGA589RzytSCt0BC55wcym0FDgC4H/DzwDDHP3PsCvaRgmAWju9pw/alw+yd17A1cfsr1I1qjApVszs95mNht4FHjI3ZfScBS9293rzOxU4KpDHlIJJIHRhyzrBRwA9plZKfCvmUkv0jLT/cCluzGzCqAEiNNQxiuAh4Bfu3vCzC4Hfgb0B14FKmh40/LqxsffDnwFCAMXAPuB3wNjgbXAg8AN7j40c69K5HAqcBGRHKUhFBGRHKUCFxHJUSpwEZEcpQIXEclRGb2QZ8CAAT5y5MhM7lJEJOctXrx4p7sPbLo8owU+cuRIysvLM7lLEZGcZ2YbmluuIRQRkRylAhcRyVEqcBGRHKW7EYqkibuzfcG7bPjjQhJ1UQZNncjIK84mVBjJdjTppjJ6KX1ZWZnrTUzpjmq27ealmf/KgYptxKvrwZ1QzyIsGGD6U7cz+JyTsh1RcpiZLXb3sqbLNYQi0knJRIIXzv4G+1ZtIn6gDhoPiuIHaontq+bPs7/LvtWbspxSuiMVuEgnbX7uDWq27sbjiWbXJ+qjvPvjRzKcSvKBClykk9Y+OIf4gdqU6z2RpOKJBSnXi3SUClykk2JV1a1uk6yPZSCJ5BsVuEgnDTh1HIGCcIvb9Bo9JENpJJ+owEU66fh/no1Z6ikyQ8WFTPr2pzOYSPKFClykk4qHDmTKT79MsEfBYeuCPQoo+dgkjrnm/Cwkk+5OF/KIpMG4f7mE3seWsuS2+9lVvhrMKBrUl4nf+hTj/s8nCASD2Y4o3ZAKXCRNSmeWUTqzjHhdFI/FGy7kaWFoRaSzVOAiaRYqjIAun5cM0Bi4iEiOUoGLiOQoFbiISI5SgYuI5CgVuIhIjlKBi4jkKBW4iEiOUoGLiOQoFbiISI5SgYuI5CgVuIhIjmq1wM3st2a2w8yWHbLsJ2a2yszeNbOnzKzvEU0pIiKHacsR+P3ABU2WzQEmuvsJwBrg5jTnEhGRVrRa4O6+ANjdZNnL7h5v/PKvwNAjkE1ERFqQjjHwLwAvpOF5RESkHTpV4Gb2XSAOPNzCNteZWbmZlVdWVnZmdyIicogOF7iZfR6YDXzW3T3Vdu5+r7uXuXvZwIEDO7o7ERFpokMz8pjZBcC3gbPdvSa9kUREpC3achrhI8AiYKyZbTazLwK/AnoBc8zsHTP79RHOKSIiTbR6BO7uVzaz+DdHIIuIiLSDrsQUEclRKnARkRylAhcRyVEqcBGRHKUCFxHJUSpwEZEcpQIXEclRKnARkRylAhcRyVEqcBGRHKUCFxHJUR26G6GIZE7dzn2sfXAO+1ZuoKikH6M/O4O+xw/PdizpAlTgIl3Yil88SflN/w/MSNTWY6Egy372GMNmn85ZD91CMBLOdkTJIg2hiHRR6/8wn/Jb7iNRFyVRWw+AxxMk6qJseu4NXr/uziwnlGxTgYt0Qe5O+c33kaipb3Z9orae9//nFWq27spwMulKVOAiXdD+dVuo3b67xW0sGGDTs4sylEi6IhW4SBcUr64jEAq2uI3HE8Sr6zKUSLoiFbhIF9Rz1NEko/EWtwlEwvSdMDIzgaRLUoGLdEGR3sWMuOxjBMKpTxQL9+rBkBmTM5hKuhoVuEgXddrPv0rR4P4Emp4qaEaouJBzH7sNC+i/cD7Tv75IF1U4oA+XLPlvxn31EkK9igiEQ1g4yIhLpzH7r/dQMnVCtiNKlpm7Z2xnZWVlXl5enrH9iXQXnkwS219DqEdhi8Mq0j2Z2WJ3L2u6XN8JIjnAAgEifXpmO4Z0MRpCERHJUa0WuJn91sx2mNmyQ5b1N7M5ZvZe48d+RzamiIg01ZYj8PuBC5osuwmY6+7HAXMbvxYRkQxqtcDdfQHQ9JreS4AHGj9/ALg0vbFERKQ1HR0DL3H3rY2fbwNKUm1oZteZWbmZlVdWVnZwdyIi0lSn38T0hvMQU56L6O73unuZu5cNHDiws7sTEZFGHS3w7WY2GKDx4470RRIRkbboaIE/A1zb+Pm1wNPpiSMiIm3VltMIHwEWAWPNbLOZfRG4AzjfzN4DZjR+LSIiGdTqlZjufmWKVdPTnEVERNpBV2KKiOQoFbiISI5SgYuI5CgVuIhIjlKBi4jkKBW4iEiOUoGLiOQozcgjInIEJBMJPnjhTVb88ikObNhOjyEDGPfVSxh+yTQCoWBa9qECFxFJs3hdlDkX3sTOxWuIH6gFoGrNZnaWr6bP8cO4cN6dhHsWdXo/GkIREUmzN2+4h8o3Vx4s7w/FD9Syd2kFC7/wX2nZjwpcRCSNolXVrH3gZRK10WbXJ+qjbHp2EbXbm86T034qcBGRNNr51moCkXCL2wQiYba/trTT+1KBi4ikkSeTYG3YLuU0OG2nAhcRSaMBp4whWd/88MmHktEYg84Y3+l9qcBFRNKooH9vRl5xDoHC5odRApEQg2dMpnho56eY1GmEItIhe1dUsPHp14nX1NH/xGMazm8Oq1IAzrjnevYur2Df6k3Eq+sOLg8VF1I8bBBn/f7mtOzHPB0DMW1UVlbm5eXlGdufiKRfbH8N8674d7a/tpRkLI7HE4R7FWGhIOc9/n0Gn3tytiN2CYlojIo/zGf5L56kZvNOikr6Me5rn2D0lecRKipo13OZ2WJ3LztsuQpcpHXJRILY/lrCPYvSdhVdLnJ3Xjj3m1S+sZJkfeyw9aEeBVy06Ff0nzQ6C+m6r1QFrjFwkRZUf1DJX677GQ/1ns2jR1/Gg70uYsHn/pP9FduyHS0rdr61ml2L1zRb3gDx2ijv3P77DKfKXxqwEklhf8U2np3yFaL7qvF44uDy9x+Zx8ZnFzH79V/Sd9yILCbMvHUP/5l4bX3qDdzZ+PTreDKJBXR8eKTpb1gkhdc+/2Oiew58pLwBPJEkVlXD/Cv/I0vJsqd+dxUkWx529WSSZCyeoUT5TQUu0owDG7az881VDRdlNMedqrUfsHvp+5kNlmX9TzyGYCtvwBUe1ZtgQSRDifKbClykGXtXbiBQ0Mrl0MEge5dXZCZQF3Hc5y9o8RLCYGGE8V//ZAYT5bdOFbiZ3WBmy81smZk9YmaF6Qomkk2h4sLWr3W2xu3ySOGAPpx611ebPQoPFoTpPWYoE264PAvJ8lOHC9zMSoGvA2XuPhEIAp9JVzCRbBp0+njMWr6hRTKeYPB5+XfO8/HXzebcx26j34mjCURChHoUEO7Vg3Ff+wQX/eUXhHrk1w+1bOrsWSghoMjMYkAPYEvnI4lkXyAc4sRbr+Ht2+7/yJV0Hwr1KGT8DZcRLu78Tflz0bBZpzFs1mnU7dxHvLaeHkf311WYWdDhI3B3/wD4KbAR2Arsc/eXm25nZteZWbmZlVdWVnY8qUiGTbjhcsZ97RMECyMHx8MDkTDBwgjHXDuTyf/++ewG7AIKB/Sh57BBKu8s6fCVmGbWD3gC+DSwF3gMeNzdH0r1GF2JKbmo+oNK1j44hwPvb6V42CCO/dxMeo4oyXYsySOprsTszI/NGcB6d69s3MGTwFQgZYGL5KLi0oGceNNV2Y4hcpjOnIWyETjdzHpYw7s904GV6YklIiKt6cwY+BvA48ASYGnjc92bplwiItKKTr3z4O63AbelKYuIiLSDrsQUEclRKnARkRylAhcRyVEqcBGRHKUCFxHJUSpwEZEcpQIXEclRKnARkRylAhcRyVG6B6R0SDIW58DGHQQiIYqHDmx18gMRST8VuLRLoj7KOz94kJX3PI3HE3giSVFJP076/rUcd+0/ZDueSF5RgUubJWNxXpzxr+xasoZEbfTg8gMbtvPXr97NvjWbKPvhl7KYUCS/aAxc2uy9+19i99trP1LeH4rX1LPi50+wb/WmLCQTyU8qcGmzZXc+Rrzm8PkhP5SMJ1h5zx8zF0gkz6nApc2qN+5ocb3HE+xZuj5DaUREBS5tFioubHWbgqN6ZSCJiIAKXNrhmKtnEIikft871LOI474wK4OJRPKbzkKRNpt44xW897sXSUbjh60LREL0Gj2Y0n84bOLsLql+z37WPvASO5e8R0Hfnoz6zHkMOmO8zmeXnKIClzYrLh3IrFd/zpzZ3yW69wDxmnoCwQAWCjKgbAzTn7qdQDCY7ZitWvvQHF7/5zvBjERNPQSM9373Iv0mjeL85++goG/PbEcUaRNz94ztrKyszMvLyzO2PzkyPJlky9wl7HxrNYFwiKEXnkq/iaOyHatNts5/hzkX3UKitv6wdYFImAGnjuWiBXdnIZlIama22N0P+/VWR+DSbhYIUHp+GaXn58ZwyaGW/Ntvmy1vgGQ0xu4la9m5eA0DThmT4WTtE6+tZ/0f5rPxqYUk4wlKZ5Zx7LUzifTRbw/5RAUueSNeU0flm6ta3qaung1PvtalC3z3u+t4cca3SNTFiB+oBWDb/L+x+N9+y/SnbmfI9MlZTiiZorNQJG8k6qJYsJVv+aQT21+TmUAdEN13gBfOu5H6nVUHyxsafjjFD9Qy99Jb2f/+liwmlEzqVIGbWV8ze9zMVpnZSjM7I13BRNIt0rdnq+eyh3oWcVQXPvpe+/uXSdbFUq5PROMsv+uJDCaSbOrsEfjdwIvufjxwIrCy85FEjgwLBBj/9U8SLIq0uN2oT52TmUAd8P4jr7R4OwOPxal4YkEGE0k2dbjAzawPcBbwGwB3j7r73jTlEjkiJn37M/SbOIpgUcFHV5gR7FHAOY/eSqjpui4kUXf4jcSaSsYSGUgiXUFnjsBHAZXA78zsbTO7z8yKm25kZteZWbmZlVdWVnZidyKdFyqMcOGrd3HS966hsKQfFghgwQDDLjqNWQvuYtis07IdsUUlZ07Ewi2fe9Bv4kje+cGDvHTBd5h72W1UPP4qydjhF19J7uvweeBmVgb8FZjm7m+Y2d1AlbvfmuoxOg9cupp4XZRgJIQFcuP9/Kq1H/DHE7/U7C19oeFcdnAsEDh4tB7qVURB/97MevXn9BxeksG0ki6pzgPvzHftZmCzu7/R+PXjgM5fkpwSKozkTHkD9D62lFN/9hVCPQ4f5gkWRkgmEiSj8Y8MtcT311KzqZIXzvkmyYSGV7qTDn/nuvs2YJOZjW1cNB1YkZZUIpLS8V++mJkv/RelF0whWBghEAkx8PRx9Bk3HBLJZh/jySR1u/ax+fk3ml0vuamzF/J8DXjYzCLA+8A/dj6SiLSmZNpEZj5/x8Gv3Z0HClqekzS+v5aKxxcw/ONTj3Q8yZBOFbi7vwPk3vXUIt2NO57i6PtQqW4jILkpdwb/RCQlCwToPaa0xW1CxYWUnHVChhJJJqjARbqJE75zZctXmjoc+7mZmQskR5wKXKSbOPZzMxk667TDStwCgYaLlP7wPSK9D7tUQ3KYClykm7BAgHMe+Tem/vc36TdpNIFIiFBxISM/fTazF/2qy1+kJO2n28mKdCMWCHDMVdM55qrp2Y4iGaAjcBGRHKUjcBHplNrtu9n+l+WQTDLwjPEUlw7MdqS8oQIXkQ6JHahl4Zd+wqanX2+4B4tBsj7GkJllfOz+71DQr1e2I3Z7GkIRkXZLxhO8cN6NbHp6EYn6GLH9NcSqakjUx/jgpbd4btrXieuioSNOBS4i7bbx6b+wb9UGEvWH3xUxGY1TvWkH7z8yLwvJ8osKXETabeU9TxM/kHpmoHh1HSt/9VQGE+UnFbiItFvNlp2tblO7fU8GkuQ3FbiItFtbJoboMWRABpLkNxW4iLTbuK9dSqhnUcr1oeJCJlz/yQwmyk8qcBFpt6GzTuOoyccRLIwcti5YEKHP2GGM/NQ5mQ+WZ1TgItJugWCQmS/+mGOuOZ9gYYRw7x6E+xQTLAgz4vKPceH8nxOMhLMds9vr8KTGHaFJjUW6n+i+A1S+uQqSzlFlYyg8qk+2I3U7qSY11pWYItIpkT49KT1fE3Nlg4ZQRERylApcRCRHqcBFRHKUClxEJEepwEVEclSnC9zMgmb2tpn9KR2BRESkbdJxBH49sDINzyMiIu3QqQI3s6HARcB96YkjIiJt1dkLee4Cvg1o7qQuLJFI8pf57/P8U8up3HaAcCTIqdNGcNEnJ1AyuHe244lIB3X4CNzMZgM73H1xK9tdZ2blZlZeWVnZ0d1JByUSSe78wTweuvdNtm6uIh5PUlsT47W567j1hudYs3JHtiOKSAd1ZghlGnCxmVUAjwLnmdlDTTdy93vdvczdywYO1GzVmfbSMytZs2IH9fWJjyxPJp36ujh3/fAV4rFEikeLSFfW4QJ395vdfai7jwQ+A8xz96vTlkw6zd158ekVRKOpCzoRT7L4jU0ZTCUi6aLzwLux6v1Rqg8cPunsoerq4qxdraEtkVyUlrsRuvt8YH46nkvSJxgO0JbbBYdDwQykEZF00xF4N1ZUFKZ0eN8WtykoDHHyaUMzE0hE0koF3s1ddtVJRAqaP8IOBI2Swb04dqzeXBbJRSrwbu6kKUO57KqTCIeDhEJ//+cuLAxRcnQvvnXbdMwsiwlFpKM0I08euOCS8ZSdMZx5L66hYu0uioojTDt3NCedUkogqJ/hIrlKBZ4jdlVW89yTy3j91fXU1cXp26+If5h9POfNGktBQev/jAMG9eRTn5ucgaQikikq8BywqWIPP7zlJaL1cRKJhrNK9uyq4clH/sbC+e9z6x0XUFikGcBF8o1+f+7i3J27fzSf2prYwfL+UDSaYNuWKv7ngSVZSici2dTlCzyZSLLh/d2sW1PZ6kUp3dHqFTuoqqpLuT4eS7LwlXXU18czmEpEuoIuO4Ti7sx5bjXP/OFdYtEEFjDi8SQnTi6lZEgvyhdtJBpNMHxkPy68dDzjJh2d7chHRMW6XSTiyRa3CQQC7Ni6n2Ej+2UolYh0BV22wB+9fzHzXnyPaJMjy/K/bvzI13t217Bq2XbOnH4M1/zTlIOnxNXWRNlUsRcLwIhR/Ym04Y2+lrg7a1bsYOnbW0gmnTHjB3HCyUOO+Fkc4VAQC7R8ml8ymSQc1tWUIvmmSxb4ti1VzH1hDbEWbsJ0kEN9fZyFc9dx3PEDmXzqUB667y0WLag4eN5zMunMmDWWyz57EsEOFO6uymp+evtcdlVWU1/X8ANl7gurKSwK863vTT+iR74nnDKER37X4h17KS6OUDJEt2QXyTddcgx8/ktrSCZaHjZoqr4+ztP/8y4/vOVlFr26nlg0QW1NjNqaGPV1ceY8t4pf/vjVNt0b5FDRaIL/uPlFtn1QdbC8Aepq4+zdXcuPvvsS+/bWtus522NgSS8mnjSYcLj5f6pIQZBLrzxRF+OI5KEuWeBbP6g67IyLtti2pYotm/cSix1e/tH6BCve3cbq5e2bwODNhRVUH4iSTDafJxZN8OfnV7c7a3t8+ZtnMuq4ARQUhqCxp4OhAOFIkJmzj+fsGcce0f2LSNfUJYdQ+vYvwgzaebCMO8SiqY/c6+vjzPnTKo6fWNLm51wwd+1HjrybisWS/GXeOi676qT2RG2XwqIwt/xwJmtW7GDB3LVU7aujdGgfzr1gLCWDNXQikq+6ZIGfNeNYXn91PdH69s0UEwhAsqWRF4cd2/a36zlrq2OtbpOJU/jMjLETShg7oe0/fESke+uSQyijjxvAmHGDCEfafmZFQUGIXr0LW92uT7+ig58nE0n2V9W1WMDDR/cj0MpZIEeX9mlzThGRdOmSR+BmxvW3nMtvf7WItxZtIBgM4EnH3YknnFAocPAMFQsY4XCA82ePpVefIp546O2UU4gVFIaYfuEY6mpjPPv4Uua+sIZ4LEEi6Rx3/EAuv/pkxowb9JHHzJw9jjcXbmjxOWddOj69fwEiIm3QJQscIBIJ8uVvnsln9pzC0iVbiMcTjBjdn5LBvVn4yjoWzltHtD7BiGP6c+El4xl17FHU1sZ4+dmV7NlVc9ibjuFwgCFDezN2/CBu//YL7Ni2/yNvdq5evoOf3PZn/vmbZ1J2+vCDy0eM7s/0WWOZ+8Lqw4Z0IgVBJp40mJNPHXZk/zJERJph7T2trjPKysq8vLz8iO5j755a7vnJAtav3UUgYBiQSCSZNLmU674xjWcfW8rLz65s9kwVaBiK+eUDl1NQ+PebQ7k7ixas54+PvsuuymowKO5ZwEWfGM/5Fx2vW7KKyBFlZovdveyw5d2twD+0bUsVa1dVYgFj3KSj6X9UD5KJJP9yzR+orUn9xmRBYYirvzSFs5o5Nc/dqd4fJZlM0rN3Yatj4yIi6ZCqwLvsEEpnHT2kN0cP6f2RZdXVUeKxls9sqa+Ls3H97mbXmRk9exekLaOISGfk1e/+kUiQRIoLcj4UCBhFxZEMJRIR6bi8KvCCwjDHtTKBbygU4LRpIzKUSESk4/KqwAEuv/pkIinOLw+HA4wZP4ihI3RbVhHp+jpc4GY2zMxeMbMVZrbczK5PZ7AjZcz4QVz3jWlECoIN9xahYdgkEgkydkIJX7/p7CwnFBFpm868iRkHbnT3JWbWC1hsZnPcfUWash0xU6aO4ITJQ/jraxVsqthDUXGEU6eO0IQIIpJTOlzg7r4V2Nr4+X4zWwmUAl2+wKFhPPzs84/LdgwRkQ5Lyxi4mY0ETgbeaGbddWZWbmbllZWV6didiIiQhgI3s57AE8A33L2q6Xp3v9fdy9y9bODAls8AERGRtutUgZtZmIbyftjdn0xPJBERaYvOnIViwG+Ale5+Z/oiiYhIW3TmCHwacA1wnpm90/hnVppyiYhIKzpzFspCDs7Q2P25O0ve2MSfnljGpoq9BILGCZOH8PHLJzFidP9sxxORPNRtb2aVTu7OA79+g9fnr//77D0xKF+0kb+Vf8A/XT+VU6eN7NBzJxJJ/lb+AetWVxKKBDlhcinHjBmQvvAi0m2pwNvg7Tc3f7S8G7lDNJrg3rtfZ+yEEvr0LUrxDM1bv3YXd/5gHtFonLraOGbw/FPLGTK0Dzfeeh692/l8IpJf8u5eKB3x7BPLWp24eP7L77XrOXfuOMAdt75M1b466mobntsdovUJNlXs4UfffZlEoqUZmkUk36nA22BTxZ4W18eiCVYt296u53zhj8sPzuvZVCLh7NlVw98Wf9Cu5xSR/JJXQyjuzrJ3tvLcE8vYsH43wWCAU04fzoWXjj9s8odDBQNG6jl8GoTCzd/hMJVFCypIJFLfm7yuLs5rc9cxWfNtikgKeXME7u48dN9b/PKO+axctp2a6hj7q+pZMHctt97wJ5a9syXlY084pRRr4XybgsIQp585sl15Uh19H6rmQLRdzyki+SVvCnzxG5t47c/rqG8ys3wy4UTrE/zijlepqW6+MD9++UTCqY6wrWEi5CntnARiQEnPFtcHg8bw0bo7ooikljcF/qfHW34j0t1ZOG9ds+uGj+rfcA/xSJDwIZNBFBaG6N27kJt/ODPlJBGpzLp0AgUFqR8TCAaYMWtsu55TRPJL3oyBp5qo+EPR+gTL/7aVmR8f1+z6KVNHMHZCCfNffo9VS7cRigQ57cyRTJk6ot3lDTD1nFG8/ur7rF1VSbTJcEqkIMjsyyZSMjj1uLyISN4UeMOtW1qZ0DjY8i8kvfsUcvEVk7j4ikmdzhMMBrjxe9N5/qnlvPTMSurr43jSGXh0Tz555Umcqnk5RaQVeVPg4yYdzdK3U79RWVgYYsrU4RlM1DCB8sVXTGL2JydQta+OYChAr96FGc0gIrkrb8bAL/7UpBaHOsIFQaZMzc5RbyAYoG//HipvEWmXvCnwMeMG8dkvTSEcCRIM/f1lFxSG6NmrgJt+MDP1mSYiIl1Q3gyhAJwz8zgmnDiYPz+/ijUrdhAKB5l69ijOOGsUhUXhbMcTEWmXvCpwgIElPbnyH8uyHUNEpNPyZghFRKS7UYGLiOQoFbiISI5SgYuI5Chzb/nqxLTuzKwS2JCxHabHAGBntkNkST6/dtDrz+fX39Ve+wh3H9h0YUYLPBeZWbm75+VpK/n82kGvP59ff668dg2hiIjkKBW4iEiOUoG37t5sB8iifH7toNefz68/J167xsBFRHKUjsBFRHKUClxEJEepwJthZsPM7BUzW2Fmy83s+mxnygYzC5rZ22b2p2xnyTQz62tmj5vZKjNbaWZnZDtTppjZDY3f98vM7BEz69Y3qjez35rZDjNbdsiy/mY2x8zea/zYJWcYV4E3Lw7c6O7jgdOBr5rZ+CxnyobrgZXZDpEldwMvuvvxwInkyd+DmZUCXwfK3H0iEAQ+k91UR9z9wAVNlt0EzHX344C5jV93OSrwZrj7Vndf0vj5fhr+85ZmN1VmmdlQ4CLgvmxnyTQz6wOcBfwGwN2j7r43q6EyKwQUmVkI6AGknouwG3D3BUDTWc8vAR5o/PwB4NJMZmorFXgrzGwkcDLwRpajZNpdwLeBZJZzZMMooBL4XeMQ0n1mVpztUJng7h8APwU2AluBfe7+cnZTZUWJu29t/HwbUJLNMKmowFtgZj2BJ4BvuHtVtvNkipnNBna4++JsZ8mSEDAZ+L/ufjJQTRf9FTrdGsd6L6Hhh9gQoNjMrs5uquzyhnOtu+T51irwFMwsTEN5P+zuT2Y7T4ZNAy42swrgUeA8M3sou5EyajOw2d0//K3rcRoKPR/MANa7e6W7x4AngalZzpQN281sMEDjxx1ZztMsFXgzzMxoGP9c6e53ZjtPprn7ze4+1N1H0vAG1jx3z5ujMHffBmwys7GNi6YDK7IYKZM2AqebWY/G/wfTyZM3cJt4Bri28fNrgaezmCUlFXjzpgHX0HDk+U7jn1nZDiUZ9TXgYTN7FzgJ+FF242RG428djwNLgKU0dEROXFbeUWb2CLAIGGtmm83si8AdwPlm9h4Nv5Xckc2MqehSehGRHKUjcBGRHKUCFxHJUSpwEZEcpQIXEclRKnARkRylAhcRyVEqcBGRHPW/W2JU8OyhZngAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# data generation\n",
    "np.random.seed(314)\n",
    "\n",
    "data_size1 = 10\n",
    "x1 = np.random.randn(data_size1, 2) + np.array([2,2])\n",
    "y1 = [-1 for _ in range(data_size1)]\n",
    "\n",
    "data_size2 = 10\n",
    "x2 = np.random.randn(data_size2, 2)*2 + np.array([8,8])\n",
    "y2 = [1 for _ in range(data_size2)]\n",
    "\n",
    "# all sample data\n",
    "x = np.concatenate((x1, x2), axis=0)\n",
    "y = np.concatenate((y1, y2), axis=0)\n",
    "\n",
    "shuffled_index = np.random.permutation(data_size1 + data_size2)\n",
    "x = x[shuffled_index]\n",
    "y = y[shuffled_index]\n",
    "\n",
    "train_data = np.concatenate((x, y[:, np.newaxis]), axis=1)\n",
    "\n",
    "label_y = y.copy()\n",
    "label_y[label_y==-1] = 2;\n",
    "print(label_y)\n",
    "# plot data\n",
    "plt.scatter(train_data[:,0], train_data[:,1], marker='.',  s = 300,\n",
    "            c=label_y, cmap=plt.cm.Spectral)\n",
    "plt.title(\"Data\")\n",
    "plt.savefig(\"perceptron_sample_data.pdf\")\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "学习模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "lines_to_end_of_cell_marker": 2
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "update weight/bias:  4.168170755009569 4.181365846630972 0.5\n",
      "update weight/bias:  3.0724185380181854 3.490747761861174 0.0\n",
      "update weight/bias:  1.4813054058811954 1.9310599013646637 -0.5\n",
      "update weight/bias:  0.592200234913389 0.7401981125197059 -1.0\n",
      "update weight/bias:  -0.8339423116876052 0.09373359199050268 -1.5\n",
      "update weight/bias:  1.3000028486797486 3.4573196134037336 -1.0\n",
      "update weight/bias:  0.5492714737908462 2.0751568705213304 -1.5\n",
      "update weight/bias:  -1.2835880698143969 0.2060627698241797 -2.0\n",
      "update weight/bias:  2.378931537209766 3.244691345864665 -1.5\n",
      "update weight/bias:  1.8447601323444536 1.8013609063723086 -2.0\n",
      "update weight/bias:  0.4186175857434593 1.1548963858431054 -2.5\n",
      "update weight/bias:  -1.4142419578617837 -0.7141977148540453 -3.0\n",
      "update weight/bias:  2.354422566641632 2.1385772796165576 -2.5\n",
      "update weight/bias:  1.2586703496502485 1.4479591948467596 -3.0\n",
      "update weight/bias:  0.39930036275602365 0.9139219094288209 -3.5\n",
      "update weight/bias:  -1.4335591808492194 -0.9551721912683298 -4.0\n",
      "update weight/bias:  2.734611574160349 3.2261936553626422 -3.5\n",
      "update weight/bias:  1.6388593571689658 2.535575570592844 -4.0\n",
      "update weight/bias:  0.5431071401775824 1.8449574858230462 -4.5\n",
      "update weight/bias:  -0.20762423471131997 0.4627947429406427 -5.0\n",
      "update weight/bias:  2.6948191352076956 3.592413861280405 -4.5\n",
      "update weight/bias:  2.081106856196456 2.160920268143234 -5.0\n",
      "update weight/bias:  0.9853546392050727 1.4703021833734358 -5.5\n",
      "update weight/bias:  -0.6057584929319173 -0.08938567712307455 -6.0\n",
      "update weight/bias:  3.5910006762733424 2.381046334670937 -5.5\n",
      "update weight/bias:  2.507957956903076 0.9900640934151144 -6.0\n",
      "update weight/bias:  1.4122057399116927 0.29944600864531634 -6.5\n",
      "w =  [1.4122057399116927, 0.29944600864531634]\n",
      "b =  -6.5\n",
      "\n",
      "\n",
      "\n",
      "ground_truth:  [-1. -1. -1. -1. -1.  1.  1. -1.  1.  1.  1.  1. -1.  1.  1.  1.  1. -1.\n",
      " -1. -1.]\n",
      "predicted:     [-1. -1. -1. -1. -1.  1.  1. -1.  1.  1.  1.  1. -1.  1.  1.  1.  1. -1.\n",
      " -1. -1.]\n",
      "accuracy:      1.0\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "def sign(v):\n",
    "    \"\"\"符号函数\"\"\"\n",
    "    if v > 0:  return 1\n",
    "    else:      return -1\n",
    "    \n",
    "def perceptron_train(train_data, eta=0.5, n_iter=100):\n",
    "    \"\"\"对感知机模型进行训练\"\"\"\n",
    "    weight = [0, 0]      # 权重\n",
    "    bias = 0             # 偏置量\n",
    "    learning_rate = eta  # 学习速率\n",
    "\n",
    "    train_num = n_iter  # 迭代次数\n",
    "\n",
    "    for i in range(train_num):\n",
    "        # select one data\n",
    "        ti = np.random.randint(len(train_data))\n",
    "        (x1, x2, y) = train_data[ti]\n",
    "        \n",
    "        y_pred = sign(weight[0] * x1 + weight[1] * x2 + bias) \n",
    "        \n",
    "        if y * y_pred <= 0:  # 判断误分类点\n",
    "            weight[0] = weight[0] + learning_rate * y * x1  # 更新权重\n",
    "            weight[1] = weight[1] + learning_rate * y * x2\n",
    "            bias      = bias      + learning_rate * y       # 更新偏置量\n",
    "            print(\"update weight/bias: \", weight[0], weight[1], bias)\n",
    "\n",
    "    return weight, bias\n",
    "\n",
    "def perceptron_pred(data, w, b):\n",
    "    \"\"\"输入数据，模型，对数据进行分类\"\"\"\n",
    "    y_pred = []\n",
    "    for d in data:\n",
    "        x1, x2, y = d\n",
    "        yi = sign(w[0]*x1 + w[1]*x2 + b)\n",
    "        y_pred.append(yi)\n",
    "        \n",
    "    return np.array(y_pred, dtype=float)\n",
    "\n",
    "\n",
    "# 训练感知机\n",
    "w, b = perceptron_train(train_data)\n",
    "print(\"w = \", w)\n",
    "print(\"b = \", b)\n",
    "\n",
    "# 预测 \n",
    "y_pred = perceptron_pred(train_data, w, b)\n",
    "\n",
    "# 计算分类精度\n",
    "c = y_pred == y\n",
    "cn = np.sum(c == True)\n",
    "acc = cn / len(y_pred)\n",
    "print()\n",
    "\n",
    "print(\"\\n\")\n",
    "print(\"ground_truth: \", train_data[:, 2])\n",
    "print(\"predicted:    \", y_pred)\n",
    "print(\"accuracy:     \", acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 参考资料\n",
    "* [感知机（Python实现）](http://www.cnblogs.com/kaituorensheng/p/3561091.html)\n",
    "* [Programming a Perceptron in Python](https://blog.dbrgn.ch/2013/3/26/perceptrons-in-python/)\n",
    "* [损失函数、风险函数、经验风险最小化、结构风险最小化](https://blog.csdn.net/zhzhx1204/article/details/70163099)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}