# nndl-exercise **Repository Path**: lepecoder/nndl-exercise ## Basic Information - **Project Name**: nndl-exercise - **Description**: 神经网络练习题 - **Primary Language**: Python - **License**: GPL-2.0 - **Default Branch**: master - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 4 - **Forks**: 1 - **Created**: 2020-06-26 - **Last Updated**: 2024-09-09 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README # 《神经网络》课程作业 [TOC] ### 1. 逻辑回归 logistic regression 逻辑回归其实并不“逻辑”也不是回归，而是分类模型，逻辑是指利用了Logistic非线性函数。 #### 逻辑回归 在logistic regression中，我们用logistic函数来预测类别标签的后验概率 $$ p(y=1|x)=\sigma(w^T \cdot x)=\frac{1}{1+exp(-w^T\cdot x)} $$ 这里$x=[x_1,\cdots,x_D,1]$和$w=[w_1,\cdots,w_D,b]$，是$D+1$维的增广的特征向量和权重向量。 标签$y=0$的后验概率 $$ p(y=0|x)=1-p(y=1|x) $$ 由公式1可以得到 $$ w^Tx=\log \frac{p(y=1|x)}{p(y=0|x)} $$ 其中$\frac{p(y=1|x)}{p(y=0|x)}$称为几率(Odds)，所以$w^Tx$就是几率的对数，所以逻辑回归也称为对数几率回归。 当正负样本概率相等时，$w^Tx=\log(1)=0$，因此逻辑回归的目的就是求参数$w$使得正样本$w^Tx>0$负样本$w^Tx<0$。 #### 损失函数 使用交叉熵损失函数，在二分类问题中 $$ J(w) = -\frac{1}{m}\sum_{i=1}^m\left(y^i\log(\hat{y}^i) + (1-y^i)\log(1-\hat{y}^i ) \right) $$ $y^i$是样本$x^i$的真实标签，$\hat{y}^i$是模型的预测标签，正样本的损失是$\log(\hat{y})$负样本的损失是$\log(1-\hat{y})$ 我们的目标是最小化损失函数。 #### 参数学习 ##### 随机梯度下降 SGD是通过对损失函数求偏导确定梯度方向，沿着梯度方向更新参数以最小化损失函数 $$ \frac{\partial J(w)}{\partial w} = -\frac{1}{N} \sum_{i=1}^N x^i(y^i-\hat{y}^i) $$ $$ w_{t+1} \leftarrow w_t - \alpha \frac{\partial J(w)}{\partial w} $$ #### 实验结果 ###### 代码实现 ```python # 梯度下降法 for _ in range(500): # 利用逻辑回归做预测 y0 = logistic(w,ex) # 计算当前的交叉熵损失 ce = cross_entropy(y,y0) # print("第{}轮，cross_entropy = {}".format(_,ce)) # 求偏导 partial = (np.sum(ex*(ey-y0),axis=0))/N # 更新参数 w = w-alpha*partial ``` ![img](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAXwAAAD4CAYAAADvsV2wAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAbWklEQVR4nO3df6xc5X3n8fcHF++yhA1JcRJ+2HG060RlEaHNlbNVVrtEjlPjkJBWKQKtuqiJ5GUblITdSJAiEZZVJaJo44WSDXELSiJRUq8CCQ1OguvtiiKVhmtkHAi/XArCMYrdpPxa0BLDd/+YmXg8nt/nmXOec87nJV3dmTNn5jwzd+73POf7fM9zFBGYmVnzHVd1A8zMrBwO+GZmLeGAb2bWEg74ZmYt4YBvZtYSv1J1A8Y55ZRTYu3atVU3w8ysNnbv3v0PEbFq2GNZB/y1a9eyvLxcdTPMzGpD0tOjHnNKx8ysJRzwzcxaYuqAL+kWSQclPdS37BpJP5G0p/uzecRzN0l6TNI+SVemaLiZmc1mlh7+14BNQ5ZvjYhzuj87Bh+UtAL4MnAecCZwsaQz52msmZnNb+qAHxH3AD+fYxvrgX0R8WREvAp8E7hgjtcxM7MCUuTwL5O0t5vyedOQx08Hnum7v7+7bChJWyQtS1o+dOhQguaZ1dze7bD1LLjm5M7vvdurbpHVVNGA/xXgXwDnAM8C/33IOhqybOQUnRGxLSKWImJp1aqhpaRm7bF3O/zFp+D5Z4Do/P6LTzno21wKBfyI+GlEvBYRrwN/Qid9M2g/sLrv/hnAgSLbNWuNXdfCL145etkvXuksN5tRoYAv6dS+u78NPDRktfuBdZLeIWklcBFwZ5HtmrXG8/tnW242xtRn2kq6DTgXOEXSfuDzwLmSzqGTonkK+I/ddU8D/jQiNkfEYUmXAT8AVgC3RMTDSd+FWVO98YxuOmfIcrMZTR3wI+LiIYtvHrHuAWBz3/0dwDElm2Y2wYarOzn7/rTO8Sd0lpvNyGfamuXs7AvhwzfAG1cD6vz+8A2d5WYzynryNDOjE9wd4C0B9/DNyuBaesuAe/hmi9arpe/l4Xu19OCeu5XKPXyzRXMtvWXCPXyzRdi7vRPQn9/PyBPLc6il72/nG8/oVP/4qKOxHPDNUhtM4YxSdS29U02t45SOWWrDUjiDcqild6qpdRzwzVIbm6rJqJbe0za0jlM6ZqmNnA5hNVw+bLqpinjahtZxD98stQ1Xd1I2/XJI4QyqSzstGQd8s9TqMh1CGe30CWdZUcTIa5FUbmlpKZaXl6tuhpnNY1i10vEn5LnzaxBJuyNiadhj7uGb2WK4Cig7DvhmthiuAsqOA76ZLcaoah9XAVXGAd+s6aoaOHUVUHZch2/WZFVOn9B7/UXM1eM5gObiHr5Zk1U9cHr2hZ2TzX5nW+f+7VuKH2X0dmLPPwPEkZ2YSz4nmjrgS7pF0kFJD/Ut+6KkRyXtlXSHpJNHPPcpST+StEeS6yzNypLDwGnqAF31TqzGZunhfw3YNLBsJ3BWRJwNPA58bszz3x8R54yqDzWzCebJxecwcJo6QOewE6upqQN+RNwD/Hxg2d0Rcbh79z7Aw+9mi/Dd/9xJh8zaS85h4DR1gM5hJ1ZTKXP4Hwe+N+KxAO6WtFvSlnEvImmLpGVJy4cOHUrYPLOa2rsdlm/hmAupTNNLzmGah9QBOoedWE0lqdKRdBVwGLh1xCrvi4gDkt4C7JT0aPeI4RgRsQ3YBp2pFVK0z0rmCoq0dl1LoatmnX1htZ//hquHT7Ewb4BeZPVPwxUO+JIuAc4HNsSIiXki4kD390FJdwDrgaEB32rOV1FKb1xQP+FNnXx+zoFvEQG66p1YTRUK+JI2AVcA/y4iXh6xzonAcRHxYvf2BwEPpzfVuAE6/4POZ9S89QCvvgSvdIfWct65OkBnYZayzNuAvwHeJWm/pE8ANwIn0UnT7JF0U3fd0yTt6D71rcC9kh4EfgjcFRHfT/ouLB+uoEhvWM4awcoT4bVXj17s8kQbY+oefkRcPGTxzSPWPQBs7t5+Enj3XK2z+qnbVZTqMN4wKiVy+4j6B+9cbQRPrWBppR6gW6Q6jTcMS4nsurZeO1ernAO+pVVmBcXe7fC9K47ksE94M5z3hem3Vffxhpx2roNHSus+CE/cnfeRUws54Ft6ZQzQ7d0O3/4DeP0XR5a98nP4ziePtGGSuo835FKeOOxIabkv25vzkVPLOOBbPe269uhg3/Paq9P30Os23jBMDtUvw46UBtXpyKnBPFum5WXa+WLG9cKn7aH7jM00pv2863Lk1GAO+JaPWWZVHNcLn7aHnsO0A00w7eddpyOnhnLAt3zMMqvihqvhuOOPXb5i5Ww99N587dc81/ntYD+7oecJDPCRUxYc8C0fswyinn0hfPR/dipzek54M1zw5SNBu6pL+02jaNtyem/DjpSWPuEjpwxpxPQ3WVhaWorlZV8vpTW2njViEHV1p/c9i8HKEej0MnMIPEXbNuz5CIjOZ+USyFaTtHvUdUfcw7d8pBxEzfmqSEXbNrQqpttx8+X+bAwHfMtHykHUnGvsi7Zt0nq57NgsO67Dt7ykqitfRI19qnl3irZt3OyZPeN2CnWYP8gWwj18y0PqQcjUNfYpL8RdtG3TVMWM2nmkvqC41YoDvlVvEUEodY19yjGBom076vl0XqPfuJ1HzmMbtnBO6Vj1FjWJWcppB4rk3UelUFK9t1lSNDmPbdjCOeBb9aoIQrPmsU9405FZOftNyruXMQXzLDuPJswfBB6HmJNTOla9UcFmUUFo1hTS3u3w/148dvk0Z/XmlkJpwvxBHoeYmwO+Va/sIDRrEB41M+fKN0zuVeaWQmnC/EG57URrZOqUjqRbgPOBgxFxVnfZm4E/B9YCTwEXRsQ/DnnuJuB6YAXwpxFxXeGWW3OUPa/7rEF41PJXjvmqHyvHFEoOUyoXkdtOtEZm6eF/Ddg0sOxKYFdErAN2de8fRdIK4MvAecCZwMWSzpyrtdZcZU5iNmsKqUjKqQkplNyUnQJskKkDfkTcAwyOWl0AfL17++vAR4c8dT2wLyKejIhXgW92n2dWjVmDcJGgXVYKJafJ1BbNO9G5Fa3SeWtEPAsQEc9KesuQdU4H+o9p9wPvLbhds/nNmkIqmnKaNoUyb+VJnS7GnkIul3asoTLKMjVk2cgpOiVtAbYArFmzZlFtsrabNY+96Lx3kaBd94uxz6Pu4xAVKVql81NJpwJ0fx8css5+YHXf/TOAA6NeMCK2RcRSRCytWrWqYPPMaqJI5cnIQcxnykvtTJNSalPaKVNFA/6dwCXd25cA3xmyzv3AOknvkLQSuKj7PMuR/ymrMS5oT/pbjBusLKM+fZq6eNfOZ2HqgC/pNuBvgHdJ2i/pE8B1wEZJTwAbu/eRdJqkHQARcRi4DPgB8AiwPSIeTvs2LAn/U1ZnbIXJhL/FuMnUyqhPn+boxLXzWZg6hx8RF494aMOQdQ8Am/vu7wB2zNw6K1cbc8G52HD1kKtYDRj2t+gN9I573qLr06epix81nbNr50vlM23tCJ/QUp3B8s1R+v8WRx2RjbHo+vRJdfF7tzPyPbl2vlQO+HbEqH++E95U77x+XcYl+k8+e+Pq4ev0/40m9eyhnPr0SXXxu65leGGeXDtfMgd8O2LYP+6KlZ2Jw+qa189lXGLWnc40JxeNPfIqcZ6cSSeXjWxnOFVYMk+PbEcMO6Hl1f977LTAdcrr5zAuMU+N/TQnF42cp2d150ihTOPq4se100rlgG9HG/zHvebk4evVJa+fw7jEvDudSScXDRvozXGKgbq0swWc0rHx6j5RVQ7tX9ROpy5THdelnS3gHr6NV/feWQ7tn3aK5Hnm0qnLFAN1aWfDuYdv49W9d5ZD+6cZgM1lcNkaTREj5zGr3NLSUiwvL1fdDLPiJvXet56VzwCs1Zqk3RGxNOwxp3TMyjAppZHD4LJlY+vOx7l84zuTv65TOmY5SDy4vHXn4wUaY1W7ftcTC3ldB3yzHCS+itOiAobVm1M6ZjnwVZxab+vOx4/aUa+98i4APr1hXbL0jgdtzcaZ97KDFRgMGD0pA4aVY+2Vd/HUdR+a67ketDWbR82uFXv5xnf+MrAXCRjWXM7hm43ii3ZYRT69Yd1CXtcB32yUGpdKLipgWDkWlYJzwDcbJYd5eObU9Jy9y07n44BvNkriUsnayvACMi47nU/hgC/pXZL29P28IOkzA+ucK+n5vnVa9h9jtZTDPDxV8xw/jZK0LFPSCuAnwHsj4um+5ecCn42I82d5PZdlmlUsozl+XHY6nTLLMjcAf9cf7M2sxjIauHbZaXGpc/gXAbeNeOw3JT0o6XuS/tWoF5C0RdKypOVDhw4lbp6ZzaTGA9d2rGQBX9JK4CPA/xry8APA2yPi3cAfA98e9ToRsS0iliJiadWqVamaZ7nJcCDQhsh04Nplp/NJ2cM/D3ggIn46+EBEvBARL3Vv7wCOl3RKwm1bnXggMIlSShMzHbh2zn4+KQP+xYxI50h6myR1b6/vbvdnCbdtdeIzWJNIXZo4cgdy9oWdAdprnuv8blOVUsMkCfiS/hmwEbi9b9mlki7t3v0Y8JCkB4EbgIsi51nbMtaIE04yGgjMQS5/U9e2N1+SKp2IeBn41YFlN/XdvhG4McW22u76XU/U/3B22ot6t8Qsf9MyptC15vJsmVa+DVcfPQslZDEQWAepSxO9A2kXz4dfA4084aRG88wvQoq/aepadNe2N8O4E68c8GvG/5QzqsGOZd6/aeoLXfu7NbtFXWy8iHEB35OnWXM1vPwzdaBxbfvs6jbQ7YBfM/6nnEFNyj9z+Zvm1lO19JzSsea65mRg2PdbnZpysznkPqbmlI61U2bzwORSb19EE95DUZdvfCdPXfehX4539G7nEOwnccC39HKZJyezeWDqlu8dpgnvoc0c8C2tnAZKM50HZlruTecvl/GXaTmHb2lldMGMHBTJ9+ZSJpl7ztqOVuYFUKztajxPziJqqptw0Y7ee+gF/jq+B+twSsfSymygdBY55Ke37nyctVfe9cspDnq3c0jv5PD5WDHu4VtanidnpGnyvbkfEdQtZ21Hc8C3tHoDoplPZ9BT5uRhdcx3D34+1+96gut3PeH8fU150NasK7cedW7ztOTy+eT2ueTGJ16Z1ZCD2nAeS5ifA75Zl/PT4/nzqT+ndMwsez4XYHqeD9/MsjVrTj6X6wfkauE5fElPSfqRpD2SjonQ6rhB0j5JeyX9Rortmlk+5j1XoKycvHP/aXP474+Ic0bsWc4D1nV/tgBfSbhdM8tAWQHVYwnzK6sO/wLgG9HJH90n6WRJp0bEsyVt3yyJtqQFFq3I+Q+zfP6+SPvRkuTwJf098I90rjbx1YjYNvD4d4HrIuLe7v1dwBURMSz9s4XOUQBr1qx5z9NPP124fZaXOgfNXGrRc5Hjxdir3k7Vypg87X0RcUDSW4Cdkh6NiHv62zDkOUP3NN2dxTboDNomap9l5PpdT9Q24NvRcp8Kwo6WJOBHxIHu74OS7gDWA/0Bfz+wuu/+GcCBFNs2WzSnBRarrJy8c/8JUjqSTgSOi4gXu7d3AtdGxPf71vkQcBmwGXgvcENErJ/02i7LbI6m1FG7FztanVN1TbLolM5bgTsk9V7vzyLi+5IuBYiIm4AddIL9PuBl4PcTbNdqpMxDfweeavgzz1/hgB8RTwLvHrL8pr7bAXyy6LbMprHIMQKnBazOPJeOla6MoLmoC4a4F2t15qkVrBGaMkZgVpTn0rFW6VXRpB4n8NiA1YHnw7dW6F0Ptif19WA9F4vVnS9xaI3RqwTqpXdcPml2NAd8a5zLN74zWW/cJ11ZkziHb42x6IFbn3RldVDGXDpmlWv6vC4eNLaiPGhrNqWqT7ryoLEV5YBvjZQqOPdX+Lh3bXXngG+NlCo4V92r7pWa9gaLU5eaNpk/o2M5h2+WsbqMS+Q4vuDrLhzLAd9sgEsxZ+fgWg8O+GYDcu1VVz1oXAfeWY/nOnyzMeYN+DmmOFLLfcK6nHbWZXIdvtmc5u1VtyHFkeuRkI3mKp0WcdXC7FIG7Zw//5zbNi+nwI7lgN8iVZcYNt2kEsqcP/+ibcsxuDb9CGsehVM6klYD3wDeBrwObIuI6wfWORf4DvD33UW3R8S1RbdtlpPcUxyLHFdwcK2HFDn8w8B/iYgHJJ0E7Ja0MyJ+PLDeX0fE+Qm2ZzNw1UK1cvr8B8cVcmqblSQikv7Q6clvHFh2LvDdWV/rPe95T1g6b7/iu1U3oTW+dPdjxyyr+vMft/2q22bpAMsxIqYmrdKRtBb4deBvhzz8m5IeBA4An42Ih0e8xhZgC8CaNWtSNs+sNLn0kN2Lt37JAr6kNwDfAj4TES8MPPwA8PaIeEnSZuDbwNBRnojYBmyDTh1+qvZZngNrbVLF5z/tuMIi29aGcxLqIkmVjqTj6QT7WyPi9sHHI+KFiHipe3sHcLykU1Js26bnf7pq5fz5L7JtOVcntU3hgC9JwM3AIxHxpRHrvK27HpLWd7f7s6LbNrMjJtXS+wjPUqR03gf8HvAjSXu6y/4QWAMQETcBHwP+k6TDwCvARd3BBTNLZNLZvWUeYXjsIE+FA35E3Atowjo3AjcW3ZaZ1UPu5yS0lefSsVrwwN9w7knbLBzwrRbaMBnZPBbZk061k/XYQT48l46ZDZWqusY76ny4h2/ZcrpiNu5J2yS+AIrVggf+ypH7RU1sMl8AxayPB4BHc3VNszmHb7WQMl3hMz+trRzwrRbcIy+fxwSaxzl8awXnpq0txuXwHfCtdXLMTXtcwVIZF/Cd0jHLgMcVrAwO+NY6zk1bWzmlY1YRjyvYIjiHb5a5HMcVpuXxh7w4h29ZmnTBjrpp2vuZlscf6sMB3yrTtEBR5P14XMHK4KkVzDJQt5SIJ7arJ+fwWyCnHGvTBiqb9n7mUefxhyby5Gktl9PFQ5o2OVfR95PTztiaL0kOX9ImSY9J2ifpyiGPS9IN3cf3SvqNFNu1PLR1sDKFJoxjePyhPgoHfEkrgC8D5wFnAhdLOnNgtfOAdd2fLcBXim7Xxtu683HWXnnXL3OrvduLCM7zBq2mBYqmvZ9p+QilPlKkdNYD+yLiSQBJ3wQuAH7ct84FwDeiM2Bwn6STJZ0aEc8m2L4NUYfUSdMCxbTvxwOeVpUUAf904Jm++/uB906xzunAMQFf0hY6RwGsWbMmQfNsERy05leHnbE1U4qAryHLBkt/plmnszBiG7ANOlU6xZpmsJhUg4OWWf2kGLTdD6zuu38GcGCOdWxB3OPOV1vz/laNFAH/fmCdpHdIWglcBNw5sM6dwH/oVuv8a+B55+/rrzcA7KA1P++MrUyFA35EHAYuA34APAJsj4iHJV0q6dLuajuAJ4F9wJ8Af1B0u1a9Xg7fQcusHpKceBURO+gE9f5lN/XdDuCTKbZl1jQ++crK4jNtbSauzkkvpzOhrdkc8G0mrs4xqy8HfLMK+EjJquCAb3Nzdc78fKRkVfAFUGxu7oma1YsDvlnFfKRkZXHAN6uYj5SsLA741liep9/saA741lhNuLiIWUqNDfju3dmi+LtlddXYgO/eXTuVcaUvf7esrlyHb43Sq2/vBXzXt5sd0aiA77MXbVH83bImUGciyzwtLS3F8vLyXM/12YvtMxiUe1IHZX+3LGeSdkfE0rDHGtXDt3bzdAVm4zV20NZnL9ooRQdw/d2yumpswHdetd3GBeWiVTb+blldNTbgW/4WWc/uoGx2rEI5fElfBD4MvAr8HfD7EfHckPWeAl4EXgMOjxpQsHYp80pPrrIxKz5ouxP4XEQclvQF4HPAFSPWfX9E/EPB7ZnNxQO6ZgUDfkTc3Xf3PuBjxZpjTeeetll1UpZlfhz48xGPBXC3pAC+GhHbRr2IpC3AFoA1a9YkbJ7lIIeetqtsrK0mBnxJfwm8bchDV0XEd7rrXAUcBm4d8TLvi4gDkt4C7JT0aETcM2zF7s5gG3ROvJriPZjNxEcS1lYTA35EfGDc45IuAc4HNsSI03Yj4kD390FJdwDrgaEB39rDPW2zchUqy5S0ic4g7Uci4uUR65wo6aTebeCDwENFtmvN4J62WbmK1uHfCJxEJ02zR9JNAJJOk7Sju85bgXslPQj8ELgrIr5fcLtmZjajolU6/3LE8gPA5u7tJ4F3F9mOGXQqfHxUYDY/n2lrteELj5gV44BvZtYSnh7ZsuYTtczSaewFUKx5PCWC2WTjLoDilI6ZWUs44Ftt+EQts2Ic8K02nLM3K8YB38ysJRzwzcxawgHfzKwlHPDNzFrCAd/MrCWyPvFK0iHg6arbMYNTgDpet9ftLk8d2wxud9mKtPvtEbFq2ANZB/y6kbQ86gy3nLnd5aljm8HtLtui2u2UjplZSzjgm5m1hAN+WtuqbsCc3O7y1LHN4HaXbSHtdg7fzKwl3MM3M2sJB3wzs5ZwwE9M0n+TtFfSHkl3Szqt6jZNQ9IXJT3abfsdkk6uuk2TSPpdSQ9Lel1S9qV3kjZJekzSPklXVt2eaUi6RdJBSQ9V3ZZZSFot6a8kPdL9jny66jZNIumfSvqhpAe7bf6vybfhHH5akv55RLzQvf0p4MyIuLTiZk0k6YPA/46Iw5K+ABARV1TcrLEk/RrwOvBV4LMRke3l0SStAB4HNgL7gfuBiyPix5U2bAJJ/xZ4CfhGRJxVdXumJelU4NSIeEDSScBu4KM5f96SBJwYES9JOh64F/h0RNyXahvu4SfWC/ZdJwK12KNGxN0Rcbh79z7gjCrbM42IeCQiHqu6HVNaD+yLiCcj4lXgm8AFFbdpooi4B/h51e2YVUQ8GxEPdG+/CDwCnF5tq8aLjpe6d4/v/iSNHw74CyDpjyQ9A/x74Oqq2zOHjwPfq7oRDXM68Ezf/f1kHoCaQtJa4NeBv622JZNJWiFpD3AQ2BkRSdvsgD8HSX8p6aEhPxcARMRVEbEauBW4rNrWHjGp3d11rgIO02l75aZpc01oyLJaHP3VmaQ3AN8CPjNw9J2liHgtIs6hc4S9XlLSNNqvpHyxtoiID0y56p8BdwGfX2Bzpjap3ZIuAc4HNkQmgzszfNa52w+s7rt/BnCgora0QjcP/i3g1oi4ver2zCIinpP0f4BNQLIBc/fwE5PUf6XtjwCPVtWWWUjaBFwBfCQiXq66PQ10P7BO0jskrQQuAu6suE2N1R0AvRl4JCK+VHV7piFpVa86TtIJwAdIHD9cpZOYpG8B76JTPfI0cGlE/KTaVk0maR/wT4CfdRfdl3t1kaTfBv4YWAU8B+yJiN+qtlWjSdoM/A9gBXBLRPxRxU2aSNJtwLl0puv9KfD5iLi50kZNQdK/Af4a+BGd/0WAP4yIHdW1ajxJZwNfp/P9OA7YHhHXJt2GA76ZWTs4pWNm1hIO+GZmLeGAb2bWEg74ZmYt4YBvZtYSDvhmZi3hgG9m1hL/H0C9AFHRtl/PAAAAAElFTkSuQmCC) ![img](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAXwAAAD4CAYAAADvsV2wAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3de2xc53nn8e8jitSF1oXSyJItiaYs62JqqLiJqjRIi9pV4siKE2+CxLGzSR23C7VFjbbZFmunBtysF4tmETTeBM7W1TZGWsBN4qJ2k8ZOYtXowhts00Q2bJGUaEmWZVuWK4ui7jfenv3jDEcz1Aw5lzNzzpn5fQCBnDOHPA9H5HPeed7nvMfcHRERaXwzog5ARETqQwlfRKRJKOGLiDQJJXwRkSahhC8i0iRmRh3AVFKplHd1dUUdhohIYrz44ouD7r6k0HOxTvhdXV3s2rUr6jBERBLDzN4o9pxKOiIiTUIJX0SkSZSc8M3scTN718z6crZ92czeNrOXM/+2FfnarWb2qpkdMLMHwghcRETKU84I/9vA1gLbH3H3mzL/np38pJm1AN8EbgO6gbvNrLuSYEVEpHIlJ3x3fwEYquAYm4ED7n7Q3YeB7wJ3VPB9RESkCmHU8O8zs92Zkk9HgeeXA2/lPD6c2VaQmW03s11mtuvYsWMhhCfSoHY/CY+k4csLg4+7n4w6Iom5ahP+XwKrgZuAd4C/KLCPFdhWdIlOd9/h7pvcfdOSJQVbSUVk95PwT38Ap94CPPj4T3+gpC9Tqirhu/tRdx9z93HgfxOUbyY7DKzMebwCOFLNcUWa3vMPw8iF/G0jF4LtIkVUlfDN7Jqch58A+grs9gtgjZmtMrM24C7gB9UcV6TpnTpc3nYRyrjS1sy+A9wMpMzsMPBnwM1mdhNBieYQ8DuZfa8F/trdt7n7qJndB/wEaAEed/f+UH8KkWazYEWmnFNgu0gRJSd8d7+7wOZvFdn3CLAt5/GzwBUtmyJSoS0PBTX73LJO65xgu0gRutJWJIk23gkf+wYsWAlY8PFj3wi2ixQR68XTRGQKG+9UgpeyaIQvEjX100udaIQvEqWJfvqJWvxEPz1o9C6h0whfJErqp5c60ghfpN52Pxkk9FOHKXrReZT99LnxLVgRdP7o3UZDUMIXqafJJZxiouqnV4mpoamkI1JPhUo4k0XZT68SU0NTwheppylLNTHop9eSDQ1NJR2Reiq6JMJK+GKhpajqTEs2NDSN8EXqactDQckmV5yWRIh7fFIVJXyReor7kgi1jk8XmUXK3IveiyRymzZt8l27dkUdhoiEoVCHUuuceJ3wGoCZvejumwo9pxG+iNSHOoAip4QvIvWhDqDIKeGLSH0U6/RRB1DdKOGLNKMoJk/VARQ59eGLNJuolk+Y+N5hr9OjtX9KphG+SLOJcvJ0453BBWaf3BE8fmp7de8wJk5ep94C/PLJS+2eBZWc8M3scTN718z6crZ91cwGzGy3mT1tZguLfO0hM+s1s5fNTH2WIlGKevI0zCStzp+ylDPC/zawddK2nUDa3TcC+4AvTfH1t7j7TcX6Q0WkQuXW46OePA0zSUd98kqYkhO+u78ADE3a9py7j2Ye/gzQdLtIPf3wPwdlkXJGy1FPnoaZpKM+eSVMmDX83wJ+VOQ5B54zsxfNbPtU38TMtpvZLjPbdezYsRDDE2kwu5+EXY9zxU1UphstR728Q5hJOuqTV8KE0qVjZg8Co8ATRXb5oLsfMbOrgZ1mNpB5x3AFd98B7IBgaYUw4pOYUndFdZ5/mIrvmLXxzuhe6y0PFV5ioZIkXavOnwZVdcI3s3uA24EtXmRhHnc/kvn4rpk9DWwGCiZ8aRK6s1L1pkrqczqCen4ck2DYSTrKk1fCVJXwzWwrcD/w6+5+vsg+7cAMdz+T+fxWQFPozW6qiTv98Zam2Nr1AMNn4UJmyi2OJ1Ml6UiU05b5HeBfgXVmdtjMfht4FJhHUKZ52cwey+x7rZk9m/nSpcBPzewV4OfAM+7+41B/CkkedVdUr1D9GoO2dhgbzt+sVkWhjBG+u99dYPO3iux7BNiW+fwg8J6KopPGlZQ7K8V5nqFYaeSpIn0ROpk2PS2tINEIc+KuVpIwz1CoNPL8w8k4mUrdKeFLNOrVXZEdob8F1gI+FrQhlnKspM4zxOVkmvvuaE5HsO3Cifi9U2oiSvgSnVpP3E0eoftY8LHUkXpS5xni0Ko4+bW/kHPNZhzfKTUJJXxpXIVG6BNKGaknZZ6hkKi7YKZ67SEZ75QakFbLlGQqZf2Y6Ubi0z2vqzgrV8q7oLi/U2pASviSPKWutjjdSHy656NegiDJSnkXlIR3Sg1GCV+Sp9TVFgv2qWeUOlKfWL/9yyeDj0r2pZnqtQe9U4qIEr4kT6mTqXkjdIIuHbg8Uof63+avFNXcfjCKWxcWMvnd0ZxFwT+9U4qUFVn+JhY2bdrku3bpfikyySPpIpOpK4NReCkmd5FAMOqMOhFVE1ehr8UAL70VVRLPzF4sdt8RjfAlecKYTI3rnZKqiatgZ0xmQKdb/wlK+JJEYUymxrXHvpq4ptsnDic0iZT68CWZqu0zr0WPfRjr7lQT11SrZ06Y6qQQ53WDJBQa4UuyhDUpGXaPfVg35q4mruk6Y6D4iSPMG4tLbCnhS3KEmZTC7rEPa06gmrgmdyVh+c9PdeKI65yGhEolHUmOsBczC3P5gUpr78XKKGHc/amcEk1c5zQkVEr4khz1TkrlJMw5HfkLhE2YqvZe6+WXyzlxJHndIM09lEwlHUmOYsmnFkmpnPLR7ifh0pkrt7e0TV17j1MZJanrBmnuoSxK+JIc9UxK5STj5x+G8ZErt7ddlZzll5O6blCcTpoJUHJJx8weB24H3nX3dGbbIuB7QBdwCLjT3U8U+NqtwNeBFuCv3f0rVUcuzaee67yXk4yL7Xvhij+FfHEro0S9pHIl4nTSTIByRvjfBrZO2vYA8Ly7rwGezzzOY2YtwDeB24Bu4G4z664oWpF6LWZWTvmo0lJTUssocVLPMl8DKDnhu/sLwORZqTuAv8l8/jfAfyjwpZuBA+5+0N2Hge9mvk4kvspJxpUm7nqUUeKymFqt6KRZlmq7dJa6+zsA7v6OmV1dYJ/lQO771sPA+6s8rkhtlVM+qqbUVGoZpZJOlCTchL1acbidY4LUoy3TCmwrukSnmW0HtgN0dnbWKiaR6ZVT065l/bvSxJ3Um7CXK4lzDxGptkvnqJldA5D5+G6BfQ4DK3MerwCOFPuG7r7D3Te5+6YlS5ZUGZ5IA6i0E6XohOZbtS/tlFJKavRyUxVqtWx9tQn/B8A9mc/vAb5fYJ9fAGvMbJWZtQF3Zb5Okk5/sPUxVeKe6nWfauKylr3qpfTGq38+69LYJV459gpP7H2C+1+4n48+9VH+ft/f1+RY5bRlfge4GUiZ2WHgz4CvAE+a2W8DbwKfzux7LUH75TZ3HzWz+4CfELRlPu7u/eH+GFJ3zVAfjoupVsGc6nXf8lCBG6Jk1LK0U0opqVnKTZOM+zivn3qd3sFe+gb76B3sZd/QPkZ9FICr515NT6qHZe3LanL8khO+u99d5KktBfY9AmzLefws8GzZ0Ul8NekfbCSmStxQ+HWfmOQt9jVQu171Unrji57AGqt//ui5o9nE3jfYR9/xPs6NnAOgvbWd9OI092y4h55UD+lUmqXtS2saj9bSkcrogpf6yetEKSFRFrzVYQG16lWf7oKy3U+SvfVivWKqgzPDZ+g/3h8k+GNBgn/3QjCtOXPGTNZ1rOP262+nJ9VDT6qHrgVdzLD6LnaghC+VKfZHPacjc8/ZhLXIxX0BrolOlKL3881JlNON7KG2veqF3pHkHu/5hyncqGeJ6Z8fGRth34l99A72Zkfvr596Hc/8XNfNv45fvuaXsyP39YvWM6tlVsRRK+FLpQr9Ube0BYuITawamZS6ftTzEeWcbKZLpjDNuyyr/Qltut74ovF5LH9P3J03z7yZV3cfOD7A8PgwAItmL2JjaiPbVm2jJ9XDhtQGFsxaEHHUhSnhS2UK/VEPn7tyieAk1PWjnI8o92RTyoVGRUsqK4PlKOphqt74qeKLgcELg/l198E+Tg+fBmDOzDl0L+7mszd+lnQqTU+qh2var8Gs0OVG8aOEL5Wb/Ef95YWF94t7XT/K+YhKTjbTXWhUyruAKMUovvMj59lzfE9egj9yLrhMqMVaWNOxhlu7bs2WZq5fcD0zZyQ3bSY3comfuK3+WKoo467FySbuyw1EFN/o+CivnXwtm9h3D+7mtZOvMe7jACy/ajkbl2zkszd+lp5UDzcuvpE5M6e5R3DCKOFLeGI0citLlHGXcrKpZEI57ssN1Dg+d+fts2/njdz3HN/DxbGLACyctZB0Ks2HOj9EOpUmnUqzaPaimsUTF0r4Ep64jyyLiTLu6U42UU8oJ8TJiyfpO55fdx+6GMwnzWqZxfpF6/nU2k9lWyJXzFuRmLp7mKxWazaEYdOmTb5r166owxCpralG8EXbMOs4ARszF0cvMjA0kNcS+daZ4DUyjNULV2dr7j2pHm7ouIHWGa0RR10/Zvaiu28q9JxG+CJRm6q80eQXuI2Nj12xFMH+E/uzSxEsa19GT6onO3rvXtxNe2t7xFFX75Gd+/jih9eG/n2V8EXiLOQJ5VolkjC4O0fPH80bufcP9nN+9DwA81rnsSG1gXvT92ZH8EvmNuaKul9/fr8SvkjTCXlCuVaJpBKnh0/TP9ifHbn3DvYyeGEQgNYZraxftJ47brgjm9yvm39d3ZciaDRK+CJxltSJ8EmGx4Z5dejVvNLModOHss93ze/iA9d8IFt3X7doHW0tbdEFHIFHdu7j68/vzz7ueuAZAP5wy5rQTtKatBWpRtzX4OHKRDIhzESSa9zHeeP0G5dH7sd6GTgxwOh4UHdPzUllu2XSqTQbUhuY3zY/9DiSrOuBZzj0lY9W9LWatBWphYS0TH7xw2uzib2aRFLMsfPH8kbu/YP9nBk5A8DcmXPZkNrA57s/n03yS+cubcqWyDhQwhepVBPeE+DcyDn2HN9z+WrVY7s5ev4oECxFsLZjLVtXbc0m91ULVtEyoyXiqJPnD7esqcn3VcIXqVQCWybLSSQj4yPsP7E/72rV106+ll0CeMVVK3jv0vdmk/v6ReuZPXN2rUJvKrWaWFfCF6lUAtcOKpZI3J3DZw5nu2V6B3sZGBrg0tglADpmdZBOpbn1ulvpWdLDhsUb6JjdUc/Qi4pzq2ncKOGLVCqpawcBQxeH8toh+wb7OHXpFACzW2bTvbibz6z7THZidflVy/Pr7jGarI5Tq2ncVZ3wzWwd8L2cTdcDD7n7/8zZ52bg+8DrmU1PufvD1R5bJFIJaZm8MHqBvcf35iX3t8++DcAMm8HqhavZ0rmFdCrNxtRGVi9cPfUSwAmZrJYrhdqWaWYtwNvA+939jZztNwN/4u63l/P91JYpUp6JJYBz6+4HTh5gzMcAuLb92myvezqVpntxN3Nb55Z3kBis71PvVtMkqWdb5hbgtdxkLyK14e68c+6dbK9772Ave4f2cmE0GHnPa5tHT6qHm1fenL31XmpOqvoDx2Cyutatpo0q7IR/F/CdIs99wMxeAY4QjPb7C+1kZtuB7QCdnZ0hhyeSXKcuncobufcO9maXAG6b0cb6xev55JpPZkfwnfM6a9PvnsDJagmElvDNrA34OPClAk+/BFzn7mfNbBvwj0DB/jB33wHsgKCkE1Z8klAxmhysp0tjlxgYGsj2uvcN9vHmmTeBYAngVQtW8avLf5WNqY2kl6RZu3AtrS11WgI4ZpPVtepZb0RhjvBvA15y96OTn3D30zmfP2tm/8vMUu4+GOLxpdE0yeTguI9fsQTwvqF92SWAr557NT2pHj6x5hPZJYDntc0r+v1q3qYYs8nqZq/ZlyPMhH83Rco5ZrYMOOrubmabgRnA8RCPLY2oQa9kPXruaPaeqn2DffQf7+fcyDkA2lvbSS9O84X0F4Jb7y1Os7R9aVnfP8w2xaInj7jfQlEKCiXhm9lc4MPA7+Rs+10Ad38M+BTwe2Y2ClwA7vI4r9rWYBJ7YUoMJgerdWb4DP3HM0sAHwtG8O9eeBeAmTNmsq5jHbdff3u2a2bVglV5SwAH/3flJfwwqce9sYSS8N39PLB40rbHcj5/FHg0jGNJ+RL7R5uwycGRsRH2ndiX1+/++qnXs0sRXDf/OjZfszlvCeBZLbOm/J6l/t/VY2ldST5daSvxFbPJwVzuzhun38i7afbeob2MjI8AsGj2IjamNrJt1bZsS+SCWQtqFk+YbYo6eTQurYffoBrmwpSYdOkMXhi8oiXyzHCwBPCcmXPoXtwddMxkRu/L2pdV3BJZ7f9dmH3p6nFPnqkuvFLCbwL6o53GpJPK+ZsfYM816/NKM++cewcIlgBe07Em72rV6xdcP/VSBFWo5P8uzDkb/e5ML25zZLoBikgRo698lwM/+WN6W5y+VAe9s0Z47ZWvML47GJ0vv2o5Ny25ic/d+Dl6lgRLAM+ZOSfiqKcWZvJRj/v0kjRHpoTfBPRHG3B33j77dl5pZs/RF7m4dBEAC8fGSF8a5kPnLpCeOZ/0f3qBRbMXRRpz1P93SUlkUhqVdKRhnbx4Mq/m3jfYx4lLJwCY1TKL7sXdpPe/QM+lS6QvXWLF6BiXq+4GXz4ZVegSc3GeI1NJRxrexdGLDAwN5NXd3zoTtHQaxuqFq7l55c3Z2vsNHTfQOqMVBtJw7sSV37DOrZ9xqwOXI8mxVyqpi7cp4Uu0KujCGRsf4+Cpg3kj9/0n9meXIljWvoyeVA+fXvvp7BLA7a3thb9ZTFo/k1QHnizJsTcbJXyJTglr5bg7R88fzRu59w/2c370PADzWueRTqW5N31vtmtmydwlpccQs3VhStGMI+o4i3qepRyq4Ut0CtxI4/QMo69jBX2//gfZBD94IVhjr3VGK+sXrc+WZXpSPXTO78xbiiBJKq0Dx6GEEOcadrNTH77E0vCXF/JqWyu9s9romzWL3lltHGq7vMTvqgWrsqP2nlQPazvW0tbSFlm8tRxZl5PE45DwJ0wk/rjEI5q0lRgY93EOnT6Ut4jYQNcKRjNXo6ZGx+i5dImPnz1HeuYCNvzO/2N+2/yIo84XZa06rssdFBrlS3wp4UtNHDt/LK8lsn+wnzMjwVIEc2fOZUNqA7+59IP07PkR6fNnWTqWaYlsnQMfewhiluxrbbo6cJy7QpJUw252KulI1c4On2XP8T15Cf7o+eA+ODNtJms61uSVZlYtWEXLjJbgi2OyVk4xcaxVR53w4/iayGWq4UtoRsZH2H9ifzax9x7r5eCpg9klgFfOW5k3qbp+0Xpmz5wdcdThiDrRTohTl07Ur0mcXou4UA1fKuLuvHXmrbyR+8DQAJfGLgHQMauDniU9fGTVR4IR/OI0C2cvjDjqxqcEd5muASiPEr5kHb9wnP7j/Xk976cunQJgdstsuhd3c+e6O7PLAC+/annFSwAnkWrVV9Jrkiwq6TSp8yPn2Tu0N+9q1bfPvg3ADJvBDQtvyKu7r164umZLAIuUQ3MIU1MNv8mNjo/y2snXLtfdB3s5cPIA4z4OwLXt1+at7969uJu5rXMjjlqaVTl1+ajvFxBHNa/hm9kh4AwwBoxOPpgF7/u/DmwDzgNfcPeXwji25HN3jpw7EozajwUJfu/QXi6MBssXzG+bT0+qh1tW3pK99V5qTiriqKURVZpYa12Xb+a6f5jv0W9x98Eiz90GrMn8ez/wl5mPUqVTl07ljdz7BvsYujgEQNuMNtYvXs8n13wyO4LvnNfZVHV3iU49EqvmEMpTr6LsHcDfelA/+pmZLTSza9z9nTodvyFMLAGcW3d/88ybQLAE8PULrufXlv9aUJpZkmbtwrW0trRO810lTI1eLqiVSq8kLvW1juuVyvUWSg3fzF4HTgAO/JW775j0/A+Br7j7TzOPnwfud/crCvRmth3YDtDZ2fm+N954o+r4kmhsfIxDpw9le917B3vzlgC+eu7VeZOq3Yu7mdc2L+Kow5XE5Bl1X3rUwphQrfVr2Oj/R/Xow/+gux8xs6uBnWY24O4v5MZQ4GsKnmkyJ4sdEEzahhRfrE0sAZw7cu8/3s+5kXMAXNV6FRtSG/hC+gukU2nSi9MsbV8acdS118y11qSK8xIQElLCd/cjmY/vmtnTwGYgN+EfBlbmPF4BHAnj2El0ZvgM/cf76RvsY/ex3fQN9nHswjEAZs6YybqOddx+/e3Zq1W7FnQldgngZqByQbhqXZdv5rp/1SUdM2sHZrj7mcznO4GH3f3HOft8FLiPoEvn/cA33H3zdN+7Edoyh8eG2XdiX97Vqq+fej37fNf8LjakNmST+7pF65jVMivCiKOV9B5rjWovS2JJrhHUuqSzFHg60/kxE/g7d/+xmf0ugLs/BjxLkOwPELRl3hvCcWNn3Md58/SbeR0zA0MDjIyPALBo9iI2pjby0VUfzbZELpi1IOKo46VeJQElo9rT6xs/VSd8dz8IvKfA9sdyPnfg96s9VtwMXhjMTqj2DfbRd7yPM8PBEsBzZs5hw+INfO7Gz2UnVpe1L1NLZEzUan6gmcsFEn+6Vr5E50fOZ+vuEyP4fz/37wC0WAtrOtbwka6PZDtnVi9YfXkJYKlIrZNnLUb5GtVKnGlphQJGxkc4cOJAXt394KmD2aUIll+1PLuAWM+SYAngOTPn1D1OKU/S5wdESqG1dKbg7hw+ezhvffe9Q3uzSwAvnLUwb52ZdCrNotmLahqT1N5EJ02YcwSaF5A40Hr4OYYuDgX19pye95OXTgIwq2VWdgngiQS/4qoVqrs3kFq2UOq6AYm7hk74F0YvMDA0kJ1Y7R3szS4BbBirF67mlpW3ZEfwN3TcQOsMLUXQyCa6gCYSv1oopZk0XMIfHhvmz3/+5/QN9rH/xH7GfAyAZe3L6En18Jl1n8kuAdze2h5xtBKVL354bcF6frl00ZUkSUPW8D/x/U+wZM6SvNr7krlLahChJE0tJ2510ZXEQdPV8J++4+moQ5CYasS1XjRZLKXSAi0iIYnqoqswSlPSHJTwpWmFkaAf2bkv+7lG2RJ3SvjStMJI0FGNrh/ZuY+uB57JThJPfJ57ApLL9LoEGrKGL9LokjAXEae5BV0jEVDCFymTWjFLoyQbP0r4ImWK2+haK3QWphPzlRqyD1+kXipJ+HEqdYQtrgvUxeHEXC9N14cvUi+VjK4budQRt3c/kk9dOlKUOhumF1bijuNrHceYKqWyV0AJX4rSBT3hma6NMo6vdbUxxSnJNuo7qnJVXdIxs5XA3wLLgHFgh7t/fdI+NwPfBybu3v2Uuz9c7bFFkiLOpY5azSkoycZPGDX8UeCP3f0lM5sHvGhmO919z6T9/q+73x7C8aSG1NlQP3F5rXPnFOISk9SIu4f6j2Ak/+FJ224Gflju93rf+97nEp3r7v9h1CE0pK899+oV26J8rYsdW///yQTs8iI5NdQuHTPrAn4J+LcCT3/AzF4BjgB/4u79Rb7HdmA7QGdnZ5jhicRCHEbKGsk3p9ASvpldBfwD8EfufnrS0y8B17n7WTPbBvwjUHBGx913ADsg6MMPKz4pX5wm3RpdvV/rUuYUahVTI1+HEHehdOmYWStBsn/C3Z+a/Ly7n3b3s5nPnwVazSwVxrGldvRHWT9xfK1rFVMcO5KaRdUJ34I7fH8L2OvuXyuyz7LMfpjZ5sxxj1d7bBGZ3nT99Hon1zzCKOl8EPg80GtmL2e2/SnQCeDujwGfAn7PzEaBC8BdmckFEamx6a7srce7C80ZxEPVCd/dfwrYNPs8Cjxa7bFEJJnifB1CM9FaOtJQNCEY0IhaClHCl4bSyAuTlaNWI+owTqiaM4iO1tIRkZKF0WGjE3J0NMKXxFP5YmoaUcsE3QBFGoomBMMX15uaSGG6AYpImTT5e5k6bBqHavjSUMIqX+hqUGlESvjSUDQqry3NBySbavgiGapVSyOYqoavhC9SQJxq1ZpPkHJMlfBV0hGJOc0nSFiU8EUKUK1aGpFKOiIxpPkEqZRq+CIJFqf5hFJoziFaquFLok13A4+kaJSfYzqac4gvJXyJvUZJIJX+HJpPkLBoaQWRmEtCeUQL2CWDavhyhTjUYBtl0rJRfo5yJG3OodFo8TQpSxxuItIoC3ZV83PE4cQrjSWUGr6ZbTWzV83sgJk9UOB5M7NvZJ7fbWbvDeO4kizNMmkZlqTOXWjOIb6qTvhm1gJ8E7gN6AbuNrPuSbvdBqzJ/NsO/GW1x5VwPbJzH10PPJOtvU58HmaSbvZJy0b5OaajdyXxFUZJZzNwwN0PApjZd4E7gD05+9wB/K0HEwY/M7OFZnaNu78TwvElBHEuoTRKAinl59Dkp9RSGAl/OfBWzuPDwPtL2Gc5cEXCN7PtBO8C6OzsDCE8iZISWHnifOKV5Asj4VuBbZNbf0rZJ9jovgPYAUGXTnWhSSXCLD0ogYnERxiTtoeBlTmPVwBHKthHYkIj73holpq/1E8YCf8XwBozW2VmbcBdwA8m7fMD4Dcz3Tq/ApxS/b65PLJznxJYmXTilbBVXdJx91Ezuw/4CdACPO7u/Wb2u5nnHwOeBbYBB4DzwL3VHleS5evP71c5RyRioVx45e7PEiT13G2P5XzuwO+HcSyRZqELryRsutJWakYdOtWJwxXP0liU8KVm1KEjEi9K+CIxondFUktK+FIX6tApjd4VSS3pBihSFxqdikRPCV8kpvSuSMKmhC8SU3pXJGFTwpempjX6pZko4UtTS+pNRkQqoYRfgEZ9Uin97kicKeEXoFFfY6vl3b30uyNxpj58aToTve4TCV+97tIslPAzdIWjVEq/O5IUFixkGU+bNm3yXbt21f24usKxcU1OzhPCSs763ZGomdmL7r6p0HMa4UtT0dIF0sw0aVuArnCUSidw9bsjcaaEX4Dqrs1hquRcabeNfnckzpTwJfZq1duu5CzNpqoavpl9FfgYMAy8Btzr7icL7HcIOAOMAaPFJhRECqnXnZ/UbSONrtpJ253Al6RICecAAAU3SURBVDI3Mv8fwJeA+4vse4u7D1Z5PJGa0YSuNLqqEr67P5fz8GfAp6oLRySg0bZI+MJsy/wt4HtFnnPgOTNz4K/cfUexb2Jm24HtAJ2dnSGGJ0kS9Whb3TbSiKZN+Gb2z8CyAk896O7fz+zzIDAKPFHk23zQ3Y+Y2dXATjMbcPcXCu2YORnsgODCqxJ+BpHQ6V2ENKJpE767f2iq583sHuB2YIsXuWzX3Y9kPr5rZk8Dm4GCCV9kMo22RcJRVVummW0lmKT9uLufL7JPu5nNm/gcuBXoq+a40lw02hYJR7V9+I8C8wjKNC+b2WMAZnatmT2b2Wcp8FMzewX4OfCMu/+4yuOKiEiZqu3SuaHI9iPAtsznB4H3VHMckXI8snOf3hWIFKArbaXh6CYkIoUp4YuINAktjywNQRdqiUxPN0CRhqNlEaSZTXUDFJV0RESahBK+NBxdqCVSmBK+NBzV7EUKU8IXEWkSSvgiIk1CCV9EpEko4YuINAklfBGRJhHrC6/M7BjwRoVfngKScg/dJMUKyYpXsdZOkuJNUqxQXbzXufuSQk/EOuFXw8x2FbvaLG6SFCskK17FWjtJijdJsULt4lVJR0SkSSjhi4g0iUZO+DuiDqAMSYoVkhWvYq2dJMWbpFihRvE2bA1fRETyNfIIX0REcijhi4g0iaZI+Gb2J2bmZpaKOpZizOy/mdluM3vZzJ4zs2ujjqkYM/uqmQ1k4n3azBZGHdNUzOzTZtZvZuNmFsvWPDPbamavmtkBM3sg6nimYmaPm9m7ZtYXdSzTMbOVZvYvZrY38zvwh1HHVIyZzTazn5vZK5lY/2vYx2j4hG9mK4EPA29GHcs0vuruG939JuCHwENRBzSFnUDa3TcC+4AvRRzPdPqATwIvRB1IIWbWAnwTuA3oBu42s+5oo5rSt4GtUQdRolHgj939RuBXgN+P8Wt7CfgNd38PcBOw1cx+JcwDNHzCBx4B/gsQ69lpdz+d87CdGMfr7s+5+2jm4c+AFVHGMx133+vur0YdxxQ2Awfc/aC7DwPfBe6IOKai3P0FYCjqOErh7u+4+0uZz88Ae4Hl0UZVmAfOZh62Zv6FmgcaOuGb2ceBt939lahjKYWZ/Xczewv4j8R7hJ/rt4AfRR1Ewi0H3sp5fJiYJqUkM7Mu4JeAf4s2kuLMrMXMXgbeBXa6e6ixzgzzm0XBzP4ZWFbgqQeBPwVurW9ExU0Vq7t/390fBB40sy8B9wF/VtcAc0wXa2afBwneMj9Rz9gKKSXeGLMC22L7Di+JzOwq4B+AP5r0bjpW3H0MuCkzL/a0maXdPbS5ksQnfHf/UKHtZtYDrAJeMTMIyg4vmdlmd//3OoaYVSzWAv4OeIYIE/50sZrZPcDtwBaPwcUcZby2cXQYWJnzeAVwJKJYGo6ZtRIk+yfc/amo4ymFu580s/9DMFcSWsJv2JKOu/e6+9Xu3uXuXQR/VO+NKtlPx8xy77z9cWAgqlimY2ZbgfuBj7v7+ajjaQC/ANaY2SozawPuAn4QcUwNwYLR3reAve7+tajjmYqZLZnoeDOzOcCHCDkPNGzCT6CvmFmfme0mKEPFtn0MeBSYB+zMtJE+FnVAUzGzT5jZYeADwDNm9pOoY8qVmQC/D/gJwaTik+7eH21UxZnZd4B/BdaZ2WEz++2oY5rCB4HPA7+R+V192cy2RR1UEdcA/5LJAb8gqOH/MMwDaGkFEZEmoRG+iEiTUMIXEWkSSvgiIk1CCV9EpEko4YuINAklfBGRJqGELyLSJP4/HCI46As+bhAAAAAASUVORK5CYII=) ##### 牛顿法 牛顿法在求解方程$f(\theta)=0$的根时主要是根据泰勒展开式进行迭代求解，假设有初始近似解$x_k$，那么$f(x)$在点$x_k$处的泰勒展开式 $$ f(x)\approx f(x_k)+f'(x_k)(x-x_k) $$ 令$f(x)=0$求解得到$x_{k+1}$ $$ x_{k+1} = x_k - \frac{f(x_k)}{f'(x_k)} $$ 牛顿法的几何解释如下图 ![这里写图片描述](img/20160607212855512.png) 在逻辑回归中，损失函数的最小值在$J'(w)=0$处。用牛顿迭代法求参数 $$ w_{t+1} \leftarrow w_t - \frac{J'(w)}{J''(w)} $$ ### 2. 感知器 perceptron 感知器是一种广泛使用的线性分类器．感知器可谓是最简单的人工神经网络，只有一个神经元． 感知器是一种简单的两类线性分类模型，其分类准则为： $$ \hat{y}=\operatorname{sgn}\left(\boldsymbol{w}^{\top} \boldsymbol{x}\right) $$ ![image-20200703091533910](img/image-20200703091533910.png) #### 参数学习 感知器的参数学习实际上也是经典的线性分类器的参数学习方法。给定N个样本 ${x^i,y^i}_{i=1}^N$，参数学习的目标是找到一组参数，使得对于每个分类样本都有： $$ y^{(n)} \boldsymbol{w}^{* \top} \boldsymbol{x}^{(n)}>0, \quad \forall n \in\{1, \cdots, N\} $$ 感知器的学习算法是一种错误驱动的在线学习算法，首先初始化一个权重，然后每错分一个样本就利用这个样本来更新权重。 $$ \boldsymbol{w} \leftarrow \boldsymbol{w}+y \boldsymbol{x} $$ ![image-20200703105227533](img/image-20200703105227533.png) #### 实验结果 ![img](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAXwAAAD4CAYAAADvsV2wAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAf0UlEQVR4nO3dfbBcdZ3n8feHPJMESQhgIAlRDI7AImoKtZwqcSJWYBBWx2WhtmZctTbrlpQPO+6CQxXDsrW7brlrChdLzIwUY62jsjUwOhCVTGqmWHdk5EJBDILAMLDEsILkAsE83uS7f3Tf0LfvOf14us/T51V163afc26fX/e993t+5/v7nt9RRGBmZtV3XN4NMDOz8XDANzOrCQd8M7OacMA3M6sJB3wzs5qYm3cDOlmxYkWsXbs272aYmZXGAw888OuIODlpXaED/tq1a5mYmMi7GWZmpSHpmbR1TumYmdWEA76ZWU30HPAl3SrpeUk7W5bdIOmXkh5qfl2S8rMbJf1C0pOSrs2i4WZm1p9+evi3ARsTlm+OiPObX1vbV0qaA3wVuBg4G7hK0tmDNNbMzAbXc8CPiHuBPQPs4wLgyYh4KiIOAd8BLh/gdczMbAhZ5PCvlrSjmfJZlrD+dODZlue7mssSSdokaULSxAsvvJBB88wqasftsPlcuOHExvcdt+fdIiu4YQP+14AzgfOB54D/nrCNEpalTtEZEVsiYn1ErD/55MRSUjPbcTv81afh5WeBaHz/q0876FtHQwX8iPhVRByJiKPAn9BI37TbBaxueb4K2D3Mfs1qb/uNcHj/zGWH9zeWm6UYKuBLWtny9EPAzoTN7gfWSXqDpPnAlcD3h9mvWe29vKu/5Wb0caWtpG8DFwIrJO0C/hi4UNL5NFI0TwP/urntacCfRsQlETEl6WrgR8Ac4NaIeCTTd2FWN69b1UznJCw3S9FzwI+IqxIWfyNl293AJS3PtwKzSjbNbEAbrm/k7FvTOvMWNZabpfCVtmZldN4V8MGvwOtWA2p8/+BXGsvNUhR68jQz6+C8KxzgrS/u4ZvlzfX0Nibu4ZvlabqefjoXP11PD+69W+bcwzfLk+vpbYzcwzcbtx23NwL6y7tIveg8z3r61va9blWj8sdnG5XggG82Tu0pnDR51dM7xVRpTumYjVNSCqddnvX0TjFVmgO+2Th1TNUUoJ7eUzZUmlM6ZuOUOiXCavhc0lRUY+YpGyrNPXyzcdpwfSNl06pIUyIUvX02FAd8s3Eq+pQIo26fLzLLlSJS70WSu/Xr18fExETezTCzLCRVKM1bVKwDXgVIeiAi1ietcw/fzMbDFUC5c8A3s/FwBVDuHPDNbDzSKn1cATQ2DvhmdZTH4KkrgHLnOnyzuslr+oTp1856nh7P/dMz9/DN6ibPwdPzrmhcYPbhLY3nd2wa7gxj+uD18rNAvHbwcrlnop4DvqRbJT0vaWfLsi9JekzSDkl3Sjox5WeflvQzSQ9Jcp2lWZ7yHjzNMki78qcv/fTwbwM2ti3bBpwbEecBjwNf6PDz74uI89PqQ81sQP3m4/MePM0ySOd98CqZngN+RNwL7Glbdk9ETDWf3gd4uN1snO76t420SD+95bwHT7MM0nkfvEomyxz+x4EfpKwL4B5JD0ja1OlFJG2SNCFp4oUXXsiweWYVs+N2mLiVWTdR6dZbznt6hyyDdN4Hr5LJpEpH0nXAFPCtlE3eExG7JZ0CbJP0WPOMYZaI2AJsgcbUClm0zwrK1RXD2X4jA98x67wr8vusN1yfPMXCIEF6VJU/FTV0wJf0UeBSYEOkTMwTEbub35+XdCdwAZAY8K0mfGel4XUK6ouWNfL5RQyCWQfpPA9eJTNUwJe0EbgGeG9E7EvZZjFwXETsbT7+AOAh9LrrNHDnf97epM1dD3DoVdjfHHIr4sHUQToX/ZRlfhv4CfBmSbskfQK4GVhKI03zkKRbmtueJmlr80dPBX4s6WHgp8DdEfHDTN+FlY+rK4aXlL9GMH8xHDk0c7FLFY0+evgRcVXC4m+kbLsbuKT5+CngrQO1zqqrLHdWKvI4Q1pq5I6UuggfTGvPUytYPrIcuBuVMowzJKVGtt9YjoOpjZ0DvuVjXNUVx3roz4LmQBxplCH2sq+yjjMU5WDaena0aFlj2f7J4p0p1YgDvuVn1AN37T30ONL43mtPvazjDEUoVWz/7Pe3XLNZxDOlmnDAt+pK6qFP66WnXpZxhiR5V8F0+uyhHGdKFeTZMq2cepk/pltPvNt6X8U5uF7Ogop+plRBDvhWPr3OttitJ95tfd5TEJRZL2dBZThTqhgHfCufXmdbTKxTb+q1pz49f/sNLzW+O9j3ptNnDz5TyokDvpVPr4OpM3roNKp04LWeOoz/Nn+9GOb2g3ncujBJ+9nRouWNL58p5Uop098Uwvr162NiwvdLsTabz00ZTF3d6IX3or2KBBq9zrwD0TDtSvpZBETvpahWepIeSLvviHv4Vj5ZDKYW9U5Jw7QrsTKm2aHzrf8MB3wroywGU4taYz9Mu7ptU4QDmuXKdfhWTsPWmY+ixj6LeXeGaVen2TOndTooFHneIMuEe/hWLlkNSmZdY5/VjbmHaVe3yhhIP3BkeWNxKywHfCuPLINS1jX2WY0JDNOu9qokNHN9pwNHUcc0LFNO6Vh5ZD2ZWZbTDwyae09Lo2Rx96d+UjRFHdOwTDngW3mMOyj1EzAXLZs5Qdi0Trn3UU+/3M+Bo8zzBnnsoWdO6Vh5pAWfUQSlftJHO26Hg3tnL58zv3PuvUhplLLOG+Sxh7444Ft5jDMo9ROMt98IRw/PXj5/SXmmXy7rvEFFOmiWQM8pHUm3ApcCz0fEuc1ly4HvAmuBp4ErImIy4Wc3AjcBc4A/jYgvDt1yq59xzvPeTzBO23b/rH+FmYqWRsl7SuVBFOmgWQL99PBvAza2LbsW2B4R64DtzeczSJoDfBW4GDgbuErS2QO11mxck5n1kz4aNNVU1jRKkYwzzVcBPQf8iLgXaB+Vuhz4s+bjPwP+acKPXgA8GRFPRcQh4DvNnzMrrn6C8aCBexxplKJMpjYqPmj2ZdgqnVMj4jmAiHhO0ikJ25wOtJ637gLeOeR+zUarn/TRMKmmXtMog1SilOEm7MMqwu0cS2QcZZlKWJY6RaekTcAmgDVr1oyqTWbd9ZPTHmX+e9DAXdabsPerjGMPORm2SudXklYCNL8/n7DNLmB1y/NVwO60F4yILRGxPiLWn3zyyUM2z6wCBq1ESR3QfHb0qZ1eUklVTzcV0LAB//vAR5uPPwp8L2Gb+4F1kt4gaT5wZfPnrOz8DzsenQJ3p8+908DlKGvVe6mNd/18LnoO+JK+DfwEeLOkXZI+AXwRuEjSE8BFzedIOk3SVoCImAKuBn4EPArcHhGPZPs2bOz8Dzs+nQJ3p8+902Rqo6xV7+WMxPXzueg5hx8RV6Ws2pCw7W7gkpbnW4GtfbfOiqsu+eEi2HB9wp2sWiR97tODvGk/A6OrVe+lNj5tGmfXz4+Ur7S1wfiCl/GZNQtmgtbPfcbZVwejqlXvVhu/43aSazk6/KxlwgHfBpP2j7loWTnz+kUfj5i+4Cwt6Lf+Prr17GG0terdauO330hyoZ5cPz9iDvg2mKR/6jnzG5OIlS2vn/d4RD8Hm14uNOp4ljWGeXK6XVCW2r5wOnDEPD2yDSbpgpdDv5k9RXAZ8vp5jkf0W2Pfy4VGqXP0rG6cJYxDp9r4Tu2zkXLAt8G1/1PfcGLydkXP6+c5HjHIwabbhUZJg7xFmm6g6O2rMKd0LDtlncgqz3aP4mBT9KmOi96+CnMP37JT1p5bnu3uZYrkQebRKfp0A0VvX0W5h2/ZKWvPLc92dxuEzXtA2SpFEanzmOVu/fr1MTExkXczzEarUw9+87n5D8BaqUh6ICLWJ61zSscsb53SG77ArZY2b3ucz110Vuav65SOWZFlPKC8edvjQzTGxuWm7U+M5HUd8M2KLOM7Oo0qkFg5OKVjVmS+o1NtbN72+IwD8tpr7wbgMxvWZZbe8aCt2TAGKZkcs/ZAMi3LQGLZWnvt3Tz9xd8d6Gc9aGs2CiW5Z+znLjrrWGAfJpBY+TmHbzYo38TDRuQzG9aN5HUd8M0GVcKSyVEFEsvWqFJtDvhmgyrh3EFVzNm71LR3Dvhmg8q4ZLJUCnTDGJea9m7ogC/pzZIeavl6RdJn27a5UNLLLdvU4D/CKq+scwcNy/P7lFamZZmS5gC/BN4ZEc+0LL8Q+HxEXNrP67ks06yACjC/j0tN042zLHMD8A+twd7MKqYAg9UuNR1M1jn8K4Fvp6x7t6SHJf1A0jlpLyBpk6QJSRMvvPBCxs0zs6GVcLDaGjIL+JLmA5cB/yth9YPAGRHxVuB/AH+Z9joRsSUi1kfE+pNPPjmr5llZFWhw0JoKNljtUtPeZdnDvxh4MCJ+1b4iIl6JiFebj7cC8yStyHDfVkUeHBzIyMsUCzZYXfecfT+yDPhXkZLOkfR6SWo+vqC53xcz3LdVka9kHUiWZYqpB4/zrmgM0N7wUuN71SuTKiKTgC/peOAi4I6WZZ+U9Mnm048AOyU9DHwFuDKKPGtbxZT2wpQCDA7mLe/fnWvcqyWTKp2I2Aec1LbslpbHNwM3Z7Ev699N258o52lvLzf4rrhef3fjmFrXRuNoHOWVg6+w58CeY19nnngmZ554Zub78myZVlwbrp85GyXU50rWPmVZpuiDx3COBfCDe9izfw+TByeZPDDJiwdeZPLAJHsO7Dn2fc+BPbx88GWOxJEZr/Hpt316JAHf8+FXVGUuTCnBfPNZG/Z3l2VdumvcZwbwpIDd/vilgy/NCuDTls5fyvKFy1m+cDnLFixj2cJlrz1vebxyyUpOmH/CQO3tdOGVA34N+J+2iwIfVAb53WV5A+wq/u0cjaPsPbQ3NWAf+97soXcM4POWoqNLeOPyU2cF7WULGwH9pIUnNR4vWMa8OfNG/v58AxSzNCW5iUk/sjyDK0ONe0TwyqFXugfv5uPJA5MdA/h00F61ZBXnrThvRiCfEdCbAXzttXfzf0pyUHTAr4Ey/NPmplPpZwECft6/uzzSf9MBfPLAJJMHJ9mzf8+MfHh7QH/pwEtMxVTiay2Zt+RYoJ4O4DOC94LlLF/0Wnpl/pz5Y3634+WUjtXbDScCSf8DatSY29Aigr2H96b2vqcHM48tOzjJ1NH0AJ7W216+aDnLF8zsjY8qgBd5jMwpHbM0BSn9zDLvPmoRwauHXz0WoG+7bye//VsLO6ZT0gL44nmLjwXnlUtWcs6KcxrBOyEfvnzh8sL0wMs6eZsDvuUr7wHTgpR+5nmtRHsAP5brPjjJi/tfPJZWmU6nTB6Y5PDRwzNeY/tPGt8Xz1t8rLe9cvFKzj7p7NTgvWzhMhbMWZDDO64vB3zLTxEGTKf3U9AqnSTdzgYigt8c/s3s2u+2/Hfr+vYAPu34uccfC9SnHn8qb1n+llnB+w/+5Of83b+7rLYBPO9xln44h2/5KcCNNPLUax64NYDvObiHD3/9Hr70z89MTp80l3UK4Gm136097+lSwrQAXuQcdt25Dt+KqWQDplnl2SOCfVP7ZgTsf/U//5ZrLl2VemHPoaOHEl9r0dxFM+u+WwYvp6tPWtcvnLtw6Pa3mg78Zclh14EHba2YCjJg2qu0PHtrAO96JWYzH94ewBeeBjc92Ajg00F6xaIVnLXsLJYvXM7OZ6e497EDxNRi4siS5vfFfPx3zsm1R+3J1crFAd/yU5AB03YRwf6p/bMC9vyT/o7/dv+jjeVtl9kfPHIw8bUWzll4rHe9YtEK1i1bl5hOueP+l/j8+9/O8fOOT25US3+taFUhZcph150DvuVnTAOmrQE8qbc9eXB2LXhSAF9wCty2cx5xZDGnHH8Sv3XKSt504ptSa8KXLViWHsDbnHtxpm95pNrz9zdtf4Kbtj/h/H0JOIdvpbTv8L5Z5YKdLq0/cORA4ussmLOgY8A+adFJx67C/O3/fD9P/5cPjfmdzlakmv28zzaK9FkUhXP4VniJPfC0C3kO7OkYwFurTd74ujemVqMsX7icRXMX0bwZW3fxcIbveHAOcK8p7b0ecuKAbyOxf2r/rFrv1gDeenHPngN72D+1P/F15h83/7Vqk0XLeePr3jgjYLfPSthXAO+Tc9Wz+TMpF6d0rCcHpg7MmDK2dVKrpMqUTgG8U8Cefjy97vi5x48sgFs5+RqAzlyHb7O0BvBebuqwb2pf4uu0B/D2gN0+K6EDuHXTT14+7/sFFNHIc/iSngb2AkeAqfadqfEffhNwCbAP+JcR8WAW+7aGg0cO9pz/3nNgT2oAn3fcvBm97TNOOGPWgGbr+sXzFjuAW6JBA+uo8/J1zvtnmcN/X0T8OmXdxcC65tc7ga81v1uK1gDeLXhPHpzkN4d/k/g6c4+bOyNYrz5h9azqk9beuAO4ZWUcgdVjCP0Z16Dt5cA3o5E/uk/SiZJWRsRzY9p/7g4dOTQrUM8YuGzLh6cGcM2dUWmyaumqjvOiLJm3xAF8jKqeLhiVQW+c3utn7RuzN2SSw5f0j8AkjYlRvh4RW9rW3wV8MSJ+3Hy+HbgmImYl6CVtAjYBrFmz5h3PPPPM0O0bhdYAPqMS5WBCQD8wyauHX018nbmaOyPvnTZ4Ob1u6byltQngZQyeedel5y2LAdVRf4ZV/x2Now7/PRGxW9IpwDZJj0XEva1tSPiZxCNN82CxBRqDthm1r6vDRw7PuPqytRolqbSwlwC+bOEyTl9x+qw78rTeUu2E+SfUJoD3q8651rIq641B6iKTgB8Ru5vfn5d0J3AB0BrwdwGrW56vAnZnse80MwJ4W287qQe+9/DexNeZozkzetvnnHTOjIDd3ht3AK8fpwuyNeq8fJ3z/kOndCQtBo6LiL3Nx9uAGyPihy3b/C5wNY0qnXcCX4mIC7q99iBlmRHBe7/7XiYPTiaunxHAF3QpJVy4nKXzl3KcjuurDTa4stdYu1f7mjKm5Kpg1CmdU4E7m73aucCfR8QPJX0SICJuAbbSCPZP0ijL/FgG+00kid876/eOzZHSfmGPA3ixjSsl4GA0ev58i2fogB8RTwFvTVh+S8vjAD417L569Zm3f2Zcu7KSGtX4QJ3TBVZ87upaYY06eG7e9njmr+lerRWZp1aw2ij7+IBZLzyXjlmb6UqaLMcIPC5gRdAp4DulY7Wyedvjx4I9NAL/2mvvziS94/u7WtF5PnyrlekqoOn0jksorU4c8K2WPnfRWZn0yH3RlZWJc/hWK6McuPVFV1YEvqetWVMV53rxYLH1yoO2ZhnJ66IrDxZbrxzwrbayCNCt1T3uZVvROeBbbWURoPPqXU+Xl04PEmdZXlpF/lwanMM3K6EyjEUUaWzB91ZocMA365NLMXvjIFs8DvhmfSpa79ozdCbzgXk21+GbDWGQgF+kVEfWijpBXREOzOPiOnyzERmkd13lVEfRzn5sJlfpWCpXNnSXVeAu4mddxDYNymmvBgd8S+ULerLTrYyyiJ/1sG0qUpCt6hlVv4ZO6UhaDXwTeD1wFNgSETe1bXMh8D3gH5uL7oiIG4fdt1lZFDnVMaoxBQfZ4skihz8F/GFEPChpKfCApG0R8fO27f53RFyawf5shFzZMD5F+axbxxSK0iYbkYjI9ItGT/6itmUXAnf1+1rveMc7wvJzxjV35d2ESvryPb+YtSzPzzpt3/79lxMwESkxNdMqHUlrgbcBf5+w+t2SHgZ2A5+PiEdSXmMTsAlgzZo1WTbPrBCK0FN2T76eMgv4kpYAfwF8NiJeaVv9IHBGRLwq6RLgL4HEEZ2I2AJsgUYdflbts/4VadCt6sb9WfcypjCqNlX5OoSiy6RKR9I8GsH+WxFxR/v6iHglIl5tPt4KzJO0Iot92+j4n3J8ivhZj6pNRaxIqouhA74kAd8AHo2IL6ds8/rmdki6oLnfF4fdt5l1162e3mdy9ZFFSuc9wO8DP5P0UHPZHwFrACLiFuAjwL+RNAXsB65sDi6Y2Yh1u7J3HGcXHjMohqEDfkT8GFCXbW4Gbh52X2ZWTkW+DqFOPJeOVYoHBBvco7YkDvhWKVWemKwfo+pRZ3FA9ZhBfjyXjpn1LIsKGx+Q8+MevpWe0xeduUdt03wDFKsUDwhmr6g3NbFkvgGKWZ88+PsaV9hUh3P4VilZpS98NahVkQO+VYp75aPl8YBycw7frMm5aquCTjl8B3yzBEXKVXs8wfrRKeA7pWNWcB5PsKw44JslcK7aqsgpHbMC8niCDco5fLMSK9J4Qi885pAv5/Ct1LrdwKMsqvI+uvGYQ3E54FvhVSWADPo+PJ5gWfHUCmYFV4b0iCewKwfn8G2WIuRgqzJoWZX30Y+yjTlUjSdPs74U4SYiVZmwa5j3UYQDr1VLJjl8SRsl/ULSk5KuTVgvSV9prt8h6e1Z7NfKpS6Dllkp69iFxxyKa+iAL2kO8FXgYuBs4CpJZ7dtdjGwrvm1CfjasPu1bG3e9jhrr737WO51+nGWQbrug5ZVeR/d+KykuLJI6VwAPBkRTwFI+g5wOfDzlm0uB74ZjQGD+ySdKGllRDyXwf4tA0VOoVQlgPTyPjz4aaOURcA/HXi25fku4J09bHM6MCvgS9pE4yyANWvWZNA8y5MDWH+KfOC18ssi4CthWXvpTy/bNBZGbAG2QKNKZ7im2SCyTD04gJkVRxaDtruA1S3PVwG7B9jGCsI972KoS87fxieLgH8/sE7SGyTNB64Evt+2zfeBP2hW67wLeNn5+3rZvO1xB7A++cBrWRs6pRMRU5KuBn4EzAFujYhHJH2yuf4WYCtwCfAksA/42LD7tXK5afsTTueY5SyTC68iYiuNoN667JaWxwF8Kot9mdWFL7yyrPlKWxsZV+gMpwhXPFu1OODbyLhCx6xYHPDNCsRnRTZKDvg2Fq7Q6Y3PimyUfAMUGwv3Ts3y54BvVlA+K7KsOeCbFZTPiixrDvhWa56j3+rEAd9qraw3GTEbhAN+Avf6bFD+27Eic8BP4F5ftY3y7l7+27Eicx2+1c50rft0wHetu9WFA36Tr3C0Qflvx8pCjYksi2n9+vUxMTEx9v36Csfqag/O07IKzv7bsbxJeiAi1ietcw/fasVTF1idedA2ga9wtEEHcP23Y0XmgJ/Aedd66BScB6228d+OFZkDvhXeqGrbHZytbobK4Uv6EvBB4BDwD8DHIuKlhO2eBvYCR4CptAEFsyTjuvOTq22s6oYdtN0GfKF5I/P/CnwBuCZl2/dFxK+H3J/ZyHhA16puqIAfEfe0PL0P+MhwzTFrcG/bLHtZlmV+HPhuyroA7pEUwNcjYkvai0jaBGwCWLNmTYbNszLJu7ftahuroq4BX9JfA69PWHVdRHyvuc11wBTwrZSXeU9E7JZ0CrBN0mMRcW/Shs2DwRZoXHjVw3swy5zPIqyKugb8iHh/p/WSPgpcCmyIlMt2I2J38/vzku4ELgASA75ZO/e2zbIxVFmmpI00Bmkvi4h9KdsslrR0+jHwAWDnMPu1enFv2ywbw9bh3wwspZGmeUjSLQCSTpO0tbnNqcCPJT0M/BS4OyJ+OOR+zcysT8NW6bwpZflu4JLm46eAtw6zH7N+bN72uM8KzBL4SlurHN+ExCyZA76ZWU14emSrBF+oZdadb4BileNpEazOOt0AxSkdM7OacMC3yvGFWmbJHPCtcpyzN0vmgG9mVhMO+GZmNeGAb2ZWEw74ZmY14YBvZlYThb7wStILwDMD/vgKoCz30C1TW6Fc7XVbR6dM7S1TW2G49p4REScnrSh0wB+GpIm0q82KpkxthXK1120dnTK1t0xthdG11ykdM7OacMA3M6uJKgf8LXk3oA9laiuUq71u6+iUqb1laiuMqL2VzeGbmdlMVe7hm5lZCwd8M7OaqEXAl/R5SSFpRd5tSSPpP0raIekhSfdIOi3vNqWR9CVJjzXbe6ekE/NuUyeS/pmkRyQdlVTI0jxJGyX9QtKTkq7Nuz2dSLpV0vOSdubdlm4krZb0N5Iebf4NfCbvNqWRtFDSTyU93Gzrf8h6H5UP+JJWAxcB/zfvtnTxpYg4LyLOB+4Crs+7QR1sA86NiPOAx4Ev5NyebnYCHwbuzbshSSTNAb4KXAycDVwl6ex8W9XRbcDGvBvRoyngDyPiLcC7gE8V+LM9CPxORLwVOB/YKOldWe6g8gEf2Az8e6DQo9MR8UrL08UUuL0RcU9ETDWf3gesyrM93UTEoxHxi7zb0cEFwJMR8VREHAK+A1yec5tSRcS9wJ6829GLiHguIh5sPt4LPAqcnm+rkkXDq82n85pfmcaBSgd8SZcBv4yIh/NuSy8k/SdJzwL/gmL38Ft9HPhB3o0oudOBZ1ue76KgQanMJK0F3gb8fb4tSSdpjqSHgOeBbRGRaVvnZvlieZD018DrE1ZdB/wR8IHxtihdp7ZGxPci4jrgOklfAK4G/nisDWzRra3Nba6jccr8rXG2LUkv7S0wJSwr7BleGUlaAvwF8Nm2s+lCiYgjwPnNcbE7JZ0bEZmNlZQ+4EfE+5OWS/onwBuAhyVBI+3woKQLIuL/jbGJx6S1NcGfA3eTY8Dv1lZJHwUuBTZEAS7m6OOzLaJdwOqW56uA3Tm1pXIkzaMR7L8VEXfk3Z5eRMRLkv6WxlhJZgG/simdiPhZRJwSEWsjYi2Nf6q35xXsu5HUeufty4DH8mpLN5I2AtcAl0XEvrzbUwH3A+skvUHSfOBK4Ps5t6kS1OjtfQN4NCK+nHd7OpF08nTFm6RFwPvJOA5UNuCX0Bcl7ZS0g0YaqrDlY8DNwFJgW7OM9Ja8G9SJpA9J2gW8G7hb0o/yblOr5gD41cCPaAwq3h4Rj+TbqnSSvg38BHizpF2SPpF3mzp4D/D7wO80/1YfknRJ3o1KsRL4m2YMuJ9GDv+uLHfgqRXMzGrCPXwzs5pwwDczqwkHfDOzmnDANzOrCQd8M7OacMA3M6sJB3wzs5r4/4n0Ht/x80AEAAAAAElFTkSuQmCC) ### 3. 多层神经网络 simple neural nerwork #### 神经网络 人工神经网络（Artificial Neural Network，ANN）是指一系列受生物学和神经科学启发的数学模型．这些模型主要是通过对人脑的神经元网络进行抽象，构建人工神经元，并按照一定拓扑结构来建立人工神经元之间的连接，来模拟生物神经网络。 神经网络最早是作为一种主要的连接主义模型，神经元是构成神经网络的基本单元，主要是模拟生物神经元的结构和特性，接收一组输入信号并产生输出。神经元接受n个输入，与权重做计算产生一个输出。 $$ \begin{aligned} z &=\sum_{d=1}^{D} w_{d} x_{d}+b \\ &=\boldsymbol{w}^{\top} \boldsymbol{x}+b \end{aligned} $$ 为了增加非线性特性，我们还需要为净输出做一个非线性运算，称为激活，得到神经元的Activation。 ![image-20200703110113654](img/image-20200703110113654.png) #### 激活函数 ##### Sigmoid型激活函数 Sigmoid 型函数是指一类S 型曲线函数，为两端饱和函数．常用的Sigmoid型函数有**Logistic** 函数和**Tanh** 函数。 Logistic函数定义为： $$ \sigma(x)=\frac{1}{1+\exp (-x)} $$ Tanh也是S型函数，可以看作是放大并平移的Logistic函数 $$ \tanh (x)=2 \sigma(2 x)-1 $$ ![image-20200703110409171](img/image-20200703110409171.png) ##### ReLU 修正线性单元是目前常用的激活函数，最大的特点是导数容易计算 $$ \begin{aligned} \operatorname{ReLU}(x) &=\left\{\begin{array}{ll} x & x \geq 0 \\ 0 & x<0 \end{array}\right.\\ &=\max (0, x) \end{aligned} $$ #### 网络结构 本次实验只会用到全连接神经网络，但除此之外，还有记忆网络，图神经网络，循环神经网络和卷积网络等网络结构。 ![image-20200703110901439](img/image-20200703110901439.png) #### 反向传播 实验中采用随机梯度下降进行神经网络参数学习，，给定一个样本(𝒙, 𝒚)，将其输入到神经网络模型中，得到网络输出为$\hat{y}$．假设损失函数为$\mathcal{L}(\boldsymbol{y}, \hat{\boldsymbol{y}})$，要进行参数学习就需要计算损失函数关于每个参数的导数． 依据链式法则对每一层参数计算偏导数 $$ \begin{array}{c} \frac{\partial \mathcal{L}(\boldsymbol{y}, \hat{\boldsymbol{y}})}{\partial w_{i j}^{(l)}}=\frac{\partial \boldsymbol{z}^{(l)}}{\partial w_{i j}^{(l)}} \frac{\partial \mathcal{L}(\boldsymbol{y}, \hat{\boldsymbol{y}})}{\partial \boldsymbol{z}^{(l)}} \\ \frac{\partial \mathcal{L}(\boldsymbol{y}, \hat{\boldsymbol{y}})}{\partial \boldsymbol{b}^{(l)}}=\frac{\partial \boldsymbol{z}^{(l)}}{\partial \boldsymbol{b}^{(l)}} \frac{\partial \mathcal{L}(\boldsymbol{y}, \hat{\boldsymbol{y}})}{\partial \boldsymbol{z}^{(l)}} \end{array} $$ #### 实验结果 本次实验使用MNIST数据集的6万的训练数据只训练一次，结果如下 ![image-20200703111453231](img/image-20200703111453231.png) ### 4. 卷积神经网络 CNN 卷积神经网络（Convolutional Neural Network，CNN 或ConvNet）是一种具有局部连接、权重共享等特性的深层前馈神经网络，主要用来处理图像信息。由于传统的全连接网络存在参数太多，而且无法处理图像的局部不变性特征，所以在感受野的启发下人们发明了CNN。 卷积神经网络一般是由卷积层、汇聚层和全连接层交叉堆叠而成。 #### 卷积 一维卷积通常用在信号处理中，本次实验中用到的是二维卷积，通常用在图像处理中。 $$ y_{i j}=\sum_{u=1}^{U} \sum_{v=1}^{V} w_{u v} x_{i-u+1, j-v+1} $$ ![image-20200703112022163](img/image-20200703112022163.png) 在图像处理中，卷积经常作为特征提取的有效方法．一幅图像在经过卷积操作后得到结果称为特征映射。 ![image-20200703112119417](img/image-20200703112119417.png) #### 池化 池化层又叫汇聚层，卷积层虽然可以显著减少网络中连接的数量，但特征映射组中的神经元个数并没有显著减少．如果后面接一个分类器，分类器的输入维数依然很高，很容易出现过拟合．为了解决这个问题，可以在卷积层之后加上一个池化层，从而降低特征维数，避免过拟合． 常见的池化有两种 ##### 最大池化 选择这个区域内所有神经元的最大活性值作为这个区域的表示。 $$ y_{m, n}^{d}=\max _{i \in R_{m, n}^{d}} x_{i} $$ ![image-20200703112416562](img/image-20200703112416562.png) ##### 平均池化 取区域内所有神经元活性值的平均值。 $$ y_{m, n}^{d}=\frac{1}{\left|R_{m, n}^{d}\right|} \sum_{i \in R_{m, n}^{d}} x_{i} $$ #### 实验结果 ![image-20200703115441833](img/image-20200703115441833.png) ### 5. 循环神经网络 RNN 在前馈神经网络中，信息的传递是单向的，这种限制虽然使得网络变得更容易学习，但在一定程度上也减弱了神经网络模型的能力．在生物神经网络中，神经元之间的连接关系要复杂得多。 RNN是一类具有短期记忆功能的神经网络，在循环神经网络中，神经元不但可以接受其他神经元的信息，也可以接受自身的信息，形成具有环路的网络结构．和前馈神经网络相比，循环神经网络更加符合生物神经网络的结构．循环神经网络已经被广泛应用在语音识别、语言模型以及自然语言生成等任务上。 #### 延时单元 为了处理这些时序数据并利用其历史信息， 我们需要让网络具有短期记忆能力。一种简单的利用历史信息的方法是建立一个额外的延时单元，用来存储网络的历史信息。 延时神经网络是在前馈网络中的非输出层都添加一个延时器，记录神经元的最近几次活性值．在第t 个时刻，第𝑙 层神经元的活性值依赖于第𝑙 − 1 层神经元的最近𝐾 个时刻的活性值，即 $$ \boldsymbol{h}_{t}^{(l)}=f\left(\boldsymbol{h}_{t}^{(l-1)}, \boldsymbol{h}_{t-1}^{(l-1)}, \cdots, \boldsymbol{h}_{t-K}^{(l-1)}\right) $$ #### 循环神经网络 通过使用带自反馈的神经元，能够处理任意长度的时序数据。对于一个输入序列$\boldsymbol{x}_{1: T}=\left(\boldsymbol{x}_{1}, \boldsymbol{x}_{2}, \ldots, \boldsymbol{x}_{t}, \ldots, \boldsymbol{x}_{T}\right)$，循环神经网络可以通过下面的公式更新隐藏层的Activation。 $$ \boldsymbol{h}_{t}=f\left(\boldsymbol{h}_{t-1}, \boldsymbol{x}_{t}\right) $$ 下图是循环神经网络的一个示例图，延时器是一个虚拟单元，记录了最近一次或几次的活性值。 ![image-20200703154200326](img/image-20200703154200326.png) 对于一个两层的前馈神经网络，只有一个隐藏层，隐层之间的节点是没有连接的，我们可以增加隐层到隐层的反馈连接，这样就构成了一个**简单循环网络**。 简单循环网络在时刻t的输出不仅和当前输入有关，还和上一时刻的隐层状态有关，在时刻t的更新公式 $$ z_{t}=U h_{t-1}+W x_{t}+b $$ #### 实验结果 ![image-20200703160124021](img/image-20200703160124021.png) ### 6. 深度信念网络 DBN > DBN既可以用于非监督学习，类似于一个自编码机；也可以用于监督学习，作为分类器来使用。 > DBN由若干层神经元构成，组成元件是**受限玻尔兹曼机**（RBM）。所以在学习DBN之前，我们必须先了解RBM。 #### 受限玻尔兹曼机 Restricted Boltzmann Machines RBM是一个二分图结构的无向图模型。 - 在RBM中，变量分为输入层和隐藏层 - 变量的取值只能是0或1 - 网络结构和两层全连接神经网络相同 ![image-20200702112007311](img/image-20200702112007311.png) 所谓的受限就是将完全图变成了二分图，更方便与神经网络结合。 在受限玻尔兹曼机中，每个可观测变量和隐变量的条件概率为： $$ p\left(v_{i}=1 \mid \mathbf{h}\right)=\sigma\left(a_{i}+\sum_j w_{i j} h_{j}\right) $$ $$ p\left(h_{j}=1 \mid \mathbf{v}\right)=\sigma\left(b_{j}+\sum_{i} w_{i j} v_{i}\right) $$ 受限玻尔兹曼机的能量函数定义为： $$ \begin{aligned} E(\boldsymbol{v}, \boldsymbol{h}) &=-\sum_{i} a_{i} v_{i}-\sum_{j} b_{j} h_{j}-\sum_{i} \sum_{j} v_{i} w_{i j} h_{j} \\ &=-\boldsymbol{a}^{\top} \boldsymbol{v}-\boldsymbol{b}^{\top} \boldsymbol{h}-\boldsymbol{v}^{\top} \boldsymbol{W} \boldsymbol{h} \end{aligned} $$ ##### 生成模型 受限玻尔兹曼机的联合概率分布$p(\boldsymbol{v},\boldsymbol{h})$定义为 $$ \begin{aligned} p(\boldsymbol{v}, \boldsymbol{h}) &=\frac{1}{Z} \exp (-E(\boldsymbol{v}, \boldsymbol{h})) \\ &=\frac{1}{Z} \exp \left(\boldsymbol{a}^{\top} \boldsymbol{v}\right) \exp \left(\boldsymbol{b}^{\top} \boldsymbol{h}\right) \exp \left(\boldsymbol{v}^{\top} \boldsymbol{W} \boldsymbol{h}\right) \end{aligned} $$ 其中$Z=\sum_{v, h} \exp (-E(\boldsymbol{v}, \boldsymbol{h}))$是分配函数。 在给定受限玻尔兹曼机的联合概率分布后可以通过**Gibbs采样**生成一组服从条件分布的样本。由于RBM同层之间没有连接，可以有如下条件概率。 $$ \begin{array}{l} p\left(v_{i} \mid \boldsymbol{v}_{\backslash i}, \boldsymbol{h}\right)=p\left(v_{i} \mid \boldsymbol{h}\right) \\ p\left(h_{j} \mid \boldsymbol{v}, \boldsymbol{h}_{\backslash j}\right)=p\left(h_{j} \mid \boldsymbol{v}\right) \end{array} $$ $$ \begin{array}{l} p\left(v_{i}=1 \mid \boldsymbol{h}\right)=\sigma\left(a_{i}+\sum_{j} w_{i j} h_{j}\right) \\ p\left(h_{j}=1 \mid \boldsymbol{v}\right)=\sigma\left(b_{j}+\sum_{i} w_{i j} v_{i}\right) \end{array} $$ ##### 参数学习 由于受限玻尔兹曼机的特殊结构，因此可以使用一种比吉布斯采样更有效的学习算法，即对比散度（Contrastive Divergence）[Hinton, 2002]．对比散度算法仅需𝑘 步吉布斯采样。 为了提高效率，对比散度算法用一个训练样本作为可观测向量的初始值．然后，交替对可观测向量和隐向量进行吉布斯采样，不需要等到收敛，只需要𝑘 步就足够了．这就是CD-𝑘 算法．通常，𝑘 = 1 就可以学得很好 ![image-20200702163757117](img/image-20200702163757117.png) #### 深度信念网络 深度信念网络（Deep Belief Network，DBN）是一种深层的概率有向图模型，其图结构由多层的节点构成． 和全连接的前馈神经网络结构相同．每层节点的内部没有连接，相邻两层的节点之间为全连接．网络的最底层为可观测变量，其他层节点都为隐变量．最顶部的两层间的连接是无向的，其他层之间的连接是有向的。 ![image-20200702164318444](img/image-20200702164318444.png) ##### 生成模型 深度信念网络是一个生成模型，可以用来生成符合特定分布的样本。在生成样本时，首先在最顶两层进行足够多次的吉布斯采样，生成𝒉(𝐿−1)，然后依次计算下一层隐变量的分布．因为在给定上一层变量取值时，下一层的变 量是条件独立的，所以可以独立采样．这样，我们可以从第𝐿 − 1 层开始，自顶向下进行逐层采样，最终得到可观测层的样本。 ##### 参数学习 深度信念网络一般采用逐层训练的方式，我们将每一层的Sigmoid 信念网络转换为受限玻尔兹曼机．这样做的好处是隐变量的后验概率是互相独立的，从而可以很容易地进行采样．这样，深度信念网络可以看作由多个受限玻尔兹曼机从下到上进行堆叠，第𝑙 层受限玻尔兹曼机的隐层作为第𝑙 + 1 层受限玻尔兹曼机的可观测层．进一步地，深度信念网络可以采用逐层训练的方式来快速训练，即从最底层开始，每次只训练一层，直到最后一层。 深度信念网络的训练过程可以分为**逐层预训练**和**精调**两个阶段．先通过逐层预训练将模型的参数初始化为较优的值，再通过传统学习方法对参数进行精调。 ##### 逐层预训练 在逐层预训练阶段，将深度信念网络的训练简化为对多个受限玻尔兹曼机的训练。 ![image-20200702174111017](img/image-20200702174111017.png) ![image-20200702174126343](img/image-20200702174126343.png) ##### 精调 经过预训练之后，再结合具体的任务（监督学习或无监督学习），通过传统的全局学习算法对网络进行精调（Fine-Tuning），使模型收敛到更好的局部最优点． #### 实验结果 原图像 ![image-20200703160923147](img/image-20200703160923147.png) 生成的图像 ![image-20200703160930245](img/image-20200703160930245.png)