to load 'ernie-1.0-base-zh'.\u001b[0m\n",
+ "\u001b[32m[2022-06-22 21:31:15,580] [ INFO]\u001b[0m - Already cached /remote-home/shxing/.paddlenlp/models/ernie-1.0-base-zh/ernie_v1_chn_base.pdparams\u001b[0m\n"
+ ]
+ }
+ ],
+ "source": [
+ "import paddle.nn as nn\n",
+ "\n",
+ "class SeqClsModel(nn.Layer):\n",
+ " def __init__(self, model_checkpoint, num_labels):\n",
+ " super(SeqClsModel, self).__init__()\n",
+ " self.model = AutoModelForSequenceClassification.from_pretrained(\n",
+ " model_checkpoint,\n",
+ " num_classes=num_labels,\n",
+ " )\n",
+ "\n",
+ " def forward(self, input_ids, attention_mask, token_type_ids):\n",
+ " logits = self.model(input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids)\n",
+ " return logits\n",
+ "\n",
+ " def train_step(self, input_ids, attention_mask, token_type_ids, label):\n",
+ " logits = self(input_ids, attention_mask, token_type_ids)\n",
+ " loss = nn.CrossEntropyLoss()(logits, label)\n",
+ " return {\"loss\": loss}\n",
+ "\n",
+ " def evaluate_step(self, input_ids, attention_mask, token_type_ids, label):\n",
+ " logits = self(input_ids, attention_mask, token_type_ids)\n",
+ " return {'pred': logits, 'target': label}\n",
+ "\n",
+ "model = SeqClsModel(model_checkpoint, num_labels=2)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### 3.2 设置参数并使用 Trainer 开始训练\n",
+ "\n",
+ "现在我们可以着手使用 ``FastNLP.Trainer`` 进行训练了。\n",
+ "\n",
+ "首先，为了高效地训练 ``ERNIE`` 模型，我们最好为学习率指定一定的策略。``paddlenlp`` 提供的 ``LinearDecayWithWarmup`` 可以令学习率在一段时间内从 0 开始线性地增长（预热），然后再线性地衰减至 0 。在本篇教程中，我们将学习率设置为 ``5e-5``，预热时间为 ``0.1``，然后将得到的的 ``lr_scheduler`` 赋值给 ``AdamW`` 优化器。\n",
+ "\n",
+ "其次，我们还可以为 ``Trainer`` 指定多个 ``Callback`` 来在基础的训练过程之外进行额外的定制操作。在本篇教程中，我们使用的 ``Callback`` 有以下三种：\n",
+ "\n",
+ "- ``RichCallback`` - 在训练和验证时显示进度条，以便观察训练的过程\n",
+ "- ``LRSchedCallback`` - 由于我们使用了 ``Scheduler``，因此需要将 ``lr_scheduler`` 传给该 ``Callback`` 以在训练中进行更新\n",
+ "- ``LoadBestModelCallback`` - 该 ``Callback`` 会评估结果中的 ``'acc#accuracy'`` 值，保存训练中出现的正确率最高的模型，并在训练结束时加载到模型上，方便对模型进行测试和评估。\n",
+ "\n",
+ "在 ``Trainer`` 中，我们还可以设置 ``metrics`` 来衡量模型的表现。``Accuracy`` 能够根据传入的预测值和真实值计算出模型预测的正确率。还记得模型中 ``evaluate_step`` 函数的返回值吗？键 ``pred`` 和 ``target`` 分别为 ``Accuracy.update`` 的参数名，在验证过程中 ``FastNLP`` 会自动将键和参数名匹配从而计算出正确率，这也是我们规定模型需要返回字典类型数据的原因。\n",
+ "\n",
+ "``Accuracy`` 的返回值包含三个部分：``acc``、``total`` 和 ``correct``，分别代表 ``正确率``、 ``数据总数`` 和 ``预测正确的数目``，这让您能够直观地知晓训练中模型的变化，``LoadBestModelCallback`` 的参数 ``'acc#accuracy'`` 也正是代表了 ``accuracy`` 指标的 ``acc`` 结果。\n",
+ "\n",
+ "在设定好参数之后，调用 ``run`` 函数便可以进行训练和验证了。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "[21:31:16] INFO Running evaluator sanity check for 2 batches. trainer.py:631\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "\u001b[2;36m[21:31:16]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m Running evaluator sanity check for \u001b[1;36m2\u001b[0m batches. \u001b]8;id=4641;file://../fastNLP/core/controllers/trainer.py\u001b\\\u001b[2mtrainer.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=822054;file://../fastNLP/core/controllers/trainer.py#631\u001b\\\u001b[2m631\u001b[0m\u001b]8;;\u001b\\\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n"
+ ],
+ "text/plain": []
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "---------------------------- Eval. results on Epoch:0, Batch:60 -----------------------------\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "---------------------------- Eval. results on Epoch:\u001b[1;36m0\u001b[0m, Batch:\u001b[1;36m60\u001b[0m -----------------------------\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "{\n",
+ " \"acc#accuracy\": 0.895833,\n",
+ " \"total#accuracy\": 1200.0,\n",
+ " \"correct#accuracy\": 1075.0\n",
+ "}\n",
+ "\n"
+ ],
+ "text/plain": [
+ "\u001b[1m{\u001b[0m\n",
+ " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.895833\u001b[0m,\n",
+ " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n",
+ " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1075.0\u001b[0m\n",
+ "\u001b[1m}\u001b[0m\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "---------------------------- Eval. results on Epoch:0, Batch:120 ----------------------------\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "---------------------------- Eval. results on Epoch:\u001b[1;36m0\u001b[0m, Batch:\u001b[1;36m120\u001b[0m ----------------------------\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "{\n",
+ " \"acc#accuracy\": 0.8975,\n",
+ " \"total#accuracy\": 1200.0,\n",
+ " \"correct#accuracy\": 1077.0\n",
+ "}\n",
+ "\n"
+ ],
+ "text/plain": [
+ "\u001b[1m{\u001b[0m\n",
+ " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.8975\u001b[0m,\n",
+ " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n",
+ " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1077.0\u001b[0m\n",
+ "\u001b[1m}\u001b[0m\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "---------------------------- Eval. results on Epoch:0, Batch:180 ----------------------------\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "---------------------------- Eval. results on Epoch:\u001b[1;36m0\u001b[0m, Batch:\u001b[1;36m180\u001b[0m ----------------------------\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "{\n",
+ " \"acc#accuracy\": 0.911667,\n",
+ " \"total#accuracy\": 1200.0,\n",
+ " \"correct#accuracy\": 1094.0\n",
+ "}\n",
+ "\n"
+ ],
+ "text/plain": [
+ "\u001b[1m{\u001b[0m\n",
+ " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.911667\u001b[0m,\n",
+ " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n",
+ " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1094.0\u001b[0m\n",
+ "\u001b[1m}\u001b[0m\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "---------------------------- Eval. results on Epoch:0, Batch:240 ----------------------------\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "---------------------------- Eval. results on Epoch:\u001b[1;36m0\u001b[0m, Batch:\u001b[1;36m240\u001b[0m ----------------------------\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "{\n",
+ " \"acc#accuracy\": 0.9225,\n",
+ " \"total#accuracy\": 1200.0,\n",
+ " \"correct#accuracy\": 1107.0\n",
+ "}\n",
+ "\n"
+ ],
+ "text/plain": [
+ "\u001b[1m{\u001b[0m\n",
+ " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.9225\u001b[0m,\n",
+ " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n",
+ " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1107.0\u001b[0m\n",
+ "\u001b[1m}\u001b[0m\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "---------------------------- Eval. results on Epoch:0, Batch:300 ----------------------------\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "---------------------------- Eval. results on Epoch:\u001b[1;36m0\u001b[0m, Batch:\u001b[1;36m300\u001b[0m ----------------------------\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "{\n",
+ " \"acc#accuracy\": 0.9275,\n",
+ " \"total#accuracy\": 1200.0,\n",
+ " \"correct#accuracy\": 1113.0\n",
+ "}\n",
+ "\n"
+ ],
+ "text/plain": [
+ "\u001b[1m{\u001b[0m\n",
+ " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.9275\u001b[0m,\n",
+ " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n",
+ " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1113.0\u001b[0m\n",
+ "\u001b[1m}\u001b[0m\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "---------------------------- Eval. results on Epoch:1, Batch:60 -----------------------------\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "---------------------------- Eval. results on Epoch:\u001b[1;36m1\u001b[0m, Batch:\u001b[1;36m60\u001b[0m -----------------------------\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "{\n",
+ " \"acc#accuracy\": 0.930833,\n",
+ " \"total#accuracy\": 1200.0,\n",
+ " \"correct#accuracy\": 1117.0\n",
+ "}\n",
+ "\n"
+ ],
+ "text/plain": [
+ "\u001b[1m{\u001b[0m\n",
+ " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.930833\u001b[0m,\n",
+ " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n",
+ " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1117.0\u001b[0m\n",
+ "\u001b[1m}\u001b[0m\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "---------------------------- Eval. results on Epoch:1, Batch:120 ----------------------------\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "---------------------------- Eval. results on Epoch:\u001b[1;36m1\u001b[0m, Batch:\u001b[1;36m120\u001b[0m ----------------------------\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "{\n",
+ " \"acc#accuracy\": 0.935833,\n",
+ " \"total#accuracy\": 1200.0,\n",
+ " \"correct#accuracy\": 1123.0\n",
+ "}\n",
+ "\n"
+ ],
+ "text/plain": [
+ "\u001b[1m{\u001b[0m\n",
+ " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.935833\u001b[0m,\n",
+ " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n",
+ " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1123.0\u001b[0m\n",
+ "\u001b[1m}\u001b[0m\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "---------------------------- Eval. results on Epoch:1, Batch:180 ----------------------------\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "---------------------------- Eval. results on Epoch:\u001b[1;36m1\u001b[0m, Batch:\u001b[1;36m180\u001b[0m ----------------------------\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "{\n",
+ " \"acc#accuracy\": 0.935833,\n",
+ " \"total#accuracy\": 1200.0,\n",
+ " \"correct#accuracy\": 1123.0\n",
+ "}\n",
+ "\n"
+ ],
+ "text/plain": [
+ "\u001b[1m{\u001b[0m\n",
+ " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.935833\u001b[0m,\n",
+ " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n",
+ " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1123.0\u001b[0m\n",
+ "\u001b[1m}\u001b[0m\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "---------------------------- Eval. results on Epoch:1, Batch:240 ----------------------------\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "---------------------------- Eval. results on Epoch:\u001b[1;36m1\u001b[0m, Batch:\u001b[1;36m240\u001b[0m ----------------------------\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "{\n",
+ " \"acc#accuracy\": 0.9375,\n",
+ " \"total#accuracy\": 1200.0,\n",
+ " \"correct#accuracy\": 1125.0\n",
+ "}\n",
+ "\n"
+ ],
+ "text/plain": [
+ "\u001b[1m{\u001b[0m\n",
+ " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.9375\u001b[0m,\n",
+ " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n",
+ " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1125.0\u001b[0m\n",
+ "\u001b[1m}\u001b[0m\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "---------------------------- Eval. results on Epoch:1, Batch:300 ----------------------------\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "---------------------------- Eval. results on Epoch:\u001b[1;36m1\u001b[0m, Batch:\u001b[1;36m300\u001b[0m ----------------------------\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "{\n",
+ " \"acc#accuracy\": 0.941667,\n",
+ " \"total#accuracy\": 1200.0,\n",
+ " \"correct#accuracy\": 1130.0\n",
+ "}\n",
+ "\n"
+ ],
+ "text/plain": [
+ "\u001b[1m{\u001b[0m\n",
+ " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.941667\u001b[0m,\n",
+ " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n",
+ " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1130.0\u001b[0m\n",
+ "\u001b[1m}\u001b[0m\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "[21:34:28] INFO Loading best model from fnlp-ernie/2022-0 load_best_model_callback.py:111\n",
+ " 6-22-21_29_12_898095/best_so_far with \n",
+ " acc#accuracy: 0.941667... \n",
+ "
\n"
+ ],
+ "text/plain": [
+ "\u001b[2;36m[21:34:28]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m Loading best model from fnlp-ernie/\u001b[1;36m2022\u001b[0m-\u001b[1;36m0\u001b[0m \u001b]8;id=340364;file://../fastNLP/core/callbacks/load_best_model_callback.py\u001b\\\u001b[2mload_best_model_callback.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=763898;file://../fastNLP/core/callbacks/load_best_model_callback.py#111\u001b\\\u001b[2m111\u001b[0m\u001b]8;;\u001b\\\n",
+ "\u001b[2;36m \u001b[0m \u001b[1;36m6\u001b[0m-\u001b[1;36m22\u001b[0m-21_29_12_898095/best_so_far with \u001b[2m \u001b[0m\n",
+ "\u001b[2;36m \u001b[0m acc#accuracy: \u001b[1;36m0.941667\u001b[0m\u001b[33m...\u001b[0m \u001b[2m \u001b[0m\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "[21:34:34] INFO Deleting fnlp-ernie/2022-06-22-21_29_12_8 load_best_model_callback.py:131\n",
+ " 98095/best_so_far... \n",
+ "
\n"
+ ],
+ "text/plain": [
+ "\u001b[2;36m[21:34:34]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m Deleting fnlp-ernie/\u001b[1;36m2022\u001b[0m-\u001b[1;36m06\u001b[0m-\u001b[1;36m22\u001b[0m-21_29_12_8 \u001b]8;id=430330;file://../fastNLP/core/callbacks/load_best_model_callback.py\u001b\\\u001b[2mload_best_model_callback.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=508566;file://../fastNLP/core/callbacks/load_best_model_callback.py#131\u001b\\\u001b[2m131\u001b[0m\u001b]8;;\u001b\\\n",
+ "\u001b[2;36m \u001b[0m 98095/best_so_far\u001b[33m...\u001b[0m \u001b[2m \u001b[0m\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n"
+ ],
+ "text/plain": []
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "from fastNLP import LRSchedCallback, RichCallback, LoadBestModelCallback\n",
+ "from fastNLP import Trainer, Accuracy\n",
+ "from paddlenlp.transformers import LinearDecayWithWarmup\n",
+ "\n",
+ "n_epochs = 2\n",
+ "num_training_steps = len(train_dataloader) * n_epochs\n",
+ "lr_scheduler = LinearDecayWithWarmup(5e-5, num_training_steps, 0.1)\n",
+ "optimizer = paddle.optimizer.AdamW(\n",
+ " learning_rate=lr_scheduler,\n",
+ " parameters=model.parameters(),\n",
+ ")\n",
+ "callbacks = [\n",
+ " LRSchedCallback(lr_scheduler, step_on=\"batch\"),\n",
+ " LoadBestModelCallback(\"acc#accuracy\", larger_better=True, save_folder=\"fnlp-ernie\"),\n",
+ " RichCallback()\n",
+ "]\n",
+ "trainer = Trainer(\n",
+ " model=model,\n",
+ " driver=\"paddle\",\n",
+ " optimizers=optimizer,\n",
+ " device=0,\n",
+ " n_epochs=n_epochs,\n",
+ " train_dataloader=train_dataloader,\n",
+ " evaluate_dataloaders=val_dataloader,\n",
+ " evaluate_every=60,\n",
+ " metrics={\"accuracy\": Accuracy()},\n",
+ " callbacks=callbacks,\n",
+ ")\n",
+ "trainer.run()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### 3.3 测试和评估\n",
+ "\n",
+ "现在我们已经得到了一个表现良好的 ``ERNIE`` 模型，接下来可以在测试集上测试模型的效果了。``FastNLP.Evaluator`` 提供了定制函数的功能。我们以 ``test_dataloader`` 初始化一个 ``Evaluator``，然后将写好的测试函数 ``test_batch_step_fn`` 传给参数 ``evaluate_batch_step_fn``，``Evaluate`` 在对每个 batch 进行评估时就会调用我们自定义的 ``test_batch_step_fn`` 函数而不是 ``evaluate_step`` 函数。在这里，我们仅测试 5 条数据并输出文本和对应的标签。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "text: ['这个宾馆比较陈旧了，特价的房间也很一般。总体来说一般']\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "text: ['这个宾馆比较陈旧了，特价的房间也很一般。总体来说一般']\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "labels: 0\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "labels: 0\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "text: ['怀着十分激动的心情放映，可是看着看着发现，在放映完毕后，出现一集米老鼠的动画片！开始\n",
+ "还怀疑是不是赠送的个别现象，可是后来发现每张DVD后面都有！真不知道生产商怎么想的，我想看的是猫\n",
+ "和老鼠，不是米老鼠！如果厂家是想赠送的话，那就全套米老鼠和唐老鸭都赠送，只在每张DVD后面添加一\n",
+ "集算什么？？简直是画蛇添足！！']\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "text: ['怀着十分激动的心情放映，可是看着看着发现，在放映完毕后，出现一集米老鼠的动画片！开始\n",
+ "还怀疑是不是赠送的个别现象，可是后来发现每张DVD后面都有！真不知道生产商怎么想的，我想看的是猫\n",
+ "和老鼠，不是米老鼠！如果厂家是想赠送的话，那就全套米老鼠和唐老鸭都赠送，只在每张DVD后面添加一\n",
+ "集算什么？？简直是画蛇添足！！']\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "labels: 0\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "labels: 0\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "text: ['还稍微重了点，可能是硬盘大的原故，还要再轻半斤就好了。其他要进一步验证。贴的几种膜气\n",
+ "泡较多，用不了多久就要更换了，屏幕膜稍好点，但比没有要强多了。建议配赠几张膜让用用户自己贴。'\n",
+ "]\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "text: ['还稍微重了点，可能是硬盘大的原故，还要再轻半斤就好了。其他要进一步验证。贴的几种膜气\n",
+ "泡较多，用不了多久就要更换了，屏幕膜稍好点，但比没有要强多了。建议配赠几张膜让用用户自己贴。'\n",
+ "]\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "labels: 0\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "labels: 0\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "text: ['交通方便；环境很好；服务态度很好 房间较小']\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "text: ['交通方便；环境很好；服务态度很好 房间较小']\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "labels: 1\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "labels: 1\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "text: ['不错，作者的观点很颠覆目前中国父母的教育方式，其实古人们对于教育已经有了很系统的体系\n",
+ "了，可是现在的父母以及祖父母们更多的娇惯纵容孩子，放眼看去自私的孩子是大多数，父母觉得自己的\n",
+ "孩子在外面只要不吃亏就是好事，完全把古人几千年总结的教育古训抛在的九霄云外。所以推荐准妈妈们\n",
+ "可以在等待宝宝降临的时候，好好学习一下，怎么把孩子教育成一个有爱心、有责任心、宽容、大度的人\n",
+ "。']\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "text: ['不错，作者的观点很颠覆目前中国父母的教育方式，其实古人们对于教育已经有了很系统的体系\n",
+ "了，可是现在的父母以及祖父母们更多的娇惯纵容孩子，放眼看去自私的孩子是大多数，父母觉得自己的\n",
+ "孩子在外面只要不吃亏就是好事，完全把古人几千年总结的教育古训抛在的九霄云外。所以推荐准妈妈们\n",
+ "可以在等待宝宝降临的时候，好好学习一下，怎么把孩子教育成一个有爱心、有责任心、宽容、大度的人\n",
+ "。']\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "labels: 1\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "labels: 1\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n"
+ ],
+ "text/plain": []
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "{}"
+ ]
+ },
+ "execution_count": 14,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from fastNLP import Evaluator\n",
+ "def test_batch_step_fn(evaluator, batch):\n",
+ " input_ids = batch[\"input_ids\"]\n",
+ " attention_mask = batch[\"attention_mask\"]\n",
+ " token_type_ids = batch[\"token_type_ids\"]\n",
+ " logits = model(input_ids, attention_mask, token_type_ids)\n",
+ " predict = logits.argmax().item()\n",
+ " print(\"text:\", batch['text'])\n",
+ " print(\"labels:\", predict)\n",
+ "\n",
+ "evaluator = Evaluator(\n",
+ " model=model,\n",
+ " dataloaders=test_dataloader,\n",
+ " driver=\"paddle\",\n",
+ " device=0,\n",
+ " evaluate_batch_step_fn=test_batch_step_fn,\n",
+ ")\n",
+ "evaluator.run(5) "
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3.7.13 ('fnlp-paddle')",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.7.13"
+ },
+ "orig_nbformat": 4,
+ "vscode": {
+ "interpreter": {
+ "hash": "31f2d9d3efc23c441973d7c4273acfea8b132b6a578f002629b6b44b8f65e720"
+ }
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/tutorials/figures/paddle-ernie-1.0-masking-levels.png b/tutorials/figures/paddle-ernie-1.0-masking-levels.png
new file mode 100644
index 0000000000000000000000000000000000000000..ff2519c4bc2c6b3ce4c63a7612fcc1e11f9cf4f8
GIT binary patch
literal 59022
zcmV*8=5ZN`P)$rG?_|Ufe0}Rv-o;N`(04zmwgN
zO|rY$B%5Tz|L!?^l6`OAd*94_a=)2*Ge*$_BuJ1TL4pJc5+q2FAVGqJ6hM~I0!WY`
zL4pJc5+q2FAVGqJ%nK>0lpsNZ1PKx(NRS{wf&>XEhLluFkRU;V1PKx(NRS{wf`k;q
zB%=k8AVGoz2@)hokRTx=;Tr(ld;~;9Wwa!^!^9YWxW0*2iBHA(;7hDoSJ|0
zm}RlPHn@GHpSlhNho*;Q
zy2Cyz(7cv;V3}m&5J0{BV<09bJyR!w|v#V%m|!St6&U+CV9gtTY;V{{s5xn=_{A65I$?wM4(L3xMQv#fHq?dBMzhnd+D=^
z=8OA2Vj*z(UdG9j6xK}vs+LK7Ty&ze7-@JNMDs@gO==`QE;v`y;rih%hKz5E1Dk)=
z{u{lFRC;&;{bm84zWTVE1PK{~y-XN4P6vwQO?*yF9}x~ZJOig;sKfsAz+W?fLJo@W
zAz=z4B6{r&87Hp;7j7#^y-4Z>&fEmXFVk;e7YXW8KBHEJ_{ZP92m1VMc%x9FGe8hF
zk?b$9A30ya}a%AecX|Q;h7Xb_6_so4FH8fvqQiM%96eCF73G{nPLgN`X7ifE}mE
z`C2DFC&=|L>kT)d6c9e^7y7L2bK;Ku^Ka%)CUP37#rd}=CuKt{uN@|FB@+N*=UVD&n
zvLqMk>Qszei|Kor9aj^ygQ4nAauC2I>+$RO#}bw^v*M~s`-mxY_g`e3EJ;C=EG&o5;7k5Gz9XeZ_`zbWI+VTSQxUL>hSbU-2Z~irv+R-Xn(GuLA0m3ZeyYP
z9#Upn
z^k0cNa{-omzNZ8U5+q2FAmJ;7F?mA{z{)721xX!@jAM$TYap
zu|Rb-%fU;K@c)b?35SFfAv4yiku~7x2sA9F*yjL#w3(07XYoYX0#+qupS$C0e&6M&oJPZAAsJpR~v2Dm;jT12R63W?2l^p
z1ujfy(x|14jHyraX8=vIq(c5QK(QvkpZ$RBnb3zA*#iCh1I{|^Etye}QMM6-fch$2
zJo~r6jKS1JEn5+#84}pQ?18{=>OevBq%Tc2m@kZ>UG!3}!fwiPA6^Ue%cuRav_K|w
zp9TQdT9?ki(^Yg_RF>&4fHsbqkC;hF862tuKX*t>Fs;h)cu(yCFFhfg>?k_41wZn`
zzcVOXs}h-1Pjw(15>kLnSg%GT4YOvGsI~z*R!QT{mIW~&QTcnYa=VwzQc0Nv=9Nga
z$7Hk=anL1ck)Q);H^nOe6|8}G9{_(EY1bjZ?=7|XnJD1k4q(Hrn15rE!-aEzW-287
z$pzrIgMgcXzvfY_5wL?vgoa&LG98R8f!cL|(gv2Ot3q^8)viI?RZjCWD*())&gFLp
zE_r2yK=EzhLn!@ATVT^@pl}*@EJVHmmTd+8$$$mt(lj1_wV*en9lHYgP>vH2hz+2_G
zc^)3G%(~qU{AU0`PdM2cRROQlzys<{t|Gv=snn4b!1S9*UoQgVoT5M}#t7CiRoYTQ>D+o3cG(nXb@
zd?N4Jn@kzRmR0Hfj=EhwprDQRSrp=&M6uEaM%0~gKLR<(iw7OaSbwW(1Ap=VeZzqU
zxq&~`EFh>aL{F#e8}I9
zn*g-Uqd0HQ^IbXtQ>p`(4l#h81j_XU`WIq5llb3YcIqLP6X)NDK8Q&OJC6c}@%&9D
zdM952j{N;RCuNTxlfZh7fnWLqk@tXwTY=ZbfCX)o=Xw42yuLq^%{|+Ii6k^9{eV%F
zv5~UZsTi+cPMQ3ej2^ts`;-AzGH@1@v?KF_&f^Z9%Q_~&bAF><+yxpp((O`pfl-%W
znI=#Yj*IWp`@vs1GGV=1i=OB}2J3Xlf<+I#75{qsrQq5G-!sg1k_&?uFzpbNw9rS!
zteVgGSEMkIMYZc)-LYgMEvwhIB)P$C;KDlK&@`ZR0mTD5kOOS}KIV19FG92%NwS+>
zGZgr|JlO$HSHPb>$Cz@hUIg@Q2y9saZ0m>^K?Y6*X4M38GDiJFj^aRpr>>%Z`ib_#
zpOkF}92l&eMr%P{)t9{H51wDm^7V{Ay*5WBmTw6;)n*dx$v_R;WXf#LI_kswE(Xq%
zH|?9m*qKXls$oCik5+MYCuRlBo5Ol$ve0=v&#_z<6q^ai|5vo
zciAZwB4;_q=o!Enp1ZINSk|4orm`Pv2|n)t6QvG(_G!v9v`8|`^Zx?cX18IiXLsPI
zErvBX5+26%f$~JE3_56Rcj`q;<`^40#Et$r^4y^@Kz^mp`%w&r!?y*AEcOg3ZpQNG56`2IC2I1d=n(T{8bY?wtyrQB=o4wFldA;4fq
zVDk0!sn4$wmUQ|f87ylS0>m``r__nrTNS!o(8}g2HtV=fEURb5vBgPeq@`A&4#I~H7LWN`arR))D!CHDeCP?ws$Dc
zb!I>xQ4KG3qOA
zKz5D}U&CIYS|JDrbw2+;z;x=a73*{)gR>Ls$MiEv17%rwCk6LS1xh;sLr8u~DxB-)^MOh@Zytvu$dVcnz(`x?*5@eGgdtH}F`@su00t0{zt^l4rJMQ~q`^^{UQ<
zxOx@HU4gnqvQvcwD!`4TioRu+a_`fUEc2M=EIS!q5Orz6Tn0jY?n|dlFL}Bi{n|YG
zGz*qx*9HY)6grp|Fe52FIGjPBH4GF9h$$7_bfqW#XzHjKiDaYxLaW=il
z7x=-6^~kc^ItomnA8s~Qz3ph!Awih=oE}wxT#VALj1hxrS2IZpTuCUGJXKbgJ)cR!
z{B$XOxJwEP8mkxd`yW}ZwoFW9OrkgOIcuLN;+P;lqiDa?^HbkMrzd6neK+uiKB?UZ
zpuH#SXD4Hz@_s$(KdMsqMxW$&E&d@PQl(#lgfVV)pezITlDI)Z!lshw5B;5fH(Ei$
zR+H=xT~7TUpTHKT^A>GL@c8?lEX%JXS&o)G)}OwQ@-O^_K5SZoVQ3B>Gf{LSG7z
z>*TUkkl~Uofbpe(iR3Q#*x2RsDcU@?E}D^3O=X+DZQang(O^`ovvPzOgadH$XE`<}i+xNkGclb3DgMD96yHhqsz
z>e}XOMddE^AB=rtSdT6SF#X1yI$fIfZ$V$(hCWpOFZEeD9oMv|_I7{hsu+MONMG?E
zm8~9QQg$%|jj?f$Km84TSs4Az$Ao@_F(U`{&r+$NHvD~E^2X&Wfrg9|ZK!`gv7S!N
z1l)_!KMkaR+{Ijj9KfD7JDqx&FNOR6N=aoZ@RrWthxbe@>G&qnQ_iO2uEv1;4I6NC
zOO+oo$mXIVe8bv)&ad9Dd98}FfAm*OsZ@vZOaSbtkjs9Ke>^LlT3}%+1rt#@ufMA@
z(jX@x@V8Q|epZC?JinPZfPYpV5Yurh(&=U=3FxINvJIa>hiZCVNfac+gN%-%9m#^q
z{>d;a9KYAshCwrk9CDS8ht23+4giKB<}0h(D_n_3X)j;Jb(^?fqwzXH5W0}bRMC5R;)F{G8UR(=KaUq=08-4;#n-bl7XKGwsT
zZNM+AzdR%?%lnet&&?Hll)nrU%3s))zq9==ur0fhRL&(4>at0Zl!@R{y&>Co
z2Yr_Z@IG3lUI|nBnnk?k)mD-ylAof@*xn>lZ%N8-GvLjnlXyTvKAt+{K;IFq=b)XG
zUS46K?Mrh0CrNyt1tbk=IIk)EKG2zjb>$4;BK`074?wUMq8Ve;4EnX1kn|j)AdRbO
zFCFPaH>)F>T6E}Lv&4isg8tw-W$0Ur_kF5p&wL~tj+AGWTk6WHjdZvb(N27>KjUwi
z2v%aka5F;Dk<_v{aO5w>BKn6x+WdnGj0>hxzoL^z!qnM9F=f&M#?Y}$7{@Y^%yo!~
z5q)1Mnf-A!$F$kZ%`okHvC$)S?Uu0oCrJ#CDU;bPELT(NPyPJVVaA;69oY`#6IYq+
z7plRe>(ht{O5QxQU+Sz?
z@uaVdq^zNliZTR$jz9HM&h}n=im{#Y`Cg_^rHvJ#T-8}O1?kU6#ZC>T%nJ_!2O0aT
zvmMVXH=K{AAN+F<@F)}Q2b!*+&D)b3o3Vam1~TKKDE;axer%|#kB$PpsZ-v{crE;o
zwsyZnTYb}p|FcZ*ve17~w+=B5)F?}Tc8mE2lhW%)b?X>zL8VynD{wA>6BKj$#eSY*
z#%Wk+{3*@ixUq&gD3LUdt5m`n|0XyQt=yl`TH(0E;@yzP#EOnoiZGtbqGj5LoEqlK
zWM%N0#PiO@o`(j|(PNM>vH(gHjbCRb306k)Vlq#z2@6}rRN}V`dgaARhc^`G69x!7
zHOEBsJuG^eNvwr3`xF@tFzrM$9%;M;-0DP~
zAZd*BP){^9>5#hy!hDMN9NZPC%HLh4Y)zR!JtIMQqlb66HIa^dNf4h4)TP@ZVPO({
z^AS+GCUuXA<_XGeR$jY*hzH};>z8{nNmx8vNC1&AMzFk5JQ)^5o>C;zX^9VvT}zbjli4s<
z>>NcKzXEJl%IlU2$Nlsqus$D?Me?nF^m7|`Q|DqkS6D8;PjSBup}`p^bDuWjT-15m
zKr=qWhxO`1o0p~Xl^}U=+>A3({?D{wYu+<2?cc{mnMBJBlo=Z$&R3??8Of%$`g0n0OfwDReGpzjf&LObBM#(?t$+JP7ZGD^DLqulAQ
zXvEb@^KW*A&)h}5Ui~e7DRr_u{m?w-AiBKB7!aUbhlPluJ%q{>D_s|$o_o8-m%*QU
z8m)DyX3db2$`nAX@;QZ#{OexD2w*rHD3Srugrwmz8@M?g_Sy4sfvXUM&KNe(PX`s>
zlVg|n7$k?*qJl9XdMGu*l!?WhZgi+ifEUUeYT^|4Zz_76D={g(8#{tRWv)t+7R+PM
zo&!$=A)vw@h+QFzTG4SRkd^Xoc$W-e|Bnzw!(XtJv09}u?MS|kwTkm+^tBwIbK
zNqk&YbWu!=Wq`M2;%Q@~J6X>{hm?(Sb!Ot)R^_-?n9BR$sU#1H!29hWNiEiw4wh|j
zEszkw4po^_xwmxpPRb&`|#WEEQ8SOE7TFb&C7Bo4>Nx
zR@tVWA62G_sX~mLFI
zM4cN)o}&+8iH3~bEocWlN76P$r)+azZ#~61F%V#C5~stWlgT_4BV&Q&W?8Z--g%AH
zM-#?rTk5Nyz8q{SAY)t+eG{JoGg^|Q5lvfL%w)3Q6krkcV$hcOzAIydqFypauPQ-4
z-wU|9fN?Eahjyo)U8ZmSj{d$r{n@|tkFHEQe8@E$)5lnn3qBMVnrVJ?QgVroS{rLc
zG8ijER_m0%`ztqv-7DxUY}|*cMXk
zVYzU*j=?Kh`R))C(YL4gxxwd95tmQpvq&z28N};Vh41=ba2p=h^nO
z&>nPhty6HT5VqlIDn(-a`CKf-a!uJY77N}hkh+sS!QQHNbY|fU8o|kM
zz@c=C;r#qIoyo77q%M{>rG`AHgAes{40X0>Ws>~fz!DPI_M6ggsmM91V?l1)T5$`cj`v|0O?3hh)XX0?2L8zf2?>bp(o0XO{|Ip~L=p(ImU!jGH;x
zj!?6Zn1fiThUI(j4s1OSbR59n>Wk4DDMon1y%j!5?9_vv$He-EEre9Se7bv
zfV}S&I|dlDJ)H|O&fHc1UbG<~+!dI*7T7R?zKFhc)Ct94ZyxgUl1w@;x@Le-)PsH7
zfaWt4GjGOT)UDkg)0al{2ij5(7l@f8^jUZK=}vvTyDRlR=N^5cC5dT=0<$lY3M=86Xw7zFbz*gMwD=?P|@rg#ZW+59n6+pc9`<1yKh6
zK$5-br{gD>VGn@GCzM}!V#9h~7p3ee;7uX6jR^??jL5rBSHC>vWALPY5ycCVgP)j6vB(iWWU+KNjm7y4C05@0Rwj%
zK<`c_hUmEzJ@lUf6m4xsr_n5>tF`e4hnW~wZ$(mT%J#XRO5y0N&OQX{wg4(Ar#Og(
zU%FJFv%MZSGN2k5q2XBYGdd-)+FBFEDwzVg>6FS^r&303;IqpVBgqw`u(YQGbc$KC
z#mt^eH<+{DEZHu;pB3^{CaEZsUV9j7DnI?wl^1MR67tf@4bn_V$Z8eUz1Q&$eUWJ;
z29;P{Wtl3IJY_LbtZ-N^hho&_JpI*e{>_n1>+O;U9rZH95KA5w4I
zVtc1RC1DP*jw8$uNk+WH!Z}Pz&s|J=Vi!YvDk_LijOc2J&u6yVdDcn4wrnRc9acGA
zFFF1}U-}h8!YFH7>i(sdiqXq2B1~mQ2erZ8npxU>RMhQ`?p^Uhp$T
zECRA#WidDYJ#qyoMDlq0;+GkfX^Jw`hhs~D{WF0!jEQrWDBd&G1(8fX&oUt%C^}t+
zC*Bu2KNFojE9w6h@bd%PeLwX^Us*ld`TQB`at+`hxziEKZ{`xm+aB`VMV4n3&mCs@
zzNha0r^3Nr&@aEs5B&L?Vx|71RKb_=eLC;iW;_%B#lZ1p^eyCZleffQw;{bzxrbs!
z(B;*E&E%oas6)jYknawqpHAj<;`D|OdF}W;)WI&SNBT-(uw9ohtr4MW-PFIMgJvw&3Vu3Oh5}}VZ2ZH3PB3s!sq|N`weGa
zag_SJts8R&-ZwIZ{9hWWOet95FYYR*pC$yA=*2zt=i~v2k4N*^a~03w#UJcS@FPDu
zw4>@q4h6AHI7OAm-Xu5{@)028uE;te_M5_(?GubpHsjSY?DOCNMTq1jd^Ofl24p
zA6xJju&_7p$>Y9s+*ujeXaB+=pGq+`f~2hlWpB-(?nP!~&u6YBiAwABaa7^UdN|$;
zn7>%Dsj(-&zeullH#C)>?Rt>yTJ}e_4};Tt>R@(8&odiksFFUr9zMOlRWY
z%||f_3TBef|Dj^)=j@-D2>FqiGeeNQ1}3T-5lty{)XPYg*)KSq-uLE10s`qj<@iE0
zzaPwcE~7Yhj!9EwQQdBp&~_$k(IG*Y3~mqI001BWNkl(3ZQG8FkD0pX=jI>zL)eN$lN+BotZ?W4PY(3gJ2
zlCav(l_U%b64uob*pM8j#I`ukGIeSP9IDG}=PR+cXx4wy|n5KbqvF44wKVJV-kqtMZ*?`C}$!&9BfSF`e3b
zD^N>$`WBO%X2$^sYv7Y7?I$*&RCDjkOscPX
zm>B>12>Pbx1LMALXS_X;N70e_mipRg4B+pJ6u~GuI`yyKwxuI@mx=Z{lORD8vXjLA
zqb3qo>YlRu*)qU8Fc7)WKEnCegx(@N1yinM`GCJ>0%u8F{z_nLg&cqbixmIQTBjg6
z$&s%hNum$$`9V#PFv}7b7=0r&9}~lbh4PZLFtEFP)cxJJVV!^a4gH9KRNy}@lI#wiRynZU~Cio;Uhc&-Xsp=O5`ZJNclZqr%lv^Y5NeVjp2Tl5=idAfz8sEl!
zUn|~KO35}wIwCve&O>=$y^kB+Q+G_0&FG3g-vJ}PSN#8ptk-Ffo(iCZ*bsik&ww@S
zYW$Vdy-!)b1h8pl;^SBD1H;qtHGtIM#4^RyT!Gc}S!+`1vkU?KS_31xDE{}O4K*B`
zvGOE^HuZs-0}~(LavWH^CH>_|3Uw+1%f}@?evI}sWo7!yqYsEJ;VM`F_dFCnAvepo
zlu7KpCBQs2Q(#3*5wS%Rq>)Mq64DTXUNQVgfyfk2ITB0bw$KiLF?e2={}
zpARQFdJ>TIG12S#E3<$tMC2f8Md0*P!#iw*^PD?SNfR~cx_#B0?>{VdRjFTfJP?a}p8m;p!wt-0WTmx>;+}YLZp_raxh{$3c
zzwHcFPIV}lC-E^+|7PjAKTR5;P=c~$rgqVW{0YkOwUkR$7i}z)CT|%Db#^0=ux$(#
zwpQ`uLema%KTphUs+
zl_w+6qP8O5YlB{r%2I{(aTf^^G9KR)R)4(rcj+pI&Jew=y;{b7Z(Bd@k~4iHUoN1L
zW8C+SX%qm}-M42;!|e=HZBJE*mHiA+W?>tqDsx_(*`|fHtPW==u(ANUHA{TlmO-;i
z0$&8aJH4v-uidC!=#{N-V*;XwK&+bby0b4Oa
zq4(+_bWD_|T1xj|(=)_sr@xH`N^96_T=1}!EKl8vU&oAtgb=nOc+}*+N&i+aTZ+7P
zqXV1;1Q^(=Z4x9T2g32LodT3ArvCiiv%t0!fVTm>&m{bo=%F9b7N}P_{yDKF!XG<<
z`!5Z1l|N${P`w;5vTOX_Na2Y#9#U+H5~znwC$m~BHfh&HfG*utY)9bgX&CjCBP;NI
zQ|-yMka_(<#THH(EwhlfOB2>{qr~UL)+c}L0xpY<=+kTeP8}f<^Rh*Op&fxT8upkJ
zy13~maNxXRY5vRt;TMYK2L`rN*jPe5e5Kfa?#msV0#$~pC$a2@LQS?^f~kCkZls9M`EikLum^U|C1W+QbjQ34%c<7u{3~)7eZcL
zlf}=WK-gEd{)LsfVvk#0pqHdl;H?L+`IsWXlprCqK};1BJ$fbo52i3kkRU;V1PKx(
zNRXgU=q;%f2n`KIM8sE?qD%O{fQg9-EG#Twq^UdV^K&5ZCJ>#TJHu&#MIj(-JzB6{
zX0S`hBnS@=M_5?Kue>0W$zW+|8S_^oBD{f5*Jv|_w21-j+p;KNQ&H=$L|7E@EHW|!
zyU0_3B>fl+M~9LFo%BtDgj7H-mm?%3B<62sfv|IPb90!QYJLe^_%czhs3^l2e$_?h
z7(|6Ieq7|=p!5+;Bj|H%Dk$QH78qJR>l4Pt#xed^TU?2SURHknHO$O3&)E?56C4~IL$VD4LZ@wPY}68GA*+v%4*~)VXLofG
zXoS6p_DqD2JAt>;7#s|=lQEkUP;wRdf<>ZJLjOK|_>k!muULro5bYw`M-@VcyuG~<
z7?_FX^Qu6UCwK1LF@8D`f}a9@r&-9%)Mm7Et1>|L#&lSTemMt6X>@+N!8h8hI
z-k$~VPhY{bAx|Hm@C>!5ME#0-{+e?YVK*YKB-I(9myt?eUtjq9r(NTJ%?%|XWk`#p
zQXor~EXa{VWl2)6-N2L1Y4R++AXiVI^yV*LMdI=4lL5T+nFIte5d<-ooCxCN;o+fI
zA|mdcD_5=j3LFD8KQGS=)NipRfm!TwfgqqPvGsWj1wxg$WjydhGme&bu`omkc@x#6-$}zy
zCyv_R8)iZ&;FCCJh-C8rA)yq|<&_YkPEl8PhJTkjGid?m=ll?E>Xyg%_H}JhsV!7;=YH6
zXIMF;q*4vS0|SxLNgUxmuJ8y*hgHwQecx-
zj3uSv`1HXIUO|Sn;%c%W4^MKFdtJzKi04Z@e&b_s$wB1H&G@nNG#vGjyhxIW2*V%0
zdZFh`FN{8#9*i;?kN0??&VpckjMqGJT-@dzBS)X*L3oi2^NMmihJ~X?V&uq?7&U4P
zCQM(5P3PVqEhe^yotuj;gV!s+T*UN-RnTSCiwv=WOrBKIi7#%Cx{X_-ZQHhJ*L4^c
zpK?i6QX}`-gAq-)y-9}$(i{BoGce?Af{9U~z~cFoV-7Ix
zw6+^W!R^RA{5WhFh7BK%A4gBXtmS)f`=g4H@+GN=58inOcOXx!NC
zjb04Ky_W>xae6dcWCZ@0`T;}EL?y79P%NIUASmNbByaH(x#v-gYttU%R_?&|Gf!cd
zOQ&Q~gq_3OE`zcDvHs&l_~QT!_?c}!%NLveOOL6J`X{ByLgX{7?b{Ti_jwyc$45Lu
zJ^Ck1nzIJSpKII7CJYy+H$;^#tDw;p=|q+z@pAKz=sfKxGBS6Ie7*%kzZ;K3hW%=+
zj*!FW!b(j3?F8J?tIr}T5*N>f!`jgrZOa%Z{rfb75sFo|Mwb%NICD81ACpo3=z!-q
zf8q{4+n)kOeWJJEZkL~L=qXktYc$^8`;2>*mgri~
z3=SC?X-rJu)X17P6NXWX
zjT>8KtiMi3N(~|7O9#Xw)cNfY@$heq=`s-Oo@*i>rueqwI7}Ma6be@qiRW8};=A#O
z;zo8dm@Fp;QNFM6_>DgzwDf0Wo^`XL5&Yr;4xD}n-}HPH)kuco6)aJ|NW!EtI#J!p
zXjp=Nhn9WKv>8r?@=)TnWov3v`DX=RFA2T#K#Ewf6Q9V)iz2Ir#c
zqg7h#U8YYeRbl2(9^Yi~#(OW-S_^~g#lJ9j+%OCtHUaau-hhYO1#H^;fM?|Jy1pJ$
zh77=nnSbJ%=!S?2#G5;h;rXRYIvnm-H{;i_gV3i}UksbH7&~vi*LgHF-1A@TUN#Gp
zCr-iKRR?h2Td7S^Z*X+^-?$lXW$s|y-ntw|UPs1WjDL48!>q{@F=g&59JueDWDiG(
zt2_KWj$*?R&-gu-@{c&W{t#S!T;URr*Cexk5+d?FE^Pb_8!x<%Uw5DHVEdZW_|N46
z*3TG$0Rx6(=EkdduM$(_{!egh?L17KGzGI)9KZu_Z`?j`2=_i2rnfb+0qRr*%4A7!
zLY{b;g)}o(mCtC||sK{09Eo
zB4rvuW>*nyx;aCGpOlb7q;>e)MNW5K=0y5ZDY=dsjMU@>lQSg2B7;pR|@TFyvAj|3@PsPY>Fdpulj{a>LqGshv
zsL{A5W*vPOGfEnKVk%noT^%NT
z@J6XEpTn?zp(m<#a)o^pSCk&%gI#WlbAc}cF?57G@-=Zm&UPLcc}Weg4GzWZk#49u
zCkX%PZ4kx`mKG*4@hCP;j{0yJD@S*Nvtvb+uT&c?`p?1PSBe&wd+*2SMqRMvt}@#Q
zyMj5b8(`q(cZxhf?tz^{8)3*+xA>05HZ{Dx#ueQw&(L2D5v#!|Zb@+j8wINGhWd;`tg6?_9v%rN82+SyMDA
z8Os2|@O0x43|RIG1)BHA@Ih@*;=_838M_plPd`Uk*d;7oa2KTqk4K;KzPNhT4N=kF
zxPSaCo_!V-9**amhhxZU4}8;g0v7x>7b9Ba#)Aa|G3`X+>7NM^{$dM$9Pm4yX0M6X
z9a^G-$#u->KN`E=P)m*TB6rYUtlaPVB~g^WKa91z{a~NhL|lw5KMug}PqWjAI-q3*
z6I`FuAES4_(K_|R)x`^ydUZhH_APjzlKF-|+KR1yZP2}vH(Xp|?~sD}QGHzSQ*0eQ
z2+QA;LZg}vYzmdS3wwpLTbE+O^y4sb8px!&H;O-)jghOL#>gM(ascD{%)mvHDrnoi
zE1DGffWIe=$AT3*aMR!LiPwW|K&k1#AB%we%>m1{z|eERtph+s{%_tG`1TjziL!Gd
z=sCcg9VI#d+1mq+mI1MS2ltNhdv6~94k$bjIHYElP~c;rX-}a48Kiigyevy06v!Tp
zK>tr#oS9s0Q1tq+1%0dc!*=&joS*85vK0qlXH2Awg2(x#7}};Ts#J1B!!8rBPdE=>jcrrgqI$*F`0YW;qz`|v3|*baLg6Qa@MPC?^lVxal`2+6-8QpuM*X&F
zX^Y^8*g3sF+BB?*%9U!ManD&e`Yt`@d1@|yl*f529ohzUt5kwx!!DSxmHN7XA)~tkzRqLY7tTWKP+N1H{?+++7$sd2rdyo7L-@&ql2Zo*r$F03y
zsMx{<=Jj0g?To-Un|T(9VPiZ{qLmA6!yS!&3s%~U9QO|SpwaiP$lllmh5PuZx0#3j
zXxhaC{ZB@uz-Gem&&2v@z2pHx?k_=$@)a>)r>jPiDW)H;?&*x-D|cdcx4I}_x)jRQ
z?TMw={1K_icC_eE(34%5-m@ucR;q}qb=zRp8GpEM8v>^`^Kku3te3-M$51rvz7kJU
zp>|1<4lQI??y(cYoZDgH%{T%Q`C>i#IRAjZ-iT<9;ox|n|M8ZIquhcA7=k#}+UVqT2dcN4ddPDYtm>#=D2D+qscfB7H`+ZKez
zvv%Ulg(Fzj!xm>J4Z?ye%9oDA1L5s<1v}2zV&IzXSlFphN}D6|AzThmj}UBoY>su?
z3gGPmYlI&4L4#jHP<3KH_#e%KA?BYkd6QzqL>?RocN;4#obP~_+jF9qJQ%}P^IT*&
z7SHv?rGj?2xH}*2%*%!rc@lIwMMhxVEH5niY>nSX*q~@?hQ*^p;r03v9^AWulN+aC
z-ccho?cM_UMOGPhAIk>~!?vKtn6>i^E*x2g9=2yOY0v_QCKnyPrLsN7xyx^3Vodmh
zvp7qgyLS}n93
zFc+%^o8j-}C*cosRO;Fq(T8^7QMjTU&v)-fQ1fnZw3OrhzD3w-GX%@0_CVtr)ljSD
z0L)!A7-oMj$4QNgw?x3zJren9^gy3d7qQ`(a$yO%H%@FkhoXIYp;q27xVeU>x_;z-
zSFw2DIGimz9qWE>iGpfoyhaC^q29z<=+od^6e(5>-;ZbwldBiu65)fRi`PTme+B0D
zYmBPpzJ+6p!T5DjU6}c2j&AZW8vUE=z>PA%<3qrKhQObTfj&2ZHJgABzcbOi0{nVi
z@%w=HK-T*F-5MsEOg81Gfhm^+*J3*jdtpKid!pKqba
z_#HTR{t#A=azefgm=7fna>EDPW|+A80M4FTh4%6Tn6da|nm<=#`NJOIkD+66!nPMy
zo;Zj7vuon*@1w9Nt}zmm5|w(@em(u{hKcz3RuPQfaSrDXt;Q&)e96+qu%IZsxf+BU
zW$p3!WPTiI6pcUUe?Xt>X4H*>_^`qjmKXi->lNVxq7mQ`g{%$jaA0FTJeg+=`AHv4
zIUj|{m!Hu8w@{QCng_4;{!&)GzB(ej`}l>V&{;uu=sWu
z_FTDyrESbKl1vfE1HIjFcGm^u{b@gL-8qU$WnNcO&SCOTLGfJdJ|C1mc+Q_afPi;6lyvC6u
z|3Z%xgKr3;WqHtf#Xnft(E%29r{ml;+MX~NRkEzsM&im5dG|0b<{g85
zH*e$U=miinhahp-K*O*pzICw$&}*`kHnYb`{%qy+x%Vv++aiBFJHv4^_HO!;o4&
zII{1`m(fZwTHA2&D72`^WVt}rEf5H~a}5vd
zYolIq9qdQ9XfT$!zS9%}n(0?!&yPQ!r%;Khu`u%zG856?P9d
z?)c)t`hn=&u_HRhd^%&$x_j`v|1X|Uzm1EuL+8R5uQI!fC=B;*
z+=1_d^%&SW_S%levr09A3YKxti2elpI1PAPj|pcVz%H3G%3trn
zq`p1Ty?tXi_uB`vp0hA0dY^V@rZy;=!OF%0rfe`{GizkAwc?R5T-&}AF6F74WlZ4x
zJ^*%g>*Je`mvK%-Fy)E}001BWNklY=n%>TjD|$Z|Kw$Z?%ey?_YXdVRr_cHAttfBm2*S2G)OL=tcTm~lI
z?-5{E7xlmSh|A}m#7)=NLQ0alK7!Achle0I$_&}^mPLblrIFlfH!fm>@y*R>GbU)<
z+=@11j8=WJ&}NKaU)CBe3q{jrgm*GR-iFyQ_d5&P3=@JV6O_ssO`C~Q+DtTpsjB8#
zO;D+}WdfTC!H*OD@U}q??C)=pY@3ONiKPuJ&5dCsGli9{Evzh(BAGH2YdZqHs^oyV
zbxE{uR})t6Uf^}4I@{5vKWyI!Il@DN5oMMQdCN9HJvOi`Zwqv&9g6eEugAP+7WL*R
zjy)@jHf@WiKKar}I+7r}vb<=~+7SWgPT-cvD#IV)=t&P$Yu^e5jdk-6vCxNpX@$CroNv9{F+hEoskB!&kdc?KG1v4ufSSSnz)>&wKi8`&-+sK!D7j28-
zy{aIGxeZFTYm4d@u6Ui?A>&9F9G~42&NUmL)6f}My6*veqveQ7&1zk;Y{>qFZyCeV
z$^uc5QILOfgZrl(C|=4gjuXjjQMjNzK7aJYmq8<$0}2*2O@#oN7}O{G*pOvp1WQ(f
zy`AjKK!=ru(U)mlk?z45Gr|q|I=)ATMgG|IM6Q_fZeoTJqq3pQLmyOV=Z>~3g78SC
zgDdh?AjTbuM(5#nXr4XgiPaYvm&1sCxAEZW4vZ-M4!hT#z&qt^n@?`;_>`kKN~sWN
znQdVdwEv8co`L*@35q$@LB6NgaozVVE?j0~HhNSis>QR4mGMTYB4Ur-=?p_-kR~^DbzpHpu
zxhE#)5KSwCiK#v6j{XyK6JHEtUL5^cjX+VZa1G3j`Z@U<^RDP!yf-!;Z-vGe&!9lh
zzfjVI>1u)8@H_9qIw=lWO6uo_q7Z6RqJDLJ+PeY=YtBKZ5|(O&VU?I$#($5*nC4GK
zjxw@_wP^tSgr~Gp2_{i~@bQx)huXUt9%MFjYD)2qSWt8RpEH)h(k87~sS(P4hX2k3
z1J?sjSg+)c2xYPu#p0Wk1eP!n9r`;^cq`DaJ20&gV57Xv{R6;d#>%7pf!ry`$BgVs
zpjN}0u(!yDft6~bHiMS$-y7-7N~v{b&tsgK`u0o&`6sx#`61-l7EBqvJMMVFT(DAk
zM@i;T5c-URCgpl#>d=>%KD{OCt*(bogU4Wa>r$}R-y?KFq|0&4o3aRpUq&Nm!Fm}N{k8+6YyF^*HZ^fUqm3v|%d%xhT<&Jcwv9q?RPts)1)f@r
z>4&T^cJD=WDPRV<+Xi%Nx>s?3!DS-C5gx0?(IE&4P5Em(GOMhxHT?u1FVWc&ncG+S5#rTXMOK-IzqNAf@>?Z>rodU?Qf3**`TH4|Iep?hVjmF#UA5iwFIA?@B
z)w1Ehc3V8Y9EkD1d819JF`oRSV5%m?vtUWFVEj1G8`YQQMDGInDY3DAH4L9O8h6|M
zhFRO|VPnt2u*zx+(@#E#-{d~<4K#;ccIy~_D=Y1U+IIVJ<>VQ7eyWaP-xfl)nmMrj
z!Z8$h8Un}Rm14Mih~`IYh0M$x=0<_|q`F05xIg@ZQ=!W#zS1BehtI$F@IHUBnCVdF
zRU4t=vmMy+T;+vRxhKx-I0@I7;6bOr1~toF9leJ%K+xTX@%E50tJDqDuX;cy{C@JY})h#$1eX!`5kBa$D~2iuY#uP#`8sWJQC%
z&G2#6P%Lw4g#L{bWXCjLK3KhXiFv;u<@IBP?6oFi&7{IOJZ2cy-1gV-eyNdq1v4#=
zmd>BBW&R=j7yD=?D0kN{!$DVhCUIQJKoBs6Nuy0yfa-+H3xRI=mFHMN&b^sz?f}-+
z2e!`#Cf$v@v_gAeFqLA^0^n5&dPt43S3s+tz0jj;Yt%1gPr@OK@62M6z?ta;nI&v(
z%wSt@684=qffKQxW7{#tQKg$g-@!O6u6Jkl`z&#gn_Q)=5n%x?OQ{3R%^oNOd?GadW
z>H-eVZHmal6EJAuwV3Ip*0nmKxz%MHJ9iU@PkN(X`=-c|hEY5|*^Na4vYV{|+B!wy
z;_=J4diWp68+U}0NZxey53$cmPyNycZMye`l;+3nN!ExsMN+yd1hHbqkka->*iKTq
z$u<4g1_%!gjr$Vm2KNs@_MABsdSzV;9h+O>^09NcdH5u}>$OMIv{asWutZ*m?1=XF
zi#wF6xS-ia^aD9U{e1D+EF0{zn2G(;aBJIPJW_HmQzq}Zy>8&_eer&96g)3(!!ecV
zwMOO^FpKuaM^7C)8LeueS#254ZCHm}ehM}f^mq%lTr@?)mbKHp!!$mk;OGAd$dXOt
ztTcmyoE-gAG|aOaBb%vsRVxguPlYSCgNlm8zmFpkYHWUqf?{>5!T8#rIP|_M>Q~co06-ILs-i}@5S-n-9nS^1
zmwV&Jn$5WK|D9Au;njgTm^g7fMi1dF#+R(Xxp_s%u97f
zhoZ3zMTYFnCt!5m-5A(wJQl55hgI{(pvQ<~DBQaSMCDHtL&;YDcKc$
z%3GjxuWtB8d5uYd0?6&@f|#{>QdK`pENlFTRSTT)@aI98`_KDiw3!*o4_SaAHe1lI
zX=`-v(;Hn{e~-lN;Zznir@yAE6blyQw9tuC-&oIko4+ZDAg>8o)
zrORf+n#ynOSPxag&tcQ%hfuUeI6Q7%z=E
z>=pBJ3tW$`#lCkX;oP{Ga`e(16+5&-?kj6B|718CcW^@5YELABec!{w{Vg7!U5`b(
zUcjMFJsqz#g1H52-}@uf=4IM!L@$uRzEO9ywmgOr&h_E!)EUdeO5qzNw=%2L9fO@i
zu&uWf>N?d&x4(>0PUY1dv(k;wr07d5>)a5He%zV#Rx+|2_-@8R^f2FpzD~8_>|6(R
zyKjSezuz&XaW=h`DtQRO=g)E`Ww&tPA2$?nbVMFG?f1w((7NSlMnh(dHtnoo{f`$)
zws%9Zf&K_BZibja@u&!Fndy!^t=&+jlRE}HG{dwWR`Kn?*c82|!51LTliF{kD4}WpR7BKi=g}TpHTJfBDAer7fyBBVqQRH
z)XZt3B6sqT$)~@X;hP={vCuOV(PI9-tpkb{cYsaYJRRhyKLK05xsQMTxruk5bHZuD
zS~%qo#1l6QWGR*lOP80&m0NEiAG;RLl^kFUdvu;Vk-=VZlPr}7V%?tRxOM3|UitzK
zZGJ|7$KtS4TXQBT)_E?*n=4+VkmYv9lx-F9?D8et^9V1Lh8{iQ>wOWyLyU?s#*kzOrFb{CmtBpB-^}mJN+2
z|AlI{aZ|Ej+X&+qR7N(Fl-3Vo$~ckaO5D@BbayP-SRD_=ULhvM;_A-0A;!%Qg@bAW
zY+e^*7FR(olR!Mb^cF=%?!>8)@9>HQ*|uOY6p)?4m~NgZnC&a*>rjD=$>iWhK%bw1
zb2(W}xq!Mwfm=Km=>hbd0(`ayEQ3jCzG1>yEB^H+`GAeHm{g7j+K|Pa7?MhPVw~9v
zjGjzqwu^da>W~I@nNl|QF))@`=*22(jOVFj$
zbj*4%95XxD!Pb2Ci1P7=b@MscUbjN(gw%q_cL(v)puGsml?xVu-jG)qfu+u=^{&Z;
z%pQ%px5i)NM!@-cURaqppmEi2VDd2SAFZkUGV4y5KI;L7&+LpkTl2v_$_M_|%`spcUI8PPIxGFU2}-nU
zjpA$P!@ut@P&44CUnt4$i_w!wc6T9n4kl4PCa5wnKG`*|)&@g*rVSHHADo9-z{=|}v<Oq->il)fM9Nm43<_3w6YN*BqEQM{lZ7PqN`
z>1IrfeQi;1!g35K9Y>;CwfFOG#LqV3PARr(jg2-MXLzW2b;A3om1#3ab`2T7;|M)21!Lh=6*sA#>Wt^ROv{#Rcz)gq&ZPgITMORZe3YTL
zuMilcz}3$mud*4T5SCf)FJdH|;)9NFu{S&c+d9Mf`1P7_X(=uSer%7`{FB~4tQ#`r$Bc%O
zwcXh8+y}orZg+My2DAEyuH3jH+PPyT{O!JD*d9kb3K+#;k(?4NT{zsneLH25eNiDP
zsTgx|n^6o%;@X$P8TD-?hQ0R+t*bebaPt&9KVHhQnzLCqqBgm$J+`zg8h?mVsiZ;$
ztyJBAcZKMEJUk3OI|UJ33Y7{N@)i>UH&yVIc)X4HKT-5qxowf^K3oYEozWhux{;?4
z%EL?GO;Jb}M~W{kQoHcoyLX9>j?TM4NwGO9GZG@Xnl>m_c1brt5{SHdiC_yas#bP+
z99EeSd7Wz^36%1zh_8!Xo4}}4Rv6A5(Mj$s;j77a
zSW&~1pZZ%BFf2HknBYXwXB0eD(Ngpojp#ELqR&{Mx|bySj9T;=E2_FH^6>m*4lfK6
zea3~0W6}twwFlHck@+ey9NxHbBm2k5KBH}9#!Dm92FpB^an`<*ZTnHh5)>H3T?+>)
zRPv&X)(fhytmMTPkI-(%HilQu{>>{@s+8$_aBwh>w8JCwg(PCLF?REUyy*x1?MoM#
z+FR5w8QejZ7v7KQMTe!lrN_vocanYUwn}x?RKzjc;y0z
z>)YDe;^pO)>6p}~Qm+~k36WR16l{T4)ylZ&a^M{LNN&(>Lve9&goo$8ceAmhbm`K#
zySqRBF58c6M+Jw9s2i6EQn}(^#Um~KMFQc0k=XirQ_cXs|FUJv;_B*}`C!ZuINOJp
zx!$s)9K(?+wNrsn4hTXVFwornK_q%cm~K%LXPsQ)z~2!jj4ZK4yjcM%J&M
z$d{qk#$5jIcl6ZC-fEpe9<9ql&AdAO1h1Gi
z7Y<$(W>2&H%a93{_V`qF=kk@w)f^K@hNFo
z+WS!LaS~Ma9{?DkIzbt+Drhio6$lG8ey#qSwt*%$iV+I{6-LEO7L)
z*6K4E#eF*UXnSYq!}ox0g7i=2@+{gB`Gl{3^g;~4gSBE8;lX1ITTmsDim
z_OMb45+go&dQOH>8j)O-DXJA%+SsKUbme@gUPr$yk%!l^J?L59?&ViJ>%R*L8yA1F
z3_%
zR^zn5P!oE14zI1bMQL{@6n8_o@6>|PAC03)0o9e6RGLvjC`IMABbd{+kg=S_1lxvi
z=tEN0LwUI8dU?@7D78NQpq)^*cFwt9ZiX2p0}I*?_h4UavS_oy058dic57?;478_R
zvV-U|rkjrc;lB6iI`0RfY@cV=`mVS>S*rvL3fo4G99pCn`q}~y-Mzt!6*@*~Nj1NR
z`*^SGJbs9><@s6b>1u8~W*9RRO8jexPf5!m`mbvR8(ee0&_rQP{kL~>Vn84_gJaQ@
z^`^3)i{9NX@(MGlG{cNMKv51F)hKBxNSWTOTD_!&FvE=8V1bj54UG#pvt>{yl~k>1
z3iC2K!?yWw7W)jrt6CLYi+U}ki9=y;MI(D-a3C0oQEJ0Z*5K^fqXqq51%rG*T+p-yEm|=`i*!xkp=<;8g@gzfGkDs~mm|=z)MGS*j
z&@$nM8D=~Q$kLW=y!2maEU?ayPd6Eju|r$lq9pU$32$;bX3Q|=y3Hb+m0QUOl3!S7
zjf<4+!kEiCuh5sstf-Qk6*n@~kL44x2Efv(q$T*z;z{W561wjp$5dl?lItyd>ylhw
zlneBX8Ba?4$qVu_J{n|{DOavsO6ITo_wQqXWU^WCqzr~;7&j!Lba2qw6UuWKgnUlk
z(x8_ByhlC$Q)YjY*lnp!cB?|1a0|cCR-aDo0Ke`
zrV2S;==S;aF*Y_fwuGe08M3Wt+W?PIx(1YO2>1UeexY)Lg=jPG{j(gFZAsc0QnxF~pc4+#woEg_+DhO7&@
zorol2x{&?G*c~$Zwy}@|J|ZF_rQA&o`o`0ANL?oTyPWVbCS4{SLsI5UTTy8v>8|M0
z?~4X?P`rc@$Zh8*`soyKY^C?KEZeB4sFdw#jM1hjEYckmvd(gg@-ir?G{cN17E-Y#
zNsphAc#kEiRAirsLMl09?b_1t%}y~%nGG@c$>--5`Xl#!lzV#`%Z#OTYDhwm({chs
zl}}-jm-0GmgHTC&(E}
zWw}gFL2XEaBlVLpIRGRXkPZtuyO$m0%$PAcWMunEk4DO_j~Mf~nH};HerardKF=Ye
znn$8v$LD@sQ6iUqstr?YKY2l3#+7uKNu?QPm|=z)W|(1y8D^Mah8YEhK^H4Z@09ee
znR%)irUdDMkzp8eO@gTZ@k*vop9^6FcsWJd8r<^~803p47q$@(C}5@kCwC$}@Bq#(m}
z<#=gKVY?D?T$gLWv(vG1IfS%fjVVktH^}kL$;m1E&~OeqAAgq%_|Kg?XOtP#q~TbvUcF3f@UC9H%AGrRN>Hl2
zK@NnHz^4y}GQ#1^nKK4m-zZ6I&6+h+3<+IGgXiqovqt48kY!P)PMxe7h|?ft4h{}Z
zNxD$dkZn&oHL^hP(zU_s&D6nC2Sw2qixbk}B=tge4i))+
zvKHMlAqj{yASL90e)_Z#X4}jL)v8s)!y|Ryr+@$bH)UqX(2&NDtWVup
zQCX+5ZjB8x3_BnoK<_NBBs#{HozUOkpYr9)XFixL41vGJ0336uWQ58qZ2h3#ucDE7
zW;!gL4aT0qGB?ObfZDZdXE`o)gltE~YV_v?(twr@jmMD5c|umFwl|F_%`7WYw@>gI
z=A!;{001BWNkl#x6zc9vW^E$Y{=uQyqb>^pMC&zK>}my|!KXmlZU_BIV^r+H0#C)X^_c}?krPKP>|6PN(t%R&k}|$x8N~$LMid+hA7x&P-))d
zL+RUzm^x(pmhYnLoRASkraoRvhmhP7;&I5|)zRXG50@rqDDIpzPQbsxBrwu8yu@g!OIUdOICVgK=nG3{}38jP_OS6#5=mRAqOnwk^ASuevsXJJP8p+lGHp7O@45}bO~iT6eN&C6%#^OTrq?fsbUBTiz}gJ%qn6+!nm*Z$5q6|nP!3uGtBsJK|^Aq
zX@^1Cmu?Yb
zYi=$k`Z>*RJBM$4h>mI?P`7lt(GTum6Mm&s!ziy|zf*FN?
zX_LytzghKq7dm(Dto7;6^zZW$KBD5!b(2_gS`Z*j4Bt-g%iv`v(@y=#x=Cv|oo{ZW
z=EgTOSbDhV4Us>5&7pAp29PG=5IcT|D4c&@eS>diEaHc|x=+y%
z^X+u{4qnE|Cy`gGL;2~md9436N~ioD^5>R$EI$xvOu5RQAehNdQ9yNSDgC>RX7i0F
zsBDGta^VU@7{oFVy7b6AhK-oVm;3f|{F>R^Feo(l2)c5EaN}QwXWURw7Qfe!1!b44o@TsQ7FwreK8g
zJE7TP&^4WpeU(av2Y(8eO>C$v38HM;n^`5RQ6<*k^>x!|m$9n?O1m=I-=hx-M_Ru5
zAr&pF8dkm%ftox2?qSQ7s+!A*d}ZP+FD3(Weg2
zcWz59=aQE$zmVi-!1SwhU%8zAex@hCvBI)y2ZoNgOI7cpD^uwy|Kwk=u!jl5p=J?}
zI7xxfXABJ645PGnl98VBWx~J`qpL>%ENq*HhiPK?W#7*zI?dygQFRNYFV#18G3$ei
zJhyxzt?df&peG5clYIKtM%?FZV02xp+^;usJd99rDPBo!U!{o~R)s+FeXgB2%-_yE
z=v>o|haqSAdgeY(gskG@{+0EXi)HmLj2InFb#Hk#iEBHiGV{Xote)DA)UIMl^)4eA
z9bFx7Q!H3HgL*R_hswSO*p;H6cs56_D6ai0)L9P2Ou`uiB}H%)t{G6hPk1&Ne<^`(F8WbcXphdo<<;a1Q|
zwk@8{d+$wS$ycWdm!1YqBqzRH!M+PI{JZaCmhZhl-0gq39FZhWo#4wA`?wJEFZ(`T
z&fdSzvw!^ou4Y<`8^@V%Hu6(YZf=W2BF^n+{jvqjTeyNv2QLtvevA4xhc|q~nM^Lc
zc>dnCfgf+BUX&PdPW<*V7R+17icJT(5S3{>PtDS!Ga%|HfBq
zxAU7QTy@-S{<$2H;u(qmN08fJZV_~H8;hsE$9vP3@YU(?OtX2KdsjHJc{^7CJW!oCJ=yFH6<5jucO)l;5=<+bq0PsT9Pk`S2^U!`Ey#7P}XUiW=
zx^-KWb?sdh+6pF_bpxGzLiGJ(N1?QMrx&UjLb8Ix+Di~+F&@U9hSLU4cK!sA%6KAg
zT}|2aJ>_%lI-zl*g{cBLzH=S(7XO50lYWePvlkUFE@#GByqh@oO
z)o>-|-=8>sEgnVbiqvmb6OU3J)NJ028s1)%3i=;k9l!VZ7d4@W*z(^H}_oB~AK`;>}(axww1=UtUdpf;#AX=8jv!Zc#yT#k^c+hJWzW=p3|n<3qYgD;?3>n~v9b;4K?)0}x^x*Zh{12X!<@yVdFJ6y
z>^>Hi>wS6+VD(*YUcSTq60nMkgZvZ-@t4@YbRxt0_oiFdUJM$yoFln9nhJ}DT;0Ef
zaf5r)wM%#U4qMKzX^tuB+}SW~7=62UrAPnK%-?nKaoCyq+7`wQoWYhoiy79l3vaBu
zMB>Gbj2ZYLKjz_n9$sbtl5q^~-IXrg`!a0#uQ`UaX(QcJ?xZ`~XtJisDKg_A%~M`K
zuU>D}aTdNjjP;iuadG`?bnY^V{dcqP*XiZ^<^C1C-M16%+IFPxxDEVuFNug<6X^Ng
zUcwSivbaYFx=h-S%wrLMX+KLR4x@kXZglN6h;hr0kluVAUfs`-Ng>8ta@={g%^Jo_
zvd#2=i$&jEPjMb=LiaGW-!N94$*=}AVf&cYf9&J2xX7T6Av{8bTcyg_i~gTnF=ksdkwhUhVFNqh{G2*3J9E=s?|JcsFvon;06Z7fRvMu9xON%#=3+qPk(yW82
z%Y7bPX3v7ryxg@NZ9DW}@Zz7fmH{Obb$mG^d$i`c<}K+yauuhFlguSuW5@f0n0?SV
zW4aR_z>SmeTManB2X?iB^^0KeDOkM`LYD}l`73;wwi4`Kh|u3H#C59$(G+A;a|kB?
zEbCDaML{sD4uh+LsP23fD(K9@CfJowzb=THAj3{VhIN5yH#~&3;(WvY;4DaQ^><)t-|VTl
zf6lw&{`uq>a-DBjfPZahQ8DYO;v~wJu7d0fa_dzDHkIj7TDSmbOMn
z^T3*bH^^!Ma&krXXNBo|C
zn#xX&H0|Din|ltX5ivE_zB@!h%P!Q)p+Yp_2l#AfX$H(3O^@bJQ@24YUY_s)1FXMc
z!%<04t*P0q4atZ02?CM&-0Si2k&%BQ~KNZQPlCi*A7I^j$2UJY7?!^97cQ|0aXjBVvj
z)&M3A53B=tb6jtlRP~`k-EO?vM#iQzDPh832nzp~G#g`jr
z(_W0YOV=GM(%)2H-OhsL=kR%B8JoAPVfNt0xXZ?%y2Pewb2(b32Q$}i=F@TY30gm!
zb*Iyeo5Tl1hn!~Lucdi;jvz*zyu`6MQMY$_l$_iBYOZcy!18lGys>OETh`2GaAWrZ
zg?Z=KQ+|3Mbbdn^0A0lZI7k=(M>0I#dGVB=_yVelb6bM*DX4X0Mz8yr{#9(KI`9*=
zZ<|WHayj(tl=3~nx4(HXWaTdQt{skl;1<^G4Z^kaOun4i5jUGgjNbe;+ora|LF&fP
z2uimY&YV?WvT=5MVX!S_oqU-3D%%$<=bX3ne4>FM6PrqdUinr<8
zNf_!K`!a6BU))PmY1!)wNlx}Nec%W_JDWjR)Hl9j!oc_0AswLVaDLk`jbVMe)3rx`
zM$g~H#ccX(79w59j%`ueie>?})UdbEIg_l%fGeso#{%zacC@XkEc{GIj!5G2-d#jC
z8Oz5T*D$k-9mki>W%Iv@DBPZ*p?45Rems*Fp`<3{r$hW%=4qN1$#Mc=SNyV`h4bbx
zao8BXwtJOnL+Yi5kE?{v3I3hk$I38Z!Ahod^-j@4QQ6<7rCm<=>{3A}WRdx_b%r;?sQRaSrrf|S|YKwI&D714FO3c}XdUJx)rHaq(W
zqIfg)TCbKcw-fk@0uB@1xvC3Xl4~4=5>|=#cp%7^O*weF3DngJ>rak_x28hibAoUV
z2FJn*EyjZ|L2$SH3u`ur=Y9Z>zJupK6vS^F+!bVVglJ%sHm6>$5po5UfrWG6iXh8_
zgsdairpmo&0sJi9@t7Ek{uJjsx$FFvxMs~61!DWjehx
zjcL=;K2zW4rP`_VM(JFQ+CCXnN9p2>eOwH&`RvrzHM-DR7>xTQ`-?x#-d~;R(7_k3
z9bn~03t6~Od=_osm+%auS#{wIf5-mI*7rv;V#J8l&&Ww^y%5QTb5~Nl%}O6(wELW5
z&&fD6@j}raccF7fUzDl~ocTMJf49ER$n<+hjAYW*3q)Qx$CVtC;Hblc2`}e~r~Na$
z*eaIYyDp3Vn8>x=JBfVmC7L>T;$s-2ck&BN%mkE8KHHau&Xfb#w9qW2x85fP(!v~l6TLgxt
zw+{v7n)KtuG%D(6O(CIR@1!Cdv%9`*-Uqhlw3l+oeD
zm1{&mrD*;*c{O9kSy7HA{a&SI4Sy<65vlkmImg_
zQ!aGPjY#M{LXb{jEPNvd-7lAl|L^C`Q+{bLy!18n83p~k@++wz0i|rPR#;$RX-DZY
zc39=suNA(u9n^zbPS)7@wWsaVb_5090xLU8*;)#Ep~R|G8Oqp6g{Z)##bAbZtc$0m
zXy0lNd|cwW6&NZ@o1{byiI3t@Tb0ADN)rP7OB+xhWJXFqJ@GBSc?@CYuDz@s?oXiT
zTYDubRQ;QSz7TKW+JL&QH#u_T@3a|IHP??E;=I$-JmaP2(xz$5Jz9nyGuN~E({a=f
zTF>ltr?X~O*GHrc*s+tY#?t1caAy85k|y15VcX&s0Sg{*;le%o54=U`=WgQEKZ12P
z()9Sl1lBGLqv4CUac*~;>Z7CBekb+Z{mb`xb94~i?QY@PBbYHi>v5Jnc*LABcWC%Q
zT#*`Z8BpN-{18U8t%iq_C-q($M%N0rIet74wZe^-?HiDA@+hfI=MG1XUZYCewiL4I
zWCIHnRyNp)w(jgx9>qUf`DELl8FrV#o~DC_(z<39{Jk>AHd@)0#@0%V6G|&el`Vsv
zo~5=)QWvSDF0#hny(&$gsgj|Kl)iL$tv7X?Y$#o+BORW$CHUs8e80iNi@RC)&R|~b
z+Mae@`Z9X)58RRU`0yv@zcQ9hm(nX$5cJ(s7&Lh&w?lU@w(~pe3{Bv~{7$rNJD%N-
zLkn}5K~;xyYQuEic(D_0TDPOyfOlASC@^D*l2CS=M!&@5FLpBj)y}jXyqbRt(WVrF
zY&oYHT^6>26pFuH(^;P=dZG46&x?2cr&QwOKT?n~p`U*jrkh%bZTXZSUE=r&p|2|n
z-TU4TWQtrlyG!VGp%4}ba^!9(mLb||zn7twkl(9;Aj00@@~n_cj3qXL5cd_tHZF~1
z302MXq0qup5Kd7C6@|GTcRTf*%p#m13O7l#EuSjT-e2oCL>E6G#+CPl8r~!bXY;aI
zq%a(wK%y*R%+`D1VQq_zwjHZbVxd-}&aYEaS)TTty?AH$aXQvN%!O*586kS5g;fon
zYvYmTIk&^3tdjIC$`wjW90P{1XhKt+bpQ$!3X3$)vOUdSXuwyS4skK|G?!}iWTcD{
zQz@}@3}DEj2{hGNM4%94!$O&TB6W30FmbNW;3VGFx@&JLyz>=5wQkODzk1R0BR`Z%
z6`roKoRrh96%|-pTcA?u4vHF7k5p;DVTF%Sjw_x%!`Dj|GX3E+K6+6cwyelA_3rWA
z7l)`nzB7K=xM8hrZOB{&7N|uXU{MM?VNAu!P-QzUt)sq6j9{)#S)ODpP~hEq6f^qy
zYRwMZWzU2@Y)${F4(8~pMXdcX5RH?U801yNCTq~BGoIn*<)zd4^c=HqN{6POvG}Mk
z!U8-vd@PuHJzM3OL}_jv;G_3WqkI_7bzxL__IRIu$p;yIrzQhGc!>rsvfLf%GI{{N
z4OqqZr{ADKGhG8X^;veMY_Uuk>q}84<)Dh+_UtFP+KY_xR}vi_N4Z7;RNDSMC$A>b
zpmu7VD#|sX->WUDVW=>ROr*+NkSaF=J-7g(1RrJ9qP6@ODB7!+2TrP3l8e|4t*p|M
zQ3C>5x?~7_HwVzE|LY8DU4e(ep~ODA#_mNQ@ZIC9J#n_xWatWETzYxwQ~4V1=Xk%8
zDm_OrAaE&5hV*4~KqvaYK8V(S1Icc8q1JYC8ToB5^DdG_!
z2Kgip@D@h#`7r3I$5SqmZfivflG<|(^qMS;i#i1oGb?iI*NU<@IHYSk1$K5esN=G;
z{?r5?UB#lcKN6^M!pp-6RjiyrQ_D)G=coY$E?vTqzMBc?)SuS}wZ^ZI`XDDLeA_ap
zM=hMhvz*$u<>}8C6L?FFDr6tu=BO_OVQF2L^_z}zPSt>h7MD2kb2xQ+x5iWb7mM}>
zQ|-l3_$#9ck1I!`fJ$ut{v=lu8&F%PJhX|lw;<96?b)KSgaya5<-8L+wm9MRcMKgS
zM)CY#HcXu2&FAG4nKd?yNgE&X`i#8M(@qmu6V8Xl^t=
z*L3CB92=|!fw-X1ah3_vvu0L=FyP9tb99*$11`ak@1hrS{P4^elay|_ia|6k_FhV8
zw0+YKH2rKbM^DES;BlB^!S(3bs$kIud6K1ZKSm9zkn;Sv;~z3`(rR|MdzyYJr`;%D
z;EX$#RDJPn247#uf`L8w;+d`t8ulvfDwj$m?8A03ZQMt!s?(S`&%_aND+t?185Wo-Hk4bTv8Y}9Gk3cO%U+vI_$yyCvyC9g
zGH0Y94yiM!&6&RLTPok(&y3Ma`E*x10$wf8`Ohct{ytY;owAB~Pg!y2@Tbh4I{~%B
z2Ht5T-%CYYbTGf~K1TO97qGH|qv42JP7o|{vLKCR1UU=rB5XOKZx_D}8{|3h`>kJr
z3q4`)4tQ(6;M>lyJgrmL6~}8Gg<*p1927d(-w=7YL^pFTm+`lquqm7!G7=C;GMqjD
zBn68x<|Eh|2p(R7*vd^FWJeIqZuBTIhKTp7+ZQ@?ggJwRtZB_h&K{2c8~TX%=~x^_
zs1{KsjVSMWCY2~llSvZgrKeIaO()74L5KtIq!Hx{spqtba%Dl3Gv(dUCdvznm?-n4
zlS&H<15L#$aCojePj6enoNJae8Q&9(u0katBpDLMk^=`CHn#k;JVz8c5H#J1YvNq2=^*KKrjih92Ks07UeY)W$J*T^^`5d#c&Wv9rcYT&SCropQ6*Yo?NX241Df#D=8MF?SV@$v0}Vz^q~hM$%$|Ce
zu~T2BcDbxlSy*IK16EaO)$|_oRtZx2Ui;Mba+>>`|7tylLNxTvOSKp&;(lJsk{|3C
zx^kPSdn+`d+Zj9fK&ERfbPW-`$Lwg(p(S=Rk8;PBj}I3MU#2n!|=3
zH#o6lH6PEP%p<1_ye<@j~z_v(-fk(2QgkPP}NNGIaHcV^p#sUjzniUud=SIA?U@N_^|HAeaiwXjy!fmxD6fA2;@jVOvr&t*5m|wK(?JYvU9$+X_UnfxLFijS
zRD2y%2~Na0LDFQBvpOlwi$BDljXmInUa(`2;Lcj<8Y~2H`DiwTjuk{n=;(vv1kowR
zh@zcqVY4XylV5?KAX0*G54sj|<`{$X0jpo)^ZXp0lhh#uZ`Sz9k50
zWAS;Vd=U%!1QKP_Cza}d*fe$syB_ODE1rIRSqjt74R)fp-aN5~6TaPaX2$G9N5094
zJty$((IhoQ#JVTJu+4gky&%Jjj99lAB$OWcNe4DdP1H|AMK1$yn001BWNkl6NtHHcD^Wb*sj{!wu<}9*d%Ipyh}mT$nbIx6d{uK#Z($S5I&%ssW>>
zS4vsCsDE{unxLR4T>R5WrTEvcTW_lWbc9CTE2ph_RO037LgW>>$jF^?P2XU|KU0`G
z`VuXkuZiQM8~k=E3fHt}sBeD5lsRYdZT=Jqr%&SAu8Sz0w8cw;SU&ygTqf=Mka_Qa
z!mHEY%vtAVf;H6#Ok%)_xr`t9KRn8pBPlu>t2!-dQsEzLtP3&flqfW)Vg=ct>J
zRmvKRb9;W|a#D&e&Nr-Uw5OHxc($%CMPjpeXq=DTaxCnt)9QsDnL?cG%9g?Mehksl
zc1mxCNq4y$XN|o>sk|Gg3LDClvZ8dr2v$yNriav}wNMKM2Cd0#NY$8vu$#g_qas!q
z%vSCWq*n7e_WXER=zl{^i|L0t^b(iwqlSkc&3bp@=$-{cMl0!D%P6@sux-4N;jo$bP*XUatGM=%NE!n_!^-Vg?ywmJVI*ne%Fy3n=AWd(`rS#rTpnP4hJ*|bTer9Zu=&E@$_YYZ(Y=ObRL%(D!e
zsKQ-)+TFSlL#EWiIpwXbsoVcudi{JAO|lZ-&f^(jo#ryIu1o)Sd-3yCt1O9TuPU%LoI+ZABHRl!T;6u!ve1HTOXRIH@kX&sLQ=1Co-mVjGy?v4CACKmx
z-5xk~HNl7GGWkFPF3#3Gh)Jf}ps6%1mHTrGgLq*KJP^vm
zO6U0==x%xRh`?Tc>OKi&8Q#K_$zi|E9
zs&Z4B`|M`p=mC7+U;s-dHKkkY9n70KfWO>?o^@+Rol2IZJWUaTg!6(m-|du
z97*>_N-m5`HT^C9%dxcb1Kye!N#kX%yy8<_#8OEG?^xCi>%&qjH8FPunSWyjeY70l
z%8Ko1>${Cr_j>bA-J&ejBra_nPETRvDy;4CuG5AIAHGQ=yS!{GVMUeZE%5#3bB4Y6
zEe&7!gjwC~wa%*vIxvF?`-LuYu^}!-L(SJ_(Y&+)#g9A6s)gTUH+(HSyL(}!31REV
zmr1F9g+jF3#FW|UR3!80QNe?o6?SDQV|ibUVbX|w?6n~#CJq~W2TI9KlD4A4qA=6D
zX=pT5TCy1s>V)65gDpgJ8NWh&(!TZ*f3z9^H-&glRS-l?mAbc~jj*Br5v1lyXzCus
zRoyc5>)iDjk1Hd{=8-*x7CV0s@^$Y9AI}pq|0QG)f_1MWN>?gZVbXm00c;Re=4At6
zqHL#5BeNYXu75=k;SfP8UlpIy3!!B(zd{z65K%U5QmH6|SJm3+&%xkCMStDv9jQ9^xkdffqzBag9<^Yt58n&n1g-hBjtRkn&~1dTxztWyY}p^eRV3Q
z{aMPZ4lTU2{qc5q`1@&nSK?H$X4-6T86KZB5l0X(HM&Q_Ia+T4fT8%HzX`vweHGKJ2a-s0_x)A@MxOYHV=L=$rlyB6=T
ztYKbgOu>*HsVaWRL@VQ2Uw;6BVTm}F_r=Ld#SnGT#`sv*AI19Zy}1*1AB%F{__&pN
z9F^eRW;|=3d!1XiB2d}6Qr^=E+jJ*XwV~)XF{|1m+4T5WPWM|=f7E)m_r62eeGAHY
z``}ip=wq1#&!XQq&_@_GKEfdNX$p1HMyzZ-}-bI*pz5XTC1idHCvSB^&|IrJyNY@z$s`w+Kr`Ap^O?IMcHM}^mH#`VyS$J*H-W56{Ri8
z_z-RcJ)n%YA0B0N+lPurDod)h5jI6pH66>Rcw^Hqypi4M8Y9^JQ|4n>zHlb
z5v=*TBiF;gy^>3c^P%pT?WAAQaO@U#^}S8#JqzrG-gPgX_A*(XZNsyZ_Hrnnub@eY
zB{tp)cc;`Sf|xVkaPXp_!!o|s${TOjXny-xNS@%wY(;8j7xo8?~yOQGNY{$A^V5Ms(|@zFX|`DH0jBLW%!tYxzDa`z0kyV89TTZ0keFjXT6D
zoG4$xolN$Jt^u4vENrPgbS0bH-5|ok6~B~&-n1FNmgio-#jOYxcCO{|bh1sGflyLy
z=z0vX#4r>2$(B@=43uR($*5*)3Rungf>l`${KQ8utex=nFT}!$Z114Ti0r=MB}FmC%}I-I+{NRhV|Bfj_%-3T|OcN@m9eU8^}
z-6BF|hiiFHoYFlVw(XenUAwd^>*q+Mixmnb-FuYgI4O)mpB&jOou}BiL|y;#uNXl`mPb=
z=)_li8`^+ZJCrX`otYQdI8~&2wrb3Za-NmMr)Yj#;n~%1fMA@qi?f9@H*vmw%kfMpFw{{BhpfYO>N>nms(a28Y^;K_plso%Vwcj4~__QHTm
z#?9AS7;wpH2Al;;x0Gk>?IdBqDe?EU%rM|acxxGOHcVXUn=#XIy$D_^uZTa){5Mu{
zJjRx1-kwjZVoK0svQW74{QFbt~}c@&Q|)4dh0c2Iq>E
zaJ5n~TBAsFDB4nY#2UWtc%3lZspO)-(k15^RE0|uhJW%6{eo|B`>qAfzB;qamNgim
zJH4AHSm0D18W(l4nY=)}PStz_FxfVQLm#GQX7JG69;291_j#Bs-fz
z|4Ep#r`)(;)(828`eq!%-cO|08_o>(&wmK2WjQPAJPz@-&@$kZRH&rPqXDPkhjo!G
zJEbBbxkQ3_Mbl|Sze%3-aj#58Cv(fm8!Rd3<)5*XNIQyv)S1+3kG&+;IyuvAaH?26
zpWC9z->X8+eDA0GWI`bdzEK{MZ59VyeZ|KR4-#E(t{{|=f+^dU@Sb_L{_x%ap`8YT
zXGOz4+e|9WFvBRJpsY{zJSAB%!wh4Esd~zd4`zLkAJ{f>}b8M~~y(3V%BD7tB^qI|X)W|(1oP}p11$le(Ap#`oLYz#Bg
zVTKv`f(dNPrZ=OlG0Sag@)iYheqV!FTi)LKQkR-xhUr1KEqi(vQ8oYrL0djoS63`8
zO(Xa~rbqYCnnG}QFPeoDE-o%8m08nL*x4C!S9QZd5@21&hJ+!~$rlx}$Y$ePGJ+HY
zHkpGMvnzj7gKXdW+KjEOt)8n*3$l(f98$*HFA54%s3U!SMAnI^IPkQ~*c8e;H|Tu_
zyDG&mNPeMDMmuAe%gY9KPh~x3E5;BDV{l*<0~Do1=g&Dt+aMZ18nacaR>j)d9ItPN
zX+jQ&{{H@1NTj{JJvD07Fgin14&FXKKKS`%Z0#ZmWxaa!C|fqK*-J_W(%`LCs}^O-
zWNg1J$-J+xuR)CxNhqaW(g#^~#z@r4moKl^F`!*m#spc0rAq0n6?5xj8pP5<7-HU|
zvmCBiv10MdVqjFLpnt`mZ1<)PsVg2kicG<9&I`2OL+Suy)J^G-vPl_rA?w7;%lH=s
z%5gz^JeRuF!J&v^0gC~$KJ=EmuS_jzyzK~UVQYElNl{-)5I5JCieH}GptKjtXH?eX
zRxaSBoA>$&MftI!9CEfrERqEb^?UMBNl8f<@R4SiQRI+HSq_Lf5gKD_+md+XB(IcJ
zqtO(jM0tZOYbkqnMw66T%3fH=vdL}MHYO&9n>TOr;K73uuUF~RkcL5a2IZqikNES?
zKT)ZQC=Yl6A<2ujW8*O%1;X`7aQ8>ic?w^jhK1-iWoirIXM#g>y;p>WhH~f5os@Dn
zCCC{INgQ*U`40~d=hm%TCiiZ#k4UFq7KhN)w;^(O$q80iSbIP%A-{FTj^t7|1qKGD
z=q6)A>C&ZBoGy-zj@e!#`%YL`7>S9A#VA?skYl`@0FioDckEQF)hYcwDk{n-$Ia{@
z9eQ%iC|53<{nI1&!mYUwe?@#T4%-C_F)ll_go<;-3uNOI(j>smSrE2GRL;F(6)y`c
zlu))Y_z8K-)YX4YvKnqL6pjCLNQ^38`HBNpLN|F06Qp`fHX+LyFes@s!wfUbFr#Q7
zNx<#fx4D1+eo2dxLf9p7adUGs>RA;t{sWLsi?A;t`k?qbp`@kB2L!PxT@yUt$Tg(Q
z3^U9y!xUg9m1dY>h8bp|(s$QayNekQuf@7BqR_U8*7mDy~PO`mMryHc1iQ(+A%p!%l9)S
z$Te_s-B^~SP-Yk%WSG9}JF=}A6LM-pGGI@xUzJM>OeuIz_H)_abIczQcTS8OpMfg4
z_yo%qay|RgLjMjIMwfHWmqvdN;af|7O{J8%T;tR*bxZr~FN^&JLN^F96kTkR6&vK5
zN{`n?IW)@kQiFuQ$_3eS;lHsd5i`ul57INL>SlVCjuN?m-sz140hCde{6tyaYGrh#x&ei4!ylIas;B
zk~(ndlUEf1XP$%ji^VTnPN=*L%DeMCl*vXGFF5|AqI(AS0{BO_DF+7uz{LE0^P9pcdk;GgcIPZ-%`
zkF7sEEl8MkHVMUUiSgsc`^76$ZYW(lb4u3U
zl)=G<_oPbWK^hU6@aQI-ZYZqmNQ3`gq2W6P{AOo8eq*W_Xy+8aJh?&cTh$~eW1x+{
z0jDZsG?@l-f@@!>u{FzKQ_PI23+cF#Tl8qdlnv)}NWxsHQl-p$&uWCheEtP^@NdZp
zRv1`0L9IPtld(s#bZp3Mq{a$KU}XE!t_L~(+_-VW=x6tJ+mIaNrQD_rx%F9=jL*`v
z|Gm1<$(Kq-sNCT59#qofd{l+QNgv}Q!V-RS#TczacGVAsyyb;G5mPcU6(_8PZQM9e
zjQiO~^ciGLrKu0VW|)x<=^fUFoH;XALMb6&6I4#B?VTB$+gJLMfr!hMt7}uD5SXl4GodQX;!C
zE$&a_B$N`ddSsSSPyYo%DIr6s6K>{si9txEsfG!eVMcB+w$LM8Xv=46>zTFXEh=L~
z>ZUACf%vOM@>>RhMi>VPL0OL{8}ZTtZP|=jPF7T*F7;#1{%VF9X8gxs>_&vK%V&leMuaDo6qsR#
z8D^Lm%%su`Gt4lfsPPCcT!9GVj}Ci`;6S)sG*QV#22+&T3^V@A(5O`8s1`LURJrNR
z_%NDMO}4z`7vB4oHFhNtKXZ$H&@MSv`CN_qauF~
zi3&&+7sn0~9TWMlZM_<2TK8uwex
zl^n`7>B^V9{qhXHjWWW1T9Sk}QRu{lsk(3Mf#Uhe^E;v0V$fBY56{ADLXLqS!sgq>
zBTJr998TZ=NL`o|sZ^bsN0;Vp>DK)PUg+MHj@@2i%GR^QrWDvc4$K_M=l>cTeM#cd
z7h{<5ZEWtB7WS7eeZiO+-=h0Vny|fW+#RBS;{;6@d%oC9SYdQ=(j~qaGnKu16uKtr
zYeu*1#LT0{RxkNS^8L#k{q7g8#_5Dxscv%WyCeJ)X(R)h`VRXSjODGj-eSy{vAjEJ
z4xfDUC*dZ^dGlseIS1hjyIR?&-X7jD03Dr%yxR%#Dzss(Y9D|Z$xqS2hf8BGzuVQ%|Re$s4
z`2LJocRBl+63bfj8uuQ<+xn(#nx1rdGo!mtWZ%Qb7f`F$INlrH7GHzys$RS(QNpW+
zV@IKMLwKQD@&CLwWa7|1LJ!BChSNFawNYIvcJh^-)7PNn%M|OP05B<1Nzx+{YYk`f
z?p^HKy_elHx6Ik@i!`$!v4wSv*$ykab|!f2?>=bVh}}H?*q<
zPP$r9lXd6HB1-@)-l;(V)X!xfXbDr3SHcu!P8BASAGRMv*?TeDH+{}K&lOyMm0aKX
z4?~^Au_G!<2bAW88p_Lc8q50&K4VHpE6yG}
zMYeoO;Yi(XeQ8xGb#G`*^ffL7#*&zv@jS=6-RRq@5;oa9LLW*jVSCbLHjnPkL}Tjg
z7T{kST2#z>DoZ)((aVN{r7N@-gvu*7Rhks`0Ns`6Kf>1fO*0L{@Fg0p*`EuJfIQHIu2&ex3^N>GwI@I
z3~oP$)!U}gt7$`eEd5&^x;8MAsD(|R?fVUMMpy|s;zt$VWIw+v5!CU`re+74n>
zo@PxHxHf!=5hF)4Zq`b^n9_o{16w&5k$X929oY3BD;^ZgsK_tQZGo!OV8usJz9ZPZ
z0B;_F-*-Vh@wZJ|s687lq$S%uy$?FS0p4AOVjcuzj%2J`II|zxydsEH7pN$%*`p`^
z>%)O%6FKZ2IG&+?bz89G>?Pf#!0P7+#;EWpT
zXhOnNcvN``M@2ScpXwYd`nPAozPp5N9>TLjzar?-QKmjymxjZ)6D{neb1VAOe!@Od
z%%Zq|`QMDebZXv+hR-}t&pAJmVJ;;TadbX|TGgj^&APN0w1i)yi@8B-qPcbj=
z=ZbsF+%x6OoHKXI#0lu(@-OC&+62J{PkMR|Tjs68-{tyXNUQQNQw}){ih^6**Ovemp_7Kown%I
zsxqY4=cE6a{dkdWMrHG?#2&V>zhd3NfcFlpD&z<@?T>_&y%b4Lc4Fkf<#=RS2W>mG
zLS^Z7%mvtoj!I{wZO|A7LwZ7+3i@
zqDBAjFt}Z5g#I)Z<5zCSnaAA0kzc{e#s8w*kcsfF6oIQ(12fFwjCFZG))oRv9+kp#l?3wTu
zm6nEm4}7&Y_8OT&F0ICQ9UG!n%{pk(V={d1rdsz3_-^t>vr!wdXGSN~t
z`@pC`EFSHj1@D&iP_ssDH0(GJ=d^6=m|Ns=z>Dj^jq<>Q!@!|Nz_z7;_g}!qUw}|v
znH%zo`iJZ4WWtNLykQ>79U>p#H{KZRT*3eHs4M)q6T=%=25L9#hcD)C1M4)kOWa1F`VfGj*LLbaIVs))`B0rrR3m=~nb_HW=Gq@K!+?
zf=jEvL&t`-P_s@G^qA~}yE-P=WsY+`D0PO2CSlQtKy?1;9R?2wfNA|#u>Ce1`(DVg
zZe|c{8@}R3T@ZR6PD(MR<@jky7;1bI0IQ|}C^v%N8<^4t#mTW{aVRRZ@rOn80Mu9(
zpV^q^`EzYsD9U=j#jc=&YP%TT{reNfbo4~c>b20g$8`MqSW~Zms`8T=9`1#8m^FPe
zhIj7=AH!i-Jfa5FHo4BWFBFmSBG`Vj3a3|hgq=ZMOh0o47nZe$rTVeA2#t*AvEyyZ
zwkeo*%LU_o&fx69wHWE?kfz&;+ct~?
z;qYdmW9n#WbH+b{=t$t$S)dkoFv9opc8C}F!J~mwZoHm73Y_N-;_sXJJ@J}th=WXk
zCG&u%ypHa}MIQc>CY(F`*^zgFg+gZ^zkhaZKrj7^coJ=10Ic9)Eawhv$-Fik1?>pq
z<2!wGL^2S@+qhkicslX8y_^puo&XyDz~vgp<>JofP_8&rd^ES~%nP1AOBVxI9D&Qc
zJ^Y7XZ)y9%yYee9a=;hoR3`ts0p6WEqccC+cbkDnb!TBhV^hsV77M+vCt|jDBUCKu
z3is|K&_?_RF8OQR9+g{1jQYkMt~N6DytDm3?uYf5jhNHB5vqMv2_7BCVbx@JC?f@B
zS0O1`2?L8_C|SKD#*J$aCyC(X{e~ZXOfYoyH1ujx8#O&zVc>%G7$Vz&)u*yvb};*A
z^-;B3Td>3Xa9^z*3OwF_5V0+Kz+FECzy0U~lcB3ItydG&uIYhR0~cWZ5E*u?KBZML
zQc?^7K?*q3?gekR%h+=KwWe1`^%f_$UO-9jKJc)YBQQXCA+I3!|A_t8W3X{fZ#1g*
z6)Jh~xUZTFcV%SCb4f^68o;7hNmS?QHLkr=N>9EDkyn>u(1i1-Fk>@jw<>|mojVM%
zvikUbjyD=sbb)Kl?ikTlimR9Xg~^vVw)7{c`me=;Z=0aTR~6yjatMByTo1CS!qj~p
zkI={WaQE)rwBtb_GEB1++oJi1ap>e;9C|U~P&l~2Htsq6f>RnIkvSR;8if{>%fijk
zEaf2)%o`5Es1}t`*3A)SQaiNhP#1}pPT_{|6TW}fggEMCjV(myz
zIOJ0WDRBZOatF~&A85k=mgXwarWbE~c@|l5N2nFAwEcvAe|a=IdjtI{^F}n5%Tx?-
z;1+dW8VN+&=PRrX%)t+
zLpN*g+`h6Zy-KS-(D}9o~h(6%6pz(7m{HVHw(+XV+15
z0sAl8W6b_*`0MaQl=It!rF&oLG?Z1bfe15cj*06I;Pmk|Xs3~#SsYwD5}pn+*m>54)2rh+dLzZu;!atwzXoA5(gC$}S{y67H@YiV?fx
z(0G;)PM^Y7fzXgr2qQH~mzvEx)TJS8P#JzPRFzHkg
z^j>`wXAaCk?N_TXZuwPY8W^ZG&iQTiFhoZvaO|`kwI(|t{GbgtreiU9_*=q3U0s|VaXypJl$QC8*p(Lz9C*6&Yj)i7@B)#=sfXlH=B)6d|FP;XeK+@#at&05DaFVN~+Q)-CGdwG%bl`O`Px9JcysqWN#P=;vQht5rK)}^>*O_^Jm
zEDS{B^|!D%l%)-?yWIif)EDr^`XqPQCWit$xRX=Q^yB0$Fh>~q)9IUIxbfVhJMa~k
z*S^|
zShe$S#0VC!xELA^8-&zOgY2G7YUKf$O9ymu?u%WQ2BM*B|7D!?7RTRG%#IGzg64^G2?Yf{99Jce(iBYH0W4%Ke2!1ph!
zV%Yj`;bAMJv=zEc8-hzs7UR&>@$hK!VZ$zjCDU?Fj%$1O!M|cZOerTtcxVhPzOE0q
ztp{=GVG=xQXmU|x-VlREwLso~Xv7_%WT7hI75IUTz*Jr=RorNB=JyEeV<`D~X&GPzcanxJ=jVNZZ+imM
zc>``Bgu$O(w;nKU1k9`T@jZGumY4JVA~1+ItPgp;RFXTviQMs0s&vb1k`m0FL%<(A
zzmN3?Z1bTD^2?A}3PTeE$iyOu^^9Q33%rDzu0lr^qbX(m~)Knd@l4$px
zV9;_6w(fv-wN_%r)8|}uT(#(q(C^e7n&2W3^hyB-7hA;L7HU{WUc{A1ijRjvW{wi&
z8$eUSTcAVTRakQB2IA{Cfd0$lIPtgw+Aehc_(oip?D9bD-Tx9*hporA9=7W8YSm>L
zhFofb#fPq9yoVQ>x2}m*>rUa;_a3M#dVu4nLr~MZH5?V!Fk_!TD)yU#a?)^w##rF%
z`fk{I@DgvUJWwNp0LePyoZpC0zNr~+pM=~nM4zvv*nG(lQ#$LziW_6K8ymrGPXc`3
zFm!M-#>}BSe-f48J_MYdB#2e_^crykj&Wm}J040AEQ{--RZ0V&`t9oN^>B1tr$DD!
zwwPI^0OxaxY|(PeB($p}RJIQ2$8DD5-<-jDzY(ZZ{u7p;3~&dw?zfmQxTLzBin%lm
zZAYxY-q!BuFI?8z_T>~QD)gR$p?ITUz&PH9CPW0`mBIlowsE-a7tAg4te%s>h}Ro>Tm-4i7)4T)T`j?ThgZ-s
z!QowwS%SkW56(u%nrNJ!_s)3!G^FN5(%K)P&Xw@rHmUWpKGY-m6ucf*@F-9G5!n3_
zD8?NuTdo4Cl=`_^N8SeV^Fw*Y{=kjT$^38klA3G6(9Q|>xMQ}6*Ibs!7g8Nzq4Wrj
z@a8{K9AUi_NBDro5$;ez<9w;Oll3w%#rb-u2_u6e%pJH^HYxI^ydyt5eRBkF^Q?-b
z{ojyhua!C8+jv6Wqzb1qFL;8C;12Ku-X2!&1~hNM?TbF#QB7%UK1E(+#TVyPLM(;8
zp}v-{U{Gs=p+Sb5Ar>pu$|2;KzCpU~Y%vc*B!^tcu=JC)RB|NBHJ_biF6}UXPi@@z
zWfrE6jKi|61K=dpgH^pT*tWn68D1s4OTv$s7iiknK{J0wWCRB@2~y|xinDoElNlCh
z(z7nMu0M=>kymlQT5n9T=GrcoLeHum#%x;vuZ$1#I7)
zsTdiz3|qbVX=!PA){4fjYkx(-dVe$v_)DtVv`amVV!`-VKF?X
zwPsvqRM7|sj6}TeE=(WyYub6oqA*PN)mCCxB3~ESkyhaPFw_sTXSPM7jrGxK@MsKc
zT_&xEv;c_ajoGj>zyj_d4zLqOAOH=$Q!esk_3FzVA>QB~;mIIbcU~Xs^fen^@lX^9IL+H?%KX^LKI;TQ(Z_ISYALSJXUOnVV6d7|y)oZB$cksYRbgU}QXecTGpe>cJ41
z27~I1^T$W+L(I*@>5OR+B)oloZ%iAAAV{Y+ri0=zabYBObGyHogA}%MRmxm05vqM-
ziK)-SF|>CeeyDAN$%D<&-8Cf;Zn3UglL%8UTg&;Ccyf{j1M{~J{AQ6
zPA8P7Whe*%1X+jV^};KJ+C
zm-~PUCsM@Zb?)@}z;|3nR&4IeT#p*ET7sD&hJj&
z{3ffsC{`Uf#MA!4RUY?MKsz~aA4h4feTuxuio%v|A5A91Jj1ng!%*~Vpm7yn?Ah}e
z>6Su4*x5Ze6_BzhPEwMv8W3vr9g2pr|KeUepjX`!6?~84RG1htnM`va5Wh|sj?GUL
zyzX#CHTPPmRf`|)6;a$M?UmU*n~JP*ceF{~fw4bDp?ME?jhU}k9iA0@ar9Ie#Hr76
zi!|`p2^hXv_mbVJr~rh>9N?&KJIv6qUvs=(KMbq<8>4??OLYOzb8vuBh(E$r`l!*k
z3ViqM!Q*r*#j3*2V$Z37ly@XSlPtB+cW6Vz{(Da|oJnK}k0~25+4%^@4&R8sbq!h7
zl|iyfXyp}+or@0RdFs2ljwyqI4(4E&vvzsgYH20tV`quO@@Jo(>{@U&+wc;){acPy{3
z![](\"./figures/paddle-ernie-1.0-masking.png\")
\n",
+ "\n",
+ "![](\"./figures/paddle-ernie-1.0-masking-levels.png\")
\n",
+ "\n",
+ "``ERNIE 2.0`` 则提出了连续学习（``Continual Learning``）的概念，即首先用一个简单的任务来初始化模型，在更新时用前一个任务训练好的参数作为下一个任务模型初始化的参数。这样在训练新的任务时，模型便可以记住之前学习到的知识，使得模型在新任务上获得更好的表现。``ERNIE 2.0`` 分别构建了词法、语法、语义不同级别的预训练任务，并使用不同的 task id 来标示不同的任务，在共计16个中英文任务上都取得了SOTA效果。\n",
+ "\n",
+ "![](\"./figures/paddle-ernie-2.0-continual-pretrain.png\")
\n",
+ "\n",
+ "``ERNIE 3.0`` 将自回归和自编码网络融合在一起进行预训练，其中自编码网络采用 ``ERNIE 2.0`` 的多任务学习增量式构建预训练任务，持续进行语义理解学习。其中自编码网络增加了知识增强的预训练任务。自回归网络则基于 ``Tranformer-XL`` 结构，支持长文本语言模型建模，并在多个自然语言处理任务中取得了SOTA的效果。\n",
+ "\n",
+ "![](\"./figures/paddle-ernie-3.0-framework.png\")
\n",
+ "\n",
+ "接下来，我们将展示如何在 ``FastNLP`` 中使用基于 ``paddle`` 的 ``ERNIE 1.0`` 框架进行中文情感分析。"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 2. 使用 tokenizer 处理数据并构造 dataloader\n",
+ "\n",
+ "#### 2.1 加载中文数据集 ChnSentiCorp\n",
+ "\n",
+ "``ChnSentiCorp`` 数据集是由中国科学院发布的中文句子级情感分析数据集，包含了从网络上获取的酒店、电影、书籍等多个领域的评论，每条评论都被划分为两个标签：消极（``0``）和积极（``1``），可以用于二分类的中文情感分析任务。通过 ``paddlenlp.datasets.load_dataset`` 函数，我们可以加载并查看 ``ChnSentiCorp`` 数据集的内容。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "训练集大小： 9600\n",
+ "{'text': '选择珠江花园的原因就是方便，有电动扶梯直接到达海边，周围餐馆、食廊、商场、超市、摊位一应俱全。酒店装修一般，但还算整洁。 泳池在大堂的屋顶，因此很小，不过女儿倒是喜欢。 包的早餐是西式的，还算丰富。 服务吗，一般', 'label': 1, 'qid': ''}\n",
+ "{'text': '15.4寸笔记本的键盘确实爽，基本跟台式机差不多了，蛮喜欢数字小键盘，输数字特方便，样子也很美观，做工也相当不错', 'label': 1, 'qid': ''}\n",
+ "{'text': '房间太小。其他的都一般。。。。。。。。。', 'label': 0, 'qid': ''}\n"
+ ]
+ }
+ ],
+ "source": [
+ "from paddlenlp.datasets import load_dataset\n",
+ "\n",
+ "train_dataset, val_dataset, test_dataset = load_dataset(\"chnsenticorp\", splits=[\"train\", \"dev\", \"test\"])\n",
+ "print(\"训练集大小：\", len(train_dataset))\n",
+ "for i in range(3):\n",
+ " print(train_dataset[i])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### 2.2 处理数据\n",
+ "\n",
+ "可以看到，原本的数据集仅包含中文的文本和标签，这样的数据是无法被模型识别的。同英文文本分类任务一样，我们需要使用 ``tokenizer`` 对文本进行分词并转换为数字形式的结果。我们可以加载已经预训练好的中文分词模型 ``ernie-1.0-base-zh``，将分词的过程写在函数 ``_process`` 中，然后调用数据集的 ``map`` 函数对每一条数据进行分词。其中：\n",
+ "- 参数 ``max_length`` 代表句子的最大长度；\n",
+ "- ``padding=\"max_length\"`` 表示将长度不足的结果 padding 至和最大长度相同；\n",
+ "- ``truncation=True`` 表示将长度过长的句子进行截断。\n",
+ "\n",
+ "至此，我们得到了每条数据长度均相同的数据集。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "\u001b[32m[2022-06-22 21:31:04,168] [ INFO]\u001b[0m - We are using