{ "cells": [ { "cell_type": "markdown", "id": "fdd7ff16", "metadata": {}, "source": [ "# T5. fastNLP 中的预定义模型\n", "\n", "  1   fastNLP 中 modules 的介绍\n", " \n", "    1.1   modules 模块、models 模块 简介\n", "\n", "    1.2   示例一：modules 实现 LSTM 分类\n", "\n", "  2   fastNLP 中 models 的介绍\n", " \n", "    2.1   示例一：models 实现 CNN 分类\n", "\n", "    2.3   示例二：models 实现 BiLSTM 标注" ] }, { "cell_type": "markdown", "id": "d3d65d53", "metadata": {}, "source": [ "## 1. fastNLP 中 modules 模块的介绍\n", "\n", "### 1.1 modules 模块、models 模块 简介\n", "\n", "在`fastNLP 0.8`中，**`modules.torch`路径下定义了一些基于`pytorch`实现的基础模块**\n", "\n", "    包括长短期记忆网络`LSTM`、条件随机场`CRF`、`transformer`的编解码器模块等，详见下表\n", "\n", "|
代码名称
|
简要介绍
|
代码路径
|\n", "|:--|:--|:--|\n", "| `LSTM` | 轻量封装`pytorch`的`LSTM` | `/modules/torch/encoder/lstm.py` |\n", "| `Seq2SeqEncoder` | 序列变换编码器，基类 | `/modules/torch/encoder/seq2seq_encoder.py` |\n", "| `LSTMSeq2SeqEncoder` | 序列变换编码器，基于`LSTM` | `/modules/torch/encoder/seq2seq_encoder.py` |\n", "| `TransformerSeq2SeqEncoder` | 序列变换编码器，基于`transformer` | `/modules/torch/encoder/seq2seq_encoder.py` |\n", "| `StarTransformer` | `Star-Transformer`的编码器部分 | `/modules/torch/encoder/star_transformer.py` |\n", "| `VarRNN` | 实现`Variational Dropout RNN` | `/modules/torch/encoder/variational_rnn.py` |\n", "| `VarLSTM` | 实现`Variational Dropout LSTM` | `/modules/torch/encoder/variational_rnn.py` |\n", "| `VarGRU` | 实现`Variational Dropout GRU` | `/modules/torch/encoder/variational_rnn.py` |\n", "| `ConditionalRandomField` | 条件随机场模型 | `/modules/torch/decoder/crf.py` |\n", "| `Seq2SeqDecoder` | 序列变换解码器，基类 | `/modules/torch/decoder/seq2seq_decoder.py` |\n", "| `LSTMSeq2SeqDecoder` | 序列变换解码器，基于`LSTM` | `/modules/torch/decoder/seq2seq_decoder.py` |\n", "| `TransformerSeq2SeqDecoder` | 序列变换解码器，基于`transformer` | `/modules/torch/decoder/seq2seq_decoder.py` |\n", "| `SequenceGenerator` | 序列生成，封装`Seq2SeqDecoder` | `/models/torch/sequence_labeling.py` |\n", "| `TimestepDropout` | 在每个`timestamp`上`dropout` | `/modules/torch/dropout.py` |" ] }, { "cell_type": "markdown", "id": "89ffcf07", "metadata": {}, "source": [ "  **`models.torch`路径下定义了一些基于`pytorch`、`modules`实现的预定义模型** \n", "\n", "    例如基于`CNN`的分类模型、基于`BiLSTM+CRF`的标注模型、基于[双仿射注意力机制](https://arxiv.org/pdf/1611.01734.pdf)的分析模型\n", "\n", "    基于`modules.torch`中的`LSTM`/`transformer`编/解码器模块的序列变换/生成模型，详见下表\n", "\n", "|
代码名称
|
简要介绍
|
代码路径
|\n", "|:--|:--|:--|\n", "| `BiaffineParser` | 句法分析模型，基于双仿射注意力 | `/models/torch/biaffine_parser.py` |\n", "| `CNNText` | 文本分类模型，基于`CNN` | `/models/torch/cnn_text_classification.py` |\n", "| `Seq2SeqModel` | 序列变换，基类`encoder+decoder` | `/models/torch/seq2seq_model.py` |\n", "| `LSTMSeq2SeqModel` | 序列变换，基于`LSTM` | `/models/torch/seq2seq_model.py` |\n", "| `TransformerSeq2SeqModel` | 序列变换，基于`transformer` | `/models/torch/seq2seq_model.py` |\n", "| `SequenceGeneratorModel` | 封装`Seq2SeqModel`，结合`SequenceGenerator` | `/models/torch/seq2seq_generator.py` |\n", "| `SeqLabeling` | 标注模型，基类`LSTM+FC+CRF` | `/models/torch/sequence_labeling.py` |\n", "| `BiLSTMCRF` | 标注模型，`BiLSTM+FC+CRF` | `/models/torch/sequence_labeling.py` |\n", "| `AdvSeqLabel` | 标注模型，`LN+BiLSTM*2+LN+FC+CRF` | `/models/torch/sequence_labeling.py` |" ] }, { "cell_type": "markdown", "id": "61318354", "metadata": {}, "source": [ "上述`fastNLP`模块，不仅**为入门级用户提供了简单易用的工具**，以解决各种`NLP`任务，或复现相关论文\n", "\n", "  同时**也为专业研究人员提供了便捷可操作的接口**，封装部分代码的同时，也能指定参数修改细节\n", "\n", "  在接下来的`tutorial`中，我们将通过`SST-2`分类和`CoNLL-2003`标注，展示相关模型使用\n", "\n", "注一：**`SST`**，**单句情感分类**数据集，包含电影评论和对应情感极性，1 对应正面情感，0 对应负面情感\n", "\n", "  数据集包括三部分：训练集 67350 条，验证集 873 条，测试集 1821 条，更多参考[下载链接](https://gluebenchmark.com/tasks)\n", "\n", "注二：**`CoNLL-2003`**，**文本语法标注**数据集，包含语句和对应的词性标签`pos_tags`（名动形数量代）\n", "\n", "  语法结构标签`chunk_tags`（主谓宾定状补）、命名实体标签`ner_tags`（人名、组织名、地名、时间等）\n", "\n", "  数据集包括三部分：训练集 14041 条，验证集 3250 条，测试集 3453 条，更多参考[原始论文](https://aclanthology.org/W03-0419.pdf)" ] }, { "cell_type": "markdown", "id": "2a36bbe4", "metadata": {}, "source": [ "### 1.2 示例一：modules 实现 LSTM 分类" ] }, { "cell_type": "code", "execution_count": 1, "id": "40e66b21", "metadata": {}, "outputs": [], "source": [ "# import sys\n", "# sys.path.append('..')\n", "\n", "# from fastNLP.io import SST2Pipe # 没有 SST2Pipe 会运行很长时间，并且还会报错\n", "\n", "# databundle = SST2Pipe(tokenizer='raw').process_from_file()\n", "\n", "# dataset = databundle.get_dataset('train')[:6000]\n", "\n", "# dataset.apply_more(lambda ins:{'words': ins['sentence'].lower().split(), 'target': ins['label']}, \n", "# progress_bar=\"tqdm\")\n", "# dataset.delete_field('sentence')\n", "# dataset.delete_field('label')\n", "# dataset.delete_field('idx')\n", "\n", "# from fastNLP import Vocabulary\n", "\n", "# vocab = Vocabulary()\n", "# vocab.from_dataset(dataset, field_name='words')\n", "# vocab.index_dataset(dataset, field_name='words')\n", "\n", "# train_dataset, evaluate_dataset = dataset.split(ratio=0.85)" ] }, { "cell_type": "code", "execution_count": 2, "id": "50960476", "metadata": {}, "outputs": [], "source": [ "# from fastNLP import prepare_torch_dataloader\n", "\n", "# train_dataloader = prepare_torch_dataloader(train_dataset, batch_size=16, shuffle=True)\n", "# evaluate_dataloader = prepare_torch_dataloader(evaluate_dataset, batch_size=16)" ] }, { "cell_type": "code", "execution_count": 3, "id": "0b25b25c", "metadata": {}, "outputs": [], "source": [ "# import torch\n", "# import torch.nn as nn\n", "\n", "# from fastNLP.modules.torch import LSTM, MLP # 没有 MLP\n", "# from fastNLP import Embedding, CrossEntropyLoss\n", "\n", "\n", "# class ClsByModules(nn.Module):\n", "# def __init__(self, vocab_size, embedding_dim, output_dim, hidden_dim=64, num_layers=2, dropout=0.5):\n", "# nn.Module.__init__(self)\n", "\n", "# self.embedding = Embedding((vocab_size, embedding_dim))\n", "# self.lstm = LSTM(embedding_dim, hidden_dim, num_layers=num_layers, bidirectional=True)\n", "# self.mlp = MLP([hidden_dim * 2, output_dim], dropout=dropout)\n", " \n", "# self.loss_fn = CrossEntropyLoss()\n", "\n", "# def forward(self, words):\n", "# output = self.embedding(words)\n", "# output, (hidden, cell) = self.lstm(output)\n", "# output = self.mlp(torch.cat((hidden[-1], hidden[-2]), dim=1))\n", "# return output\n", " \n", "# def train_step(self, words, target):\n", "# pred = self(words)\n", "# return {\"loss\": self.loss_fn(pred, target)}\n", "\n", "# def evaluate_step(self, words, target):\n", "# pred = self(words)\n", "# pred = torch.max(pred, dim=-1)[1]\n", "# return {\"pred\": pred, \"target\": target}" ] }, { "cell_type": "code", "execution_count": 4, "id": "9dbbf50d", "metadata": {}, "outputs": [], "source": [ "# model = ClsByModules(vocab_size=len(vocabulary), embedding_dim=100, output_dim=2)\n", "\n", "# from torch.optim import AdamW\n", "\n", "# optimizers = AdamW(params=model.parameters(), lr=5e-5)" ] }, { "cell_type": "code", "execution_count": 5, "id": "7a93432f", "metadata": {}, "outputs": [], "source": [ "# from fastNLP import Trainer, Accuracy\n", "\n", "# trainer = Trainer(\n", "# model=model,\n", "# driver='torch',\n", "# device=0, # 'cuda'\n", "# n_epochs=10,\n", "# optimizers=optimizers,\n", "# train_dataloader=train_dataloader,\n", "# evaluate_dataloaders=evaluate_dataloader,\n", "# metrics={'acc': Accuracy()}\n", "# )" ] }, { "cell_type": "code", "execution_count": 6, "id": "31102e0f", "metadata": {}, "outputs": [], "source": [ "# trainer.run(num_eval_batch_per_dl=10)" ] }, { "cell_type": "code", "execution_count": 7, "id": "8bc4bfb2", "metadata": {}, "outputs": [], "source": [ "# trainer.evaluator.run()" ] }, { "cell_type": "markdown", "id": "d9443213", "metadata": {}, "source": [ "## 2. fastNLP 中 models 模块的介绍\n", "\n", "### 2.1 示例一：models 实现 CNN 分类\n", "\n", "  本示例使用`fastNLP 0.8`中预定义模型`models`中的`CNNText`模型，实现`SST-2`文本二分类任务\n", "\n", "模型使用方面，如上所述，这里使用**基于卷积神经网络`CNN`的预定义文本分类模型`CNNText`**，结构如下所示\n", "\n", "  首先是内置的`100`维嵌入层、`dropout`层、紧接着是三个一维卷积，将`100`维嵌入特征，分别通过\n", "\n", "    **感受野为`1`、`3`、`5`的卷积算子变换至`30`维、`40`维、`50`维的卷积特征**，再将三者拼接\n", "\n", "  最终再次通过`dropout`层、线性变换层，映射至二元的输出值，对应两个分类结果上的几率`logits`\n", "\n", "```\n", "CNNText(\n", " (embed): Embedding(\n", " (embed): Embedding(5194, 100)\n", " (dropout): Dropout(p=0.0, inplace=False)\n", " )\n", " (conv_pool): ConvMaxpool(\n", " (convs): ModuleList(\n", " (0): Conv1d(100, 30, kernel_size=(1,), stride=(1,), bias=False)\n", " (1): Conv1d(100, 40, kernel_size=(3,), stride=(1,), padding=(1,), bias=False)\n", " (2): Conv1d(100, 50, kernel_size=(5,), stride=(1,), padding=(2,), bias=False)\n", " )\n", " )\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " (fc): Linear(in_features=120, out_features=2, bias=True)\n", ")\n", "```\n", "\n", "数据使用方面，此处**使用`datasets`模块中的`load_dataset`函数**，以如下形式，指定`SST-2`数据集自动加载\n", "\n", "  首次下载后会保存至`~/.cache/huggingface/modules/datasets_modules/datasets/glue/`目录下" ] }, { "cell_type": "code", "execution_count": 8, "id": "1aa5cf6d", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Using the latest cached version of the module from /remote-home/xrliu/.cache/huggingface/modules/datasets_modules/datasets/glue/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad (last modified on Thu May 26 15:30:15 2022) since it couldn't be found locally at glue., or remotely on the Hugging Face Hub.\n", "Reusing dataset glue (/remote-home/xrliu/.cache/huggingface/datasets/glue/sst2/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad)\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "70cde65067c64fdba1d5e798e2b8d631", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/3 [00:00\n", "\n" ], "text/plain": [ "\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Processing: 0%| | 0/6000 [00:00[17:45:59] INFO Running evaluator sanity check for 2 batches. trainer.py:592\n", "\n" ], "text/plain": [ "\u001b[2;36m[17:45:59]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m Running evaluator sanity check for \u001b[1;36m2\u001b[0m batches. \u001b]8;id=147745;file://../fastNLP/core/controllers/trainer.py\u001b\\\u001b[2mtrainer.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=708408;file://../fastNLP/core/controllers/trainer.py#592\u001b\\\u001b[2m592\u001b[0m\u001b]8;;\u001b\\\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Output()" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n",
       "
\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
----------------------------- Eval. results on Epoch:1, Batch:0 -----------------------------\n",
       "
\n"
      ],
      "text/plain": [
       "----------------------------- Eval. results on Epoch:\u001b[1;36m1\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
{\n",
       "  \"acc#acc\": 0.575,\n",
       "  \"total#acc\": 160.0,\n",
       "  \"correct#acc\": 92.0\n",
       "}\n",
       "
\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "  \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.575\u001b[0m,\n",
       "  \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n",
       "  \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m92.0\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n",
       "
\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
----------------------------- Eval. results on Epoch:2, Batch:0 -----------------------------\n",
       "
\n"
      ],
      "text/plain": [
       "----------------------------- Eval. results on Epoch:\u001b[1;36m2\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
{\n",
       "  \"acc#acc\": 0.75625,\n",
       "  \"total#acc\": 160.0,\n",
       "  \"correct#acc\": 121.0\n",
       "}\n",
       "
\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "  \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.75625\u001b[0m,\n",
       "  \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n",
       "  \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m121.0\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n",
       "
\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
----------------------------- Eval. results on Epoch:3, Batch:0 -----------------------------\n",
       "
\n"
      ],
      "text/plain": [
       "----------------------------- Eval. results on Epoch:\u001b[1;36m3\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
{\n",
       "  \"acc#acc\": 0.78125,\n",
       "  \"total#acc\": 160.0,\n",
       "  \"correct#acc\": 125.0\n",
       "}\n",
       "
\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "  \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.78125\u001b[0m,\n",
       "  \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n",
       "  \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m125.0\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n",
       "
\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
----------------------------- Eval. results on Epoch:4, Batch:0 -----------------------------\n",
       "
\n"
      ],
      "text/plain": [
       "----------------------------- Eval. results on Epoch:\u001b[1;36m4\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
{\n",
       "  \"acc#acc\": 0.8,\n",
       "  \"total#acc\": 160.0,\n",
       "  \"correct#acc\": 128.0\n",
       "}\n",
       "
\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "  \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.8\u001b[0m,\n",
       "  \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n",
       "  \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m128.0\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n",
       "
\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
----------------------------- Eval. results on Epoch:5, Batch:0 -----------------------------\n",
       "
\n"
      ],
      "text/plain": [
       "----------------------------- Eval. results on Epoch:\u001b[1;36m5\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
{\n",
       "  \"acc#acc\": 0.79375,\n",
       "  \"total#acc\": 160.0,\n",
       "  \"correct#acc\": 127.0\n",
       "}\n",
       "
\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "  \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.79375\u001b[0m,\n",
       "  \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n",
       "  \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m127.0\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n",
       "
\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
----------------------------- Eval. results on Epoch:6, Batch:0 -----------------------------\n",
       "
\n"
      ],
      "text/plain": [
       "----------------------------- Eval. results on Epoch:\u001b[1;36m6\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
{\n",
       "  \"acc#acc\": 0.80625,\n",
       "  \"total#acc\": 160.0,\n",
       "  \"correct#acc\": 129.0\n",
       "}\n",
       "
\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "  \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.80625\u001b[0m,\n",
       "  \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n",
       "  \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m129.0\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n",
       "
\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
----------------------------- Eval. results on Epoch:7, Batch:0 -----------------------------\n",
       "
\n"
      ],
      "text/plain": [
       "----------------------------- Eval. results on Epoch:\u001b[1;36m7\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
{\n",
       "  \"acc#acc\": 0.81875,\n",
       "  \"total#acc\": 160.0,\n",
       "  \"correct#acc\": 131.0\n",
       "}\n",
       "
\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "  \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.81875\u001b[0m,\n",
       "  \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n",
       "  \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m131.0\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n",
       "
\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
----------------------------- Eval. results on Epoch:8, Batch:0 -----------------------------\n",
       "
\n"
      ],
      "text/plain": [
       "----------------------------- Eval. results on Epoch:\u001b[1;36m8\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
{\n",
       "  \"acc#acc\": 0.825,\n",
       "  \"total#acc\": 160.0,\n",
       "  \"correct#acc\": 132.0\n",
       "}\n",
       "
\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "  \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.825\u001b[0m,\n",
       "  \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n",
       "  \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m132.0\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n",
       "
\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
----------------------------- Eval. results on Epoch:9, Batch:0 -----------------------------\n",
       "
\n"
      ],
      "text/plain": [
       "----------------------------- Eval. results on Epoch:\u001b[1;36m9\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
{\n",
       "  \"acc#acc\": 0.81875,\n",
       "  \"total#acc\": 160.0,\n",
       "  \"correct#acc\": 131.0\n",
       "}\n",
       "
\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "  \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.81875\u001b[0m,\n",
       "  \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n",
       "  \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m131.0\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n",
       "
\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
---------------------------- Eval. results on Epoch:10, Batch:0 -----------------------------\n",
       "
\n"
      ],
      "text/plain": [
       "---------------------------- Eval. results on Epoch:\u001b[1;36m10\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
{\n",
       "  \"acc#acc\": 0.81875,\n",
       "  \"total#acc\": 160.0,\n",
       "  \"correct#acc\": 131.0\n",
       "}\n",
       "
\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "  \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.81875\u001b[0m,\n",
       "  \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n",
       "  \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m131.0\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n",
       "
\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "trainer.run(num_eval_batch_per_dl=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "f47a6a35",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "{'acc#acc': 0.79, 'total#acc': 900.0, 'correct#acc': 711.0}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.evaluator.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c811257",
   "metadata": {},
   "source": [
    "  注：此处使用`gc`模块删除相关变量，释放内存，为接下来新的模型训练预留存储空间"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "c1a2e2ca",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "342"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import gc\n",
    "\n",
    "del model\n",
    "del trainer\n",
    "del dataset\n",
    "del sst2data\n",
    "\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6aec2a19",
   "metadata": {},
   "source": [
    "### 2.2  示例二：models 实现 BiLSTM 标注\n",
    "\n",
    "  通过两个示例一的对比可以发现，得益于`models`对模型结构的封装，使用`models`明显更加便捷\n",
    "\n",
    "    针对更加复杂的模型时，编码更加轻松；本示例将使用`models`中的`BiLSTMCRF`模型\n",
    "\n",
    "  避免`CRF`和`Viterbi`算法代码书写的困难，轻松实现`CoNLL-2003`中的命名实体识别`NER`任务\n",
    "\n",
    "模型使用方面，如上所述，这里使用**基于双向`LSTM`+条件随机场`CRF`的标注模型`BiLSTMCRF`**，结构如下所示\n",
    "\n",
    "  其中，隐藏层维度默认`100`维，因此对应双向`LSTM`输出`200`维，`dropout`层退学概率、`LSTM`层数可调\n",
    "\n",
    "```\n",
    "BiLSTMCRF(\n",
    "  (embed): Embedding(7590, 100)\n",
    "  (lstm): LSTM(\n",
    "    (lstm): LSTM(100, 100, batch_first=True, bidirectional=True)\n",
    "  )\n",
    "  (dropout): Dropout(p=0.1, inplace=False)\n",
    "  (fc): Linear(in_features=200, out_features=9, bias=True)\n",
    "  (crf): ConditionalRandomField()\n",
    ")\n",
    "```\n",
    "\n",
    "数据使用方面，此处仍然**使用`datasets`模块中的`load_dataset`函数**，以如下形式，加载`CoNLL-2003`数据集\n",
    "\n",
    "  首次下载后会保存至`~.cache/huggingface/datasets/conll2003/conll2003/1.0.0/`目录下"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "03e66686",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Reusing dataset conll2003 (/remote-home/xrliu/.cache/huggingface/datasets/conll2003/conll2003/1.0.0/63f4ebd1bcb7148b1644497336fd74643d4ce70123334431a3c053b7ee4e96ee)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3ec9e0ce9a054339a2453420c2c9f28b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00[17:49:16] INFO     Running evaluator sanity check for 2 batches.              trainer.py:592\n",
       "\n"
      ],
      "text/plain": [
       "\u001b[2;36m[17:49:16]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Running evaluator sanity check for \u001b[1;36m2\u001b[0m batches.              \u001b]8;id=766109;file://../fastNLP/core/controllers/trainer.py\u001b\\\u001b[2mtrainer.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=787419;file://../fastNLP/core/controllers/trainer.py#592\u001b\\\u001b[2m592\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n",
       "
\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
----------------------------- Eval. results on Epoch:1, Batch:0 -----------------------------\n",
       "
\n"
      ],
      "text/plain": [
       "----------------------------- Eval. results on Epoch:\u001b[1;36m1\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
{\n",
       "  \"f#F1\": 0.220374,\n",
       "  \"pre#F1\": 0.25,\n",
       "  \"rec#F1\": 0.197026\n",
       "}\n",
       "
\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "  \u001b[1;34m\"f#F1\"\u001b[0m: \u001b[1;36m0.220374\u001b[0m,\n",
       "  \u001b[1;34m\"pre#F1\"\u001b[0m: \u001b[1;36m0.25\u001b[0m,\n",
       "  \u001b[1;34m\"rec#F1\"\u001b[0m: \u001b[1;36m0.197026\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n",
       "
\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
----------------------------- Eval. results on Epoch:2, Batch:0 -----------------------------\n",
       "
\n"
      ],
      "text/plain": [
       "----------------------------- Eval. results on Epoch:\u001b[1;36m2\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
{\n",
       "  \"f#F1\": 0.442857,\n",
       "  \"pre#F1\": 0.426117,\n",
       "  \"rec#F1\": 0.460967\n",
       "}\n",
       "
\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "  \u001b[1;34m\"f#F1\"\u001b[0m: \u001b[1;36m0.442857\u001b[0m,\n",
       "  \u001b[1;34m\"pre#F1\"\u001b[0m: \u001b[1;36m0.426117\u001b[0m,\n",
       "  \u001b[1;34m\"rec#F1\"\u001b[0m: \u001b[1;36m0.460967\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n",
       "
\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
----------------------------- Eval. results on Epoch:3, Batch:0 -----------------------------\n",
       "
\n"
      ],
      "text/plain": [
       "----------------------------- Eval. results on Epoch:\u001b[1;36m3\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
{\n",
       "  \"f#F1\": 0.572954,\n",
       "  \"pre#F1\": 0.549488,\n",
       "  \"rec#F1\": 0.598513\n",
       "}\n",
       "
\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "  \u001b[1;34m\"f#F1\"\u001b[0m: \u001b[1;36m0.572954\u001b[0m,\n",
       "  \u001b[1;34m\"pre#F1\"\u001b[0m: \u001b[1;36m0.549488\u001b[0m,\n",
       "  \u001b[1;34m\"rec#F1\"\u001b[0m: \u001b[1;36m0.598513\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n",
       "
\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
----------------------------- Eval. results on Epoch:4, Batch:0 -----------------------------\n",
       "
\n"
      ],
      "text/plain": [
       "----------------------------- Eval. results on Epoch:\u001b[1;36m4\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
{\n",
       "  \"f#F1\": 0.665399,\n",
       "  \"pre#F1\": 0.680934,\n",
       "  \"rec#F1\": 0.650558\n",
       "}\n",
       "
\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "  \u001b[1;34m\"f#F1\"\u001b[0m: \u001b[1;36m0.665399\u001b[0m,\n",
       "  \u001b[1;34m\"pre#F1\"\u001b[0m: \u001b[1;36m0.680934\u001b[0m,\n",
       "  \u001b[1;34m\"rec#F1\"\u001b[0m: \u001b[1;36m0.650558\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n",
       "
\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
----------------------------- Eval. results on Epoch:5, Batch:0 -----------------------------\n",
       "
\n"
      ],
      "text/plain": [
       "----------------------------- Eval. results on Epoch:\u001b[1;36m5\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
{\n",
       "  \"f#F1\": 0.734694,\n",
       "  \"pre#F1\": 0.733333,\n",
       "  \"rec#F1\": 0.736059\n",
       "}\n",
       "
\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "  \u001b[1;34m\"f#F1\"\u001b[0m: \u001b[1;36m0.734694\u001b[0m,\n",
       "  \u001b[1;34m\"pre#F1\"\u001b[0m: \u001b[1;36m0.733333\u001b[0m,\n",
       "  \u001b[1;34m\"rec#F1\"\u001b[0m: \u001b[1;36m0.736059\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n",
       "
\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
----------------------------- Eval. results on Epoch:6, Batch:0 -----------------------------\n",
       "
\n"
      ],
      "text/plain": [
       "----------------------------- Eval. results on Epoch:\u001b[1;36m6\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
{\n",
       "  \"f#F1\": 0.742647,\n",
       "  \"pre#F1\": 0.734545,\n",
       "  \"rec#F1\": 0.750929\n",
       "}\n",
       "
\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "  \u001b[1;34m\"f#F1\"\u001b[0m: \u001b[1;36m0.742647\u001b[0m,\n",
       "  \u001b[1;34m\"pre#F1\"\u001b[0m: \u001b[1;36m0.734545\u001b[0m,\n",
       "  \u001b[1;34m\"rec#F1\"\u001b[0m: \u001b[1;36m0.750929\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n",
       "
\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
----------------------------- Eval. results on Epoch:7, Batch:0 -----------------------------\n",
       "
\n"
      ],
      "text/plain": [
       "----------------------------- Eval. results on Epoch:\u001b[1;36m7\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
{\n",
       "  \"f#F1\": 0.773585,\n",
       "  \"pre#F1\": 0.785441,\n",
       "  \"rec#F1\": 0.762082\n",
       "}\n",
       "
\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "  \u001b[1;34m\"f#F1\"\u001b[0m: \u001b[1;36m0.773585\u001b[0m,\n",
       "  \u001b[1;34m\"pre#F1\"\u001b[0m: \u001b[1;36m0.785441\u001b[0m,\n",
       "  \u001b[1;34m\"rec#F1\"\u001b[0m: \u001b[1;36m0.762082\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n",
       "
\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
----------------------------- Eval. results on Epoch:8, Batch:0 -----------------------------\n",
       "
\n"
      ],
      "text/plain": [
       "----------------------------- Eval. results on Epoch:\u001b[1;36m8\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
{\n",
       "  \"f#F1\": 0.770115,\n",
       "  \"pre#F1\": 0.794466,\n",
       "  \"rec#F1\": 0.747212\n",
       "}\n",
       "
\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "  \u001b[1;34m\"f#F1\"\u001b[0m: \u001b[1;36m0.770115\u001b[0m,\n",
       "  \u001b[1;34m\"pre#F1\"\u001b[0m: \u001b[1;36m0.794466\u001b[0m,\n",
       "  \u001b[1;34m\"rec#F1\"\u001b[0m: \u001b[1;36m0.747212\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n",
       "
\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
----------------------------- Eval. results on Epoch:9, Batch:0 -----------------------------\n",
       "
\n"
      ],
      "text/plain": [
       "----------------------------- Eval. results on Epoch:\u001b[1;36m9\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
{\n",
       "  \"f#F1\": 0.7603,\n",
       "  \"pre#F1\": 0.766038,\n",
       "  \"rec#F1\": 0.754647\n",
       "}\n",
       "
\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "  \u001b[1;34m\"f#F1\"\u001b[0m: \u001b[1;36m0.7603\u001b[0m,\n",
       "  \u001b[1;34m\"pre#F1\"\u001b[0m: \u001b[1;36m0.766038\u001b[0m,\n",
       "  \u001b[1;34m\"rec#F1\"\u001b[0m: \u001b[1;36m0.754647\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n",
       "
\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
---------------------------- Eval. results on Epoch:10, Batch:0 -----------------------------\n",
       "
\n"
      ],
      "text/plain": [
       "---------------------------- Eval. results on Epoch:\u001b[1;36m10\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
{\n",
       "  \"f#F1\": 0.743682,\n",
       "  \"pre#F1\": 0.722807,\n",
       "  \"rec#F1\": 0.765799\n",
       "}\n",
       "
\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "  \u001b[1;34m\"f#F1\"\u001b[0m: \u001b[1;36m0.743682\u001b[0m,\n",
       "  \u001b[1;34m\"pre#F1\"\u001b[0m: \u001b[1;36m0.722807\u001b[0m,\n",
       "  \u001b[1;34m\"rec#F1\"\u001b[0m: \u001b[1;36m0.765799\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n",
       "
\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "trainer.run(num_eval_batch_per_dl=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "37871d6b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "{'f#F1': 0.75283, 'pre#F1': 0.727438, 'rec#F1': 0.780059}"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.evaluator.run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "96bae094",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}