{ "cells": [ { "cell_type": "markdown", "id": "213d538c", "metadata": {}, "source": [ "# T3. dataloader 的内部结构和基本使用\n", "\n", "  1   fastNLP 中的 dataloader\n", " \n", "    1.1   dataloader 的基本介绍\n", "\n", "    1.2   dataloader 的函数创建\n", "\n", "  2   fastNLP 中 dataloader 的延伸\n", "\n", "    2.1   collator 的概念与使用\n", "\n", "    2.2   结合 datasets 框架" ] }, { "cell_type": "markdown", "id": "85857115", "metadata": {}, "source": [ "## 1. fastNLP 中的 dataloader\n", "\n", "### 1.1 dataloader 的基本介绍\n", "\n", "在`fastNLP 1.0`的开发中，最关键的开发目标就是**实现 fastNLP 对当前主流机器学习框架**，例如\n", "\n", "  **当下流行的 pytorch**，以及**国产的 paddle 、jittor 和 oneflow 的兼容**，扩大受众的同时，也是助力国产\n", "\n", "本着分而治之的思想，我们可以将`fastNLP 1.0`对`pytorch`、`paddle`、`jittor`、`oneflow`框架的兼容，划分为\n", "\n", "    **对数据预处理**、**批量 batch 的划分与补齐**、**模型训练**、**模型评测**，**四个部分的兼容**\n", "\n", "  针对数据预处理，我们已经在`tutorial-1`中介绍了`dataset`和`vocabulary`的使用\n", "\n", "    而结合`tutorial-0`，我们可以发现**数据预处理环节本质上是框架无关的**\n", "\n", "    因为在不同框架下，读取的原始数据格式都差异不大，彼此也很容易转换\n", "\n", "只有涉及到张量、模型，不同框架才展现出其各自的特色：**pytorch 和 oneflow 中的 tensor 和 nn.Module**\n", "\n", "    **在 paddle 中称为 tensor 和 nn.Layer**，**在 jittor 中则称为 Var 和 Module**\n", "\n", "    因此，**模型训练、模型评测**，**是兼容的重难点**，我们将会在`tutorial-5`中详细介绍\n", "\n", "  针对批量`batch`的处理，作为`fastNLP 1.0`中框架无关部分想框架相关部分的过渡\n", "\n", "    就是`dataloader`模块的职责，这也是本篇教程`tutorial-3`讲解的重点\n", "\n", "**dataloader 模块的职责**，详细划分可以包含以下三部分，**采样划分、补零对齐、框架匹配**\n", "\n", "    第一，确定`batch`大小，确定采样方式，划分后通过迭代器即可得到`batch`序列\n", "\n", "    第二，对于序列处理，这也是`fastNLP`主要针对的，将同个`batch`内的数据对齐\n", "\n", "    第三，**batch 内数据格式要匹配框架**，**但 batch 结构需保持一致**，**参数匹配机制**\n", "\n", "  对此，`fastNLP 1.0`给出了 **TorchDataLoader 、 PaddleDataLoader 、 JittorDataLoader 和 OneflowDataLoader**\n", "\n", "    分别针对并匹配不同框架，但彼此之间参数名、属性、方法仍然类似，前两者大致如下表所示\n", "\n", "名称|参数|属性|功能|内容\n", "----|----|----|----|----|\n", " `dataset` | √ | √ | 指定`dataloader`的数据内容 | |\n", " `batch_size` | √ | √ | 指定`dataloader`的`batch`大小 | 默认`16` |\n", " `shuffle` | √ | √ | 指定`dataloader`的数据是否打乱 | 默认`False` |\n", " `collate_fn` | √ | √ | 指定`dataloader`的`batch`打包方法 | 视框架而定 |\n", " `sampler` | √ | √ | 指定`dataloader`的`__len__`和`__iter__`函数的实现 | 默认`None` |\n", " `batch_sampler` | √ | √ | 指定`dataloader`的`__len__`和`__iter__`函数的实现 | 默认`None` |\n", " `drop_last` | √ | √ | 指定`dataloader`划分`batch`时是否丢弃剩余的 | 默认`False` |\n", " `cur_batch_indices` | | √ | 记录`dataloader`当前遍历批量序号 | |\n", " `num_workers` | √ | √ | 指定`dataloader`开启子进程数量 | 默认`0` |\n", " `worker_init_fn` | √ | √ | 指定`dataloader`子进程初始方法 | 默认`None` |\n", " `generator` | √ | √ | 指定`dataloader`子进程随机种子 | 默认`None` |\n", " `prefetch_factor` | | √ | 指定为每个`worker`装载的`sampler`数量 | 默认`2` |" ] }, { "cell_type": "markdown", "id": "60a8a224", "metadata": {}, "source": [ "  论及`dataloader`的函数，其中，`get_batch_indices`用来获取当前遍历到的`batch`序号，其他函数\n", "\n", "    包括`set_ignore`、`set_pad`和`databundle`类似，请参考`tutorial-2`，此处不做更多介绍\n", "\n", "    以下是`tutorial-2`中已经介绍过的数据预处理流程，接下来是对相关数据进行`dataloader`处理" ] }, { "cell_type": "code", "execution_count": 1, "id": "aca72b49", "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;5;2m[i 0604 15:44:29.773860 92 log.cc:351] Load log_sync: 1\u001b[m\n" ] }, { "data": { "text/html": [ "
\n",
       "
\n" ], "text/plain": [ "\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Processing: 0%| | 0/4 [00:00\n", " ['input_ids', 'token_type_ids', 'attention_mask', 'target']\n", "{'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n", " [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n", " 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],\n", " [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n", " [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n", " 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]),\n", " 'input_ids': tensor([[ 101, 1037, 4038, 1011, 3689, 1997, 3053, 8680, 19173, 15685,\n", " 1999, 1037, 18006, 2836, 2011, 1996, 2516, 2839, 14996, 3054,\n", " 15509, 5325, 1012, 102, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0],\n", " [ 101, 1037, 2186, 1997, 9686, 17695, 18673, 14313, 1996, 15262,\n", " 3351, 2008, 2054, 2003, 2204, 2005, 1996, 13020, 2003, 2036,\n", " 2204, 2005, 1996, 25957, 4063, 1010, 2070, 1997, 2029, 5681,\n", " 2572, 25581, 2021, 3904, 1997, 2029, 8310, 2000, 2172, 1997,\n", " 1037, 2466, 1012, 102],\n", " [ 101, 2130, 4599, 1997, 19214, 6432, 1005, 1055, 2147, 1010,\n", " 1045, 8343, 1010, 2052, 2031, 1037, 2524, 2051, 3564, 2083,\n", " 2023, 2028, 1012, 102, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0],\n", " [ 101, 1037, 13567, 26162, 5257, 1997, 3802, 7295, 9888, 1998,\n", " 2035, 1996, 20014, 27611, 1010, 14583, 1010, 11703, 20175, 1998,\n", " 4028, 1997, 1037, 8101, 2319, 10576, 2030, 1037, 28900, 7815,\n", " 3850, 1012, 102, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0]]),\n", " 'target': tensor([0, 1, 1, 2]),\n", " 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n", " [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n", " [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n", " [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])}\n" ] } ], "source": [ "from fastNLP import prepare_torch_dataloader\n", "\n", "train_dataset = data_bundle.get_dataset('train')\n", "evaluate_dataset = data_bundle.get_dataset('dev')\n", "\n", "train_dataloader = prepare_torch_dataloader(train_dataset, batch_size=16, shuffle=True)\n", "evaluate_dataloader = prepare_torch_dataloader(evaluate_dataset, batch_size=16)\n", "\n", "print(type(train_dataloader))\n", "\n", "import pprint\n", "\n", "for batch in train_dataloader:\n", " print(type(batch), type(batch['input_ids']), list(batch))\n", " pprint.pprint(batch, width=1)" ] }, { "cell_type": "markdown", "id": "9f457a6e", "metadata": {}, "source": [ "之所以说`prepare_xx_dataloader`函数更方便，是因为其**导入对象不仅可也是 DataSet 类型**，**还可以**\n", "\n", "  **是 DataBundle 类型**，不过数据集名称需要是`'train'`、`'dev'`、`'test'`供`fastNLP`识别\n", "\n", "例如下方就是**直接通过 prepare_paddle_dataloader 函数生成基于 PaddleDataLoader 的字典**\n" ] }, { "cell_type": "code", "execution_count": 3, "id": "7827557d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n" ] } ], "source": [ "from fastNLP import prepare_paddle_dataloader\n", "\n", "dl_bundle = prepare_paddle_dataloader(data_bundle, batch_size=16, shuffle=True)\n", "\n", "print(type(dl_bundle['train']))" ] }, { "cell_type": "markdown", "id": "d898cf40", "metadata": {}, "source": [ "  而在接下来`trainer`的初始化过程中，按如下方式使用即可，除了初始化时`driver='paddle'`外\n", "\n", "  这里也可以看出`trainer`模块中，**evaluate_dataloaders 的设计允许评测可以针对多个数据集**\n", "\n", "```python\n", "trainer = Trainer(\n", " model=model,\n", " train_dataloader=dl_bundle['train'],\n", " optimizers=optimizer,\n", "\t...\n", "\tdriver='paddle',\n", "\tdevice='gpu',\n", "\t...\n", " evaluate_dataloaders={'dev': dl_bundle['dev'], 'test': dl_bundle['test']}, \n", " metrics={'acc': Accuracy()},\n", "\t...\n", ")\n", "```" ] }, { "cell_type": "markdown", "id": "d74d0523", "metadata": {}, "source": [ "## 2. fastNLP 中 dataloader 的延伸\n", "\n", "### 2.1 collator 的概念与使用\n", "\n", "在`fastNLP 1.0`中，在数据加载模块`dataloader`内部，如之前表格所列举的，还存在其他的一些模块\n", "\n", "  例如，**实现序列的补零对齐的核对器 collator 模块**；注：`collate vt. 整理（文件或书等）；核对，校勘`\n", "\n", "在`fastNLP 1.0`中，虽然`dataloader`随框架不同，但`collator`模块却是统一的，主要属性、方法如下表所示\n", "\n", "名称|属性|方法|功能|内容\n", " ----|----|----|----|----|\n", " `backend` | √ | | 记录`collator`对应框架 | 字符串型，如`'torch'` |\n", " `padders` | √ | | 记录各字段对应的`padder`，每个负责具体补零对齐  | 字典类型 |\n", " `ignore_fields` | √ | | 记录`dataloader`采样`batch`时不予考虑的字段 | 集合类型 |\n", " `input_fields` | √ | | 记录`collator`每个字段的补零值、数据类型等 | 字典类型 |\n", " `set_backend` | | √ | 设置`collator`对应框架 | 字符串型，如`'torch'` |\n", " `set_ignore` | | √ | 设置`dataloader`采样`batch`时不予考虑的字段 | 字符串型，表示`field_name`  |\n", " `set_pad` | | √ | 设置`collator`每个字段的补零值、数据类型等 | |" ] }, { "cell_type": "code", "execution_count": 4, "id": "d0795b3e", "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n" ] } ], "source": [ "train_dataloader.collate_fn\n", "\n", "print(type(train_dataloader.collate_fn))" ] }, { "cell_type": "markdown", "id": "5f816ef5", "metadata": {}, "source": [ "此外，还可以 **手动定义 dataloader 中的 collate_fn**，而不是使用`fastNLP 1.0`中自带的`collator`模块\n", "\n", "  该函数的定义可以大致如下，需要注意的是，**定义 collate_fn 之前需要了解 batch 作为字典的格式**\n", "\n", "  该函数通过`collate_fn`参数传入`dataloader`，**在 batch 分发**（**而不是 batch 划分**）**时调用**" ] }, { "cell_type": "code", "execution_count": 5, "id": "ff8e405e", "metadata": {}, "outputs": [], "source": [ "import torch\n", "\n", "def collate_fn(batch):\n", " input_ids, atten_mask, labels = [], [], []\n", " max_length = [0] * 3\n", " for each_item in batch:\n", " input_ids.append(each_item['input_ids'])\n", " max_length[0] = max(len(each_item['input_ids']), max_length[0])\n", " atten_mask.append(each_item['token_type_ids'])\n", " max_length[1] = max(len(each_item['token_type_ids']), max_length[1])\n", " labels.append(each_item['attention_mask'])\n", " max_length[2] = max(len(each_item['attention_mask']), max_length[2])\n", "\n", " for i in range(3):\n", " each = (input_ids, atten_mask, labels)[i]\n", " for item in each:\n", " item.extend([0] * (max_length[i] - len(item)))\n", " return {'input_ids': torch.cat([torch.tensor([item]) for item in input_ids], dim=0),\n", " 'token_type_ids': torch.cat([torch.tensor([item]) for item in atten_mask], dim=0),\n", " 'attention_mask': torch.cat([torch.tensor(item) for item in labels], dim=0)}" ] }, { "cell_type": "markdown", "id": "487b75fb", "metadata": {}, "source": [ "注意：使用自定义的`collate_fn`函数，`trainer`的`collate_fn`变量也会自动调整为`function`类型" ] }, { "cell_type": "code", "execution_count": 6, "id": "e916d1ac", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "{'attention_mask': tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1,\n", " 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n", " 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n", " 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n", " 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0]),\n", " 'input_ids': tensor([[ 101, 1037, 4038, 1011, 3689, 1997, 3053, 8680, 19173, 15685,\n", " 1999, 1037, 18006, 2836, 2011, 1996, 2516, 2839, 14996, 3054,\n", " 15509, 5325, 1012, 102, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0],\n", " [ 101, 1037, 2186, 1997, 9686, 17695, 18673, 14313, 1996, 15262,\n", " 3351, 2008, 2054, 2003, 2204, 2005, 1996, 13020, 2003, 2036,\n", " 2204, 2005, 1996, 25957, 4063, 1010, 2070, 1997, 2029, 5681,\n", " 2572, 25581, 2021, 3904, 1997, 2029, 8310, 2000, 2172, 1997,\n", " 1037, 2466, 1012, 102],\n", " [ 101, 2130, 4599, 1997, 19214, 6432, 1005, 1055, 2147, 1010,\n", " 1045, 8343, 1010, 2052, 2031, 1037, 2524, 2051, 3564, 2083,\n", " 2023, 2028, 1012, 102, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0],\n", " [ 101, 1037, 13567, 26162, 5257, 1997, 3802, 7295, 9888, 1998,\n", " 2035, 1996, 20014, 27611, 1010, 14583, 1010, 11703, 20175, 1998,\n", " 4028, 1997, 1037, 8101, 2319, 10576, 2030, 1037, 28900, 7815,\n", " 3850, 1012, 102, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0]]),\n", " 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n", " [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n", " [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n", " [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])}\n" ] } ], "source": [ "train_dataloader = prepare_torch_dataloader(train_dataset, collate_fn=collate_fn, shuffle=True)\n", "evaluate_dataloader = prepare_torch_dataloader(evaluate_dataset, collate_fn=collate_fn, shuffle=True)\n", "\n", "print(type(train_dataloader))\n", "print(type(train_dataloader.collate_fn))\n", "\n", "for batch in train_dataloader:\n", " pprint.pprint(batch, width=1)" ] }, { "cell_type": "markdown", "id": "0bd98365", "metadata": {}, "source": [ "### 2.2 fastNLP 与 datasets 的结合\n", "\n", "从`tutorial-1`至`tutorial-3`，我们已经完成了对`fastNLP v1.0`数据读取、预处理、加载，整个流程的介绍\n", "\n", "  不过在实际使用中，我们往往也会采取更为简便的方法读取数据，例如使用`huggingface`的`datasets`模块\n", "\n", "**使用 datasets 模块中的 load_dataset 函数**，通过指定数据集两级的名称，示例中即是**GLUE 标准中的 SST-2 数据集**\n", "\n", "  即可以快速从网上下载好`SST-2`数据集读入，之后以`pandas.DataFrame`作为中介，再转化成`fastNLP.DataSet`\n", "\n", "  之后的步骤就和其他关于`dataset`、`databundle`、`vocabulary`、`dataloader`中介绍的相关使用相同了" ] }, { "cell_type": "code", "execution_count": 7, "id": "91879c30", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Reusing dataset glue (/remote-home/xrliu/.cache/huggingface/datasets/glue/sst2/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad)\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "639a0ad3c63944c6abef4e8ee1f7bf7c", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/3 [00:00