fastNLP/tutorials/fastnlp_tutorial_3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "213d538c",
   "metadata": {},
   "source": [
    "# T3. dataloader 的内部结构和基本使用\n",
    "\n",
    "  1   fastNLP 中的 dataloader\n",
    " \n",
    "    1.1   dataloader 的职责描述\n",
    "\n",
    "    1.2   dataloader 的基本使用\n",
    "\n",
    "  2   fastNLP 中 dataloader 的延伸\n",
    "\n",
    "    2.1   collator 的概念与使用\n",
    "\n",
    "    2.2   sampler 的概念与使用"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85857115",
   "metadata": {},
   "source": [
    "## 1. fastNLP 中的 dataloader\n",
    "\n",
    "### 1.1 dataloader 的职责描述\n",
    "\n",
    "在`fastNLP 0.8`中，在数据加载模块`DataLoader`之前，还存在其他的一些模块，负责例如对文本数据\n",
    "\n",
    "  进行补零对齐，即 **核对器`collator`模块**，进行分词标注，即 **分词器`tokenizer`模块**\n",
    "\n",
    "  本节将对`fastNLP`中的核对器`collator`等展开介绍，分词器`tokenizer`将在下一节中详细介绍\n",
    "\n",
    "在`fastNLP 0.8`中，**核对器`collator`模块负责文本序列的补零对齐**，通过"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb8fb51c",
   "metadata": {},
   "source": [
    "### 1.2 dataloader 的基本使用\n",
    "\n",
    "在`fastNLP 0.8`中，在数据加载模块`DataLoader`之前，还存在其他的一些模块，负责例如对文本数据\n",
    "\n",
    "  进行补零对齐，即 **核对器`collator`模块**，进行分词标注，即 **分词器`tokenizer`模块**\n",
    "\n",
    "  本节将对`fastNLP`中的核对器`collator`等展开介绍，分词器`tokenizer`将在下一节中详细介绍\n",
    "\n",
    "在`fastNLP 0.8`中，**核对器`collator`模块负责文本序列的补零对齐**，通过"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aca72b49",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from functools import partial\n",
    "from fastNLP.transformers.torch import BertTokenizer\n",
    "\n",
    "from fastNLP import DataSet\n",
    "from fastNLP import Vocabulary\n",
    "from fastNLP.io import DataBundle\n",
    "\n",
    "\n",
    "class PipeDemo:\n",
    "    def __init__(self, tokenizer='bert-base-uncased', num_proc=1):\n",
    "        self.tokenizer = BertTokenizer.from_pretrained(tokenizer)\n",
    "        self.num_proc = num_proc\n",
    "\n",
    "    def process_from_file(self, path='./data/test4dataset.tsv'):\n",
    "        datasets = DataSet.from_pandas(pd.read_csv(path))\n",
    "        train_ds, test_ds = datasets.split(ratio=0.7)\n",
    "        train_ds, dev_ds = datasets.split(ratio=0.8)\n",
    "        data_bundle = DataBundle(datasets={'train': train_ds, 'dev': dev_ds, 'test': test_ds})\n",
    "\n",
    "        encode = partial(self.tokenizer.encode_plus, max_length=100, truncation=True,\n",
    "                         return_attention_mask=True)\n",
    "        data_bundle.apply_field_more(encode, field_name='text', num_proc=self.num_proc)\n",
    "\n",
    "        target_vocab = Vocabulary(padding=None, unknown=None)\n",
    "\n",
    "        target_vocab.from_dataset(*[ds for _, ds in data_bundle.iter_datasets()], field_name='label')\n",
    "        target_vocab.index_dataset(*[ds for _, ds in data_bundle.iter_datasets()], field_name='label',\n",
    "                                   new_field_name='target')\n",
    "\n",
    "        data_bundle.set_pad('input_ids', pad_val=self.tokenizer.pad_token_id)\n",
    "        data_bundle.set_ignore('label', 'text')  \n",
    "        return data_bundle"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de53bff4",
   "metadata": {},
   "source": [
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57a29cb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = PipeDemo(tokenizer='bert-base-uncased', num_proc=4)\n",
    "\n",
    "data_bundle = pipe.process_from_file('./data/test4dataset.tsv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "226bb081",
   "metadata": {},
   "source": [
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7827557d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastNLP import prepare_torch_dataloader\n",
    "\n",
    "dl_bundle = prepare_torch_dataloader(data_bundle, batch_size=arg.batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d898cf40",
   "metadata": {},
   "source": [
    "  \n",
    "\n",
    "```python\n",
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    train_dataloader=dl_bundle['train'],\n",
    "    optimizers=optimizer,\n",
    "\t...\n",
    "\tdriver=\"torch\",\n",
    "\tdevice='cuda',\n",
    "\t...\n",
    "    evaluate_dataloaders={'dev': dl_bundle['dev'], 'test': dl_bundle['test']},     \n",
    "    metrics={'acc': Accuracy()},\n",
    "\t...\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d74d0523",
   "metadata": {},
   "source": [
    "## 2. fastNLP 中 dataloader 的延伸\n",
    "\n",
    "### 2.1 collator 的概念与使用\n",
    "\n",
    "在`fastNLP 0.8`中，在数据加载模块`DataLoader`之前，还存在其他的一些模块，负责例如对文本数据\n",
    "\n",
    "  进行补零对齐，即 **核对器`collator`模块**，进行分词标注，即 **分词器`tokenizer`模块**\n",
    "\n",
    "  本节将对`fastNLP`中的核对器`collator`等展开介绍，分词器`tokenizer`将在下一节中详细介绍\n",
    "\n",
    "在`fastNLP 0.8`中，**核对器`collator`模块负责文本序列的补零对齐**，通过"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "651baef6",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from fastNLP import prepare_torch_dataloader\n",
    "\n",
    "dl_bundle = prepare_torch_dataloader(data_bundle, train_batch_size=2)\n",
    "\n",
    "print(type(dl_bundle), type(dl_bundle['train']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "726ba357",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "dataloader = prepare_torch_dataloader(datasets['train'], train_batch_size=2)\n",
    "print(type(dataloader))\n",
    "print(dir(dataloader))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0795b3e",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "dataloader.collate_fn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9bbd9a7",
   "metadata": {},
   "source": [
    "### 2.2 sampler 的概念与使用"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0c3c58d",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "dataloader.batch_sampler"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51bf0878",
   "metadata": {},
   "source": [
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3fd2486f",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.13"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}