fastNLP/tutorials/fastnlp_tutorial_4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fdd7ff16",
   "metadata": {},
   "source": [
    "# T4. trainer 和 evaluator 的深入介绍\n",
    "\n",
    "  1   fastNLP 中的更多 metric 类型\n",
    "\n",
    "    1.1   预定义的 metric 类型\n",
    "\n",
    "    1.2   自定义的 metric 类型\n",
    "\n",
    "  2   fastNLP 中 trainer 的补充介绍\n",
    " \n",
    "    2.1   trainer 的提出构想 \n",
    "\n",
    "    2.2   trainer 的内部结构\n",
    "\n",
    "    2.3   实例：\n",
    "\n",
    "  3   fastNLP 中的 driver 与 device\n",
    "\n",
    "    3.1   driver 的提出构想\n",
    "\n",
    "    3.2   device 与多卡训练"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d19220c",
   "metadata": {},
   "source": [
    "## 1. fastNLP 中的更多 metric 类型\n",
    "\n",
    "### 1.1  预定义的 metric 类型\n",
    "\n",
    "在`fastNLP 0.8`中，除了前几篇`tutorial`中经常见到的**正确率`Accuracy`**，还有其他**预定义的评价标准`metric`**\n",
    "\n",
    "  包括**所有`metric`的基类`Metric`**、适配`Transformers`中相关模型的正确率`TransformersAccuracy`\n",
    "\n",
    "    **适用于分类语境下的`F1`值`ClassifyFPreRecMetric`**（其中也包括**召回率`Pre`**、**精确率`Rec`**\n",
    "\n",
    "    **适用于抽取语境下的`F1`值`SpanFPreRecMetric`**；相关基本信息内容见下表，之后是详细分析\n",
    "\n",
    "| <div align=\"center\">代码名称</div> | <div align=\"center\">简要介绍</div> | <div align=\"center\">代码路径</div> |\n",
    "|:--|:--|:--|\n",
    "| `Metric` | 定义`metrics`时继承的基类 | `/core/metrics/metric.py` |\n",
    "| `Accuracy` | 正确率，最为常用 | `/core/metrics/accuracy.py` |\n",
    "| `TransformersAccuracy` | 正确率，为了兼容`Transformers`中相关模型 | `/core/metrics/accuracy.py` |\n",
    "| `ClassifyFPreRecMetric` | 召回率、精确率、F1值，适用于**分类问题** | `/core/metrics/classify_f1_pre_rec_metric.py` |\n",
    "| `SpanFPreRecMetric` | 召回率、精确率、F1值，适用于**抽取问题** | `/core/metrics/span_f1_pre_rec_metric.py` |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdc083a3",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "source": [
    "大概的描述一下，给出各个正确率的计算公式"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9775ea5e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "8a22f522",
   "metadata": {},
   "source": [
    "### 2.2  自定义的 metric 类型\n",
    "\n",
    "在`fastNLP 0.8`中，&emsp; 给一个案例，训练部分留到trainer部分"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8caba1d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e6247dd",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "08752c5a",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## 2. fastNLP 中 trainer 的补充介绍\n",
    "\n",
    "### 2.1  trainer 的提出构想\n",
    "\n",
    "在`fastNLP 0.8`中，&emsp; "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "977a6355",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69203cdc",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "ab1cea7d",
   "metadata": {},
   "source": [
    "### 2.2  trainer 的内部结构\n",
    "\n",
    "在`fastNLP 0.8`中，&emsp; \n",
    "\n",
    "'accumulation_steps', 'add_callback_fn', 'backward', 'batch_idx_in_epoch', 'batch_step_fn',\n",
    "'callback_manager', 'check_batch_step_fn', 'cur_epoch_idx', 'data_device', 'dataloader',\n",
    "'device', 'driver', 'driver_name', 'epoch_evaluate', 'evaluate_batch_step_fn', 'evaluate_dataloaders',\n",
    "'evaluate_every', 'evaluate_fn', 'evaluator', 'extract_loss_from_outputs', 'fp16',\n",
    "'get_no_sync_context', 'global_forward_batches', 'has_checked_train_batch_loop',\n",
    "'input_mapping', 'kwargs', 'larger_better', 'load_checkpoint', 'load_model', 'marker',\n",
    "'metrics', 'model', 'model_device', 'monitor', 'move_data_to_device', 'n_epochs', 'num_batches_per_epoch',\n",
    "'on', 'on_after_backward', 'on_after_optimizers_step', 'on_after_trainer_initialized',\n",
    "'on_after_zero_grad', 'on_before_backward', 'on_before_optimizers_step', 'on_before_zero_grad',\n",
    "'on_evaluate_begin', 'on_evaluate_end', 'on_exception', 'on_fetch_data_begin', 'on_fetch_data_end',\n",
    "'on_load_checkpoint', 'on_load_model', 'on_sanity_check_begin', 'on_sanity_check_end',\n",
    "'on_save_checkpoint', 'on_save_model', 'on_train_batch_begin', 'on_train_batch_end',\n",
    "'on_train_begin', 'on_train_end', 'on_train_epoch_begin', 'on_train_epoch_end',\n",
    "'optimizers', 'output_mapping', 'progress_bar', 'run', 'run_evaluate',\n",
    "'save_checkpoint', 'save_model', 'start_batch_idx_in_epoch', 'state',\n",
    "'step', 'step_evaluate', 'total_batches', 'train_batch_loop', 'train_dataloader', 'train_fn', 'train_step',\n",
    "'trainer_state', 'zero_grad'\n",
    "\n",
    "&emsp; run(num_train_batch_per_epoch: int = -1, num_eval_batch_per_dl: int = -1, num_eval_sanity_batch: int = 2, resume_from: str = None, resume_training: bool = True, catch_KeyboardInterrupt=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3c8342e",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d28f2624",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "ce6322b4",
   "metadata": {},
   "source": [
    "### 2.3  实例：\n",
    "\n",
    "在`fastNLP 0.8`中，&emsp; "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43be274f",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c348864c",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "175d6ebb",
   "metadata": {},
   "source": [
    "## 3. fastNLP 中的 driver 与 device\n",
    "\n",
    "### 3.1  driver 的提出构想\n",
    "\n",
    "在`fastNLP 0.8`中，&emsp; "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47100e7a",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0204a223",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "6e723b87",
   "metadata": {},
   "source": [
    "### 3.2  device 与多卡训练\n",
    "\n",
    "在`fastNLP 0.8`中，&emsp; "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ad81ac7",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cfb28b1b",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.13"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}