{ "cells": [ { "cell_type": "markdown", "id": "fdd7ff16", "metadata": {}, "source": [ "# T4. trainer 和 evaluator 的深入介绍\n", "\n", "  1   fastNLP 中的更多 metric 类型\n", "\n", "    1.1   预定义的 metric 类型\n", "\n", "    1.2   自定义的 metric 类型\n", "\n", "  2   fastNLP 中 trainer 的补充介绍\n", " \n", "    2.1   trainer 的提出构想 \n", "\n", "    2.2   trainer 的内部结构\n", "\n", "    2.3   实例：\n", "\n", "  3   fastNLP 中的 driver 与 device\n", "\n", "    3.1   driver 的提出构想\n", "\n", "    3.2   device 与多卡训练" ] }, { "cell_type": "markdown", "id": "8d19220c", "metadata": {}, "source": [ "## 1. fastNLP 中的更多 metric 类型\n", "\n", "### 1.1 预定义的 metric 类型\n", "\n", "在`fastNLP 0.8`中，除了前几篇`tutorial`中经常见到的**正确率`Accuracy`**，还有其他**预定义的评价标准`metric`**\n", "\n", "  包括**所有`metric`的基类`Metric`**、适配`Transformers`中相关模型的正确率`TransformersAccuracy`\n", "\n", "    **适用于分类语境下的`F1`值`ClassifyFPreRecMetric`**（其中也包括**召回率`Pre`**、**精确率`Rec`**\n", "\n", "    **适用于抽取语境下的`F1`值`SpanFPreRecMetric`**；相关基本信息内容见下表，之后是详细分析\n", "\n", "|
代码名称
|
简要介绍
|
代码路径
|\n", "|:--|:--|:--|\n", "| `Metric` | 定义`metrics`时继承的基类 | `/core/metrics/metric.py` |\n", "| `Accuracy` | 正确率，最为常用 | `/core/metrics/accuracy.py` |\n", "| `TransformersAccuracy` | 正确率，为了兼容`Transformers`中相关模型 | `/core/metrics/accuracy.py` |\n", "| `ClassifyFPreRecMetric` | 召回率、精确率、F1值，适用于**分类问题** | `/core/metrics/classify_f1_pre_rec_metric.py` |\n", "| `SpanFPreRecMetric` | 召回率、精确率、F1值，适用于**抽取问题** | `/core/metrics/span_f1_pre_rec_metric.py` |" ] }, { "cell_type": "markdown", "id": "fdc083a3", "metadata": { "pycharm": { "name": "#%%\n" } }, "source": [ "大概的描述一下，给出各个正确率的计算公式" ] }, { "cell_type": "code", "execution_count": null, "id": "9775ea5e", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "8a22f522", "metadata": {}, "source": [ "### 2.2 自定义的 metric 类型\n", "\n", "在`fastNLP 0.8`中，  给一个案例，训练部分留到trainer部分" ] }, { "cell_type": "code", "execution_count": null, "id": "d8caba1d", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "id": "4e6247dd", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "08752c5a", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "## 2. fastNLP 中 trainer 的补充介绍\n", "\n", "### 2.1 trainer 的提出构想\n", "\n", "在`fastNLP 0.8`中，  " ] }, { "cell_type": "code", "execution_count": null, "id": "977a6355", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "id": "69203cdc", "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "ab1cea7d", "metadata": {}, "source": [ "### 2.2 trainer 的内部结构\n", "\n", "在`fastNLP 0.8`中，  \n", "\n", "'accumulation_steps', 'add_callback_fn', 'backward', 'batch_idx_in_epoch', 'batch_step_fn',\n", "'callback_manager', 'check_batch_step_fn', 'cur_epoch_idx', 'data_device', 'dataloader',\n", "'device', 'driver', 'driver_name', 'epoch_evaluate', 'evaluate_batch_step_fn', 'evaluate_dataloaders',\n", "'evaluate_every', 'evaluate_fn', 'evaluator', 'extract_loss_from_outputs', 'fp16',\n", "'get_no_sync_context', 'global_forward_batches', 'has_checked_train_batch_loop',\n", "'input_mapping', 'kwargs', 'larger_better', 'load_checkpoint', 'load_model', 'marker',\n", "'metrics', 'model', 'model_device', 'monitor', 'move_data_to_device', 'n_epochs', 'num_batches_per_epoch',\n", "'on', 'on_after_backward', 'on_after_optimizers_step', 'on_after_trainer_initialized',\n", "'on_after_zero_grad', 'on_before_backward', 'on_before_optimizers_step', 'on_before_zero_grad',\n", "'on_evaluate_begin', 'on_evaluate_end', 'on_exception', 'on_fetch_data_begin', 'on_fetch_data_end',\n", "'on_load_checkpoint', 'on_load_model', 'on_sanity_check_begin', 'on_sanity_check_end',\n", "'on_save_checkpoint', 'on_save_model', 'on_train_batch_begin', 'on_train_batch_end',\n", "'on_train_begin', 'on_train_end', 'on_train_epoch_begin', 'on_train_epoch_end',\n", "'optimizers', 'output_mapping', 'progress_bar', 'run', 'run_evaluate',\n", "'save_checkpoint', 'save_model', 'start_batch_idx_in_epoch', 'state',\n", "'step', 'step_evaluate', 'total_batches', 'train_batch_loop', 'train_dataloader', 'train_fn', 'train_step',\n", "'trainer_state', 'zero_grad'\n", "\n", "  run(num_train_batch_per_epoch: int = -1, num_eval_batch_per_dl: int = -1, num_eval_sanity_batch: int = 2, resume_from: str = None, resume_training: bool = True, catch_KeyboardInterrupt=None)" ] }, { "cell_type": "code", "execution_count": null, "id": "b3c8342e", "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "id": "d28f2624", "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "ce6322b4", "metadata": {}, "source": [ "### 2.3 实例：\n", "\n", "在`fastNLP 0.8`中，  " ] }, { "cell_type": "code", "execution_count": null, "id": "43be274f", "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "id": "c348864c", "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "175d6ebb", "metadata": {}, "source": [ "## 3. fastNLP 中的 driver 与 device\n", "\n", "### 3.1 driver 的提出构想\n", "\n", "在`fastNLP 0.8`中，  " ] }, { "cell_type": "code", "execution_count": null, "id": "47100e7a", "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "id": "0204a223", "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "6e723b87", "metadata": {}, "source": [ "### 3.2 device 与多卡训练\n", "\n", "在`fastNLP 0.8`中，  " ] }, { "cell_type": "code", "execution_count": null, "id": "5ad81ac7", "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "id": "cfb28b1b", "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.13" }, "pycharm": { "stem_cell": { "cell_type": "raw", "metadata": { "collapsed": false }, "source": [] } } }, "nbformat": 4, "nbformat_minor": 5 }