Lhy_Machine_Learning/2022 ML/07 Self-supervised learning for Speech and Image/ML2022Spring_HW7.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "ML2022Spring - HW7.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xvSGDbExff_I"
      },
      "source": [
        "# **Homework 7 - Bert (Question Answering)**\n",
        "\n",
        "If you have any questions, feel free to email us at mlta-2022-spring@googlegroups.com\n",
        "\n",
        "\n",
        "\n",
        "Slide:    [Link](https://docs.google.com/presentation/d/1H5ZONrb2LMOCixLY7D5_5-7LkIaXO6AGEaV2mRdTOMY/edit?usp=sharing)　Kaggle: [Link](https://www.kaggle.com/c/ml2022spring-hw7)　Data: [Link](https://drive.google.com/uc?id=1AVgZvy3VFeg0fX-6WQJMHPVrx3A-M1kb)\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WGOr_eS3wJJf"
      },
      "source": [
        "## Task description\n",
        "- Chinese Extractive Question Answering\n",
        "  - Input: Paragraph + Question\n",
        "  - Output: Answer\n",
        "\n",
        "- Objective: Learn how to fine tune a pretrained model on downstream task using transformers\n",
        "\n",
        "- Todo\n",
        "    - Fine tune a pretrained chinese BERT model\n",
        "    - Change hyperparameters (e.g. doc_stride)\n",
        "    - Apply linear learning rate decay\n",
        "    - Try other pretrained models\n",
        "    - Improve preprocessing\n",
        "    - Improve postprocessing\n",
        "- Training tips\n",
        "    - Automatic mixed precision\n",
        "    - Gradient accumulation\n",
        "    - Ensemble\n",
        "\n",
        "- Estimated training time (tesla t4 with automatic mixed precision enabled)\n",
        "    - Simple: 8mins\n",
        "    - Medium: 8mins\n",
        "    - Strong: 25mins\n",
        "    - Boss: 2.5hrs\n",
        "  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TJ1fSAJE2oaC"
      },
      "source": [
        "## Download Dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YPrc4Eie9Yo5"
      },
      "source": [
        "# Download link 1\n",
        "!gdown --id '1AVgZvy3VFeg0fX-6WQJMHPVrx3A-M1kb' --output hw7_data.zip\n",
        "\n",
        "# Download Link 2 (if the above link fails) \n",
        "# !gdown --id '1qwjbRjq481lHsnTrrF4OjKQnxzgoLEFR' --output hw7_data.zip\n",
        "\n",
        "# Download Link 3 (if the above link fails) \n",
        "# !gdown --id '1QXuWjNRZH6DscSd6QcRER0cnxmpZvijn' --output hw7_data.zip\n",
        "\n",
        "!unzip -o hw7_data.zip\n",
        "\n",
        "# For this HW, K80 < P4 < T4 < P100 <= T4(fp16) < V100\n",
        "!nvidia-smi"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TevOvhC03m0h"
      },
      "source": [
        "## Install transformers\n",
        "\n",
        "Documentation for the toolkit:　https://huggingface.co/transformers/"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tbxWFX_jpDom"
      },
      "source": [
        "# You are allowed to change version of transformers or use other toolkits\n",
        "!pip install transformers==4.5.0"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8dKM4yCh4LI_"
      },
      "source": [
        "## Import Packages"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WOTHHtWJoahe"
      },
      "source": [
        "import json\n",
        "import numpy as np\n",
        "import random\n",
        "import torch\n",
        "from torch.utils.data import DataLoader, Dataset \n",
        "from transformers import AdamW, BertForQuestionAnswering, BertTokenizerFast\n",
        "\n",
        "from tqdm.auto import tqdm\n",
        "\n",
        "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
        "\n",
        "# Fix random seed for reproducibility\n",
        "def same_seeds(seed):\n",
        "\t  torch.manual_seed(seed)\n",
        "\t  if torch.cuda.is_available():\n",
        "\t\t    torch.cuda.manual_seed(seed)\n",
        "\t\t    torch.cuda.manual_seed_all(seed)\n",
        "\t  np.random.seed(seed)\n",
        "\t  random.seed(seed)\n",
        "\t  torch.backends.cudnn.benchmark = False\n",
        "\t  torch.backends.cudnn.deterministic = True\n",
        "same_seeds(0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7pBtSZP1SKQO"
      },
      "source": [
        "# Change \"fp16_training\" to True to support automatic mixed precision training (fp16)\t\n",
        "fp16_training = False\n",
        "\n",
        "if fp16_training:\n",
        "    !pip install accelerate==0.2.0\n",
        "    from accelerate import Accelerator\n",
        "    accelerator = Accelerator(fp16=True)\n",
        "    device = accelerator.device\n",
        "\n",
        "# Documentation for the toolkit:  https://huggingface.co/docs/accelerate/"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2YgXHuVLp_6j"
      },
      "source": [
        "## Load Model and Tokenizer\n",
        "\n",
        "\n",
        "\n",
        "\n",
        " "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xyBCYGjAp3ym"
      },
      "source": [
        "model = BertForQuestionAnswering.from_pretrained(\"bert-base-chinese\").to(device)\n",
        "tokenizer = BertTokenizerFast.from_pretrained(\"bert-base-chinese\")\n",
        "\n",
        "# You can safely ignore the warning message (it pops up because new prediction heads for QA are initialized randomly)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3Td-GTmk5OW4"
      },
      "source": [
        "## Read Data\n",
        "\n",
        "- Training set: 31690 QA pairs\n",
        "- Dev set: 4131  QA pairs\n",
        "- Test set: 4957  QA pairs\n",
        "\n",
        "- {train/dev/test}_questions:\t\n",
        "  - List of dicts with the following keys:\n",
        "   - id (int)\n",
        "   - paragraph_id (int)\n",
        "   - question_text (string)\n",
        "   - answer_text (string)\n",
        "   - answer_start (int)\n",
        "   - answer_end (int)\n",
        "- {train/dev/test}_paragraphs: \n",
        "  - List of strings\n",
        "  - paragraph_ids in questions correspond to indexs in paragraphs\n",
        "  - A paragraph may be used by several questions "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NvX7hlepogvu"
      },
      "source": [
        "def read_data(file):\n",
        "    with open(file, 'r', encoding=\"utf-8\") as reader:\n",
        "        data = json.load(reader)\n",
        "    return data[\"questions\"], data[\"paragraphs\"]\n",
        "\n",
        "train_questions, train_paragraphs = read_data(\"hw7_train.json\")\n",
        "dev_questions, dev_paragraphs = read_data(\"hw7_dev.json\")\n",
        "test_questions, test_paragraphs = read_data(\"hw7_test.json\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fm0rpTHq0e4N"
      },
      "source": [
        "## Tokenize Data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rTZ6B70Hoxie"
      },
      "source": [
        "# Tokenize questions and paragraphs separately\n",
        "# 「add_special_tokens」 is set to False since special tokens will be added when tokenized questions and paragraphs are combined in datset __getitem__ \n",
        "\n",
        "train_questions_tokenized = tokenizer([train_question[\"question_text\"] for train_question in train_questions], add_special_tokens=False)\n",
        "dev_questions_tokenized = tokenizer([dev_question[\"question_text\"] for dev_question in dev_questions], add_special_tokens=False)\n",
        "test_questions_tokenized = tokenizer([test_question[\"question_text\"] for test_question in test_questions], add_special_tokens=False) \n",
        "\n",
        "train_paragraphs_tokenized = tokenizer(train_paragraphs, add_special_tokens=False)\n",
        "dev_paragraphs_tokenized = tokenizer(dev_paragraphs, add_special_tokens=False)\n",
        "test_paragraphs_tokenized = tokenizer(test_paragraphs, add_special_tokens=False)\n",
        "\n",
        "# You can safely ignore the warning message as tokenized sequences will be futher processed in datset __getitem__ before passing to model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ws8c8_4d5UCI"
      },
      "source": [
        "## Dataset and Dataloader"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xjooag-Swnuh"
      },
      "source": [
        "class QA_Dataset(Dataset):\n",
        "    def __init__(self, split, questions, tokenized_questions, tokenized_paragraphs):\n",
        "        self.split = split\n",
        "        self.questions = questions\n",
        "        self.tokenized_questions = tokenized_questions\n",
        "        self.tokenized_paragraphs = tokenized_paragraphs\n",
        "        self.max_question_len = 40\n",
        "        self.max_paragraph_len = 150\n",
        "        \n",
        "        ##### TODO: Change value of doc_stride #####\n",
        "        self.doc_stride = 150\n",
        "\n",
        "        # Input sequence length = [CLS] + question + [SEP] + paragraph + [SEP]\n",
        "        self.max_seq_len = 1 + self.max_question_len + 1 + self.max_paragraph_len + 1\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.questions)\n",
        "\n",
        "    def __getitem__(self, idx):\n",
        "        question = self.questions[idx]\n",
        "        tokenized_question = self.tokenized_questions[idx]\n",
        "        tokenized_paragraph = self.tokenized_paragraphs[question[\"paragraph_id\"]]\n",
        "\n",
        "        ##### TODO: Preprocessing #####\n",
        "        # Hint: How to prevent model from learning something it should not learn\n",
        "\n",
        "        if self.split == \"train\":\n",
        "            # Convert answer's start/end positions in paragraph_text to start/end positions in tokenized_paragraph  \n",
        "            answer_start_token = tokenized_paragraph.char_to_token(question[\"answer_start\"])\n",
        "            answer_end_token = tokenized_paragraph.char_to_token(question[\"answer_end\"])\n",
        "\n",
        "            # A single window is obtained by slicing the portion of paragraph containing the answer\n",
        "            mid = (answer_start_token + answer_end_token) // 2\n",
        "            paragraph_start = max(0, min(mid - self.max_paragraph_len // 2, len(tokenized_paragraph) - self.max_paragraph_len))\n",
        "            paragraph_end = paragraph_start + self.max_paragraph_len\n",
        "            \n",
        "            # Slice question/paragraph and add special tokens (101: CLS, 102: SEP)\n",
        "            input_ids_question = [101] + tokenized_question.ids[:self.max_question_len] + [102] \n",
        "            input_ids_paragraph = tokenized_paragraph.ids[paragraph_start : paragraph_end] + [102]\t\t\n",
        "            \n",
        "            # Convert answer's start/end positions in tokenized_paragraph to start/end positions in the window  \n",
        "            answer_start_token += len(input_ids_question) - paragraph_start\n",
        "            answer_end_token += len(input_ids_question) - paragraph_start\n",
        "            \n",
        "            # Pad sequence and obtain inputs to model \n",
        "            input_ids, token_type_ids, attention_mask = self.padding(input_ids_question, input_ids_paragraph)\n",
        "            return torch.tensor(input_ids), torch.tensor(token_type_ids), torch.tensor(attention_mask), answer_start_token, answer_end_token\n",
        "\n",
        "        # Validation/Testing\n",
        "        else:\n",
        "            input_ids_list, token_type_ids_list, attention_mask_list = [], [], []\n",
        "            \n",
        "            # Paragraph is split into several windows, each with start positions separated by step \"doc_stride\"\n",
        "            for i in range(0, len(tokenized_paragraph), self.doc_stride):\n",
        "                \n",
        "                # Slice question/paragraph and add special tokens (101: CLS, 102: SEP)\n",
        "                input_ids_question = [101] + tokenized_question.ids[:self.max_question_len] + [102]\n",
        "                input_ids_paragraph = tokenized_paragraph.ids[i : i + self.max_paragraph_len] + [102]\n",
        "                \n",
        "                # Pad sequence and obtain inputs to model\n",
        "                input_ids, token_type_ids, attention_mask = self.padding(input_ids_question, input_ids_paragraph)\n",
        "                \n",
        "                input_ids_list.append(input_ids)\n",
        "                token_type_ids_list.append(token_type_ids)\n",
        "                attention_mask_list.append(attention_mask)\n",
        "            \n",
        "            return torch.tensor(input_ids_list), torch.tensor(token_type_ids_list), torch.tensor(attention_mask_list)\n",
        "\n",
        "    def padding(self, input_ids_question, input_ids_paragraph):\n",
        "        # Pad zeros if sequence length is shorter than max_seq_len\n",
        "        padding_len = self.max_seq_len - len(input_ids_question) - len(input_ids_paragraph)\n",
        "        # Indices of input sequence tokens in the vocabulary\n",
        "        input_ids = input_ids_question + input_ids_paragraph + [0] * padding_len\n",
        "        # Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]\n",
        "        token_type_ids = [0] * len(input_ids_question) + [1] * len(input_ids_paragraph) + [0] * padding_len\n",
        "        # Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]\n",
        "        attention_mask = [1] * (len(input_ids_question) + len(input_ids_paragraph)) + [0] * padding_len\n",
        "        \n",
        "        return input_ids, token_type_ids, attention_mask\n",
        "\n",
        "train_set = QA_Dataset(\"train\", train_questions, train_questions_tokenized, train_paragraphs_tokenized)\n",
        "dev_set = QA_Dataset(\"dev\", dev_questions, dev_questions_tokenized, dev_paragraphs_tokenized)\n",
        "test_set = QA_Dataset(\"test\", test_questions, test_questions_tokenized, test_paragraphs_tokenized)\n",
        "\n",
        "train_batch_size = 32\n",
        "\n",
        "# Note: Do NOT change batch size of dev_loader / test_loader !\n",
        "# Although batch size=1, it is actually a batch consisting of several windows from the same QA pair\n",
        "train_loader = DataLoader(train_set, batch_size=train_batch_size, shuffle=True, pin_memory=True)\n",
        "dev_loader = DataLoader(dev_set, batch_size=1, shuffle=False, pin_memory=True)\n",
        "test_loader = DataLoader(test_set, batch_size=1, shuffle=False, pin_memory=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5_H1kqhR8CdM"
      },
      "source": [
        "## Function for Evaluation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SqeA3PLPxOHu"
      },
      "source": [
        "def evaluate(data, output):\n",
        "    ##### TODO: Postprocessing #####\n",
        "    # There is a bug and room for improvement in postprocessing \n",
        "    # Hint: Open your prediction file to see what is wrong \n",
        "    \n",
        "    answer = ''\n",
        "    max_prob = float('-inf')\n",
        "    num_of_windows = data[0].shape[1]\n",
        "    \n",
        "    for k in range(num_of_windows):\n",
        "        # Obtain answer by choosing the most probable start position / end position\n",
        "        start_prob, start_index = torch.max(output.start_logits[k], dim=0)\n",
        "        end_prob, end_index = torch.max(output.end_logits[k], dim=0)\n",
        "        \n",
        "        # Probability of answer is calculated as sum of start_prob and end_prob\n",
        "        prob = start_prob + end_prob\n",
        "        \n",
        "        # Replace answer if calculated probability is larger than previous windows\n",
        "        if prob > max_prob:\n",
        "            max_prob = prob\n",
        "            # Convert tokens to chars (e.g. [1920, 7032] --> \"大 金\")\n",
        "            answer = tokenizer.decode(data[0][0][k][start_index : end_index + 1])\n",
        "    \n",
        "    # Remove spaces in answer (e.g. \"大 金\" --> \"大金\")\n",
        "    return answer.replace(' ','')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rzHQit6eMnKG"
      },
      "source": [
        "## Training"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3Q-B6ka7xoCM"
      },
      "source": [
        "num_epoch = 1\n",
        "validation = True\n",
        "logging_step = 100\n",
        "learning_rate = 1e-4\n",
        "optimizer = AdamW(model.parameters(), lr=learning_rate)\n",
        "\n",
        "if fp16_training:\n",
        "    model, optimizer, train_loader = accelerator.prepare(model, optimizer, train_loader) \n",
        "\n",
        "model.train()\n",
        "\n",
        "print(\"Start Training ...\")\n",
        "\n",
        "for epoch in range(num_epoch):\n",
        "    step = 1\n",
        "    train_loss = train_acc = 0\n",
        "    \n",
        "    for data in tqdm(train_loader):\t\n",
        "        # Load all data into GPU\n",
        "        data = [i.to(device) for i in data]\n",
        "        \n",
        "        # Model inputs: input_ids, token_type_ids, attention_mask, start_positions, end_positions (Note: only \"input_ids\" is mandatory)\n",
        "        # Model outputs: start_logits, end_logits, loss (return when start_positions/end_positions are provided)  \n",
        "        output = model(input_ids=data[0], token_type_ids=data[1], attention_mask=data[2], start_positions=data[3], end_positions=data[4])\n",
        "\n",
        "        # Choose the most probable start position / end position\n",
        "        start_index = torch.argmax(output.start_logits, dim=1)\n",
        "        end_index = torch.argmax(output.end_logits, dim=1)\n",
        "        \n",
        "        # Prediction is correct only if both start_index and end_index are correct\n",
        "        train_acc += ((start_index == data[3]) & (end_index == data[4])).float().mean()\n",
        "        train_loss += output.loss\n",
        "        \n",
        "        if fp16_training:\n",
        "            accelerator.backward(output.loss)\n",
        "        else:\n",
        "            output.loss.backward()\n",
        "        \n",
        "        optimizer.step()\n",
        "        optimizer.zero_grad()\n",
        "        step += 1\n",
        "\n",
        "        ##### TODO: Apply linear learning rate decay #####\n",
        "        \n",
        "        \n",
        "        # Print training loss and accuracy over past logging step\n",
        "        if step % logging_step == 0:\n",
        "            print(f\"Epoch {epoch + 1} | Step {step} | loss = {train_loss.item() / logging_step:.3f}, acc = {train_acc / logging_step:.3f}\")\n",
        "            train_loss = train_acc = 0\n",
        "\n",
        "    if validation:\n",
        "        print(\"Evaluating Dev Set ...\")\n",
        "        model.eval()\n",
        "        with torch.no_grad():\n",
        "            dev_acc = 0\n",
        "            for i, data in enumerate(tqdm(dev_loader)):\n",
        "                output = model(input_ids=data[0].squeeze(dim=0).to(device), token_type_ids=data[1].squeeze(dim=0).to(device),\n",
        "                       attention_mask=data[2].squeeze(dim=0).to(device))\n",
        "                # prediction is correct only if answer text exactly matches\n",
        "                dev_acc += evaluate(data, output) == dev_questions[i][\"answer_text\"]\n",
        "            print(f\"Validation | Epoch {epoch + 1} | acc = {dev_acc / len(dev_loader):.3f}\")\n",
        "        model.train()\n",
        "\n",
        "# Save a model and its configuration file to the directory 「saved_model」 \n",
        "# i.e. there are two files under the direcory 「saved_model」: 「pytorch_model.bin」 and 「config.json」\n",
        "# Saved model can be re-loaded using 「model = BertForQuestionAnswering.from_pretrained(\"saved_model\")」\n",
        "print(\"Saving Model ...\")\n",
        "model_save_dir = \"saved_model\" \n",
        "model.save_pretrained(model_save_dir)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kMmdLOKBMsdE"
      },
      "source": [
        "## Testing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U5scNKC9xz0C"
      },
      "source": [
        "print(\"Evaluating Test Set ...\")\n",
        "\n",
        "result = []\n",
        "\n",
        "model.eval()\n",
        "with torch.no_grad():\n",
        "    for data in tqdm(test_loader):\n",
        "        output = model(input_ids=data[0].squeeze(dim=0).to(device), token_type_ids=data[1].squeeze(dim=0).to(device),\n",
        "                       attention_mask=data[2].squeeze(dim=0).to(device))\n",
        "        result.append(evaluate(data, output))\n",
        "\n",
        "result_file = \"result.csv\"\n",
        "with open(result_file, 'w') as f:\t\n",
        "\t  f.write(\"ID,Answer\\n\")\n",
        "\t  for i, test_question in enumerate(test_questions):\n",
        "        # Replace commas in answers with empty strings (since csv is separated by comma)\n",
        "        # Answers in kaggle are processed in the same way\n",
        "\t\t    f.write(f\"{test_question['id']},{result[i].replace(',','')}\\n\")\n",
        "\n",
        "print(f\"Completed! Result is in {result_file}\")"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}