![](\"./figures/T0-fig-training-structure.png\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fb3272eb",
+ "metadata": {},
+ "source": [
+ "### 2.2 trainer 和 evaluator 的参数匹配\n",
+ "\n",
+ "在`fastNLP 1.0`中，参数匹配涉及到两个方面，分别是在\n",
+ "\n",
+ " 一方面，**在模型的前向传播中**，**dataloader 向 train_step 或 evaluate_step 函数传递 batch**\n",
+ "\n",
+ " 另方面，**在模型的评测过程中**，**evaluate_dataloader 向 metric 的 update 函数传递 batch**\n",
+ "\n",
+ "对于前者，在`Trainer`和`Evaluator`中的参数`model_wo_auto_param_call`被设置为`False`时\n",
+ "\n",
+ " **fastNLP 1.0 要求 dataloader 生成的每个 batch **，**满足如 {\"x\": x, \"y\": y} 的形式**\n",
+ "\n",
+ " 同时，`fastNLP 1.0`会查看模型的`train_step`和`evaluate_step`方法的参数签名，并为对应参数传入对应数值\n",
+ "\n",
+ " **字典形式的定义**，**对应在 Dataset 定义的 \\_\\_getitem\\_\\_ 方法中**，例如下方的`ArgMaxDatset`\n",
+ "\n",
+ " 而在`Trainer`和`Evaluator`中的参数`model_wo_auto_param_call`被设置为`True`时\n",
+ "\n",
+ " `fastNLP 1.0`会将`batch`直接传给模型的`train_step`、`evaluate_step`或`forward`函数\n",
+ "\n",
+ "```python\n",
+ "class Dataset(torch.utils.data.Dataset):\n",
+ " def __init__(self, x, y):\n",
+ " self.x = x\n",
+ " self.y = y\n",
+ "\n",
+ " def __len__(self):\n",
+ " return len(self.x)\n",
+ "\n",
+ " def __getitem__(self, item):\n",
+ " return {\"x\": self.x[item], \"y\": self.y[item]}\n",
+ "```"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f5f1a6aa",
+ "metadata": {},
+ "source": [
+ "对于后者，首先要明确，在`Trainer`和`Evaluator`中，`metrics`的计算分为`update`和`get_metric`两步\n",
+ "\n",
+ " **update 函数**，**针对一个 batch 的预测结果**，计算其累计的评价指标\n",
+ "\n",
+ " **get_metric 函数**，**统计 update 函数累计的评价指标**，来计算最终的评价结果\n",
+ "\n",
+ " 例如对于`Accuracy`来说，`update`函数会更新一个`batch`的正例数量`right_num`和负例数量`total_num`\n",
+ "\n",
+ " 而`get_metric`函数则会返回所有`batch`的评测值`right_num / total_num`\n",
+ "\n",
+ " 在此基础上，**fastNLP 1.0 要求 evaluate_dataloader 生成的每个 batch 传递给对应的 metric**\n",
+ "\n",
+ " **以 {\"pred\": y_pred, \"target\": y_true} 的形式**，对应其`update`函数的函数签名\n",
+ "\n",
+ "![](\"./figures/T0-fig-parameter-matching.png\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f62b7bb1",
+ "metadata": {},
+ "source": [
+ "### 2.3 示例：argmax 模型的搭建\n",
+ "\n",
+ "下文将通过训练`argmax`模型，简单介绍如何`Trainer`模块的使用方式\n",
+ "\n",
+ " 首先，使用`pytorch.nn.Module`定义`argmax`模型，目标是输入一组固定维度的向量，输出其中数值最大的数的索引"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "5314482b",
+ "metadata": {
+ "pycharm": {
+ "is_executing": true
+ }
+ },
+ "outputs": [],
+ "source": [
+ "import torch\n",
+ "import torch.nn as nn\n",
+ "\n",
+ "class ArgMaxModel(nn.Module):\n",
+ " def __init__(self, num_labels, feature_dimension):\n",
+ " nn.Module.__init__(self)\n",
+ " self.num_labels = num_labels\n",
+ "\n",
+ " self.linear1 = nn.Linear(in_features=feature_dimension, out_features=10)\n",
+ " self.ac1 = nn.ReLU()\n",
+ " self.linear2 = nn.Linear(in_features=10, out_features=10)\n",
+ " self.ac2 = nn.ReLU()\n",
+ " self.output = nn.Linear(in_features=10, out_features=num_labels)\n",
+ " self.loss_fn = nn.CrossEntropyLoss()\n",
+ "\n",
+ " def forward(self, x):\n",
+ " pred = self.ac1(self.linear1(x))\n",
+ " pred = self.ac2(self.linear2(pred))\n",
+ " pred = self.output(pred)\n",
+ " return pred\n",
+ "\n",
+ " def train_step(self, x, y):\n",
+ " pred = self(x)\n",
+ " return {\"loss\": self.loss_fn(pred, y)}\n",
+ "\n",
+ " def evaluate_step(self, x, y):\n",
+ " pred = self(x)\n",
+ " pred = torch.max(pred, dim=-1)[1]\n",
+ " return {\"pred\": pred, \"target\": y}"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "71f3fa6b",
+ "metadata": {},
+ "source": [
+ " 接着，使用`torch.utils.data.Dataset`定义`ArgMaxDataset`数据集\n",
+ "\n",
+ " 数据集包含三个参数：维度`feature_dimension`、数据量`data_num`和随机种子`seed`\n",
+ "\n",
+ " 数据及初始化是，自动生成指定维度的向量，并为每个向量标注出其中最大值的索引作为预测标签"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "fe612e61",
+ "metadata": {
+ "pycharm": {
+ "is_executing": false
+ }
+ },
+ "outputs": [],
+ "source": [
+ "from torch.utils.data import Dataset\n",
+ "\n",
+ "class ArgMaxDataset(Dataset):\n",
+ " def __init__(self, feature_dimension, data_num=1000, seed=0):\n",
+ " self.num_labels = feature_dimension\n",
+ " self.feature_dimension = feature_dimension\n",
+ " self.data_num = data_num\n",
+ " self.seed = seed\n",
+ "\n",
+ " g = torch.Generator()\n",
+ " g.manual_seed(1000)\n",
+ " self.x = torch.randint(low=-100, high=100, size=[data_num, feature_dimension], generator=g).float()\n",
+ " self.y = torch.max(self.x, dim=-1)[1]\n",
+ "\n",
+ " def __len__(self):\n",
+ " return self.data_num\n",
+ "\n",
+ " def __getitem__(self, item):\n",
+ " return {\"x\": self.x[item], \"y\": self.y[item]}"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "2cb96332",
+ "metadata": {},
+ "source": [
+ " 然后，根据`ArgMaxModel`类初始化模型实例，保持输入维度`feature_dimension`和输出标签数量`num_labels`一致\n",
+ "\n",
+ " 再根据`ArgMaxDataset`类初始化两个数据集实例，分别用来模型测试和模型评测，数据量各1000笔"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "76172ef8",
+ "metadata": {
+ "pycharm": {
+ "is_executing": false
+ }
+ },
+ "outputs": [],
+ "source": [
+ "model = ArgMaxModel(num_labels=10, feature_dimension=10)\n",
+ "\n",
+ "train_dataset = ArgMaxDataset(feature_dimension=10, data_num=1000)\n",
+ "evaluate_dataset = ArgMaxDataset(feature_dimension=10, data_num=100)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "4e7d25ee",
+ "metadata": {},
+ "source": [
+ " 此外，使用`torch.utils.data.DataLoader`初始化两个数据加载模块，批量大小同为8，分别用于训练和测评"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "363b5b09",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from torch.utils.data import DataLoader\n",
+ "\n",
+ "train_dataloader = DataLoader(train_dataset, batch_size=8, shuffle=True)\n",
+ "evaluate_dataloader = DataLoader(evaluate_dataset, batch_size=8)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "c8d4443f",
+ "metadata": {},
+ "source": [
+ " 最后，使用`torch.optim.SGD`初始化一个优化模块，基于随机梯度下降法"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "dc28a2d9",
+ "metadata": {
+ "pycharm": {
+ "is_executing": false
+ }
+ },
+ "outputs": [],
+ "source": [
+ "from torch.optim import SGD\n",
+ "\n",
+ "optimizer = SGD(model.parameters(), lr=0.001)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "eb8ca6cf",
+ "metadata": {},
+ "source": [
+ "## 3. 使用 fastNLP 1.0 训练 argmax 模型\n",
+ "\n",
+ "### 3.1 trainer 外部初始化的 evaluator"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "55145553",
+ "metadata": {},
+ "source": [
+ "通过从`fastNLP`库中导入`Trainer`类，初始化`trainer`实例，对模型进行训练\n",
+ "\n",
+ " 需要导入预先定义好的模型`model`、对应的数据加载模块`train_dataloader`、优化模块`optimizer`\n",
+ "\n",
+ " 通过`progress_bar`设定进度条格式，默认为`\"auto\"`，此外还有`\"rich\"`、`\"raw\"`和`None`\n",
+ "\n",
+ " 但对于`\"auto\"`和`\"rich\"`格式，在`jupyter`中，进度条会在训练结束后会被丢弃\n",
+ "\n",
+ " 通过`n_epochs`设定优化迭代轮数，默认为20；全部`Trainer`的全部变量与函数可以通过`dir(trainer)`查询"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "b51b7a2d",
+ "metadata": {
+ "pycharm": {
+ "is_executing": false
+ }
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " SentenceId Sentence Sentiment\n",
+ "0 1 A series of escapades demonstrating the adage ... negative\n",
+ "1 2 This quiet , introspective and entertaining in... positive\n",
+ "2 3 Even fans of Ismail Merchant 's work , I suspe... negative\n",
+ "3 4 A positively thrilling combination of ethnogra... neutral\n",
+ "4 5 A comedy-drama of nearly epic proportions root... positive\n",
+ "5 6 The Importance of Being Earnest , so thick wit... neutral"
+ ]
+ },
+ "execution_count": 24,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "import pandas as pd\n",
+ "\n",
+ "df = pd.read_csv('./data/test4dataset.tsv', sep='\\t')\n",
+ "df"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "919ab350",
+ "metadata": {},
+ "source": [
+ "接着，通过`dataset`中的`from_pandas`方法填充数据集，并使用`apply_more`方法对文本进行分词操作"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 25,
+ "id": "4f634586",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ "Processing: 0%| | 0/6 [00:00': 0, '| \n", + " | SentenceId | \n", + "Sentence | \n", + "Sentiment | \n", + "
|---|---|---|---|
| 0 | \n", + "1 | \n", + "A series of escapades demonstrating the adage ... | \n", + "negative | \n", + "
| 1 | \n", + "2 | \n", + "This quiet , introspective and entertaining in... | \n", + "positive | \n", + "
| 2 | \n", + "3 | \n", + "Even fans of Ismail Merchant 's work , I suspe... | \n", + "negative | \n", + "
| 3 | \n", + "4 | \n", + "A positively thrilling combination of ethnogra... | \n", + "neutral | \n", + "
| 4 | \n", + "5 | \n", + "A comedy-drama of nearly epic proportions root... | \n", + "positive | \n", + "
| 5 | \n", + "6 | \n", + "The Importance of Being Earnest , so thick wit... | \n", + "neutral | \n", + "
![](\"./figures/T1-fig-dataset-and-vocabulary.png\")
"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "35b4f0f7",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.7.13"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/docs/source/tutorials/fastnlp_tutorial_2.ipynb b/docs/source/tutorials/fastnlp_tutorial_2.ipynb
new file mode 100644
index 00000000..546e471d
--- /dev/null
+++ b/docs/source/tutorials/fastnlp_tutorial_2.ipynb
@@ -0,0 +1,884 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# T2. databundle 和 tokenizer 的基本使用\n",
+ "\n",
+ " 1 fastNLP 中 dataset 的延伸\n",
+ "\n",
+ " 1.1 databundle 的概念与使用\n",
+ "\n",
+ " 2 fastNLP 中的 tokenizer\n",
+ " \n",
+ " 2.1 PreTrainedTokenizer 的概念\n",
+ "\n",
+ " 2.2 BertTokenizer 的基本使用\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 1. fastNLP 中 dataset 的延伸\n",
+ "\n",
+ "### 1.1 databundle 的概念与使用\n",
+ "\n",
+ "在`fastNLP 1.0`中，在常用的数据加载模块`DataLoader`和数据集`DataSet`模块之间，还存在\n",
+ "\n",
+ " 一个中间模块，即 **数据包 DataBundle 模块**，可以从`fastNLP.io`路径中导入该模块\n",
+ "\n",
+ "在`fastNLP 1.0`中，**一个 databundle 数据包包含若干 dataset 数据集和 vocabulary 词汇表**\n",
+ "\n",
+ " 分别存储在`datasets`和`vocabs`两个变量中，所以了解`databundle`数据包之前\n",
+ "\n",
+ "需要首先**复习 dataset 数据集和 vocabulary 词汇表**，**下面的一串代码**，**你知道其大概含义吗？**\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Processing: 0%| | 0/6 [00:00': 0, '
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Processing: 0%| | 0/4 [00:00\n", + "
/remote-home/xrliu/anaconda3/envs/demo/lib/python3.7/site-packages/ipywidgets/widgets/widget_\n", + "output.py:111: DeprecationWarning: Kernel._parent_header is deprecated in ipykernel 6. Use \n", + ".get_parent()\n", + " if ip and hasattr(ip, 'kernel') and hasattr(ip.kernel, '_parent_header'):\n", + "\n" + ], + "text/plain": [ + "/remote-home/xrliu/anaconda3/envs/demo/lib/python3.7/site-packages/ipywidgets/widgets/widget_\n", + "output.py:111: DeprecationWarning: Kernel._parent_header is deprecated in ipykernel 6. Use \n", + ".get_parent()\n", + " if ip and hasattr(ip, 'kernel') and hasattr(ip.kernel, '_parent_header'):\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
/remote-home/xrliu/anaconda3/envs/demo/lib/python3.7/site-packages/ipywidgets/widgets/widget_\n", + "output.py:112: DeprecationWarning: Kernel._parent_header is deprecated in ipykernel 6. Use \n", + ".get_parent()\n", + " self.msg_id = ip.kernel._parent_header['header']['msg_id']\n", + "\n" + ], + "text/plain": [ + "/remote-home/xrliu/anaconda3/envs/demo/lib/python3.7/site-packages/ipywidgets/widgets/widget_\n", + "output.py:112: DeprecationWarning: Kernel._parent_header is deprecated in ipykernel 6. Use \n", + ".get_parent()\n", + " self.msg_id = ip.kernel._parent_header['header']['msg_id']\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "\n" + ], + "text/plain": [] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Output()" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:1, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m1\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.525,\n", + " \"total#acc\": 160.0,\n", + " \"correct#acc\": 84.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.525\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m84.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:2, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m2\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.54375,\n", + " \"total#acc\": 160.0,\n", + " \"correct#acc\": 87.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.54375\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m87.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:3, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m3\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.55,\n", + " \"total#acc\": 160.0,\n", + " \"correct#acc\": 88.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.55\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m88.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:4, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m4\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.625,\n", + " \"total#acc\": 160.0,\n", + " \"correct#acc\": 100.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.625\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m100.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:5, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m5\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.65,\n", + " \"total#acc\": 160.0,\n", + " \"correct#acc\": 104.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.65\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m104.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:6, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m6\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.69375,\n", + " \"total#acc\": 160.0,\n", + " \"correct#acc\": 111.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.69375\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m111.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:7, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m7\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.675,\n", + " \"total#acc\": 160.0,\n", + " \"correct#acc\": 108.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.675\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m108.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:8, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m8\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.66875,\n", + " \"total#acc\": 160.0,\n", + " \"correct#acc\": 107.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.66875\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m107.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:9, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m9\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.675,\n", + " \"total#acc\": 160.0,\n", + " \"correct#acc\": 108.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.675\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m108.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
---------------------------- Eval. results on Epoch:10, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "---------------------------- Eval. results on Epoch:\u001b[1;36m10\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.68125,\n", + " \"total#acc\": 160.0,\n", + " \"correct#acc\": 109.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.68125\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m109.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "\n" + ], + "text/plain": [] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "trainer.run(num_eval_batch_per_dl=10)" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "8bc4bfb2", + "metadata": {}, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Output()" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "\n" + ], + "text/plain": [] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/plain": [ + "{'acc#acc': 0.712222, 'total#acc': 900.0, 'correct#acc': 641.0}" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "trainer.evaluator.run()" + ] + }, + { + "cell_type": "markdown", + "id": "07538876", + "metadata": {}, + "source": [ + " 注：此处使用`gc`模块删除相关变量，释放内存，为接下来新的模型训练预留存储空间，下同" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "1b52eafd", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "383" + ] + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "import gc\n", + "\n", + "del model\n", + "del trainer\n", + "del dataset\n", + "del sst2data\n", + "\n", + "gc.collect()" + ] + }, + { + "cell_type": "markdown", + "id": "d9443213", + "metadata": {}, + "source": [ + "## 2. fastNLP 中 models 模块的介绍\n", + "\n", + "### 2.1 示例一：models 实现 CNN 分类\n", + "\n", + " 本示例使用`fastNLP 1.0`中预定义模型`models`中的`CNNText`模型，实现`SST-2`文本二分类任务\n", + "\n", + "数据使用方面，此处沿用在上个示例中展示的`SST-2`数据集，数据加载过程相同且已经执行过了，因此简略\n", + "\n", + "模型使用方面，如上所述，这里使用**基于卷积神经网络 CNN 的预定义文本分类模型 CNNText**，结构如下所示\n", + "\n", + " 首先是内置的`100`维嵌入层、`dropout`层、紧接着是三个一维卷积，将`100`维嵌入特征，分别通过\n", + "\n", + " **感受野为 1 、 3 、 5 的卷积算子变换至 30 维、 40 维、 50 维的卷积特征**，再将三者拼接\n", + "\n", + " 最终再次通过`dropout`层、线性变换层，映射至二元的输出值，对应两个分类结果上的几率`logits`\n", + "\n", + "```\n", + "CNNText(\n", + " (embed): Embedding(\n", + " (embed): Embedding(5194, 100)\n", + " (dropout): Dropout(p=0.0, inplace=False)\n", + " )\n", + " (conv_pool): ConvMaxpool(\n", + " (convs): ModuleList(\n", + " (0): Conv1d(100, 30, kernel_size=(1,), stride=(1,), bias=False)\n", + " (1): Conv1d(100, 40, kernel_size=(3,), stride=(1,), padding=(1,), bias=False)\n", + " (2): Conv1d(100, 50, kernel_size=(5,), stride=(1,), padding=(2,), bias=False)\n", + " )\n", + " )\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " (fc): Linear(in_features=120, out_features=2, bias=True)\n", + ")\n", + "```\n", + "\n", + "对应到代码上，**从 fastNLP.models.torch 路径下导入 CNNText**，初始化`CNNText`和`optimizer`实例\n", + "\n", + " 注意：初始化`CNNText`时，**二元组参数 embed 、分类数量 num_classes 是必须传入的**，其中\n", + "\n", + " **embed 表示嵌入层的嵌入抽取矩阵大小**，因此第二个元素对应的是默认隐藏层维度 `100` 维" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "f6e76e2e", + "metadata": {}, + "outputs": [], + "source": [ + "from fastNLP.models.torch import CNNText\n", + "\n", + "model = CNNText(embed=(len(vocab), 100), num_classes=2, dropout=0.1)\n", + "\n", + "from torch.optim import AdamW\n", + "\n", + "optimizers = AdamW(params=model.parameters(), lr=5e-4)" + ] + }, + { + "cell_type": "markdown", + "id": "0cc5ca10", + "metadata": {}, + "source": [ + " 最后，使用`trainer`模块，集成`model`、`optimizer`、`dataloader`、`metric`训练" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "50a13ee5", + "metadata": {}, + "outputs": [], + "source": [ + "from fastNLP import Trainer, Accuracy\n", + "\n", + "trainer = Trainer(\n", + " model=model,\n", + " driver='torch',\n", + " device=0, # 'cuda'\n", + " n_epochs=10,\n", + " optimizers=optimizers,\n", + " train_dataloader=train_dataloader,\n", + " evaluate_dataloaders=evaluate_dataloader,\n", + " metrics={'acc': Accuracy()}\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "28903a7d", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
[16:21:57] INFO Running evaluator sanity check for 2 batches. trainer.py:596\n", + "\n" + ], + "text/plain": [ + "\u001b[2;36m[16:21:57]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m Running evaluator sanity check for \u001b[1;36m2\u001b[0m batches. \u001b]8;id=813103;file://../fastNLP/core/controllers/trainer.py\u001b\\\u001b[2mtrainer.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=271516;file://../fastNLP/core/controllers/trainer.py#596\u001b\\\u001b[2m596\u001b[0m\u001b]8;;\u001b\\\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Output()" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "\n" + ], + "text/plain": [] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Output()" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:1, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m1\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.654444,\n", + " \"total#acc\": 900.0,\n", + " \"correct#acc\": 589.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.654444\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m900.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m589.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:2, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m2\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.767778,\n", + " \"total#acc\": 900.0,\n", + " \"correct#acc\": 691.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.767778\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m900.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m691.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:3, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m3\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.797778,\n", + " \"total#acc\": 900.0,\n", + " \"correct#acc\": 718.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.797778\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m900.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m718.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:4, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m4\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.803333,\n", + " \"total#acc\": 900.0,\n", + " \"correct#acc\": 723.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.803333\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m900.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m723.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:5, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m5\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.807778,\n", + " \"total#acc\": 900.0,\n", + " \"correct#acc\": 727.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.807778\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m900.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m727.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:6, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m6\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.812222,\n", + " \"total#acc\": 900.0,\n", + " \"correct#acc\": 731.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.812222\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m900.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m731.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:7, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m7\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.804444,\n", + " \"total#acc\": 900.0,\n", + " \"correct#acc\": 724.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.804444\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m900.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m724.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:8, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m8\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.811111,\n", + " \"total#acc\": 900.0,\n", + " \"correct#acc\": 730.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.811111\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m900.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m730.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:9, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m9\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.811111,\n", + " \"total#acc\": 900.0,\n", + " \"correct#acc\": 730.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.811111\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m900.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m730.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
---------------------------- Eval. results on Epoch:10, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "---------------------------- Eval. results on Epoch:\u001b[1;36m10\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.806667,\n", + " \"total#acc\": 900.0,\n", + " \"correct#acc\": 726.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.806667\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m900.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m726.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "\n" + ], + "text/plain": [] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "trainer.run()" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "f47a6a35", + "metadata": {}, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Output()" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "\n" + ], + "text/plain": [] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/plain": [ + "{'acc#acc': 0.806667, 'total#acc': 900.0, 'correct#acc': 726.0}" + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "trainer.evaluator.run()" + ] + }, + { + "cell_type": "markdown", + "id": "5b5c0446", + "metadata": {}, + "source": [ + " 注：此处使用`gc`模块删除相关变量，释放内存，为接下来新的模型训练预留存储空间，下同" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "e9e70f88", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "344" + ] + }, + "execution_count": 14, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "import gc\n", + "\n", + "del model\n", + "del trainer\n", + "\n", + "gc.collect()" + ] + }, + { + "cell_type": "markdown", + "id": "6aec2a19", + "metadata": {}, + "source": [ + "### 2.2 示例二：models 实现 BiLSTM 标注\n", + "\n", + " 通过两个示例一的对比可以发现，得益于`models`对模型结构的封装，使用`models`明显更加便捷\n", + "\n", + " 针对更加复杂的模型时，编码更加轻松；本示例将使用`models`中的`BiLSTMCRF`模型\n", + "\n", + " 避免`CRF`和`Viterbi`算法代码书写的困难，轻松实现`CoNLL-2003`中的命名实体识别`NER`任务\n", + "\n", + "模型使用方面，如上所述，这里使用**基于双向 LSTM +条件随机场 CRF 的标注模型 BiLSTMCRF**，结构如下所示\n", + "\n", + " 其中，隐藏层维度默认`100`维，因此对应双向`LSTM`输出`200`维，`dropout`层退学概率、`LSTM`层数可调\n", + "\n", + "```\n", + "BiLSTMCRF(\n", + " (embed): Embedding(7590, 100)\n", + " (lstm): LSTM(\n", + " (lstm): LSTM(100, 100, batch_first=True, bidirectional=True)\n", + " )\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " (fc): Linear(in_features=200, out_features=9, bias=True)\n", + " (crf): ConditionalRandomField()\n", + ")\n", + "```\n", + "\n", + "数据使用方面，此处仍然**使用 datasets 模块中的 load_dataset 函数**，以如下形式，加载`CoNLL-2003`数据集\n", + "\n", + " 首次下载后会保存至`~.cache/huggingface/datasets/conll2003/conll2003/1.0.0/`目录下" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "03e66686", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Reusing dataset conll2003 (/remote-home/xrliu/.cache/huggingface/datasets/conll2003/conll2003/1.0.0/63f4ebd1bcb7148b1644497336fd74643d4ce70123334431a3c053b7ee4e96ee)\n" + ] + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "593bc03ed5914953ab94268ff2f01710", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + " 0%| | 0/3 [00:00[16:23:41] INFO Running evaluator sanity check for 2 batches. trainer.py:596\n", + "\n" + ], + "text/plain": [ + "\u001b[2;36m[16:23:41]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m Running evaluator sanity check for \u001b[1;36m2\u001b[0m batches. \u001b]8;id=565652;file://../fastNLP/core/controllers/trainer.py\u001b\\\u001b[2mtrainer.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=224849;file://../fastNLP/core/controllers/trainer.py#596\u001b\\\u001b[2m596\u001b[0m\u001b]8;;\u001b\\\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Output()" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "\n" + ], + "text/plain": [] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Output()" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:1, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m1\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"f#F1\": 0.169014,\n", + " \"pre#F1\": 0.170732,\n", + " \"rec#F1\": 0.167331\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"f#F1\"\u001b[0m: \u001b[1;36m0.169014\u001b[0m,\n", + " \u001b[1;34m\"pre#F1\"\u001b[0m: \u001b[1;36m0.170732\u001b[0m,\n", + " \u001b[1;34m\"rec#F1\"\u001b[0m: \u001b[1;36m0.167331\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:2, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m2\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"f#F1\": 0.361809,\n", + " \"pre#F1\": 0.312139,\n", + " \"rec#F1\": 0.430279\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"f#F1\"\u001b[0m: \u001b[1;36m0.361809\u001b[0m,\n", + " \u001b[1;34m\"pre#F1\"\u001b[0m: \u001b[1;36m0.312139\u001b[0m,\n", + " \u001b[1;34m\"rec#F1\"\u001b[0m: \u001b[1;36m0.430279\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:3, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m3\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"f#F1\": 0.525,\n", + " \"pre#F1\": 0.475728,\n", + " \"rec#F1\": 0.585657\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"f#F1\"\u001b[0m: \u001b[1;36m0.525\u001b[0m,\n", + " \u001b[1;34m\"pre#F1\"\u001b[0m: \u001b[1;36m0.475728\u001b[0m,\n", + " \u001b[1;34m\"rec#F1\"\u001b[0m: \u001b[1;36m0.585657\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:4, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m4\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"f#F1\": 0.627306,\n", + " \"pre#F1\": 0.584192,\n", + " \"rec#F1\": 0.677291\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"f#F1\"\u001b[0m: \u001b[1;36m0.627306\u001b[0m,\n", + " \u001b[1;34m\"pre#F1\"\u001b[0m: \u001b[1;36m0.584192\u001b[0m,\n", + " \u001b[1;34m\"rec#F1\"\u001b[0m: \u001b[1;36m0.677291\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:5, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m5\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"f#F1\": 0.710937,\n", + " \"pre#F1\": 0.697318,\n", + " \"rec#F1\": 0.7251\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"f#F1\"\u001b[0m: \u001b[1;36m0.710937\u001b[0m,\n", + " \u001b[1;34m\"pre#F1\"\u001b[0m: \u001b[1;36m0.697318\u001b[0m,\n", + " \u001b[1;34m\"rec#F1\"\u001b[0m: \u001b[1;36m0.7251\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:6, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m6\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"f#F1\": 0.739563,\n", + " \"pre#F1\": 0.738095,\n", + " \"rec#F1\": 0.741036\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"f#F1\"\u001b[0m: \u001b[1;36m0.739563\u001b[0m,\n", + " \u001b[1;34m\"pre#F1\"\u001b[0m: \u001b[1;36m0.738095\u001b[0m,\n", + " \u001b[1;34m\"rec#F1\"\u001b[0m: \u001b[1;36m0.741036\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:7, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m7\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"f#F1\": 0.748491,\n", + " \"pre#F1\": 0.756098,\n", + " \"rec#F1\": 0.741036\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"f#F1\"\u001b[0m: \u001b[1;36m0.748491\u001b[0m,\n", + " \u001b[1;34m\"pre#F1\"\u001b[0m: \u001b[1;36m0.756098\u001b[0m,\n", + " \u001b[1;34m\"rec#F1\"\u001b[0m: \u001b[1;36m0.741036\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:8, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m8\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"f#F1\": 0.716763,\n", + " \"pre#F1\": 0.69403,\n", + " \"rec#F1\": 0.741036\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"f#F1\"\u001b[0m: \u001b[1;36m0.716763\u001b[0m,\n", + " \u001b[1;34m\"pre#F1\"\u001b[0m: \u001b[1;36m0.69403\u001b[0m,\n", + " \u001b[1;34m\"rec#F1\"\u001b[0m: \u001b[1;36m0.741036\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:9, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m9\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"f#F1\": 0.768293,\n", + " \"pre#F1\": 0.784232,\n", + " \"rec#F1\": 0.752988\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"f#F1\"\u001b[0m: \u001b[1;36m0.768293\u001b[0m,\n", + " \u001b[1;34m\"pre#F1\"\u001b[0m: \u001b[1;36m0.784232\u001b[0m,\n", + " \u001b[1;34m\"rec#F1\"\u001b[0m: \u001b[1;36m0.752988\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
---------------------------- Eval. results on Epoch:10, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "---------------------------- Eval. results on Epoch:\u001b[1;36m10\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"f#F1\": 0.757692,\n", + " \"pre#F1\": 0.732342,\n", + " \"rec#F1\": 0.784861\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"f#F1\"\u001b[0m: \u001b[1;36m0.757692\u001b[0m,\n", + " \u001b[1;34m\"pre#F1\"\u001b[0m: \u001b[1;36m0.732342\u001b[0m,\n", + " \u001b[1;34m\"rec#F1\"\u001b[0m: \u001b[1;36m0.784861\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "\n" + ], + "text/plain": [] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "trainer.run(num_eval_batch_per_dl=10)" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "37871d6b", + "metadata": {}, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Output()" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "\n" + ], + "text/plain": [] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/plain": [ + "{'f#F1': 0.766798, 'pre#F1': 0.741874, 'rec#F1': 0.793456}" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "trainer.evaluator.run()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "96bae094", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/docs/source/tutorials/fastnlp_tutorial_5.ipynb b/docs/source/tutorials/fastnlp_tutorial_5.ipynb new file mode 100644 index 00000000..ab759feb --- /dev/null +++ b/docs/source/tutorials/fastnlp_tutorial_5.ipynb @@ -0,0 +1,1242 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "fdd7ff16", + "metadata": {}, + "source": [ + "# T5. trainer 和 evaluator 的深入介绍\n", + "\n", + " 1 fastNLP 中 driver 的补充介绍\n", + " \n", + " 1.1 trainer 和 driver 的构想 \n", + "\n", + " 1.2 device 与 多卡训练\n", + "\n", + " 2 fastNLP 中的更多 metric 类型\n", + "\n", + " 2.1 预定义的 metric 类型\n", + "\n", + " 2.2 自定义的 metric 类型\n", + "\n", + " 3 fastNLP 中 trainer 的补充介绍\n", + "\n", + " 3.1 trainer 的内部结构" + ] + }, + { + "cell_type": "markdown", + "id": "08752c5a", + "metadata": { + "pycharm": { + "name": "#%% md\n" + } + }, + "source": [ + "## 1. fastNLP 中 driver 的补充介绍\n", + "\n", + "### 1.1 trainer 和 driver 的构想\n", + "\n", + "在`fastNLP 1.0`中，模型训练最关键的模块便是**训练模块 trainer 、评测模块 evaluator 、驱动模块 driver**，\n", + "\n", + " 在`tutorial 0`中，已经简单介绍过上述三个模块：**driver 用来控制训练评测中的 model 的最终运行**\n", + "\n", + " **evaluator 封装评测的 metric**，**trainer 封装训练的 optimizer**，**也可以包括 evaluator**\n", + "\n", + "之所以做出上述的划分，其根本目的在于要**达成对于多个 python 学习框架**，**例如 pytorch 、 paddle 、 jittor 的兼容**\n", + "\n", + " 对于训练环节，其伪代码如下方左边紫色一栏所示，由于**不同框架对模型、损失、张量的定义各有不同**，所以将训练环节\n", + "\n", + " 划分为**框架无关的循环控制、批量分发部分**，**由 trainer 模块负责**实现，对应的伪代码如下方中间一栏所示\n", + "\n", + " 以及**随框架不同的模型调用、数值优化部分**，**由 driver 模块负责**实现，对应的伪代码如下方右边一栏所示\n", + "\n", + "|训练过程|框架无关 对应`Trainer`|框架相关 对应`Driver`\n", + "|----|----|----|\n", + "| try: | try: | |\n", + "| for epoch in 1:n_eoochs: | for epoch in 1:n_eoochs: | |\n", + "| for step in 1:total_steps: | for step in 1:total_steps: | |\n", + "| batch = fetch_batch() | batch = fetch_batch() | |\n", + "| loss = model.forward(batch) | | loss = model.forward(batch) |\n", + "| loss.backward() | | loss.backward() |\n", + "| model.clear_grad() | | model.clear_grad() |\n", + "| model.update() | | model.update() |\n", + "| if need_save: | if need_save: | |\n", + "| model.save() | | model.save() |\n", + "| except: | except: | |\n", + "| process_exception() | process_exception() | |" + ] + }, + { + "cell_type": "markdown", + "id": "3e55f07b", + "metadata": {}, + "source": [ + " 对于评测环节，其伪代码如下方左边紫色一栏所示，同样由于不同框架对模型、损失、张量的定义各有不同，所以将评测环节\n", + "\n", + " 划分为**框架无关的循环控制、分发汇总部分**，**由 evaluator 模块负责**实现，对应的伪代码如下方中间一栏所示\n", + "\n", + " 以及**随框架不同的模型调用、评测计算部分**，同样**由 driver 模块负责**实现，对应的伪代码如下方右边一栏所示\n", + "\n", + "|评测过程|框架无关 对应`Evaluator`|框架相关 对应`Driver`\n", + "|----|----|----|\n", + "| try: | try: | |\n", + "| model.set_eval() | model.set_eval() | |\n", + "| for step in 1:total_steps: | for step in 1:total_steps: | |\n", + "| batch = fetch_batch() | batch = fetch_batch() | |\n", + "| outputs = model.evaluate(batch) | | outputs = model.evaluate(batch) |\n", + "| metric.compute(batch, outputs) | | metric.compute(batch, outputs) |\n", + "| results = metric.get_metric() | results = metric.get_metric() | |\n", + "| except: | except: | |\n", + "| process_exception() | process_exception() | |" + ] + }, + { + "cell_type": "markdown", + "id": "94ba11c6", + "metadata": { + "pycharm": { + "name": "#%%\n" + } + }, + "source": [ + "由此，从程序员的角度，`fastNLP v1.0` **通过一个 driver 让基于 pytorch 、 paddle 、 jittor 、 oneflow 框架的模型**\n", + "\n", + " **都能在相同的 trainer 和 evaluator 上运行**，这也**是 fastNLP v1.0 相比于之前版本的一大亮点**\n", + "\n", + " 而从`driver`的角度，`fastNLP v1.0`通过定义一个`driver`基类，**将所有张量转化为 numpy.tensor**\n", + "\n", + " 并由此泛化出`torch_driver`、`paddle_driver`、`jittor_driver`三个子类，从而实现了\n", + "\n", + " 对`pytorch`、`paddle`、`jittor`的兼容，有关后两者的实践请参考接下来的`tutorial-6`" + ] + }, + { + "cell_type": "markdown", + "id": "ab1cea7d", + "metadata": {}, + "source": [ + "### 1.2 device 与 多卡训练\n", + "\n", + "**fastNLP v1.0 支持多卡训练**，实现方法则是**通过将 trainer 中的 device 设置为对应显卡的序号列表**\n", + "\n", + " 由单卡切换成多卡，无论是数据、模型还是评测都会面临一定的调整，`fastNLP v1.0`保证：\n", + "\n", + " 数据拆分时，不同卡之间相互协调，所有数据都可以被训练，且不会使用到相同的数据\n", + "\n", + " 模型训练时，模型之间需要交换梯度；评测计算时，每张卡先各自计算，再汇总结果\n", + "\n", + " 例如，在评测计算运行`get_metric`函数时，`fastNLP v1.0`将自动按照`self.right`和`self.total`\n", + "\n", + " 指定的 **aggregate_method 方法**，默认为`sum`，将每张卡上结果汇总起来，因此最终\n", + "\n", + " 在调用`get_metric`方法时，`Accuracy`类能够返回全部的统计结果，代码如下\n", + " \n", + "```python\n", + "trainer = Trainer(\n", + " model=model, # model 基于 pytorch 实现 \n", + " train_dataloader=train_dataloader,\n", + " optimizers=optimizer,\n", + " ...\n", + " driver='torch', # driver 使用 torch_driver \n", + " device=[0, 1], # gpu 选择 cuda:0 + cuda:1\n", + " ...\n", + " evaluate_dataloaders=evaluate_dataloader,\n", + " metrics={'acc': Accuracy()},\n", + " ...\n", + " )\n", + "\n", + "class Accuracy(Metric):\n", + " def __init__(self):\n", + " super().__init__()\n", + " self.register_element(name='total', value=0, aggregate_method='sum')\n", + " self.register_element(name='right', value=0, aggregate_method='sum')\n", + "```\n" + ] + }, + { + "cell_type": "markdown", + "id": "e2e0a210", + "metadata": { + "pycharm": { + "name": "#%%\n" + } + }, + "source": [ + "注：`fastNLP v1.0`中要求`jupyter`不能多卡，仅能单卡，故在所有`tutorial`中均不作相关演示" + ] + }, + { + "cell_type": "markdown", + "id": "8d19220c", + "metadata": {}, + "source": [ + "## 2. fastNLP 中的更多 metric 类型\n", + "\n", + "### 2.1 预定义的 metric 类型\n", + "\n", + "在`fastNLP 1.0`中，除了前几篇`tutorial`中经常见到的**正确率 Accuracy**，还有其他**预定义的评测标准 metric**\n", + "\n", + " 包括**所有 metric 的基类 Metric**、适配`Transformers`中相关模型的正确率`TransformersAccuracy`\n", + "\n", + " **适用于分类语境下的 F1 值 ClassifyFPreRecMetric**（其中也包括召回率`Pre`、精确率`Rec`\n", + "\n", + " **适用于抽取语境下的 F1 值 SpanFPreRecMetric**；相关基本信息内容见下表，之后是详细分析\n", + "\n", + "代码名称|简要介绍|代码路径\n", + "----|----|----|\n", + " `Metric` | 定义`metrics`时继承的基类 | `/core/metrics/metric.py` |\n", + " `Accuracy` | 正确率，最为常用 | `/core/metrics/accuracy.py` |\n", + " `TransformersAccuracy` | 正确率，为了兼容`Transformers`中相关模型 | `/core/metrics/accuracy.py` |\n", + " `ClassifyFPreRecMetric` | 召回率、精确率、F1值，适用于**分类问题** | `/core/metrics/classify_f1_pre_rec_metric.py` |\n", + " `SpanFPreRecMetric` | 召回率、精确率、F1值，适用于**抽取问题** | `/core/metrics/span_f1_pre_rec_metric.py` |" + ] + }, + { + "cell_type": "markdown", + "id": "fdc083a3", + "metadata": { + "pycharm": { + "name": "#%%\n" + } + }, + "source": [ + " 如`tutorial-0`中所述，所有的`metric`都包含`get_metric`和`update`函数，其中\n", + "\n", + " **update 函数更新单个 batch 的统计量**，**get_metric 函数返回最终结果**，并打印显示\n", + "\n", + "\n", + "### 2.1.1 Accuracy 与 TransformersAccuracy\n", + "\n", + "`Accuracy`，正确率，预测正确的数据`right_num`在总数据`total_num`，中的占比（公式就不用列了\n", + "\n", + " `get_metric`函数打印格式为 **{\"acc#xx\": float, 'total#xx': float, 'correct#xx': float}**\n", + "\n", + " 一般在初始化时不需要传参，`fastNLP`会根据`update`函数的传入参数确定对应后台框架`backend`\n", + "\n", + " **update 函数的参数包括 pred 、 target 、 seq_len**，**后者用来标记批次中每笔数据的长度**\n", + "\n", + "`TransformersAccuracy`，继承自`Accuracy`，只是为了兼容`Transformers`框架中相关模型\n", + "\n", + " 在`update`函数中，将`Transformers`框架输出的`attention_mask`参数转化为`seq_len`参数\n", + "\n", + "\n", + "### 2.1.2 ClassifyFPreRecMetric 与 SpanFPreRecMetric\n", + "\n", + "`ClassifyFPreRecMetric`，分类评价，`SpanFPreRecMetric`，抽取评价，后者在`tutorial-4`中已出现\n", + "\n", + " 两者的相同之处在于：**第一**，**都包括召回率/查全率 ec**、**精确率/查准率 Pre**、**F1 值**这三个指标\n", + "\n", + " `get_metric`函数打印格式为 **{\"f#xx\": float, 'pre#xx': float, 'rec#xx': float}**\n", + "\n", + " 三者的计算公式如下，其中`beta`默认为`1`，即`F1`值是召回率`Rec`和精确率`Pre`的调和平均数\n", + "\n", + "$$\\text{召回率}\\ Rec=\\dfrac{\\text{正确预测为正例的数量}}{\\text{所有本来是正例的数量}}\\qquad \\text{精确率}\\ Pre=\\dfrac{\\text{正确预测为正例的数量}}{\\text{所有预测为正例的数量}}$$\n", + "\n", + "$$F_{beta} = \\frac{(1 + {beta}^{2})*(Pre*Rec)}{({beta}^{2}*Pre + Rec)}$$\n", + "\n", + " **第二**，可以通过参数`only_gross`为`False`，要求返回所有类别的`Rec-Pre-F1`，同时`F1`值又根据参数`f_type`又分为\n", + "\n", + " **micro F1**（**直接统计所有类别的 Rec-Pre-F1**）、**macro F1**（**统计各类别的 Rec-Pre-F1 再算术平均**）\n", + "\n", + " **第三**，两者在初始化时还可以**传入基于 fastNLP.Vocabulary 的 tag_vocab 参数记录数据集中的标签序号**\n", + "\n", + " **与标签名称之间的映射**，通过字符串列表`ignore_labels`参数，指定若干标签不用于`Rec-Pre-F1`的计算\n", + "\n", + "两者的不同之处在于：`ClassifyFPreRecMetric`针对简单的分类问题，每个分类标签之间彼此独立，不构成标签对\n", + "\n", + " **SpanFPreRecMetric 针对更复杂的抽取问题**，**规定标签 B-xx 和 I-xx 或 B-xx 和 E-xx 构成标签对**\n", + "\n", + " 在计算`Rec-Pre-F1`时，`ClassifyFPreRecMetric`只需要考虑标签本身是否正确这就足够了，但是\n", + "\n", + " 对于`SpanFPreRecMetric`，需要保证**标签符合规则且覆盖的区间与正确结果重合才算正确**\n", + "\n", + " 因此回到`tutorial-4`中`CoNLL-2003`的`NER`任务，如果评测方法选择`ClassifyFPreRecMetric`\n", + "\n", + " 或者`Accuracy`，会发现虽然评测结果显示很高，这是因为选择的评测方法要求太低\n", + "\n", + " 最后通过`CoNLL-2003`的词性标注`POS`任务简单演示下`ClassifyFPreRecMetric`相关的使用\n", + "\n", + "```python\n", + "from fastNLP import Vocabulary\n", + "from fastNLP import ClassifyFPreRecMetric\n", + "\n", + "tag_vocab = Vocabulary(padding=None, unknown=None) # 记录序号与标签之间的映射\n", + "tag_vocab.add_word_lst(['\"', \"''\", '#', '$', '(', ')', ',', '.', ':', '``', \n", + " 'CC', 'CD', 'DT', 'EX', 'FW', 'IN', 'JJ', 'JJR', 'JJS', 'LS', \n", + " 'MD', 'NN', 'NNP', 'NNPS', 'NNS', 'NN|SYM', 'PDT', 'POS', 'PRP', 'PRP$', \n", + " 'RB', 'RBR', 'RBS', 'RP', 'SYM', 'TO', 'UH', 'VB', 'VBD', 'VBG', \n", + " 'VBN', 'VBP', 'VBZ', 'WDT', 'WP', 'WP+', 'WRB', ]) # CoNLL-2003 中的 pos_tags\n", + "ignore_labels = ['\"', \"''\", '#', '$', '(', ')', ',', '.', ':', '``', ]\n", + "\n", + "FPreRec = ClassifyFPreRecMetric(tag_vocab=tag_vocab, \n", + " ignore_labels=ignore_labels, # 表示评测/优化中不考虑上述标签的正误/损失\n", + " only_gross=True, # 默认为 True 表示输出所有类别的综合统计结果\n", + " f_type='micro') # 默认为 'micro' 表示统计所有类别的 Rec-Pre-F1\n", + "metrics = {'F1': FPreRec}\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "8a22f522", + "metadata": {}, + "source": [ + "### 2.2 自定义的 metric 类型\n", + "\n", + "如上文所述，`Metric`作为所有`metric`的基类，`Accuracy`等都是其子类，同样地，对于**自定义的 metric 类型**\n", + "\n", + " 也**需要继承自 Metric 类**，同时**内部自定义好 __init__ 、 update 和 get_metric 函数**\n", + "\n", + " 在`__init__`函数中，根据需求定义评测时需要用到的变量，此处沿用`Accuracy`中的`total_num`和`right_num`\n", + "\n", + " 在`update`函数中，根据需求定义评测变量的更新方式，需要注意的是如`tutorial-0`中所述，**update`的参数名**\n", + "\n", + " **需要待评估模型在 evaluate_step 中的输出名称一致**，由此**和数据集中对应字段名称一致**，即**参数匹配**\n", + "\n", + " 在`fastNLP v1.0`中，`update`函数的默认输入参数：`pred`，对应预测值；`target`，对应真实值\n", + "\n", + " 此处仍然沿用，因为接下来会需要使用`fastNLP`函数的与定义模型，其输入参数格式即使如此\n", + "\n", + " 在`get_metric`函数中，根据需求定义评测指标最终的计算，此处直接计算准确率，该函数必须返回一个字典\n", + "\n", + " 其中，字串`'prefix'`表示该`metric`的名称，会对应显示到`trainer`的`progress bar`中\n", + "\n", + "根据上述要求，这里简单定义了一个名为`MyMetric`的评测模块，用于分类问题的评测，以此展开一个实例展示" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "08a872e9", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "import sys\n", + "sys.path.append('..')\n", + "\n", + "from fastNLP import Metric\n", + "\n", + "class MyMetric(Metric):\n", + "\n", + " def __init__(self):\n", + " Metric.__init__(self)\n", + " self.total_num = 0\n", + " self.right_num = 0\n", + "\n", + " def update(self, pred, target):\n", + " self.total_num += target.size(0)\n", + " self.right_num += target.eq(pred).sum().item()\n", + "\n", + " def get_metric(self, reset=True):\n", + " acc = self.right_num / self.total_num\n", + " if reset:\n", + " self.total_num = 0\n", + " self.right_num = 0\n", + " return {'prefix': acc}" + ] + }, + { + "cell_type": "markdown", + "id": "0155f447", + "metadata": {}, + "source": [ + " 数据使用方面，此处仍然使用`datasets`模块中的`load_dataset`函数，加载`SST-2`二分类数据集" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "5ad81ac7", + "metadata": { + "pycharm": { + "name": "#%%\n" + } + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Reusing dataset glue (/remote-home/xrliu/.cache/huggingface/datasets/glue/sst2/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad)\n" + ] + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "ef923b90b19847f4916cccda5d33fc36", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + " 0%| | 0/3 [00:00 0: # 如果设置了 num_eval_sanity_batch\n", + "\t\ton_sanity_check_begin(trainer)\n", + "\t\ton_sanity_check_end(trainer, sanity_check_res)\n", + "\ttry:\n", + "\t\ton_train_begin(trainer)\n", + "\t\twhile cur_epoch_idx < n_epochs:\n", + "\t\t\ton_train_epoch_begin(trainer)\n", + "\t\t\twhile batch_idx_in_epoch<=num_batches_per_epoch:\n", + "\t\t\t\ton_fetch_data_begin(trainer)\n", + "\t\t\t\tbatch = next(dataloader)\n", + "\t\t\t\ton_fetch_data_end(trainer)\n", + "\t\t\t\ton_train_batch_begin(trainer, batch, indices)\n", + "\t\t\t\ton_before_backward(trainer, outputs) # 其中 outputs 是经过 output_mapping 后的\n", + "\t\t\t\ton_after_backward(trainer)\n", + "\t\t\t\ton_before_zero_grad(trainer, optimizers) # 实际调用受到 accumulation_steps 影响\n", + "\t\t\t\ton_after_zero_grad(trainer, optimizers) # 实际调用受到 accumulation_steps 影响\n", + "\t\t\t\ton_before_optimizers_step(trainer, optimizers) # 实际调用受到 accumulation_steps 影响\n", + "\t\t\t\ton_after_optimizers_step(trainer, optimizers) # 实际调用受到 accumulation_steps 影响\n", + "\t\t\t\ton_train_batch_end(trainer)\n", + "\t\t\ton_train_epoch_end(trainer)\n", + "\texcept BaseException:\n", + "\t\tself.on_exception(trainer, exception)\n", + "\tfinally:\n", + "\t\ton_train_end(trainer)\n", + "``` -->" + ] + }, + { + "cell_type": "markdown", + "id": "1e21df35", + "metadata": {}, + "source": [ + "紧接着，初始化`trainer`实例，继续完成`SST-2`分类，其中`metrics`输入的键值对，字串`'suffix'`和之前定义的\n", + "\n", + " 字串`'prefix'`将拼接在一起显示到`progress bar`中，故完整的输出形式为`{'prefix#suffix': float}`" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "926a9c50", + "metadata": {}, + "outputs": [], + "source": [ + "from fastNLP import Trainer\n", + "\n", + "trainer = Trainer(\n", + " model=model,\n", + " driver='torch',\n", + " device=0, # 'cuda'\n", + " n_epochs=10,\n", + " optimizers=optimizers,\n", + " input_mapping=input_mapping,\n", + " train_dataloader=train_dataloader,\n", + " evaluate_dataloaders=evaluate_dataloader,\n", + " metrics={'suffix': MyMetric()}\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "b1b2e8b7", + "metadata": { + "pycharm": { + "name": "#%%\n" + } + }, + "source": [ + "最后就是`run`函数的使用，关于其参数，这里也以表格形式列出，由此就解答了`num_eval_batch_per_dl=10`的含义\n", + "\n", + "|名称|功能|默认值|\n", + "|----|----|----|\n", + "| `num_train_batch_per_epoch` | 指定`trainer`训练时，每个循环计算批量数目 | 整数类型，默认`-1`，表示训练时，每个循环计算所有批量 |\n", + "| `num_eval_batch_per_dl` | 指定`trainer`评测时，每个循环计算批量数目 | 整数类型，默认`-1`，表示评测时，每个循环计算所有批量 |\n", + "| `num_eval_sanity_batch` | 指定`trainer`训练开始前，试探性评测批量数目 | 整数类型，默认`2`，表示训练开始前评估两个批量 |\n", + "| `resume_from` | 指定`trainer`恢复状态的路径，需要是文件夹 | 字符串型，默认`None`，使用可参考`CheckpointCallback` |\n", + "| `resume_training` | 指定`trainer`恢复状态的程度 | 布尔类型，默认`True`恢复所有状态，`False`仅恢复`model`和`optimizers`状态 |" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "43be274f", + "metadata": { + "pycharm": { + "name": "#%%\n" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
[09:30:35] INFO Running evaluator sanity check for 2 batches. trainer.py:596\n", + "\n" + ], + "text/plain": [ + "\u001b[2;36m[09:30:35]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m Running evaluator sanity check for \u001b[1;36m2\u001b[0m batches. \u001b]8;id=954293;file://../fastNLP/core/controllers/trainer.py\u001b\\\u001b[2mtrainer.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=366534;file://../fastNLP/core/controllers/trainer.py#596\u001b\\\u001b[2m596\u001b[0m\u001b]8;;\u001b\\\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Output()" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
/remote-home/xrliu/anaconda3/envs/demo/lib/python3.7/site-packages/ipywidgets/widgets/widget_\n", + "output.py:111: DeprecationWarning: Kernel._parent_header is deprecated in ipykernel 6. Use \n", + ".get_parent()\n", + " if ip and hasattr(ip, 'kernel') and hasattr(ip.kernel, '_parent_header'):\n", + "\n" + ], + "text/plain": [ + "/remote-home/xrliu/anaconda3/envs/demo/lib/python3.7/site-packages/ipywidgets/widgets/widget_\n", + "output.py:111: DeprecationWarning: Kernel._parent_header is deprecated in ipykernel 6. Use \n", + ".get_parent()\n", + " if ip and hasattr(ip, 'kernel') and hasattr(ip.kernel, '_parent_header'):\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
/remote-home/xrliu/anaconda3/envs/demo/lib/python3.7/site-packages/ipywidgets/widgets/widget_\n", + "output.py:112: DeprecationWarning: Kernel._parent_header is deprecated in ipykernel 6. Use \n", + ".get_parent()\n", + " self.msg_id = ip.kernel._parent_header['header']['msg_id']\n", + "\n" + ], + "text/plain": [ + "/remote-home/xrliu/anaconda3/envs/demo/lib/python3.7/site-packages/ipywidgets/widgets/widget_\n", + "output.py:112: DeprecationWarning: Kernel._parent_header is deprecated in ipykernel 6. Use \n", + ".get_parent()\n", + " self.msg_id = ip.kernel._parent_header['header']['msg_id']\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "\n" + ], + "text/plain": [] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Output()" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:1, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m1\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"prefix#suffix\": 0.6875\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"prefix#suffix\"\u001b[0m: \u001b[1;36m0.6875\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:2, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m2\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"prefix#suffix\": 0.8125\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"prefix#suffix\"\u001b[0m: \u001b[1;36m0.8125\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:3, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m3\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"prefix#suffix\": 0.80625\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"prefix#suffix\"\u001b[0m: \u001b[1;36m0.80625\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:4, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m4\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"prefix#suffix\": 0.825\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"prefix#suffix\"\u001b[0m: \u001b[1;36m0.825\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:5, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m5\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"prefix#suffix\": 0.8125\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"prefix#suffix\"\u001b[0m: \u001b[1;36m0.8125\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:6, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m6\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"prefix#suffix\": 0.80625\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"prefix#suffix\"\u001b[0m: \u001b[1;36m0.80625\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:7, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m7\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"prefix#suffix\": 0.80625\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"prefix#suffix\"\u001b[0m: \u001b[1;36m0.80625\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:8, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m8\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"prefix#suffix\": 0.8\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"prefix#suffix\"\u001b[0m: \u001b[1;36m0.8\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:9, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m9\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"prefix#suffix\": 0.80625\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"prefix#suffix\"\u001b[0m: \u001b[1;36m0.80625\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
---------------------------- Eval. results on Epoch:10, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "---------------------------- Eval. results on Epoch:\u001b[1;36m10\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"prefix#suffix\": 0.80625\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"prefix#suffix\"\u001b[0m: \u001b[1;36m0.80625\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "\n" + ], + "text/plain": [] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "trainer.run(num_eval_batch_per_dl=10)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f1abfa0a", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.13" + }, + "pycharm": { + "stem_cell": { + "cell_type": "raw", + "metadata": { + "collapsed": false + }, + "source": [] + } + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/docs/source/tutorials/fastnlp_tutorial_6.ipynb b/docs/source/tutorials/fastnlp_tutorial_6.ipynb new file mode 100644 index 00000000..63f7481e --- /dev/null +++ b/docs/source/tutorials/fastnlp_tutorial_6.ipynb @@ -0,0 +1,1646 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "fdd7ff16", + "metadata": {}, + "source": [ + "# T6. fastNLP 与 paddle 或 jittor 的结合\n", + "\n", + " 1 fastNLP 结合 paddle 训练模型\n", + " \n", + " 1.1 关于 paddle 的简单介绍\n", + "\n", + " 1.2 使用 paddle 搭建并训练模型\n", + "\n", + " 2 fastNLP 结合 jittor 训练模型\n", + "\n", + " 2.1 关于 jittor 的简单介绍\n", + "\n", + " 2.2 使用 jittor 搭建并训练模型\n", + "\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "08752c5a", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Reusing dataset glue (/remote-home/xrliu/.cache/huggingface/datasets/glue/sst2/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad)\n" + ] + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "6b13d42c39ba455eb370bf2caaa3a264", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + " 0%| | 0/3 [00:00\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Processing: 0%| | 0/6000 [00:00 True\n" + ] + } + ], + "source": [ + "import sys\n", + "sys.path.append('..')\n", + "\n", + "from fastNLP import DataSet\n", + "\n", + "dataset = DataSet.from_pandas(sst2data['train'].to_pandas())[:6000]\n", + "\n", + "dataset.apply_more(lambda ins:{'words': ins['sentence'].lower().split(), 'target': ins['label']}, \n", + " progress_bar=\"tqdm\")\n", + "dataset.delete_field('sentence')\n", + "dataset.delete_field('label')\n", + "dataset.delete_field('idx')\n", + "\n", + "from fastNLP import Vocabulary\n", + "\n", + "vocab = Vocabulary()\n", + "vocab.from_dataset(dataset, field_name='words')\n", + "vocab.index_dataset(dataset, field_name='words')\n", + "\n", + "train_dataset, evaluate_dataset = dataset.split(ratio=0.85)\n", + "print(type(train_dataset), isinstance(train_dataset, DataSet))\n", + "\n", + "from fastNLP.io import DataBundle\n", + "\n", + "data_bundle = DataBundle(datasets={'train': train_dataset, 'dev': evaluate_dataset})" + ] + }, + { + "cell_type": "markdown", + "id": "57a3272f", + "metadata": {}, + "source": [ + "## 1. fastNLP 结合 paddle 训练模型\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "e31b3198", + "metadata": {}, + "outputs": [], + "source": [ + "import paddle\n", + "import paddle.nn as nn\n", + "import paddle.nn.functional as F\n", + "\n", + "\n", + "class ClsByPaddle(nn.Layer):\n", + " def __init__(self, vocab_size, embedding_dim, output_dim, hidden_dim=64, dropout=0.5):\n", + " nn.Layer.__init__(self)\n", + " self.hidden_dim = hidden_dim\n", + "\n", + " self.embedding = nn.Embedding(num_embeddings=vocab_size, embedding_dim=embedding_dim)\n", + " \n", + " self.conv1 = nn.Sequential(nn.Conv1D(embedding_dim, 30, 1, padding=0), nn.ReLU())\n", + " self.conv2 = nn.Sequential(nn.Conv1D(embedding_dim, 40, 3, padding=1), nn.ReLU())\n", + " self.conv3 = nn.Sequential(nn.Conv1D(embedding_dim, 50, 5, padding=2), nn.ReLU())\n", + "\n", + " self.mlp = nn.Sequential(('dropout', nn.Dropout(p=dropout)),\n", + " ('linear_1', nn.Linear(120, hidden_dim)),\n", + " ('activate', nn.ReLU()),\n", + " ('linear_2', nn.Linear(hidden_dim, output_dim)))\n", + " \n", + " self.loss_fn = nn.MSELoss()\n", + "\n", + " def forward(self, words):\n", + " output = self.embedding(words).transpose([0, 2, 1])\n", + " conv1, conv2, conv3 = self.conv1(output), self.conv2(output), self.conv3(output)\n", + "\n", + " pool1 = F.max_pool1d(conv1, conv1.shape[-1]).squeeze(2)\n", + " pool2 = F.max_pool1d(conv2, conv2.shape[-1]).squeeze(2)\n", + " pool3 = F.max_pool1d(conv3, conv3.shape[-1]).squeeze(2)\n", + "\n", + " pool = paddle.concat([pool1, pool2, pool3], axis=1)\n", + " output = self.mlp(pool)\n", + " return output\n", + " \n", + " def train_step(self, words, target):\n", + " pred = self(words)\n", + " target = paddle.stack((1 - target, target), axis=1).cast(pred.dtype)\n", + " return {'loss': self.loss_fn(pred, target)}\n", + "\n", + " def evaluate_step(self, words, target):\n", + " pred = self(words)\n", + " pred = paddle.argmax(pred, axis=-1)\n", + " return {'pred': pred, 'target': target}" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "c63b030f", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "W0604 21:02:25.453869 19014 gpu_context.cc:278] Please NOTE: device: 0, GPU Compute Capability: 6.1, Driver API Version: 11.1, Runtime API Version: 10.2\n", + "W0604 21:02:26.061690 19014 gpu_context.cc:306] device: 0, cuDNN Version: 7.6.\n" + ] + }, + { + "data": { + "text/plain": [ + "ClsByPaddle(\n", + " (embedding): Embedding(8458, 100, sparse=False)\n", + " (conv1): Sequential(\n", + " (0): Conv1D(100, 30, kernel_size=[1], data_format=NCL)\n", + " (1): ReLU()\n", + " )\n", + " (conv2): Sequential(\n", + " (0): Conv1D(100, 40, kernel_size=[3], padding=1, data_format=NCL)\n", + " (1): ReLU()\n", + " )\n", + " (conv3): Sequential(\n", + " (0): Conv1D(100, 50, kernel_size=[5], padding=2, data_format=NCL)\n", + " (1): ReLU()\n", + " )\n", + " (mlp): Sequential(\n", + " (dropout): Dropout(p=0.5, axis=None, mode=upscale_in_train)\n", + " (linear_1): Linear(in_features=120, out_features=64, dtype=float32)\n", + " (activate): ReLU()\n", + " (linear_2): Linear(in_features=64, out_features=2, dtype=float32)\n", + " )\n", + " (loss_fn): MSELoss()\n", + ")" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "model = ClsByPaddle(vocab_size=len(vocab), embedding_dim=100, output_dim=2)\n", + "\n", + "model" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "2997c0aa", + "metadata": {}, + "outputs": [], + "source": [ + "from paddle.optimizer import AdamW\n", + "\n", + "optimizers = AdamW(parameters=model.parameters(), learning_rate=5e-4)" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "ead35fb8", + "metadata": {}, + "outputs": [], + "source": [ + "from fastNLP import prepare_paddle_dataloader\n", + "\n", + "train_dataloader = prepare_paddle_dataloader(train_dataset, batch_size=16, shuffle=True)\n", + "evaluate_dataloader = prepare_paddle_dataloader(evaluate_dataset, batch_size=16)\n", + "\n", + "# dl_bundle = prepare_paddle_dataloader(data_bundle, batch_size=16, shuffle=True)" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "25e8da83", + "metadata": {}, + "outputs": [], + "source": [ + "from fastNLP import Trainer, Accuracy\n", + "\n", + "trainer = Trainer(\n", + " model=model,\n", + " driver='paddle',\n", + " device='gpu', # 'cpu', 'gpu', 'gpu:x'\n", + " n_epochs=10,\n", + " optimizers=optimizers,\n", + " train_dataloader=train_dataloader, # dl_bundle['train'],\n", + " evaluate_dataloaders=evaluate_dataloader, # dl_bundle['dev'], \n", + " metrics={'acc': Accuracy()}\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "d63c5d74", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
[21:03:08] INFO Running evaluator sanity check for 2 batches. trainer.py:596\n", + "\n" + ], + "text/plain": [ + "\u001b[2;36m[21:03:08]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m Running evaluator sanity check for \u001b[1;36m2\u001b[0m batches. \u001b]8;id=894986;file://../fastNLP/core/controllers/trainer.py\u001b\\\u001b[2mtrainer.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=567751;file://../fastNLP/core/controllers/trainer.py#596\u001b\\\u001b[2m596\u001b[0m\u001b]8;;\u001b\\\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Output()" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
/remote-home/xrliu/anaconda3/envs/demo/lib/python3.7/site-packages/ipywidgets/widgets/widget_\n", + "output.py:111: DeprecationWarning: Kernel._parent_header is deprecated in ipykernel 6. Use \n", + ".get_parent()\n", + " if ip and hasattr(ip, 'kernel') and hasattr(ip.kernel, '_parent_header'):\n", + "\n" + ], + "text/plain": [ + "/remote-home/xrliu/anaconda3/envs/demo/lib/python3.7/site-packages/ipywidgets/widgets/widget_\n", + "output.py:111: DeprecationWarning: Kernel._parent_header is deprecated in ipykernel 6. Use \n", + ".get_parent()\n", + " if ip and hasattr(ip, 'kernel') and hasattr(ip.kernel, '_parent_header'):\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
/remote-home/xrliu/anaconda3/envs/demo/lib/python3.7/site-packages/ipywidgets/widgets/widget_\n", + "output.py:112: DeprecationWarning: Kernel._parent_header is deprecated in ipykernel 6. Use \n", + ".get_parent()\n", + " self.msg_id = ip.kernel._parent_header['header']['msg_id']\n", + "\n" + ], + "text/plain": [ + "/remote-home/xrliu/anaconda3/envs/demo/lib/python3.7/site-packages/ipywidgets/widgets/widget_\n", + "output.py:112: DeprecationWarning: Kernel._parent_header is deprecated in ipykernel 6. Use \n", + ".get_parent()\n", + " self.msg_id = ip.kernel._parent_header['header']['msg_id']\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
/remote-home/xrliu/anaconda3/envs/demo/lib/python3.7/site-packages/paddle/tensor/creation.py:\n", + "125: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To \n", + "silence this warning, use `object` by itself. Doing this will not modify any behavior and is \n", + "safe. \n", + "Deprecated in NumPy 1.20; for more details and guidance: \n", + "https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n", + " if data.dtype == np.object:\n", + "\n" + ], + "text/plain": [ + "/remote-home/xrliu/anaconda3/envs/demo/lib/python3.7/site-packages/paddle/tensor/creation.py:\n", + "125: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To \n", + "silence this warning, use `object` by itself. Doing this will not modify any behavior and is \n", + "safe. \n", + "Deprecated in NumPy 1.20; for more details and guidance: \n", + "https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n", + " if data.dtype == np.object:\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "\n" + ], + "text/plain": [] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Output()" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:1, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m1\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.78125,\n", + " \"total#acc\": 160.0,\n", + " \"correct#acc\": 125.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.78125\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m125.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:2, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m2\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.7875,\n", + " \"total#acc\": 160.0,\n", + " \"correct#acc\": 126.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.7875\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m126.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:3, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m3\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.8,\n", + " \"total#acc\": 160.0,\n", + " \"correct#acc\": 128.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.8\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m128.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:4, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m4\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.79375,\n", + " \"total#acc\": 160.0,\n", + " \"correct#acc\": 127.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.79375\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m127.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:5, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m5\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.81875,\n", + " \"total#acc\": 160.0,\n", + " \"correct#acc\": 131.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.81875\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m131.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:6, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m6\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.8,\n", + " \"total#acc\": 160.0,\n", + " \"correct#acc\": 128.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.8\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m128.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:7, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m7\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.80625,\n", + " \"total#acc\": 160.0,\n", + " \"correct#acc\": 129.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.80625\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m129.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:8, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m8\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.79375,\n", + " \"total#acc\": 160.0,\n", + " \"correct#acc\": 127.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.79375\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m127.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:9, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m9\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.7875,\n", + " \"total#acc\": 160.0,\n", + " \"correct#acc\": 126.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.7875\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m126.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
---------------------------- Eval. results on Epoch:10, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "---------------------------- Eval. results on Epoch:\u001b[1;36m10\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.8,\n", + " \"total#acc\": 160.0,\n", + " \"correct#acc\": 128.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.8\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m128.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "\n" + ], + "text/plain": [] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "trainer.run(num_eval_batch_per_dl=10) " + ] + }, + { + "cell_type": "markdown", + "id": "cb9a0b3c", + "metadata": {}, + "source": [ + "## 2. fastNLP 结合 jittor 训练模型" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "c600191d", + "metadata": {}, + "outputs": [], + "source": [ + "import jittor\n", + "import jittor.nn as nn\n", + "\n", + "from jittor import Module\n", + "\n", + "\n", + "class ClsByJittor(Module):\n", + " def __init__(self, vocab_size, embedding_dim, output_dim, hidden_dim=64, num_layers=2, dropout=0.5):\n", + " Module.__init__(self)\n", + " self.hidden_dim = hidden_dim\n", + "\n", + " self.embedding = nn.Embedding(num=vocab_size, dim=embedding_dim)\n", + " self.lstm = nn.LSTM(input_size=embedding_dim, hidden_size=hidden_dim, batch_first=True, # 默认 batch_first=False\n", + " num_layers=num_layers, bidirectional=True, dropout=dropout)\n", + " self.mlp = nn.Sequential([nn.Dropout(p=dropout),\n", + " nn.Linear(hidden_dim * 2, hidden_dim * 2),\n", + " nn.ReLU(),\n", + " nn.Linear(hidden_dim * 2, output_dim),\n", + " nn.Sigmoid(),])\n", + "\n", + " self.loss_fn = nn.MSELoss()\n", + "\n", + " def execute(self, words):\n", + " output = self.embedding(words)\n", + " output, (hidden, cell) = self.lstm(output)\n", + " output = self.mlp(jittor.concat((hidden[-1], hidden[-2]), dim=1))\n", + " return output\n", + " \n", + " def train_step(self, words, target):\n", + " pred = self(words)\n", + " target = jittor.stack((1 - target, target), dim=1)\n", + " return {'loss': self.loss_fn(pred, target)}\n", + "\n", + " def evaluate_step(self, words, target):\n", + " pred = self(words)\n", + " pred = jittor.argmax(pred, dim=-1)[0]\n", + " return {'pred': pred, 'target': target}" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "a94ed8c4", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "ClsByJittor(\n", + " embedding: Embedding(8458, 100)\n", + " lstm: LSTM(100, 64, 2, bias=True, batch_first=True, dropout=0.5, bidirectional=True, proj_size=0)\n", + " mlp: Sequential(\n", + " 0: Dropout(0.5, is_train=False)\n", + " 1: Linear(128, 128, float32[128,], None)\n", + " 2: relu()\n", + " 3: Linear(128, 2, float32[2,], None)\n", + " 4: Sigmoid()\n", + " )\n", + " loss_fn: MSELoss(mean)\n", + ")" + ] + }, + "execution_count": 12, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "model = ClsByJittor(vocab_size=len(vocab), embedding_dim=100, output_dim=2)\n", + "\n", + "model" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "6d15ebc1", + "metadata": {}, + "outputs": [], + "source": [ + "from jittor.optim import AdamW\n", + "\n", + "optimizers = AdamW(params=model.parameters(), lr=5e-3)" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "95d8d09e", + "metadata": {}, + "outputs": [], + "source": [ + "from fastNLP import prepare_jittor_dataloader\n", + "\n", + "train_dataloader = prepare_jittor_dataloader(train_dataset, batch_size=16, shuffle=True)\n", + "evaluate_dataloader = prepare_jittor_dataloader(evaluate_dataset, batch_size=16)\n", + "\n", + "# dl_bundle = prepare_jittor_dataloader(data_bundle, batch_size=16, shuffle=True)" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "917eab81", + "metadata": {}, + "outputs": [], + "source": [ + "from fastNLP import Trainer, Accuracy\n", + "\n", + "trainer = Trainer(\n", + " model=model,\n", + " driver='jittor',\n", + " device='gpu', # 'cpu', 'gpu', 'cuda'\n", + " n_epochs=10,\n", + " optimizers=optimizers,\n", + " train_dataloader=train_dataloader, # dl_bundle['train'],\n", + " evaluate_dataloaders=evaluate_dataloader, # dl_bundle['dev'],\n", + " metrics={'acc': Accuracy()}\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "f7c4ac5a", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
[21:05:51] INFO Running evaluator sanity check for 2 batches. trainer.py:596\n", + "\n" + ], + "text/plain": [ + "\u001b[2;36m[21:05:51]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m Running evaluator sanity check for \u001b[1;36m2\u001b[0m batches. \u001b]8;id=69759;file://../fastNLP/core/controllers/trainer.py\u001b\\\u001b[2mtrainer.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=202322;file://../fastNLP/core/controllers/trainer.py#596\u001b\\\u001b[2m596\u001b[0m\u001b]8;;\u001b\\\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Output()" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "\n" + ], + "text/plain": [] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Output()" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\n", + "Compiling Operators(5/6) used: 8.31s eta: 1.66s 6/6) used: 9.33s eta: 0s \n", + "\n", + "Compiling Operators(31/31) used: 7.31s eta: 0s \n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:1, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m1\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.61875,\n", + " \"total#acc\": 160,\n", + " \"correct#acc\": 99\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.61875\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m99\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:2, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m2\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.7,\n", + " \"total#acc\": 160,\n", + " \"correct#acc\": 112\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.7\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m112\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:3, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m3\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.725,\n", + " \"total#acc\": 160,\n", + " \"correct#acc\": 116\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.725\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m116\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:4, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m4\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.74375,\n", + " \"total#acc\": 160,\n", + " \"correct#acc\": 119\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.74375\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m119\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:5, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m5\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.75625,\n", + " \"total#acc\": 160,\n", + " \"correct#acc\": 121\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.75625\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m121\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:6, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m6\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.75625,\n", + " \"total#acc\": 160,\n", + " \"correct#acc\": 121\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.75625\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m121\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:7, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m7\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.73125,\n", + " \"total#acc\": 160,\n", + " \"correct#acc\": 117\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.73125\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m117\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:8, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m8\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.7625,\n", + " \"total#acc\": 160,\n", + " \"correct#acc\": 122\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.7625\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m122\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:9, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m9\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.74375,\n", + " \"total#acc\": 160,\n", + " \"correct#acc\": 119\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.74375\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m119\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
---------------------------- Eval. results on Epoch:10, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "---------------------------- Eval. results on Epoch:\u001b[1;36m10\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.7625,\n", + " \"total#acc\": 160,\n", + " \"correct#acc\": 122\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.7625\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m160\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m122\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "\n" + ], + "text/plain": [] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "trainer.run(num_eval_batch_per_dl=10)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3df5f425", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/docs/source/tutorials/fastnlp_tutorial_e1.ipynb b/docs/source/tutorials/fastnlp_tutorial_e1.ipynb new file mode 100644 index 00000000..af8e60a0 --- /dev/null +++ b/docs/source/tutorials/fastnlp_tutorial_e1.ipynb @@ -0,0 +1,1280 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + " 从这篇开始，我们将开启 **fastNLP v1.0 tutorial 的 example 系列**，在接下来的\n", + "\n", + " 每篇`tutorial`里，我们将会介绍`fastNLP v1.0`在自然语言处理任务上的应用实例" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "4.18.0\n" + ] + } + ], + "source": [ + "import torch\n", + "import torch.nn as nn\n", + "from torch.optim import AdamW\n", + "from torch.utils.data import DataLoader, Dataset\n", + "\n", + "import transformers\n", + "from transformers import AutoTokenizer\n", + "from transformers import AutoModelForSequenceClassification\n", + "\n", + "import sys\n", + "sys.path.append('..')\n", + "\n", + "import fastNLP\n", + "from fastNLP import Trainer\n", + "from fastNLP import Accuracy\n", + "\n", + "print(transformers.__version__)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 1. 基础介绍：GLUE 通用语言理解评估、SST-2 文本情感二分类数据集\n", + "\n", + " 本示例使用`GLUE`评估基准中的`SST-2`数据集，通过`fine-tuning`方式\n", + "\n", + " 调整`distilbert-bert`分类模型，以下首先简单介绍下`GLUE`和`SST-2`\n", + "\n", + "**GLUE**，**全称 General Language Understanding Evaluation**，**通用语言理解评估**，\n", + "\n", + " 包含9个数据集，各语料的语言均为英语，涉及多个自然语言理解`NLU`任务，包括\n", + "\n", + " **CoLA**，文本分类任务，预测单句语法正误分类；**SST-2**，文本分类任务，预测单句情感二分类\n", + "\n", + " **MRPC**，句对分类任务，预测句对语义一致性；**STS-B**，相似度打分任务，预测句对语义相似度回归\n", + "\n", + " **QQP**，句对分类任务，预测问题对语义一致性；**MNLI**，文本推理任务，预测句对蕴含/矛盾/中立预测\n", + "\n", + " **QNLI / RTE / WNLI**，文本推理，预测是否蕴含二分类（其中，`QNLI`从`SQuAD`转化而来\n", + "\n", + " 诸如`BERT`、`T5`等经典模型都会在此基准上验证效果，更多参考[GLUE论文](https://arxiv.org/pdf/1804.07461v3.pdf)\n", + "\n", + " 此处，我们使用`SST-2`来训练`bert`，实现文本分类，其他任务描述见下图" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "GLUE_TASKS = ['cola', 'mnli', 'mrpc', 'qnli', 'qqp', 'rte', 'sst2', 'stsb', 'wnli']\n", + "\n", + "task = 'sst2'" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
![](\"./figures/E1-fig-glue-benchmark.png\")
\n",
+ "\n",
+ "**SST**，**全称`Stanford Sentiment Treebank**，**斯坦福情感树库**，**单句情感分类**数据集\n",
+ "\n",
+ " 包含电影评论语句和对应的情感极性，1 对应`positive` 正面情感，0 对应`negative` 负面情感\n",
+ "\n",
+ " 数据集包括三部分：训练集 67350 条，验证集 873 条，测试集 1821 条，更多参考[下载链接](https://gluebenchmark.com/tasks)\n",
+ "\n",
+ "对应到代码上，此处使用`datasets`模块中的`load_dataset`函数，指定`SST-2`数据集，自动加载\n",
+ "\n",
+ " 首次下载后会保存至`~/.cache/huggingface/modules/datasets_modules/datasets/glue/`目录下"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "scrolled": false
+ },
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Reusing dataset glue (/remote-home/xrliu/.cache/huggingface/datasets/glue/sst2/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad)\n"
+ ]
+ },
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "c5915debacf9443986b5b3b34870b303",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ " 0%| | 0/3 [00:00[09:12:45] INFO Running evaluator sanity check for 2 batches. trainer.py:592\n",
+ "\n"
+ ],
+ "text/plain": [
+ "\u001b[2;36m[09:12:45]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m Running evaluator sanity check for \u001b[1;36m2\u001b[0m batches. \u001b]8;id=408427;file://../fastNLP/core/controllers/trainer.py\u001b\\\u001b[2mtrainer.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=303634;file://../fastNLP/core/controllers/trainer.py#592\u001b\\\u001b[2m592\u001b[0m\u001b]8;;\u001b\\\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ "Output()"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n"
+ ],
+ "text/plain": []
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ "Output()"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:1, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m1\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.884375,\n", + " \"total#acc\": 320.0,\n", + " \"correct#acc\": 283.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.884375\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m320.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m283.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:2, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m2\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.878125,\n", + " \"total#acc\": 320.0,\n", + " \"correct#acc\": 281.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.878125\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m320.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m281.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:3, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m3\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.884375,\n", + " \"total#acc\": 320.0,\n", + " \"correct#acc\": 283.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.884375\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m320.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m283.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:4, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m4\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.9,\n", + " \"total#acc\": 320.0,\n", + " \"correct#acc\": 288.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.9\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m320.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m288.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:5, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m5\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.8875,\n", + " \"total#acc\": 320.0,\n", + " \"correct#acc\": 284.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.8875\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m320.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m284.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:6, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m6\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.88125,\n", + " \"total#acc\": 320.0,\n", + " \"correct#acc\": 282.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.88125\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m320.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m282.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:7, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m7\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.875,\n", + " \"total#acc\": 320.0,\n", + " \"correct#acc\": 280.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.875\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m320.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m280.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:8, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m8\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.865625,\n", + " \"total#acc\": 320.0,\n", + " \"correct#acc\": 277.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.865625\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m320.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m277.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:9, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m9\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.884375,\n", + " \"total#acc\": 320.0,\n", + " \"correct#acc\": 283.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.884375\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m320.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m283.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
---------------------------- Eval. results on Epoch:10, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "---------------------------- Eval. results on Epoch:\u001b[1;36m10\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.878125,\n", + " \"total#acc\": 320.0,\n", + " \"correct#acc\": 281.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.878125\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m320.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m281.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "\n" + ], + "text/plain": [] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "trainer.run(num_eval_batch_per_dl=10)" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": {}, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Output()" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "\n" + ], + "text/plain": [] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/plain": [ + "{'acc#acc': 0.884174, 'total#acc': 872.0, 'correct#acc': 771.0}" + ] + }, + "execution_count": 14, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "trainer.evaluator.run()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### 附：`DistilBertForSequenceClassification`模块结构\n", + "\n", + "```\n", + "
![](\"./figures/E2-fig-pet-model.png\")
\n",
+ "\n",
+ "**prompt-based tuning**，**基于提示的微调**，将`prompt`应用于**参数高效微调**，**parameter-efficient tuning**\n",
+ "\n",
+ " 通过**设计模板调整模型输入**或者**调整模型内部状态**，**固定预训练模型**，**诱导/刺激模型**调整输出以适应\n",
+ "\n",
+ " 当前任务，极大降低了训练开销，也省去了`verbalizer`的构造，更多参考[prompt综述](https://arxiv.org/pdf/2107.13586.pdf)、[DeltaTuning综述](https://arxiv.org/pdf/2203.06904.pdf)\n",
+ "\n",
+ " 以下列举些经典的`prompt-based tuning`案例，简单地介绍下`prompt-based tuning`的脉络\n",
+ "\n",
+ " **案例一**：**PrefixTuning**，详细内容参考[PrefixTuning论文](https://arxiv.org/pdf/2101.00190.pdf)\n",
+ "\n",
+ " 其主要贡献在于，**提出连续的、非人工构造的、任务导向的 prompt**，即**前缀 prefix**，**调整**\n",
+ "\n",
+ " **模型内部更新状态**，诱导模型在特定任务下生成期望目标，降低优化难度，提升微调效果\n",
+ "\n",
+ " 其主要研究对象，是`GPT2`和`BART`，主要面向生成任务`NLG`，如`table-to-text`和摘要\n",
+ "\n",
+ " **案例二**：**P-Tuning v1**，详细内容参考[P-Tuning-v1论文](https://arxiv.org/pdf/2103.10385.pdf)\n",
+ "\n",
+ " 其主要贡献在于，**通过连续的、非人工构造的 prompt 调整模型输入**，取代原先基于单词设计的\n",
+ "\n",
+ " 但离散且不易于优化的`prompt`；同时也**证明了 GPT2 在语言理解任务上仍然是可以胜任的**\n",
+ "\n",
+ " 其主要研究对象，是`GPT2`，主要面向知识探测`knowledge probing`和自然语言理解`NLU`\n",
+ "\n",
+ " **案例三**：**PromptTuning**，详细内容参考[PromptTuning论文](https://arxiv.org/pdf/2104.08691.pdf)\n",
+ "\n",
+ " 其主要贡献在于，通过连续的`prompt`调整模型输入，**证明了 prompt-based tuning 的效果**\n",
+ "\n",
+ " **随模型参数量的增加而提升**，最终**在 10B 左右追上了全参数微调 fine-tuning 的效果**\n",
+ "\n",
+ " 其主要面向自然语言理解`NLU`，通过为每个任务定义不同的`prompt`，从而支持多任务语境\n",
+ "\n",
+ "通过上述介绍可以发现`prompt-based tuning`只是模型微调方式，独立于预训练模型基础`backbone`\n",
+ "\n",
+ " 目前，加载预训练模型的主流方法是使用**transformers 模块**，而实现微调的框架则\n",
+ "\n",
+ " 可以是`pytorch`、`paddle`、`jittor`等，而不同框架间又存在不兼容的问题\n",
+ "\n",
+ " 因此，**使用 fastNLP v1.0 实现 prompt-based tuning**，可以**很好地解决 paddle 等框架**\n",
+ "\n",
+ " **和 transformers 模块之间的桥接**（`transformers`模块基于`pytorch`实现）\n",
+ "\n",
+ "本示例仍使用了`tutorial-E1`的`SST-2`数据集、`distilbert-base-uncased`模型（便于比较\n",
+ "\n",
+ " 使用`pytorch`框架，通过将连续的`prompt`与`model`拼接，解决`SST-2`二分类任务"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "4.18.0\n" + ] + } + ], + "source": [ + "import torch\n", + "import torch.nn as nn\n", + "from torch.optim import AdamW\n", + "from torch.utils.data import DataLoader, Dataset\n", + "\n", + "import transformers\n", + "from transformers import AutoTokenizer\n", + "from transformers import AutoModelForSequenceClassification\n", + "\n", + "import sys\n", + "sys.path.append('..')\n", + "\n", + "import fastNLP\n", + "from fastNLP import Trainer\n", + "from fastNLP.core.metrics import Accuracy\n", + "\n", + "print(transformers.__version__)\n", + "\n", + "task = 'sst2'\n", + "model_checkpoint = 'distilbert-base-uncased' # 'bert-base-uncased'" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 2. 准备工作：P-Tuning v2 原理概述、P-Tuning v2 模型搭建\n", + "\n", + " 本示例使用`P-Tuning v2`作为`prompt-based tuning`与`fastNLP v1.0`结合的案例\n", + "\n", + " 以下首先简述`P-Tuning v2`的论文原理，并由此引出`fastNLP v1.0`的代码实践\n", + "\n", + "**P-Tuning v2**出自论文[Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://arxiv.org/pdf/2110.07602.pdf)\n", + "\n", + " 其主要贡献在于，**在 PrefixTuning 等深度提示学习基础上**，**提升了其在分类标注等 NLU 任务的表现**\n", + "\n", + " 并使之在中等规模模型，主要是**参数量在 100M-1B 区间的模型上**，**获得与全参数微调相同的效果**\n", + "\n", + " 其结构如图所示，通过**在输入序列的分类符 [CLS] 之前**，**加入前缀序列**（**序号对应嵌入是待训练的连续值向量**\n", + "\n", + " **刺激模型在新任务下**，从`[CLS]`对应位置，**输出符合微调任务的输出**，从而达到适应微调任务的目的\n", + "\n", + "
![](\"./figures/E2-fig-p-tuning-v2-model.png\")
\n",
+ "\n",
+ "本示例使用`bert-base-uncased`模型，作为`P-Tuning v2`的基础`backbone`，设置`requires_grad=False`\n",
+ "\n",
+ " 固定其参数不参与训练，**设置 pre_seq_len 长的 prefix_tokens 作为输入的提示前缀序列**\n",
+ "\n",
+ " **使用基于 nn.Embedding 的 prefix_encoder 为提示前缀嵌入**，通过`get_prompt`函数获取，再将之\n",
+ "\n",
+ " 拼接至批量内每笔数据前得到`inputs_embeds`，同时更新自注意力掩码`attention_mask`\n",
+ "\n",
+ " 将`inputs_embeds`、`attention_mask`和`labels`输入`backbone`，**得到输出包括 loss 和 logits**"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "class SeqClsModel(nn.Module):\n",
+ " def __init__(self, model_checkpoint, num_labels, pre_seq_len):\n",
+ " nn.Module.__init__(self)\n",
+ " self.num_labels = num_labels\n",
+ " self.back_bone = AutoModelForSequenceClassification.from_pretrained(model_checkpoint, \n",
+ " num_labels=num_labels)\n",
+ " self.embeddings = self.back_bone.get_input_embeddings()\n",
+ "\n",
+ " for param in self.back_bone.parameters():\n",
+ " param.requires_grad = False\n",
+ " \n",
+ " self.pre_seq_len = pre_seq_len\n",
+ " self.prefix_tokens = torch.arange(self.pre_seq_len).long()\n",
+ " self.prefix_encoder = nn.Embedding(self.pre_seq_len, self.embeddings.embedding_dim)\n",
+ " \n",
+ " def get_prompt(self, batch_size):\n",
+ " prefix_tokens = self.prefix_tokens.unsqueeze(0).expand(batch_size, -1).to(self.back_bone.device)\n",
+ " prompts = self.prefix_encoder(prefix_tokens)\n",
+ " return prompts\n",
+ "\n",
+ " def forward(self, input_ids, attention_mask, labels=None):\n",
+ " \n",
+ " batch_size = input_ids.shape[0]\n",
+ " raw_embedding = self.embeddings(input_ids)\n",
+ " \n",
+ " prompts = self.get_prompt(batch_size=batch_size)\n",
+ " inputs_embeds = torch.cat((prompts, raw_embedding), dim=1)\n",
+ " prefix_attention_mask = torch.ones(batch_size, self.pre_seq_len).to(self.back_bone.device)\n",
+ " attention_mask = torch.cat((prefix_attention_mask, attention_mask), dim=1)\n",
+ "\n",
+ " outputs = self.back_bone(inputs_embeds=inputs_embeds, \n",
+ " attention_mask=attention_mask, labels=labels)\n",
+ " return outputs\n",
+ "\n",
+ " def train_step(self, input_ids, attention_mask, labels):\n",
+ " loss = self(input_ids, attention_mask, labels).loss\n",
+ " return {'loss': loss}\n",
+ "\n",
+ " def evaluate_step(self, input_ids, attention_mask, labels):\n",
+ " pred = self(input_ids, attention_mask, labels).logits\n",
+ " pred = torch.max(pred, dim=-1)[1]\n",
+ " return {'pred': pred, 'target': labels}"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "接着，通过确定分类数量初始化模型实例，同时调用`torch.optim.AdamW`模块初始化优化器\n",
+ "\n",
+ " 根据`P-Tuning v2`论文：`Generally, simple classification tasks prefer shorter prompts (less than 20)`\n",
+ "\n",
+ " 此处`pre_seq_len`参数设定为`20`，学习率相应做出调整，其他内容和`tutorial-E1`中的内容一致"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: ['vocab_layer_norm.bias', 'vocab_layer_norm.weight', 'vocab_projector.weight', 'vocab_transform.bias', 'vocab_transform.weight', 'vocab_projector.bias']\n",
+ "- This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
+ "- This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
+ "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['pre_classifier.weight', 'classifier.weight', 'pre_classifier.bias', 'classifier.bias']\n",
+ "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
+ ]
+ }
+ ],
+ "source": [
+ "model = SeqClsModel(model_checkpoint=model_checkpoint, num_labels=2, pre_seq_len=20)\n",
+ "\n",
+ "optimizers = AdamW(params=model.parameters(), lr=1e-2)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 3. 模型训练：加载 tokenizer、预处理 dataset、模型训练与分析\n",
+ "\n",
+ " 本示例沿用`tutorial-E1`中的数据集，即使用`GLUE`评估基准中的`SST-2`数据集\n",
+ "\n",
+ " 以`bert-base-uncased`模型作为基准，基于`P-Tuning v2`方式微调\n",
+ "\n",
+ " 数据集加载相关代码流程见下，内容和`tutorial-E1`中的内容基本一致\n",
+ "\n",
+ "首先，使用`datasets.load_dataset`加载数据集，使用`transformers.AutoTokenizer`\n",
+ "\n",
+ " 构建`tokenizer`实例，通过`dataset.map`使用`tokenizer`将文本替换为词素序号序列"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {
+ "scrolled": false
+ },
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Reusing dataset glue (/remote-home/xrliu/.cache/huggingface/datasets/glue/sst2/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad)\n"
+ ]
+ },
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "21cbd92c3397497d84dc10f017ec96f4",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ " 0%| | 0/3 [00:00[22:53:00] INFO Running evaluator sanity check for 2 batches. trainer.py:592\n",
+ "\n"
+ ],
+ "text/plain": [
+ "\u001b[2;36m[22:53:00]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m Running evaluator sanity check for \u001b[1;36m2\u001b[0m batches. \u001b]8;id=406635;file://../fastNLP/core/controllers/trainer.py\u001b\\\u001b[2mtrainer.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=951504;file://../fastNLP/core/controllers/trainer.py#592\u001b\\\u001b[2m592\u001b[0m\u001b]8;;\u001b\\\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ "Output()"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n"
+ ],
+ "text/plain": []
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ "Output()"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:1, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m1\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.540625,\n", + " \"total#acc\": 320.0,\n", + " \"correct#acc\": 173.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.540625\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m320.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m173.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:2, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m2\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.5,\n", + " \"total#acc\": 320.0,\n", + " \"correct#acc\": 160.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.5\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m320.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m160.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:3, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m3\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.509375,\n", + " \"total#acc\": 320.0,\n", + " \"correct#acc\": 163.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.509375\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m320.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m163.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:4, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m4\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.634375,\n", + " \"total#acc\": 320.0,\n", + " \"correct#acc\": 203.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.634375\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m320.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m203.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:5, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m5\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.6125,\n", + " \"total#acc\": 320.0,\n", + " \"correct#acc\": 196.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.6125\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m320.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m196.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:6, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m6\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.675,\n", + " \"total#acc\": 320.0,\n", + " \"correct#acc\": 216.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.675\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m320.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m216.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:7, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m7\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.64375,\n", + " \"total#acc\": 320.0,\n", + " \"correct#acc\": 206.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.64375\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m320.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m206.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:8, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m8\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.665625,\n", + " \"total#acc\": 320.0,\n", + " \"correct#acc\": 213.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.665625\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m320.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m213.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
----------------------------- Eval. results on Epoch:9, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "----------------------------- Eval. results on Epoch:\u001b[1;36m9\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.659375,\n", + " \"total#acc\": 320.0,\n", + " \"correct#acc\": 211.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.659375\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m320.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m211.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
---------------------------- Eval. results on Epoch:10, Batch:0 -----------------------------\n", + "\n" + ], + "text/plain": [ + "---------------------------- Eval. results on Epoch:\u001b[1;36m10\u001b[0m, Batch:\u001b[1;36m0\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#acc\": 0.696875,\n", + " \"total#acc\": 320.0,\n", + " \"correct#acc\": 223.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#acc\"\u001b[0m: \u001b[1;36m0.696875\u001b[0m,\n", + " \u001b[1;34m\"total#acc\"\u001b[0m: \u001b[1;36m320.0\u001b[0m,\n", + " \u001b[1;34m\"correct#acc\"\u001b[0m: \u001b[1;36m223.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "\n" + ], + "text/plain": [] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "trainer.run(num_eval_batch_per_dl=10)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "可以发现，其效果远远逊色于`fine-tuning`，这是因为`P-Tuning v2`虽然能够适应参数量\n", + "\n", + " 在`100M-1B`区间的模型，但是，**distilbert-base 的参数量仅为 66M**，无法触及其下限\n", + "\n", + "另一方面，**fastNLP v1.0 不支持 jupyter 多卡**，所以无法在笔者的电脑/服务器上，完成\n", + "\n", + " 合适规模模型的学习，例如`110M`的`bert-base`模型，以及`340M`的`bert-large`模型" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Output()" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "\n" + ], + "text/plain": [] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/plain": [ + "{'acc#acc': 0.737385, 'total#acc': 872.0, 'correct#acc': 643.0}" + ] + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "trainer.evaluator.run()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.13" + }, + "pycharm": { + "stem_cell": { + "cell_type": "raw", + "metadata": { + "collapsed": false + }, + "source": [] + } + } + }, + "nbformat": 4, + "nbformat_minor": 1 +} diff --git a/docs/source/tutorials/fastnlp_tutorial_paddle_e1.ipynb b/docs/source/tutorials/fastnlp_tutorial_paddle_e1.ipynb new file mode 100644 index 00000000..a5883416 --- /dev/null +++ b/docs/source/tutorials/fastnlp_tutorial_paddle_e1.ipynb @@ -0,0 +1,1086 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# E3. 使用 paddlenlp 和 fastNLP 实现中文文本情感分析\n", + "\n", + "本篇教程属于 **fastNLP v1.0 tutorial 的 paddle examples 系列**。在本篇教程中，我们将为您展示如何使用 `paddlenlp` 自然语言处理库和 `fastNLP` 来完成比较简单的情感分析任务。\n", + "\n", + "1. 基础介绍：飞桨自然语言处理库 ``paddlenlp`` 和语义理解框架 ``ERNIE``\n", + "\n", + "2. 准备工作：使用 ``tokenizer`` 处理数据并构造 ``dataloader``\n", + "\n", + "3. 模型训练：加载 ``ERNIE`` 预训练模型，使用 ``fastNLP`` 进行训练" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 1. 基础介绍：飞桨自然语言处理库 paddlenlp 和语义理解框架 ERNIE\n", + "\n", + "#### 1.1 飞桨自然语言处理库 paddlenlp\n", + "\n", + "``paddlenlp`` 是由百度以飞桨 ``PaddlePaddle`` 为核心开发的自然语言处理库，集成了多个数据集和 NLP 模型，包括百度自研的语义理解框架 ``ERNIE`` 。在本篇教程中，我们会以 ``paddlenlp`` 为基础，使用模型 ``ERNIE`` 完成中文情感分析任务。" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "2.3.3\n" + ] + } + ], + "source": [ + "import sys\n", + "sys.path.append(\"../\")\n", + "\n", + "import paddle\n", + "import paddlenlp\n", + "from paddlenlp.transformers import AutoTokenizer\n", + "from paddlenlp.transformers import AutoModelForSequenceClassification\n", + "\n", + "print(paddlenlp.__version__)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### 1.2 语义理解框架 ERNIE\n", + "\n", + "``ERNIE（Enhanced Representation from kNowledge IntEgration）`` 是百度提出的基于知识增强的持续学习语义理解框架，至今已有 ``ERNIE 2.0``、``ERNIE 3.0``、``ERNIE-M``、``ERNIE-tiny`` 等多种预训练模型。``ERNIE 1.0`` 采用``Transformer Encoder`` 作为其语义表示的骨架，并改进了两种 ``mask`` 策略，分别为基于**短语**和**实体**（人名、组织等）的策略。在 ``ERNIE`` 中，由多个字组成的短语或者实体将作为一个统一单元，在训练的时候被统一地 ``mask`` 掉，这样可以潜在地学习到知识的依赖以及更长的语义依赖来让模型更具泛化性。\n", + "\n", + "
![](\"./figures/paddle-ernie-1.0-masking.png\")
\n",
+ "\n",
+ "![](\"./figures/paddle-ernie-1.0-masking-levels.png\")
\n",
+ "\n",
+ "``ERNIE 2.0`` 则提出了连续学习（``Continual Learning``）的概念，即首先用一个简单的任务来初始化模型，在更新时用前一个任务训练好的参数作为下一个任务模型初始化的参数。这样在训练新的任务时，模型便可以记住之前学习到的知识，使得模型在新任务上获得更好的表现。``ERNIE 2.0`` 分别构建了词法、语法、语义不同级别的预训练任务，并使用不同的 task id 来标示不同的任务，在共计16个中英文任务上都取得了SOTA效果。\n",
+ "\n",
+ "![](\"./figures/paddle-ernie-2.0-continual-pretrain.png\")
\n",
+ "\n",
+ "``ERNIE 3.0`` 将自回归和自编码网络融合在一起进行预训练，其中自编码网络采用 ``ERNIE 2.0`` 的多任务学习增量式构建预训练任务，持续进行语义理解学习。其中自编码网络增加了知识增强的预训练任务。自回归网络则基于 ``Tranformer-XL`` 结构，支持长文本语言模型建模，并在多个自然语言处理任务中取得了SOTA的效果。\n",
+ "\n",
+ "![](\"./figures/paddle-ernie-3.0-framework.png\")
\n",
+ "\n",
+ "接下来，我们将展示如何在 ``fastNLP`` 中使用基于 ``paddle`` 的 ``ERNIE 1.0`` 框架进行中文情感分析。"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 2. 使用 tokenizer 处理数据并构造 dataloader\n",
+ "\n",
+ "#### 2.1 加载中文数据集 ChnSentiCorp\n",
+ "\n",
+ "``ChnSentiCorp`` 数据集是由中国科学院发布的中文句子级情感分析数据集，包含了从网络上获取的酒店、电影、书籍等多个领域的评论，每条评论都被划分为两个标签：消极（``0``）和积极（``1``），可以用于二分类的中文情感分析任务。通过 ``paddlenlp.datasets.load_dataset`` 函数，我们可以加载并查看 ``ChnSentiCorp`` 数据集的内容。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "训练集大小： 9600\n",
+ "{'text': '选择珠江花园的原因就是方便，有电动扶梯直接到达海边，周围餐馆、食廊、商场、超市、摊位一应俱全。酒店装修一般，但还算整洁。 泳池在大堂的屋顶，因此很小，不过女儿倒是喜欢。 包的早餐是西式的，还算丰富。 服务吗，一般', 'label': 1, 'qid': ''}\n",
+ "{'text': '15.4寸笔记本的键盘确实爽，基本跟台式机差不多了，蛮喜欢数字小键盘，输数字特方便，样子也很美观，做工也相当不错', 'label': 1, 'qid': ''}\n",
+ "{'text': '房间太小。其他的都一般。。。。。。。。。', 'label': 0, 'qid': ''}\n"
+ ]
+ }
+ ],
+ "source": [
+ "from paddlenlp.datasets import load_dataset\n",
+ "\n",
+ "train_dataset, val_dataset, test_dataset = load_dataset(\"chnsenticorp\", splits=[\"train\", \"dev\", \"test\"])\n",
+ "print(\"训练集大小：\", len(train_dataset))\n",
+ "for i in range(3):\n",
+ " print(train_dataset[i])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### 2.2 处理数据\n",
+ "\n",
+ "可以看到，原本的数据集仅包含中文的文本和标签，这样的数据是无法被模型识别的。同英文文本分类任务一样，我们需要使用 ``tokenizer`` 对文本进行分词并转换为数字形式的结果。我们可以加载已经预训练好的中文分词模型 ``ernie-1.0-base-zh``，将分词的过程写在函数 ``_process`` 中，然后调用数据集的 ``map`` 函数对每一条数据进行分词。其中：\n",
+ "- 参数 ``max_length`` 代表句子的最大长度；\n",
+ "- ``padding=\"max_length\"`` 表示将长度不足的结果 padding 至和最大长度相同；\n",
+ "- ``truncation=True`` 表示将长度过长的句子进行截断。\n",
+ "\n",
+ "至此，我们得到了每条数据长度均相同的数据集。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "\u001b[32m[2022-06-22 21:31:04,168] [ INFO]\u001b[0m - We are using [21:31:16] INFO Running evaluator sanity check for 2 batches. trainer.py:631\n", + "\n" + ], + "text/plain": [ + "\u001b[2;36m[21:31:16]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m Running evaluator sanity check for \u001b[1;36m2\u001b[0m batches. \u001b]8;id=4641;file://../fastNLP/core/controllers/trainer.py\u001b\\\u001b[2mtrainer.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=822054;file://../fastNLP/core/controllers/trainer.py#631\u001b\\\u001b[2m631\u001b[0m\u001b]8;;\u001b\\\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "\n" + ], + "text/plain": [] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
---------------------------- Eval. results on Epoch:0, Batch:60 -----------------------------\n", + "\n" + ], + "text/plain": [ + "---------------------------- Eval. results on Epoch:\u001b[1;36m0\u001b[0m, Batch:\u001b[1;36m60\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#accuracy\": 0.895833,\n", + " \"total#accuracy\": 1200.0,\n", + " \"correct#accuracy\": 1075.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.895833\u001b[0m,\n", + " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n", + " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1075.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
---------------------------- Eval. results on Epoch:0, Batch:120 ----------------------------\n", + "\n" + ], + "text/plain": [ + "---------------------------- Eval. results on Epoch:\u001b[1;36m0\u001b[0m, Batch:\u001b[1;36m120\u001b[0m ----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#accuracy\": 0.8975,\n", + " \"total#accuracy\": 1200.0,\n", + " \"correct#accuracy\": 1077.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.8975\u001b[0m,\n", + " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n", + " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1077.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
---------------------------- Eval. results on Epoch:0, Batch:180 ----------------------------\n", + "\n" + ], + "text/plain": [ + "---------------------------- Eval. results on Epoch:\u001b[1;36m0\u001b[0m, Batch:\u001b[1;36m180\u001b[0m ----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#accuracy\": 0.911667,\n", + " \"total#accuracy\": 1200.0,\n", + " \"correct#accuracy\": 1094.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.911667\u001b[0m,\n", + " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n", + " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1094.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
---------------------------- Eval. results on Epoch:0, Batch:240 ----------------------------\n", + "\n" + ], + "text/plain": [ + "---------------------------- Eval. results on Epoch:\u001b[1;36m0\u001b[0m, Batch:\u001b[1;36m240\u001b[0m ----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#accuracy\": 0.9225,\n", + " \"total#accuracy\": 1200.0,\n", + " \"correct#accuracy\": 1107.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.9225\u001b[0m,\n", + " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n", + " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1107.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
---------------------------- Eval. results on Epoch:0, Batch:300 ----------------------------\n", + "\n" + ], + "text/plain": [ + "---------------------------- Eval. results on Epoch:\u001b[1;36m0\u001b[0m, Batch:\u001b[1;36m300\u001b[0m ----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#accuracy\": 0.9275,\n", + " \"total#accuracy\": 1200.0,\n", + " \"correct#accuracy\": 1113.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.9275\u001b[0m,\n", + " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n", + " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1113.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
---------------------------- Eval. results on Epoch:1, Batch:60 -----------------------------\n", + "\n" + ], + "text/plain": [ + "---------------------------- Eval. results on Epoch:\u001b[1;36m1\u001b[0m, Batch:\u001b[1;36m60\u001b[0m -----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#accuracy\": 0.930833,\n", + " \"total#accuracy\": 1200.0,\n", + " \"correct#accuracy\": 1117.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.930833\u001b[0m,\n", + " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n", + " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1117.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
---------------------------- Eval. results on Epoch:1, Batch:120 ----------------------------\n", + "\n" + ], + "text/plain": [ + "---------------------------- Eval. results on Epoch:\u001b[1;36m1\u001b[0m, Batch:\u001b[1;36m120\u001b[0m ----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#accuracy\": 0.935833,\n", + " \"total#accuracy\": 1200.0,\n", + " \"correct#accuracy\": 1123.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.935833\u001b[0m,\n", + " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n", + " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1123.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
---------------------------- Eval. results on Epoch:1, Batch:180 ----------------------------\n", + "\n" + ], + "text/plain": [ + "---------------------------- Eval. results on Epoch:\u001b[1;36m1\u001b[0m, Batch:\u001b[1;36m180\u001b[0m ----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#accuracy\": 0.935833,\n", + " \"total#accuracy\": 1200.0,\n", + " \"correct#accuracy\": 1123.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.935833\u001b[0m,\n", + " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n", + " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1123.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
---------------------------- Eval. results on Epoch:1, Batch:240 ----------------------------\n", + "\n" + ], + "text/plain": [ + "---------------------------- Eval. results on Epoch:\u001b[1;36m1\u001b[0m, Batch:\u001b[1;36m240\u001b[0m ----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#accuracy\": 0.9375,\n", + " \"total#accuracy\": 1200.0,\n", + " \"correct#accuracy\": 1125.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.9375\u001b[0m,\n", + " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n", + " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1125.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
---------------------------- Eval. results on Epoch:1, Batch:300 ----------------------------\n", + "\n" + ], + "text/plain": [ + "---------------------------- Eval. results on Epoch:\u001b[1;36m1\u001b[0m, Batch:\u001b[1;36m300\u001b[0m ----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"acc#accuracy\": 0.941667,\n", + " \"total#accuracy\": 1200.0,\n", + " \"correct#accuracy\": 1130.0\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.941667\u001b[0m,\n", + " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n", + " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1130.0\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
[21:34:28] INFO Loading best model from fnlp-ernie/2022-0 load_best_model_callback.py:111\n", + " 6-22-21_29_12_898095/best_so_far with \n", + " acc#accuracy: 0.941667... \n", + "\n" + ], + "text/plain": [ + "\u001b[2;36m[21:34:28]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m Loading best model from fnlp-ernie/\u001b[1;36m2022\u001b[0m-\u001b[1;36m0\u001b[0m \u001b]8;id=340364;file://../fastNLP/core/callbacks/load_best_model_callback.py\u001b\\\u001b[2mload_best_model_callback.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=763898;file://../fastNLP/core/callbacks/load_best_model_callback.py#111\u001b\\\u001b[2m111\u001b[0m\u001b]8;;\u001b\\\n", + "\u001b[2;36m \u001b[0m \u001b[1;36m6\u001b[0m-\u001b[1;36m22\u001b[0m-21_29_12_898095/best_so_far with \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m acc#accuracy: \u001b[1;36m0.941667\u001b[0m\u001b[33m...\u001b[0m \u001b[2m \u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
[21:34:34] INFO Deleting fnlp-ernie/2022-06-22-21_29_12_8 load_best_model_callback.py:131\n", + " 98095/best_so_far... \n", + "\n" + ], + "text/plain": [ + "\u001b[2;36m[21:34:34]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m Deleting fnlp-ernie/\u001b[1;36m2022\u001b[0m-\u001b[1;36m06\u001b[0m-\u001b[1;36m22\u001b[0m-21_29_12_8 \u001b]8;id=430330;file://../fastNLP/core/callbacks/load_best_model_callback.py\u001b\\\u001b[2mload_best_model_callback.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=508566;file://../fastNLP/core/callbacks/load_best_model_callback.py#131\u001b\\\u001b[2m131\u001b[0m\u001b]8;;\u001b\\\n", + "\u001b[2;36m \u001b[0m 98095/best_so_far\u001b[33m...\u001b[0m \u001b[2m \u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "\n" + ], + "text/plain": [] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "from fastNLP import LRSchedCallback, LoadBestModelCallback\n", + "from fastNLP import Trainer, Accuracy\n", + "from paddlenlp.transformers import LinearDecayWithWarmup\n", + "\n", + "n_epochs = 2\n", + "num_training_steps = len(train_dataloader) * n_epochs\n", + "lr_scheduler = LinearDecayWithWarmup(5e-5, num_training_steps, 0.1)\n", + "optimizer = paddle.optimizer.AdamW(\n", + " learning_rate=lr_scheduler,\n", + " parameters=model.parameters(),\n", + ")\n", + "callbacks = [\n", + " LRSchedCallback(lr_scheduler, step_on=\"batch\"),\n", + " LoadBestModelCallback(\"acc#accuracy\", larger_better=True, save_folder=\"fnlp-ernie\"),\n", + "]\n", + "trainer = Trainer(\n", + " model=model,\n", + " driver=\"paddle\",\n", + " optimizers=optimizer,\n", + " device=0,\n", + " n_epochs=n_epochs,\n", + " train_dataloader=train_dataloader,\n", + " evaluate_dataloaders=val_dataloader,\n", + " evaluate_every=60,\n", + " metrics={\"accuracy\": Accuracy()},\n", + " callbacks=callbacks,\n", + ")\n", + "trainer.run()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### 3.3 测试和评估\n", + "\n", + "现在我们已经得到了一个表现良好的 ``ERNIE`` 模型，接下来可以在测试集上测试模型的效果了。``fastNLP.Evaluator`` 提供了定制函数的功能。我们以 ``test_dataloader`` 初始化一个 ``Evaluator``，然后将写好的测试函数 ``test_batch_step_fn`` 传给参数 ``evaluate_batch_step_fn``，``Evaluate`` 在对每个 batch 进行评估时就会调用我们自定义的 ``test_batch_step_fn`` 函数而不是 ``evaluate_step`` 函数。在这里，我们仅测试 5 条数据并输出文本和对应的标签。" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
text: ['这个宾馆比较陈旧了，特价的房间也很一般。总体来说一般']\n", + "\n" + ], + "text/plain": [ + "text: ['这个宾馆比较陈旧了，特价的房间也很一般。总体来说一般']\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
labels: 0\n", + "\n" + ], + "text/plain": [ + "labels: 0\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
text: ['怀着十分激动的心情放映，可是看着看着发现，在放映完毕后，出现一集米老鼠的动画片！开始\n", + "还怀疑是不是赠送的个别现象，可是后来发现每张DVD后面都有！真不知道生产商怎么想的，我想看的是猫\n", + "和老鼠，不是米老鼠！如果厂家是想赠送的话，那就全套米老鼠和唐老鸭都赠送，只在每张DVD后面添加一\n", + "集算什么？？简直是画蛇添足！！']\n", + "\n" + ], + "text/plain": [ + "text: ['怀着十分激动的心情放映，可是看着看着发现，在放映完毕后，出现一集米老鼠的动画片！开始\n", + "还怀疑是不是赠送的个别现象，可是后来发现每张DVD后面都有！真不知道生产商怎么想的，我想看的是猫\n", + "和老鼠，不是米老鼠！如果厂家是想赠送的话，那就全套米老鼠和唐老鸭都赠送，只在每张DVD后面添加一\n", + "集算什么？？简直是画蛇添足！！']\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
labels: 0\n", + "\n" + ], + "text/plain": [ + "labels: 0\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
text: ['还稍微重了点，可能是硬盘大的原故，还要再轻半斤就好了。其他要进一步验证。贴的几种膜气\n", + "泡较多，用不了多久就要更换了，屏幕膜稍好点，但比没有要强多了。建议配赠几张膜让用用户自己贴。'\n", + "]\n", + "\n" + ], + "text/plain": [ + "text: ['还稍微重了点，可能是硬盘大的原故，还要再轻半斤就好了。其他要进一步验证。贴的几种膜气\n", + "泡较多，用不了多久就要更换了，屏幕膜稍好点，但比没有要强多了。建议配赠几张膜让用用户自己贴。'\n", + "]\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
labels: 0\n", + "\n" + ], + "text/plain": [ + "labels: 0\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
text: ['交通方便；环境很好；服务态度很好 房间较小']\n", + "\n" + ], + "text/plain": [ + "text: ['交通方便；环境很好；服务态度很好 房间较小']\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
labels: 1\n", + "\n" + ], + "text/plain": [ + "labels: 1\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
text: ['不错，作者的观点很颠覆目前中国父母的教育方式，其实古人们对于教育已经有了很系统的体系\n", + "了，可是现在的父母以及祖父母们更多的娇惯纵容孩子，放眼看去自私的孩子是大多数，父母觉得自己的\n", + "孩子在外面只要不吃亏就是好事，完全把古人几千年总结的教育古训抛在的九霄云外。所以推荐准妈妈们\n", + "可以在等待宝宝降临的时候，好好学习一下，怎么把孩子教育成一个有爱心、有责任心、宽容、大度的人\n", + "。']\n", + "\n" + ], + "text/plain": [ + "text: ['不错，作者的观点很颠覆目前中国父母的教育方式，其实古人们对于教育已经有了很系统的体系\n", + "了，可是现在的父母以及祖父母们更多的娇惯纵容孩子，放眼看去自私的孩子是大多数，父母觉得自己的\n", + "孩子在外面只要不吃亏就是好事，完全把古人几千年总结的教育古训抛在的九霄云外。所以推荐准妈妈们\n", + "可以在等待宝宝降临的时候，好好学习一下，怎么把孩子教育成一个有爱心、有责任心、宽容、大度的人\n", + "。']\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
labels: 1\n", + "\n" + ], + "text/plain": [ + "labels: 1\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "\n" + ], + "text/plain": [] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/plain": [ + "{}" + ] + }, + "execution_count": 14, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from fastNLP import Evaluator\n", + "def test_batch_step_fn(evaluator, batch):\n", + " input_ids = batch[\"input_ids\"]\n", + " attention_mask = batch[\"attention_mask\"]\n", + " token_type_ids = batch[\"token_type_ids\"]\n", + " logits = model(input_ids, attention_mask, token_type_ids)\n", + " predict = logits.argmax().item()\n", + " print(\"text:\", batch['text'])\n", + " print(\"labels:\", predict)\n", + "\n", + "evaluator = Evaluator(\n", + " model=model,\n", + " dataloaders=test_dataloader,\n", + " driver=\"paddle\",\n", + " device=0,\n", + " evaluate_batch_step_fn=test_batch_step_fn,\n", + ")\n", + "evaluator.run(5) " + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.13 ('fnlp-paddle')", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.13" + }, + "orig_nbformat": 4, + "vscode": { + "interpreter": { + "hash": "31f2d9d3efc23c441973d7c4273acfea8b132b6a578f002629b6b44b8f65e720" + } + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/docs/source/tutorials/fastnlp_tutorial_paddle_e2.ipynb b/docs/source/tutorials/fastnlp_tutorial_paddle_e2.ipynb new file mode 100644 index 00000000..439d7f9f --- /dev/null +++ b/docs/source/tutorials/fastnlp_tutorial_paddle_e2.ipynb @@ -0,0 +1,1510 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# E4. 使用 paddlenlp 和 fastNLP 训练中文阅读理解任务\n", + "\n", + "本篇教程属于 **fastNLP v1.0 tutorial 的 paddle examples 系列**。在本篇教程中，我们将为您展示如何在 `fastNLP` 中通过自定义 `Metric` 和 损失函数来完成进阶的问答任务。\n", + "\n", + "1. 基础介绍：自然语言处理中的阅读理解任务\n", + "\n", + "2. 准备工作：加载 `DuReader-robust` 数据集，并使用 `tokenizer` 处理数据\n", + "\n", + "3. 模型训练：自己定义评测用的 `Metric` 实现更加自由的任务评测" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 1. 基础介绍：自然语言处理中的阅读理解任务\n", + "\n", + "阅读理解任务，顾名思义，就是给出一段文字，然后让模型理解这段文字所含的语义。大部分机器阅读理解任务都采用问答式测评，即设计与文章内容相关的自然语言式问题，让模型理解问题并根据文章作答。与文本分类任务不同的是，在阅读理解任务中我们有时需要需要输入“一对”句子，分别代表问题和上下文；答案的格式也分为多种：\n", + "\n", + "- 多项选择：让模型从多个答案选项中选出正确答案\n", + "- 区间答案：答案为上下文的一段子句，需要模型给出答案的起始位置\n", + "- 自由回答：不做限制，让模型自行生成答案\n", + "- 完形填空：在原文中挖空部分关键词，让模型补全；这类答案往往不需要问题\n", + "\n", + "如果您对 `transformers` 有所了解的话，其中的 `ModelForQuestionAnswering` 系列模型就可以用于这项任务。阅读理解模型的泛用性是衡量该技术能否在实际应用中大规模落地的重要指标之一，随着当前技术的进步，许多模型虽然能够在一些测试集上取得较好的性能，但在实际应用中，这些模型仍然难以让人满意。在本篇教程中，我们将会为您展示如何训练一个问答模型。\n", + "\n", + "在这一领域，`SQuAD` 数据集是一个影响深远的数据集。它的全称是斯坦福问答数据集（Stanford Question Answering Dataset），每条数据包含 `（问题，上下文，答案）` 三部分，规模大（约十万条，2.0又新增了五万条），在提出之后很快成为训练问答任务的经典数据集之一。`SQuAD` 数据集有两个指标来衡量模型的表现：`EM`（Exact Match，精确匹配）和 `F1`（模糊匹配）。前者反应了模型给出的答案中有多少和正确答案完全一致，后者则反应了模型给出的答案中与正确答案重叠的部分，均为越高越好。" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 2. 准备工作：加载 DuReader-robust 数据集，并使用 tokenizer 处理数据" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/remote-home/shxing/anaconda3/envs/fnlp-paddle/lib/python3.7/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n", + " from .autonotebook import tqdm as notebook_tqdm\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "2.3.3\n" + ] + } + ], + "source": [ + "import sys\n", + "sys.path.append(\"../\")\n", + "import paddle\n", + "import paddlenlp\n", + "\n", + "print(paddlenlp.__version__)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "在数据集方面，我们选用 `DuReader-robust` 中文数据集作为训练数据。它是一种抽取式问答数据集，采用 `SQuAD` 数据格式，能够评估真实应用场景下模型的泛用性。" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Reusing dataset dureader_robust (/remote-home/shxing/.cache/huggingface/datasets/dureader_robust/plain_text/1.0.0/d462ecadc8c010cee20f57632f1413f272867cd802a91a602df48c7d34eb0c27)\n", + "Reusing dataset dureader_robust (/remote-home/shxing/.cache/huggingface/datasets/dureader_robust/plain_text/1.0.0/d462ecadc8c010cee20f57632f1413f272867cd802a91a602df48c7d34eb0c27)\n", + "\u001b[32m[2022-06-27 19:22:46,998] [ INFO]\u001b[0m - Already cached /remote-home/shxing/.paddlenlp/models/ernie-1.0-base-zh/vocab.txt\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'id': '0a25cb4bc1ab6f474c699884e04601e4', 'title': '', 'context': '第35集雪见缓缓张开眼睛，景天又惊又喜之际，长卿和紫萱的仙船驶至，见众人无恙，也十分高兴。众人登船，用尽合力把自身的真气和水分输给她。雪见终于醒过来了，但却一脸木然，全无反应。众人向常胤求助，却发现人世界竟没有雪见的身世纪录。长卿询问清微的身世，清微语带双关说一切上了天界便有答案。长卿驾驶仙船，众人决定立马动身，往天界而去。众人来到一荒山，长卿指出，魔界和天界相连。由魔界进入通过神魔之井，便可登天。众人至魔界入口，仿若一黑色的蝙蝠洞，但始终无法进入。后来花楹发现只要有翅膀便能飞入。于是景天等人打下许多乌鸦，模仿重楼的翅膀，制作数对翅膀状巨物。刚佩戴在身，便被吸入洞口。众人摔落在地，抬头发现魔界守卫。景天和众魔套交情，自称和魔尊重楼相熟，众魔不理，打了起来。', 'question': '仙剑奇侠传3第几集上天界', 'answers': {'text': ['第35集'], 'answer_start': [0]}}\n", + "{'id': '7de192d6adf7d60ba73ba25cf590cc1e', 'title': '', 'context': '选择燃气热水器时，一定要关注这几个问题：1、出水稳定性要好，不能出现忽热忽冷的现象2、快速到达设定的需求水温3、操作要智能、方便4、安全性要好，要装有安全报警装置 市场上燃气热水器品牌众多，购买时还需多加对比和仔细鉴别。方太今年主打的磁化恒温热水器在使用体验方面做了全面升级：9秒速热，可快速进入洗浴模式；水温持久稳定，不会出现忽热忽冷的现象，并通过水量伺服技术将出水温度精确控制在±0.5℃，可满足家里宝贝敏感肌肤洗护需求；配备CO和CH4双气体报警装置更安全（市场上一般多为CO单气体报警）。另外，这款热水器还有智能WIFI互联功能，只需下载个手机APP即可用手机远程操作热水器，实现精准调节水温，满足家人多样化的洗浴需求。当然方太的磁化恒温系列主要的是增加磁化功能，可以有效吸附水中的铁锈、铁屑等微小杂质，防止细菌滋生，使沐浴水质更洁净，长期使用磁化水沐浴更利于身体健康。', 'question': '燃气热水器哪个牌子好', 'answers': {'text': ['方太'], 'answer_start': [110]}}\n", + "{'id': 'b9e74d4b9228399b03701d1fe6d52940', 'title': '', 'context': '迈克尔.乔丹在NBA打了15个赛季。他在84年进入nba，期间在1993年10月6日第一次退役改打棒球，95年3月18日重新回归，在99年1月13日第二次退役，后于2001年10月31日复出，在03年最终退役。迈克尔·乔丹（Michael Jordan），1963年2月17日生于纽约布鲁克林，美国著名篮球运动员，司职得分后卫，历史上最伟大的篮球运动员。1984年的NBA选秀大会，乔丹在首轮第3顺位被芝加哥公牛队选中。 1986-87赛季，乔丹场均得到37.1分，首次获得分王称号。1990-91赛季，乔丹连夺常规赛MVP和总决赛MVP称号，率领芝加哥公牛首次夺得NBA总冠军。 1997-98赛季，乔丹获得个人职业生涯第10个得分王，并率领公牛队第六次夺得总冠军。2009年9月11日，乔丹正式入选NBA名人堂。', 'question': '乔丹打了多少个赛季', 'answers': {'text': ['15个'], 'answer_start': [12]}}\n", + "训练集大小： 14520\n", + "验证集大小： 1417\n" + ] + } + ], + "source": [ + "from paddlenlp.datasets import load_dataset\n", + "train_dataset = load_dataset(\"PaddlePaddle/dureader_robust\", splits=\"train\")\n", + "val_dataset = load_dataset(\"PaddlePaddle/dureader_robust\", splits=\"validation\")\n", + "for i in range(3):\n", + " print(train_dataset[i])\n", + "print(\"训练集大小：\", len(train_dataset))\n", + "print(\"验证集大小：\", len(val_dataset))\n", + "\n", + "MODEL_NAME = \"ernie-1.0-base-zh\"\n", + "from paddlenlp.transformers import ErnieTokenizer\n", + "tokenizer =ErnieTokenizer.from_pretrained(MODEL_NAME)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### 2.1 处理训练集\n", + "\n", + "对于阅读理解任务，数据处理的方式较为麻烦。接下来我们会为您详细讲解处理函数 `_process_train` 的功能，同时也将通过实践展示关于 `tokenizer` 的更多功能，让您更加深入地了解自然语言处理任务。首先让我们向 `tokenizer` 输入一条数据（以列表的形式）：" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "2\n", + "dict_keys(['offset_mapping', 'input_ids', 'token_type_ids', 'overflow_to_sample'])\n" + ] + } + ], + "source": [ + "result = tokenizer(\n", + " [train_dataset[0][\"question\"]],\n", + " [train_dataset[0][\"context\"]],\n", + " stride=128,\n", + " max_length=256,\n", + " padding=\"max_length\",\n", + " return_dict=False\n", + ")\n", + "\n", + "print(len(result))\n", + "print(result[0].keys())" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "首先不难理解的是，模型必须要同时接受问题（`question`）和上下文（`context`）才能够进行阅读理解，因此我们需要将二者同时进行分词（`tokenize`）。所幸，`Tokenizer` 提供了这一功能，当我们调用 `tokenizer` 的时候，其第一个参数名为 `text`，第二个参数名为 `text_pair`，这使得我们可以同时对一对文本进行分词。同时，`tokenizer` 还需要标记出一条数据中哪些属于问题，哪些属于上下文，这一功能则由 `token_type_ids` 完成。`token_type_ids` 会将输入的第一个文本（问题）标记为 `0`，第二个文本（上下文）标记为 `1`，这样模型在训练时便可以将问题和上下文区分开来：" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[1, 1034, 1189, 734, 2003, 241, 284, 131, 553, 271, 28, 125, 280, 2, 131, 1773, 271, 1097, 373, 1427, 1427, 501, 88, 662, 1906, 4, 561, 125, 311, 1168, 311, 692, 46, 430, 4, 84, 2073, 14, 1264, 3967, 5, 1034, 1020, 1829, 268, 4, 373, 539, 8, 154, 5210, 4, 105, 167, 59, 69, 685, 12043, 539, 8, 883, 1020, 4, 29, 720, 95, 90, 427, 67, 262, 5, 384, 266, 14, 101, 59, 789, 416, 237, 12043, 1097, 373, 616, 37, 1519, 93, 61, 15, 4, 255, 535, 7, 1529, 619, 187, 4, 62, 154, 451, 149, 12043, 539, 8, 253, 223, 3679, 323, 523, 4, 535, 34, 87, 8, 203, 280, 1186, 340, 9, 1097, 373, 5, 262, 203, 623, 704, 12043, 84, 2073, 1137, 358, 334, 702, 5, 262, 203, 4, 334, 702, 405, 360, 653, 129, 178, 7, 568, 28, 15, 125, 280, 518, 9, 1179, 487, 12043, 84, 2073, 1621, 1829, 1034, 1020, 4, 539, 8, 448, 91, 202, 466, 70, 262, 4, 638, 125, 280, 83, 299, 12043, 539, 8, 61, 45, 7, 1537, 176, 4, 84, 2073, 288, 39, 4, 889, 280, 14, 125, 280, 156, 538, 12043, 190, 889, 280, 71, 109, 124, 93, 292, 889, 46, 1248, 4, 518, 48, 883, 125, 12043, 539, 8, 268, 889, 280, 109, 270, 4, 1586, 845, 7, 669, 199, 5, 3964, 3740, 1084, 4, 255, 440, 616, 154, 72, 71, 109, 12043, 49, 61, 283, 3591, 34, 87, 297, 41, 9, 1993, 2602, 518, 52, 706, 109, 2]\n", + "['[CLS]', '仙', '剑', '奇', '侠', '传', '3', '第', '几', '集', '上', '天', '界', '[SEP]', '第', '35', '集', '雪', '见', '缓', '缓', '张', '开', '眼', '睛', '，', '景', '天', '又', '惊', '又', '喜', '之', '际', '，', '长', '卿', '和', '紫', '萱', '的', '仙', '船', '驶', '至', '，', '见', '众', '人', '无', '恙', '，', '也', '十', '分', '高', '兴', '。', '众', '人', '登', '船', '，', '用', '尽', '合', '力', '把', '自', '身', '的', '真', '气', '和', '水', '分', '输', '给', '她', '。', '雪', '见', '终', '于', '醒', '过', '来', '了', '，', '但', '却', '一', '脸', '木', '然', '，', '全', '无', '反', '应', '。', '众', '人', '向', '常', '胤', '求', '助', '，', '却', '发', '现', '人', '世', '界', '竟', '没', '有', '雪', '见', '的', '身', '世', '纪', '录', '。', '长', '卿', '询', '问', '清', '微', '的', '身', '世', '，', '清', '微', '语', '带', '双', '关', '说', '一', '切', '上', '了', '天', '界', '便', '有', '答', '案', '。', '长', '卿', '驾', '驶', '仙', '船', '，', '众', '人', '决', '定', '立', '马', '动', '身', '，', '往', '天', '界', '而', '去', '。', '众', '人', '来', '到', '一', '荒', '山', '，', '长', '卿', '指', '出', '，', '魔', '界', '和', '天', '界', '相', '连', '。', '由', '魔', '界', '进', '入', '通', '过', '神', '魔', '之', '井', '，', '便', '可', '登', '天', '。', '众', '人', '至', '魔', '界', '入', '口', '，', '仿', '若', '一', '黑', '色', '的', '蝙', '蝠', '洞', '，', '但', '始', '终', '无', '法', '进', '入', '。', '后', '来', '花', '楹', '发', '现', '只', '要', '有', '翅', '膀', '便', '能', '飞', '入', '[SEP]']\n", + "[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n" + ] + } + ], + "source": [ + "print(result[0][\"input_ids\"])\n", + "print(tokenizer.convert_ids_to_tokens(result[0][\"input_ids\"]))\n", + "print(result[0][\"token_type_ids\"])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "根据上面的输出我们可以看出，`tokenizer` 会将数据开头用 `[CLS]` 标记，用 `[SEP]` 来分割句子。同时，根据 `token_type_ids` 得到的 0、1 串，我们也很容易将问题和上下文区分开。顺带一提，如果一条数据进行了 `padding`，那么这部分会被标记为 `0` 。\n", + "\n", + "在输出的 `keys` 中还有一项名为 `offset_mapping` 的键。该项数据能够表示分词后的每个 `token` 在原文中对应文字或词语的位置。比如我们可以像下面这样将数据打印出来：" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[(0, 0), (0, 1), (1, 2), (2, 3), (3, 4), (4, 5), (5, 6), (6, 7), (7, 8), (8, 9), (9, 10), (10, 11), (11, 12), (0, 0), (0, 1), (1, 3), (3, 4), (4, 5), (5, 6), (6, 7)]\n", + "[1, 1034, 1189, 734, 2003, 241, 284, 131, 553, 271, 28, 125, 280, 2, 131, 1773, 271, 1097, 373, 1427]\n", + "['[CLS]', '仙', '剑', '奇', '侠', '传', '3', '第', '几', '集', '上', '天', '界', '[SEP]', '第', '35', '集', '雪', '见', '缓']\n" + ] + } + ], + "source": [ + "print(result[0][\"offset_mapping\"][:20])\n", + "print(result[0][\"input_ids\"][:20])\n", + "print(tokenizer.convert_ids_to_tokens(result[0][\"input_ids\"])[:20])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "`[CLS]` 由于是 `tokenizer` 自己添加进去用于标记数据的 `token`，因此它在原文中找不到任何对应的词语，所以给出的位置范围就是 `(0, 0)`；第二个 `token` 对应第一个 `“仙”` 字，因此映射的位置就是 `(0, 1)`；同理，后面的 `[SEP]` 也不对应任何文字，映射的位置为 `(0, 0)`；而接下来的 `token` 对应 **上下文** 中的第一个字 `“第”`，映射出的位置为 `(0, 1)`；再后面的 `token` 对应原文中的两个字符 `35`，因此其位置映射为 `(1, 3)` 。通过这种手段，我们可以更方便地获取 `token` 与原文的对应关系。\n", + "\n", + "最后，您也许会注意到我们获取的 `result` 长度为 2 。这是文本在分词后长度超过了 `max_length` 256 ，`tokenizer` 将数据分成了两部分所致。在阅读理解任务中，我们不可能像文本分类那样轻易地将一条数据截断，因为答案很可能就出现在后面被丢弃的那部分数据中，因此，我们需要保留所有的数据（当然，您也可以直接丢弃这些超长的数据）。`overflow_to_sample` 则可以标识当前数据在原数据的索引：" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[CLS]仙剑奇侠传3第几集上天界[SEP]第35集雪见缓缓张开眼睛，景天又惊又喜之际，长卿和紫萱的仙船驶至，见众人无恙，也十分高兴。众人登船，用尽合力把自身的真气和水分输给她。雪见终于醒过来了，但却一脸木然，全无反应。众人向常胤求助，却发现人世界竟没有雪见的身世纪录。长卿询问清微的身世，清微语带双关说一切上了天界便有答案。长卿驾驶仙船，众人决定立马动身，往天界而去。众人来到一荒山，长卿指出，魔界和天界相连。由魔界进入通过神魔之井，便可登天。众人至魔界入口，仿若一黑色的蝙蝠洞，但始终无法进入。后来花楹发现只要有翅膀便能飞入[SEP]\n", + "overflow_to_sample: 0\n", + "[CLS]仙剑奇侠传3第几集上天界[SEP]说一切上了天界便有答案。长卿驾驶仙船，众人决定立马动身，往天界而去。众人来到一荒山，长卿指出，魔界和天界相连。由魔界进入通过神魔之井，便可登天。众人至魔界入口，仿若一黑色的蝙蝠洞，但始终无法进入。后来花楹发现只要有翅膀便能飞入。于是景天等人打下许多乌鸦，模仿重楼的翅膀，制作数对翅膀状巨物。刚佩戴在身，便被吸入洞口。众人摔落在地，抬头发现魔界守卫。景天和众魔套交情，自称和魔尊重楼相熟，众魔不理，打了起来。[SEP][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD]\n", + "overflow_to_sample: 0\n" + ] + } + ], + "source": [ + "for res in result:\n", + " tokens = tokenizer.convert_ids_to_tokens(res[\"input_ids\"])\n", + " print(\"\".join(tokens))\n", + " print(\"overflow_to_sample: \", res[\"overflow_to_sample\"])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "将两条数据均输出之后可以看到，它们都出自我们传入的数据，并且存在一部分重合。`tokenizer` 的 `stride` 参数可以设置重合部分的长度，这也可以帮助模型识别被分割开的两条数据；`overflow_to_sample` 的 `0` 则代表它们来自于第 `0` 条数据。\n", + "\n", + "基于以上信息，我们处理训练集的思路如下：\n", + "\n", + "1. 通过 `overflow_to_sample` 来获取原来的数据\n", + "2. 通过原数据的 `answers` 找到答案的起始位置\n", + "3. 通过 `offset_mapping` 给出的映射关系在分词处理后的数据中找到答案的起始位置，分别记录在 `start_pos` 和 `end_pos` 中；如果没有找到答案（比如答案被截断了），那么答案的起始位置就被标记为 `[CLS]` 的位置。\n", + "\n", + "这样 `_process_train` 函数就呼之欲出了，我们调用 `train_dataset.map` 函数，并将 `batched` 参数设置为 `True` ，将所有数据批量地进行更新。有一点需要注意的是，**在处理过后数据量会增加**。" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'offset_mapping': [(0, 0), (0, 1), (1, 2), (2, 3), (3, 4), (4, 5), (5, 6), (6, 7), (7, 8), (8, 9), (9, 10), (10, 11), (11, 12), (0, 0), (0, 1), (1, 3), (3, 4), (4, 5), (5, 6), (6, 7), (7, 8), (8, 9), (9, 10), (10, 11), (11, 12), (12, 13), (13, 14), (14, 15), (15, 16), (16, 17), (17, 18), (18, 19), (19, 20), (20, 21), (21, 22), (22, 23), (23, 24), (24, 25), (25, 26), (26, 27), (27, 28), (28, 29), (29, 30), (30, 31), (31, 32), (32, 33), (33, 34), (34, 35), (35, 36), (36, 37), (37, 38), (38, 39), (39, 40), (40, 41), (41, 42), (42, 43), (43, 44), (44, 45), (45, 46), (46, 47), (47, 48), (48, 49), (49, 50), (50, 51), (51, 52), (52, 53), (53, 54), (54, 55), (55, 56), (56, 57), (57, 58), (58, 59), (59, 60), (60, 61), (61, 62), (62, 63), (63, 64), (64, 65), (65, 66), (66, 67), (67, 68), (68, 69), (69, 70), (70, 71), (71, 72), (72, 73), (73, 74), (74, 75), (75, 76), (76, 77), (77, 78), (78, 79), (79, 80), (80, 81), (81, 82), (82, 83), (83, 84), (84, 85), (85, 86), (86, 87), (87, 88), (88, 89), (89, 90), (90, 91), (91, 92), (92, 93), (93, 94), (94, 95), (95, 96), (96, 97), (97, 98), (98, 99), (99, 100), (100, 101), (101, 102), (102, 103), (103, 104), (104, 105), (105, 106), (106, 107), (107, 108), (108, 109), (109, 110), (110, 111), (111, 112), (112, 113), (113, 114), (114, 115), (115, 116), (116, 117), (117, 118), (118, 119), (119, 120), (120, 121), (121, 122), (122, 123), (123, 124), (124, 125), (125, 126), (126, 127), (127, 128), (128, 129), (129, 130), (130, 131), (131, 132), (132, 133), (133, 134), (134, 135), (135, 136), (136, 137), (137, 138), (138, 139), (139, 140), (140, 141), (141, 142), (142, 143), (143, 144), (144, 145), (145, 146), (146, 147), (147, 148), (148, 149), (149, 150), (150, 151), (151, 152), (152, 153), (153, 154), (154, 155), (155, 156), (156, 157), (157, 158), (158, 159), (159, 160), (160, 161), (161, 162), (162, 163), (163, 164), (164, 165), (165, 166), (166, 167), (167, 168), (168, 169), (169, 170), (170, 171), (171, 172), (172, 173), (173, 174), (174, 175), (175, 176), (176, 177), (177, 178), (178, 179), (179, 180), (180, 181), (181, 182), (182, 183), (183, 184), (184, 185), (185, 186), (186, 187), (187, 188), (188, 189), (189, 190), (190, 191), (191, 192), (192, 193), (193, 194), (194, 195), (195, 196), (196, 197), (197, 198), (198, 199), (199, 200), (200, 201), (201, 202), (202, 203), (203, 204), (204, 205), (205, 206), (206, 207), (207, 208), (208, 209), (209, 210), (210, 211), (211, 212), (212, 213), (213, 214), (214, 215), (215, 216), (216, 217), (217, 218), (218, 219), (219, 220), (220, 221), (221, 222), (222, 223), (223, 224), (224, 225), (225, 226), (226, 227), (227, 228), (228, 229), (229, 230), (230, 231), (231, 232), (232, 233), (233, 234), (234, 235), (235, 236), (236, 237), (237, 238), (238, 239), (239, 240), (240, 241), (241, 242), (0, 0)], 'input_ids': [1, 1034, 1189, 734, 2003, 241, 284, 131, 553, 271, 28, 125, 280, 2, 131, 1773, 271, 1097, 373, 1427, 1427, 501, 88, 662, 1906, 4, 561, 125, 311, 1168, 311, 692, 46, 430, 4, 84, 2073, 14, 1264, 3967, 5, 1034, 1020, 1829, 268, 4, 373, 539, 8, 154, 5210, 4, 105, 167, 59, 69, 685, 12043, 539, 8, 883, 1020, 4, 29, 720, 95, 90, 427, 67, 262, 5, 384, 266, 14, 101, 59, 789, 416, 237, 12043, 1097, 373, 616, 37, 1519, 93, 61, 15, 4, 255, 535, 7, 1529, 619, 187, 4, 62, 154, 451, 149, 12043, 539, 8, 253, 223, 3679, 323, 523, 4, 535, 34, 87, 8, 203, 280, 1186, 340, 9, 1097, 373, 5, 262, 203, 623, 704, 12043, 84, 2073, 1137, 358, 334, 702, 5, 262, 203, 4, 334, 702, 405, 360, 653, 129, 178, 7, 568, 28, 15, 125, 280, 518, 9, 1179, 487, 12043, 84, 2073, 1621, 1829, 1034, 1020, 4, 539, 8, 448, 91, 202, 466, 70, 262, 4, 638, 125, 280, 83, 299, 12043, 539, 8, 61, 45, 7, 1537, 176, 4, 84, 2073, 288, 39, 4, 889, 280, 14, 125, 280, 156, 538, 12043, 190, 889, 280, 71, 109, 124, 93, 292, 889, 46, 1248, 4, 518, 48, 883, 125, 12043, 539, 8, 268, 889, 280, 109, 270, 4, 1586, 845, 7, 669, 199, 5, 3964, 3740, 1084, 4, 255, 440, 616, 154, 72, 71, 109, 12043, 49, 61, 283, 3591, 34, 87, 297, 41, 9, 1993, 2602, 518, 52, 706, 109, 2], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'overflow_to_sample': 0, 'start_pos': 14, 'end_pos': 16}\n", + "处理后的训练集大小： 26198\n" + ] + } + ], + "source": [ + "max_length = 256\n", + "doc_stride = 128\n", + "def _process_train(data):\n", + "\n", + " contexts = [data[i][\"context\"] for i in range(len(data))]\n", + " questions = [data[i][\"question\"] for i in range(len(data))]\n", + "\n", + " tokenized_data_list = tokenizer(\n", + " questions,\n", + " contexts,\n", + " stride=doc_stride,\n", + " max_length=max_length,\n", + " padding=\"max_length\",\n", + " return_dict=False\n", + " )\n", + "\n", + " for i, tokenized_data in enumerate(tokenized_data_list):\n", + " # 获取 [CLS] 对应的位置\n", + " input_ids = tokenized_data[\"input_ids\"]\n", + " cls_index = input_ids.index(tokenizer.cls_token_id)\n", + "\n", + " # 在 tokenize 的过程中，汉字和 token 在位置上并非一一对应的\n", + " # 而 offset mapping 记录了每个 token 在原文中对应的起始位置\n", + " offsets = tokenized_data[\"offset_mapping\"]\n", + " # token_type_ids 记录了一条数据中哪些是问题，哪些是上下文\n", + " token_type_ids = tokenized_data[\"token_type_ids\"]\n", + "\n", + " # 一条数据可能因为长度过长而在 tokenized_data 中存在多个结果\n", + " # overflow_to_sample 表示了当前 tokenize_example 属于 data 中的哪一条数据\n", + " sample_index = tokenized_data[\"overflow_to_sample\"]\n", + " answers = data[sample_index][\"answers\"]\n", + "\n", + " # answers 和 answer_starts 均为长度为 1 的 list\n", + " # 我们可以计算出答案的结束位置\n", + " start_char = answers[\"answer_start\"][0]\n", + " end_char = start_char + len(answers[\"text\"][0])\n", + "\n", + " token_start_index = 0\n", + " while token_type_ids[token_start_index] != 1:\n", + " token_start_index += 1\n", + "\n", + " token_end_index = len(input_ids) - 1\n", + " while token_type_ids[token_end_index] != 1:\n", + " token_end_index -= 1\n", + " # 分词后一条数据的结尾一定是 [SEP]，因此还需要减一\n", + " token_end_index -= 1\n", + "\n", + " if not (offsets[token_start_index][0] <= start_char and\n", + " offsets[token_end_index][1] >= end_char):\n", + " # 如果答案不在这条数据中，则将答案位置标记为 [CLS] 的位置\n", + " tokenized_data_list[i][\"start_pos\"] = cls_index\n", + " tokenized_data_list[i][\"end_pos\"] = cls_index\n", + " else:\n", + " # 否则，我们可以找到答案对应的 token 的起始位置，记录在 start_pos 和 end_pos 中\n", + " while token_start_index < len(offsets) and offsets[\n", + " token_start_index][0] <= start_char:\n", + " token_start_index += 1\n", + " tokenized_data_list[i][\"start_pos\"] = token_start_index - 1\n", + " while offsets[token_end_index][1] >= end_char:\n", + " token_end_index -= 1\n", + " tokenized_data_list[i][\"end_pos\"] = token_end_index + 1\n", + "\n", + " return tokenized_data_list\n", + "\n", + "train_dataset.map(_process_train, batched=True, num_workers=5)\n", + "print(train_dataset[0])\n", + "print(\"处理后的训练集大小：\", len(train_dataset))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### 2.2 处理验证集\n", + "\n", + "对于验证集的处理则简单得多，我们只需要保存原数据的 `id` 并将 `offset_mapping` 中不属于上下文的部分设置为 `None` 即可。" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "from fastNLP.core import PaddleDataLoader\n", + "\n", + "train_dataloader = PaddleDataLoader(train_dataset, batch_size=32, shuffle=True)\n", + "val_dataloader = PaddleDataLoader(val_dataset, batch_size=16)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 3. 模型训练：自己定义评测用的 Metric 实现更加自由的任务评测\n", + "\n", + "#### 3.1 损失函数\n", + "\n", + "对于阅读理解任务，我们使用的是 `ErnieForQuestionAnswering` 模型。该模型在接受输入后会返回两个值：`start_logits` 和 `end_logits` ，大小均为 `(batch_size, sequence_length)`，反映了每条数据每个词语为答案起始位置的可能性，因此我们需要自定义一个损失函数来计算 `loss`。 `CrossEntropyLossForSquad` 会分别对答案起始位置的预测值和真实值计算交叉熵，最后返回其平均值作为最终的损失。" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [], + "source": [ + "class CrossEntropyLossForSquad(paddle.nn.Layer):\n", + " def __init__(self):\n", + " super(CrossEntropyLossForSquad, self).__init__()\n", + "\n", + " def forward(self, start_logits, end_logits, start_pos, end_pos):\n", + " start_pos = paddle.unsqueeze(start_pos, axis=-1)\n", + " end_pos = paddle.unsqueeze(end_pos, axis=-1)\n", + " start_loss = paddle.nn.functional.softmax_with_cross_entropy(\n", + " logits=start_logits, label=start_pos)\n", + " start_loss = paddle.mean(start_loss)\n", + " end_loss = paddle.nn.functional.softmax_with_cross_entropy(\n", + " logits=end_logits, label=end_pos)\n", + " end_loss = paddle.mean(end_loss)\n", + "\n", + " loss = (start_loss + end_loss) / 2\n", + " return loss" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### 3.2 定义模型\n", + "\n", + "模型的核心则是 `ErnieForQuestionAnswering` 的 `ernie-1.0-base-zh` 预训练模型，同时按照 `fastNLP` 的规定定义 `train_step` 和 `evaluate_step` 函数。这里 `evaluate_step` 函数并没有像文本分类那样直接返回该批次数据的评测结果，这一点我们将在下面为您讲解。" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m[2022-06-27 19:00:15,825] [ INFO]\u001b[0m - Already cached /remote-home/shxing/.paddlenlp/models/ernie-1.0-base-zh/ernie_v1_chn_base.pdparams\u001b[0m\n", + "W0627 19:00:15.831080 21543 gpu_context.cc:278] Please NOTE: device: 0, GPU Compute Capability: 7.5, Driver API Version: 11.2, Runtime API Version: 11.2\n", + "W0627 19:00:15.843276 21543 gpu_context.cc:306] device: 0, cuDNN Version: 8.1.\n" + ] + } + ], + "source": [ + "from paddlenlp.transformers import ErnieForQuestionAnswering\n", + "\n", + "class QAModel(paddle.nn.Layer):\n", + " def __init__(self, model_checkpoint):\n", + " super(QAModel, self).__init__()\n", + " self.model = ErnieForQuestionAnswering.from_pretrained(model_checkpoint)\n", + " self.loss_func = CrossEntropyLossForSquad()\n", + "\n", + " def forward(self, input_ids, token_type_ids):\n", + " start_logits, end_logits = self.model(input_ids, token_type_ids)\n", + " return start_logits, end_logits\n", + "\n", + " def train_step(self, input_ids, token_type_ids, start_pos, end_pos):\n", + " start_logits, end_logits = self(input_ids, token_type_ids)\n", + " loss = self.loss_func(start_logits, end_logits, start_pos, end_pos)\n", + " return {\"loss\": loss}\n", + "\n", + " def evaluate_step(self, input_ids, token_type_ids):\n", + " start_logits, end_logits = self(input_ids, token_type_ids)\n", + " return {\"start_logits\": start_logits, \"end_logits\": end_logits}\n", + "\n", + "model = QAModel(MODEL_NAME)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### 3.3 自定义 Metric 进行数据的评估\n", + "\n", + "`paddlenlp` 为我们提供了评测 `SQuAD` 格式数据集的函数 `compute_prediction` 和 `squad_evaluate`：\n", + "- `compute_prediction` 函数要求传入原数据 `examples` 、处理后的数据 `features` 和 `features` 对应的结果 `predictions`（一个包含所有数据 `start_logits` 和 `end_logits` 的元组）\n", + "- `squad_evaluate` 要求传入原数据 `examples` 和预测结果 `all_predictions`（通常来自于 `compute_prediction`）\n", + "\n", + "在使用这两个函数的时候，我们需要向其中传入数据集，但显然根据 `fastNLP` 的设计，我们无法在 `evaluate_step` 里实现这一过程，并且 `fastNLP` 也并没有提供计算 `F1` 和 `EM` 的 `Metric`，故我们需要自己定义用于评测的 `Metric`。\n", + "\n", + "在初始化之外，一个 `Metric` 还需要实现三个函数：\n", + "\n", + "1. `reset` - 该函数会在验证数据集的迭代之前被调用，用于清空数据；在我们自定义的 `Metric` 中，我们需要将 `all_start_logits` 和 `all_end_logits` 清空，重新收集每个 `batch` 的结果。\n", + "2. `update` - 该函数会在在每个 `batch` 得到结果后被调用，用于更新 `Metric` 的状态；它的参数即为 `evaluate_step` 返回的内容。我们在这里将得到的 `start_logits` 和 `end_logits` 收集起来。\n", + "3. `get_metric` - 该函数会在数据集被迭代完毕后调用，用于计算评测的结果。现在我们有了整个验证集的 `all_start_logits` 和 `all_end_logits` ，将他们传入 `compute_predictions` 函数得到预测的结果，并继续使用 `squad_evaluate` 函数得到评测的结果。\n", + " - 注：`suqad_evaluate` 函数会自己输出评测结果，为了不让其干扰 `fastNLP` 输出，这里我们使用 `contextlib.redirect_stdout(None)` 将函数的标准输出屏蔽掉。\n", + "\n", + "综上，`SquadEvaluateMetric` 实现的评估过程是：将验证集中所有数据的 `logits` 收集起来，然后统一传入 `compute_prediction` 和 `squad_evaluate` 中进行评估。值得一提的是，`paddlenlp.datasets.load_dataset` 返回的结果是一个 `MapDataset` 类型，其 `data` 成员为加载时的数据，`new_data` 为经过 `map` 函数处理后更新的数据，因此可以分别作为 `examples` 和 `features` 传入。" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": {}, + "outputs": [], + "source": [ + "from fastNLP.core import Metric\n", + "from paddlenlp.metrics.squad import squad_evaluate, compute_prediction\n", + "import contextlib\n", + "\n", + "class SquadEvaluateMetric(Metric):\n", + " def __init__(self, examples, features, testing=False):\n", + " super(SquadEvaluateMetric, self).__init__(\"paddle\", False)\n", + " self.examples = examples\n", + " self.features = features\n", + " self.all_start_logits = []\n", + " self.all_end_logits = []\n", + " self.testing = testing\n", + "\n", + " def reset(self):\n", + " self.all_start_logits = []\n", + " self.all_end_logits = []\n", + "\n", + " def update(self, start_logits, end_logits):\n", + " for start, end in zip(start_logits, end_logits):\n", + " self.all_start_logits.append(start.numpy())\n", + " self.all_end_logits.append(end.numpy())\n", + "\n", + " def get_metric(self):\n", + " all_predictions, _, _ = compute_prediction(\n", + " self.examples, self.features[:len(self.all_start_logits)],\n", + " (self.all_start_logits, self.all_end_logits),\n", + " False, 20, 30\n", + " )\n", + " with contextlib.redirect_stdout(None):\n", + " result = squad_evaluate(\n", + " examples=self.examples,\n", + " preds=all_predictions,\n", + " is_whitespace_splited=False\n", + " )\n", + "\n", + " if self.testing:\n", + " self.print_predictions(all_predictions)\n", + " return result\n", + "\n", + " def print_predictions(self, preds):\n", + " for i, data in enumerate(self.examples):\n", + " if i >= 5:\n", + " break\n", + " print()\n", + " print(\"原文：\", data[\"context\"])\n", + " print(\"问题：\", data[\"question\"], \\\n", + " \"答案：\", preds[data[\"id\"]], \\\n", + " \"正确答案：\", data[\"answers\"][\"text\"])\n", + "\n", + "metric = SquadEvaluateMetric(\n", + " val_dataloader.dataset.data,\n", + " val_dataloader.dataset.new_data,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### 3.4 训练\n", + "\n", + "至此所有的准备工作已经完成，可以使用 `Trainer` 进行训练了。学习率我们依旧采用线性预热策略 `LinearDecayWithWarmup`，优化器为 `AdamW`；回调模块我们选择 `LRSchedCallback` 更新学习率和 `LoadBestModelCallback` 监视评测结果的 `f1` 分数。初始化好 `Trainer` 之后，就将训练的过程交给 `fastNLP` 吧。" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
[19:04:54] INFO Running evaluator sanity check for 2 batches. trainer.py:631\n", + "\n" + ], + "text/plain": [ + "\u001b[2;36m[19:04:54]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m Running evaluator sanity check for \u001b[1;36m2\u001b[0m batches. \u001b]8;id=367046;file://../fastNLP/core/controllers/trainer.py\u001b\\\u001b[2mtrainer.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=96810;file://../fastNLP/core/controllers/trainer.py#631\u001b\\\u001b[2m631\u001b[0m\u001b]8;;\u001b\\\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "\n" + ], + "text/plain": [] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
---------------------------- Eval. results on Epoch:0, Batch:100 ----------------------------\n", + "\n" + ], + "text/plain": [ + "---------------------------- Eval. results on Epoch:\u001b[1;36m0\u001b[0m, Batch:\u001b[1;36m100\u001b[0m ----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"exact#squad\": 49.25899788285109,\n", + " \"f1#squad\": 66.55559127349602,\n", + " \"total#squad\": 1417,\n", + " \"HasAns_exact#squad\": 49.25899788285109,\n", + " \"HasAns_f1#squad\": 66.55559127349602,\n", + " \"HasAns_total#squad\": 1417\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"exact#squad\"\u001b[0m: \u001b[1;36m49.25899788285109\u001b[0m,\n", + " \u001b[1;34m\"f1#squad\"\u001b[0m: \u001b[1;36m66.55559127349602\u001b[0m,\n", + " \u001b[1;34m\"total#squad\"\u001b[0m: \u001b[1;36m1417\u001b[0m,\n", + " \u001b[1;34m\"HasAns_exact#squad\"\u001b[0m: \u001b[1;36m49.25899788285109\u001b[0m,\n", + " \u001b[1;34m\"HasAns_f1#squad\"\u001b[0m: \u001b[1;36m66.55559127349602\u001b[0m,\n", + " \u001b[1;34m\"HasAns_total#squad\"\u001b[0m: \u001b[1;36m1417\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
---------------------------- Eval. results on Epoch:0, Batch:200 ----------------------------\n", + "\n" + ], + "text/plain": [ + "---------------------------- Eval. results on Epoch:\u001b[1;36m0\u001b[0m, Batch:\u001b[1;36m200\u001b[0m ----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"exact#squad\": 57.37473535638673,\n", + " \"f1#squad\": 70.93036525200617,\n", + " \"total#squad\": 1417,\n", + " \"HasAns_exact#squad\": 57.37473535638673,\n", + " \"HasAns_f1#squad\": 70.93036525200617,\n", + " \"HasAns_total#squad\": 1417\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"exact#squad\"\u001b[0m: \u001b[1;36m57.37473535638673\u001b[0m,\n", + " \u001b[1;34m\"f1#squad\"\u001b[0m: \u001b[1;36m70.93036525200617\u001b[0m,\n", + " \u001b[1;34m\"total#squad\"\u001b[0m: \u001b[1;36m1417\u001b[0m,\n", + " \u001b[1;34m\"HasAns_exact#squad\"\u001b[0m: \u001b[1;36m57.37473535638673\u001b[0m,\n", + " \u001b[1;34m\"HasAns_f1#squad\"\u001b[0m: \u001b[1;36m70.93036525200617\u001b[0m,\n", + " \u001b[1;34m\"HasAns_total#squad\"\u001b[0m: \u001b[1;36m1417\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
---------------------------- Eval. results on Epoch:0, Batch:300 ----------------------------\n", + "\n" + ], + "text/plain": [ + "---------------------------- Eval. results on Epoch:\u001b[1;36m0\u001b[0m, Batch:\u001b[1;36m300\u001b[0m ----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"exact#squad\": 63.86732533521524,\n", + " \"f1#squad\": 78.62546663568186,\n", + " \"total#squad\": 1417,\n", + " \"HasAns_exact#squad\": 63.86732533521524,\n", + " \"HasAns_f1#squad\": 78.62546663568186,\n", + " \"HasAns_total#squad\": 1417\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"exact#squad\"\u001b[0m: \u001b[1;36m63.86732533521524\u001b[0m,\n", + " \u001b[1;34m\"f1#squad\"\u001b[0m: \u001b[1;36m78.62546663568186\u001b[0m,\n", + " \u001b[1;34m\"total#squad\"\u001b[0m: \u001b[1;36m1417\u001b[0m,\n", + " \u001b[1;34m\"HasAns_exact#squad\"\u001b[0m: \u001b[1;36m63.86732533521524\u001b[0m,\n", + " \u001b[1;34m\"HasAns_f1#squad\"\u001b[0m: \u001b[1;36m78.62546663568186\u001b[0m,\n", + " \u001b[1;34m\"HasAns_total#squad\"\u001b[0m: \u001b[1;36m1417\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
---------------------------- Eval. results on Epoch:0, Batch:400 ----------------------------\n", + "\n" + ], + "text/plain": [ + "---------------------------- Eval. results on Epoch:\u001b[1;36m0\u001b[0m, Batch:\u001b[1;36m400\u001b[0m ----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"exact#squad\": 64.92589978828511,\n", + " \"f1#squad\": 79.36746074079691,\n", + " \"total#squad\": 1417,\n", + " \"HasAns_exact#squad\": 64.92589978828511,\n", + " \"HasAns_f1#squad\": 79.36746074079691,\n", + " \"HasAns_total#squad\": 1417\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"exact#squad\"\u001b[0m: \u001b[1;36m64.92589978828511\u001b[0m,\n", + " \u001b[1;34m\"f1#squad\"\u001b[0m: \u001b[1;36m79.36746074079691\u001b[0m,\n", + " \u001b[1;34m\"total#squad\"\u001b[0m: \u001b[1;36m1417\u001b[0m,\n", + " \u001b[1;34m\"HasAns_exact#squad\"\u001b[0m: \u001b[1;36m64.92589978828511\u001b[0m,\n", + " \u001b[1;34m\"HasAns_f1#squad\"\u001b[0m: \u001b[1;36m79.36746074079691\u001b[0m,\n", + " \u001b[1;34m\"HasAns_total#squad\"\u001b[0m: \u001b[1;36m1417\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
---------------------------- Eval. results on Epoch:0, Batch:500 ----------------------------\n", + "\n" + ], + "text/plain": [ + "---------------------------- Eval. results on Epoch:\u001b[1;36m0\u001b[0m, Batch:\u001b[1;36m500\u001b[0m ----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"exact#squad\": 65.70218772053634,\n", + " \"f1#squad\": 80.33295482054824,\n", + " \"total#squad\": 1417,\n", + " \"HasAns_exact#squad\": 65.70218772053634,\n", + " \"HasAns_f1#squad\": 80.33295482054824,\n", + " \"HasAns_total#squad\": 1417\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"exact#squad\"\u001b[0m: \u001b[1;36m65.70218772053634\u001b[0m,\n", + " \u001b[1;34m\"f1#squad\"\u001b[0m: \u001b[1;36m80.33295482054824\u001b[0m,\n", + " \u001b[1;34m\"total#squad\"\u001b[0m: \u001b[1;36m1417\u001b[0m,\n", + " \u001b[1;34m\"HasAns_exact#squad\"\u001b[0m: \u001b[1;36m65.70218772053634\u001b[0m,\n", + " \u001b[1;34m\"HasAns_f1#squad\"\u001b[0m: \u001b[1;36m80.33295482054824\u001b[0m,\n", + " \u001b[1;34m\"HasAns_total#squad\"\u001b[0m: \u001b[1;36m1417\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
---------------------------- Eval. results on Epoch:0, Batch:600 ----------------------------\n", + "\n" + ], + "text/plain": [ + "---------------------------- Eval. results on Epoch:\u001b[1;36m0\u001b[0m, Batch:\u001b[1;36m600\u001b[0m ----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"exact#squad\": 65.41990119971771,\n", + " \"f1#squad\": 79.7483487059053,\n", + " \"total#squad\": 1417,\n", + " \"HasAns_exact#squad\": 65.41990119971771,\n", + " \"HasAns_f1#squad\": 79.7483487059053,\n", + " \"HasAns_total#squad\": 1417\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"exact#squad\"\u001b[0m: \u001b[1;36m65.41990119971771\u001b[0m,\n", + " \u001b[1;34m\"f1#squad\"\u001b[0m: \u001b[1;36m79.7483487059053\u001b[0m,\n", + " \u001b[1;34m\"total#squad\"\u001b[0m: \u001b[1;36m1417\u001b[0m,\n", + " \u001b[1;34m\"HasAns_exact#squad\"\u001b[0m: \u001b[1;36m65.41990119971771\u001b[0m,\n", + " \u001b[1;34m\"HasAns_f1#squad\"\u001b[0m: \u001b[1;36m79.7483487059053\u001b[0m,\n", + " \u001b[1;34m\"HasAns_total#squad\"\u001b[0m: \u001b[1;36m1417\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
---------------------------- Eval. results on Epoch:0, Batch:700 ----------------------------\n", + "\n" + ], + "text/plain": [ + "---------------------------- Eval. results on Epoch:\u001b[1;36m0\u001b[0m, Batch:\u001b[1;36m700\u001b[0m ----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"exact#squad\": 66.61961891319689,\n", + " \"f1#squad\": 80.32432238994133,\n", + " \"total#squad\": 1417,\n", + " \"HasAns_exact#squad\": 66.61961891319689,\n", + " \"HasAns_f1#squad\": 80.32432238994133,\n", + " \"HasAns_total#squad\": 1417\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"exact#squad\"\u001b[0m: \u001b[1;36m66.61961891319689\u001b[0m,\n", + " \u001b[1;34m\"f1#squad\"\u001b[0m: \u001b[1;36m80.32432238994133\u001b[0m,\n", + " \u001b[1;34m\"total#squad\"\u001b[0m: \u001b[1;36m1417\u001b[0m,\n", + " \u001b[1;34m\"HasAns_exact#squad\"\u001b[0m: \u001b[1;36m66.61961891319689\u001b[0m,\n", + " \u001b[1;34m\"HasAns_f1#squad\"\u001b[0m: \u001b[1;36m80.32432238994133\u001b[0m,\n", + " \u001b[1;34m\"HasAns_total#squad\"\u001b[0m: \u001b[1;36m1417\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
---------------------------- Eval. results on Epoch:0, Batch:800 ----------------------------\n", + "\n" + ], + "text/plain": [ + "---------------------------- Eval. results on Epoch:\u001b[1;36m0\u001b[0m, Batch:\u001b[1;36m800\u001b[0m ----------------------------\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " \"exact#squad\": 65.84333098094567,\n", + " \"f1#squad\": 79.23169801265415,\n", + " \"total#squad\": 1417,\n", + " \"HasAns_exact#squad\": 65.84333098094567,\n", + " \"HasAns_f1#squad\": 79.23169801265415,\n", + " \"HasAns_total#squad\": 1417\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[1;34m\"exact#squad\"\u001b[0m: \u001b[1;36m65.84333098094567\u001b[0m,\n", + " \u001b[1;34m\"f1#squad\"\u001b[0m: \u001b[1;36m79.23169801265415\u001b[0m,\n", + " \u001b[1;34m\"total#squad\"\u001b[0m: \u001b[1;36m1417\u001b[0m,\n", + " \u001b[1;34m\"HasAns_exact#squad\"\u001b[0m: \u001b[1;36m65.84333098094567\u001b[0m,\n", + " \u001b[1;34m\"HasAns_f1#squad\"\u001b[0m: \u001b[1;36m79.23169801265415\u001b[0m,\n", + " \u001b[1;34m\"HasAns_total#squad\"\u001b[0m: \u001b[1;36m1417\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "\n" + ], + "text/plain": [] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
[19:20:28] INFO Loading best model from fnlp-ernie-squad/ load_best_model_callback.py:111\n", + " 2022-06-27-19_00_15_388554/best_so_far \n", + " with f1#squad: 80.33295482054824... \n", + "\n" + ], + "text/plain": [ + "\u001b[2;36m[19:20:28]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m Loading best model from fnlp-ernie-squad/ \u001b]8;id=163935;file://../fastNLP/core/callbacks/load_best_model_callback.py\u001b\\\u001b[2mload_best_model_callback.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=31503;file://../fastNLP/core/callbacks/load_best_model_callback.py#111\u001b\\\u001b[2m111\u001b[0m\u001b]8;;\u001b\\\n", + "\u001b[2;36m \u001b[0m \u001b[1;36m2022\u001b[0m-\u001b[1;36m06\u001b[0m-\u001b[1;36m27\u001b[0m-19_00_15_388554/best_so_far \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m with f1#squad: \u001b[1;36m80.33295482054824\u001b[0m\u001b[33m...\u001b[0m \u001b[2m \u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
INFO Deleting fnlp-ernie-squad/2022-06-27-19_0 load_best_model_callback.py:131\n", + " 0_15_388554/best_so_far... \n", + "\n" + ], + "text/plain": [ + "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m Deleting fnlp-ernie-squad/\u001b[1;36m2022\u001b[0m-\u001b[1;36m06\u001b[0m-\u001b[1;36m27\u001b[0m-19_0 \u001b]8;id=560859;file://../fastNLP/core/callbacks/load_best_model_callback.py\u001b\\\u001b[2mload_best_model_callback.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=573263;file://../fastNLP/core/callbacks/load_best_model_callback.py#131\u001b\\\u001b[2m131\u001b[0m\u001b]8;;\u001b\\\n", + "\u001b[2;36m \u001b[0m 0_15_388554/best_so_far\u001b[33m...\u001b[0m \u001b[2m \u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "from fastNLP import Trainer, LRSchedCallback, LoadBestModelCallback\n", + "from paddlenlp.transformers import LinearDecayWithWarmup\n", + "\n", + "n_epochs = 1\n", + "num_training_steps = len(train_dataloader) * n_epochs\n", + "lr_scheduler = LinearDecayWithWarmup(3e-5, num_training_steps, 0.1)\n", + "optimizer = paddle.optimizer.AdamW(\n", + " learning_rate=lr_scheduler,\n", + " parameters=model.parameters(),\n", + ")\n", + "callbacks=[\n", + " LRSchedCallback(lr_scheduler, step_on=\"batch\"),\n", + " LoadBestModelCallback(\"f1#squad\", larger_better=True, save_folder=\"fnlp-ernie-squad\")\n", + "]\n", + "trainer = Trainer(\n", + " model=model,\n", + " train_dataloader=train_dataloader,\n", + " evaluate_dataloaders=val_dataloader,\n", + " device=1,\n", + " optimizers=optimizer,\n", + " n_epochs=n_epochs,\n", + " callbacks=callbacks,\n", + " evaluate_every=100,\n", + " metrics={\"squad\": metric},\n", + ")\n", + "trainer.run()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### 3.5 测试\n", + "\n", + "最后，我们可以使用 `Evaluator` 查看我们训练的结果。我们在之前为 `SquadEvaluateMetric` 设置了 `testing` 参数来在测试阶段进行输出，可以看到，训练的结果还是比较不错的。" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
原文： 爬行垫根据中间材料的不同可以分为:XPE爬行垫、EPE爬行垫、EVA爬行垫、PVC爬行垫;其中XPE爬\n", + "行垫、EPE爬行垫都属于PE材料加保鲜膜复合而成,都是无异味的环保材料,但是XPE爬行垫是品质较好的爬\n", + "行垫,韩国进口爬行垫都是这种爬行垫,而EPE爬行垫是国内厂家为了减低成本,使用EPE(珍珠棉)作为原料生\n", + "产的一款爬行垫,该材料弹性差,易碎,开孔发泡防水性弱。EVA爬行垫、PVC爬行垫是用EVA或PVC作为原材料\n", + "与保鲜膜复合的而成的爬行垫,或者把图案转印在原材料上,这两款爬行垫通常有异味,如果是图案转印的爬\n", + "行垫,油墨外露容易脱落。 \n", + "当时我儿子爬的时候,我们也买了垫子,但是始终有味。最后就没用了,铺的就的薄毯子让他爬。\n", + "\n" + ], + "text/plain": [ + "原文： 爬行垫根据中间材料的不同可以分为:XPE爬行垫、EPE爬行垫、EVA爬行垫、PVC爬行垫;其中XPE爬\n", + "行垫、EPE爬行垫都属于PE材料加保鲜膜复合而成,都是无异味的环保材料,但是XPE爬行垫是品质较好的爬\n", + "行垫,韩国进口爬行垫都是这种爬行垫,而EPE爬行垫是国内厂家为了减低成本,使用EPE(珍珠棉)作为原料生\n", + "产的一款爬行垫,该材料弹性差,易碎,开孔发泡防水性弱。EVA爬行垫、PVC爬行垫是用EVA或PVC作为原材料\n", + "与保鲜膜复合的而成的爬行垫,或者把图案转印在原材料上,这两款爬行垫通常有异味,如果是图案转印的爬\n", + "行垫,油墨外露容易脱落。 \n", + "当时我儿子爬的时候,我们也买了垫子,但是始终有味。最后就没用了,铺的就的薄毯子让他爬。\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
问题： 爬行垫什么材质的好 答案： EPE(珍珠棉 正确答案： ['XPE']\n", + "\n" + ], + "text/plain": [ + "问题： 爬行垫什么材质的好 答案： EPE(珍珠棉 正确答案： ['XPE']\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
原文： 真实情况是160-162。她平时谎报的168是因为不离脚穿高水台恨天高(15厘米) 图1她穿着高水台恨\n", + "天高和刘亦菲一样高,(刘亦菲对外报身高172)范冰冰礼服下厚厚的高水台暴露了她的心机,对比一下两者的\n", + "鞋子吧 图2 穿着高水台恨天高才和刘德华谢霆锋持平,如果她真的有168,那么加上鞋高,刘和谢都要有180?\n", + "明显是不可能的。所以刘德华对外报的身高174减去10-15厘米才是范冰冰的真实身高 图3,范冰冰有一次脱\n", + "鞋上场,这个最说明问题了,看看她的身体比例吧。还有目测一下她手上鞋子的鞋跟有多高多厚吧,至少超过\n", + "10厘米。\n", + "\n" + ], + "text/plain": [ + "原文： 真实情况是160-162。她平时谎报的168是因为不离脚穿高水台恨天高(15厘米) 图1她穿着高水台恨\n", + "天高和刘亦菲一样高,(刘亦菲对外报身高172)范冰冰礼服下厚厚的高水台暴露了她的心机,对比一下两者的\n", + "鞋子吧 图2 穿着高水台恨天高才和刘德华谢霆锋持平,如果她真的有168,那么加上鞋高,刘和谢都要有180?\n", + "明显是不可能的。所以刘德华对外报的身高174减去10-15厘米才是范冰冰的真实身高 图3,范冰冰有一次脱\n", + "鞋上场,这个最说明问题了,看看她的身体比例吧。还有目测一下她手上鞋子的鞋跟有多高多厚吧,至少超过\n", + "10厘米。\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
问题： 范冰冰多高真实身高 答案： 160-162 正确答案： ['160-162']\n", + "\n" + ], + "text/plain": [ + "问题： 范冰冰多高真实身高 答案： 160-162 正确答案： ['160-162']\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
原文： 防水作为目前高端手机的标配,特别是苹果也支持防水之后,国产大多数高端旗舰手机都已经支持防\n", + "水。虽然我们真的不会故意把手机放入水中,但是有了防水之后,用户心里会多一重安全感。那么近日最为\n", + "火热的小米6防水吗?小米6的防水级别又是多少呢? 小编查询了很多资料发现,小米6确实是防水的,但是为\n", + "了保持低调,同时为了不被别人说防水等级不够,很多资料都没有标注小米是否防水。根据评测资料显示,小\n", + "米6是支持IP68级的防水,是绝对能够满足日常生活中的防水需求的。\n", + "\n" + ], + "text/plain": [ + "原文： 防水作为目前高端手机的标配,特别是苹果也支持防水之后,国产大多数高端旗舰手机都已经支持防\n", + "水。虽然我们真的不会故意把手机放入水中,但是有了防水之后,用户心里会多一重安全感。那么近日最为\n", + "火热的小米6防水吗?小米6的防水级别又是多少呢? 小编查询了很多资料发现,小米6确实是防水的,但是为\n", + "了保持低调,同时为了不被别人说防水等级不够,很多资料都没有标注小米是否防水。根据评测资料显示,小\n", + "米6是支持IP68级的防水,是绝对能够满足日常生活中的防水需求的。\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
问题： 小米6防水等级 答案： IP68级 正确答案： ['IP68级']\n", + "\n" + ], + "text/plain": [ + "问题： 小米6防水等级 答案： IP68级 正确答案： ['IP68级']\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
原文： 这位朋友你好,女性出现妊娠反应一般是从6-12周左右,也就是女性怀孕1个多月就会开始出现反应,\n", + "第3个月的时候,妊辰反应基本结束。 而大部分女性怀孕初期都会出现恶心、呕吐的感觉,这些症状都是因\n", + "人而异的,除非恶心、呕吐的非常厉害,才需要就医,否则这些都是刚怀孕的的正常症状。1-3个月的时候可\n", + "以观察一下自己的皮肤,一般女性怀孕初期可能会产生皮肤色素沉淀或是腹壁产生妊娠纹,特别是在怀孕的\n", + "后期更加明显。 还有很多女性怀孕初期会出现疲倦、嗜睡的情况。怀孕三个月的时候,膀胱会受到日益胀\n", + "大的子宫的压迫,容量会变小,所以怀孕期间也会有尿频的现象出现。月经停止也是刚怀孕最容易出现的症\n", + "状,只要是平时月经正常的女性,在性行为后超过正常经期两周,就有可能是怀孕了。 如果你想判断自己是\n", + "否怀孕,可以看看自己有没有这些反应。当然这也只是多数人的怀孕表现,也有部分女性怀孕表现并不完全\n", + "是这样,如果你无法确定自己是否怀孕,最好去医院检查一下。\n", + "\n" + ], + "text/plain": [ + "原文： 这位朋友你好,女性出现妊娠反应一般是从6-12周左右,也就是女性怀孕1个多月就会开始出现反应,\n", + "第3个月的时候,妊辰反应基本结束。 而大部分女性怀孕初期都会出现恶心、呕吐的感觉,这些症状都是因\n", + "人而异的,除非恶心、呕吐的非常厉害,才需要就医,否则这些都是刚怀孕的的正常症状。1-3个月的时候可\n", + "以观察一下自己的皮肤,一般女性怀孕初期可能会产生皮肤色素沉淀或是腹壁产生妊娠纹,特别是在怀孕的\n", + "后期更加明显。 还有很多女性怀孕初期会出现疲倦、嗜睡的情况。怀孕三个月的时候,膀胱会受到日益胀\n", + "大的子宫的压迫,容量会变小,所以怀孕期间也会有尿频的现象出现。月经停止也是刚怀孕最容易出现的症\n", + "状,只要是平时月经正常的女性,在性行为后超过正常经期两周,就有可能是怀孕了。 如果你想判断自己是\n", + "否怀孕,可以看看自己有没有这些反应。当然这也只是多数人的怀孕表现,也有部分女性怀孕表现并不完全\n", + "是这样,如果你无法确定自己是否怀孕,最好去医院检查一下。\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
问题： 怀孕多久会有反应 答案： 6-12周左右 正确答案： ['6-12周左右', '6-12周', '1个多月']\n", + "\n" + ], + "text/plain": [ + "问题： 怀孕多久会有反应 答案： 6-12周左右 正确答案： ['6-12周左右', '6-12周', '1个多月']\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
原文： 【东奥会计在线——中级会计职称频道推荐】根据《关于提高科技型中小企业研究开发费用税前加计\n", + "扣除比例的通知》的规定,研发费加计扣除比例提高到75%。|财政部、国家税务总局、科技部发布《关于提\n", + "高科技型中小企业研究开发费用税前加计扣除比例的通知》。|通知称,为进一步激励中小企业加大研发投\n", + "入,支持科技创新,就提高科技型中小企业研究开发费用(以下简称研发费用)税前加计扣除比例有关问题发\n", + "布通知。|通知明确,科技型中小企业开展研发活动中实际发生的研发费用,未形成无形资产计入当期损益的\n", + ",在按规定据实扣除的基础上,在2017年1月1日至2019年12月31日期间,再按照实际发生额的75%在税前加计\n", + "扣除;形成无形资产的,在上述期间按照无形资产成本的175%在税前摊销。|科技型中小企业享受研发费用税\n", + "前加计扣除政策的其他政策口径按照《财政部国家税务总局科技部关于完善研究开发费用税前加计扣除政\n", + "策的通知》(财税〔2015〕119号)规定执行。|科技型中小企业条件和管理办法由科技部、财政部和国家税\n", + "务总局另行发布。科技、财政和税务部门应建立信息共享机制,及时共享科技型中小企业的相关信息,加强\n", + "协调配合,保障优惠政策落实到位。|上一篇文章:关于2016年度企业研究开发费用税前加计扣除政策企业所\n", + "得税纳税申报问题的公告 下一篇文章:关于提高科技型中小企业研究开发费用税前加计扣除比例的通知\n", + "\n" + ], + "text/plain": [ + "原文： 【东奥会计在线——中级会计职称频道推荐】根据《关于提高科技型中小企业研究开发费用税前加计\n", + "扣除比例的通知》的规定,研发费加计扣除比例提高到75%。|财政部、国家税务总局、科技部发布《关于提\n", + "高科技型中小企业研究开发费用税前加计扣除比例的通知》。|通知称,为进一步激励中小企业加大研发投\n", + "入,支持科技创新,就提高科技型中小企业研究开发费用(以下简称研发费用)税前加计扣除比例有关问题发\n", + "布通知。|通知明确,科技型中小企业开展研发活动中实际发生的研发费用,未形成无形资产计入当期损益的\n", + ",在按规定据实扣除的基础上,在2017年1月1日至2019年12月31日期间,再按照实际发生额的75%在税前加计\n", + "扣除;形成无形资产的,在上述期间按照无形资产成本的175%在税前摊销。|科技型中小企业享受研发费用税\n", + "前加计扣除政策的其他政策口径按照《财政部国家税务总局科技部关于完善研究开发费用税前加计扣除政\n", + "策的通知》(财税〔2015〕119号)规定执行。|科技型中小企业条件和管理办法由科技部、财政部和国家税\n", + "务总局另行发布。科技、财政和税务部门应建立信息共享机制,及时共享科技型中小企业的相关信息,加强\n", + "协调配合,保障优惠政策落实到位。|上一篇文章:关于2016年度企业研究开发费用税前加计扣除政策企业所\n", + "得税纳税申报问题的公告 下一篇文章:关于提高科技型中小企业研究开发费用税前加计扣除比例的通知\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
问题： 研发费用加计扣除比例 答案： 75% 正确答案： ['75%']\n", + "\n" + ], + "text/plain": [ + "问题： 研发费用加计扣除比例 答案： 75% 正确答案： ['75%']\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "\n" + ], + "text/plain": [] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
{\n", + " 'exact#squad': 65.70218772053634,\n", + " 'f1#squad': 80.33295482054824,\n", + " 'total#squad': 1417,\n", + " 'HasAns_exact#squad': 65.70218772053634,\n", + " 'HasAns_f1#squad': 80.33295482054824,\n", + " 'HasAns_total#squad': 1417\n", + "}\n", + "\n" + ], + "text/plain": [ + "\u001b[1m{\u001b[0m\n", + " \u001b[32m'exact#squad'\u001b[0m: \u001b[1;36m65.70218772053634\u001b[0m,\n", + " \u001b[32m'f1#squad'\u001b[0m: \u001b[1;36m80.33295482054824\u001b[0m,\n", + " \u001b[32m'total#squad'\u001b[0m: \u001b[1;36m1417\u001b[0m,\n", + " \u001b[32m'HasAns_exact#squad'\u001b[0m: \u001b[1;36m65.70218772053634\u001b[0m,\n", + " \u001b[32m'HasAns_f1#squad'\u001b[0m: \u001b[1;36m80.33295482054824\u001b[0m,\n", + " \u001b[32m'HasAns_total#squad'\u001b[0m: \u001b[1;36m1417\u001b[0m\n", + "\u001b[1m}\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "from fastNLP import Evaluator\n", + "evaluator = Evaluator(\n", + " model=model,\n", + " dataloaders=val_dataloader,\n", + " device=1,\n", + " metrics={\n", + " \"squad\": SquadEvaluateMetric(\n", + " val_dataloader.dataset.data,\n", + " val_dataloader.dataset.new_data,\n", + " testing=True,\n", + " ),\n", + " },\n", + ")\n", + "result = evaluator.run()" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3.7.13 ('fnlp-paddle')", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.13" + }, + "orig_nbformat": 4, + "vscode": { + "interpreter": { + "hash": "31f2d9d3efc23c441973d7c4273acfea8b132b6a578f002629b6b44b8f65e720" + } + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/docs/source/tutorials/figures/E1-fig-glue-benchmark.png b/docs/source/tutorials/figures/E1-fig-glue-benchmark.png new file mode 100644 index 00000000..515db700 Binary files /dev/null and b/docs/source/tutorials/figures/E1-fig-glue-benchmark.png differ diff --git a/docs/source/tutorials/figures/E2-fig-p-tuning-v2-model.png b/docs/source/tutorials/figures/E2-fig-p-tuning-v2-model.png new file mode 100644 index 00000000..b5a9c1b8 Binary files /dev/null and b/docs/source/tutorials/figures/E2-fig-p-tuning-v2-model.png differ diff --git a/docs/source/tutorials/figures/E2-fig-pet-model.png b/docs/source/tutorials/figures/E2-fig-pet-model.png new file mode 100644 index 00000000..c3c377c0 Binary files /dev/null and b/docs/source/tutorials/figures/E2-fig-pet-model.png differ diff --git a/docs/source/tutorials/figures/T0-fig-parameter-matching.png b/docs/source/tutorials/figures/T0-fig-parameter-matching.png new file mode 100644 index 00000000..24013cc1 Binary files /dev/null and b/docs/source/tutorials/figures/T0-fig-parameter-matching.png differ diff --git a/docs/source/tutorials/figures/T0-fig-trainer-and-evaluator.png b/docs/source/tutorials/figures/T0-fig-trainer-and-evaluator.png new file mode 100644 index 00000000..38222ee8 Binary files /dev/null and b/docs/source/tutorials/figures/T0-fig-trainer-and-evaluator.png differ diff --git a/docs/source/tutorials/figures/T0-fig-training-structure.png b/docs/source/tutorials/figures/T0-fig-training-structure.png new file mode 100644 index 00000000..edc2e2ff Binary files /dev/null and b/docs/source/tutorials/figures/T0-fig-training-structure.png differ diff --git a/docs/source/tutorials/figures/T1-fig-dataset-and-vocabulary.png b/docs/source/tutorials/figures/T1-fig-dataset-and-vocabulary.png new file mode 100644 index 00000000..803cf34a Binary files /dev/null and b/docs/source/tutorials/figures/T1-fig-dataset-and-vocabulary.png differ diff --git a/docs/source/tutorials/figures/paddle-ernie-1.0-masking-levels.png b/docs/source/tutorials/figures/paddle-ernie-1.0-masking-levels.png new file mode 100644 index 00000000..ff2519c4 Binary files /dev/null and b/docs/source/tutorials/figures/paddle-ernie-1.0-masking-levels.png differ diff --git a/docs/source/tutorials/figures/paddle-ernie-1.0-masking.png b/docs/source/tutorials/figures/paddle-ernie-1.0-masking.png new file mode 100644 index 00000000..ed003a2f Binary files /dev/null and b/docs/source/tutorials/figures/paddle-ernie-1.0-masking.png differ diff --git a/docs/source/tutorials/figures/paddle-ernie-2.0-continual-pretrain.png b/docs/source/tutorials/figures/paddle-ernie-2.0-continual-pretrain.png new file mode 100644 index 00000000..d45f65d8 Binary files /dev/null and b/docs/source/tutorials/figures/paddle-ernie-2.0-continual-pretrain.png differ diff --git a/docs/source/tutorials/figures/paddle-ernie-3.0-framework.png b/docs/source/tutorials/figures/paddle-ernie-3.0-framework.png new file mode 100644 index 00000000..f50ddb1c Binary files /dev/null and b/docs/source/tutorials/figures/paddle-ernie-3.0-framework.png differ diff --git a/fastNLP/__init__.py b/fastNLP/__init__.py index 9885a175..31249c80 100644 --- a/fastNLP/__init__.py +++ b/fastNLP/__init__.py @@ -2,4 +2,4 @@ from fastNLP.envs import * from fastNLP.core import * -__version__ = '0.8.0beta' +__version__ = '1.0.0alpha' diff --git a/fastNLP/core/callbacks/callback_event.py b/fastNLP/core/callbacks/callback_event.py index 8a51b6de..f632cf3c 100644 --- a/fastNLP/core/callbacks/callback_event.py +++ b/fastNLP/core/callbacks/callback_event.py @@ -35,14 +35,14 @@ class Event: :param value: Trainer 的 callback 时机； :param every: 每触发多少次才真正运行一次； - :param once: 在第一次运行后时候再次执行； + :param once: 是否仅运行一次； :param filter_fn: 输入参数的应该为 ``(filter, trainer)``，其中 ``filter`` 对象中包含了 `filter.num_called` 和 `filter.num_executed` 两个变量分别获取当前被调用了多少次，真正执行了多少次；``trainer`` 对象即为当前正在运行的 Trainer； """ every: Optional[int] - once: Optional[int] + once: Optional[bool] - def __init__(self, value: str, every: Optional[int] = None, once: Optional[int] = None, + def __init__(self, value: str, every: Optional[int] = None, once: Optional[bool] = None, filter_fn: Optional[Callable] = None): self.every = every self.once = once @@ -68,7 +68,6 @@ class Event: return Event(value='on_after_trainer_initialized', every=every, once=once, filter_fn=filter_fn) @staticmethod - def on_sanity_check_begin(every=None, once=None, filter_fn=None): """ 当 Trainer 运行到 :func:`on_sanity_check_begin` 时触发； @@ -85,7 +84,6 @@ class Event: return Event(value='on_sanity_check_begin', every=every, once=once, filter_fn=filter_fn) @staticmethod - def on_sanity_check_end(every=None, once=None, filter_fn=None): """ 当 Trainer 运行到 :func:`on_sanity_check_end` 时触发； @@ -101,7 +99,6 @@ class Event: return Event(value='on_sanity_check_end', every=every, once=once, filter_fn=filter_fn) @staticmethod - def on_train_begin(every=None, once=None, filter_fn=None): """ 当 Trainer 运行到 :func:`on_train_begin` 时触发； @@ -117,7 +114,6 @@ class Event: return Event(value='on_train_begin', every=every, once=once, filter_fn=filter_fn) @staticmethod - def on_train_end(every=None, once=None, filter_fn=None): """ 当 Trainer 运行到 :func:`on_train_end` 时触发； @@ -133,7 +129,6 @@ class Event: return Event(value='on_train_end', every=every, once=once, filter_fn=filter_fn) @staticmethod - def on_train_epoch_begin(every=None, once=None, filter_fn=None): """ 当 Trainer 运行到 :func:`on_train_epoch_begin` 时触发； @@ -149,7 +144,6 @@ class Event: return Event(value='on_train_epoch_begin', every=every, once=once, filter_fn=filter_fn) @staticmethod - def on_train_epoch_end(every=None, once=None, filter_fn=None): """ 当 Trainer 运行到 :func:`on_train_epoch_end` 时触发； @@ -165,7 +159,6 @@ class Event: return Event(value='on_train_epoch_end', every=every, once=once, filter_fn=filter_fn) @staticmethod - def on_fetch_data_begin(every=None, once=None, filter_fn=None): """ 当 Trainer 运行到 :func:`on_fetch_data_begin` 时触发； @@ -181,7 +174,6 @@ class Event: return Event(value='on_fetch_data_begin', every=every, once=once, filter_fn=filter_fn) @staticmethod - def on_fetch_data_end(every=None, once=None, filter_fn=None): """ 当 Trainer 运行到 :func:`on_fetch_data_end` 时触发； @@ -197,7 +189,6 @@ class Event: return Event(value='on_fetch_data_end', every=every, once=once, filter_fn=filter_fn) @staticmethod - def on_train_batch_begin(every=None, once=None, filter_fn=None): """ 当 Trainer 运行到 :func:`on_train_batch_begin` 时触发； @@ -213,7 +204,6 @@ class Event: return Event(value='on_train_batch_begin', every=every, once=once, filter_fn=filter_fn) @staticmethod - def on_train_batch_end(every=None, once=None, filter_fn=None): """ 当 Trainer 运行到 :func:`on_train_batch_end` 时触发； @@ -229,7 +219,6 @@ class Event: return Event(value='on_train_batch_end', every=every, once=once, filter_fn=filter_fn) @staticmethod - def on_exception(every=None, once=None, filter_fn=None): """ 当 Trainer 运行到 :func:`on_exception` 时触发； @@ -245,7 +234,6 @@ class Event: return Event(value='on_exception', every=every, once=once, filter_fn=filter_fn) @staticmethod - def on_save_model(every=None, once=None, filter_fn=None): """ 当 Trainer 运行到 :func:`on_save_model` 时触发； @@ -261,7 +249,6 @@ class Event: return Event(value='on_save_model', every=every, once=once, filter_fn=filter_fn) @staticmethod - def on_load_model(every=None, once=None, filter_fn=None): """ 当 Trainer 运行到 :func:`on_load_model` 时触发； @@ -277,7 +264,6 @@ class Event: return Event(value='on_load_model', every=every, once=once, filter_fn=filter_fn) @staticmethod - def on_save_checkpoint(every=None, once=None, filter_fn=None): """ 当 Trainer 运行到 :func:`on_save_checkpoint` 时触发； @@ -293,7 +279,6 @@ class Event: return Event(value='on_save_checkpoint', every=every, once=once, filter_fn=filter_fn) @staticmethod - def on_load_checkpoint(every=None, once=None, filter_fn=None): """ 当 Trainer 运行到 :func:`on_load_checkpoint` 时触发； @@ -309,7 +294,6 @@ class Event: return Event(value='on_load_checkpoint', every=every, once=once, filter_fn=filter_fn) @staticmethod - def on_load_checkpoint(every=None, once=None, filter_fn=None): """ 当 Trainer 运行到 :func:`on_load_checkpoint` 时触发； @@ -325,7 +309,6 @@ class Event: return Event(value='on_load_checkpoint', every=every, once=once, filter_fn=filter_fn) @staticmethod - def on_before_backward(every=None, once=None, filter_fn=None): """ 当 Trainer 运行到 :func:`on_before_backward` 时触发； @@ -341,7 +324,6 @@ class Event: return Event(value='on_before_backward', every=every, once=once, filter_fn=filter_fn) @staticmethod - def on_after_backward(every=None, once=None, filter_fn=None): """ 当 Trainer 运行到 :func:`on_after_backward` 时触发； @@ -357,7 +339,6 @@ class Event: return Event(value='on_after_backward', every=every, once=once, filter_fn=filter_fn) @staticmethod - def on_before_optimizers_step(every=None, once=None, filter_fn=None): """ 当 Trainer 运行到 :func:`on_before_optimizers_step` 时触发； @@ -373,7 +354,6 @@ class Event: return Event(value='on_before_optimizers_step', every=every, once=once, filter_fn=filter_fn) @staticmethod - def on_after_optimizers_step(every=None, once=None, filter_fn=None): """ 当 Trainer 运行到 :func:`on_after_optimizers_step` 时触发； @@ -389,7 +369,6 @@ class Event: return Event(value='on_after_optimizers_step', every=every, once=once, filter_fn=filter_fn) @staticmethod - def on_before_zero_grad(every=None, once=None, filter_fn=None): """ 当 Trainer 运行到 :func:`on_before_zero_grad` 时触发； @@ -405,7 +384,6 @@ class Event: return Event(value='on_before_zero_grad', every=every, once=once, filter_fn=filter_fn) @staticmethod - def on_after_zero_grad(every=None, once=None, filter_fn=None): """ 当 Trainer 运行到 :func:`on_after_zero_grad` 时触发； @@ -421,7 +399,6 @@ class Event: return Event(value='on_after_zero_grad', every=every, once=once, filter_fn=filter_fn) @staticmethod - def on_evaluate_begin(every=None, once=None, filter_fn=None): """ 当 Trainer 运行到 :func:`on_evaluate_begin` 时触发； @@ -437,7 +414,6 @@ class Event: return Event(value='on_evaluate_begin', every=every, once=once, filter_fn=filter_fn) @staticmethod - def on_evaluate_end(every=None, once=None, filter_fn=None): """ 当 Trainer 运行到 :func:`on_evaluate_end` 时触发； diff --git a/fastNLP/core/collators/padders/oneflow_padder.py b/fastNLP/core/collators/padders/oneflow_padder.py index 5e235a0f..30d73e26 100644 --- a/fastNLP/core/collators/padders/oneflow_padder.py +++ b/fastNLP/core/collators/padders/oneflow_padder.py @@ -7,6 +7,7 @@ from inspect import isclass import numpy as np from fastNLP.envs.imports import _NEED_IMPORT_ONEFLOW +from fastNLP.envs.utils import _module_available if _NEED_IMPORT_ONEFLOW: import oneflow diff --git a/fastNLP/core/controllers/trainer.py b/fastNLP/core/controllers/trainer.py index 14dad89b..454db3d0 100644 --- a/fastNLP/core/controllers/trainer.py +++ b/fastNLP/core/controllers/trainer.py @@ -84,13 +84,13 @@ class Trainer(TrainerEventTrigger): .. warning:: 当使用分布式训练时， **fastNLP** 会默认将 ``dataloader`` 中的 ``Sampler`` 进行处理，以使得在一个 epoch 中，不同卡 - 用以训练的数据是不重叠的。如果你对 sampler 有特殊处理，那么请将 ``use_dist_sampler`` 参数设置为 ``False`` ，此刻需要由 - 你自身保证每张卡上所使用的数据是不同的。 + 用以训练的数据是不重叠的。如果您对 sampler 有特殊处理，那么请将 ``use_dist_sampler`` 参数设置为 ``False`` ，此刻需要由 + 您自身保证每张卡上所使用的数据是不同的。 :param optimizers: 训练所需要的优化器；可以是单独的一个优化器实例，也可以是多个优化器组成的 List； :param device: 该参数用来指定具体训练时使用的机器；注意当该参数仅当您通过 ``torch.distributed.launch/run`` 启动时可以为 ``None``， - 此时 fastNLP 不会对模型和数据进行设备之间的移动处理，但是你可以通过参数 ``input_mapping`` 和 ``output_mapping`` 来实现设备之间 - 数据迁移的工作（通过这两个参数传入两个处理数据的函数）；同时你也可以通过在 kwargs 添加参数 ``data_device`` 来让我们帮助您将数据 + 此时 fastNLP 不会对模型和数据进行设备之间的移动处理，但是您可以通过参数 ``input_mapping`` 和 ``output_mapping`` 来实现设备之间 + 数据迁移的工作（通过这两个参数传入两个处理数据的函数）；同时您也可以通过在 kwargs 添加参数 ``data_device`` 来让我们帮助您将数据 迁移到指定的机器上（注意这种情况理应只出现在用户在 Trainer 实例化前自己构造 DDP 的场景）； device 的可选输入如下所示： @@ -196,7 +196,7 @@ class Trainer(TrainerEventTrigger): 3. 如果此时 batch 此时是其它类型，那么我们将会直接报错； 2. 如果 ``input_mapping`` 是一个函数，那么对于取出的 batch，我们将不会做任何处理，而是直接将其传入该函数里； - 注意该参数会被传进 ``Evaluator`` 中；因此你可以通过该参数来实现将训练数据 batch 移到对应机器上的工作（例如当参数 ``device`` 为 ``None`` 时）； + 注意该参数会被传进 ``Evaluator`` 中；因此您可以通过该参数来实现将训练数据 batch 移到对应机器上的工作（例如当参数 ``device`` 为 ``None`` 时）； 如果 ``Trainer`` 和 ``Evaluator`` 需要使用不同的 ``input_mapping``, 请使用 ``train_input_mapping`` 与 ``evaluate_input_mapping`` 分别进行设置。 :param output_mapping: 应当为一个字典或者函数。作用和 ``input_mapping`` 类似，区别在于其用于转换输出： @@ -367,7 +367,7 @@ class Trainer(TrainerEventTrigger): .. note:: ``Trainer`` 是通过在内部直接初始化一个 ``Evaluator`` 来进行验证； - ``Trainer`` 内部的 ``Evaluator`` 默认是 None，如果您需要在训练过程中进行验证，你需要保证这几个参数得到正确的传入： + ``Trainer`` 内部的 ``Evaluator`` 默认是 None，如果您需要在训练过程中进行验证，您需要保证这几个参数得到正确的传入： 必须的参数：``metrics`` 与 ``evaluate_dataloaders``； @@ -898,7 +898,7 @@ class Trainer(TrainerEventTrigger): 这段代码意味着 ``fn1`` 和 ``fn2`` 会被加入到 ``trainer1``，``fn3`` 会被加入到 ``trainer2``； - 注意如果你使用该函数修饰器来为你的训练添加 callback，请务必保证你加入 callback 函数的代码在实例化 `Trainer` 之前； + 注意如果您使用该函数修饰器来为您的训练添加 callback，请务必保证您加入 callback 函数的代码在实例化 `Trainer` 之前； 补充性的解释见 :meth:`~fastNLP.core.controllers.Trainer.add_callback_fn`； diff --git a/fastNLP/core/dataset/dataset.py b/fastNLP/core/dataset/dataset.py index 862c7ee7..8ef521eb 100644 --- a/fastNLP/core/dataset/dataset.py +++ b/fastNLP/core/dataset/dataset.py @@ -584,7 +584,7 @@ class DataSet: 将 :class:`DataSet` 每个 ``instance`` 中为 ``field_name`` 的 field 传给函数 ``func``，并写入到 ``new_field_name`` 中。 - :param func: 对指定 fiel` 进行处理的函数，注意其输入应为 ``instance`` 中名为 ``field_name`` 的 field 的内容； + :param func: 对指定 field 进行处理的函数，注意其输入应为 ``instance`` 中名为 ``field_name`` 的 field 的内容； :param field_name: 传入 ``func`` 的 field 名称； :param new_field_name: 函数执行结果写入的 ``field`` 名称。该函数会将 ``func`` 返回的内容放入到 ``new_field_name`` 对 应的 ``field`` 中，注意如果名称与已有的 field 相同则会进行覆盖。如果为 ``None`` 则不会覆盖和创建 field ； @@ -624,10 +624,9 @@ class DataSet: ``apply_field_more`` 与 ``apply_field`` 的区别参考 :meth:`~fastNLP.core.dataset.DataSet.apply_more` 中关于 ``apply_more`` 与 ``apply`` 区别的介绍。 - :param func: 对指定 fiel` 进行处理的函数，注意其输入应为 ``instance`` 中名为 ``field_name`` 的 field 的内容； - :param field_name: 传入 ``func`` 的 fiel` 名称； - :param new_field_name: 函数执行结果写入的 ``field`` 名称。该函数会将 ``func`` 返回的内容放入到 ``new_field_name`` 对 - 应的 ``field`` 中，注意如果名称与已有的 field 相同则会进行覆盖。如果为 ``None`` 则不会覆盖和创建 field ； + :param func: 对指定 field 进行处理的函数，注意其输入应为 ``instance`` 中名为 ``field_name`` 的 field 的内容； + :param field_name: 传入 ``func`` 的 field 名称； + :param modify_fields: 是否用结果修改 ``DataSet`` 中的 ``Field`` ， 默认为 ``True`` :param num_proc: 使用进程的数量。 .. note:: @@ -751,8 +750,8 @@ class DataSet: 3. ``apply_more`` 默认修改 ``DataSet`` 中的 field ，``apply`` 默认不修改。 - :param modify_fields: 是否用结果修改 ``DataSet`` 中的 ``Field`` ， 默认为 True :param func: 参数是 ``DataSet`` 中的 ``Instance`` ，返回值是一个字典，key 是field 的名字，value 是对应的结果 + :param modify_fields: 是否用结果修改 ``DataSet`` 中的 ``Field`` ， 默认为 ``True`` :param num_proc: 使用进程的数量。 .. note:: diff --git a/fastNLP/core/drivers/torch_driver/torch_fsdp.py b/fastNLP/core/drivers/torch_driver/torch_fsdp.py index 9359615a..0b1948e8 100644 --- a/fastNLP/core/drivers/torch_driver/torch_fsdp.py +++ b/fastNLP/core/drivers/torch_driver/torch_fsdp.py @@ -1,15 +1,17 @@ -from fastNLP.envs.imports import _TORCH_GREATER_EQUAL_1_12 +from fastNLP.envs.imports import _TORCH_GREATER_EQUAL_1_12, _NEED_IMPORT_TORCH if _TORCH_GREATER_EQUAL_1_12: from torch.distributed.fsdp import FullyShardedDataParallel, StateDictType, FullStateDictConfig, OptimStateKeyType +if _NEED_IMPORT_TORCH: + import torch + import torch.distributed as dist + from torch.nn.parallel import DistributedDataParallel + import os -import torch -import torch.distributed as dist -from torch.nn.parallel import DistributedDataParallel from typing import Optional, Union, List, Dict, Mapping from pathlib import Path diff --git a/fastNLP/core/utils/cache_results.py b/fastNLP/core/utils/cache_results.py index d462c06c..dc114b48 100644 --- a/fastNLP/core/utils/cache_results.py +++ b/fastNLP/core/utils/cache_results.py @@ -1,3 +1,14 @@ +""" +:func:`cache_results` 函数是 **fastNLP** 中用于缓存数据的装饰器，通过该函数您可以省去调试代码过程中一些耗时过长程序 +带来的时间开销。比如在加载并处理较大的数据时，每次修改训练参数都需要从头开始执行处理数据的过程，那么 :func:`cache_results` +便可以跳过这部分漫长的时间。详细的使用方法和原理请参见下面的说明。 + +.. warning:: + + 如果您发现对代码进行修改之后程序执行的结果没有变化，很有可能是这个函数的原因；届时删除掉缓存数据即可。 + +""" + from datetime import datetime import hashlib import _pickle diff --git a/fastNLP/embeddings/torch/static_embedding.py b/fastNLP/embeddings/torch/static_embedding.py index 12e7294c..6980c851 100644 --- a/fastNLP/embeddings/torch/static_embedding.py +++ b/fastNLP/embeddings/torch/static_embedding.py @@ -86,7 +86,7 @@ class StaticEmbedding(TokenEmbedding): :param requires_grad: 是否需要梯度。 :param init_method: 如何初始化没有找到的值。可以使用 :mod:`torch.nn.init` 中的各种方法，传入的方法应该接受一个 tensor，并 inplace 地修改其值。 - :param lower: 是否将 ``vocab`` 中的词语小写后再和预训练的词表进行匹配。如果你的词表中包含大写的词语，或者就是需要单独 + :param lower: 是否将 ``vocab`` 中的词语小写后再和预训练的词表进行匹配。如果您的词表中包含大写的词语，或者就是需要单独 为大写的词语开辟一个 vector 表示，则将 ``lower`` 设置为 ``False``。 :param dropout: 以多大的概率对 embedding 的表示进行 Dropout。0.1 即随机将 10% 的值置为 0。 :param word_dropout: 按照一定概率随机将 word 设置为 ``unk_index`` ，这样可以使得 ``
@@ -493,9 +421,10 @@ class IMDBPipe(CLSBasePipe): def process_from_file(self, paths=None): r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` - :param paths: 支持路径类型参见 :class:`fastNLP.io.loader.Loader` 的load函数。 - :return: DataBundle + :param paths: + :return: """ # 读取数据 data_bundle = IMDBLoader().load(paths) @@ -506,7 +435,7 @@ class IMDBPipe(CLSBasePipe): class ChnSentiCorpPipe(Pipe): r""" - 处理之后的DataSet有以下的结构 + 处理 **ChnSentiCorp** 的数据，处理之后 :class:`~fastNLP.core.DataSet` 中的内容为： .. csv-table:: :header: "raw_chars", "target", "chars", "seq_len" @@ -515,30 +444,18 @@ class ChnSentiCorpPipe(Pipe): "<荐书> 推荐所有喜欢<红楼>...", 1, "[10, 21, ....]", 25 "..." - 其中chars, seq_len是input，target是target - dataset的print_field_meta()函数输出的各个field的被设置成input和target的情况为:: - - +-------------+-----------+--------+-------+---------+ - | field_names | raw_chars | target | chars | seq_len | - +-------------+-----------+--------+-------+---------+ - | is_input | False | True | True | True | - | is_target | False | True | False | False | - | ignore_type | | False | False | False | - | pad_value | | 0 | 0 | 0 | - +-------------+-----------+--------+-------+---------+ + 其中 ``chars`` , ``seq_len`` 是 input， ``target`` 是 target。 + :param bigrams: 是否增加一列 ``bigrams`` 。 ``bigrams`` 会对原文进行如下转化： ``['复', '旦', '大', '学', ...]->["复旦", "旦大", ...]`` 。如果 + 设置为 ``True`` ，返回的 `~fastNLP.core.DataSet` 将有一列名为 ``bigrams`` ，且已经转换为了 index 并设置为 input，对应的词表可以通过 + ``data_bundle.get_vocab('bigrams')`` 获取。 + :param trigrams: 是否增加一列 ``trigrams`` 。 ``trigrams`` 会对原文进行如下转化 ``['复', '旦', '大', '学', ...]->["复旦大", "旦大学", ...]`` 。 + 如果设置为 ``True`` ，返回的 `~fastNLP.core.DataSet` 将有一列名为 ``trigrams`` ，且已经转换为了 index 并设置为 input，对应的词表可以通过 + ``data_bundle.get_vocab('trigrams')`` 获取。 + :param num_proc: 处理数据时使用的进程数目。 """ - def __init__(self, bigrams=False, trigrams=False, num_proc: int = 0): - r""" - - :param bool bigrams: 是否增加一列bigrams. bigrams的构成是['复', '旦', '大', '学', ...]->["复旦", "旦大", ...]。如果 - 设置为True，返回的DataSet将有一列名为bigrams, 且已经转换为了index并设置为input，对应的vocab可以通过 - data_bundle.get_vocab('bigrams')获取. - :param bool trigrams: 是否增加一列trigrams. trigrams的构成是 ['复', '旦', '大', '学', ...]->["复旦大", "旦大学", ...] - 。如果设置为True，返回的DataSet将有一列名为trigrams, 且已经转换为了index并设置为input，对应的vocab可以通过 - data_bundle.get_vocab('trigrams')获取. - """ + def __init__(self, bigrams: bool=False, trigrams: bool=False, num_proc: int = 0): super().__init__() self.bigrams = bigrams @@ -557,7 +474,7 @@ class ChnSentiCorpPipe(Pipe): def process(self, data_bundle: DataBundle): r""" - 可以处理的DataSet应该具备以下的field + ``data_bunlde`` 中的 :class:`~fastNLP.core.DataSet` 应该具备如下的结构： .. csv-table:: :header: "raw_chars", "target" @@ -567,7 +484,7 @@ class ChnSentiCorpPipe(Pipe): "..." :param data_bundle: - :return: + :return: 处理后的 ``data_bundle`` """ _add_chars_field(data_bundle, lower=False) @@ -601,9 +518,10 @@ class ChnSentiCorpPipe(Pipe): def process_from_file(self, paths=None): r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` - :param paths: 支持路径类型参见 :class:`fastNLP.io.loader.Loader` 的load函数。 - :return: DataBundle + :param paths: + :return: """ # 读取数据 data_bundle = ChnSentiCorpLoader().load(paths) @@ -614,7 +532,7 @@ class ChnSentiCorpPipe(Pipe): class THUCNewsPipe(CLSBasePipe): r""" - 处理之后的DataSet有以下的结构 + 处理 **THUCNews** 的数据，处理之后 :class:`~fastNLP.core.DataSet` 中的内容为： .. csv-table:: :header: "raw_chars", "target", "chars", "seq_len" @@ -622,27 +540,18 @@ class THUCNewsPipe(CLSBasePipe): "马晓旭意外受伤让国奥警惕 无奈大雨格外青睐殷家军记者傅亚雨沈阳报道...", 0, "[409, 1197, 2146, 213, ...]", 746 "..." - 其中chars, seq_len是input，target是target - dataset的print_field_meta()函数输出的各个field的被设置成input和target的情况为:: - - +-------------+-----------+--------+-------+---------+ - | field_names | raw_chars | target | chars | seq_len | - +-------------+-----------+--------+-------+---------+ - | is_input | False | True | True | True | - | is_target | False | True | False | False | - | ignore_type | | False | False | False | - | pad_value | | 0 | 0 | 0 | - +-------------+-----------+--------+-------+---------+ - - :param bool bigrams: 是否增加一列bigrams. bigrams的构成是['复', '旦', '大', '学', ...]->["复旦", "旦大", ...]。如果 - 设置为True，返回的DataSet将有一列名为bigrams, 且已经转换为了index并设置为input，对应的vocab可以通过 - data_bundle.get_vocab('bigrams')获取. - :param bool trigrams: 是否增加一列trigrams. trigrams的构成是 ['复', '旦', '大', '学', ...]->["复旦大", "旦大学", ...] - 。如果设置为True，返回的DataSet将有一列名为trigrams, 且已经转换为了index并设置为input，对应的vocab可以通过 - data_bundle.get_vocab('trigrams')获取. + 其中 ``chars`` , ``seq_len`` 是 input， ``target`` 是target + + :param bigrams: 是否增加一列 ``bigrams`` 。 ``bigrams`` 会对原文进行如下转化： ``['复', '旦', '大', '学', ...]->["复旦", "旦大", ...]`` 。如果 + 设置为 ``True`` ，返回的 `~fastNLP.core.DataSet` 将有一列名为 ``bigrams`` ，且已经转换为了 index 并设置为 input，对应的词表可以通过 + ``data_bundle.get_vocab('bigrams')`` 获取。 + :param trigrams: 是否增加一列 ``trigrams`` 。 ``trigrams`` 会对原文进行如下转化 ``['复', '旦', '大', '学', ...]->["复旦大", "旦大学", ...]`` 。 + 如果设置为 ``True`` ，返回的 `~fastNLP.core.DataSet` 将有一列名为 ``trigrams`` ，且已经转换为了 index 并设置为 input，对应的词表可以通过 + ``data_bundle.get_vocab('trigrams')`` 获取。 + :param num_proc: 处理数据时使用的进程数目。 """ - def __init__(self, bigrams=False, trigrams=False, num_proc=0): + def __init__(self, bigrams: int=False, trigrams: int=False, num_proc: int=0): super().__init__(num_proc=num_proc) self.bigrams = bigrams @@ -663,7 +572,7 @@ class THUCNewsPipe(CLSBasePipe): def process(self, data_bundle: DataBundle): r""" - 可处理的DataSet应具备如下的field + ``data_bunlde`` 中的 :class:`~fastNLP.core.DataSet` 应该具备如下的结构： .. csv-table:: :header: "raw_words", "target" @@ -672,7 +581,7 @@ class THUCNewsPipe(CLSBasePipe): "...", "..." :param data_bundle: - :return: + :return: 处理后的 ``data_bundle`` """ # 根据granularity设置tag tag_map = {'体育': 0, '财经': 1, '房产': 2, '家居': 3, '教育': 4, '科技': 5, '时尚': 6, '时政': 7, '游戏': 8, '娱乐': 9} @@ -713,8 +622,10 @@ class THUCNewsPipe(CLSBasePipe): def process_from_file(self, paths=None): r""" - :param paths: 支持路径类型参见 :class:`fastNLP.io.loader.Loader` 的load函数。 - :return: DataBundle + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param paths: + :return: """ data_loader = THUCNewsLoader() # 此处需要实例化一个data_loader，否则传入load()的参数为None data_bundle = data_loader.load(paths) @@ -724,32 +635,23 @@ class THUCNewsPipe(CLSBasePipe): class WeiboSenti100kPipe(CLSBasePipe): r""" - 处理之后的DataSet有以下的结构 + 处理 **WeiboSenti100k** 的数据，处理之后 :class:`~fastNLP.core.DataSet` 中的内容为： .. csv-table:: :header: "raw_chars", "target", "chars", "seq_len" - "六一出生的？好讽刺…… //@祭春姬:他爸爸是外星人吧 //@面孔小高:现在的孩子都怎么了 [怒][怒][怒]", 0, "[0, 690, 18, ...]", 56 + "马晓旭意外受伤让国奥警惕 无奈大雨格外青睐殷家军记者傅亚雨沈阳报道...", 0, "[409, 1197, 2146, 213, ...]", 746 "..." - 其中chars, seq_len是input，target是target - dataset的print_field_meta()函数输出的各个field的被设置成input和target的情况为:: - - +-------------+-----------+--------+-------+---------+ - | field_names | raw_chars | target | chars | seq_len | - +-------------+-----------+--------+-------+---------+ - | is_input | False | True | True | True | - | is_target | False | True | False | False | - | ignore_type | | False | False | False | - | pad_value | | 0 | 0 | 0 | - +-------------+-----------+--------+-------+---------+ - - :param bool bigrams: 是否增加一列bigrams. bigrams的构成是['复', '旦', '大', '学', ...]->["复旦", "旦大", ...]。如果 - 设置为True，返回的DataSet将有一列名为bigrams, 且已经转换为了index并设置为input，对应的vocab可以通过 - data_bundle.get_vocab('bigrams')获取. - :param bool trigrams: 是否增加一列trigrams. trigrams的构成是 ['复', '旦', '大', '学', ...]->["复旦大", "旦大学", ...] - 。如果设置为True，返回的DataSet将有一列名为trigrams, 且已经转换为了index并设置为input，对应的vocab可以通过 - data_bundle.get_vocab('trigrams')获取. + 其中 ``chars`` , ``seq_len`` 是 input， ``target`` 是target + + :param bigrams: 是否增加一列 ``bigrams`` 。 ``bigrams`` 会对原文进行如下转化： ``['复', '旦', '大', '学', ...]->["复旦", "旦大", ...]`` 。如果 + 设置为 ``True`` ，返回的 `~fastNLP.core.DataSet` 将有一列名为 ``bigrams`` ，且已经转换为了 index 并设置为 input，对应的词表可以通过 + ``data_bundle.get_vocab('bigrams')`` 获取。 + :param trigrams: 是否增加一列 ``trigrams`` 。 ``trigrams`` 会对原文进行如下转化 ``['复', '旦', '大', '学', ...]->["复旦大", "旦大学", ...]`` 。 + 如果设置为 ``True`` ，返回的 `~fastNLP.core.DataSet` 将有一列名为 ``trigrams`` ，且已经转换为了 index 并设置为 input，对应的词表可以通过 + ``data_bundle.get_vocab('trigrams')`` 获取。 + :param num_proc: 处理数据时使用的进程数目。 """ def __init__(self, bigrams=False, trigrams=False, num_proc=0): @@ -770,7 +672,7 @@ class WeiboSenti100kPipe(CLSBasePipe): def process(self, data_bundle: DataBundle): r""" - 可处理的DataSet应具备以下的field + ``data_bunlde`` 中的 :class:`~fastNLP.core.DataSet` 应该具备如下的结构： .. csv-table:: :header: "raw_chars", "target" @@ -779,7 +681,7 @@ class WeiboSenti100kPipe(CLSBasePipe): "...", "..." :param data_bundle: - :return: + :return: 处理后的 ``data_bundle`` """ # clean,lower @@ -811,8 +713,10 @@ class WeiboSenti100kPipe(CLSBasePipe): def process_from_file(self, paths=None): r""" - :param paths: 支持路径类型参见 :class:`fastNLP.io.loader.Loader` 的load函数。 - :return: DataBundle + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param paths: + :return: """ data_loader = WeiboSenti100kLoader() # 此处需要实例化一个data_loader，否则传入load()的参数为None data_bundle = data_loader.load(paths) @@ -820,20 +724,23 @@ class WeiboSenti100kPipe(CLSBasePipe): return data_bundle class MRPipe(CLSBasePipe): - def __init__(self, lower: bool = False, tokenizer: str = 'spacy', num_proc=0): - r""" + """ + 加载 **MR** 的数据。 - :param bool lower: 是否将words列的数据小写。 - :param str tokenizer: 使用什么tokenizer来将句子切分为words. 支持spacy, raw两种。raw即使用空格拆分。 - """ + :param lower: 是否对输入进行小写化。 + :param tokenizer: 使用哪种 tokenize 方式将数据切成单词。支持 ``['spacy', 'raw']`` 。``'raw'`` 表示使用空格作为切分，``'spacy'`` 则使用 :mod:`spacy` 库进行分词。 + :param num_proc: 处理数据时使用的进程数目。 + """ + def __init__(self, lower: bool = False, tokenizer: str = 'spacy', num_proc=0): super().__init__(tokenizer=tokenizer, lang='en', num_proc=num_proc) self.lower = lower def process_from_file(self, paths=None): r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` - :param paths: 支持路径类型参见 :class:`fastNLP.io.loader.Loader` 的load函数。 - :return: DataBundle + :param paths: + :return: """ # 读取数据 data_bundle = MRLoader().load(paths) @@ -843,20 +750,23 @@ class MRPipe(CLSBasePipe): class R8Pipe(CLSBasePipe): - def __init__(self, lower: bool = False, tokenizer: str = 'spacy', num_proc = 0): - r""" + """ + 加载 **R8** 的数据。 - :param bool lower: 是否将words列的数据小写。 - :param str tokenizer: 使用什么tokenizer来将句子切分为words. 支持spacy, raw两种。raw即使用空格拆分。 - """ + :param lower: 是否对输入进行小写化。 + :param tokenizer: 使用哪种 tokenize 方式将数据切成单词。支持 ``['spacy', 'raw']`` 。``'raw'`` 表示使用空格作为切分，``'spacy'`` 则使用 :mod:`spacy` 库进行分词。 + :param num_proc: 处理数据时使用的进程数目。 + """ + def __init__(self, lower: bool = False, tokenizer: str = 'spacy', num_proc = 0): super().__init__(tokenizer=tokenizer, lang='en', num_proc=num_proc) self.lower = lower def process_from_file(self, paths=None): r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` - :param paths: 支持路径类型参见 :class:`fastNLP.io.loader.Loader` 的load函数。 - :return: DataBundle + :param paths: + :return: """ # 读取数据 data_bundle = R8Loader().load(paths) @@ -866,20 +776,23 @@ class R8Pipe(CLSBasePipe): class R52Pipe(CLSBasePipe): - def __init__(self, lower: bool = False, tokenizer: str = 'spacy', num_proc: int = 0): - r""" + """ + 加载 **R52** 的数据。 - :param bool lower: 是否将words列的数据小写。 - :param str tokenizer: 使用什么tokenizer来将句子切分为words. 支持spacy, raw两种。raw即使用空格拆分。 - """ + :param lower: 是否对输入进行小写化。 + :param tokenizer: 使用哪种 tokenize 方式将数据切成单词。支持 ``['spacy', 'raw']`` 。``'raw'`` 表示使用空格作为切分，``'spacy'`` 则使用 :mod:`spacy` 库进行分词。 + :param num_proc: 处理数据时使用的进程数目。 + """ + def __init__(self, lower: bool = False, tokenizer: str = 'spacy', num_proc: int = 0): super().__init__(tokenizer=tokenizer, lang='en', num_proc=num_proc) self.lower = lower def process_from_file(self, paths=None): r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` - :param paths: 支持路径类型参见 :class:`fastNLP.io.loader.Loader` 的load函数。 - :return: DataBundle + :param paths: + :return: """ # 读取数据 data_bundle = R52Loader().load(paths) @@ -889,20 +802,23 @@ class R52Pipe(CLSBasePipe): class OhsumedPipe(CLSBasePipe): - def __init__(self, lower: bool = False, tokenizer: str = 'spacy', num_proc: int = 0): - r""" + """ + 加载 **Ohsumed** 的数据。 - :param bool lower: 是否将words列的数据小写。 - :param str tokenizer: 使用什么tokenizer来将句子切分为words. 支持spacy, raw两种。raw即使用空格拆分。 - """ + :param lower: 是否对输入进行小写化。 + :param tokenizer: 使用哪种 tokenize 方式将数据切成单词。支持 ``['spacy', 'raw']`` 。``'raw'`` 表示使用空格作为切分，``'spacy'`` 则使用 :mod:`spacy` 库进行分词。 + :param num_proc: 处理数据时使用的进程数目。 + """ + def __init__(self, lower: bool = False, tokenizer: str = 'spacy', num_proc: int = 0): super().__init__(tokenizer=tokenizer, lang='en', num_proc=num_proc) self.lower = lower def process_from_file(self, paths=None): r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` - :param paths: 支持路径类型参见 :class:`fastNLP.io.loader.Loader` 的load函数。 - :return: DataBundle + :param paths: + :return: """ # 读取数据 data_bundle = OhsumedLoader().load(paths) @@ -912,20 +828,23 @@ class OhsumedPipe(CLSBasePipe): class NG20Pipe(CLSBasePipe): - def __init__(self, lower: bool = False, tokenizer: str = 'spacy', num_proc: int = 0): - r""" + """ + 加载 **NG20** 的数据。 - :param bool lower: 是否将words列的数据小写。 - :param str tokenizer: 使用什么tokenizer来将句子切分为words. 支持spacy, raw两种。raw即使用空格拆分。 - """ + :param lower: 是否对输入进行小写化。 + :param tokenizer: 使用哪种 tokenize 方式将数据切成单词。支持 ``['spacy', 'raw']`` 。``'raw'`` 表示使用空格作为切分，``'spacy'`` 则使用 :mod:`spacy` 库进行分词。 + :param num_proc: 处理数据时使用的进程数目。 + """ + def __init__(self, lower: bool = False, tokenizer: str = 'spacy', num_proc: int = 0): super().__init__(tokenizer=tokenizer, lang='en', num_proc=num_proc) self.lower = lower def process_from_file(self, paths=None): r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` - :param paths: 支持路径类型参见 :class:`fastNLP.io.loader.Loader` 的load函数。 - :return: DataBundle + :param paths: + :return: """ # 读取数据 data_bundle = NG20Loader().load(paths) diff --git a/fastNLP/io/pipe/conll.py b/fastNLP/io/pipe/conll.py index efe05de0..d2d4730e 100644 --- a/fastNLP/io/pipe/conll.py +++ b/fastNLP/io/pipe/conll.py @@ -1,5 +1,3 @@ -r"""undocumented""" - __all__ = [ "Conll2003NERPipe", "Conll2003Pipe", @@ -60,7 +58,7 @@ class _NERPipe(Pipe): "[...]", "[...]" :param ~fastNLP.DataBundle data_bundle: 传入的DataBundle中的DataSet必须包含raw_words和ner两个field，且两个field的内容均为List[str]在传入DataBundle基础上原位修改。 - :return DataBundle: + :return: 处理后的 ``data_bundle`` """ # 转换tag for name, dataset in data_bundle.iter_datasets(): @@ -79,10 +77,14 @@ class _NERPipe(Pipe): class Conll2003NERPipe(_NERPipe): r""" - Conll2003的NER任务的处理Pipe, 该Pipe会（1）复制raw_words列，并命名为words; (2）在words, target列建立词表 - (创建 :class:`fastNLP.Vocabulary` 对象，所以在返回的DataBundle中将有两个Vocabulary); (3）将words，target列根据相应的 - Vocabulary转换为index。 - 经过该Pipe过后，DataSet中的内容如下所示 + **Conll2003** 的 **NER** 任务的处理 **Pipe** ， 该Pipe会： + + 1. 复制 ``raw_words`` 列，并命名为 ``words`` ； + 2. 在 ``words`` , ``target`` 列建立词表，即创建 :class:`~fastNLP.core.Vocabulary` 对象，所以在返回的 + :class:`~fastNLP.io.DataBundle` 中将有两个 ``Vocabulary`` ； + 3. 将 ``words`` , ``target`` 列根据相应的词表转换为 index。 + + 处理之后 :class:`~fastNLP.core.DataSet` 中的内容如下： .. csv-table:: Following is a demo layout of DataSet returned by Conll2003Loader :header: "raw_words", "target", "words", "seq_len" @@ -91,27 +93,21 @@ class Conll2003NERPipe(_NERPipe): "[AL-AIN, United, Arab, ...]", "[3, 4,...]", "[4, 5, 6,...]", 6 "[...]", "[...]", "[...]", . - raw_words列为List[str], 是未转换的原始数据; words列为List[int]，是转换为index的输入数据; target列是List[int]，是转换为index的 - target。返回的DataSet中被设置为input有words, target, seq_len; 设置为target有target。 - - dataset的print_field_meta()函数输出的各个field的被设置成input和target的情况为:: - - +-------------+-----------+--------+-------+---------+ - | field_names | raw_words | target | words | seq_len | - +-------------+-----------+--------+-------+---------+ - | is_input | False | True | True | True | - | is_target | False | True | False | True | - | ignore_type | | False | False | False | - | pad_value | | 0 | 0 | 0 | - +-------------+-----------+--------+-------+---------+ + ``raw_words`` 列为 :class:`List` [ :class:`str` ], 是未转换的原始数据； ``words`` 列为 :class:`List` [ :class:`int` ]， + 是转换为 index 的输入数据； ``target`` 列是 :class:`List` [ :class:`int` ] ，是转换为 index 的 target。返回的 :class:`~fastNLP.core.DataSet` + 中被设置为 input 有 ``words`` , ``target``, ``seq_len``；target 有 ``target`` 。 + :param encoding_type: ``target`` 列使用什么类型的 encoding 方式，支持 ``['bioes', 'bio']`` 两种。 + :param lower: 是否将 ``words`` 小写化后再建立词表，绝大多数情况都不需要设置为 ``True`` 。 + :param num_proc: 处理数据时使用的进程数目。 """ def process_from_file(self, paths) -> DataBundle: r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` - :param paths: 支持路径类型参见 :class:`fastNLP.io.loader.ConllLoader` 的load函数。 - :return: DataBundle + :param paths: + :return: """ # 读取数据 data_bundle = Conll2003NERLoader().load(paths) @@ -122,7 +118,7 @@ class Conll2003NERPipe(_NERPipe): class Conll2003Pipe(Pipe): r""" - 经过该Pipe后，DataSet中的内容如下 + 处理 **Conll2003** 的数据，处理之后 :class:`~fastNLP.core.DataSet` 中的内容如下： .. csv-table:: :header: "raw_words" , "pos", "chunk", "ner", "words", "seq_len" @@ -131,27 +127,14 @@ class Conll2003Pipe(Pipe): "[AL-AIN, United, Arab, ...]", "[1, 2...]", "[3, 4...]", "[3, 4...]", "[4, 5, 6,...]", 6 "[...]", "[...]", "[...]", "[...]", "[...]", . - 其中words, seq_len是input; pos, chunk, ner, seq_len是target - dataset的print_field_meta()函数输出的各个field的被设置成input和target的情况为:: - - +-------------+-----------+-------+-------+-------+-------+---------+ - | field_names | raw_words | pos | chunk | ner | words | seq_len | - +-------------+-----------+-------+-------+-------+-------+---------+ - | is_input | False | False | False | False | True | True | - | is_target | False | True | True | True | False | True | - | ignore_type | | False | False | False | False | False | - | pad_value | | 0 | 0 | 0 | 0 | 0 | - +-------------+-----------+-------+-------+-------+-------+---------+ - + 其中``words``, ``seq_len`` 是 input; ``pos``, ``chunk``, ``ner``, ``seq_len`` 是 target + :param chunk_encoding_type: ``chunk`` 列使用什么类型的 encoding 方式，支持 ``['bioes', 'bio']`` 两种。 + :param ner_encoding_type: ``ner`` 列使用什么类型的 encoding 方式，支持 ``['bioes', 'bio']`` 两种。 + :param lower: 是否将 ``words`` 小写化后再建立词表，绝大多数情况都不需要设置为 ``True`` 。 + :param num_proc: 处理数据时使用的进程数目。 """ - def __init__(self, chunk_encoding_type='bioes', ner_encoding_type='bioes', lower: bool = False, num_proc: int = 0): - r""" - - :param str chunk_encoding_type: 支持bioes, bio。 - :param str ner_encoding_type: 支持bioes, bio。 - :param bool lower: 是否将words列小写化后再建立词表 - """ + def __init__(self, chunk_encoding_type: str='bioes', ner_encoding_type: str='bioes', lower: bool = False, num_proc: int = 0): if chunk_encoding_type == 'bio': self.chunk_convert_tag = iob2 elif chunk_encoding_type == 'bioes': @@ -175,7 +158,7 @@ class Conll2003Pipe(Pipe): def process(self, data_bundle) -> DataBundle: r""" - 输入的DataSet应该类似于如下的形式 + 输入的 `~fastNLP.core.DataSet` 应该类似于如下的形式： .. csv-table:: :header: "raw_words", "pos", "chunk", "ner" @@ -185,7 +168,7 @@ class Conll2003Pipe(Pipe): "[...]", "[...]", "[...]", "[...]", . :param data_bundle: - :return: 传入的DataBundle + :return: 处理后的 ``data_bundle`` """ # 转换tag for name, dataset in data_bundle.datasets.items(): @@ -210,6 +193,7 @@ class Conll2003Pipe(Pipe): def process_from_file(self, paths): r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` :param paths: :return: @@ -220,7 +204,7 @@ class Conll2003Pipe(Pipe): class OntoNotesNERPipe(_NERPipe): r""" - 处理OntoNotes的NER数据，处理之后DataSet中的field情况为 + 处理 **OntoNotes** 的 **NER** 数据，处理之后 :class:`~fastNLP.core.DataSet` 中的内容如下： .. csv-table:: :header: "raw_words", "target", "words", "seq_len" @@ -229,23 +213,19 @@ class OntoNotesNERPipe(_NERPipe): "[AL-AIN, United, Arab, ...]", "[3, 4]", "[4, 5, 6,...]", 6 "[...]", "[...]", "[...]", . - raw_words列为List[str], 是未转换的原始数据; words列为List[int]，是转换为index的输入数据; target列是List[int]，是转换为index的 - target。返回的DataSet中被设置为input有words, target, seq_len; 设置为target有target。 - - dataset的print_field_meta()函数输出的各个field的被设置成input和target的情况为:: - - +-------------+-----------+--------+-------+---------+ - | field_names | raw_words | target | words | seq_len | - +-------------+-----------+--------+-------+---------+ - | is_input | False | True | True | True | - | is_target | False | True | False | True | - | ignore_type | | False | False | False | - | pad_value | | 0 | 0 | 0 | - +-------------+-----------+--------+-------+---------+ + ``raw_words`` 列为 :class:`List` [ :class:`str` ], 是未转换的原始数据； ``words`` 列为 :class:`List` [ :class:`int` ]， + 是转换为 index 的输入数据； ``target`` 列是 :class:`List` [ :class:`int` ] ，是转换为 index 的 target。返回的 :class:`~fastNLP.core.DataSet` + 中被设置为 input 有 ``words`` , ``target``, ``seq_len``；target 有 ``target`` 。 """ def process_from_file(self, paths): + r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param paths: + :return: + """ data_bundle = OntoNotesNERLoader().load(paths) return self.process(data_bundle) @@ -301,7 +281,7 @@ class _CNNERPipe(Pipe): 是转换为index的target。返回的DataSet中被设置为input有chars, target, seq_len; 设置为target有target。 :param ~fastNLP.DataBundle data_bundle: 传入的DataBundle中的DataSet必须包含raw_words和ner两个field，且两个field的内容均为List[str]。在传入DataBundle基础上原位修改。 - :return: DataBundle + :return: 处理后的 ``data_bundle`` """ # 转换tag for name, dataset in data_bundle.datasets.items(): @@ -338,7 +318,7 @@ class _CNNERPipe(Pipe): class MsraNERPipe(_CNNERPipe): r""" - 处理MSRA-NER的数据，处理之后的DataSet的field情况为 + 处理 **MSRA-NER** 的数据，处理之后 :class:`~fastNLP.core.DataSet` 中的内容如下： .. csv-table:: :header: "raw_chars", "target", "chars", "seq_len" @@ -347,30 +327,34 @@ class MsraNERPipe(_CNNERPipe): "[青, 岛, 海, 牛, 队, 和, ...]", "[1, 2, 3, ...]", "[10, 21, ....]", 21 "[...]", "[...]", "[...]", . - raw_chars列为List[str], 是未转换的原始数据; chars列为List[int]，是转换为index的输入数据; target列是List[int]，是转换为index的 - target。返回的DataSet中被设置为input有chars, target, seq_len; 设置为target有target。 - - dataset的print_field_meta()函数输出的各个field的被设置成input和target的情况为:: - - +-------------+-----------+--------+-------+---------+ - | field_names | raw_chars | target | chars | seq_len | - +-------------+-----------+--------+-------+---------+ - | is_input | False | True | True | True | - | is_target | False | True | False | True | - | ignore_type | | False | False | False | - | pad_value | | 0 | 0 | 0 | - +-------------+-----------+--------+-------+---------+ - + ``raw_chars`` 列为 :class:`List` [ :class:`str` ], 是未转换的原始数据； ``chars`` 列为 :class:`List` [ :class:`int` ]， + 是转换为 index 的输入数据； ``target`` 列是 :class:`List` [ :class:`int` ] ，是转换为 index 的 target。返回的 :class:`~fastNLP.core.DataSet` + 中被设置为 input 有 ``chars`` , ``target``, ``seq_len``；target 有 ``target`` 。 + + :param encoding_type: ``target`` 列使用什么类型的 encoding 方式，支持 ``['bioes', 'bio']`` 两种。 + :param bigrams: 是否增加一列 ``bigrams`` 。 ``bigrams`` 会对原文进行如下转化： ``['复', '旦', '大', '学', ...]->["复旦", "旦大", ...]`` 。如果 + 设置为 ``True`` ，返回的 :class:`~fastNLP.core.DataSet` 将有一列名为 ``bigrams`` ，且已经转换为了 index 并设置为 input，对应的词表可以通过 + ``data_bundle.get_vocab('bigrams')`` 获取。 + :param trigrams: 是否增加一列 ``trigrams`` 。 ``trigrams`` 会对原文进行如下转化 ``['复', '旦', '大', '学', ...]->["复旦大", "旦大学", ...]`` 。 + 如果设置为 ``True`` ，返回的 :class:`~fastNLP.core.DataSet` 将有一列名为 ``trigrams`` ，且已经转换为了 index 并设置为 input，对应的词表可以通过 + ``data_bundle.get_vocab('trigrams')`` 获取。 + :param num_proc: 处理数据时使用的进程数目。 """ def process_from_file(self, paths=None) -> DataBundle: + r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param paths: + :return: + """ data_bundle = MsraNERLoader().load(paths) return self.process(data_bundle) class PeopleDailyPipe(_CNNERPipe): r""" - 处理people daily的ner的数据，处理之后的DataSet的field情况为 + 处理 **People's Daily NER** 的 **ner** 的数据，处理之后 :class:`~fastNLP.core.DataSet` 中的内容如下： .. csv-table:: :header: "raw_chars", "target", "chars", "seq_len" @@ -379,30 +363,34 @@ class PeopleDailyPipe(_CNNERPipe): "[青, 岛, 海, 牛, 队, 和, ...]", "[1, 2, 3, ...]", "[10, 21, ....]", 21 "[...]", "[...]", "[...]", . - raw_chars列为List[str], 是未转换的原始数据; chars列为List[int]，是转换为index的输入数据; target列是List[int]，是转换为index的 - target。返回的DataSet中被设置为input有chars, target, seq_len; 设置为target有target。 - - dataset的print_field_meta()函数输出的各个field的被设置成input和target的情况为:: - - +-------------+-----------+--------+-------+---------+ - | field_names | raw_chars | target | chars | seq_len | - +-------------+-----------+--------+-------+---------+ - | is_input | False | True | True | True | - | is_target | False | True | False | True | - | ignore_type | | False | False | False | - | pad_value | | 0 | 0 | 0 | - +-------------+-----------+--------+-------+---------+ - + ``raw_chars`` 列为 :class:`List` [ :class:`str` ], 是未转换的原始数据； ``chars`` 列为 :class:`List` [ :class:`int` ]， + 是转换为 index 的输入数据； ``target`` 列是 :class:`List` [ :class:`int` ] ，是转换为 index 的 target。返回的 :class:`~fastNLP.core.DataSet` + 中被设置为 input 有 ``chars`` , ``target``, ``seq_len``；target 有 ``target`` 。 + + :param encoding_type: ``target`` 列使用什么类型的 encoding 方式，支持 ``['bioes', 'bio']`` 两种。 + :param bigrams: 是否增加一列 ``bigrams`` 。 ``bigrams`` 会对原文进行如下转化： ``['复', '旦', '大', '学', ...]->["复旦", "旦大", ...]`` 。如果 + 设置为 ``True`` ，返回的 :class:`~fastNLP.core.DataSet` 将有一列名为 ``bigrams`` ，且已经转换为了 index 并设置为 input，对应的词表可以通过 + ``data_bundle.get_vocab('bigrams')`` 获取。 + :param trigrams: 是否增加一列 ``trigrams`` 。 ``trigrams`` 会对原文进行如下转化 ``['复', '旦', '大', '学', ...]->["复旦大", "旦大学", ...]`` 。 + 如果设置为 ``True`` ，返回的 :class:`~fastNLP.core.DataSet` 将有一列名为 ``trigrams`` ，且已经转换为了 index 并设置为 input，对应的词表可以通过 + ``data_bundle.get_vocab('trigrams')`` 获取。 + :param num_proc: 处理数据时使用的进程数目。 """ def process_from_file(self, paths=None) -> DataBundle: + r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param paths: + :return: + """ data_bundle = PeopleDailyNERLoader().load(paths) return self.process(data_bundle) class WeiboNERPipe(_CNNERPipe): r""" - 处理weibo的ner的数据，处理之后的DataSet的field情况为 + 处理 **Weibo** 的 **BER** 的数据，处理之后 :class:`~fastNLP.core.DataSet` 中的内容如下： .. csv-table:: :header: "raw_chars", "chars", "target", "seq_len" @@ -411,22 +399,26 @@ class WeiboNERPipe(_CNNERPipe): "['心']", "[0]", "[41]", 1 "[...]", "[...]", "[...]", . - raw_chars列为List[str], 是未转换的原始数据; chars列为List[int]，是转换为index的输入数据; target列是List[int]，是转换为index的 - target。返回的DataSet中被设置为input有chars, target, seq_len; 设置为target有target。 - - dataset的print_field_meta()函数输出的各个field的被设置成input和target的情况为:: - - +-------------+-----------+--------+-------+---------+ - | field_names | raw_chars | target | chars | seq_len | - +-------------+-----------+--------+-------+---------+ - | is_input | False | True | True | True | - | is_target | False | True | False | True | - | ignore_type | | False | False | False | - | pad_value | | 0 | 0 | 0 | - +-------------+-----------+--------+-------+---------+ - + ``raw_chars`` 列为 :class:`List` [ :class:`str` ], 是未转换的原始数据； ``chars`` 列为 :class:`List` [ :class:`int` ]， + 是转换为 index 的输入数据； ``target`` 列是 :class:`List` [ :class:`int` ] ，是转换为 index 的 target。返回的 :class:`~fastNLP.core.DataSet` + 中被设置为 input 有 ``chars`` , ``target``, ``seq_len``；target 有 ``target`` 。 + + :param encoding_type: ``target`` 列使用什么类型的 encoding 方式，支持 ``['bioes', 'bio']`` 两种。 + :param bigrams: 是否增加一列 ``bigrams`` 。 ``bigrams`` 会对原文进行如下转化： ``['复', '旦', '大', '学', ...]->["复旦", "旦大", ...]`` 。如果 + 设置为 ``True`` ，返回的 :class:`~fastNLP.core.DataSet` 将有一列名为 ``bigrams`` ，且已经转换为了 index 并设置为 input，对应的词表可以通过 + ``data_bundle.get_vocab('bigrams')`` 获取。 + :param trigrams: 是否增加一列 ``trigrams`` 。 ``trigrams`` 会对原文进行如下转化 ``['复', '旦', '大', '学', ...]->["复旦大", "旦大学", ...]`` 。 + 如果设置为 ``True`` ，返回的 :class:`~fastNLP.core.DataSet` 将有一列名为 ``trigrams`` ，且已经转换为了 index 并设置为 input，对应的词表可以通过 + ``data_bundle.get_vocab('trigrams')`` 获取。 + :param num_proc: 处理数据时使用的进程数目。 """ def process_from_file(self, paths=None) -> DataBundle: + r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param paths: + :return: + """ data_bundle = WeiboNERLoader().load(paths) return self.process(data_bundle) diff --git a/fastNLP/io/pipe/construct_graph.py b/fastNLP/io/pipe/construct_graph.py index 1448765e..26846002 100644 --- a/fastNLP/io/pipe/construct_graph.py +++ b/fastNLP/io/pipe/construct_graph.py @@ -164,7 +164,7 @@ class GraphBuilderBase: def build_graph_from_file(self, path: str): r""" - 传入文件路径，生成处理好的scipy_sparse_matrix对象。paths支持的路径形式可以参考 ：:meth:`fastNLP.io.Loader.load()` + 传入文件路径，生成处理好的scipy_sparse_matrix对象。paths支持的路径形式可以参考 ：:meth:`fastNLP.io.Loader.load` :param path: :return: scipy_sparse_matrix @@ -173,14 +173,20 @@ class GraphBuilderBase: class MRPmiGraphPipe(GraphBuilderBase): + """ + 构建 **MR** 数据集的 **Graph** 。 + :param graph_type: + :param widow_size: + :param threshold: + """ def __init__(self, graph_type='pmi', widow_size=10, threshold=0.): super().__init__(graph_type=graph_type, widow_size=widow_size, threshold=threshold) def build_graph(self, data_bundle: DataBundle): r""" - params: ~fastNLP.DataBundle data_bundle: 需要处理的DataBundle对象. - return 返回csr类型的稀疏矩阵图;训练集，验证集，测试集，在图中的index. + :param data_bundle: 需要处理的 :class:`~fastNLP.io.DataBundle` 对象。 + :return: 返回 ``csr`` 类型的稀疏矩阵图；包含训练集，验证集，测试集，在图中的 index 。 """ self._get_doc_edge(data_bundle) self._get_word_edge() @@ -190,19 +196,31 @@ class MRPmiGraphPipe(GraphBuilderBase): self.tr_doc_index, self.dev_doc_index, self.te_doc_index) def build_graph_from_file(self, path: str): + r""" + 传入文件路径，生成处理好的 ``scipy_sparse_matrix`` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param path: 数据集的路径。 + :return: 返回 ``csr`` 类型的稀疏矩阵图；包含训练集，验证集，测试集，在图中的 index 。 + """ data_bundle = MRLoader().load(path) return self.build_graph(data_bundle) class R8PmiGraphPipe(GraphBuilderBase): + """ + 构建 **R8** 数据集的 **Graph** 。 + :param graph_type: + :param widow_size: + :param threshold: + """ def __init__(self, graph_type='pmi', widow_size=10, threshold=0.): super().__init__(graph_type=graph_type, widow_size=widow_size, threshold=threshold) def build_graph(self, data_bundle: DataBundle): r""" - params: ~fastNLP.DataBundle data_bundle: 需要处理的DataBundle对象. - return 返回csr类型的稀疏矩阵图;训练集，验证集，测试集，在图中的index. + :param data_bundle: 需要处理的 :class:`~fastNLP.io.DataBundle` 对象。 + :return: 返回 ``csr`` 类型的稀疏矩阵图；包含训练集，验证集，测试集，在图中的 index 。 """ self._get_doc_edge(data_bundle) self._get_word_edge() @@ -212,19 +230,31 @@ class R8PmiGraphPipe(GraphBuilderBase): self.tr_doc_index, self.dev_doc_index, self.te_doc_index) def build_graph_from_file(self, path: str): + r""" + 传入文件路径，生成处理好的 ``scipy_sparse_matrix`` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param path: 数据集的路径。 + :return: 返回 ``csr`` 类型的稀疏矩阵图；包含训练集，验证集，测试集，在图中的 index 。 + """ data_bundle = R8Loader().load(path) return self.build_graph(data_bundle) class R52PmiGraphPipe(GraphBuilderBase): + """ + 构建 **R52** 数据集的 **Graph** 。 + :param graph_type: + :param widow_size: + :param threshold: + """ def __init__(self, graph_type='pmi', widow_size=10, threshold=0.): super().__init__(graph_type=graph_type, widow_size=widow_size, threshold=threshold) def build_graph(self, data_bundle: DataBundle): r""" - params: ~fastNLP.DataBundle data_bundle: 需要处理的DataBundle对象. - return 返回csr类型的稀疏矩阵;训练集，验证集，测试集，在图中的index. + :param data_bundle: 需要处理的 :class:`~fastNLP.io.DataBundle` 对象。 + :return: 返回 ``csr`` 类型的稀疏矩阵图；包含训练集，验证集，测试集，在图中的 index 。 """ self._get_doc_edge(data_bundle) self._get_word_edge() @@ -234,19 +264,31 @@ class R52PmiGraphPipe(GraphBuilderBase): self.tr_doc_index, self.dev_doc_index, self.te_doc_index) def build_graph_from_file(self, path: str): + r""" + 传入文件路径，生成处理好的 ``scipy_sparse_matrix`` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param path: 数据集的路径。 + :return: 返回 ``csr`` 类型的稀疏矩阵图；包含训练集，验证集，测试集，在图中的 index 。 + """ data_bundle = R52Loader().load(path) return self.build_graph(data_bundle) class OhsumedPmiGraphPipe(GraphBuilderBase): + """ + 构建 **Ohsuned** 数据集的 **Graph** 。 + :param graph_type: + :param widow_size: + :param threshold: + """ def __init__(self, graph_type='pmi', widow_size=10, threshold=0.): super().__init__(graph_type=graph_type, widow_size=widow_size, threshold=threshold) def build_graph(self, data_bundle: DataBundle): r""" - params: ~fastNLP.DataBundle data_bundle: 需要处理的DataBundle对象. - return 返回csr类型的稀疏矩阵图;训练集，验证集，测试集，在图中的index. + :param data_bundle: 需要处理的 :class:`~fastNLP.io.DataBundle` 对象。 + :return: 返回 ``csr`` 类型的稀疏矩阵图；包含训练集，验证集，测试集，在图中的 index 。 """ self._get_doc_edge(data_bundle) self._get_word_edge() @@ -256,19 +298,31 @@ class OhsumedPmiGraphPipe(GraphBuilderBase): self.tr_doc_index, self.dev_doc_index, self.te_doc_index) def build_graph_from_file(self, path: str): + r""" + 传入文件路径，生成处理好的 ``scipy_sparse_matrix`` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param path: 数据集的路径。 + :return: 返回 ``csr`` 类型的稀疏矩阵图；包含训练集，验证集，测试集，在图中的 index 。 + """ data_bundle = OhsumedLoader().load(path) return self.build_graph(data_bundle) class NG20PmiGraphPipe(GraphBuilderBase): + """ + 构建 **NG20** 数据集的 **Graph** 。 + :param graph_type: + :param widow_size: + :param threshold: + """ def __init__(self, graph_type='pmi', widow_size=10, threshold=0.): super().__init__(graph_type=graph_type, widow_size=widow_size, threshold=threshold) def build_graph(self, data_bundle: DataBundle): r""" - params: ~fastNLP.DataBundle data_bundle: 需要处理的DataBundle对象. - return 返回csr类型的稀疏矩阵图;训练集，验证集，测试集，在图中的index. + :param data_bundle: 需要处理的 :class:`~fastNLP.io.DataBundle` 对象。 + :return: 返回 ``csr`` 类型的稀疏矩阵图；包含训练集，验证集，测试集，在图中的 index 。 """ self._get_doc_edge(data_bundle) self._get_word_edge() @@ -279,8 +333,10 @@ class NG20PmiGraphPipe(GraphBuilderBase): def build_graph_from_file(self, path: str): r""" - param: path->数据集的路径. - return: 返回csr类型的稀疏矩阵图;训练集，验证集，测试集，在图中的index. + 传入文件路径，生成处理好的 ``scipy_sparse_matrix`` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param path: 数据集的路径。 + :return: 返回 ``csr`` 类型的稀疏矩阵图；包含训练集，验证集，测试集，在图中的 index 。 """ data_bundle = NG20Loader().load(path) return self.build_graph(data_bundle) diff --git a/fastNLP/io/pipe/cws.py b/fastNLP/io/pipe/cws.py index 2937f147..27068348 100644 --- a/fastNLP/io/pipe/cws.py +++ b/fastNLP/io/pipe/cws.py @@ -1,5 +1,3 @@ -r"""undocumented""" - __all__ = [ "CWSPipe" ] @@ -135,7 +133,7 @@ def _find_and_replace_digit_spans(line): class CWSPipe(Pipe): r""" - 对CWS数据进行预处理, 处理之后的数据，具备以下的结构 + 对 **CWS** 数据进行处理，处理之后 :class:`~fastNLP.core.DataSet` 中的内容如下： .. csv-table:: :header: "raw_words", "chars", "target", "seq_len" @@ -144,30 +142,21 @@ class CWSPipe(Pipe): "2001年 新年 钟声...", "[8, 9, 9, 7, ...]", "[0, 1, 1, 1, 2...]", 20 "...", "[...]","[...]", . - dataset的print_field_meta()函数输出的各个field的被设置成input和target的情况为:: - - +-------------+-----------+-------+--------+---------+ - | field_names | raw_words | chars | target | seq_len | - +-------------+-----------+-------+--------+---------+ - | is_input | False | True | True | True | - | is_target | False | False | True | True | - | ignore_type | | False | False | False | - | pad_value | | 0 | 0 | 0 | - +-------------+-----------+-------+--------+---------+ - + :param dataset_name: data 的名称，支持 ``['pku', 'msra', 'cityu'(繁体), 'as'(繁体), None]`` + :param encoding_type: ``target`` 列使用什么类型的 encoding 方式，支持 ``['bmes', 'segapp']`` 两种。``"我 来自 复旦大学..."`` 这句话 ``bmes``的 + tag为 ``[S, B, E, B, M, M, E...]`` ； ``segapp`` 的 tag 为 ``[seg, app, seg, app, app, app, seg, ...]`` 。 + :param replace_num_alpha: 是否将数字和字母用特殊字符替换。 + :param bigrams: 是否增加一列 ``bigrams`` 。 ``bigrams`` 会对原文进行如下转化： ``['复', '旦', '大', '学', ...]->["复旦", "旦大", ...]`` 。如果 + 设置为 ``True`` ，返回的 :class:`~fastNLP.core.DataSet` 将有一列名为 ``bigrams`` ，且已经转换为了 index 并设置为 input，对应的词表可以通过 + ``data_bundle.get_vocab('bigrams')`` 获取。 + :param trigrams: 是否增加一列 ``trigrams`` 。 ``trigrams`` 会对原文进行如下转化 ``['复', '旦', '大', '学', ...]->["复旦大", "旦大学", ...]`` 。 + 如果设置为 ``True`` ，返回的 :class:`~fastNLP.core.DataSet` 将有一列名为 ``trigrams`` ，且已经转换为了 index 并设置为 input，对应的词表可以通过 + ``data_bundle.get_vocab('trigrams')`` 获取。 + :param num_proc: 处理数据时使用的进程数目。 """ - def __init__(self, dataset_name=None, encoding_type='bmes', replace_num_alpha=True, - bigrams=False, trigrams=False, num_proc: int = 0): - r""" - - :param str,None dataset_name: 支持'pku', 'msra', 'cityu', 'as', None - :param str encoding_type: 可以选择'bmes', 'segapp'两种。"我 来自 复旦大学...", bmes的tag为[S, B, E, B, M, M, E...]; segapp - 的tag为[seg, app, seg, app, app, app, seg, ...] - :param bool replace_num_alpha: 是否将数字和字母用特殊字符替换。 - :param bool bigrams: 是否增加一列bigram. bigram的构成是['复', '旦', '大', '学', ...]->["复旦", "旦大", ...] - :param bool trigrams: 是否增加一列trigram. trigram的构成是 ['复', '旦', '大', '学', ...]->["复旦大", "旦大学", ...] - """ + def __init__(self, dataset_name: str=None, encoding_type: str='bmes', replace_num_alpha: bool=True, + bigrams: bool=False, trigrams: bool=False, num_proc: int = 0): if encoding_type == 'bmes': self.word_lens_to_tags = _word_lens_to_bmes else: @@ -220,7 +209,7 @@ class CWSPipe(Pipe): def process(self, data_bundle: DataBundle) -> DataBundle: r""" - 可以处理的DataSet需要包含raw_words列 + ``data_bunlde`` 中的 :class:`~fastNLP.core.DataSet` 应该包含 ``raw_words`` ： .. csv-table:: :header: "raw_words" @@ -230,7 +219,7 @@ class CWSPipe(Pipe): "..." :param data_bundle: - :return: + :return: 处理后的 ``data_bundle`` """ data_bundle.copy_field('raw_words', 'chars') @@ -276,8 +265,9 @@ class CWSPipe(Pipe): def process_from_file(self, paths=None) -> DataBundle: r""" - - :param str paths: + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param paths: :return: """ if self.dataset_name is None and paths is None: diff --git a/fastNLP/io/pipe/matching.py b/fastNLP/io/pipe/matching.py index 5b9981c2..a9abd943 100644 --- a/fastNLP/io/pipe/matching.py +++ b/fastNLP/io/pipe/matching.py @@ -1,5 +1,3 @@ -r"""undocumented""" - __all__ = [ "MatchingBertPipe", "RTEBertPipe", @@ -21,7 +19,7 @@ __all__ = [ "BQCorpusPipe", "RenamePipe", "GranularizePipe", - "MachingTruncatePipe", + "TruncateBertPipe", ] from functools import partial @@ -31,14 +29,13 @@ from .utils import get_tokenizer from ..data_bundle import DataBundle from ..loader.matching import SNLILoader, MNLILoader, QNLILoader, RTELoader, QuoraLoader, BQCorpusLoader, CNXNLILoader, \ LCQMCLoader -# from ...core._logger import log # from ...core.const import Const from ...core.vocabulary import Vocabulary class MatchingBertPipe(Pipe): r""" - Matching任务的Bert pipe，输出的DataSet将包含以下的field + **Matching** 任务的 Bert pipe ，处理之后 :class:`~fastNLP.core.DataSet` 中的内容如下： .. csv-table:: :header: "raw_words1", "raw_words2", "target", "words", "seq_len" @@ -47,29 +44,17 @@ class MatchingBertPipe(Pipe): "This site includes a...", "The Government Executive...", 0, "[11, 12, 13,...]", 5 "...", "...", ., "[...]", . - words列是将raw_words1(即premise), raw_words2(即hypothesis)使用"[SEP]"链接起来转换为index的。 - words列被设置为input，target列被设置为target和input(设置为input以方便在forward函数中计算loss， - 如果不在forward函数中计算loss也不影响，fastNLP将根据forward函数的形参名进行传参). - - dataset的print_field_meta()函数输出的各个field的被设置成input和target的情况为:: - - +-------------+------------+------------+--------+-------+---------+ - | field_names | raw_words1 | raw_words2 | target | words | seq_len | - +-------------+------------+------------+--------+-------+---------+ - | is_input | False | False | False | True | True | - | is_target | False | False | True | False | False | - | ignore_type | | | False | False | False | - | pad_value | | | 0 | 0 | 0 | - +-------------+------------+------------+--------+-------+---------+ + ``words`` 列是将 ``raw_words1`` （即 ``premise`` ）， ``raw_words2`` （即 ``hypothesis`` ）使用 ``[SEP]`` + 链接起来转换为 index 的。``words`` 列被设置为 input， ``target`` 列被设置为 target 和 input （设置为 input 以 + 方便在 :func:`forward` 函数中计算 loss，如果不在也不影响， **fastNLP** 将根据 :func:`forward` 函数的形参名进行 + 传参）。 + :param lower: 是否对输入进行小写化。 + :param tokenizer: 使用哪种 tokenize 方式将数据切成单词。支持 ``['spacy', 'raw']`` 。``'raw'`` 表示使用空格作为切分，``'spacy'`` 则使用 :mod:`spacy` 库进行分词。 + :param num_proc: 处理数据时使用的进程数目。 """ def __init__(self, lower=False, tokenizer: str = 'raw', num_proc: int = 0): - r""" - - :param bool lower: 是否将word小写化。 - :param str tokenizer: 使用什么tokenizer来将句子切分为words. 支持spacy, raw两种。raw即使用空格拆分。 - """ super().__init__() self.lower = bool(lower) @@ -89,9 +74,9 @@ class MatchingBertPipe(Pipe): dataset.apply_field(self.tokenizer, field_name=field_name, new_field_name=new_field_name, num_proc=self.num_proc) return data_bundle - def process(self, data_bundle): + def process(self, data_bundle: DataBundle): r""" - 输入的data_bundle中的dataset需要具有以下结构： + ``data_bunlde`` 中的 :class:`~fastNLP.core.DataSet` 应该具备以下结构： .. csv-table:: :header: "raw_words1", "raw_words2", "target" @@ -100,7 +85,7 @@ class MatchingBertPipe(Pipe): "...","..." :param data_bundle: - :return: + :return: 处理后的 ``data_bundle`` """ for dataset in data_bundle.datasets.values(): if dataset.has_field('target'): @@ -164,38 +149,103 @@ class MatchingBertPipe(Pipe): class RTEBertPipe(MatchingBertPipe): + """ + 处理 **RTE** 数据。 + + :param lower: 是否对输入进行小写化。 + :param tokenizer: 使用哪种 tokenize 方式将数据切成单词。支持 ``['spacy', 'raw']`` 。``'raw'`` 表示使用空格作为切分，``'spacy'`` 则使用 :mod:`spacy` 库进行分词。 + :param num_proc: 处理数据时使用的进程数目。 + """ def process_from_file(self, paths=None): + r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param paths: + :return: + """ data_bundle = RTELoader().load(paths) return self.process(data_bundle) class SNLIBertPipe(MatchingBertPipe): + """ + 处理 **SNLI** 数据。 + + :param lower: 是否对输入进行小写化。 + :param tokenizer: 使用哪种 tokenize 方式将数据切成单词。支持 ``['spacy', 'raw']`` 。``'raw'`` 表示使用空格作为切分，``'spacy'`` 则使用 :mod:`spacy` 库进行分词。 + :param num_proc: 处理数据时使用的进程数目。 + """ def process_from_file(self, paths=None): + r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param paths: + :return: + """ data_bundle = SNLILoader().load(paths) return self.process(data_bundle) class QuoraBertPipe(MatchingBertPipe): + """ + 处理 **Quora** 数据。 + + :param lower: 是否对输入进行小写化。 + :param tokenizer: 使用哪种 tokenize 方式将数据切成单词。支持 ``['spacy', 'raw']`` 。``'raw'`` 表示使用空格作为切分，``'spacy'`` 则使用 :mod:`spacy` 库进行分词。 + :param num_proc: 处理数据时使用的进程数目。 + """ def process_from_file(self, paths): + r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param paths: + :return: + """ data_bundle = QuoraLoader().load(paths) return self.process(data_bundle) class QNLIBertPipe(MatchingBertPipe): + """ + 处理 **QNNLI** 数据。 + + :param lower: 是否对输入进行小写化。 + :param tokenizer: 使用哪种 tokenize 方式将数据切成单词。支持 ``['spacy', 'raw']`` 。``'raw'`` 表示使用空格作为切分，``'spacy'`` 则使用 :mod:`spacy` 库进行分词。 + :param num_proc: 处理数据时使用的进程数目。 + """ def process_from_file(self, paths=None): + r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param paths: + :return: + """ data_bundle = QNLILoader().load(paths) return self.process(data_bundle) class MNLIBertPipe(MatchingBertPipe): + """ + 处理 **MNLI** 数据。 + + :param lower: 是否对输入进行小写化。 + :param tokenizer: 使用哪种 tokenize 方式将数据切成单词。支持 ``['spacy', 'raw']`` 。``'raw'`` 表示使用空格作为切分，``'spacy'`` 则使用 :mod:`spacy` 库进行分词。 + :param num_proc: 处理数据时使用的进程数目。 + """ def process_from_file(self, paths=None): + r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param paths: + :return: + """ data_bundle = MNLILoader().load(paths) return self.process(data_bundle) class MatchingPipe(Pipe): r""" - Matching任务的Pipe。输出的DataSet将包含以下的field + **Matching** 任务的 Pipe，处理之后 :class:`~fastNLP.core.DataSet` 中的内容如下： .. csv-table:: :header: "raw_words1", "raw_words2", "target", "words1", "words2", "seq_len1", "seq_len2" @@ -204,21 +254,14 @@ class MatchingPipe(Pipe): "This site includes a...", "The Government Executive...", 0, "[11, 12, 13,...]", "[2, 7, ...]", 6, 7 "...", "...", ., "[...]", "[...]", ., . - words1是premise，words2是hypothesis。其中words1,words2,seq_len1,seq_len2被设置为input；target被设置为target - 和input(设置为input以方便在forward函数中计算loss，如果不在forward函数中计算loss也不影响，fastNLP将根据forward函数 - 的形参名进行传参)。 - - dataset的print_field_meta()函数输出的各个field的被设置成input和target的情况为:: - - +-------------+------------+------------+--------+--------+--------+----------+----------+ - | field_names | raw_words1 | raw_words2 | target | words1 | words2 | seq_len1 | seq_len2 | - +-------------+------------+------------+--------+--------+--------+----------+----------+ - | is_input | False | False | False | True | True | True | True | - | is_target | False | False | True | False | False | False | False | - | ignore_type | | | False | False | False | False | False | - | pad_value | | | 0 | 0 | 0 | 0 | 0 | - +-------------+------------+------------+--------+--------+--------+----------+----------+ + ``words1`` 是 ``premise`` ，``words2`` 是 ``hypothesis`` 。其中 ``words1`` , ``words2`` , ``seq_len1``, ``seq_len2`` + 被设置为 input； ``target`` 列被设置为 target 和 input （设置为 input 以 + 方便在 :func:`forward` 函数中计算 loss，如果不在也不影响， **fastNLP** 将根据 :func:`forward` 函数的形参名进行 + 传参）。 + :param lower: 是否对输入进行小写化。 + :param tokenizer: 使用哪种 tokenize 方式将数据切成单词。支持 ``['spacy', 'raw']`` 。``'raw'`` 表示使用空格作为切分，``'spacy'`` 则使用 :mod:`spacy` 库进行分词。 + :param num_proc: 处理数据时使用的进程数目。 """ def __init__(self, lower=False, tokenizer: str = 'raw', num_proc: int = 0): @@ -246,9 +289,9 @@ class MatchingPipe(Pipe): dataset.apply_field(self.tokenizer, field_name=field_name, new_field_name=new_field_name, num_proc=self.num_proc) return data_bundle - def process(self, data_bundle): + def process(self, data_bundle: DataBundle): r""" - 接受的DataBundle中的DataSet应该具有以下的field, target列可以没有 + ``data_bunlde`` 中的 :class:`~fastNLP.core.DataSet` 应该具备以下结构，可以没有 ``target`` 列： .. csv-table:: :header: "raw_words1", "raw_words2", "target" @@ -257,8 +300,8 @@ class MatchingPipe(Pipe): "This site includes a...", "The Government Executive...", "not_entailment" "...", "..." - :param ~fastNLP.DataBundle data_bundle: 通过loader读取得到的data_bundle，里面包含了数据集的原始数据内容 - :return: data_bundle + :param data_bundle: + :return: 处理后的 ``data_bundle`` """ data_bundle = self._tokenize(data_bundle, ['raw_words1', 'raw_words2'], ['words1', 'words2']) @@ -307,40 +350,117 @@ class MatchingPipe(Pipe): class RTEPipe(MatchingPipe): + """ + 处理 **RTE** 数据。 + + :param lower: 是否对输入进行小写化。 + :param tokenizer: 使用哪种 tokenize 方式将数据切成单词。支持 ``['spacy', 'raw']`` 。``'raw'`` 表示使用空格作为切分，``'spacy'`` 则使用 :mod:`spacy` 库进行分词。 + :param num_proc: 处理数据时使用的进程数目。 + """ def process_from_file(self, paths=None): + r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param paths: + :return: + """ data_bundle = RTELoader().load(paths) return self.process(data_bundle) class SNLIPipe(MatchingPipe): + """ + 处理 **SNLI** 数据。 + + :param lower: 是否对输入进行小写化。 + :param tokenizer: 使用哪种 tokenize 方式将数据切成单词。支持 ``['spacy', 'raw']`` 。``'raw'`` 表示使用空格作为切分，``'spacy'`` 则使用 :mod:`spacy` 库进行分词。 + :param num_proc: 处理数据时使用的进程数目。 + """ def process_from_file(self, paths=None): + r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param paths: + :return: + """ data_bundle = SNLILoader().load(paths) return self.process(data_bundle) class QuoraPipe(MatchingPipe): + """ + 处理 **Quora** 数据。 + + :param lower: 是否对输入进行小写化。 + :param tokenizer: 使用哪种 tokenize 方式将数据切成单词。支持 ``['spacy', 'raw']`` 。``'raw'`` 表示使用空格作为切分，``'spacy'`` 则使用 :mod:`spacy` 库进行分词。 + :param num_proc: 处理数据时使用的进程数目。 + """ def process_from_file(self, paths): + r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param paths: + :return: + """ data_bundle = QuoraLoader().load(paths) return self.process(data_bundle) class QNLIPipe(MatchingPipe): + """ + 处理 **QNLI** 数据。 + + :param lower: 是否对输入进行小写化。 + :param tokenizer: 使用哪种 tokenize 方式将数据切成单词。支持 ``['spacy', 'raw']`` 。``'raw'`` 表示使用空格作为切分，``'spacy'`` 则使用 :mod:`spacy` 库进行分词。 + :param num_proc: 处理数据时使用的进程数目。 + """ def process_from_file(self, paths=None): + r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param paths: + :return: + """ data_bundle = QNLILoader().load(paths) return self.process(data_bundle) class MNLIPipe(MatchingPipe): + """ + 处理 **MNLI** 数据。 + + :param lower: 是否对输入进行小写化。 + :param tokenizer: 使用哪种 tokenize 方式将数据切成单词。支持 ``['spacy', 'raw']`` 。``'raw'`` 表示使用空格作为切分，``'spacy'`` 则使用 :mod:`spacy` 库进行分词。 + :param num_proc: 处理数据时使用的进程数目。 + """ def process_from_file(self, paths=None): + r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param paths: + :return: + """ data_bundle = MNLILoader().load(paths) return self.process(data_bundle) class LCQMCPipe(MatchingPipe): - def __init__(self, tokenizer='cn=char', num_proc=0): + """ + 处理 **LCQMC** 数据。 + + :param tokenizer: 使用哪种 tokenize 方式将数据切成单词。支持 ``['cn-char']`` ，按字分词。 + :param num_proc: 处理数据时使用的进程数目。 + """ + def __init__(self, tokenizer='cn-char', num_proc=0): super().__init__(tokenizer=tokenizer, num_proc=num_proc) def process_from_file(self, paths=None): + r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param paths: + :return: + """ data_bundle = LCQMCLoader().load(paths) data_bundle = RenamePipe().process(data_bundle) data_bundle = self.process(data_bundle) @@ -349,10 +469,22 @@ class LCQMCPipe(MatchingPipe): class CNXNLIPipe(MatchingPipe): + """ + 处理 **XNLI Chinese** 数据。 + + :param tokenizer: 使用哪种 tokenize 方式将数据切成单词。支持 ``['cn-char']`` ，按字分词。 + :param num_proc: 处理数据时使用的进程数目。 + """ def __init__(self, tokenizer='cn-char', num_proc=0): super().__init__(tokenizer=tokenizer, num_proc=num_proc) def process_from_file(self, paths=None): + r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param paths: + :return: + """ data_bundle = CNXNLILoader().load(paths) data_bundle = GranularizePipe(task='XNLI').process(data_bundle) data_bundle = RenamePipe().process(data_bundle) # 使中文数据的field @@ -362,10 +494,22 @@ class CNXNLIPipe(MatchingPipe): class BQCorpusPipe(MatchingPipe): + """ + 处理 **BQ Corpus** 数据。 + + :param tokenizer: 使用哪种 tokenize 方式将数据切成单词。支持 ``['cn-char']`` ，按字分词。 + :param num_proc: 处理数据时使用的进程数目。 + """ def __init__(self, tokenizer='cn-char', num_proc=0): super().__init__(tokenizer=tokenizer, num_proc=num_proc) def process_from_file(self, paths=None): + r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param paths: + :return: + """ data_bundle = BQCorpusLoader().load(paths) data_bundle = RenamePipe().process(data_bundle) data_bundle = self.process(data_bundle) @@ -374,12 +518,23 @@ class BQCorpusPipe(MatchingPipe): class RenamePipe(Pipe): + """ + 重命名数据集的 Pipe ，经过处理后会将数据集中的 ``chars``, ``raw_chars1`` 等列重命名为 ``words``, + ``raw_words1``，反之亦然。 + + :param task: 任务类型，可选 ``['cn-nli', 'cn-nli-bert']`` 。 + :param num_proc: 处理数据时使用的进程数目。 + """ def __init__(self, task='cn-nli', num_proc=0): super().__init__() self.task = task self.num_proc = num_proc def process(self, data_bundle: DataBundle): # rename field name for Chinese Matching dataset + """ + :param data_bundle: + :return: 处理后的 ``data_bundle`` + """ if (self.task == 'cn-nli'): for name, dataset in data_bundle.datasets.items(): if (dataset.has_field('raw_chars1')): @@ -415,6 +570,16 @@ class RenamePipe(Pipe): class GranularizePipe(Pipe): + """ + 将数据集中 ``target`` 列中的 tag 按照一定的映射进行重命名，并丢弃不在映射中的 tag。 + + :param task: 任务类型，目前仅支持 ``['XNLI']``。 + + - ``'XNLI'`` -- 将 ``neutral``, ``entailment``, ``contradictory``, ``contradiction`` 分别 + 映射为 0, 1, 2, 3； + + :param num_proc: 处理数据时使用的进程数目。 + """ def __init__(self, task=None, num_proc=0): super().__init__() self.task = task @@ -437,6 +602,10 @@ class GranularizePipe(Pipe): return data_bundle def process(self, data_bundle: DataBundle): + """ + :param data_bundle: + :return: 处理后的 ``data_bundle`` + """ task_tag_dict = { 'XNLI': {'neutral': 0, 'entailment': 1, 'contradictory': 2, 'contradiction': 2} } @@ -446,22 +615,23 @@ class GranularizePipe(Pipe): raise RuntimeError(f"Only support {task_tag_dict.keys()} task_tag_map.") return data_bundle - -class MachingTruncatePipe(Pipe): # truncate sentence for bert, modify seq_len - def __init__(self): - super().__init__() - - def process(self, data_bundle: DataBundle): - for name, dataset in data_bundle.datasets.items(): - pass - return None - - class LCQMCBertPipe(MatchingBertPipe): - def __init__(self, tokenizer='cn=char', num_proc=0): + """ + 处理 **LCQMC** 数据 + + :param tokenizer: 使用哪种 tokenize 方式将数据切成单词。支持 ``['cn-char']`` ，按字分词。 + :param num_proc: 处理数据时使用的进程数目。 + """ + def __init__(self, tokenizer='cn-char', num_proc=0): super().__init__(tokenizer=tokenizer, num_proc=num_proc) def process_from_file(self, paths=None): + r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param paths: + :return: + """ data_bundle = LCQMCLoader().load(paths) data_bundle = RenamePipe(task='cn-nli-bert').process(data_bundle) data_bundle = self.process(data_bundle) @@ -471,10 +641,22 @@ class LCQMCBertPipe(MatchingBertPipe): class BQCorpusBertPipe(MatchingBertPipe): + """ + 处理 **BQ Corpus** 数据。 + + :param tokenizer: 使用哪种 tokenize 方式将数据切成单词。支持 ``['cn-char']`` ，按字分词。 + :param num_proc: 处理数据时使用的进程数目。 + """ def __init__(self, tokenizer='cn-char', num_proc=0): super().__init__(tokenizer=tokenizer, num_proc=num_proc) def process_from_file(self, paths=None): + r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param paths: + :return: + """ data_bundle = BQCorpusLoader().load(paths) data_bundle = RenamePipe(task='cn-nli-bert').process(data_bundle) data_bundle = self.process(data_bundle) @@ -484,10 +666,22 @@ class BQCorpusBertPipe(MatchingBertPipe): class CNXNLIBertPipe(MatchingBertPipe): + """ + 处理 **XNLI Chinese** 数据。 + + :param tokenizer: 使用哪种 tokenize 方式将数据切成单词。支持 ``['cn-char']`` ，按字分词。 + :param num_proc: 处理数据时使用的进程数目。 + """ def __init__(self, tokenizer='cn-char', num_proc=0): super().__init__(tokenizer=tokenizer, num_proc=num_proc) def process_from_file(self, paths=None): + r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param paths: + :return: + """ data_bundle = CNXNLILoader().load(paths) data_bundle = GranularizePipe(task='XNLI').process(data_bundle) data_bundle = RenamePipe(task='cn-nli-bert').process(data_bundle) @@ -498,6 +692,13 @@ class CNXNLIBertPipe(MatchingBertPipe): class TruncateBertPipe(Pipe): + """ + 对数据进行截断的 **Pipe** 。该 **Pipe** 将会寻找每条数据中的第一个分隔符 ``[SEP]`` ，对其前后的数据分别进行截断。 + 对于中文任务会将前后的文本分别截断至长度 **250** ，对于英文任务会分别截断至 **215** 。 + + :param task: 任务类型，可选 ``['cn', 'en']`` ，分别表示 **中文任务** 和 **英文任务** 。 + :param num_proc: 处理数据时使用的进程数目。 + """ def __init__(self, task='cn', num_proc=0): super().__init__() self.task = task @@ -522,6 +723,10 @@ class TruncateBertPipe(Pipe): return words_before_sep + words_after_sep def process(self, data_bundle: DataBundle) -> DataBundle: + """ + :param data_bundle: + :return: 处理后的 ``data_bundle`` + """ for name in data_bundle.datasets.keys(): dataset = data_bundle.get_dataset(name) sep_index_vocab = data_bundle.get_vocab('words').to_index('[SEP]') diff --git a/fastNLP/io/pipe/pipe.py b/fastNLP/io/pipe/pipe.py index 4916bf09..0ee43ae8 100644 --- a/fastNLP/io/pipe/pipe.py +++ b/fastNLP/io/pipe/pipe.py @@ -1,5 +1,3 @@ -r"""undocumented""" - __all__ = [ "Pipe", ] @@ -9,33 +7,38 @@ from fastNLP.io.data_bundle import DataBundle class Pipe: r""" - Pipe是fastNLP中用于处理DataBundle的类，但实际是处理DataBundle中的DataSet。所有Pipe都会在其process()函数的文档中指出该Pipe可处理的DataSet应该具备怎样的格式；在Pipe - 文档中说明该Pipe返回后DataSet的格式以及其field的信息；以及新增的Vocabulary的信息。 + :class:`Pipe` 是 **fastNLP** 中用于处理 :class:`~fastNLP.io.DataBundle` 的类，但实际是处理其中的 :class:`~fastNLP.core.DataSet` 。 + 所有 ``Pipe`` 都会在其 :meth:`process` 函数的文档中指出该 ``Pipe`` 可处理的 :class:`~fastNLP.core.DataSet` 应该具备怎样的格式；在 + ``Pipe`` 文档中说明该 ``Pipe`` 返回后 :class:`~fastNLP.core.DataSet` 的格式以及其 field 的信息；以及新增的 :class:`~fastNLP.core.Vocabulary` + 的信息。 - 一般情况下Pipe处理包含以下的几个过程，(1)将raw_words或raw_chars进行tokenize以切分成不同的词或字; - (2) 再建立词或字的 :class:`~fastNLP.Vocabulary` , 并将词或字转换为index; (3)将target列建立词表并将target列转为index; + 一般情况下 **Pipe** 处理包含以下的几个过程： + + 1. 将 ``raw_words`` 或 ``raw_chars`` 进行 tokenize 以切分成不同的词或字； + 2. 建立词或字的 :class:`~fastNLP.core.Vocabulary` ，并将词或字转换为 index； + 3. 将 ``target`` 列建立词表并将 ``target`` 列转为 index； - Pipe中提供了两个方法 + **Pipe** 中提供了两个方法： - -process()函数，输入为DataBundle - -process_from_file()函数，输入为对应Loader的load函数可接受的类型。 + - :meth:`process` 函数，输入为 :class:`~fastNLP.io.DataBundle` + - :meth:`process_from_file` 函数，输入为对应 :meth:`fastNLP.io.Loader.load` 函数可接受的类型。 """ def process(self, data_bundle: DataBundle) -> DataBundle: r""" - 对输入的DataBundle进行处理，然后返回该DataBundle。 + 对输入的 ``data_bundle`` 进行处理，然后返回该 ``data_bundle`` - :param ~fastNLP.DataBundle data_bundle: 需要处理的DataBundle对象 - :return: DataBundle + :param data_bundle: + :return: 处理后的 ``data_bundle`` """ raise NotImplementedError def process_from_file(self, paths: str) -> DataBundle: r""" - 传入文件路径，生成处理好的DataBundle对象。paths支持的路径形式可以参考 ：:meth:`fastNLP.io.Loader.load()` + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` - :param str paths: - :return: DataBundle + :param paths: + :return: """ raise NotImplementedError diff --git a/fastNLP/io/pipe/qa.py b/fastNLP/io/pipe/qa.py index 23fe1367..0d646263 100644 --- a/fastNLP/io/pipe/qa.py +++ b/fastNLP/io/pipe/qa.py @@ -1,5 +1,5 @@ r""" -本文件中的Pipe主要用于处理问答任务的数据。 +本文件中的 **Pipe** 主要用于处理问答任务的数据。 """ @@ -78,32 +78,20 @@ def _concat_clip(data_bundle, max_len, concat_field_name='raw_chars'): class CMRC2018BertPipe(Pipe): r""" - 处理之后的DataSet将新增以下的field(传入的field仍然保留) + 处理 **CMRC2018** 的数据，处理之后 :class:`~fastNLP.core.DataSet` 中新增的内容如下（原有的 field 仍然保留）： .. csv-table:: :header: "context_len", "raw_chars", "target_start", "target_end", "chars" - 492, ['范', '廷', '颂... ], 30, 34, "[21, 25, ...]" - 491, ['范', '廷', '颂... ], 41, 61, "[21, 25, ...]" - + 492, "['范', '廷', '颂... ]", 30, 34, "[21, 25, ...]" + 491, "['范', '廷', '颂... ]", 41, 61, "[21, 25, ...]" ".", "...", "...","...", "..." - raw_words列是context与question拼起来的结果(连接的地方加入了[SEP])，words是转为index的值, target_start为答案start的index，target_end为答案end的index - （闭区间）；context_len指示的是words列中context的长度。 - - 其中各列的meta信息如下: - - .. code:: - - +-------------+-------------+-----------+--------------+------------+-------+---------+ - | field_names | context_len | raw_chars | target_start | target_end | chars | answers | - +-------------+-------------+-----------+--------------+------------+-------+---------| - | is_input | False | False | False | False | True | False | - | is_target | True | True | True | True | False | True | - | ignore_type | False | True | False | False | False | True | - | pad_value | 0 | 0 | 0 | 0 | 0 | 0 | - +-------------+-------------+-----------+--------------+------------+-------+---------+ - + ``raw_chars`` 列是 ``context`` 与 ``question`` 拼起来的结果（连接的地方加入了 ``[SEP]`` ）， ``chars`` 是转为 + index 的值， ``target_start`` 为答案开始的位置， ``target_end`` 为答案结束的位置（闭区间）； ``context_len`` + 指示的是 ``chars`` 列中 context 的长度。 + + :param max_len: """ def __init__(self, max_len=510): @@ -112,17 +100,17 @@ class CMRC2018BertPipe(Pipe): def process(self, data_bundle: DataBundle) -> DataBundle: r""" - 传入的DataSet应该具备以下的field + ``data_bunlde`` 中的 :class:`~fastNLP.core.DataSet` 应该包含 ``raw_words`` ： .. csv-table:: - :header:"title", "context", "question", "answers", "answer_starts", "id" + :header: "title", "context", "question", "answers", "answer_starts", "id" "范廷颂", "范廷颂枢机（，），圣名保禄·若瑟（）...", "范廷颂是什么时候被任为主教的？", ["1963年"], ["30"], "TRAIN_186_QUERY_0" "范廷颂", "范廷颂枢机（，），圣名保禄·若瑟（）...", "1990年，范廷颂担任什么职务？", ["1990年被擢升为天..."], ["41"],"TRAIN_186_QUERY_1" "...", "...", "...","...", ".", "..." :param data_bundle: - :return: + :return: 处理后的 ``data_bundle`` """ data_bundle = _concat_clip(data_bundle, max_len=self.max_len, concat_field_name='raw_chars') @@ -138,5 +126,11 @@ class CMRC2018BertPipe(Pipe): return data_bundle def process_from_file(self, paths=None) -> DataBundle: + r""" + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param paths: + :return: + """ data_bundle = CMRC2018Loader().load(paths) return self.process(data_bundle) diff --git a/fastNLP/io/pipe/summarization.py b/fastNLP/io/pipe/summarization.py index b413890b..dc27651c 100644 --- a/fastNLP/io/pipe/summarization.py +++ b/fastNLP/io/pipe/summarization.py @@ -1,4 +1,3 @@ -r"""undocumented""" import os import numpy as np from functools import partial @@ -20,21 +19,23 @@ TAG_UNK = "X" class ExtCNNDMPipe(Pipe): r""" - 对CNN/Daily Mail数据进行适用于extractive summarization task的预处理，预处理之后的数据，具备以下结构： + 对 **CNN/Daily Mail** 数据进行适用于 ``extractive summarization task`` 的预处理，预处理之后的数据具备以下结构： .. csv-table:: :header: "text", "summary", "label", "publication", "text_wd", "words", "seq_len", "target" - + + "['I got new tires from them and... ','...']", "['The new tires...','...']", "[0, 1]", "cnndm", "[['I','got',...'.'],...,['...']]", "[[54,89,...,5],...,[9,43,..,0]]", "[1,1,...,0]", "[0,1,...,0]" + "['Don't waste your time. We had two...','...']", "['Time is precious','...']", "[1]", "cnndm", "[['Don't','waste',...,'.'],...,['...']]", "[[5234,653,...,5],...,[87,234,..,0]]", "[1,1,...,0]", "[1,1,...,0]" + "['...']", "['...']", "[]", "cnndm", "[[''],...,['']]", "[[],...,[]]", "[]", "[]" + + :param vocab_size: 词表大小 + :param sent_max_len: 句子最大长度，不足的句子将 padding ，超出的将截断 + :param doc_max_timesteps: 文章最多句子个数，不足的将 padding，超出的将截断 + :param vocab_path: 外部词表路径 + :param domain: 是否需要建立 domain 词表 + :param num_proc: 处理数据时使用的进程数目。 """ - def __init__(self, vocab_size, sent_max_len, doc_max_timesteps, vocab_path=None, domain=False, num_proc=0): - r""" - - :param vocab_size: int, 词表大小 - :param sent_max_len: int, 句子最大长度，不足的句子将padding，超出的将截断 - :param doc_max_timesteps: int, 文章最多句子个数，不足的将padding，超出的将截断 - :param vocab_path: str, 外部词表路径 - :param domain: bool, 是否需要建立domain词表 - """ + def __init__(self, vocab_size: int, sent_max_len: int, doc_max_timesteps: int, vocab_path=None, domain=False, num_proc=0): self.vocab_size = vocab_size self.vocab_path = vocab_path self.sent_max_len = sent_max_len @@ -44,23 +45,24 @@ class ExtCNNDMPipe(Pipe): def process(self, data_bundle: DataBundle): r""" - 传入的DataSet应该具备如下的结构 + ``data_bunlde`` 中的 :class:`~fastNLP.core.DataSet` 应该具备以下结构： .. csv-table:: :header: "text", "summary", "label", "publication" - ["I got new tires from them and... ","..."], ["The new tires...","..."], [0, 1], "cnndm" - ["Don't waste your time. We had two...","..."], ["Time is precious","..."], [1], "cnndm" - ["..."], ["..."], [], "cnndm" + "['I got new tires from them and... ','...']", "['The new tires...','...']", "[0, 1]", "cnndm" + "['Don't waste your time. We had two...','...']", "['Time is precious','...']", "[1]", "cnndm" + "['...']", ['...']", "[]", "cnndm" :param data_bundle: - :return: 处理得到的数据包括 + :return: 处理后的 ``data_bundle``，新增以下列： + .. csv-table:: :header: "text_wd", "words", "seq_len", "target" - [["I","got",..."."],...,["..."]], [[54,89,...,5],...,[9,43,..,0]], [1,1,...,0], [0,1,...,0] - [["Don't","waste",...,"."],...,["..."]], [[5234,653,...,5],...,[87,234,..,0]], [1,1,...,0], [1,1,...,0] - [[""],...,[""]], [[],...,[]], [], [] + "[['I','got',...'.'],...,['...']]", "[[54,89,...,5],...,[9,43,..,0]]", "[1,1,...,0]", "[0,1,...,0]" + "[['Don't','waste',...,'.'],...,['...']]", "[[5234,653,...,5],...,[87,234,..,0]]", "[1,1,...,0]", "[1,1,...,0]" + "[[''],...,['']]", "[[],...,[]]", "[]", "[]" """ if self.vocab_path is None: @@ -117,8 +119,10 @@ class ExtCNNDMPipe(Pipe): def process_from_file(self, paths=None): r""" - :param paths: dict or string - :return: DataBundle + 传入文件路径，生成处理好的 :class:`~fastNLP.io.DataBundle` 对象。``paths`` 支持的路径形式可以参考 :meth:`fastNLP.io.Loader.load` + + :param paths: + :return: """ loader = ExtCNNDMLoader() if self.vocab_path is None: diff --git a/fastNLP/io/pipe/utils.py b/fastNLP/io/pipe/utils.py index c5c32d95..3f1b8563 100644 --- a/fastNLP/io/pipe/utils.py +++ b/fastNLP/io/pipe/utils.py @@ -1,5 +1,3 @@ -r"""undocumented""" - __all__ = [ "iob2", "iob2bioes", @@ -17,10 +15,10 @@ from pkg_resources import parse_version def iob2(tags: List[str]) -> List[str]: r""" - 检查数据是否是合法的IOB数据，如果是IOB1会被自动转换为IOB2。两种格式的区别见 + 检查数据是否是合法的 ``IOB`` 数据，如果是 ``IOB1`` 会被自动转换为 ``IOB2`` 。两种格式的区别见 https://datascience.stackexchange.com/questions/37824/difference-between-iob-and-iob2-format - :param tags: 需要转换的tags + :param tags: 需要转换的 tags """ for i, tag in enumerate(tags): if tag == "O": @@ -41,8 +39,9 @@ def iob2(tags: List[str]) -> List[str]: def iob2bioes(tags: List[str]) -> List[str]: r""" - 将iob的tag转换为bioes编码 - :param tags: + 将 ``iob`` 的 tag 转换为 ``bioes`` 编码 + + :param tags: 需要转换的 tags :return: """ new_tags = [] @@ -69,9 +68,10 @@ def iob2bioes(tags: List[str]) -> List[str]: def get_tokenizer(tokenize_method: str, lang='en'): r""" - :param str tokenize_method: 获取tokenzier方法 - :param str lang: 语言，当前仅支持en - :return: tokenize函数 + :param tokenize_method: 获取 tokenzier 方法，支持 ``['spacy', 'raw', 'cn-char']`` 。``'raw'`` 表示使用空格作为切分， ``'cn-char'`` 表示 + 按字符切分，``'spacy'`` 则使用 :mod:`spacy` 库进行分词。 + :param lang: :mod:`spacy` 使用的语言，当前仅支持 ``'en'`` 。 + :return: tokenize 函数 """ tokenizer_dict = { 'spacy': None, @@ -82,7 +82,7 @@ def get_tokenizer(tokenize_method: str, lang='en'): import spacy spacy.prefer_gpu() if lang != 'en': - raise RuntimeError("Spacy only supports en right right.") + raise RuntimeError("Spacy only supports en right now.") if parse_version(spacy.__version__) >= parse_version('3.0'): en = spacy.load('en_core_web_sm') else: diff --git a/fastNLP/io/utils.py b/fastNLP/io/utils.py index 5c0b63ce..a3af71f5 100644 --- a/fastNLP/io/utils.py +++ b/fastNLP/io/utils.py @@ -1,8 +1,3 @@ -r""" -.. todo:: - doc -""" - __all__ = [ "check_loader_paths" ] @@ -16,7 +11,7 @@ from typing import Union, Dict def check_loader_paths(paths: Union[str, Dict[str, str]]) -> Dict[str, str]: r""" - 检查传入dataloader的文件的合法性。如果为合法路径，将返回至少包含'train'这个key的dict。类似于下面的结果:: + 检查传入 ``dataloader`` 的文件的合法性。如果为合法路径，将返回至少包含 ``'train'`` 这个 key 的字典。类似于下面的结果:: { 'train': '/some/path/to/', # 一定包含，建词表应该在这上面建立，剩下的其它文件应该只需要处理并index。 @@ -24,10 +19,13 @@ def check_loader_paths(paths: Union[str, Dict[str, str]]) -> Dict[str, str]: ... } - 如果paths为不合法的，将直接进行raise相应的错误. 如果paths内不包含train也会报错。 + 如果 ``paths`` 为不合法的，将直接进行 raise 相应的错误。如果 ``paths`` 内不包含 ``'train'`` 也会报错。 - :param str paths: 路径. 可以为一个文件路径(则认为该文件就是train的文件); 可以为一个文件目录，将在该目录下寻找包含train(文件名 - 中包含train这个字段), test, dev这三个字段的文件或文件夹; 可以为一个dict, 则key是用户自定义的某个文件的名称，value是这个文件的路径。 + :param str paths: 路径。可以为： + + - 一个文件路径，此时认为该文件就是 train 的文件； + - 一个文件目录，将在该目录下寻找包含 ``train`` （文件名中包含 train 这个字段）， ``test`` ，``dev`` 这三个字段的文件或文件夹； + - 一个 dict, 则 key 是用户自定义的某个文件的名称，value 是这个文件的路径。 :return: """ if isinstance(paths, (str, Path)): diff --git a/fastNLP/transformers/__init__.py b/fastNLP/transformers/__init__.py index 6403f6b9..6b175b28 100644 --- a/fastNLP/transformers/__init__.py +++ b/fastNLP/transformers/__init__.py @@ -1 +1,3 @@ -"""基于 transformers-4.11.3 版本迁移""" \ No newline at end of file +""" +:mod:`transformers` 模块，包含了常用的预训练模型。 +""" diff --git a/fastNLP/transformers/torch/__init__.py b/fastNLP/transformers/torch/__init__.py index 9ce4fb10..3b564cd4 100644 --- a/fastNLP/transformers/torch/__init__.py +++ b/fastNLP/transformers/torch/__init__.py @@ -1,9 +1,15 @@ """ -为了防止因 https://github.com/huggingface/transformers 版本变化导致代码不兼容，当前 folder 以及子 folder -都复制自 https://github.com/huggingface/transformers 的4.11.3版本。 -In order to avoid the code change of https://github.com/huggingface/transformers to cause version -mismatch, we copy code from https://github.com/huggingface/transformers(version:4.11.3) in this +为了防止因 `transformers