to load 'ernie-1.0-base-zh'.\u001b[0m\n",
+ "\u001b[32m[2022-06-22 21:31:15,580] [ INFO]\u001b[0m - Already cached /remote-home/shxing/.paddlenlp/models/ernie-1.0-base-zh/ernie_v1_chn_base.pdparams\u001b[0m\n"
+ ]
+ }
+ ],
+ "source": [
+ "import paddle.nn as nn\n",
+ "\n",
+ "class SeqClsModel(nn.Layer):\n",
+ " def __init__(self, model_checkpoint, num_labels):\n",
+ " super(SeqClsModel, self).__init__()\n",
+ " self.model = AutoModelForSequenceClassification.from_pretrained(\n",
+ " model_checkpoint,\n",
+ " num_classes=num_labels,\n",
+ " )\n",
+ "\n",
+ " def forward(self, input_ids, attention_mask, token_type_ids):\n",
+ " logits = self.model(input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids)\n",
+ " return logits\n",
+ "\n",
+ " def train_step(self, input_ids, attention_mask, token_type_ids, label):\n",
+ " logits = self(input_ids, attention_mask, token_type_ids)\n",
+ " loss = nn.CrossEntropyLoss()(logits, label)\n",
+ " return {\"loss\": loss}\n",
+ "\n",
+ " def evaluate_step(self, input_ids, attention_mask, token_type_ids, label):\n",
+ " logits = self(input_ids, attention_mask, token_type_ids)\n",
+ " return {'pred': logits, 'target': label}\n",
+ "\n",
+ "model = SeqClsModel(model_checkpoint, num_labels=2)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### 3.2 设置参数并使用 Trainer 开始训练\n",
+ "\n",
+ "现在我们可以着手使用 ``FastNLP.Trainer`` 进行训练了。\n",
+ "\n",
+ "首先，为了高效地训练 ``ERNIE`` 模型，我们最好为学习率指定一定的策略。``paddlenlp`` 提供的 ``LinearDecayWithWarmup`` 可以令学习率在一段时间内从 0 开始线性地增长（预热），然后再线性地衰减至 0 。在本篇教程中，我们将学习率设置为 ``5e-5``，预热时间为 ``0.1``，然后将得到的的 ``lr_scheduler`` 赋值给 ``AdamW`` 优化器。\n",
+ "\n",
+ "其次，我们还可以为 ``Trainer`` 指定多个 ``Callback`` 来在基础的训练过程之外进行额外的定制操作。在本篇教程中，我们使用的 ``Callback`` 有以下三种：\n",
+ "\n",
+ "- ``RichCallback`` - 在训练和验证时显示进度条，以便观察训练的过程\n",
+ "- ``LRSchedCallback`` - 由于我们使用了 ``Scheduler``，因此需要将 ``lr_scheduler`` 传给该 ``Callback`` 以在训练中进行更新\n",
+ "- ``LoadBestModelCallback`` - 该 ``Callback`` 会评估结果中的 ``'acc#accuracy'`` 值，保存训练中出现的正确率最高的模型，并在训练结束时加载到模型上，方便对模型进行测试和评估。\n",
+ "\n",
+ "在 ``Trainer`` 中，我们还可以设置 ``metrics`` 来衡量模型的表现。``Accuracy`` 能够根据传入的预测值和真实值计算出模型预测的正确率。还记得模型中 ``evaluate_step`` 函数的返回值吗？键 ``pred`` 和 ``target`` 分别为 ``Accuracy.update`` 的参数名，在验证过程中 ``FastNLP`` 会自动将键和参数名匹配从而计算出正确率，这也是我们规定模型需要返回字典类型数据的原因。\n",
+ "\n",
+ "``Accuracy`` 的返回值包含三个部分：``acc``、``total`` 和 ``correct``，分别代表 ``正确率``、 ``数据总数`` 和 ``预测正确的数目``，这让您能够直观地知晓训练中模型的变化，``LoadBestModelCallback`` 的参数 ``'acc#accuracy'`` 也正是代表了 ``accuracy`` 指标的 ``acc`` 结果。\n",
+ "\n",
+ "在设定好参数之后，调用 ``run`` 函数便可以进行训练和验证了。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "[21:31:16] INFO Running evaluator sanity check for 2 batches. trainer.py:631\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "\u001b[2;36m[21:31:16]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m Running evaluator sanity check for \u001b[1;36m2\u001b[0m batches. \u001b]8;id=4641;file://../fastNLP/core/controllers/trainer.py\u001b\\\u001b[2mtrainer.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=822054;file://../fastNLP/core/controllers/trainer.py#631\u001b\\\u001b[2m631\u001b[0m\u001b]8;;\u001b\\\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n"
+ ],
+ "text/plain": []
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "---------------------------- Eval. results on Epoch:0, Batch:60 -----------------------------\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "---------------------------- Eval. results on Epoch:\u001b[1;36m0\u001b[0m, Batch:\u001b[1;36m60\u001b[0m -----------------------------\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "{\n",
+ " \"acc#accuracy\": 0.895833,\n",
+ " \"total#accuracy\": 1200.0,\n",
+ " \"correct#accuracy\": 1075.0\n",
+ "}\n",
+ "\n"
+ ],
+ "text/plain": [
+ "\u001b[1m{\u001b[0m\n",
+ " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.895833\u001b[0m,\n",
+ " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n",
+ " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1075.0\u001b[0m\n",
+ "\u001b[1m}\u001b[0m\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "---------------------------- Eval. results on Epoch:0, Batch:120 ----------------------------\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "---------------------------- Eval. results on Epoch:\u001b[1;36m0\u001b[0m, Batch:\u001b[1;36m120\u001b[0m ----------------------------\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "{\n",
+ " \"acc#accuracy\": 0.8975,\n",
+ " \"total#accuracy\": 1200.0,\n",
+ " \"correct#accuracy\": 1077.0\n",
+ "}\n",
+ "\n"
+ ],
+ "text/plain": [
+ "\u001b[1m{\u001b[0m\n",
+ " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.8975\u001b[0m,\n",
+ " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n",
+ " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1077.0\u001b[0m\n",
+ "\u001b[1m}\u001b[0m\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "---------------------------- Eval. results on Epoch:0, Batch:180 ----------------------------\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "---------------------------- Eval. results on Epoch:\u001b[1;36m0\u001b[0m, Batch:\u001b[1;36m180\u001b[0m ----------------------------\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "{\n",
+ " \"acc#accuracy\": 0.911667,\n",
+ " \"total#accuracy\": 1200.0,\n",
+ " \"correct#accuracy\": 1094.0\n",
+ "}\n",
+ "\n"
+ ],
+ "text/plain": [
+ "\u001b[1m{\u001b[0m\n",
+ " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.911667\u001b[0m,\n",
+ " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n",
+ " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1094.0\u001b[0m\n",
+ "\u001b[1m}\u001b[0m\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "---------------------------- Eval. results on Epoch:0, Batch:240 ----------------------------\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "---------------------------- Eval. results on Epoch:\u001b[1;36m0\u001b[0m, Batch:\u001b[1;36m240\u001b[0m ----------------------------\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "{\n",
+ " \"acc#accuracy\": 0.9225,\n",
+ " \"total#accuracy\": 1200.0,\n",
+ " \"correct#accuracy\": 1107.0\n",
+ "}\n",
+ "\n"
+ ],
+ "text/plain": [
+ "\u001b[1m{\u001b[0m\n",
+ " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.9225\u001b[0m,\n",
+ " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n",
+ " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1107.0\u001b[0m\n",
+ "\u001b[1m}\u001b[0m\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "---------------------------- Eval. results on Epoch:0, Batch:300 ----------------------------\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "---------------------------- Eval. results on Epoch:\u001b[1;36m0\u001b[0m, Batch:\u001b[1;36m300\u001b[0m ----------------------------\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "{\n",
+ " \"acc#accuracy\": 0.9275,\n",
+ " \"total#accuracy\": 1200.0,\n",
+ " \"correct#accuracy\": 1113.0\n",
+ "}\n",
+ "\n"
+ ],
+ "text/plain": [
+ "\u001b[1m{\u001b[0m\n",
+ " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.9275\u001b[0m,\n",
+ " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n",
+ " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1113.0\u001b[0m\n",
+ "\u001b[1m}\u001b[0m\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "---------------------------- Eval. results on Epoch:1, Batch:60 -----------------------------\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "---------------------------- Eval. results on Epoch:\u001b[1;36m1\u001b[0m, Batch:\u001b[1;36m60\u001b[0m -----------------------------\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "{\n",
+ " \"acc#accuracy\": 0.930833,\n",
+ " \"total#accuracy\": 1200.0,\n",
+ " \"correct#accuracy\": 1117.0\n",
+ "}\n",
+ "\n"
+ ],
+ "text/plain": [
+ "\u001b[1m{\u001b[0m\n",
+ " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.930833\u001b[0m,\n",
+ " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n",
+ " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1117.0\u001b[0m\n",
+ "\u001b[1m}\u001b[0m\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "---------------------------- Eval. results on Epoch:1, Batch:120 ----------------------------\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "---------------------------- Eval. results on Epoch:\u001b[1;36m1\u001b[0m, Batch:\u001b[1;36m120\u001b[0m ----------------------------\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "{\n",
+ " \"acc#accuracy\": 0.935833,\n",
+ " \"total#accuracy\": 1200.0,\n",
+ " \"correct#accuracy\": 1123.0\n",
+ "}\n",
+ "\n"
+ ],
+ "text/plain": [
+ "\u001b[1m{\u001b[0m\n",
+ " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.935833\u001b[0m,\n",
+ " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n",
+ " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1123.0\u001b[0m\n",
+ "\u001b[1m}\u001b[0m\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "---------------------------- Eval. results on Epoch:1, Batch:180 ----------------------------\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "---------------------------- Eval. results on Epoch:\u001b[1;36m1\u001b[0m, Batch:\u001b[1;36m180\u001b[0m ----------------------------\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "{\n",
+ " \"acc#accuracy\": 0.935833,\n",
+ " \"total#accuracy\": 1200.0,\n",
+ " \"correct#accuracy\": 1123.0\n",
+ "}\n",
+ "\n"
+ ],
+ "text/plain": [
+ "\u001b[1m{\u001b[0m\n",
+ " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.935833\u001b[0m,\n",
+ " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n",
+ " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1123.0\u001b[0m\n",
+ "\u001b[1m}\u001b[0m\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "---------------------------- Eval. results on Epoch:1, Batch:240 ----------------------------\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "---------------------------- Eval. results on Epoch:\u001b[1;36m1\u001b[0m, Batch:\u001b[1;36m240\u001b[0m ----------------------------\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "{\n",
+ " \"acc#accuracy\": 0.9375,\n",
+ " \"total#accuracy\": 1200.0,\n",
+ " \"correct#accuracy\": 1125.0\n",
+ "}\n",
+ "\n"
+ ],
+ "text/plain": [
+ "\u001b[1m{\u001b[0m\n",
+ " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.9375\u001b[0m,\n",
+ " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n",
+ " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1125.0\u001b[0m\n",
+ "\u001b[1m}\u001b[0m\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "---------------------------- Eval. results on Epoch:1, Batch:300 ----------------------------\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "---------------------------- Eval. results on Epoch:\u001b[1;36m1\u001b[0m, Batch:\u001b[1;36m300\u001b[0m ----------------------------\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "{\n",
+ " \"acc#accuracy\": 0.941667,\n",
+ " \"total#accuracy\": 1200.0,\n",
+ " \"correct#accuracy\": 1130.0\n",
+ "}\n",
+ "\n"
+ ],
+ "text/plain": [
+ "\u001b[1m{\u001b[0m\n",
+ " \u001b[1;34m\"acc#accuracy\"\u001b[0m: \u001b[1;36m0.941667\u001b[0m,\n",
+ " \u001b[1;34m\"total#accuracy\"\u001b[0m: \u001b[1;36m1200.0\u001b[0m,\n",
+ " \u001b[1;34m\"correct#accuracy\"\u001b[0m: \u001b[1;36m1130.0\u001b[0m\n",
+ "\u001b[1m}\u001b[0m\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "[21:34:28] INFO Loading best model from fnlp-ernie/2022-0 load_best_model_callback.py:111\n",
+ " 6-22-21_29_12_898095/best_so_far with \n",
+ " acc#accuracy: 0.941667... \n",
+ "
\n"
+ ],
+ "text/plain": [
+ "\u001b[2;36m[21:34:28]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m Loading best model from fnlp-ernie/\u001b[1;36m2022\u001b[0m-\u001b[1;36m0\u001b[0m \u001b]8;id=340364;file://../fastNLP/core/callbacks/load_best_model_callback.py\u001b\\\u001b[2mload_best_model_callback.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=763898;file://../fastNLP/core/callbacks/load_best_model_callback.py#111\u001b\\\u001b[2m111\u001b[0m\u001b]8;;\u001b\\\n",
+ "\u001b[2;36m \u001b[0m \u001b[1;36m6\u001b[0m-\u001b[1;36m22\u001b[0m-21_29_12_898095/best_so_far with \u001b[2m \u001b[0m\n",
+ "\u001b[2;36m \u001b[0m acc#accuracy: \u001b[1;36m0.941667\u001b[0m\u001b[33m...\u001b[0m \u001b[2m \u001b[0m\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "[21:34:34] INFO Deleting fnlp-ernie/2022-06-22-21_29_12_8 load_best_model_callback.py:131\n",
+ " 98095/best_so_far... \n",
+ "
\n"
+ ],
+ "text/plain": [
+ "\u001b[2;36m[21:34:34]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m Deleting fnlp-ernie/\u001b[1;36m2022\u001b[0m-\u001b[1;36m06\u001b[0m-\u001b[1;36m22\u001b[0m-21_29_12_8 \u001b]8;id=430330;file://../fastNLP/core/callbacks/load_best_model_callback.py\u001b\\\u001b[2mload_best_model_callback.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=508566;file://../fastNLP/core/callbacks/load_best_model_callback.py#131\u001b\\\u001b[2m131\u001b[0m\u001b]8;;\u001b\\\n",
+ "\u001b[2;36m \u001b[0m 98095/best_so_far\u001b[33m...\u001b[0m \u001b[2m \u001b[0m\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n"
+ ],
+ "text/plain": []
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "from fastNLP import LRSchedCallback, RichCallback, LoadBestModelCallback\n",
+ "from fastNLP import Trainer, Accuracy\n",
+ "from paddlenlp.transformers import LinearDecayWithWarmup\n",
+ "\n",
+ "n_epochs = 2\n",
+ "num_training_steps = len(train_dataloader) * n_epochs\n",
+ "lr_scheduler = LinearDecayWithWarmup(5e-5, num_training_steps, 0.1)\n",
+ "optimizer = paddle.optimizer.AdamW(\n",
+ " learning_rate=lr_scheduler,\n",
+ " parameters=model.parameters(),\n",
+ ")\n",
+ "callbacks = [\n",
+ " LRSchedCallback(lr_scheduler, step_on=\"batch\"),\n",
+ " LoadBestModelCallback(\"acc#accuracy\", larger_better=True, save_folder=\"fnlp-ernie\"),\n",
+ " RichCallback()\n",
+ "]\n",
+ "trainer = Trainer(\n",
+ " model=model,\n",
+ " driver=\"paddle\",\n",
+ " optimizers=optimizer,\n",
+ " device=0,\n",
+ " n_epochs=n_epochs,\n",
+ " train_dataloader=train_dataloader,\n",
+ " evaluate_dataloaders=val_dataloader,\n",
+ " evaluate_every=60,\n",
+ " metrics={\"accuracy\": Accuracy()},\n",
+ " callbacks=callbacks,\n",
+ ")\n",
+ "trainer.run()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### 3.3 测试和评估\n",
+ "\n",
+ "现在我们已经得到了一个表现良好的 ``ERNIE`` 模型，接下来可以在测试集上测试模型的效果了。``FastNLP.Evaluator`` 提供了定制函数的功能。我们以 ``test_dataloader`` 初始化一个 ``Evaluator``，然后将写好的测试函数 ``test_batch_step_fn`` 传给参数 ``evaluate_batch_step_fn``，``Evaluate`` 在对每个 batch 进行评估时就会调用我们自定义的 ``test_batch_step_fn`` 函数而不是 ``evaluate_step`` 函数。在这里，我们仅测试 5 条数据并输出文本和对应的标签。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "text: ['这个宾馆比较陈旧了，特价的房间也很一般。总体来说一般']\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "text: ['这个宾馆比较陈旧了，特价的房间也很一般。总体来说一般']\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "labels: 0\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "labels: 0\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "text: ['怀着十分激动的心情放映，可是看着看着发现，在放映完毕后，出现一集米老鼠的动画片！开始\n",
+ "还怀疑是不是赠送的个别现象，可是后来发现每张DVD后面都有！真不知道生产商怎么想的，我想看的是猫\n",
+ "和老鼠，不是米老鼠！如果厂家是想赠送的话，那就全套米老鼠和唐老鸭都赠送，只在每张DVD后面添加一\n",
+ "集算什么？？简直是画蛇添足！！']\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "text: ['怀着十分激动的心情放映，可是看着看着发现，在放映完毕后，出现一集米老鼠的动画片！开始\n",
+ "还怀疑是不是赠送的个别现象，可是后来发现每张DVD后面都有！真不知道生产商怎么想的，我想看的是猫\n",
+ "和老鼠，不是米老鼠！如果厂家是想赠送的话，那就全套米老鼠和唐老鸭都赠送，只在每张DVD后面添加一\n",
+ "集算什么？？简直是画蛇添足！！']\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "labels: 0\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "labels: 0\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "text: ['还稍微重了点，可能是硬盘大的原故，还要再轻半斤就好了。其他要进一步验证。贴的几种膜气\n",
+ "泡较多，用不了多久就要更换了，屏幕膜稍好点，但比没有要强多了。建议配赠几张膜让用用户自己贴。'\n",
+ "]\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "text: ['还稍微重了点，可能是硬盘大的原故，还要再轻半斤就好了。其他要进一步验证。贴的几种膜气\n",
+ "泡较多，用不了多久就要更换了，屏幕膜稍好点，但比没有要强多了。建议配赠几张膜让用用户自己贴。'\n",
+ "]\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "labels: 0\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "labels: 0\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "text: ['交通方便；环境很好；服务态度很好 房间较小']\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "text: ['交通方便；环境很好；服务态度很好 房间较小']\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "labels: 1\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "labels: 1\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "text: ['不错，作者的观点很颠覆目前中国父母的教育方式，其实古人们对于教育已经有了很系统的体系\n",
+ "了，可是现在的父母以及祖父母们更多的娇惯纵容孩子，放眼看去自私的孩子是大多数，父母觉得自己的\n",
+ "孩子在外面只要不吃亏就是好事，完全把古人几千年总结的教育古训抛在的九霄云外。所以推荐准妈妈们\n",
+ "可以在等待宝宝降临的时候，好好学习一下，怎么把孩子教育成一个有爱心、有责任心、宽容、大度的人\n",
+ "。']\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "text: ['不错，作者的观点很颠覆目前中国父母的教育方式，其实古人们对于教育已经有了很系统的体系\n",
+ "了，可是现在的父母以及祖父母们更多的娇惯纵容孩子，放眼看去自私的孩子是大多数，父母觉得自己的\n",
+ "孩子在外面只要不吃亏就是好事，完全把古人几千年总结的教育古训抛在的九霄云外。所以推荐准妈妈们\n",
+ "可以在等待宝宝降临的时候，好好学习一下，怎么把孩子教育成一个有爱心、有责任心、宽容、大度的人\n",
+ "。']\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "labels: 1\n",
+ "
\n"
+ ],
+ "text/plain": [
+ "labels: 1\n"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n"
+ ],
+ "text/plain": []
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "{}"
+ ]
+ },
+ "execution_count": 14,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from fastNLP import Evaluator\n",
+ "def test_batch_step_fn(evaluator, batch):\n",
+ " input_ids = batch[\"input_ids\"]\n",
+ " attention_mask = batch[\"attention_mask\"]\n",
+ " token_type_ids = batch[\"token_type_ids\"]\n",
+ " logits = model(input_ids, attention_mask, token_type_ids)\n",
+ " predict = logits.argmax().item()\n",
+ " print(\"text:\", batch['text'])\n",
+ " print(\"labels:\", predict)\n",
+ "\n",
+ "evaluator = Evaluator(\n",
+ " model=model,\n",
+ " dataloaders=test_dataloader,\n",
+ " driver=\"paddle\",\n",
+ " device=0,\n",
+ " evaluate_batch_step_fn=test_batch_step_fn,\n",
+ ")\n",
+ "evaluator.run(5) "
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3.7.13 ('fnlp-paddle')",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.7.13"
+ },
+ "orig_nbformat": 4,
+ "vscode": {
+ "interpreter": {
+ "hash": "31f2d9d3efc23c441973d7c4273acfea8b132b6a578f002629b6b44b8f65e720"
+ }
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/tutorials/figures/paddle-ernie-1.0-masking-levels.png b/tutorials/figures/paddle-ernie-1.0-masking-levels.png
new file mode 100644
index 0000000000000000000000000000000000000000..ff2519c4bc2c6b3ce4c63a7612fcc1e11f9cf4f8
GIT binary patch
literal 59022
zcmZ77b8sd>yD0G3w)w`k%{O+kv9U=uwr$(i#@HmA4c^$c?PO!??sx7vRp;LMqpPc@
zr)%n|>8_fme;uWwB!i4VfB*&th76RIR09KpO#4@_g@gH5lG*6R|0}_Ol42U3xfce|
zt{O6gU90Wj_x2Ma>`vounk`*NlA+qKu7-oUZobyNRr}**Y8{lZzRu`
zihBKQV@62&GzcPRz{k3>7znvlMn`gOybNAiTJ&X4v{w-L_2@$D_#ljik^ksiM$=>k
z{xd`UL{}{_S?)hCJOY5TXyEsQbo4?szelhQ9^kBoH*eMq_}pdKL4dR?4MOL1hDv3E
zS*V5@X_;8Op@tW5LcT^-;F(oJR&6bQ0YCm8m}Rqfr)n32=n+)$WzqQwfL%%#ohK08
z%FPbA%e3flUB13oB#Z^leNTg>PgMypO#EIHV(5R$kiiePNw0}#ts+I<#@+5$TF`+4
z6;U2x-qA12Fw>dtUEp>C!RkYaIW}^i!SIWfJf&a;@#KW)>siLgctmhVW=)BM%|mjh}n1%s!$YT^jA{1
zRsc4*=aUGCkiNbpzb147Y8I2qyNG|5{J3@b{SPiPsJl6Ey5c@Xkr3gad?F6x={ahv
z$?rH2mIZkpM@qTOjBn=;e^k`f=5h26Nh$1b?jVfyF;pr;`R6+<>1z!~V0DGbLJFDS
zB_RYA;!hEl`yU{pummnF^F|mgWe7Rkr~uzpl%45*e9DwMG@*KVe+b42jDkNOSa{u{
zNds2Wsq|1aSxDx;q6DbNZucp6miA*)7`V{74(65T4R>Rv9{KFDFzx%a42MQiPq)P#hM+r9iK+#u8ffOBuUl6jwq_Y3>4rIg;
zD2@_+MQ(yXLE(Lgj89V!iTi^=woCR5dTIk|h-Kcv7#IFbO}IoW?0I;*3-aiD9?ofgOIlnRGf@*03r5A&usrKV1KjyD7BvmGJ4$c+
zi2LPfp+h^*lE=;yx<5nJL??uA?`MUdi!;K!mq-&agk=P@Atq8YROt~B9V-_0tE6y-^
z%!lQ{2^SuNo#crV>bAmRUNhEQ3~EFNuPOKJ-`w5!3ZAHd))%%8s4Eg{spvRZL-YAz
zR`g1uW~qB1zWJuUn{}N>?LQ^H
z0a6)o!D^cvABAAuKHGZA>J5Fc-lQ~Vc9S*nSut>__vuk`wYZ*8F1-UTTp
zJ@hhsj1|yD5XkXNRP3&lMUgv`CAP@mUahY0$CxvzA>#db#Ya<%D79+=4A_!x#uR8?
z(^X$1{gfq;J3cER6DW&M^b=CVLso&ccI?_y`4XH(h&=a#mcIpBflWiF`Z^3e!p|VLT
zgtX=vWB|ji3GyHX3m~vrCI?+H{d&%#-Pu7g?yt=ZFiY+aa|yJh2fU9$@3e&k<^=T|
zsn(!3gaAXs4Yb*bV;O_i+ydud(krXQT~!wA#MU*H?l>60CW@vGJz$x=EJ8(Z>~|z+
zKjig{=dBAJ2|O4jyTQ}#p+83hp^xV{NdkTqif6K)_~OXJ$v}SnJ(4+Wi6XuxhL?r*
zAzQ)Zbwcqh)wh`lF8%Z&I69Rr%)rFNBKh-z+Hi?ZM)U1#ndF(9A+ZZrr1M9C&S;8Z
zyP5%YL+4WNbf+IO?`L)oE>CIyBf{GtyN_lc~d$nkXzmDdhAYLC8lVaNn}0j)lJTsh{xY&ile3y~B1d5|a9fj!l$z
zD=;x--cH=KMs`J}
z^YkOW!$7#OFz)di)?Th;;~b3iLY|Wd1m8#uEb*vLK=pUXfSWJB{vep&K_ZKcWR)zE
zuxv8C)w|{d4EYPB85oIvmz1gw!dTYYWS?~N?*P*MK!;38k}@9Ok?r^#u$IXZUWchW
z1AcJzpO6_SY=L=DV-HZTxTXRr&)dfCXXNRa$J-IYkw7?s1c>mPcSUM%8YA%d_YXlD
ziv-dXkat{=(JN%qQ(%~0(DGm)w0-Ka*T++5w@AvQ(0X7LfRpAUP&KtNscE?iuii*0
z>pe4;q;u*m6nqizpt%4CBb2b0g
zYsANH(xn@de)c?o(VT**dbt(wyVq`~kAG)|v(OEE5KoMn!4ASMQ%a_6!}c#qihKe&$XLes_Tj#2x*k2<
z^_3Ez(G~|{fb*+jA!=0*yK9F+l=0xDXB8&Ic82YJ?3E?sdBxw~NS-+Jg~L(rH{71PgLc}@Qj}CZ9WWq^
zj_08ERmV>)%zY;KkigFw2;Svp12bp5Gv{PHiyzhvf7=jza5%(v#$Ye%9*<(wZ&;{H
z4hQQd{3QZ%J*&e$cLjtrW1bJ+a{wVNzIoC>KPf`X34lw;`pX@fyTLSk=+tfhD4mWh
z7`kn?|DNKBTsz!hYV7TYs`Jistuyfxt=9tLW05l1s?|dM)1CF<=TvCBH};4@P={2L
zdKeTvD6TZR+akmHYCf6qF7t>{0N)VJ&8Al$M%?eOe1#&5R|m<*n&2xZV!FjL473Rr
ze3J|9+edWelS2M|FYNSh(3sg)sA0PK5rrSof-M?fT~
z#=URwFy_)MRRUNV^(yCdGZPBN1i{Au2`Q|$_1){(g$e)
z2&eoG$6ogO`7NJuydLl^K3#CBpgZA~22;vN@}rumIUdz8g|p)K8Rp3rB0$%s#bI>I<_g$)I(L}C7s
z&UIh~tDyqvvTA$Wzl4O&%6XMtWo>q=6OU%ilVjH$<5v*iRU=Z|2qEeGph+`vH<>B$
zrYtb(fca@<Uc&vTmW(z^@RD)iOQJN=3Cxv*G|6xyK2@#
z7PlO!vBbv&Y5ANO=$4n4b?WYSj&%$P-~h4msdn@wSmTkElTPeQL1Z&!RGj3q)u7|R)*E6GPdEQwpVQ*h@Knwco^r@YQCk4jSXX-iisq+20p}==!LPr<*G*b4Fqhbo*
z{x$!p`mK7e+UTZ$du@d(jfh(Xu#3vXasT)LL54hCV-tF4OjW-6o@(|vG8*#hHnkdm
ztA(
z7)RuR?L9w61`&X+W`CEZl7VLd7EXX`SlE{$3h>)bN!(oz@3-f0HTQbH`H35Q#Y*&P
zLmZI6k5*#!(G>vrd%iUDq85RWtn(olh!PYO06uys%oW6To1JY8%p+k%u$DaLNgr4!
z2=e2;G~s=4BDsP@J|L%xq#|M~O>5HfhrTzG{xILfQM}rcZIuvS>d&$VvDEYk0vN~N
zk~d~xbCu2~dw!+%AYqOft)%xfe=s*N{Ylc8qC!7w0w$$K?WlE;uY17yWx!qw>y{b9
ztG{MTbsjBhCDSGs6i=>f0n{CUV?Q`{CQDO1Ma6~l`42?5(QA(j^HL2BO$smNoj+-S
zz6H<#WEj#Y<{*Y<8Wr${*^?&RBBRj#jA*kyoVbQt
zRwpJ|FTERQDcvs98{TPvNC|W)t4xgA>SxLy?+YeFx#z^ICSW%b#I!O>TW94aDeKhzbFV|5n{hJk7w9n@~>r
z?t*$vi+Kr~T~|1{oyd6QJ6L!iKUyD<9w<^8%AHcP`_0j^{1!xBjAbv)=z6U>&dqu%|`cw-tx6
z_sp#@!XRd+s}Z5AQlJt>9PT7unGHskU`G=x)45@AWDdf_a(Nx}&o>>i(XoD~o7`(}
z4CkXw;*77>=)?b_=0&gFD45aRZSUXPOdKAFj@O80%^xqeSc}LdI2t#%xXvE9n;OiI
zfSH*!OJr>Vlf#NLvZ999lD;!#SFVFc)Em^;YeVur9F&XT!8*o-`Oln^ycc5h&H~mk
zttjoHC?Em5<7;Od?VmT6L=ru3hfzA22}7qhqx2*RY
zoyranUfOdeeu#-r@!%gpsd!D1@o{AGYM8m6b(V0T6`#VI4*QUJ)ms~
znQ}K&%=vWZ4X+^4P9K5HPZwiT6Me^%4HL
zlRqc(yserzkvUZiMe;0}=L6i&pms}$I)Wdw
z*E0MQ5DB}#aGv4A4%89UlLbp+m2T)%6~xCX=EbQx(y?dacDW1NDE!}E{~=8%2xx-*+ikn8px+!rL3}QT@XnT_CVNajh&|>
z41e4M8iYwxq;6~d?_CHz)Q?RCv~`$!l%0Bzi+b^CTnf1N^%uO>L^*`}BOcBqax%(p
z)5yFWH)1(dKc`zvY_~XAF1LrJ4vEl5BsVhL^W_yxG^o>x6%9BdQwKYvtwhy;sQHuLxFeDYe9HW>~bBOB^#&p)&q-jcMk4h(;HG<
zh969RbZ$+8A2WhSWF4I&cxZ+C5n++VLZM?=3sO;f#3+X_LWcLrZtr3ETOIA6M_3f~wI%rtr9Dlwq;AQ_M1+@Bk4@#ldW3
z-n3kOsZoQuKU^#2198U)zwco^InaCOGO*Eg5efv0Thz%V<4l-M=R+z(kr(!nL>iFC
z=*~LPCZ^r~@XSCm`AR^O>NjdgZPYDeyOiyY{}ug~pWw|}%b}9`aH%wGY)L3WTZM>r
z7N6Gx_dc?t6%s!b63Vo#4x{XF0yFkRN<6E`p9;p`dJgdW1n3FX{L3C{x~eL~Sp>G3
z@7b7@3VfRuk4TMkcC*}D(>iR|&K!H&sRE!zf!;33fJ{pnDCzc&hk^~k@o&Gfw=Os#
z*BjA+4~}c)Nh0fE4>^kEzg=YOXH-7F7_rsakDM1{w0
zGFih4$;@#}RoJ;0I|vQ(v7s4%$EfHpP8k)%rZfvWu>96UzvUITD1$j=5&zjO{IS||pHQd3?>oXpI6thDj3|tSn{DIos^%OOuS;kv#g4}yp>>|x^TtG#
zMh80nlHI{0|H2yqn+DSgk^G_#IT4@-{iy}Dm7H9hwAUv>t>CC#nEZD99pB5K`|=kJ
zbGEJG`FHQJ3VMd}VY4iAF#f=~=E2=(ybq0fb&bY|Q8d_m1$Y~t>@t-tA&AQ~)pV>o^Zd;du42WroDnxmMk=8m)EL
zhF;37HO7jIzf@?{^F^O=z*%vX0|#-7A+aMxsCVn9yQ}3c5<%~%BdEwVdiVXEJ8!f;
zyOcSdrz~8sG6eO(1<9?|+9LUV3xt8O{FpQ6@UssQH$pXUtDn7sQVG^#e=`FR#1-m9YFza01!Oz;}i2E
zl}`x<>&(=mr(7YL$UtSk(XNH4FH&8=J$o8n4ovm>Hr|8Gh(<
zg<>a!>PiIKK8u+ppp*9O@4^-aUd*plX%U&=*kZ1v$oP%(21LXeR;#|-Tle*2GsqmX
zM{nQM!1Ihu9{2@rl?5(Ne{)~bJ07y*upHr;Yi{NQXT;O)(@Cdy3+%kY7J$KP7Rp0p
zpW4v;GHf9y+rZB3z*%7p?vCtsTYw)lb|AUEO;+G%>97n`SqSl44?N<);&2!;T&Ue7
zozIZIhh0-c#f{7Y+<)_UK2`GT{KN1dd`#k3OciAl0ntLa6hnModEU0LEkpOGprzYN
z4y3hl&A^(0a1$lnYDvzjw|Y+FZI3$%aby7de%exCP>5L|53LIGDVp4MTCq=)X@VL!
zN|S2jY1;2?h+B3+x1-3y8uY%)vcnaXcBO}bv=uN&X~9B?Y<@~W9r6>NI^=_t0ZV-c
z>MyA!h*In&qbWRE)8-36XA0(9g4Gu@YTmO70`i|sBke^jfkS!_ie4#EhoAh$61p#d
zz_PcOshDLftNwKZ6Oo0dW=a_9tQCx&u7Wo*%&0@7u(^wbhrN&h
zuV`22nuDLJaZML$Y{Xi81}uD^Vn(mn5`DC`
zdS{DMpAULkl*ZPa$V$
zTFG0_*S!qo4#4XU@nwWWVRp#~h9@hSy1MH>A?7u&0SFU%D&)epU4K8qTov
zjHAB46b>FJTJ
z_u%17E4cJdywrV^X+;g0_c=3mm!9Sz47B|^bw5Wv4l)8t(`;p%x#I7x-*|=Map6Cc!4eo!q^1
zvd)PU+1mJ>T_pRU3@PuGN>EmSwE8yB;>C>GVzPdQk}Yq|he1W&>lG&@*K!yf)^;a_
z=8}fHNW-2a)Y-IT&kF#ZNECloiXg6w+T&)YqKN}fkzQB
z&Rbf?eiEv3$IYwd2p{el#y)fT^*|4+Q_l9W-;MCKp*0BenThxH3in!m1i9bsVAUy<
zBr}`zlK#48jkI$WgVjsR&;L-r)mldBnGyBKmqI3n*_1NySwfAWb(Y`sw#p6?U`6{q
zBi#409J)mz2(4ac{u0<@31J1)|CBSf4I>JKjKTVv)~z7p8%?Sw0~VuRy@)9$vW{cm
z;`vc>l4ACUhTSH&ur~bhUHI5+Gsz1iO=Z#JOZJpLc@4(IWQY)q4y)G}!h%nuj`Z-H
zqPf1f9A*P5UsB+k20^#R;AcJP!9c2GaC?QClKx!Sd)$t~4YL-E?KODu*HAm0R>~Wh
zMk>u6>o7e@fq|{`2p^9Wtf!+!p1^WgQmyHkJda?4*=u4#QLgV8wYq8-4;|u
zm)@%xIWmqHitv51-6&ZdN1X+*7~2sSSnFoe;-b!hN7>25psXILNMu6CW;RSfoxP9u
zL~!*eSDoQPy>cY|Nr?t%r0P-L01^0KVHID?qBZSu2)2j(j2N~w2rUSXozrUv~bSZ!9hH9{}+}qdw
z3)w$^!xgx&Gr&H$pdYk&|IJqIv>6HfLl=^=gZA^O1euE{xA7Jb#{CeK1&K+z?M$T8
zN%e6EL~x^r7$2q*_K$t!w6J&`FF}oD+NIY4dlJMFCg&M%PX%6dQGo=x)QzPS&>N*N
zOOyVwlsU8@L$-I>pb;!3J`DN&Hb8G_5b0;3$SPE*Z$cMm0V)ljH%8(;+wA!(vkS>PMD3UwD!QRPLp`
zg2R~2Ex31#6uG^$4;RnwcXCvy`}MDM1`Y}iY_VV6X4|mS7b9hAD7J{8IU=}G5EC$2
zTnV{az?MIMcBYiZDCQFYjcGQx=e1Ut!6ap#HJT!uSO^V|D-$I}@^+s_$)}RWq#Pfc
zLP@2P#$!uRd&kroBzvz9`D1!xb@99?Gv73F#FV(_5=m3ZjW(0!hE4U$^7OZeB0-T#
zaFa#3`)v3-{MgN}RC!HYd&U%M2Yh%pcgn;DdnD{ut($%Ed4h0*?GDlB9ZUN0egx8`
zUcUVUAOeo1C?U>Ao#__!F!`|=16u%Gn@o;Oc{2>Z~)*6%GR9B`vESO=7pxC
zu_<(1Xw^5-nl+e1pQ;9{T1-gOf92W1_1MIUp3cZz6;dm7<(`b^IR?I0^u)o)*{;~?
zUpJ&NDsM&`4abIrqP0?ZE*x|NzqDBNqC4G7AakLi@K%1>V*miKeK&P^5dQ_2wwgh$
zn1OBJozsB#^phge8JtROzE8eJsy;#XQSRL)Bw2nKSrZ%4w5WBe1hp;fM>MJ_pg1%)d
z{`;djvC9PqqOlvl%*FW%4c<{=b>RH;eiOEP{ueP*I6<`4Rt&^d*L{ExuG^VFJuX#@UBgRk%IZ&ia0!&AXR+
zFcPBPf>}V4Avr{JTAC&WELEc;!oJUmrx^7*
zk&kGX6l*fr;t`!)UFdk3QAc+_MN?@!{%)}O^P>kBK=@aD8<|+JY;mb(7kwOZ)rs2A
z+R^M2!B>;%qTa4W@OyRg6Mxo3$6TP|pfE&0$;rZy(=?yV#(sQ^I#x#f%jla`G|;XC
z_c5BzM0jz*$(Tdn3G2C40DY}&)Z0R?Oj>x2^s7ShBij2j+XdT%hoTO!h+mkIBxht~
z^!!#~upcmqOhs-X8&j%ACZEnYk|bdPRIA?vrA*n}GIo(^D;M4$u^371-JGU?civ|~
zL9wAjv8Ct2?H%my7HLyZ{LN#~uEV29LJQC4h3j&6bd)`4m!jA?JWLZ)vk6aZ6Agm6
z0a}g_?gTX#OAI9@IpC=n+ZFediulKp4`$&PyNI83(<5pA`zC(9KJF^-T-L;5(*GJ!
z!+Z|6NE~#Vf}6Vg{O}2pqctmPz0-uBEuwq>SwwAhW*VHhI5;?HY_Wt*kxw1#R?H+E
zu+$F$Z3BgVJ^-g97#Tm}^DCxmdCd?EmNQd=senu>`#HR*UM8j3jDKzz2PLS&Mogwm9$%(_)0}|~D*ImQOMrcm=`R$yc2gd3cF_p&_EQs)NxzUgd3vNNM
zAX(ou$A%H@;{S%c)sCsJGCaKY#0v+5=&LK`+VPvY$z7~Ywvp#8LF~#ijTpFbL6qSp#Q#^t#1QoC0T{rW)nbY6!X#)Ae0g>IN0YG*807gqcl`j@-fBBp%-#9)>XRuNPhpO1HMM
z|3N-)%L}WZ`r$m5MXvh_%`AtSwnuo}H4P~6x2Uo|cNZ`Mc8>7WE|RCDO#_*POBrn`ye?!-@^bsDV6!d~$n9^F#C
zY`w8^8E1$z)Rh6X>*UR;XAbq5`3f#EB}vtgrNWqBOGdk|1aP>){C%t%YdcS}uk{!7
z$f0bKDPLi(7Mj<;!5q@~+4#9H^}G*;^qgfcJdT_5DW6ak|4LIfDt^FGbGeRH^5jJ*
zIV>Y6F7!&HklhS3TwuFCMva79!6TW*TwOSeeJYtwdEu^Ac)fu6mk+9s_qetbc8mzme+-kh%`m@
z^Ca+?H((D%_>eZZFO1mu&;y?V#$#Y-vVPv3b>v>pm52vkA_r$~IzMStl3JO*%4c_%
zR>&9=g@`vW50P+Za&L8|Ai4%6$wZ&d{=6z&`RF?7|P8(;em~)fNiFFoB(?}@z_%;d(I}0P(+?dh*(BKD-+*)thwU{$;8ub<;YC|d5V%=6uvi_zcrAW(As;kNI6u@_~-_pljNdu01c
z+4)sBM-OeNGSP4S%%X9u=LlAVzOEMr6+2?
z$}>lmiL4fQ7mPRUN1uKcO219u?f68~_tW3SNLU6m`(set3NhRLV;3x)2}T|TSH^GI
z^Uu?V5KGf|LEw<#W=BKSp7j{&2cOE0c9((*d7RMA+C(5o2Zs$|N4O_p{PH%gCh#*2
z=rMDkqj0nRI)0wjIW3tlyrPf9G6~2iP#LAozM;2==lQ7AgxiS
z_dCqz8ifUcn58zleFvXJNI`0%c&S^L<)lMAb=qX2kVz>ye53<#m}wGTHbUx$@wMJuAuI-1tv>
zf?~@7rDyKVv!>v|FD$@Y083H
zY31Wxoi2O&HLd!{r5+(!r%`r73L`)9k}1-gyB`hX1k%~R*Zjj#AlH42yQl$O0Cx*WO|+M-PMl9$?&-vK9M$P
zIH+S^hx-%ES4(KTbL0^%DEc=jPlC{7M62yN0X`tm!XS_BF_GeTmfK`d!=Iz2C
z=gM^#y1EyJ2j&ap_{r0laxVJfVl#ClohF!OpO*d$7rJ%lO9ol`67;)}?~M6=IM^id$2k#ZBi~Vd=iMN1-nnrpbq|I@pdr~VCHGDX;L$%#5&em-
z1@bNHqn3zui-SkqCQd~N{%otr$K6qQ%ox^eT)k4MruK-${`&J_T(B&uAIl_y0=iOe
zo{!`&w1CFcFPJAS@#vZ5f9PKvL`?H`6Ww`a@+?e#2^U$5uJyN{=}P9&*EhXN41QJh
z)4x`?BNl=i-%b@T$wsk5Nb5lK32WBNFzvhFXLTYNDed5i7RiiDsxBBBAw6{l+kc$^
zNu|(RuXizIWBp6B(Zmf0V+%OWZ(m0(huDi&oq|YX0^cXW$6G-&E}&XSQK?UdExhMP
z+?}W<7m5=;P%beQKiKLzWL!FUqaW;3{H>1z$rlQ(AR2F}WxC3aS)q+`;TL|XXMQK6
zJUP|~Nu21HvTkmczP-x*YXh`j^;5t5lS*#ES!5G7tg)3sX>9QDOBvo;IrC2qXUTqM
z%!WfTJ8+5{J2z!sRHHb1_4Q)!!KLR*v9VIrfCeOkykYTOnviSrW!HNwWqRC#W360h~FP^O7nCV(8x!JiwrR5qW4Vdsrct_suZ3=@8
z&xaqb?J}%f>4QFc(7(Qn1@Y#1;Y6C4391yQA=>v7$N3{_t;G^Op}Z8%KGltEtw4vp
zJy*76MSNPX4MKh>OP7Ky`>v-4Uah+hz{UTvh{fDuvl&-ZODEHnlh*w`jooGNFm-S1
zpf&2i8e6=KDqDU>uNrl4IZ>WTG05C%ru(k-GxDd~y73ZzK)VAWL*@M$gh_9LpIM|quO#iy2jQD=Y-Ai|9fvFYKo$O1Ul`0jv9LhU#eH7&c
zG&>)=c(-+gol6MYP?86+5F>FFH&fwf(#a6=qXIUs6w?7o5+ZmHLI%d%tH6wPm*NaT
zynd}aeRE!%>4(9xVE6WR0n)z{97Nw-yd%{<=eACi)gm(rU$KGXoJxjZ55J)+99a
zd)&dyRN**1!uP4(&HO<-RfTz-8Fya9S_d*sa{bIz+FV`A@xe
z7gvn2PW`yB^)F2Eov9`hynsFkgACH4EXcc=m3=Vx4PW>KP^#OoFEr2skpDM3wh`0P
zbarW&)u};JHnEt!c*EUn_Ae1$mnDujcTKGGp>5HJ@;K`#8-o8H&8?%PTR(8FSZ=iTrJCnUbjyxZPpBS;f%uz
zByp-Gx=bxZthm)MIlEVfG`pxS8fR++vk~sQs_n%@(;incDtA8|TT=Hwx}{dWb3lz(
zJc{lUEUHu=f=w83;aLW-3^C;N~Mil#b7in#=
zw*L;V=&G}6C}(N#FZ0>
z+QTv13Gwc2aPt=UB)atazDYoxt-yU)5k!{(wQ5p40@b`gwu6ni_~926%7IR)A85#N
zDnS4iLqHy;9&m8o<42US=~DRKR?x!Bk+oKs$)Rjj^K3=M&#zPX_ui%5-G*+BO)XsL
z8B$_6KhVBDjo0aih~=_x%(e_uANHZOr)MG0M%@Jne?3U2*^itbTdrsv9@W_N9C4S1
zvfu}3punMMTN6>KVBH|+5iGPCj7JX=KD?4_S%
zC!xs9sjn4RE1x&wquF-v@~WhPwvG)W1tZOS(O>A}B0TE(MGHN4+X>|uwRVWM!gq9T
zu%VE)^EW{#p~dCRv$;Q!HCCL+ae!#Sx#dD!@N+{SPCD+#^*Qz~WHI^`;
z3hRy420c58z*OFF(PGI?>c77LB{!9mO;w`qI-)<3&V!AR+DT>M9CD@7Unyk7Ya?}x
z%J#hcokK~KU*bu1tntJ`fq(dH`>G0^l0V5&s+L(UEfi4b7HfP?mE=;wWChn6q4SXg
zdZ*?i8jGvBKC{?h7_K5Dj&?%%!?t!$cF)I>>S^0a1OltSeFyPrj+H<=d%QRM@|P>%
z1y6N-g&#IZ;eAtN8>?hy4u$pH4RL@gw(y$q8Who4odw5ntJvS*TFYm}(`>3)lHMCb
z_j+PwJ|YhB<8jzOIbiqv<&WOlBX^W5@$CmEDIZGSPuxIYAW!ivwp9Mk*Y%uG6kuZl
zfg7j%AYzEMc?v?00oxhC5Z&7EQzVDtvXW{S(YnaA^P26BuVaZ_Dlw;6MlF4wfFpdR*x!)y`3)ir5ub8I>UzssQc{wVQ6pZK!wieo
zBCMQBgBR$ErS$%Yp0cI4d-bxa!n0e`av=VRATabW^kmRsveNXm)8fm5^J|JQMIh8*
zGv2^~!DQruROOE>%F1DNaNT@(Vfr-s#h*SL9_Bz8x0oa=!$BD@K)yjIeDxLoPR_Bm
zBNH0jZx>|L3)>PyC*lijEhKvx+WV;vbDx%Kn=lU+L-k0)6(10?Cd{yEW6NJ@`C2U{dp
zhm>e3)_R+SVV;^n;f9-RZf|~o2`dVcjiEnt52`{zrlWQjiIWQV*MIvvtJGO75JcwF
z_a}U?E3n;BdHz1{^}Yt3Y&25x5Bc(&&2;!&cYd+ttMtUhv`8HH;Wk`~gB|p0H(c5=
zyPm0}*3lw8{n!Y)3VFa=;d@+*KM}Mh{_C|h+@s8?&>iQEn(?%nDAGfm``Q|~y;REv
zZ~NSAzp}TA^^pmOOPfecy5%(x%Uj;*PvL0$ASh_xkqUIyT(Oy<_%?N0sm3`~L~uL>
z|3}p5Ib5i(7ME&fg-=@$nv+fg)oqR*R0bY9Iy%f_(lo-mni(M9P^b3*_H%xDonzJR
zTVT|lz@Wn&Ke&tDzvHh$-(Jd`!KMd;MUvQjVS5kQj~1HcxPlPUou(r{q276sJ@pdc
ztw=%4S?ps*SG}d(LbF|dX1q`9u+90+thC+*?pEd8wAsC5LlEV5tY#dx=8eDhl34ZE
zyD^y$t8y$9c0Po`+}+Oia}qL~l>TM+qv8I?0?1gnBeQvGAZjogAV4>r^&Uz}l8KYV
zC+^T@T8zguA)ghsXr4Lb=#Mz_439YFsdNRxLv)rT)?1T6eLb;XlGYTxn=H4QRmgj}
zUg*P6Z(X4qYvVrjyhM&4=MN|@FN@11cL)bhr$t1N@s1&_-lj{1c$Xz@eCJ^{4Y`?|
zwC13;-q?u#KLB??h`%;tj8=WJ&}NKaU)CBe3q{jrgm*GR-iFyQ_d5&P3=@JV6O_ss
zO`C~Q+DtTpsjB8#O;D+}WdfTC!H*OD@U}q??C)=pY@3ONiKPuJ&5dCsGli9{Evzh(
zBAGH2YdZqHs^oyVbxE{uR})t6Uf^}4I@{5vKWyI!Il@DN5oMMQdCN9HJvOi`Zwqv&
z9g6eEugAP+7WL*Rjy)@jHf@WiKKar}I+7r}vb<=~+7SWgPT-cvD#IV)=t&P$Yu^e5
zjdk-6vCxNpX@$CroNv9{F+hEoskB!&kdc?KG1v4ufSSSnz)>&wK
zi8`&-+sK!D7j28-y{aIGxeZFTYm4d@u6Ui?A>&9F9G~42&NUmL)6f}My6*veqveQ7
z&1zk;Y{>qFZyCeV$^uc5QILOfgZrl(C|=4gjuXjjQMjNzK7aJYmq8<$0}2*2O@#oN
z7}O{G*pOvp1WQ(fy`AjKK!=ru(U)mlk?z45Gr|q|I=)ATMgG|IM6Q_fZeoTJqq3pQ
zLmyOV=Z>~3g78SCgDdh?AjTbuM(5#nXr4XgiPaYvm&1sCxAEZW4vZ-M4!hT#z&qt^
zn@?`;_>`kKN~sWNnQdVdwEv8co`L*@35q$@LB6NgaozVVE?j0~HhNSis>QR4mGMTYB4Ur-=
z?p_-kR~^DbzpHpuxhE#)5KSwCiK#v6j{XyK6JHEtUL5^cjX+VZa1G3j`Z@U<^RDP!
zyf-!;Z-vGe&!9lhzfjVI>1u)8@H_9qIw=lWO6uo_q7Z6RqJDLJ+PeY=YtBKZ5|(O&
zVU?I$#($5*nC4GKjxw@_wP^tSgr~Gp2_{i~@bQx)huXUt9%MFjYD)2qSWt8RpEH)h
z(k87~sS(P4hX2k31J?sjSg+)c2xYPu#p0Wk1eP!n9r`;^cq`DaJ20&gV57Xv{R6;d
z#>%7pf!ry`$BgVspjN}0u(!yDft6~bHiMS$-y7-7N~v{b&tsgK`u0o&`6sx#`61-l
z7EBqvJMMVFT(DAkM@i;T5c-URCgpl#>d=>%KD{OCt*(bogU4Wa>r$}R-y?KFq|0&4
zo3aRpUq&Nm!Fm}N{k8+6YyF^*HZ^fUqm3v|%d%xhT<&Jc
zwv9q?RPts)1)f@r>4&T^cJD=WDPRV<+Xi%Nx>s?3!DS-C5gx0?(IE&4P5Em(GOMhx
zHT?u1FVWc&ncG+S5#rTXMOK-IzqNAf@>?Z>rodU?Qf3**`TH4|Iep?hV
zjmF#UA5iwFIA?@B)w1Ehc3V8Y9EkD1d819JF`oRSV5%m?vtUWFVEj1G8`YQQMDGIn
zDY3DAH4L9O8h6|MhFRO|VPnt2u*zx+(@#E#-{d~<4K#;ccIy~_D=Y1U+IIVJ<>VQ7
zeyWaP-xfl)nmMrj!Z8$h8Un}Rm14Mih~`IYh0M$x=0<_|q`F05xIg@ZQ=!W#zS1Be
zhtI$F@IHUBnCVdFRU4t=vmMy+T;+vRxhKx-I0@I7;6bOr1~toF9leJ%K+xTX@%E50
ztJDqDuX;cy{C@JY})h#$1eX!`5kBa$D~2
ziuY#uP#`8sWJQC%&G2#6P%Lw4g#L{bWXCjLK3KhXiFv;u<@IBP?6oFi&7{IOJZ2cy
z-1gV-eyNdq1v4#=md>BBW&R=j7yD=?D0kN{!$DVhCUIQJKoBs6Nuy0yfa-+H3xRI=
zmFHMN&b^sz?f}-+2e!`#Cf$v@v_gAeFqLA^0^n5&dPt43S3s+tz0jj;Yt%1gPr@OK
z@62M6z?ta;nI&v(%wSt@684=qffKQxW7{#tQKg$g-@!O6u6Jkl`z&#gn_Q)=5n%x?O
zQ{3R%^oNOd?GadW>H-eVZHmal6EJAuwV3Ip*0nmKxz%MHJ9iU@PkN(X`=-c|hEY5|
z*^Na4vYV{|+B!wy;_=J4diWp68+U}0NZxey53$cmPyNycZMye`l;+3nN!ExsMN+yd
z1hHbqkka->*iKTq$u<4g1_%!gjr$Vm2KNs@_MABsdSzV;9h+O>^09NcdH5u}>$OMI
zv{asWutZ*m?1=XFi#wF6xS-ia^aD9U{e1D+EF0{zn2G(;aBJIPJW_HmQzq}Zy>8&_
zeer&96g)3(!!ecVwMOO^FpKuaM^7C)8LeueS#254ZCHm}ehM}f^mq%lTr@?)mbKHp
z!!$mk;OGAd$dXOttTcmyoE-gAG|aOaBb%vsRVxguPlYSCgNlm8zmFpkYHWUqf?{>5!T8#rIP|_M>Q~co06-IL
zs-i}@5S-n-9nS^1mwV&Jn$5WK|D9Au;njgTm^g7fMi1d
zF#+R(Xxp_s%u97fhoZ3zMTYFnCt!5m-5A(wJQl55hgI{(pvQ<~DBQa
zSMCDHtL&;YDcKc$%3GjxuWtB8d5uYd0?6&@f|#{>QdK`pENlFTRSTT)@aI98`_KDi
zw3!*o4_SaAHe1lIX=`-v(;Hn{e~-lN;Zznir@yAE6blyQw9tuC-
z&oIko4+ZDAg>8o)rORf+n#ynOSPxag&tcQ%hfuUeI6Q7%z=E>=pBJ3tW$`#lCkX;oP{Ga`e(16+5&-?kj6B|718CcW^@5YELAB
zec!{w{Vg7!U5`b(UcjMFJsqz#g1H52-}@uf=4IM!L@$uRzEO9ywmgOr&h_E!)EUde
zO5qzNw=%2L9fO@iu&uWf>N?d&x4(>0PUY1dv(k;wr07d5>)a5He%zV#Rx+|2_-@8R
z^f2FpzD~8_>|6(RyKjSezuz&XaW=h`DtQRO=g)E`Ww&tPA2$?nbVMFG?f1w((7NSl
zMnh(dHtnoo{f`$)ws%9Zf&K_BZibja@u&!Fndy!^t=&+jlRE}HG{dwWR`Kn?*c82|
z!51LTliF{kD4}WpR7BKi=g}TpHTJf
zBDAer7fyBBVqQRH)XZt3B6sqT$)~@X;hP={vCuOV(PI9-tpkb{cYsaYJRRhyKLK05
zxsQMTxruk5bHZuDS~%qo#1l6QWGR*lOP80&m0NEiAG;RLl^kFUdvu;Vk-=VZlPr}7
zV%?tRxOM3|UitzKZGJ|7$KtS4TXQBT)_E?*n=4+VkmYv9lx-F9?D8et^9V1Lh8{iQ>wOWyLyU?s#*kzOrFb
z{CmtBpB-^}mJN+2|AlI{aZ|Ej+X&+qR7N(Fl-3Vo$~ckaO5D@BbayP-SRD_=ULhvM
z;_A-0A;!%Qg@bAWY+e^*7FR(olR!Mb^cF=%?!>8)@9>HQ*|uOY6p)?4m~NgZnC&a*
z>rjD=$>iWhK%bw1b2(W}xq!Mwfm=Km=>hbd0(`ayEQ3jCzG1>yEB^H+`GAeHm{g7j
z+K|Pa7?MhPVw~9vjGjzqwu^da>W~I@nNl|QF
z))@`=*22(jOVFj$bj*4%95XxD!Pb2Ci1P7=b@MscUbjN(gw%q_cL(v)puGsml?xVu
z-jG)qfu+u=^{&Z;%pQ%px5i)NM!@-cURaqppmEi2VDd2SAFZkUGV4y5KI;L7&+Lpk
zTl2v_$_M_|%`spc
zUI8PPIxGFU2}-nUjpA$P!@ut@P&44CUnt4$i_w!wc6T9n4kl4PCa5wnKG`*|)&@g*rVSHHADo9-z{=|}v<Oq->il)fM9Nm43<_3w6Y
zN*BqEQM{lZ7PqN`>1IrfeQi;1!g35K9Y>;CwfFOG#LqV3PARr(jg2-MXLzW2b;A3om1#3ab`2T7;|M)21!Lh=6*sA#>Wt^ROv{#Rcz)gq
z&ZPgITMORZe3YTLuMilcz}3$mud*4T5SCf)FJdH|;)9NFu{S&c+d9Mf`1P7_X(=uSer%7`
z{FB~4tQ#`r$Bc%OwcXh8+y}orZg+My2DAEyuH3jH+PPyT{O!JD*d9kb3K+#;k(?4N
zT{zsneLH25eNiDPsTgx|n^6o%;@X$P8TD-?hQ0R+t*bebaPt&9KVHhQnzLCqqBgm$
zJ+`zg8h?mVsiZ;$tyJBAcZKMEJUk3OI|UJ33Y7{N@)i>UH&yVIc)X4HKT-5qxowf^
zK3oYEozWhux{;?4%EL?GO;Jb}M~W{kQoHcoyLX9>j?TM4NwGO9GZG@Xnl>m_c1brt
z5{SHdiC_yas#bP+99EeSd7Wz^36%1zh_8!Xo4}}4
zRv6A5(Mj$s;j77aSW&~1pZZ%BFf2HknBYXwXB0eD(Ngpojp#ELqR&{Mx|bySj9T;=
zE2_FH^6>m*4lfK6ea3~0W6}twwFlHck@+ey9NxHbBm2k5KBH}9#!Dm92FpB^an`<*
zZTnHh5)>H3T?+>)RPv&X)(fhytmMTPkI-(%HilQu{>>{@s+8$_aBwh>w8JCwg(PCL
zF?REUyy*x1?MoM#+FR5w8QejZ7v7KQMTe!lrN_vocanY
zUwn}x?RKzjc;y0z>)YDe;^pO)>6p}~Qm+~k36WR16l{T4)ylZ&a^M{LNN&(>Lve9&
zgoo$8ceAmhbm`K#ySqRBF58c6M+Jw9s2i6EQn}(^#Um~KMFQc0k=XirQ_cXs|FUJv
z;_B*}`C!ZuINOJpx!$s)9K(?+wNrsn4hTXVFwornK_q%cm~K%LXPsQ)z~2
z!jj4ZK4yjcM%J&M$d{qk#$5jIcl6ZC-fEpe
z9<9ql&AdAO1h1Gi7Y<$(W>2&H%a93{_V`q
zF=kk@w)f^K@hNFo+WS!LaS~Ma9{?DkIzbt+Drhio6$lG8ey#qSwt*
z%$iV+I{6-LEO7L)*6K4E#eF*UXnSYq!}ox0g7i=2@+{gB`Gl{3^g;~4gS
zBE8;lX1ITTmsDim_OMb45+go&dQOH>8j)O-DXJA%+SsKUbme@gUPr$yk%!l^J?L59
z?&ViJ>%R*L8yA1F3_%R^zn5P!oE14zI1bMQL{@6n8_o@6>|PAC03)0o9e6RGLvjC`IMA
zBbd{+kg=S_1lxvi=tEN0LwUI8dU?@7D78NQpq)^*cFwt9ZiX2p0}I*?_h4UavS_oy
z058dic57?;478_RvV-U|rkjrc;lB6iI`0RfY@cV=`mVS>S*rvL3fo4G99pCn`q}~y
z-Mzt!6*@*~Nj1NR`*^SGJbs9><@s6b>1u8~W*9RRO8jexPf5!m`mbvR8(ee0&_rQP
z{kL~>Vn84_gJaQ@^`^3)i{9NX@(MGlG{cNMKv51F)hKBxNSWTOTD_!&FvE=8V1bj5
z4UG#pvt>{yl~k>13iC2K!?yWw7W)jrt6CLYi+U}ki9=y;MI(D-a3C0oQEJ0Z*5K^fqXqq51%rG*T+p-yEm|=`i*!xkp=<;8g@gzfG
zkDs~mm|=z)MGS*j&@$nM8D=~Q$kLW=y!2maEU?ayPd6Eju|r$lq9pU$32$;bX3Q|=
zy3Hb+m0QUOl3!S7jf<4+!kEiCuh5sstf-Qk6*n@~kL44x2Efv(q$T*z;z{W561wjp
z$5dl?lItyd>ylhwlneBX8Ba?4$qVu_J{n|{DOavsO6ITo_wQqXWU^WCqzr~;7&j!L
zba2qw6UuWKgnUlk(x8_ByhlC$Q)YjY*lnp!cB?|
z1a0|cCR-aDo0Ke`rV2S;==S;aF*Y_fwuGe08M3Wt+W?PIx(1YO2>1UeexY)Lg=jPG
z{j(gFZAsc0QnxF~pc
z4+#woEg_+DhO7&@orol2x{&?G*c~$Zwy}@|J|ZF_rQA&o`o`0ANL?oTyPWVbCS4{S
zLsI5UTTy8v>8|M0?~4X?P`rc@$Zh8*`soyKY^C?KEZeB4sFdw#jM1hjEYckmvd(gg
z@-ir?G{cN17E-Y#NsphAc#kEiRAirsLMl09?b_1t%}y~%nGG@c$>--5`Xl#!lzV#`
z%Z#OTYDhwm({chsl}}-jm-0GmgHTC&(E}Ww}gFL2XEaBlVLpIRGRXkPZtuyO$m0%$PAcWMunEk4DO_j~Mf~
znH};HerardKF=Yenn$8v$LD@sQ6iUqstr?YKY2l3#+7uKNu?QPm|=z)W|(1y8D^Ma
zh8YEhK^H4Z@09eenR%)irUdDMkzp8eO@gTZ@k*vop9^6FcsWJd8r<^~803p47q$@(C}
z5@kCwC$}@Bq#(m}<#=gKVY?D?T$gLWv(vG1IfS%fjVVktH^}kL$;m1E&~OeqAAgq%_|Kg?XOtP#q~TbvUcF3f
z@UC9H%AGrRN>Hl2K@NnHz^4y}GQ#1^nKK4m-zZ6I&6+h+3<+IGgXiqovqt48kY!P)
zPMxe7h|?ft4h{}ZNxD$dkZn&oHL^hP(zU_s&D6nC2Sw2q
zixbk}B=tge4i))+vKHMlAqj{yASL90e)_Z#X4}jL)v8s)!y|Ryr+@$b
zH)UqX(2&NDtWVupQCX+5ZjB8x3_BnoK<_NBBs#{HozUOkpYr9)XFixL41vGJ0336u
zWQ58qZ2h3#ucDE7W;!gL4aT0qGB?ObfZDZdXE`o)gltE~YV_v?(twr@jmMD5c|umF
zwl|F_%`7WYw@>gI=A!;{001BWNkl#x6zc9vW^E$Y{=uQyqb>^pMC
z&zK>}my|!KXmlZU_BIV^r+H0#C)X^_c}?krPKP>|6PN(t%R&k}|$x8N~$
zLMid+hA7x&P-))dL+RUzm^x(pmhYnLoRASkraoRvhmhP7;&I5|)zRXG50@rqDDIpzPQbsxBrwu8yu@g!OIUdOICVgK=nG3{}38jP_OS6#5
z=mRAqOnwk^ASuevsXJJP8p+lGHp7O@45}bO~iT
z6eN&C6%#^OTrq?fsbUBTiz}gJ%qn6+!nm*Z$5q6|
znP!3uGtBsJK|^AqX@^1Cmu?YbYi=$k`Z>*RJBM$4h>mI?P`7lt(GTum6
zMm&s!ziy|zf*FN?X_LytzghKq7dm(Dto7;6^zZW$KBD5!b(2_gS`Z*j4Bt-g%iv`v
z(@y=#x=Cv|oo{ZW=EgTOSbDhV4Us>5&7pAp29PG=5IcT|D4c&@
zeS>diEaHc|x=+y%^X+u{4qnE|Cy`gGL;2~md9436N~ioD^5>R$EI$xvOu5RQAehNd
zQ9yNSDgC>RX7i0FsBDGta^VU@7{oFVy7b6AhK-oVm;3f|{F>R^Feo(l2)c5EaN}Qw
zXWURw7Qfe!1!b
z44o@TsQ7FwreK8gJE7TP&^4WpeU(av2Y(8eO>C$v38HM;n^`5RQ6<*k^>x!|m$9n?
zO1m=I-=hx-M_Ru5Ar&pF8dkm%ftox2?qSQ7s+!
zA*d}ZP+FD3(Weg2cWz59=aQE$zmVi-!1SwhU%8zAex@hCvBI)y2ZoNgOI7cpD^uwy
z|Kwk=u!jl5p=J?}I7xxfXABJ645PGnl98VBWx~J`qpL>%ENq*HhiPK?W#7*zI?dyg
zQFRNYFV#18G3$eiJhyxzt?df&peG5clYIKtM%?FZV02xp+^;usJd99rDPBo!U!{o~
zR)s+FeXgB2%-_yE=v>o|haqSAdgeY(gskG@{+0EXi)HmLj2InFb#Hk#iEBHiGV{Xo
zte)DA)UIMl^)4eA9bFx7Q!H3HgL*R_hswSO*p;H6cs56_D6ai0)L9P2Ou`uiB}H%)t{G6hPk1&Ne<^`(F8
zWbcXphdo<<;a1Q|wk@8{d+$wS$ycWdm!1YqBqzRH!M+PI{JZaCmhZhl-0gq39FZhW
zo#4wA`?wJEFZ(`T&fdSzvw!^ou4Y<`8^@V%Hu6(YZf=W2BF^n+{jvqjTeyNv2QLtv
zevA4xhc|q~nM^Lcc>dnCfgf+BUX&PdPW<*V7R+17icJT(5S3
z{>PtDS!Ga%|HfBqxAU7QTy@-S{<$2H;u(qmN08fJZV_~H8;hsE$9vP3@YU(?OtX2K
zdsjHJc{^7CJW!oCJ=yFH6<5jucO)l;5=<+bq0PsT9Pk`S2^U!`Ey#7P}
zXUiW=x^-KWb?sdh+6pF_bpxGzLiGJ(N1?QMrx&UjLb8Ix+Di~+F&@U9
zhSLU4cK!sA%6KAgT}|2aJ>_%lI-zl*g{cBLzH=S(7XO50lYWePvlkUFE@#GByqh@oO)o>-|-=8>sEgnVbiqvmb6OU3J)NJ028s1)%3i=;k9l!VZ7d4@W
z*z(^H}_oB~AK`;>}(axww1=UtUdp
zf;#AX=8jv!Zc#yT#k^c+hJWzW=p3|n<3qYgD;?3>n~v9b;4K?)0}x^x*Z
zh{12X!<@yVdFJ6y>^>Hi>wS6+VD(*YUcSTq60nMkgZvZ-@t4@YbRxt0_oiFdUJM$y
zoFln9nhJ}DT;0Efaf5r)wM%#U4qMKzX^tuB+}SW~7=62UrAPnK%-?nKaoCyq+7`wQ
zoWYhoiy79l3vaBuMB>Gbj2ZYLKjz_n9$sbtl5q^~-IXrg`!a0#uQ`UaX(QcJ?xZ`~
zXtJisDKg_A%~M`KuU>D}aTdNjjP;iuadG`?bnY^V{dcqP*XiZ^<^C1C-M16%+IFPx
zxDEVuFNug<6X^NgUcwSivbaYFx=h-S%wrLMX+KLR4x@kXZglN6h;hr0kluVAUfs`<
zaf9jIwF}+*4rBSx*9)x=@&--Ghfo>-Ng>8ta@={g%^Jo_vd#2=i$&jEPjMb=LiaGW-!N94$*=}AVf&cYf9&J2xX7T6Av{8b
zTcyg_i~gTnF=ksdkwhUhVFNqh{G2*3J9E=s?|JcsFvon;06Z7fRvMu9x
zON%#=3+qPk(yW82%Y7bPX3v7ryxg@NZ9DW}@Zz7fmH{Obb$mG^d$i`c<}K+yauuhF
zlguSuW5@f0n0?SVW4aR_z>SmeTManB2X?iB^^0KeDOkM`LYD}l`73;wwi4`Kh|u3H
z#C59$(G+A;a|kB?EbCDaML{sD4uh+LsP23fD(K9@CfJowzb=THAj3{VhIN5yH#~&3;(WvY;4DaQ^><)t-|VTlf6lw&{`uq>a-DBjfPZahQ8DYO;v~wJu7d0fa_dzDHkIj7TDSmbOMn^T3*bH^^!Ma&krXXNBo|Cn#xX&H0|Din|ltX5ivE_zB@!h%P!Q)p+Yp_2l#AfX$H(3O^@bJ
zQ@24YUY_s)1FXMc!%<04t*P0q4atZ02?CM&-0Si2k&%BQ~KNZQPlCi*A7I^j$2UJY7?
z!^97cQ|0aXjBVvj)&M3A53B=tb6jtlRP~`k-EO?vM#iQzDPh832nzp~G#g`jr(_W0YOV=GM(%)2H-OhsL=kR%B8JoAPVfNt0xXZ?%y2Pewb2(b3
z2Q$}i=F@TY30gm!b*Iyeo5Tl1hn!~Lucdi;jvz*zyu`6MQMY$_l$_iBYOZcy!18lG
zys>OETh`2GaAWrZg?Z=KQ+|3Mbbdn^0A0lZI7k=(M>0I#dGVB=_yVelb6bM*DX4X0
zMz8yr{#9(KI`9*=Z<|WHayj(tl=3~nx4(HXWaTdQt{skl;1<^G4Z^kaOun4i5jUGg
zjNbe;+ora|LF&fP2uimY&YV?WvT=5MVX!S_oqU-3D%%$<=bX3ne
z4>FM6PrqdUinr<8Nf_!K`!a6BU))PmY1!)wNlx}Nec%W_JDWjR)Hl9j!oc_0AswLV
zaDLk`jbVMe)3rx`M$g~H#ccX(79w59j%`ueie>?})UdbEIg_l%fGeso#{%zacC@Xk
zEc{GIj!5G2-d#jC8Oz5T*D$k-9mki>W%Iv@DBPZ*p?45Rems*Fp`<3{r$hW%=4qN1
z$#Mc=SNyV`h4bbxao8BXwtJOnL+Yi5kE?{v3I3hk$I38Z!Ahod^-j@4QQ6<7rCm<=>{3A}WRdx_b%r;?sQRaSrrf|S|YKwI&D714FO
z3c}XdUJx)rHaq(WqIfg)TCbKcw-fk@0uB@1xvC3Xl4~4=5>|=#cp%7^O*weF3DngJ
z>rak_x28hibAoUV2FJn*EyjZ|L2$SH3u`ur=Y9Z>zJupK6vS^F+!bVVglJ%sHm6>$
z5po5UfrWG6iXh8_gsdairpmo&0sJi9@t7Ek{uJjsx$FFvxMs~61!DWjehxjcL=;K2zW4rP`_VM(JFQ+CCXnN9p2>eOwH&`RvrzHM-DR7>xTQ
z`-?x#-d~;R(7_k39bn~03t6~Od=_osm+%auS#{wIf5-mI*7rv;V#J8l&&Ww^y%5QT
zb5~Nl%}O6(wELW5&&fD6@j}raccF7fUzDl~ocTMJf49ER$n<+hjAYW*3q)Qx$CVtC
z;Hblc2`}e~r~Na$*eaIYyDp3Vn8>x=JBfVmC7L>T;$s-2ck&BN%mkE8KHHau&Xfb#w9qW2x
z85fP(!v~l6TLgxtw+{v7n)KtuG%D(6O(CIR@1!Cdv%
z9`*-Uqhlw3l+oeDm1{&mrD*;*c{O9kSy7HA{a&SI4Sy<65vlkmImg_Q!aGPjY#M{LXb{jEPNvd-7lAl|L^C`Q+{bLy!18n83p~k@++wz
z0i|rPR#;$RX-DZYc39=suNA(u9n^zbPS)7@wWsaVb_5090xLU8*;)#Ep~R|G8Oqp6
zg{Z)##bAbZtc$0mXy0lNd|cwW6&NZ@o1{byiI3t@Tb0ADN)rP7OB+xhWJXFqJ@GBS
zc?@CYuDz@s?oXiTTYDubRQ;QSz7TKW+JL&QH#u_T@3a|IHP??E;=I$-JmaP2(xz$5
zJz9nyGuN~E({a=fTF>ltr?X~O*GHrc*s+tY#?t1caAy85k|y15VcX&s0Sg{*;le%o
z54=U`=WgQEKZ12P()9Sl1lBGLqv4CUac*~;>Z7CBekb+Z{mb`xb94~i?QY@PBbYHi
z>v5Jnc*LABcWC%QT#*`Z8BpN-{18U8t%iq_C-q($M%N0rIet74wZe^-?HiDA@+hfI
z=MG1XUZYCewiL4IWCIHnRyNp)w(jgx9>qUf`DELl8FrV#o~DC_(z<39{Jk>AHd@)0
z#@0%V6G|&el`Vsvo~5=)QWvSDF0#hny(&$gsgj|Kl)iL$tv7X?Y$#o+BORW$CHUs8
ze80iNi@RC)&R|~b+Mae@`Z9X)58RRU`0yv@zcQ9hm(nX$5cJ(s7&Lh&w?lU@w(~pe
z3{Bv~{7$rNJD%N-Lkn}5K~;xyYQuEic(D_0TDPOyfOlASC@^D*l2CS=M!&@5FLpBj
z)y}jXyqbRt(WVrFY&oYHT^6>26pFuH(^;P=dZG46&x?2cr&QwOKT?n~p`U*jrkh%b
zZTXZSUE=r&p|2|n-TU4TWQtrlyG!VGp%4}ba^!9(mLb||zn7twkl(9;Aj00@@~n_c
zj3qXL5cd_tHZF~1302MXq0qup5Kd7C6@|GTcRTf*%p#m13O7l#EuSjT-e2oCL>E6G
z#+CPl8r~!bXY;aIq%a(wK%y*R%+`D1VQq_zwjHZbVxd-}&aYEaS)TTty?AH$aXQvN
z%!O*586kS5g;fonYvYmTIk&^3tdjIC$`wjW90P{1XhKt+bpQ$!3X3$)vOUdSXuwyS
z4skK|G?!}iWTcD{Qz@}@3}DEj2{hGNM4%94!$O&TB6W30FmbNW;3VGFx@&JLyz>=5
zwQkODzk1R0BR`Z%6`roKoRrh96%|-pTcA?u4vHF7k5p;DVTF%Sjw_x%!`Dj|GX3E+
zK6+6cwyelA_3rWA7l)`nzB7K=xM8hrZOB{&7N|uXU{MM?VNAu!P-QzUt)sq6j9{)#
zS)ODpP~hEq6f^qyYRwMZWzU2@Y)${F4(8~pMXdcX5RH?U801yNCTq~BGoIn*<)zd4
z^c=HqN{6POvG}Mk!U8-vd@PuHJzM3OL}_jv;G_3WqkI_7bzxL__IRIu$p;yIrzQhG
zc!>rsvfLf%GI{{N4OqqZr{ADKGhG8X^;veMY_Uuk>q}84<)Dh+_UtFP+KY_xR}vi_
zN4Z7;RNDSMC$A>bpmu7VD#|sX->WUDVW=>ROr*+NkSaF=J-7g(1RrJ9qP6@ODB7!+
z2TrP3l8e|4t*p|MQ3C>5x?~7_HwVzE|LY8DU4e(ep~ODA#_mNQ@ZIC9J#n_xWatWE
zTzYxwQ~4V1=Xk%8Dm_OrAaE&5hV*4~KqvaYK8V(S1Icc8q
z1JYC8ToB5^DdG_!2Kgip@D@h#`7r3I$5SqmZfivflG<|(^qMS;i#i1oGb?iI*NU<@
zIHYSk1$K5esN=G;{?r5?UB#lcKN6^M!pp-6RjiyrQ_D)G=coY$E?vTqzMBc?)SuS}
zwZ^ZI`XDDLeA_apM=hMhvz*$u<>}8C6L?FFDr6tu=BO_OVQF2L^_z}zPSt>h7MD2k
zb2xQ+x5iWb7mM}>Q|-l3_$#9ck1I!`fJ$ut{v=lu8&F%PJhX|lw;<96?b)KSgaya5
z<-8L+wm9MRcMKgSM)CY#HcXu2&FAG4nKd?yNgE&X`i#
z8M(@qmu6V8Xl^t=*L3CB92=|!fw-X1ah3_vvu0L=FyP9tb99*$11`ak@1hrS{P4^e
zlay|_ia|6k_FhV8w0+YKH2rKbM^DES;BlB^!S(3bs$kIud6K1ZKSm9zkn;Sv;~z3`
z(rR|MdzyYJr`;%D;EX$#RDJPn247#uf`L8w;+d`t8ulvfDwj$m?8A03ZQMt!s?(S`&%_aND+t?185Wo-Hk4bTv8Y}9Gk3cO
z%U+vI_$yyCvyC9gGH0Y94yiM!&6&RLTPok(&y3Ma`E*x10$wf8`Ohct{ytY;owAB~
zPg!y2@Tbh4I{~%B2Ht5T-%CYYbTGf~K1TO97qGH|qv42JP7o|{vLKCR1UU=rB5XOK
zZx_D}8{|3h`>kJr3q4`)4tQ(6;M>lyJgrmL6~}8Gg<*p1927d(-w=7YL^pFTm+`lq
zuqm7!G7=C;GMqjDBn68x<|Eh|2p(R7*vd^FWJeIqZuBTIhKTp7+ZQ@?ggJwRtZB_h
z&K{2c8~TX%=~x^_s1{KsjVSMWCY2~llSvZgrKeIaO()74L5KtIq!Hx{spqtba%Dl3
zGv(dUCdvznm?-n4lS&H<15L#$aCojePj6enoNJae8Q&9(u0katBpDLMk^=`CHn#k;JVz8c5H#J1YvNq2=^*KKr
zjih92Ks07UeY)W$J*T^^`5d#c&Wv9rcYT&SCropQ6*Yo?NX241Df#D=8MF?SV@$v
z0}Vz^q~hM$%$|Ceu~T2BcDbxlSy*IK16EaO)$|_oRtZx2Ui;Mba+>>`|7tylLNxTv
zOSKp&;(lJsk{|3Cx^kPSdn+`d+Zj9fK&ERfbPW-`$Lwg(p(S=Rk8;
zPBj}I3MU#2n!|=3H#o6lH6PEP%p<1_ye<@j~z_v
z(-fk(2QgkPP}NN
zGIaHcV^p#sUjzniUud=SIA?U@N_^|HAeaiwXjy!fmxD6fA2;@jVOvr&t*5m|wK(?JYv
zU9$+X_UnfxLFijSRD2y%2~Na0LDFQBvpOlwi$BDljXmInUa(`2;Lcj<8Y~2H`DiwT
zjuk{n=;(vv1kowRh@zcqVY4XylV5?KAX0*G54sj|<`{$X0jpo)^ZX
zp0lhh#uZ`Sz9k50WAS;Vd=U%!1QKP_Cza}d*fe$syB_ODE1rIRSqjt74R)fp-aN5~
z6TaPaX2$G9N5094Jty$((IhoQ#JVTJu+4gky&%Jjj99lAB$OWcNe4DdP1H|AM
zK1$yn001BWNkl6NtHHcD^Wb*sj{!wu<}9*d%Ipyh}mT$nbIx6d{u
zK#Z($S5I&%ssW>>S4vsCsDE{unxLR4T>R5WrTEvcTW_lWbc9CTE2ph_RO037LgW>>
z$jF^?P2XU|KU0`G`VuXkuZiQM8~k=E3fHt}sBeD5lsRYdZT=Jqr%&SAu8Sz0w8cw;
zSU&ygTqf=Mka_Qa!mHEY%vtAVf;H6#Ok%)_xr`t9KRn8pBPlu>t2!-dQsEzLtP3&f
zlqfW)Vg=ct>JRmvKRb9;W|a#D&e&Nr-Uw5OHxcmUWyC%?9?E=6LqcW9iC-Eu7K
ztJCU*9+^U%?8=tG@_r1_(soL3hDmq18)uEZL#ezQs0tg(l(M39zz9}OYNm(OrL|BC
z1qQ9jY)I9Zg0P#yK%*j77|d4g4y0D|IrjW`S?GU5O^fM=I`k5k@S}!@AI*Aq;^>|Q
zL`Nl2s!jm464$U}=Oya*tBPnXgrEJ3D8GhO*W>KbMb7sp6ME^A7@ce2;n$4bosP0+
zK{Qcvuq(o1kNriE#zQby5X2##qKre}`N3%?v+> A2*un>rXm%)<3+
z3u_y!l4FR95dA`QNR29%*hf~lJCqUKP{ZAGd-yT^VZvfdEp89|n)RmB(LF4Pj!LmD
zN)>5FpD(6;fXe0=l|PFtNo(^SYMj7mjZ8A@4|CSb%$Ch7B3WJ*DvJ&`t8
z6BD|~**^VmiV3waRnTX#qcGr#a|}!3;=!-^qs<5coHT?S-pawCigaw69#LaW
zjrOf@9>0~BvUU$Q+@51(6;a^1?m1|IwT-Cp=tyK&
zsgT9gQWNs;MW|F0&q^)m)9na5KZqpKpb6xX2s-=;^Hv??MzS;Bo=zm(l@SSrh&jz>
z232!El8}2&_*Ztw7<(0EC|}-*_^T1fK(2IB=~mvmSPgcCAd|z}z~Gs1#8otXH)t*h
z*YBhfoPm>5b*|k5p$@AFvh!@_`z^iU^I3vaP7uVcG#nUFOoEFti;M*uA+JNKs%F)X
zke0fHKZI_rCHO(j=3;y}CO%PQBtH{fwzZIRX*7(Q1Z4$@>sfNaP?=yVMA@`SrKLZ;
zr_JU0Olu4+DCZ+ytIV?uo2bHFd)nQ)5kscb!a3!wt*P7pU3&d|6-}}d-_GM1VV&kO
zu&zu0cYE>kRjtr59uRmdktSVR#_73q*&oJV{CV0kOCdAeb
zrE?!<%&gAu=kAdF#ypzVbQ8Z4Pv`L?uud^n?eKqLIt%JuBqAw3Rjv`%$Ww}EhcQv*
zPUb2Y!jxJv9bX*R|GIQKq@V!eH{PXmg3-oc${bzlPPx
z1CJ>XvLjVR%5L;Q;Xu=Ft=TbeCIkL*!%pc&vpSV1Xf@{*>fl4noqT`<7iX*=#E@KV
z5L259n)~c#mAISIe@?1gr0S4Mx9EQ
zq&!U>%%QoHIgsFjv-JaFlB+RjYEx2*+rauDH{?ZM2s%ciw$Ea@U<;cpZD}#3PO2eX
zsty6Q5?QljCztB?BUMG5<*z6|8dfitc^u4vNEd(7XAL`7MF|7Wk&`>i3Ii^gz^$Rw
z+%L~r;L*r|U0cg?{+IhqSR6_BM@lY?OEvv1{mZem@&n$Q7fIt~uDs$?T*OjI1@Bna
z4eP^FD>X591(|m
zBfGq8D`7>I<}L93=5vO<_$>`z`Gi^B?X}LU2|6%?3HyaEaf&)JG*;fr3qo{$d^f}euYA`+r*UF>Qp52=uyFgn-z9tDPwtGjA7D5aAF(Dqj_!(hH$wF~33m8|3B_j`&(xoAm3
zy0l>Y`uYqA3?tF8Jibm=sD_C5wHDe+QH`PNbwcKpdX8y9u5Y!h)`oU@m?6&%3a3`Q
zH8GGm3*V#Px6+XjON?bL`c8S1rfF>=ivc{PGtO3(Y1-U}-5W1^=S%q(nhRoJ>^emlMIgmK@Z9Ns=|XcxC3^+9gPi@r#C
z+d6G&X0wl-WnSmKr_zjLg*%QK5_)A4S<^
z&h&IIVq&R$iq}@}<`tza%J>j&1U;aPw;vv5blZoDM=DFIwGlQ&Q8gXQr+8!2FT9c6
z=^7*0{8Q#*R&~d)UF(=_-4U$$x+B-az`c@7iu0lFnC+xr(s1k+cJ;kY=sgSUh2C{9
zo%S+Wo^8XkllF2bpRb@vh$S}O3U{Z}D1w+X-*E7vpu;l0*2)`i*JytG^*nJ6pT;iY
zBs&k^ASs>HO6P0JM5}bZR><9vW!H$70SZ3ax|uT#hEUI5!GlXX*m2U5rV|?(r2Sv~
zUXVkf7iYbbp+EBjM^CNnt=1XjQO=#z%UZ72<1?LhfFKSGKBYis#F
zEBhrSOJKkdmTc=o$c;P1Dx4@^!JSO@hpqvfLM&{lJ#;0T+ub0-!WF-igWj|mzn14-
zzs0Qx6?U%W@pQ6Hn}JYLZRmOovBWSF`N@`4l?;?+J;|tMYYJG+`GQqh5B$VOFRY#L
z^)JN2iEQt;qO6A>+5K!@kzY^wn$Ka+C>zDSX`
z79+m+5#0zkrFR>~uYHc!Z`~q7Wru5dPn^;{9k%V5^If~NE9>V-q>B{_CEa_J=HuBB
zJat+r-qn)s^{lb_SDaHP^5u+hVZbSc0jCiLoDEZFmC`ZbtaxRz8zJw8@#+KxrI$Ds
z@l1z8^og{#Q^phjs`{=Gh(;S38t1QJt9=*f>?BdbVoJigKQn#HVO}TjAK8
z9__a?d&*0kmQe)m&r#3cl7CZvx)wbLHRI?Z62IT4AGDKs8S}VrtXdG^VS*g>5_Zh+%DG>!3ro={>u7b%$NEGN%nE+ulgH=$
zuo0I14iN_4b*Q9?GCAxbQKSQ}EXEA_N4&qdyYL?8r)=s_@hR5U=Ig%A
zvwwalDeEg}8E_U<
z9^lD?0jb}-o_FEz2KK^$OUBLDS{QK2X$G7HOShC~?Cm6Bz$x+fwahT!MtEx(a5hX_
z>6(%_XqLRF6VR&>TnHG(9C
zbagB6>GA8B~Q!6NZ2C
z4gG>|aQm(W&b~Uc%$7A6p*y{sCs^QA9vT;QvYEUO2nd
zwa_x)lvJps%%cIP;fHmREIXwlBDq9@c}3G{M88R%^l`6DMJIF1$r~&w=jETVlt?>@
zfYh1PYLC4n);c-UZE&hsJ)hg6$=|C&&3x~t`(#2P3cgVul5G|TTz$pI5DyYvZ>}Je
zk%B4Pmhhf=w*K(m0HK`*f@ej;KHE$x%`n3#p`ff!^*kk6F~baFg{gYVjSpshkRRAK
za^%n=HENkyr$mDqEU4SYp2>}qnPOvE_?eD^Kv5Q->gJYXh8ZOa1^8D(e~y%5hSC87
z9E#@I1|gL;Ha5h^#~b2NW|&c;kZoBTR#qip1>_yt@)^6Vy3m$ap(wg(tfG9j8D^Ma
zd{Ee1(a7Ey^q~c=6>JPM(_w}g`GN^-%ceJ@tuf1OYVsBZa(-WfSXY_-
zeMHuYsW|Yo%h(jkJ2&Wk2fHf8FGzl&Pewaqn9Iusc28wJW-G=J3uAC#6$2EdMCZ>r
zM%y47KpL}Et5(I@+8nQMhG{|$i2nZmSxBV4y*)K*)G#_jQx4ufK0f&QWo+#t31z){
z^(b35uh~mV2GZcIRjU?d%4BT6Ey=vEudhLk5=khfUeX6ycE(85%9k&%*D;`7R>lNb
zhNVjBtQB+XV;aQLLKtG+qq7{YSg~U9%VJ
zSXg^NEg`>k#*XAtHw6X;rsyVPLg~__Q=Bf2j*i)0Bl}KRSQv?kiNz>c?vP`=oB)w}
zR(I@FtJNv}Jt`{7D96q0ART&g%qUkboBh)x_rk5Y5PwB{F%H`W3o$M`w1kRt#0zBO
z6w)NX%~=q(MO4ncVihk7ER;~TG5862%hc6>O|lwpFBFacb4ZLTUipdxRzf#<4HKk#
zOg16Q889fRG{X!t%rK*9AW6XO+qb!Y|9(k}l0w)eadC5VGwN9tGyVgRPK&TFA^M>B
zJE5ed$p-|nDP0ph-^ewj%nUQkFvApJCY5HGVTKuIm|=z)W|(1y8D^rip85449Lo#4bu3wc)3rs0^
zPxf=!-*e0#5O+?D8=rwHxcCIi7IHoN(?b6a7e<$J&X-1i58+!&eodv6xm@GaFm+4&
z>@SP`1wuCnG8A2Gk`)`|no5t?ML9Ie^-_a`zsd#Ka^b(RDG@Ww$Pdyps_JHXm5vg*
zfZv!5F`0BgCQH!G3M%_xbab@QC3w(=OeW&w62F#0cW0r_>09aTTZCF49dTi`PlXM;r6_ekteT^
zgJiv9f?#IN4*c-pLxO^WOv<>_MtXKm-EDIuL6&QBjV*ynejpRA=uHxn7z4p8;NA%%
zObnGBgt1zxMruY$yGmKk%Dum(E;qLNlB7)PZ+%2YlCqGHkP?t8@6gwVA|oSH%Gwkm
z>p|KrdL81?2jHLXqE8svWRI;sJS|9=bv6mbZi(^Z#{0!9Q*J0-J9W&k)RQSVbUj?s
zlgajpg3wph1<7~IIWw<8NTtm7b?;tDdlSqsqwtU!k`&9#q3MQ+B$UCyhWDgO<3Sn`negZ)oNg$r>_~(EUZLST1^i}bJ$_@V
z7-;7dzdX4??pxI)C}W_FzX7KzV>FotbAoGMsIfK6VN=YEstf73kz4d=!;}r@bV$Nn
zsZyoPd(Ucw!F>J&c<^t@304?bIYF&GV3V;&vUF_7Y^25tNnm9A(yj+N{@l26!{}%C
zb=!~}v<9t+w!$}|GBf=7XbHx~~Lw3~<
zg}mj3JrPqfG8HGRg>Bq8P>lQ8NAwwFO{J+1z-E|{4(T1%hMYMwRzfKuXHe52XD3a8
zP)f+zNEsrP4jGbH6p+u&;x`LgVVs0g;?5!=n;r+OP{j{Co15`LEMaXx`qT^x?nl7P~kYVj%n=&2J*6YRyNyKy~5+s>4c0wtk
z+lHQm{;s!gOOj)(gi<2AF)i*-<0O<4v3g{dQBVH`LMb6bs1t7Hc!@zsrKyGqnPEn5
zFt*SmU1-Z^YU`P`MCzt2PJ#HVMe$@4p*
z*<#RDnGesxYeJ5JAHwF_#Uo3eQ5;U+|43b!6sc65nn#!BZRyti1zzahm5$wBVanFC
z#HJM3Jr2wq$>;wX8+}RQ(idZx@ojAGmlpPyE`7n68Q-G&OPa8~Y}_59f8zvA7<<0h
zOITraandEe7&DcT5=~?8MBY##S%+NAms49R2PWuEyzvTd8hx>boQS6KNy^
zn)(j=7mVetx87pRn6bP&X%3%!^C#gZ$$9f;ROAmD0?)x2Tj<*iJWEmwT0+a`p>90v
zy`0@eNd*2GX+rX*!nDd*7aGF8ok-6Y-(%JGZTxuABs-7sBH<_I^?Q?b7m8?HNK9bG
zhY<{%5y9B~#WVIKUB6GCVNrY)lX-l2_*X2i2{{JNj%M?%q6yVay116tJHN}Wk{pAQ
zW4V0v0Ds+c!LMR@994hw<@o-LSa&)5nG(xd^cwdb!`u3%Y?_{Qc{8KCPh{W2#}`nm
z*ErrA-WFej?W$h9C{edP;^Fxy*IS02Tr0MyTAA7}|vlUKqNWlj|)k{`Ao
zMA>^W+c$mAJI@tdf0bO{`42;##IYkPN(YqYg&NA-uP;Jad)v~djWuzobf!zy4~p8suDKY
zJVGBzEMa@nWj2rQ&O~GC>=xi(8(LJ%dMZmf>CwxEf~7087lg_yH&vPx_T-YUeAGoz
zw&6V2MFC)1q*4dg?oH|1)Rv2XYh_DJ3gy@Jvlu&kIB(D1$jPu&wiN%@Ha^|+BVWxJ
z#rXASNKe3=aQhe==8k9N@R7VXe>1-Y-{I)S9h?&cOcQaEwUb9Mbo30?p9t3qD^(Hp
z$N%_v)+8oQp3A4Z&JvmDu#+6_Ro`I$@=ct|l&Sjxr#CNW-?h~DP=)=$|303@q=}Q6
z`{^#u=HZ9qZwC<*yq|UZ!ZIriO*s44?%{USZEk7nz=Ycz+c1~$BZo8cz4>hZEtoq;
zH?l((ig-fA$+b)#F_h6W)^j2}>+dIp9Od)HpK~-s$gIB4jX%#5mIx33+{)@h*SU3c
zEz?I2W$36Ge11G6lNVES_X6L3HjgRqPiD?1yEqpa$?08tI1_6mvw(TR1OC}Qlc7D^
z(mbFM%{mTd&bPNx-ZSaqXAEvXhSl4q(W_}gdMy209^|vZ?HRLr8`F9f!}vQJ@L0qTc|x7E~F*fJ-rV)zX9G|g<>8AV~%93
zTR5{H+Por&R2QfyuGym}{_DgSP)(fMQoyY_^DA%MB}89_D!qhUJ%xN9Lz+s-lf3uO
zi$Q20y#7PM*JqfN%*k=Y37zQZ;zoWcNfzUv#QIMDVZ^&9S#j3RCrW*3P(jYW1s3AEBd!*!oItNZ63n2L%$;E(NU&8
zTbG8zw-YVwrE@F#(|*D}Qp}>bfBD~x!E|cgh=$KRPtQ3&l3^|-6LEAtgId+6cFnr9
z7_@|6ql>vgYofV!A&}S{I(v!sF`)?OzXQEDz_2%=tkAjc!(e+LtQ5M}y#v?^o!Wmt
zsY{lG4qUqg8ukZgalE>?cYC_IEzYeI=V}W5?ATpgBlL5Y*aH2(8;o|PL-Ett@mRV23?9qnxUzCF{w+5I6TK@T;_6id
zs{N-VJlQ!C1D8L7Wu3O@)T%P1*XN`EnEiN>ZANADti&F+vA<&7!GQM;tSaOPHtmmu
zmAw>6Pj+JDz~y*kSqE)9wL)d-bt}9Z&0;QC&cXDiM#1W0wvw|flp*R
z^sM?8{{AUp$RA-VcNkasIif}X?=ZMsX@veX7UNfL$C=06!I59V%EkYp+>nXzt`vc*
zR|7N5;f#CagE50wPOotM=#7*P5G4Xn{*G%!YN1i7+B9o1^qN85o5aqOA!7L}HUeGh!KHuf5sLN2YwcO4s|R?RwS(ql4w?xtGz3ixjF
zMzc{Hv1djn)UDA0EAHi`Ci}psKr9~Zp9Sxh^-!}$Z8YpS59hRO>zG^Qalnh~z>V_2
zgTug~M!>eEfcIa(#$SL?UYQ&6iu#A^>SV%;x4dB<${ivf;5XhF>|DYB@~A8PxD&%0
zS2x~J`*A1gAa4|$KFF@eM}RH|G{bFr*cX7joOdCBnJ2$*BX6|0@t`^bOudpbd6WHt
zKC^hk)e!ixvHHW`e_Lkxc}#885v%Vd;O?q6s8(YL_C;jhQM6&-0`zVu7_?Q;(0etm
zM=5j|%AC0S*f)D1IyR{X_nKd$h4(xhPpN}rAMM91@0O@nvj%E6?1*{iq7bxaIJ`P8
zMp_?}3c>rnLz6xm@G&EEb<3Vop((@g=Z10U*3<*ltJOsPwga*7*fVvVBy@6(Y}Oe|
zZ>HNC=;>DUZ#Ed)U+`8z8G=izzeC4{wNSH86ZDwugS$E=*kz7$J}7mDh$dmth(L7y
z=^X|S2!Lt*SFrsy9Q$6#v2JD%Y#YAfMqLnk9!^Rzrseo)Nf>H;69B8G0Vp?u-y4|H
z2F1y-WpOAfwDE^U^8nOX7N6Oe=J|7NTPVtUzs0Vgf@-@M-u?R%#&q;V&FZz#xW{z-
z`dCx1f2#7686NJ1b(l4MGKP2W2Oq;>SUjQz)Hb=!wl5Ts@gmrMvkIqIcZ8imT}(f7
z1s9gJho$5!(|irYAP*$$)kUBivTlTiNI
z7A)KIEKjkByT1>!2X;i0dhV$CbqjdUI}Wva#+}BD_5-l#VQQlBc6r|zv>UYtuY!Fr
zz+)ix1|{OsbPrTk*BhxJX5sK=p=0W3X>-Oug6K%#*jb&y$DK1&w^R~&)MygmGfUvFvq!MpM+FLJ;a=Ts*Dy8+&vJEJo{+IO3QM|Edm
zLSs|SMHUOauP0)*cOz6R=?eGmBhW_t2QK+*+#Z!%M~wQ$9j-Pq^}MtFKkkS1n2ngz
zyAi5=RS6y)$6?iEcPJwTWmh37SqTG+VklX?BgTzu4=0J>_Lbmm@Gh
zcpgp={2ssQ%X<13XxZr
zV$g*1s4!zQX16MV%$++7v9kL3evUU9R&;@D&F&b{R*I{a{e{VwIJWdBsQRzPf^VCk
z##a^L-f{?jm|PFCsKV5J9*@w+_i*>_-L&IDATms|6x*Wth;iuTUL1Nc;ZQiZz&7qV
z{DM;&B9S>74jP3PmCM4-(JbX55zHG7!l)LNQP#~7W>Pz}=uj7lmrmh^@D*A4630(G
zMuoO*;hNhDK^296PX37?>x!sq`w(k~j==7?#+c=E8s`qJKo3)HL=XEBSL4K58upGT
ztE;iUw+mt)@mSkMx1;b|4dt3x+y?%ck}US1Z>4TtHx!K=@mHy&q;j}!UJkXgSzrXAje
z!4(Yf)zH1TbYU6Vn`hTibOHM>+hffBYxwK%M3nQ}gr$35=`@s8v4IFPX^x5O4&e0h
zHE6Foj5*6st4A$tn=%i#T*kxa%vl^++)a32g$=|P`NUcW5
zbstlDbjmIybQ12a9f}dVss(Ep~Ss)BQWVy5%gYt6lV_1LG4$oFmCx(WEvQ#HO~2M^)N(7C~)kw9JMAp
zApD>WH>P7Tc=%hKDr=30XB=_9qX@rmh}Ia>aSCq08DYsHJ3QT8lpAny7``E19nPKI
z;TU;G4-3~j;Mp%WnC&6WXiRe+|Na(T_AqoEZGj#RqI`~v4ixvXcUKr1%-V$$hgPAh
z@deDDumN|I#GkJGXh9@1fU&VDY@D4SzP$}ge!hYCO}N&!F9n$qObqmR^(uwTxCl&*
zGFuSBzRb#`c*HAYFt;m@2KCCqDD|S)1?`6SLN#lBn3Qghw(f=qe4dv*OjU>t41-DY
ziCA~wG>)%n57ps0SS~2LDh6RM0}zprW|JgE2H|B;BowyaVAbi>=xAqvy3^0#icoJ@
zrre~{;mumdG%wKRjDH;E8*`_mBzMMmo!ZBPJ2N~hrgY$SFi-N@Uc9d5#l-h5Pf~t;
z=0L#HjypNr8Fl8bj!l_cm@EuLa$+u0ob^rJ&MTmA^UJti?}E%RWF(5%t)+ZRpcG~A_S*W1Pdn@
z6iu~jlkVW}fGTLyz$$}hZCIfR8n_6XqZ`6=$hWvSt`B-%^Ty=yT~XOw4EgOF_$T54
z!UuK1mUPdlqQVeu`VjH1p;)!^Z^Q@|u(%i+4jY8jPJ`^8O={%kz=QBX=#ueaghiX;VNxr&YfCU?2>e6LQM|YX8uV+4ZIgX)
zzvUQ|_1lG=Va+h7krj;mEfM-En7fn?xbt@asHGT1{Pr!6nlbzKA{O-(YjvZyuMEOQvJ_Q9r1x9pPXtM&t76#tZ{cApq_h>fOdEnrO%~(O)$#CX@?pa+geB8*PL6AP
z_rbqnKTIhnMR;foEWWM}x2*?p>0uH)YG`s%WZn>iMzuiY{2D&Ojf!O6lEud>ATuw4
z@(t|rDPW$pykbu+_aXycsq_CAP4!5)U1-D|p=6;d;T8CSjlfi1ELGfSaOU?2>tiVS
zd1)D71$UB$E$8QbfNy&O(|H4KAcVo6UAG=EZ3N7#_3=G=IhL36{30-jH>?kNy;PDr
z!HL}QQmS;zYmySookPGMJim|i2W<193-Zg5SqeiF1IWZ8i1mzM$_u=No327f5mX6?
z2zr&$QFL3~QPfl&v65)_onX*%4YuxpcC}Vw#?$9qbzHURj?nMa9Gc)F5cEm`2Nzq!
z-4<$CM_$C0NQ#e#LS~K<IUNLH-P@j<2doS0@^Ng{rE;)m+bOD
z?A`woRfny|w;s0Y@@my(8irhIg2jifV!VeJnzycrRqIaS*7qK$D|&$Ar$bQFyEPmY
z*DzzBKPvW{f^yPugvMCl>-uikdhil&t2|I6g8<1o;+)@zP`;@dZ=ZzRFhrlPrPzGQ
z5K}tq!-^YYwHq73ZBGJx-!OD=GRDlIJbx0E;64PLog|1=_w*WZ1CDWHnmZm!5iE=A
zqg6@+pZe|U?e%bUT&F;%S+Fu0_;or<|M4Q)rPz~0vG=r3H>+VjjI~K#x
zx=e~IBQ}TS`^no}V~q@;jLXYsSTSsykH9$Ih9*P=;g!MxF1B&F?H9}~@~obd!HCxz
zdRzpl%os&dlwB>sdxuxhF~Q+oj#+}kD-X^_$C_xIp7+jp{xqcKMbg?Iq0W`?-!`fB
zvOd%!`4qe!R`4iK{1Mpw5-7$UEL*Mus+9V$guR2
zwN!E>$~B*zWG?M6e@|`P_+=KRj*P>ytpngB)q_>NG1#`i3mIM|yi3B5m=|c;)#220I!S>^EgU`
zWYIpZTp56X7+ZL9Z(5??4Sh@X#nuyT(B$%2IQIS-Wuy|w9c>YLQ5eeQ2z{9d^2~_>
zd7?aJOLiGt+RewF+Bfjatf?3ow+vgo`Dtlsc-D%>uWNrto%!FOw4oMRggnTYL6C6m
zQK&P>7{+?<5E;W`VU*4iR7N5^QUz;m!4*M!$ugp?>xFV%Ag9|%ALW3>Fy9|
zbtpt1o;3DAhZe>dbMhi?7CVi%^+%zJbUt_R$BgZp}C5Z+n4
zp|nL>IAW6$C}Ev|$6+x%r?qBWW>nD#2#iF$?=DOq_iNgD$D%My_tjQnS0Y~**pXJ?
z`Y_ZFvuCzNqmA{^Y4B(aYh5O-hqM5Q=8f5~Gr$7wAP%q-Mj!wUy;CmoWcBLH9Uq@Bb7-%}@|#%R`11zGgg3M=Tl05v6Jl(`F>snUw52}m*8P8xIlGP`u^B8aJ}_uwl68f~gFA^O)0g9@AFtaS
z9bhew1QRR-X$AD2G8BHZXQFMRji}#gFh&n+jWWg-Xw=aQKTbb^t8rhW__0$6aPN&)
z4*B4;=#X6%9fqKH)^ICrkyc-cO}JBFjfBTxh*61Q)1)mtmMy`_zY@EFd#<84$^
zZmC6|Mqp$-e0NPp#_GWknFfREjPu7w?L*AX#OaJ_5hT2Qes4@0h#*L(Hl~B(FmYid
zc5}PGn1d9ya#hM)E)lAIV~MHH!!fjXAbzNAg2{u;(cLvA5N@%qTayS=FI&v3BPrOS
zV3J~RbWE#krFO-5yUE*B^=HCQSbnm=kjr4fZXrt8Z%i$k_O>slh>Ln)_`2ih_4GWp
zFP?|*hbF-K#EcI*DLxhj0!}W$^yNq4r?Q5lgEi!lj2rN}^*LK~IJ~Op*V*V8)FT@{
z&Kb`;2Y&*5c-wV-FW|!K(U<#x3MW#;HU$cn<2ZXZo1
z!#u;abi+{eYoKuzU+mfQ80nTmLD<>z-hr_{MWJ~Q
zca52^R~?=ed~x(t7{sa1a*H(Z*9jQDS@)9Ns;B^j$QVD#tz?zzjX~+)s;cAN@(R3jh%}Q<9X`4x#RNhPpj~IfC>dPfG8hOAu%!%
z2{Ln76_KeY?%&#TERXy2$xx+zE7)A!j3uX&(4<3scPORm
zb4F&SkiL71x9L~eS41M>ojxoqjdNZhhz((CEQ4wNN!Wkv7>=bLM|NU#&2+9(RtlLH
z#6{sTd>xLTKZvp;&soV7IGDhyvnxrEVj%p1hb}ISl-rlE2DO;08Y=Jfqro
z0zCQOknpr;EqLYs8+R@!KnPszZ*r-4DoIiu#cW=e))~VA7
z#3<@99P5ss$HBQxk$m)f3|)F%Jy*!sqXSwPU&YCbH*xgzTh#B+1i9#A$~9smGnh)>
z;cdps0+A8#pl@LbW8vNAo@nc-#FbNj;_9*UPM)-H`u&}a%ZM2cRHu=Ht%)8iJj_vUFFW#ziYV6>E_g@
zI>IkDf$Ofu5l+)xO{cF3k7NInC|rB1XO$ORRc`UNE|JS$p&pmL3J?*VPW}SUi~pQc
znRCqHHDMCUpBj$tW0vBl?c1<=T7P_ZNDt+N`5ek;*g3QdhAdc%)pHl&qMaw)jd_Au
zw!ow@_SiqD4<`P&9-G%M!q{FTaNM~sy0~Rsapw40NeQ&?RtfrUUC=3Yn5EbfEhdeD
z{r*AdGx0~P-@G1+#`eO9![](\"./figures/paddle-ernie-1.0-masking.png\")
\n",
+ "\n",
+ "![](\"./figures/paddle-ernie-1.0-masking-levels.png\")
\n",
+ "\n",
+ "``ERNIE 2.0`` 则提出了连续学习（``Continual Learning``）的概念，即首先用一个简单的任务来初始化模型，在更新时用前一个任务训练好的参数作为下一个任务模型初始化的参数。这样在训练新的任务时，模型便可以记住之前学习到的知识，使得模型在新任务上获得更好的表现。``ERNIE 2.0`` 分别构建了词法、语法、语义不同级别的预训练任务，并使用不同的 task id 来标示不同的任务，在共计16个中英文任务上都取得了SOTA效果。\n",
+ "\n",
+ "![](\"./figures/paddle-ernie-2.0-continual-pretrain.png\")
\n",
+ "\n",
+ "``ERNIE 3.0`` 将自回归和自编码网络融合在一起进行预训练，其中自编码网络采用 ``ERNIE 2.0`` 的多任务学习增量式构建预训练任务，持续进行语义理解学习。其中自编码网络增加了知识增强的预训练任务。自回归网络则基于 ``Tranformer-XL`` 结构，支持长文本语言模型建模，并在多个自然语言处理任务中取得了SOTA的效果。\n",
+ "\n",
+ "![](\"./figures/paddle-ernie-3.0-framework.png\")
\n",
+ "\n",
+ "接下来，我们将展示如何在 ``FastNLP`` 中使用基于 ``paddle`` 的 ``ERNIE 1.0`` 框架进行中文情感分析。"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 2. 使用 tokenizer 处理数据并构造 dataloader\n",
+ "\n",
+ "#### 2.1 加载中文数据集 ChnSentiCorp\n",
+ "\n",
+ "``ChnSentiCorp`` 数据集是由中国科学院发布的中文句子级情感分析数据集，包含了从网络上获取的酒店、电影、书籍等多个领域的评论，每条评论都被划分为两个标签：消极（``0``）和积极（``1``），可以用于二分类的中文情感分析任务。通过 ``paddlenlp.datasets.load_dataset`` 函数，我们可以加载并查看 ``ChnSentiCorp`` 数据集的内容。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "训练集大小： 9600\n",
+ "{'text': '选择珠江花园的原因就是方便，有电动扶梯直接到达海边，周围餐馆、食廊、商场、超市、摊位一应俱全。酒店装修一般，但还算整洁。 泳池在大堂的屋顶，因此很小，不过女儿倒是喜欢。 包的早餐是西式的，还算丰富。 服务吗，一般', 'label': 1, 'qid': ''}\n",
+ "{'text': '15.4寸笔记本的键盘确实爽，基本跟台式机差不多了，蛮喜欢数字小键盘，输数字特方便，样子也很美观，做工也相当不错', 'label': 1, 'qid': ''}\n",
+ "{'text': '房间太小。其他的都一般。。。。。。。。。', 'label': 0, 'qid': ''}\n"
+ ]
+ }
+ ],
+ "source": [
+ "from paddlenlp.datasets import load_dataset\n",
+ "\n",
+ "train_dataset, val_dataset, test_dataset = load_dataset(\"chnsenticorp\", splits=[\"train\", \"dev\", \"test\"])\n",
+ "print(\"训练集大小：\", len(train_dataset))\n",
+ "for i in range(3):\n",
+ " print(train_dataset[i])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### 2.2 处理数据\n",
+ "\n",
+ "可以看到，原本的数据集仅包含中文的文本和标签，这样的数据是无法被模型识别的。同英文文本分类任务一样，我们需要使用 ``tokenizer`` 对文本进行分词并转换为数字形式的结果。我们可以加载已经预训练好的中文分词模型 ``ernie-1.0-base-zh``，将分词的过程写在函数 ``_process`` 中，然后调用数据集的 ``map`` 函数对每一条数据进行分词。其中：\n",
+ "- 参数 ``max_length`` 代表句子的最大长度；\n",
+ "- ``padding=\"max_length\"`` 表示将长度不足的结果 padding 至和最大长度相同；\n",
+ "- ``truncation=True`` 表示将长度过长的句子进行截断。\n",
+ "\n",
+ "至此，我们得到了每条数据长度均相同的数据集。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "\u001b[32m[2022-06-22 21:31:04,168] [ INFO]\u001b[0m - We are using