非标准化疾病诉求的简单分诊挑战赛2.0

赛题名称：非标准化疾病诉求的简单分诊挑战赛2.0

举办方：好大夫

赛题链接：点击

Fork项目运行请使用选择V100/32GB的GPU环境，显存不够适当减少batch大小

赛事概要

一、赛事背景

人民对于医疗健康的需求在不断增长，但社会现阶段医疗资源紧缺，往往排队一上午看病十分钟，时间和精神成本巨大。如何更好地优化医疗资源配置，找到合适的方向，进行分级诊疗，是当前社会的重要课题。
大众自觉身体状态异常，有时不能准确判断自己是否患有疾病，需要寻求有专业知识的人进行判断，但是主诉者一般进行口语化表述，不容易进行精准高效的指引。

二、赛事任务

进行简单分诊需要一定的数据和经验知识进行支撑。本次比赛提供了部分好大夫在线的真实问诊数据，经过严格脱敏，提供给参赛者进行单分类任务。具体为：通过处理文字诉求，给出20个常见的就诊方向之一和61个疾病方向之一。

三、评审规则

1. 数据说明

比赛共提供约22800条训练数据，约7600条测试数据。单条数据包含年龄段age、主诉diseaseName、标题title、希望获得的帮助hopeHelp和其他描述字段文本conditionDesc，以及就诊方向标签label_i∈int [0,19]，疾病方向标签label_j∈int [0,60] (训练数据中疾病方向标签存在缺失，以-1标记，测试数据中没有)。

2. 评估指标

本模型依据提交的结果文件，采用两种标签类别求和评价，就诊方向label_i用F1-macro score进行评价，疾病方向label_j用F1-micro score进行评价, 最终结果取两者之和。

3. 评测及排行

1、比赛提供下载数据，选手在本地进行算法调试，在比赛页面提交结果。

2、每支团队每天最多提交3次。

3、排行按照得分从高到低排序，排行榜将选择团队的历史最优成绩进行排名。

4、要求最终预测两种标签使用的模型参数总量小于4e8(含模型融合)，前十名需提交代码和相关说明以备核验。

查看当前挂载的数据集目录, 该目录下的变更重启环境后会自动还原

# 查看当前挂载的数据集目录, 该目录下的变更重启环境后会自动还原
# View dataset directory.
# This directory will be recovered automatically after resetting environment.
!tree data/data155594

data/data155594
├── data_test.xlsx
├── data_train.xlsx
├── spo.txt
└── 提交示例.csv0 directories, 4 files

#升级paddlenlp达到最高
!pip install pyzmq==18.1.1
!pip install seaborn
!pip install --upgrade paddlenlp -i https://pypi.tuna.tsinghua.edu.cn/simple

正式代码

import time
import os
import random
import numpy as np
import pandas as pd
from tqdm import tqdm
import paddle
import paddle.nn as nn
import paddlenlp as ppnlp
from functools import partial
from paddlenlp.datasets import load_dataset
from paddlenlp.transformers import LinearDecayWithWarmup
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
import seaborn as sns%pylab inline
# windows解决中文乱码
plt.rcParams['font.sans-serif']=['SimHei'] # 用来正常显示中文标签
# mac解决中文乱码
# plt.rcParams['font.sans-serif'] = ['Arial Unicode MS']
plt.rcParams['axes.unicode_minus']=False # 用来正常显示负号

Populating the interactive namespace from numpy and matplotlib

1.加载并查看数据情况

train = pd.read_excel('data/data155594/data_train.xlsx')
test = pd.read_excel('data/data155594/data_test.xlsx')
test_submit = pd.read_csv('data/data155594/提交示例.csv')train.head(3)

	id	age	diseaseName	conditionDesc	title	hopeHelp	label_i	label_j
0	0	20+	刨腹产可以养三胎吗？	刨腹产可以养三胎吗？希望医生能帮我看看，谢谢！	刨腹产可以养三胎吗？	前两个都是刨腹产，时间已经隔了有6年了，前两个都很顺利，想问问医生现在可以养三胎吗？	1	8
1	1	20+	右膝前交叉韧带断裂，半月板三度重度损伤	右膝受力小腿骨和大腿骨就会滑开，走路会有隐隐作痛	大概做韧带重建手术需要多少费用？	大概做韧带重建手术需要多少费用？	18	57
2	2	30+	没有不适，就是左眼球有红色充血	昨天我朋友跟我说，我的左眼睛有红色的问我怎么啦，我自己都不知道，所以现在问医生看需要去医院吗...	需要怎么治疗	需要怎么治疗	15	45

1.1查看数据详细信息，检查缺失情况

# 查看数据详细信息，检查缺失情况
train.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 22868 entries, 0 to 22867
Data columns (total 8 columns):#   Column         Non-Null Count  Dtype
---  ------         --------------  ----- 0   id             22868 non-null  int64 1   age            22868 non-null  object2   diseaseName    22868 non-null  object3   conditionDesc  22868 non-null  object4   title          21627 non-null  object5   hopeHelp       21182 non-null  object6   label_i        22868 non-null  int64 7   label_j        22868 non-null  int64
dtypes: int64(3), object(5)
memory usage: 1.4+ MB

1.2统计并查看label_i和label_j的个数

train['label_i'].value_counts().plot(kind='barh')

<matplotlib.axes._subplots.AxesSubplot at 0x7f441be7f6d0>

train['label_j'].value_counts().plot(kind='barh')

<matplotlib.axes._subplots.AxesSubplot at 0x7f441bbe0450>

%matplotlib inline# 统计 label_i 的数目
sns.countplot(x='label_i',data=train)
plt.show()
# train['label_i'].value_counts().plot(kind='barh')

%matplotlib inline# 统计 label_j 的数目
plt.figure(figsize=(16,8))
sns.countplot(x='label_j',data=train,)
plt.show()

1.3统计文本长度

%matplotlib inline
# 文本长度
merge_col = ["diseaseName","conditionDesc","title","hopeHelp"]
train["text_length"] = [len("".join([str(rows[jol]) for jol in merge_col])) for _,rows in train.iterrows()]
print(train["text_length"].max())
sns.distplot(train["text_length"],bins=10)

255<matplotlib.axes._subplots.AxesSubplot at 0x7f441b7a2310>

2.数据预处理

seed = 1234# 设置随机种子 固定结果
def set_seed(seed):np.random.seed(seed)random.seed(seed)paddle.seed(seed)set_seed(seed)

填补缺失值+拼接文本内容+划分数据集**（9：1）**+转换为模型需要的编码格式

def clean_str(x):return x.replace('\r', '').replace('\t', ' ').replace('\n', ' ')# 清洗和填补缺失值
merge_col = ["diseaseName","conditionDesc","title","hopeHelp"]
for col in merge_col:train[col].fillna('', inplace=True)test[col].fillna('', inplace=True)train[col] = train[col].apply(lambda x: clean_str(x))test[col] = test[col].apply(lambda x: clean_str(x))# 拼接文本
train["sentence"] = [",".join([rows[jol] for jol in merge_col if len(rows[jol])>0]) for _,rows in train.iterrows()]
test["sentence"] = [",".join([rows[jol] for jol in merge_col if len(rows[jol])>0]) for _,rows in test.iterrows()]# 划分数据集 9:1
train_ds,dev_ds,_,_ = train_test_split(train,train["id"],test_size=0.2,random_state=seed)# 转为json格式
def read(df,label_name=None):if label_name:for idx,rows in df.iterrows():yield {'sentence':str(rows["sentence"]),'label': rows[label_name ]}else:for idx,rows in df.iterrows():yield {'sentence':str(rows["sentence"])}# label_i 训练集
train_i_ds = train_ds[["sentence","label_i"]]
dev_i_ds = dev_ds[["sentence","label_i"]]# label_j 训练集 直接删除包含-1未知标签的样本
train_j_ds = train_ds[["sentence","label_j"]]
dev_j_ds = dev_ds[["sentence","label_j"]]train_j_ds = train_j_ds[train_j_ds["label_j"]!=-1]
dev_j_ds = dev_j_ds[dev_j_ds["label_j"]!=-1]class_i_nums = train_i_ds["label_i"].value_counts().shape[0]
class_j_nums = train_j_ds["label_j"].value_counts().shape[0]
print("label_i_num:{} label_j_num:{}".format(class_i_nums,class_j_nums))# 转变为json格式
train_i_ds = load_dataset(read,df=train_i_ds,label_name="label_i",lazy=False)
dev_i_ds = load_dataset(read,df=dev_i_ds,label_name="label_i",lazy=False)train_j_ds = load_dataset(read,df=train_j_ds,label_name="label_j",lazy=False)
dev_j_ds = load_dataset(read,df=dev_j_ds,label_name="label_j",lazy=False)# 测试集
test_ds = load_dataset(read,df=test[["sentence"]],label_name=None,lazy=False)# 文本到模型需要的编码的函数
def convert_example(example, tokenizer):encoded_inputs = tokenizer(text=example["sentence"],max_seq_len=max_seq_len,pad_to_max_seq_len=True)return tuple([np.array(x, dtype="int64") for x in [encoded_inputs["input_ids"], encoded_inputs["token_type_ids"], [example["label"]]]])def convert_predict_example(example, tokenizer):encoded_inputs = tokenizer(text=example["sentence"],max_seq_len=max_seq_len,pad_to_max_seq_len=True)return tuple([np.array(x, dtype="int64") for x in [encoded_inputs["input_ids"], encoded_inputs["token_type_ids"]]])

label_i_num:20 label_j_num:61

3.搭建模型

class TextClassification(nn.Layer):def __init__(self, pretrained_model,num_class,dropout=None):super().__init__()self.ptm = pretrained_modelself.dropout = nn.Dropout(dropout if dropout is not None else 0.1)self.classifier = nn.Linear(self.ptm.config["hidden_size"], num_class)def forward(self,input_ids,token_type_ids=None,position_ids=None,attention_mask=None):_, cls_embedding = self.ptm(input_ids=input_ids, token_type_ids=token_type_ids,attention_mask=attention_mask)cls_embedding = self.dropout(cls_embedding)logits = self.classifier(cls_embedding)return logits

4.模型配置

4.1超参数

epochs = 15
train_batch_size = 16
dev_batch_size = 32
test_batch_size = 16
lr = 5E-5
max_seq_len = 256
logging_print = 50
logging_eval = 200
model_name = 'ernie-1.0'
save_i_dir = "checkpoint/label_i_model"
save_j_dir = "checkpoint/label_j_model"
submit_dir = "work/baseline.csv"
step1 = "i"
step2 = "j"
rdrop_coef = 0.2

4.2模型配置

# 加载tokenizer
tokenizer = ppnlp.transformers.ErnieTokenizer.from_pretrained(model_name)# 把验证集合转换成id
test_ds = test_ds.map(partial(convert_predict_example, tokenizer=tokenizer))
# 构建训练集合的dataloader
test_batch_sampler = paddle.io.DistributedBatchSampler(dataset=test_ds,batch_size=test_batch_size,shuffle=False)test_data_loader = paddle.io.DataLoader(dataset=test_ds,batch_sampler=test_batch_sampler,return_list=True)

[2022-07-05 23:54:42,962] [    INFO] - Already cached /home/aistudio/.paddlenlp/models/ernie-1.0/vocab.txt
[2022-07-05 23:54:42,976] [    INFO] - tokenizer config file saved in /home/aistudio/.paddlenlp/models/ernie-1.0/tokenizer_config.json
[2022-07-05 23:54:42,979] [    INFO] - Special tokens file saved in /home/aistudio/.paddlenlp/models/ernie-1.0/special_tokens_map.json

5.模型训练

5.1定义评估函数

# 因为训练过程中同时要在验证集进行模型评估，因此我们先定义评估函数
@paddle.no_grad()
def evaluate(model, criterion, metric, data_loader, phase="dev"):model.eval()metric.reset()losses = []for batch in data_loader:input_ids, token_type_ids, labels = batchprobs = model(input_ids=input_ids, token_type_ids=token_type_ids)# 计算损失loss = criterion(probs, labels)losses.append(loss.numpy())# 计算准确率correct = metric.compute(probs, labels)#准确率更新metric.update(correct)acc,p,r,f1,_ = metric.accumulate()print("eval {} loss: {:.5}, dev_accu: {:.5}, dev_p: {:.5}, dev_r: {:.5}, dev_f1: {:.5}".format(phase,np.mean(losses), acc,p,r,f1))return np.mean(losses),f1,acc

5.2定义模型训练

采用Rdrop_loss函数增加扰动。
可以考虑将CE loss 改为 focal_loss 但是得自己写一下focal_loss 由于focal比较简单paddle没有focal的api。

def do_train(model, criterion, metric, dev_data_loader,train_data_loader):global_step = 0tic_train = time.time()best_accuracy = 0.0best_f1 = 0.0model.train()# --创建Rdrop_loss函数rdrop_loss = ppnlp.losses.RDropLoss()print("do train ...")for epoch in range(1, epochs + 1):model.train()for step, batch in enumerate(train_data_loader, start=1):input_ids, token_type_ids, labels = batchprobs = model(input_ids=input_ids, token_type_ids=token_type_ids)# --修改loss / 当rdrop_coef大于0时启用Rdropif rdrop_coef > 0:probs_2 = model(input_ids=input_ids, token_type_ids=token_type_ids)ce_loss = (criterion(probs, labels) + criterion(probs, labels)) * 0.5kl_loss = rdrop_loss(probs, probs_2)loss = ce_loss + kl_loss * rdrop_coefelse:loss = criterion(probs, labels)correct = metric.compute(probs, labels)metric.update(correct)acc,p,r,f1,_ = metric.accumulate()global_step += 1# 每间隔 100 step 输出训练指标if global_step % logging_print == 0:print("global step %d, epoch: %d, batch: %d, loss: %.5f, train_accu: %.5f, train_p: %.5f,train_r: %.5f,train_f1: %.5f,speed: %.2f step/s"% (global_step, epoch, step, loss, acc, p, r, f1, 10 / (time.time() - tic_train)))tic_train = time.time()loss.backward()optimizer.step()lr_scheduler.step()optimizer.clear_grad()# 每间隔 100 step 在验证集和测试集上进行评估if global_step % logging_eval == 0 :eval_loss,eval_f1,eval_acc = evaluate(model, criterion, metric, dev_data_loader, "dev")if(best_f1 <= eval_f1 or (best_f1 == eval_f1 and best_accuracy < eval_acc)):print("best f1 %.5f, update %.5f ---> %.5f" % (eval_f1,best_f1,eval_f1))best_f1 = eval_f1best_accuracy = eval_acc# 保存模型save_param_path = os.path.join(save_dir, 'model_best.pdparams')paddle.save(model.state_dict(), save_param_path)# 保存tokenizertokenizer.save_pretrained(save_dir)eval_loss,eval_f1,eval_acc = evaluate(model, criterion, metric, dev_data_loader, "dev")if(best_f1 <= eval_f1 or (best_f1 == eval_f1 and best_accuracy < eval_acc)):print("best f1 %.5f, update %.5f ---> %.5f" % (eval_f1,best_f1,eval_f1))best_f1 = eval_f1best_accuracy = eval_acc# 保存模型save_param_path = os.path.join(save_dir, 'model_best.pdparams')paddle.save(model.state_dict(), save_param_path)# 保存tokenizertokenizer.save_pretrained(save_dir)def do_sample_predict(model,data_loader):model.eval()outputs = []for batch in tqdm(data_loader):input_ids, token_type_ids = batchpooled_output = model(input_ids=input_ids, token_type_ids=token_type_ids)# 取概率值最大的索引out2 = paddle.argmax(pooled_output, axis=1)outputs.extend(out2.numpy())return outputs

5.3 label_i 模型训练

step_1_start = True
if step1 and step_1_start:print("start i ...")train_ds = train_i_ds.map(partial(convert_example, tokenizer=tokenizer))# 把验证集合转换成iddev_ds = dev_i_ds.map(partial(convert_example, tokenizer=tokenizer))# 构建训练集合的dataloadertrain_batch_sampler = paddle.io.DistributedBatchSampler(dataset=train_ds,batch_size=train_batch_size,shuffle=True)train_data_loader = paddle.io.DataLoader(dataset=train_ds,batch_sampler=train_batch_sampler,return_list=True)# 针对验证集数据加载，我们使用单卡进行评估，所以采用 paddle.io.BatchSampler 即可# 定义验证集的dataloaderdev_batch_sampler = paddle.io.BatchSampler(dev_ds,batch_size=dev_batch_size,shuffle=False)dev_data_loader = paddle.io.DataLoader(dataset=dev_ds,batch_sampler=dev_batch_sampler,return_list=True)class_nums = class_i_nums# 模型创建pretrained_model = ppnlp.transformers.ErnieModel.from_pretrained(model_name)model = TextClassification(pretrained_model,num_class=class_nums)num_training_steps = len(train_data_loader) * epochs# 定义 learning_rate_scheduler，负责在训练过程中对 lr 进行调度lr_scheduler = LinearDecayWithWarmup(lr, num_training_steps, 0.0)# 训练结束后，存储模型参数save_dir = save_i_dir# 创建保存的文件夹os.makedirs(save_dir,exist_ok=True)# Generate parameter names needed to perform weight decay.# All bias and LayerNorm parameters are excluded.decay_params = [p.name for n, p in model.named_parameters()if not any(nd in n for nd in ["bias", "norm"])]# 定义 Optimizeroptimizer = paddle.optimizer.AdamW(learning_rate=lr_scheduler,parameters=model.parameters(),weight_decay=0.0,apply_decay_param_fun=lambda x: x in decay_params)# --修改损失函数# 交叉熵损失criterion = paddle.nn.loss.CrossEntropyLoss()# 评估的时候采用准确率指标metric = ppnlp.metrics.AccuracyAndF1()do_train(model, criterion, metric, dev_data_loader,train_data_loader)state_dict = paddle.load('checkpoint/label_i_model/model_best.pdparams')model.load_dict(state_dict)predict_label = do_sample_predict(model,test_data_loader)predict_submit = pd.DataFrame([],columns=["id","label_i","label_j"])for i, pred in enumerate(predict_label):predict_submit.loc[len(predict_submit),:] = [i,pred,0]predict_submit.to_csv(submit_dir,index=False)

start i ...[2022-07-05 23:54:43,035] [    INFO] - Already cached /home/aistudio/.paddlenlp/models/ernie-1.0/ernie_v1_chn_base.pdparams
[2022-07-05 23:54:43,831] [    INFO] - Weights from pretrained model not used in ErnieModel: ['cls.predictions.layer_norm.weight', 'cls.predictions.decoder_bias', 'cls.predictions.transform.bias', 'cls.predictions.transform.weight', 'cls.predictions.layer_norm.bias']do train ...---------------------------------------------------------------------------KeyboardInterrupt                         Traceback (most recent call last)/tmp/ipykernel_103/1298338812.py in <module>60     metric = ppnlp.metrics.AccuracyAndF1()61
---> 62     do_train(model, criterion, metric, dev_data_loader,train_data_loader)63 64     state_dict = paddle.load('checkpoint/label_i_model/model_best.pdparams')/tmp/ipykernel_103/824462548.py in do_train(model, criterion, metric, dev_data_loader, train_data_loader)19             if rdrop_coef > 0:20                 probs_2 = model(input_ids=input_ids, token_type_ids=token_type_ids)
---> 21                 ce_loss = (criterion(probs, labels) + criterion(probs, labels)) * 0.522                 kl_loss = rdrop_loss(probs, probs_2)23                 loss = ce_loss + kl_loss * rdrop_coef/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py in __call__(self, *inputs, **kwargs)928             return self.forward(*inputs, **kwargs)929         else:
--> 930             return self._dygraph_call_func(*inputs, **kwargs)931 932     def forward(self, *inputs, **kwargs):/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py in _dygraph_call_func(self, *inputs, **kwargs)913                 outputs = self.forward(*inputs, **kwargs)914         else:
--> 915             outputs = self.forward(*inputs, **kwargs)916 917         for forward_post_hook in self._forward_post_hooks.values():/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/nn/layer/loss.py in forward(self, input, label)404             axis=self.axis,405             use_softmax=self.use_softmax,
--> 406             name=self.name)407 408         return ret/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/nn/functional/loss.py in cross_entropy(input, label, weight, ignore_index, reduction, soft_label, axis, use_softmax, name)1709         if soft_label == False:1710             valid_label = paddle.cast(
-> 1711                 label != ignore_index, dtype=label.dtype) * label1712             label_min = paddle.min(valid_label)1713             label_max = paddle.max(valid_label)/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/math_op_patch.py in __impl__(self, other_var)297             else:298                 math_op = getattr(_C_ops, op_type)
--> 299                 return math_op(self, other_var, 'axis', axis)300 301         comment = OpProtoHolder.instance().get_op_proto(op_type).commentKeyboardInterrupt:

5.4label_j 模型训练

由于引入了Rdrop后训练时间和显存开销增加,一键Run可能会爆显存,设置一个参数控制是否训练
所以建议采用V100 32G进行跑模型。


step_2_start = True
if step2 and step_2_start:print("start j ...")# 把训练集合转换成idtrain_ds = train_j_ds.map(partial(convert_example, tokenizer=tokenizer))# 把验证集合转换成iddev_ds = dev_j_ds.map(partial(convert_example, tokenizer=tokenizer))# 构建训练集合的dataloadertrain_batch_sampler = paddle.io.DistributedBatchSampler(dataset=train_ds,batch_size=train_batch_size,shuffle=True)train_data_loader = paddle.io.DataLoader(dataset=train_ds,batch_sampler=train_batch_sampler,return_list=True)# 针对验证集数据加载，我们使用单卡进行评估，所以采用 paddle.io.BatchSampler 即可# 定义验证集的dataloaderdev_batch_sampler = paddle.io.BatchSampler(dev_ds,batch_size=dev_batch_size,shuffle=False)dev_data_loader = paddle.io.DataLoader(dataset=dev_ds,batch_sampler=dev_batch_sampler,return_list=True)class_nums = class_j_nums# 模型创建pretrained_model = ppnlp.transformers.ErnieModel.from_pretrained(model_name)model = TextClassification(pretrained_model,num_class=class_nums)num_training_steps = len(train_data_loader) * epochs# 定义 learning_rate_scheduler，负责在训练过程中对 lr 进行调度lr_scheduler = LinearDecayWithWarmup(lr, num_training_steps, 0.0)# 训练结束后，存储模型参数save_dir = save_j_dir# 创建保存的文件夹os.makedirs(save_dir,exist_ok=True)# Generate parameter names needed to perform weight decay.# All bias and LayerNorm parameters are excluded.decay_params = [p.name for n, p in model.named_parameters()if not any(nd in n for nd in ["bias", "norm"])]# 定义 Optimizeroptimizer = paddle.optimizer.AdamW(learning_rate=lr_scheduler,parameters=model.parameters(),weight_decay=0.0,apply_decay_param_fun=lambda x: x in decay_params)# --修改损失函数# 交叉熵损失criterion = paddle.nn.loss.CrossEntropyLoss()# 评估的时候采用准确率指标metric = ppnlp.metrics.AccuracyAndF1()do_train(model, criterion, metric, dev_data_loader,train_data_loader)state_dict = paddle.load('checkpoint/label_j_model/model_best.pdparams')model.load_dict(state_dict)predict_label = do_sample_predict(model,test_data_loader)predict_submit = pd.read_csv(submit_dir)predict_submit["label_j"] = predict_labelpredict_submit.to_csv(submit_dir,index=False)

提分小建议

引入更多外部知识，官方给出的14000条三元组信息，构建知识图谱
更换模型（bert和ernie使用）、使用医疗语料进行预训练
使用FGM、PGD、R-drop等对抗训练方法
数据类别不平衡，更换损失函数如Focal_loss、Dice_loss
数据增广，如拼接变换次序、同义词替换、随机删减等
参数、模型修改、scheduler 调优等
标签有明确语义可以尝试使用Prompt base learning的方法如PET、P-tuning等
独立训练并还没有考虑label_i和label_j的关系，通过eda可以发现标签存在明显的层次关系或联系，考虑层次多标签、多任务等模型方法

模型修改比较

ennie-1.0 lr=5*10-5 epochs = 10 分数：1.6332
macbert-base-chinese lr=5*10-5 epochs =10 分数：1.50144
由于优化器用的adamw 所以建议不要增加轮数来提分 bert分低的原因有一部分是因为轮数过高

此文章为搬运
原项目链接