机器学习识别系统的度量指标

分类指标

混淆矩阵
ROC曲线
AUC
EER
改进版本
参考

混淆矩阵

在使用支持向量机模型进行分类时：如果调整判断的阈值，如

则TP，FP均增加。这是因为将判别的阈值改为-1，则会有更多的测试样本被判别为正样本，则TP，FP均增加。

ROC曲线

AUC

代表右下角面积，越大代表分类效果越好。

EER

等错误率。FP=FN

下面是对于兵王问题利用SVM来进行二分类的程序：
首先是对数据集进行一个分析和处理：

'''
总样本数28056，其中正样本2796，负样本25260.
随机选取5000个样本训练，其余测试。
将draw设置为+1，其他为-1
'''
s = open('krkopt.data','r',encoding='utf-8')
x_data = []
y_data = []
for i in s.readlines():i = i.rstrip().split(',')if i[0]=='??': breaky_data.append(1 if i[-1] == "draw" else -1)xs =[]for k in i[:-1]:xs.append(int(k) if k.isalpha()==False else (ord(k)-ord('a')+1))x_data.append(xs)
s.close()

然后进行归一化

#归一化
import pandas as pd
ds = pd.DataFrame(x_data,columns={'pbx','pby','pr1x','pr1y','pr2x','pr2y'})
columnname = [i for i in ds]
for i in columnname:ds[i] = (ds[i]-ds.describe()[i]['mean'])/(ds.describe()[i]['std'])
ds = ds.values
print(ds.shape)
print(len(y_data)

进行随机选取，得到训练集

import numpy as np
ds_x =[]
ds_y =[]
for i in range(2500):ind = np.random.randint(0, 2800)ds_x.append(ds[ind])ds_y.append(y_data[ind])
for i in range(2500):ind = np.random.randint(2800, 25260)ds_x.append(ds[ind])ds_y.append(y_data[ind])
print(ds_x[:5])
print(ds_y[:5])

进行SVM的训练和评估

from sklearn import svm
#3.训练svm分类器
classifier=svm.SVC(C=2,kernel='rbf',gamma=10,decision_function_shape='ovr') # ovr:一对多策略
#参数详解见https://blog.csdn.net/weixin_41990278/article/details/93137009
classifier.fit(ds_x,ds_y)

#4.计算svc分类器的准确率
print("训练集：",classifier.score(ds_x,ds_y))
print("测试集：",classifier.score(ds,y_data))
训练集： 1.0
测试集： 0.9668876532648988

#查看决策函数
print('train_decision_function:',classifier.decision_function(ds_x))
print('predict_result:',classifier.predict(ds_x))

计算混淆矩阵

#计算评价指标。
pred = classifier.predict(ds)
print(pred.shape)
correct = np.equal(y_data[:2796], pred[:2796])
TP = np.mean(correct.astype(np.float))
FN = 1-TP
correct_TN = np.equal(y_data[2796:], pred[2796:])
TN = np.mean(correct_TN.astype(np.float))
FP = 1-TN
matrix = [[TP, FN], [FP, TN]]
matrix = pd.DataFrame(matrix, columns=['正样本','负样本'])
matrix.index=['实际正样本','实际负样本']
print(matrix)
'''正样本       负样本
实际正样本  0.879471  0.120529
实际负样本  0.023436  0.976564
'''

改进版本

参考了一份代码，地址：https://www.yuque.com/docs/share/97eb71c5-2715-40af-b873-a61785175b98?#

#数据预处理与导包，里面有几个非常好用的函数。
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib%matplotlib inline
matplotlib.rcParams.update({'font.size': 20})from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import confusion_matrixfrom sklearn.metrics import roc_curve# 读取数据
names = ['x1', 'y1', 'x2', 'y2', 'x3', 'y3', 'label']
data = pd.read_csv('krkopt.data', names=names)
data.head()# 处理数据
letter_num_map = {"a": 1,"b": 2,"c": 3,"d": 4,"e": 5,"f": 6,"g": 7,"h": 8,"draw": 1,"zero": -1,"one": -1,"two": -1,"three": -1,"four": -1,"five": -1,"six": -1,"seven": -1,"eight": -1,"nine": -1,"ten": -1,"eleven": -1,"twelve": -1,"thirteen": -1,"fourteen": -1,"fifteen": -1,"sixteen": -1
}
for i in ["x1", "x2", "x3", "label"]:data[i] = data[i].map(lambda x: letter_num_map.get(x, 0))# 查看数据
data.head()
data.info()
data.describe()

# 查看label分布
data.label.value_counts()
data.label.hist(figsize=(10, 5))
plt.title('Class counts')# 将数据集分割为训练集和测试集
X = data.iloc[:, :-1]
y = data.iloc[:, -1]
# random_state 指定随机数seed
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=5000, random_state=0)print(X_train.shape)
print(y_train.shape)

在进行SVM训练时，用到了GridSearchCV()方法。
参考资料：https://blog.csdn.net/foneone/article/details/89985045
下面这里C与gamma的范围可以根据需要调整，这里我没有改变参数，训练的时间比较长。

# 归一化
scaler = StandardScaler()
scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)# Grid Search with Cross-Validation
# 选用高斯核函数
param_grid = {'C': list(map(lambda x: 2 ** x, range(-5, 16))),'gamma': list(map(lambda x: 2 ** x, range(-15, 4)))}
print(f"Parameter grid:\n {param_grid}")# 训练 耗时操作需要等待
grid_search = GridSearchCV(SVC(), param_grid, cv=5)
grid_search.fit(X_train_scaled, y_train)

#评估 包括最佳参数得到的准确率和混淆矩阵，以及多个参数得到的ROC曲线。
# 测试准确率
print("Test set score: {:.4f}".format(grid_search.score(X_test_scaled, y_test)))
print("Best parameters: {}".format(grid_search.best_params_))# 查看混淆矩阵def plot_confusion_matrix(grid_search, x_test, y_test):fig = plt.figure(figsize=(10,10))ax = fig.add_subplot(1, 1, 1)y_pred = grid_search.predict(x_test)Carray = confusion_matrix(y_test, y_pred) #C中为TP FP TN FN 的值TP = Carray[0][0] / (Carray[0][0]+Carray[0][1])FN = 1-TPFP = Carray[1][0] /(Carray[1][0] +Carray[1][1])TN = 1-FPCmat = [[TP, FN] ,[FP , TN]]Cmat = pd.DataFrame(Cmat, columns=['预测正样本','预测负样本'])Cmat.index=['实际正样本','实际负样本']print(Cmat)plot_confusion_matrix(grid_search, X_test_scaled, y_test)# 绘制ROC曲线
def plot_roc_curve(grid_search, x_test, y_test):y_score = grid_search.decision_function(x_test)fpr,tpr,threshold = roc_curve(y_test, y_score) ###计算真正率和假正率roc_auc = auc(fpr,tpr) ###计算auc的值lw = 2plt.plot(fpr, tpr, color='darkorange',lw=lw, label='ROC curve (area = %0.2f)' % roc_auc) ###假正率为横坐标，真正率为纵坐标做曲线plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')plt.xlim([0.0, 1.0])plt.ylim([0.0, 1.05])plt.xlabel('False Positive Rate')plt.ylabel('True Positive Rate')plt.title('Receiver operating characteristic example')plt.legend(loc="lower right")plt.show()print(roc_auc) #输出roc的值。
plot_roc_curve(grid_search, X_test_scaled, y_test)

AUC值为0.997313.

参考

https://blog.csdn.net/u012679707/article/details/80511968
//利用SVM对Iris.data进行分类。
https://blog.csdn.net/weixin_41990278/article/details/93137009
//参数SVM参数详解
https://blog.csdn.net/xyz1584172808/article/details/81839230//ROC
https://blog.csdn.net/foneone/article/details/89985045//GridSearchCV
https://blog.csdn.net/u011734144/article/details/80277225//confusion_matrix