一, 前言

1.Imdb影评的数据集介绍与下载
2.贝叶斯原理介绍
3.TF-IDF是什么

二，代码介绍

preprocess_data() 方法包含以下步骤：

加载Imdb影评数据。
用数据训练CountVectorizer或者TfidfVectorizer。
保持CountVectorizer与TfidfVectorizer模型，因为预测的时候要用这两个模型把要预测的词转化为词向量。
保持转换为词向量的数据集。

def preprocess_data():X_orig, Y_orig = get_data(aclImdb_train_dir)X_orig_test, Y_orig_test = get_data(aclImdb_test_dir)X_orig = np.concatenate([X_orig, X_orig_test])Y_orig = np.concatenate([Y_orig, Y_orig_test])cv = CountVectorizer(max_features=vocab_size)tfidf = TfidfVectorizer(max_features=vocab_size)cv.fit(X_orig)tfidf.fit(X_orig)print(cv.vocabulary_)# 3. 4.transform training data into a 'document-term matrix' (which is a sparse matrix) use “transform()”train_data = cv.transform(X_orig)tfidf_train_data = tfidf.transform(X_orig)# (the index of the list , the index of the dict ) the frequency of the list[index]print(cv.get_feature_names())print(train_data)train_data = train_data.toarray()tfidf_train_data = tfidf_train_data.toarray()print(train_data)joblib.dump(cv, "data/CountVectorizer.joblib")joblib.dump(tfidf, "data/TfidfVectorizer.joblib")np.savez(r'data\CountVectorizer_trainData', x=train_data, y=Y_orig)np.savez(r'data\TfidfVectorizer_trainData', x=tfidf_train_data, y=Y_orig)

运行后会生成下面四个文件

train_my_module()方法:

加载已经生成的词向量。
训练你MultinomialNB贝叶斯模型。
测试你的模型

def train_my_module(is_tfidf):if is_tfidf:trainDataNew = np.load(r'data/TfidfVectorizer_trainData.npz')else:trainDataNew = np.load('data/CountVectorizer_trainData.npz')x = trainDataNew['x']y = trainDataNew['y']x_train,x_test,y_train,y_test=train_test_split(x, y, test_size=0.3)x_train,x_test,y_train,y_test=np.array(x_train),np.array(x_test),np.array(y_train),np.array(y_test)print(x_train.shape,x_test.shape)#训练数据module=MultinomialNB()module.fit(x_train,y_train)#测试数据y_pred=module.predict(x_test)if is_tfidf:joblib.dump(module, r'data/Tfidf_bayes_module.joblib')else:joblib.dump(module, r'data/bayes_module.joblib')#输出print("正确值：{0}".format(y_test))print("预测值：{0}".format(y_pred))print("准确率：%f%%"%(accuracy_score(y_test, y_pred)*100))

predict_my_module()方法

加载训练好的贝叶斯模型
加载训练好的词向量模型
把要预测的数据转为词向量
预测

def predict_my_module(is_tfidf):if is_tfidf:model = joblib.load(r'data/Tfidf_bayes_module.joblib')else:model = joblib.load(r'data/bayes_module.joblib')#neg:0 postive:1review =["the character is so poorly written.", "this is bad movie ",  "I'm not very disappoint for this movie", "I'm very happy for this movie" ]if is_tfidf:cv = joblib.load(r'data/TfidfVectorizer.joblib')else:cv = joblib.load(r'data/CountVectorizer.joblib')train_data = cv.transform(review)train_data = train_data.toarray()s = model.predict(train_data)print(s)

三，CountVectorizer与TfidfVectorizer的结果对比

如果is_tfidf为False，那么就是用CountVectorizer生成的词向量训练的模型
如果is_tfidf为True，那么就是用TfidfVectorizer生成的词向量训练的模型

if __name__ == '__main__':preprocess_data()is_tfidf =Truetrain_my_module(is_tfidf)#predict_my_module(is_tfidf)

运行结果如下，可知TfidfVectorizer 生成的词向量准确率最高。

Count Vectorizer	Tfidf Vectorizer
准确率：84.326667%	准确率： 85.893333%

四，情感分析的完整代码如下

import numpy as np
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
import numpy as np
import re
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
import os as os
import joblibvocab_size = 30000
aclImdb_train_dir = r'D:\train_data\aclImdb\aclImdb\train'
aclImdb_test_dir = r'D:\train_data\aclImdb\aclImdb\test'
# remove html tag like '<br /><br />'
def  rm_tags(text):re_tag = re.compile(r'<[^>]+>')return re_tag.sub(' ', text)def clean_str(string):return string.strip().lower()def process(text):text = clean_str(text)text = rm_tags(text)return  textdef get_data(datapath):pos_files = os.listdir(datapath + '/pos')neg_files = os.listdir(datapath + '/neg')print(len(pos_files))print(len(neg_files))pos_all = []neg_all = []for pf, nf in zip(pos_files, neg_files):with open(datapath + '/pos' + '/' + pf, encoding='utf-8') as f:s = f.read()s = process(s)pos_all.append(s)with open(datapath + '/neg' + '/' + nf, encoding='utf-8') as f:s = f.read()s = process(s)neg_all.append(s)X_orig= np.array(pos_all + neg_all)Y_orig = np.array([1 for _ in range(len(pos_all))] + [0 for _ in range(len(neg_all))])return X_orig, Y_origdef preprocess_data():X_orig, Y_orig = get_data(aclImdb_train_dir)X_orig_test, Y_orig_test = get_data(aclImdb_test_dir)X_orig = np.concatenate([X_orig, X_orig_test])Y_orig = np.concatenate([Y_orig, Y_orig_test])cv = CountVectorizer(max_features=vocab_size)tfidf = TfidfVectorizer(max_features=vocab_size)cv.fit(X_orig)tfidf.fit(X_orig)print(cv.vocabulary_)# 3. 4.transform training data into a 'document-term matrix' (which is a sparse matrix) use “transform()”train_data = cv.transform(X_orig)tfidf_train_data = tfidf.transform(X_orig)# (the index of the list , the index of the dict ) the frequency of the list[index]print(cv.get_feature_names())print(train_data)train_data = train_data.toarray()tfidf_train_data = tfidf_train_data.toarray()print(train_data)joblib.dump(cv, "data/CountVectorizer.joblib")joblib.dump(tfidf, "data/TfidfVectorizer.joblib")np.savez(r'data\CountVectorizer_trainData', x=train_data, y=Y_orig)np.savez(r'data\TfidfVectorizer_trainData', x=tfidf_train_data, y=Y_orig)def train_my_module(is_tfidf):if is_tfidf:trainDataNew = np.load(r'data/TfidfVectorizer_trainData.npz')else:trainDataNew = np.load('data/CountVectorizer_trainData.npz')x = trainDataNew['x']y = trainDataNew['y']x_train,x_test,y_train,y_test=train_test_split(x, y, test_size=0.3)x_train,x_test,y_train,y_test=np.array(x_train),np.array(x_test),np.array(y_train),np.array(y_test)print(x_train.shape,x_test.shape)#训练数据module=MultinomialNB()module.fit(x_train,y_train)#测试数据y_pred=module.predict(x_test)if is_tfidf:joblib.dump(module, r'data/Tfidf_bayes_module.joblib')else:joblib.dump(module, r'data/bayes_module.joblib')#输出print("正确值：{0}".format(y_test))print("预测值：{0}".format(y_pred))print("准确率：%f%%"%(accuracy_score(y_test, y_pred)*100))def predict_my_module(is_tfidf):if is_tfidf:model = joblib.load(r'data/Tfidf_bayes_module.joblib')else:model = joblib.load(r'data/bayes_module.joblib')#neg:0 postive:1review =["the character is so poorly written.", "this is bad movie ",  "I'm not very disappoint for this movie", "I'm very happy for this movie" ]if is_tfidf:cv = joblib.load(r'data/TfidfVectorizer.joblib')else:cv = joblib.load(r'data/CountVectorizer.joblib')train_data = cv.transform(review)train_data = train_data.toarray()s = model.predict(train_data)print(s)if __name__ == '__main__':preprocess_data()is_tfidf =Truetrain_my_module(is_tfidf)#predict_my_module(is_tfidf)

[机器学习-实战篇]Imdb数据集情感分析之贝叶斯相关推荐

【深度学习】-Imdb数据集情感分析之模型对比（4）- CNN-LSTM 集成模型
[深度学习]-Imdb数据集情感分析之模型对比(4)- CNN-LSTM 集成模型前言 [深度学习]-Imdb数据集情感分析之模型对比(3)- CNN [深度学习]-Imdb数据集情感分析之模型对比 ...
【深度学习】-Imdb数据集情感分析之模型对比（2）- LSTM
[深度学习]-Imdb数据集情感分析之模型对比(2)-LSTM 文章目录前言一.LSTM是什么? 算法介绍二.训练LSTM模型 1.数据预处理 2.构建LSTM模型设定模型参数构建并训练模型 ...
[深度学习-NLP]Imdb数据集情感分析之模型对比(贝叶斯, LSTM, GRU, TextCNN, Transformer, BERT)
一,详细原理以及代码请看下面博客 1.Imdb数据集情感分析之离散贝叶斯 2.Imdb数据集情感分析之LSTM长短期记忆 3.Imdb数据集情感分析之卷积神经网络-TextCNN 4.Imdb数据集情 ...
情感分析朴素贝叶斯_朴素贝叶斯推文的情感分析
情感分析朴素贝叶斯 Millions of tweets are posted every second. It helps us know how the public is responding ...
机器学习 · 总览篇 III 统计推断: 频率学派和贝叶斯学派
统计推断作为重要的机器学习基础,对它的了解十分必要,否则我们做机器学习只是在黑盒操作,对其原理和结果难以解释本文首发于我的知乎专栏<机器怎么学习>中机器学习·总览篇(3) 统计推断: ...
电影评论情感分析-IMDB数据集 | python 利用朴素贝叶斯、神经网络模型
载入包 import torch # torch==1.7.1 import torch.nn as nn from torch.utils.data import Dataset from torc ...
【阿旭机器学习实战】【33】中文文本分类之情感分析--朴素贝叶斯、KNN、逻辑回归
[阿旭机器学习实战]系列文章主要介绍机器学习的各种算法模型及其实战案例,欢迎点赞,关注共同学习交流. 目录 1.查看原始数据结构 2.导入数据并进行数据处理 2.1 提取数据与标签 2.2 过滤停用词 ...
机器学习实战教程（四）：朴素贝叶斯基础篇之言论过滤器
一.前言朴素贝叶斯算法是有监督的学习算法,解决的是分类问题,如客户是否流失.是否值得投资.信用等级评定等多分类问题.该算法的优点在于简单易懂.学习效率高.在某些领域的分类问题中能够与决策树.神经网络 ...
机器学习实战之分类算法(K-近邻/朴素贝叶斯/决策树/随机森林)
机器学习分类算法 1. 机器学习算法简介 1.1 按照学习方式分类 1.2 区别 1.3 关于监督学习中的分类与回归区别 1.4 机器学习开发流程 2. 数据集介绍与划分 2.1 sklearn数据集 ...

[机器学习-实战篇]Imdb数据集情感分析之贝叶斯

一, 前言

二，代码介绍

preprocess_data() 方法包含以下步骤：

train_my_module()方法:

predict_my_module()方法

三，CountVectorizer与TfidfVectorizer的结果对比

四，情感分析的完整代码如下

[机器学习-实战篇]Imdb数据集情感分析之贝叶斯相关推荐

最新文章

热门文章