NLP-基础知识-003(词性标注)

目标：词性标注s = w1w2w3......wn 单词z = (z1z2......zn)  词性目的：argmax p(z|s)  -> Noisy Channel Model= argmax p(s|z) p(z)p(s|z) - Translation Modelp(z) - Language Model= argmax p(w1w2...wn|z1z2....zn)p(z1z2....zn) (假设条件独立)= argmax p(w1|z1) p(w2|z2) ..... p(wn|zn)p(z1)p(z2|z1)p(z3|z1z2)......(马尔科夫假设)= argmax Pi(i=1..n) P(wi|zi) * p(z1)p(z2|z1)p(z3|z2)......=> argmax logpi(i=1...n)p(wi|zi)p(z1) pi(i=2..n)p(zj|zj-1)
= argmax sum(i=1..n) log p(wi|zi) + logp(z1) + sum(j=2..n)logp(zj|zj-1)z' = argmax sum(i=1..n) log p(wi|zi) + logp(z1) + sum(j=2..n)logp(zj|zj-1)

"计算pi、A、B代码,traindata.txt文件数据见文章结尾"tag2id, id2tag = {}, {}  # maps tag to id . tag2id: {"VB": 0, "NNP":1,..} , id2tag: {0: "VB", 1: "NNP"....}
word2id, id2word = {}, {} # maps word to idfor line in open('traindata.txt'):items = line.split('/')word, tag = items[0], items[1].rstrip()  # 抽取每一行里的单词和词性if word not in word2id:word2id[word] = len(word2id)id2word[len(id2word)] = wordif tag not in tag2id:tag2id[tag] = len(tag2id)id2tag[len(id2tag)] = tagM = len(word2id)  # M: 词典的大小、# of words in dictionary
N = len(tag2id)   # N: 词性的种类个数  # of tags in tag set# 构建 pi, A, B
import numpy as np
pi = np.zeros(N)   # 每个词性出现在句子中第一个位置的概率,  N: # of tags  pi[i]: tag i出现在句子中第一个位置的概率
A = np.zeros((N, M)) # A[i][j]: 给定tag i, 出现单词j的概率。 N: # of tags M: # of words in dictionary
B = np.zeros((N,N))  # B[i][j]: 之前的状态是i, 之后转换成转态j的概率 N: # of tagsprev_tag = ""
for line in open('traindata.txt'):items = line.split('/')wordId, tagId = word2id[items[0]], tag2id[items[1].rstrip()]if prev_tag == "":  # 这意味着是句子的开始pi[tagId] += 1A[tagId][wordId] += 1else:  # 如果不是句子的开头A[tagId][wordId] += 1B[tag2id[prev_tag]][tagId] += 1if items[0] == ".":prev_tag = ""else:prev_tag = items[1].rstrip()# normalize
pi = pi/sum(pi)
for i in range(N):A[i] /= sum(A[i])B[i] /= sum(B[i])#  到此为止计算完了模型的所有的参数： pi, A, B

知道了pi、A、B，需要求出最优的z

维特比算法最终为一个动态规划寻找最优路径的问题，最终代码如下：

def log(v):if v == 0:return np.log(v+0.000001)return np.log(v)def viterbi(x, pi, A, B):"""x: user input string/sentence: x: "I like playing soccer"pi: initial probability of tagsA: 给定tag, 每个单词出现的概率B: tag之间的转移概率"""x = [word2id[word] for word in x.split(" ")]  # x: [4521, 412, 542 ..]T = len(x)dp = np.zeros((T,N))  # dp[i][j]: w1...wi, 假设wi的tag是第j个tagptr = np.array([[0 for x in range(N)] for y in range(T)] ) # T*N# TODO: ptr = np.zeros((T,N), dtype=int)for j in range(N): # basecase for DP算法dp[0][j] = log(pi[j]) + log(A[j][x[0]])for i in range(1,T): # 每个单词for j in range(N):  # 每个词性# TODO: 以下几行代码可以写成一行（vectorize的操作， 会使得效率变高）dp[i][j] = -9999999for k in range(N): # 从每一个k可以到达jscore = dp[i-1][k] + log(B[k][j]) + log(A[j][x[i]])if score > dp[i][j]:dp[i][j] = scoreptr[i][j] = k# decoding: 把最好的tag sequence 打印出来best_seq = [0]*T  # best_seq = [1,5,2,23,4,...]  # step1: 找出对应于最后一个单词的词性best_seq[T-1] = np.argmax(dp[T-1])# step2: 通过从后到前的循环来依次求出每个单词的词性for i in range(T-2, -1, -1): # T-2, T-1,... 1, 0best_seq[i] = ptr[i+1][best_seq[i+1]]# 到目前为止, best_seq存放了对应于x的 词性序列for i in range(len(best_seq)):print (id2tag[best_seq[i]])

最终验证，输入一句话，可以得出对应的词性:

x = "Social Security number , passport number and details about the services provided for the payment"
print(viterbi(x, pi, A, B))NNP
NNP
NN
,
NN
NN
CC
NNS
IN
DT
NNS
VBN
IN
DT
NN

traindata.txt 部分训练语料如下所示：Newsweek/NNP
,/,
trying/VBG
to/TO
keep/VB
pace/NN
with/IN
rival/JJ
Time/NNP
magazine/NN
,/,
announced/VBD
new/JJ
advertising/NN
rates/NNS
for/IN
1990/CD
and/CC
said/VBD
it/PRP
will/MD
introduce/VB
a/DT
new/JJ
incentive/NN
plan/NN
for/IN
advertisers/NNS
./.
The/DT
new/JJ
ad/NN
plan/NN
from/IN
Newsweek/NNP
,/,
a/DT
unit/NN
of/IN
the/DT
Washington/NNP
Post/NNP
Co./NNP
,/,
is/VBZ
the/DT
second/JJ
incentive/NN
plan/NN
the/DT
magazine/NN
has/VBZ
offered/VBN
advertisers/NNS
in/IN
three/CD
years/NNS
./.
Plans/NNS
that/WDT
give/VBP
advertisers/NNS
discounts/NNS
for/IN
maintaining/VBG
or/CC
increasing/VBG
ad/NN
spending/NN
have/VBP
become/VBN
permanent/JJ
fixtures/NNS
at/IN
the/DT
news/NN
weeklies/NNS
and/CC
underscore/VBP
the/DT
fierce/JJ
competition/NN
between/IN
Newsweek/NNP
,/,
Time/NNP
Warner/NNP
Inc./NNP
's/POS
Time/NNP
magazine/NN
,/,
and/CC
Mortimer/NNP
B./NNP
Zuckerman/NNP
's/POS
U.S./NNP
News/NNP
&/CC
World/NNP
Report/NNP
./.
Alan/NNP
Spoon/NNP
,/,
recently/RB
named/VBN
Newsweek/NNP
president/NN
,/,
said/VBD
Newsweek/NNP
's/POS
ad/NN
rates/NNS
would/MD
increase/VB
5/CD
%/NN
in/IN
January/NNP
./.
A/DT
full/JJ
,/,
four-color/JJ
page/NN
in/IN
Newsweek/NNP
will/MD
cost/VB
$/$
100,980/CD
./.
In/IN
mid-October/NNP
,/,
Time/NNP
magazine/NN
lowered/VBD
its/PRP$
guaranteed/VBN
circulation/NN
rate/NN
base/NN
for/IN
1990/CD
while/IN
not/RB
increasing/VBG
ad/NN
page/NN
rates/NNS
;/:
with/IN
a/DT
lower/JJR
circulation/NN
base/NN
,/,
Time/NNP
's/POS
ad/NN
rate/NN
will/MD
be/VB
effectively/RB
7.5/CD
%/NN
higher/JJR
per/IN
subscriber/NN
;/:
a/DT
full/JJ
page/NN
in/IN
Time/NNP
costs/VBZ
about/IN
$/$
120,000/CD
./.
U.S./NNP
News/NNP
has/VBZ
yet/RB
to/TO
announce/VB
its/PRP$
1990/CD
ad/NN
rates/NNS
./.
Newsweek/NNP
said/VBD
it/PRP
will/MD
introduce/VB
the/DT
Circulation/NNP
Credit/NNP
Plan/NNP
,/,
which/WDT
awards/VBZ
space/NN
credits/NNS
to/TO
advertisers/NNS
on/IN
``/``
renewal/NN
advertising/NN
./.
''/''
The/DT
magazine/NN
will/MD
reward/VB
with/IN
``/``
page/NN
bonuses/NNS
''/''
advertisers/NNS
who/WP
in/IN
1990/CD
meet/VBP
or/CC
exceed/VBP
their/PRP$
1989/CD
spending/NN
,/,
as/RB
long/RB
as/IN
they/PRP
spent/VBD
$/$
325,000/CD
in/IN
1989/CD
and/CC
$/$
340,000/CD
in/IN
1990/CD
./.

NLP-基础知识-003(词性标注)相关推荐

NLP基础知识（语法语义、LDA、N-gram、词嵌入）
文章目录本节课大纲 Hyper-simplified linguistics Term spotting + handling negation, uncertainty ML to expand ...
NLP基础知识（三）-AI应用篇
文本间的推理关系,又称为文本蕴含关系 (TextualEntailment),作为一种基本的文本间语义联系,广泛存在于自然语言文本中. 简单的来说文本蕴含关系描述的是两个文本之间的推理关系,其中一个文 ...
【NLP基础知识】1.前言及研究方向概述
目录 1.前言什么是NLP? 学习课程前的重点申明如何学? 教材推荐录读过的很有收获的笔记(research篇) 读过的很有收获的笔记(工程实验篇) 重要模型一览语言模型自然语言处理的基本方 ...
NLP基础知识之语音识别
1. 语音识别的输出类别: 1)phoneme:输出为发音,比较简单,因为语音跟发音是一一对应的,但是需要一个词汇表,表示发音跟word的对应. 2)Grapheme:字母或者token 3)word ...
NLP汉语自然语言处理入门基础知识介绍
NLP汉语自然语言处理入门基础知识介绍自然语言处理定义: 自然语言处理是一门计算机科学.人工智能以及语言学的交叉学科.虽然语言只是人工智能的一部分(人工智能还包括计算机视觉等),但它是非常独特的一部 ...
Penn Treebank数据集介绍+句法分析parsed的基本语法+句法分析基础知识+NLP常用公开数据集汇总及下载
Penn Treebank数据集介绍+句法分析parsed的基本语法+句法分析基础知识+NLP常用公开数据集汇总及下载 Penn Treebank数据集介绍 NLP底层技术之句法分析 NLP常用公开数 ...
[003]Reference in C++---C++引用基础知识篇
最近在搞C++的引用,闲来无事,顺便写写引用的基础知识引言:什么是引用引用就是某一变量(目标)的一个别名,对引用的操作与对变量直接操作完全一样. 引用的格式: 数据类型 &引用变量名 = ...
matlab中if语句多个_MATLAB入门教程 | 003基础知识
一.认识MATLAB 1. MATLAB概述 (1)在欧美各高等学校, Matlab成为线性代数.自动控制理论.数字信号处理.时间序列分析.动态系统仿真.图像处理等诸多课程的基本教学工具,成为本科生. ...
《统计自然语言处理》读书笔记一.基础知识及概念介绍
最近准备学习自然语言处理相关的知识,主要参考<统计自然语言处理·宗成庆>和<Natural Language Processing with Python>,推荐大家阅读.第一 ...

NLP-基础知识-003(词性标注)

NLP-基础知识-003(词性标注)相关推荐

最新文章

热门文章