古德图灵估计（Good-Turing Estimation)

古德-图灵估计最早发表于1953年。其核心思想是用r*取代原始的r。其中r*的定义为：
r ∗ = ( r + 1 ) n r + 1 n r r^{*}=(r+1)\frac{n_{r+1}}{n_{r}} r∗=(r+1)nrnr+1
新的频次计算为：
P r = r ∗ N P_{r}=\frac{r^{*}}{N} Pr=Nr∗
N可以表示为：
N = ∑ r = 0 ∞ n r r ∗ = ∑ r = 0 ∞ ( r + 1 ) n r + 1 = ∑ r = 1 ∞ n r r N= \sum_{r=0}^{\infty} n_{r}r^{*}=\sum_{r=0}^{\infty}(r+1)n_{r+1}=\sum_{r=1}^{\infty}n_{r}r N=r=0∑∞nrr∗=r=0∑∞(r+1)nr+1=r=1∑∞nrr
在加上 P r P_{r} Pr的情况下，有：
∑ r = 1 ∞ n r P r = 1 N ∑ r = 1 ∞ ( r + 1 ) n r + 1 = 1 N ( ∑ r = 1 ∞ n r r − n 1 ) = 1 − n 1 N \sum_{r=1}^{\infty} n_{r}P_{r}=\frac{1}{N}\sum_{r=1}^{\infty}(r+1)n_{r+1}=\frac{1}{N}(\sum_{r=1}^{\infty}n_{r}r-n_{1})=1-\frac{n_{1}}{N} r=1∑∞nrPr=N1r=1∑∞(r+1)nr+1=N1(r=1∑∞nrr−n1)=1−Nn1
因此古德图灵估计相当于把 n 1 N \frac {n_{1}}{N} Nn1 的概率分配给了未出现的词汇。
为了保证计算概率之和为1，在计算的时候需要把概率归一化。也就是说在不考虑未出现单词的情况下，保证所有出现单词的概率之和为 1 − n 1 N 1-\frac{n_{1}}{N} 1−Nn1。方法为每个概率都乘以一个归一化因子。归一化因子的公式见后面的说明。

举例说明如下：
假设有单词组“the the the big big dog”，其中the出现3次，big出现2次，dog出现1次。正常的情况下计算单词出现的频次 = 单词出现次数总单词数 =\frac{单词出现次数}{总单词数} =总单词数单词出现次数。比如the的频次=3/6=0.5。
使用古德-图灵算法的情况下，对应的参数为：

单词出现次数	总次数	r+1	n_r	n_r+1	P_r	归一化因子	概率
3	6	4	1	1	4/6	5/9	10/27
2	6	3	1	1	3/6	5/9	5/18
1	6	2	1	1	2/6	5/9	5/27

其中归一化因子:
1 − n 1 N Σ P r n r N = N − n 1 Σ P r n r = N − n 1 Σ ( r + 1 ) n r + 1 \frac{1-\frac{n_{1}}{N}}{\frac{\Sigma P_{r}n_{r}}{N}}=\frac{N-n_{1}}{\Sigma P_{r}n_{r}}=\frac{N-n_{1}}{\Sigma (r+1)n_{r+1}} NΣPrnr1−Nn1=ΣPrnrN−n1=Σ(r+1)nr+1N−n1
对于本例为：（6-1)/( 4 ∗ 1 6 \frac{4*1}{6} 64∗1+ 3 ∗ 1 6 \frac{3*1}{6} 63∗1+ 2 ∗ 1 6 \frac{2*1}{6} 62∗1)= 5 6 \frac{5}{6} 65/ 9 6 \frac{9}{6} 69= 5 9 \frac{5}{9} 95
概率= P r 归一化因子 \frac{P_{r}}{归一化因子} 归一化因子Pr。
上述情况比较特殊，每个频率出现的单词数量都是1个。考虑一种复杂的情况，比如单词组“the the the big big are are dog some other”，对应的结果为：

出现次数	总次数	r+1	n_r	n_r+1	P_r	归一化因子	概率
3	10	4	1	1	2/5	7/11	14/55
2	10	3	2	1	3/10	7/11	21/110
1	10	2	3	2	2/15	7/11	14/165

以上是最简单的古德图灵估计。但是假如某个频次的词汇为0的情况，比如出现13次的词汇没有，要如何处理呢？最简单的方法当然是进行插值。在https://www.grsampson.net/AGtf1.html中，Gale & Sampson提出了另外一种方法是使用对数的形式进行插值。其算法为：
l o g ( n r ) = a + b ∗ l o g ( r ) n r = e a + b ∗ l o g ( r ) \begin{aligned} log(n_{r})&=a+b*log(r)\\ n_{r}&=e^{a+b*log(r)} \end{aligned} log(nr)nr=a+b∗log(r)=ea+b∗log(r)

其中：
x _ m e a n = ∑ r = 1 τ l o g ( r ) τ y _ m e a n = ∑ r = 1 τ l o g ( z r ) τ x y _ c o v = ∑ r = 1 τ l o g ( r − x _ m e a n ) ∗ l o g ( z r − y _ m e a n ) x _ v a r = ∑ r = 1 τ ( l o g ( r ) − x _ m e a n ) 2 b = { x y _ c o v x _ v a r i f x _ v a r ≠ 0 0 o t h e r w i s e a = y _ m e a n − b ∗ x _ m e a n \begin{aligned} x\_mean &= \frac{\sum_{r=1}^{\tau} log(r)}{\tau} \\ y\_mean &= \frac{\sum_{r=1}^{\tau} log(z_{r})}{\tau} \\ xy\_cov &= \sum_{r=1}^{\tau} log(r-x\_mean)*log(z_{r} - y\_mean)\\ x\_var&=\sum_{r=1}^{\tau} {(log(r) - x\_mean) }^{ 2}\\ b&=\left\{ \begin{matrix} \frac{xy\_cov}{x\_var} & \quad \quad if\quad x\_var\neq 0\\ 0 & \quad otherwise \end{matrix} \right. \\ a&=y\_mean - b * x\_mean \end{aligned} x_meany_meanxy_covx_varba=τ∑r=1τlog(r)=τ∑r=1τlog(zr)=r=1∑τlog(r−x_mean)∗log(zr−y_mean)=r=1∑τ(log(r)−x_mean)2={x_varxy_cov0ifx_var=0otherwise=y_mean−b∗x_mean
对应nltk中代码为：

    def find_best_fit(self, r, nr):"""Use simple linear regression to tune parameters self._slope andself._intercept in the log-log space based on count and Nr(count)(Work in log space to avoid floating point underflow.)"""# For higher sample frequencies the data points becomes horizontal# along line Nr=1. To create a more evident linear model in log-log# space, we average positive Nr values with the surrounding zero# values. (Church and Gale, 1991)if not r or not nr:# Empty r or nr?returnzr = []for j in range(len(r)):i = r[j - 1] if j > 0 else 0k = 2 * r[j] - i if j == len(r) - 1 else r[j + 1]zr_ = 2.0 * nr[j] / (k - i)zr.append(zr_)log_r = [math.log(i) for i in r]log_zr = [math.log(i) for i in zr]xy_cov = x_var = 0.0x_mean = sum(log_r) / len(log_r)y_mean = sum(log_zr) / len(log_zr)for (x, y) in zip(log_r, log_zr):xy_cov += (x - x_mean) * (y - y_mean)x_var += (x - x_mean) ** 2self._slope = xy_cov / x_var if x_var != 0 else 0.0if self._slope >= -1:print("SimpleGoodTuring did not find a proper best fit ""line for smoothing probabilities of occurrences. ""The probability estimates are likely to be ""unreliable.")self._intercept = y_mean - self._slope * x_mean

例子如下：

from nltk.probability import SimpleGoodTuringProbDist,FreqDist,ProbDistI
from nltk.tokenize import word_tokenize
sent = "the the the big big are are dog some other"
words = word_tokenize(sent)
fdist = FreqDist(words)
gt = SimpleGoodTuringProbDist(fdist)
n = fdist.N()
for k in fdist:print(f'{k:6},{fdist[k]/n:0.6f},{gt.prob(k):0.6f}')

结果：

the   ,0.300000,0.208417
big   ,0.200000,0.139678
are   ,0.200000,0.139678
dog   ,0.100000,0.070743
some  ,0.100000,0.070743
other ,0.100000,0.070743

古德图灵估计（Good-Turing Estimation)相关推荐

Python+OpenCV：姿态估计（Pose Estimation）
Python+OpenCV:姿态估计(Pose Estimation) ################################################################ ...
信道估计（channel estimation）图解——从SISO到MIMO原理介绍
文章目录 1. 引言 2. 通用算法 3. SISO的信道估计 3.1 信道系数的估计 3.2 噪声的估计 4. 2 x 2 MIMO的信道估计 4.1 信道系数的估计 4.2 噪声的估计参考 1. ...
SISO/MIMO信道估计（channel estimation）原理详细图解
1. 引言在所有通信中,信号都会通过一个介质(称为信道),并且信号会失真,或者在信号通过信道时会向信号中添加各种噪声.正确解码接收到的信号而没有太多错误的方法是从接收到的信号中消除信道施加的失真和噪 ...
什么是图灵完备性（Turing complete）？
作者:Ran C 链接:https://www.zhihu.com/question/20115374/answer/288346717 来源:知乎著作权归作者所有.商业转载请联系作者获得授权,非商 ...
人体姿态估计(Human Pose Estimation)技巧方法汇总
关注上方"深度学习技术前沿",选择"星标公众号", 资源干货,第一时间送达! 作者:Poeroz https://zhuanlan.zhihu.com/p/10 ...
视线估计（Gaze Estimation）简介概述
©PaperWeekly 原创 · 作者|俞雨单位|瑞士洛桑联邦理工学院博士研究方向|视线估计.头部姿态估计本文七个篇章总计涵盖 29 篇论文,总结了自深度学习以来,视线估计领域近五年的发展. ...
混合波束成形| 宽带系统基于码本的信道估计《Channel Estimation for Hybrid Architecture-Based Wideband Millimete
关于 <Channel Estimation for Hybrid Architecture-Based Wideband Millimeter Wave System>一文的中文笔记. ...
人体姿态估计(Human Pose Estimation)文献综述
一.研究背景维基百科对人体姿态估计的定义如下: Articulated body pose estimation in computer vision is the study of algorit ...
单目深度估计(Monocular Depth Estimation)论文阅读 2021-01-15
单目深度估计问题公式化:求非线性映射函数一.数据集: NYU Depth:视频序列和dense depth map通过RGB-D采集的,但是不是每一种图像都有深度图,因为映射是离散的. KITTI ...

古德图灵估计（Good-Turing Estimation)

古德图灵估计（Good-Turing Estimation)相关推荐

最新文章

热门文章