序

本文主要研究下如何使用opennlp进行文档分类

DoccatModel

要对文档进行分类，需要一个最大熵模型(Maximum Entropy Model)，在opennlp中对应DoccatModel

    @Testpublic void testSimpleTraining() throws IOException {ObjectStream<DocumentSample> samples = ObjectStreamUtils.createObjectStream(new DocumentSample("1", new String[]{"a", "b", "c"}),new DocumentSample("1", new String[]{"a", "b", "c", "1", "2"}),new DocumentSample("1", new String[]{"a", "b", "c", "3", "4"}),new DocumentSample("0", new String[]{"x", "y", "z"}),new DocumentSample("0", new String[]{"x", "y", "z", "5", "6"}),new DocumentSample("0", new String[]{"x", "y", "z", "7", "8"}));TrainingParameters params = new TrainingParameters();params.put(TrainingParameters.ITERATIONS_PARAM, 100);params.put(TrainingParameters.CUTOFF_PARAM, 0);DoccatModel model = DocumentCategorizerME.train("x-unspecified", samples,params, new DoccatFactory());DocumentCategorizer doccat = new DocumentCategorizerME(model);double[] aProbs = doccat.categorize(new String[]{"a"});Assert.assertEquals("1", doccat.getBestCategory(aProbs));double[] bProbs = doccat.categorize(new String[]{"x"});Assert.assertEquals("0", doccat.getBestCategory(bProbs));//test to make sure sorted map's last key is cat 1 because it has the highest score.SortedMap<Double, Set<String>> sortedScoreMap = doccat.sortedScoreMap(new String[]{"a"});Set<String> cat = sortedScoreMap.get(sortedScoreMap.lastKey());Assert.assertEquals(1, cat.size());}

这里为了方便测试，先手工编写DocumentSample来做训练文本
categorize方法返回的是一个概率，getBestCategory可以根据概率来返回最为匹配的分类

输出如下：

Indexing events with TwoPass using cutoff of 0Computing event counts...  done. 6 eventsIndexing...  done.
Sorting and merging events... done. Reduced 6 events to 6.
Done indexing in 0.13 s.
Incorporating indexed data for training...
done.Number of Event Tokens: 6Number of Outcomes: 2Number of Predicates: 14
...done.
Computing model parameters ...
Performing 100 iterations.1:  ... loglikelihood=-4.1588830833596715    0.52:  ... loglikelihood=-2.6351991759048894    1.03:  ... loglikelihood=-1.9518912133474995    1.04:  ... loglikelihood=-1.5599038834410852    1.05:  ... loglikelihood=-1.3039748361952568    1.06:  ... loglikelihood=-1.1229511041438864    1.07:  ... loglikelihood=-0.9877356230661396    1.08:  ... loglikelihood=-0.8826624290652341    1.09:  ... loglikelihood=-0.7985244514476817    1.010:  ... loglikelihood=-0.729543972551105    1.0
//...95:  ... loglikelihood=-0.0933856684859806    1.096:  ... loglikelihood=-0.09245907503183291    1.097:  ... loglikelihood=-0.09155090064000486    1.098:  ... loglikelihood=-0.09066059844628399    1.099:  ... loglikelihood=-0.08978764309881068    1.0
100:  ... loglikelihood=-0.08893152970793908    1.0

小结

opennlp的categorize方法需要自己先切词好，单独调用不是很方便，不过如果是基于pipeline设计的，也可以理解，在pipeline前面先经过切词等操作。本文仅仅是使用官方的测试源码来做介绍，读者可以下载个中文分类文本训练集来训练，然后对中文文本进行分类。

doc

• Document Categorizer API