IKAnalyzer2012FF + Lucene4.9 TokenStream contract violation: reset()/close() call missing

异常信息如下：

java.lang.IllegalStateException: TokenStream contract violation: reset()/close() call missing, reset() called multiple times, or subclass does not call super.reset(). Please see Javadocs of TokenStream class for more information about the correct consuming workflow.at org.apache.lucene.analysis.Tokenizer$1.read(Tokenizer.java:111)at java.io.Reader.read(Reader.java:140)at org.wltea.analyzer.core.AnalyzeContext.fillBuffer(AnalyzeContext.java:124)at org.wltea.analyzer.core.IKSegmenter.next(IKSegmenter.java:122)at org.wltea.analyzer.lucene.IKTokenizer.incrementToken(IKTokenizer.java:78)at unit.test.IKAnalyzerTest.test01(IKAnalyzerTest.java:29)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:601)at junit.framework.TestCase.runTest(TestCase.java:168)at junit.framework.TestCase.runBare(TestCase.java:134)at junit.framework.TestResult$1.protect(TestResult.java:110)at junit.framework.TestResult.runProtected(TestResult.java:128)at junit.framework.TestResult.run(TestResult.java:113)at junit.framework.TestCase.run(TestCase.java:124)at junit.framework.TestSuite.runTest(TestSuite.java:243)at junit.framework.TestSuite.run(TestSuite.java:238)at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

代码：

public void test01() throws IOException {
<span style="white-space:pre"> </span>String text="基于java语言开发的轻量级的中文分词工具包，一个轻量级框架!"; // 创建分词对象  Analyzer analyzer = new IKAnalyzer(true);       StringReader reader = new StringReader(text);  // 分词  TokenStream ts = analyzer.tokenStream("", reader);  CharTermAttribute term = ts.getAttribute(CharTermAttribute.class);  // 遍历分词数据  while(ts.incrementToken()){  System.out.print(term.toString()+" | ");  }  reader.close();  System.out.println();
}

上面的代码为旧的分词步骤，按照新的API，调用TokenStream的流程如下：

1、Instantiation of TokenStream/TokenFilters which add/get attributes to/ the AttributeSource.
2、The consumer calls reset（）.
3、The consumer retrieves attributes the stream and stores local references to all attributes it wants to access.
4、The consumer calls incrementToken（） until it returns false consuming the attributes after each call.
5、The consumer calls end（） so that any end-of-stream operations can be performed.
6、The consumer calls close（） to release any resource when finished using the TokenStream.

所以在调用incrementToken（）之前需要调用一次reset()，如下面的第10行代码：

public void test01() throws IOException {String text="基于java语言开发的轻量级的中文分词工具包，一个轻量级框架!"; // 创建分词对象  Analyzer analyzer = new IKAnalyzer(true);       StringReader reader = new StringReader(text);  // 分词  TokenStream ts = analyzer.tokenStream("", reader); ts.reset();CharTermAttribute term = ts.getAttribute(CharTermAttribute.class);  // 遍历分词数据  while(ts.incrementToken()){  System.out.print(term.toString()+" | ");  }  reader.close();  System.out.println();  }

IKAnalyzer2012FF + Lucene4.9 TokenStream contract violation: reset()/close() call missing相关推荐

遇到java.lang.IllegalStateException: TokenStream contract violation: reset()
解决办法 : 加 ts.reset();方法举个栗子: TokenStream ts = analyzer.tokenStream("text", reader);ts.rese ...
Lucene 4.6.1 java.lang.IllegalStateException: TokenStream contract violation
这是旧代码在新版本Lucene中出现的异常,异常如下: Exception in thread "main" java.lang.IllegalStateException: To ...
Lucene分词报错：”TokenStream contract violation: close() call missing”
Lucene使用IKAnalyzer分词时报错:"TokenStream contract violation: close() call missing" 解决办法是每次完成后 ...
map contract violation
map: contract violation 违反约定 expected: list? given: 3
｛UserDetailsService returned null, which is an interface contract violation｝
关于遇到的:"UserDetailsService returned null, which is an interface contract violation" 在做项目的时候 ...
lucene 7.x 分词 TokenStream的使用及源码分析
一.使用步骤 1 //将一个字符串创建成token流,第一个参数---fiedName,是一种标志性参数,可以写空字符串,不建议用null,因为null对于IKAnalyzer会包错 2 TokenS ...
Lucene IKAnalyzer(V2012)
IKAnalyzer:IK分词器,采用特有的"正向迭代最细粒度切分算法",支持细粒度和智能分词两种切分方式. 支持中文.数字.中文词汇等分词处理,兼容韩文.日语字符. 例子:主要展 ...
lucene 分词源码分析
lucene 分词源码分析流程分析借助于 ES 分词DSL 看一下 analyzer的组成: # 可以看出一个 analyzer 由 char_filter(任意个) + tokenier(固定一 ...
lucene java maven_Lucene 6.5.0整合IKAnalyzer至Maven
1. 下载IKAnalyzer6.5.0.jar 度盘链接:http://pan.baidu.com/s/1jH4NY66 2. 打开cmd, 执行以下命令手动将jar包安装到本地maven仓 ...

IKAnalyzer2012FF + Lucene4.9 TokenStream contract violation: reset()/close() call missing

IKAnalyzer2012FF + Lucene4.9 TokenStream contract violation: reset()/close() call missing相关推荐

最新文章

热门文章