Java访问Elasticsearch报错Request cannot be executed; I/O reactor status: STOPPED
简介
使用ES过程中遇到一个Request cannot be executed; I/O reactor status: STOPPED 的异常,大概意思是和server端的连接异常终止了。开始以为是引用的版本不对,或者自己使用问题,后来发现就是因为OOM导致程序宕机,进而引发连接终止。
环境
功能
SpringBoot 的程序通过 SpringDataElasticsearch 访问ES-server 获取数据。
ES-SERVER
- 版本:7.15.2
ES-CLIENT
ES-CLIENT 就是 SpringBoot 程序,核心pom依赖:
<dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-data-elasticsearch</artifactId><version>2.5.3</version>
</dependency>
问题
- 部署到测试环境之后,用户量大了之后接口报错,报错信息为:
Request cannot be executed; I/O reactor status: STOPPED
- 后端程序处于假死状态,访问其他接口一直转圈,无法响应内容
- 重启程序之后一切正常,用户量大了之后又有此问题
- 详细报错日志为:
2021-12-31/19:26:31.748||||^_^|[pool-4-thread-1] ERROR o.a.http.impl.nio.client.InternalHttpAsyncClient 66 - I/O reactor terminated abnormally
org.apache.http.nio.reactor.IOReactorException: I/O dispatch worker terminated abnormallyat org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:359)at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.OutOfMemoryError: Java heap space
2021-12-31/19:26:32.783||||^_^|[http-nio-8092-exec-3] ERROR com.nghsmart.datacenter.handler.ExceptionHandler 67 - handleRuntimeException:
java.lang.RuntimeException: Request cannot be executed; I/O reactor status: STOPPEDat org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:888)at org.elasticsearch.client.RestClient.performRequest(RestClient.java:283)at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1654)at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1624)at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1594)at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1110)at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.lambda$search$11(ElasticsearchRestTemplate.java:296)at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.execute(ElasticsearchRestTemplate.java:383)at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.search(ElasticsearchRestTemplate.java:296)at org.springframework.data.elasticsearch.core.AbstractElasticsearchTemplate.search(AbstractElasticsearchTemplate.java:513)at com.nghsmart.datacenter.domain.area.service.impl.AreaDashboradServiceImpl.getRotationStatisticsInfo(AreaDashboradServiceImpl.java:166)at com.nghsmart.datacenter.domain.area.controller.AreaDashboradController.getRotationStatisticsInfo(AreaDashboradController.java:76)at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.base/java.lang.reflect.Method.invoke(Method.java:566)at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197)at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141)at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1064)at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)at javax.servlet.http.HttpServlet.service(HttpServlet.java:655)at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:228)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)at com.nghsmart.commonauthentication.filter.JwtAuthenticationTokenFilter.doFilterInternal(JwtAuthenticationTokenFilter.java:52)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382)at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPEDat org.apache.http.util.Asserts.check(Asserts.java:46)at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)at org.elasticsearch.client.RestClient.performRequest(RestClient.java:279)
... 69 common frames omitted
- 测试环境用户量一大就报,本地IDEA中一次也没有报这个问题
- 使用JMETER设置50个线程并发访问可以稳定复现此问题
原因
程序接口中将一块很大的数据存进JAVA集合中引发了oom,oom异常导致程序宕机,处于假死状态,进而导致 ES-CLIENT 和 ES-SERVER 端的 http 连接异常终止,然后org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning 方法报异常。
ensureRunning:
protected void ensureRunning() {final Status currentStatus = this.status.get();Asserts.check(currentStatus == Status.ACTIVE, "Request cannot be executed; " +"I/O reactor status: %s", currentStatus);
}
check:
public static void check(final boolean expression, final String message, final Object arg) {if (!expression) {throw new IllegalStateException(String.format(message, arg));}
}
- SpringDataElasticsearch 和ES-SERVER 是长链接,只要报了OOM,当前和 ES-SERVER 的连接线程都将报异常,也就是说,虽然OOM只报了一次,但是可能有多个线程都在 Asserts.check 方法中报异常
- 其实不仅仅是和ES-SERVER 的连接异常关闭,观察大量日志后发现和nacos 的连接也有问题:
2022-01-04 15:59:07.366 [com.alibaba.nacos.client.remote.worker] INFO com.alibaba.nacos.common.remote.client.printIfInfoEnabled:60 - [117be10e-119f-4253-a348-71c95a8978a1]Server healthy check fail,currentConnection=1641282430796_192.168.1.90_43108
2022-01-04 15:59:07.367 [com.alibaba.nacos.client.remote.worker] INFO com.alibaba.nacos.common.remote.client.printIfInfoEnabled:60 - [117be10e-119f-4253-a348-71c95a8978a1] try to re connect to a new server ,server is not appointed,will choose a random server.
- 测试环境运行SpringBoot内存配置为:-Xms256m -Xmx512m,使用 java -jar 命令启动。测试环境能复现,本地idea无法复现的原因是内存配置不同导致的,本地idea内存给的2g。编辑 run/debug configuration ,设置 VM options 和测试环境相同,使用jemter压测本地idea中跑的程序,也能复现此问题
解决办法
- 优化接口,解决OOM问题
- 增大测试环境内存配置
解决过程
HttpClient问题
这个思路并没有解决我的问题,但是解决问题的过程中花了大量时间研究。其实应该仔细阅读报错log的上下文和接口代码上下文,一开始没有留意到oom的异常
大概意思是说低版本 httpClient 有这个问题,高版本就没有这个问题,是apache-httpclient 导致的。具体是因为捕获到线程异常之后http连接就主动终止了,解决思路是升级版本或者是重写处理器,遇到异常之后不终止连接。
升级httpclient相关pom依赖:
<dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-data-elasticsearch</artifactId><!-- 排除 httpclient、httpcore、 httpcore-nio这三个依赖,并手动引入最新版本,解决es客户端报Request cannot be executed; I/O reactor status: STOPPED问题 --><exclusions>
<!-- <exclusion>-->
<!-- <groupId>org.apache.httpcomponents</groupId>-->
<!-- <artifactId>httpclient</artifactId>-->
<!-- </exclusion>-->
<!-- <exclusion>-->
<!-- <groupId>org.apache.httpcomponents</groupId>-->
<!-- <artifactId>httpcore</artifactId>-->
<!-- </exclusion>-->
<!-- <exclusion>-->
<!-- <groupId>org.apache.httpcomponents</groupId>-->
<!-- <artifactId>httpcore-nio</artifactId>-->
<!-- </exclusion>--><exclusion><groupId>org.elasticsearch.client</groupId><artifactId>elasticsearch-rest-client</artifactId></exclusion></exclusions>
</dependency><!-- start -->
<dependency><groupId>org.elasticsearch.client</groupId><artifactId>elasticsearch-rest-client</artifactId><version>7.15.2</version><exclusions><exclusion><groupId>org.apache.httpcomponents</groupId><artifactId>httpclient</artifactId></exclusion><exclusion><groupId>org.apache.httpcomponents</groupId><artifactId>httpcore</artifactId></exclusion><exclusion><groupId>org.apache.httpcomponents</groupId><artifactId>httpcore-nio</artifactId></exclusion></exclusions>
</dependency>
<!-- end -->
<!-- start:排除 httpclient、httpcore、 httpcore-nio这三个依赖,并手动引入最新版本,解决es客户端报Request cannot be executed; I/O reactor status: STOPPED问题 -->
<dependency><groupId>org.apache.httpcomponents</groupId><artifactId>httpclient</artifactId><version>4.5.13</version>
</dependency><dependency><groupId>org.apache.httpcomponents</groupId><artifactId>httpcore-nio</artifactId><version>4.4.14</version>
</dependency><dependency><groupId>org.apache.httpcomponents</groupId><artifactId>httpcore</artifactId><version>4.4.14</version>
</dependency><!-- end -->
重写ElasticsearchRestTemplate:
import com.alibaba.fastjson.JSON;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.Header;
import org.apache.http.HttpHost;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.impl.client.DefaultConnectionKeepAliveStrategy;
import org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager;
import org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor;
import org.apache.http.message.BasicHeader;
import org.apache.http.nio.reactor.IOReactorException;
import org.apache.http.nio.reactor.IOReactorExceptionHandler;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestClientBuilder;
import org.elasticsearch.client.RestHighLevelClient;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
@Slf4j
@Configuration
public class EsClientConfig {@Value("${es.host}")private String host;@Value("${es.port}")private Integer port;@Beanpublic ElasticsearchRestTemplate initElasticsearchRestTemplate() {RestClientBuilder builder = RestClient.builder(new HttpHost(host, port)).setRequestConfigCallback(config -> config.setConnectTimeout(180000).setConnectionRequestTimeout(180000).setSocketTimeout(180000)).setHttpClientConfigCallback(httpClientBuilder -> {httpClientBuilder.setMaxConnTotal(100);httpClientBuilder.setMaxConnPerRoute(50);List<Header> headers = new ArrayList<>(2);headers.add(new BasicHeader("Connection", "keep-alive"));headers.add(new BasicHeader("Keep-Alive", "720"));httpClientBuilder.setDefaultHeaders(headers);httpClientBuilder.setKeepAliveStrategy(DefaultConnectionKeepAliveStrategy.INSTANCE);try {DefaultConnectingIOReactor ioReactor = new DefaultConnectingIOReactor();ioReactor.setExceptionHandler(new IOReactorExceptionHandler() {@Overridepublic boolean handle(IOException e) {log.debug("System may be unstable: IOReactor encountered a checked exception : " + e.getMessage());log.debug("start setHttpClientConfigCallback handle IOException e printStackTrace");e.printStackTrace();log.debug("end setHttpClientConfigCallback handle IOException e printStackTrace");// Return true to note this exception as handled, it will not be re-thrownreturn true;}@Overridepublic boolean handle(RuntimeException e) {log.debug("System may be unstable: IOReactor encountered a runtime exception : " + e.getMessage() + ",e is {}", e);log.debug("start setHttpClientConfigCallback handle RuntimeException e printStackTrace ");e.printStackTrace();log.debug("end setHttpClientConfigCallback handle RuntimeException e printStackTrace ");// Return true to note this exception as handled, it will not be re-thrownreturn true;}});httpClientBuilder.setConnectionManager(new PoolingNHttpClientConnectionManager(ioReactor));} catch (IOReactorException e) {throw new RuntimeException(e);}return httpClientBuilder;});ElasticsearchRestTemplate elasticsearchRestTemplate = new ElasticsearchRestTemplate(new RestHighLevelClient(builder));return elasticsearchRestTemplate;}
}
关于此问题相关参考:
- https://www.cnblogs.com/yangchongxing/p/15440197.html
- https://github.com/elastic/elasticsearch/issues/42133
- https://zhuanlan.zhihu.com/p/384269417
- https://cloud.tencent.com/developer/article/1806886
引用
https://www.cnblogs.com/Naylor/p/15763941.html
Java访问Elasticsearch报错Request cannot be executed; I/O reactor status: STOPPED相关推荐
- ES 查询时报错 I/O 异常: Request cannot be executed; I/O reactor status: STOPPED
问题: ES 查询时报错 I/O 异常: java.lang.IllegalStateException: Request cannot be executed; I/O reactor status ...
- Request cannot be executed; I/O reactor status: STOPPED
使用ES报错: java.lang.RuntimeException: Request cannot be executed; I/O reactor status: STOPPED at o ...
- 关于ES Request cannot be executed; I/O reactor status: STOPPED 异常原因查找
项目场景: 项目用到ES 存储日志数据, 问题描述 程序使用RestHighLevelClient客户端,经常出现Request cannot be executed; I/O reactor sta ...
- Nacos 报错:Request cannot be executed; I/O reactor status: STOPPED
报错信息 Nacos集群报错如下: 2021-07-13 16:21:56,363 ERROR [IP-DEAD] failed to delete ip automatically, ip: {&q ...
- java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
es7.3.2 运行时候java 客户端报错如下: java.lang.RuntimeException: Request cannot be executed; I/O reactor status ...
- [正解]异常信息:java.lang.RuntimeException: Request cannot be executed; I/O reactor status: STOPPED
笔者,从事大型项目监控设计流程,有一次需要将ES整合到项目中,ELK的那一套日志作为用户接口的调用基础,类似于采集.清洗.转换.存储. 刚开始设计代码及流程,之后把代码进行优化抽取,封装client等 ...
- ElasticSearch:Request cannot be executed; I/O reactor status: STOPPED
项目场景: 项目添加Elasticsearch7.8统计部分 2020年10月10日部署到测试环境 10月14日突然报错 问题描述: java.lang.RuntimeException: Reque ...
- 记一次错误:java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
这两天测试给提了个BUG,具体现象是:调用es查询数据的接口进行过一次查询,过几分钟之后再次用这个接口进行查询就不行了,日志抛错堆栈如下: java.lang.IllegalStateExceptio ...
- 【Elasticsearch】Request cannot be executed;I/O reactor status :STOPPED.
1.概述 转载:https://bbs.huaweicloud.com/forum/thread-70234-1-1.html 我也曾经遇到这个错误,但是忘记记录了,现在转载记录 多线程并发写入ES时 ...
最新文章
- 设计模式之命令模式、举例分析、通俗易懂
- Spark编程指南笔记
- 个人博客 SEO 优化(2):站内优化
- Java并发编程之:Vector和ArrayList的区别
- 日积月累-从细节做起
- python-进程、线程
- 文件上怎么盖章_投标文件该怎么盖章呢?投标人必看!
- 面向对象的三大特性————继承,多态
- Several ports (8080, 8009) required by Tomcat v9.0 Server at localhost are already in use.解决方案
- 谷歌披露影响多个苹果操作系统的零点击Image I/O 漏洞和开源库 OpenEXR漏洞
- 中兴f477v2超级管理员_中兴本机电信光猫超级密码获取-中兴F412/F460/F612/F660超级密码获取下载V1.0最新版-西西软件下载...
- UART协议就应该这么理解
- MVC/POJO/POJI/DAO/DTO/VO
- 副业做淘宝可以么?淘宝可以当做副业来做吗?
- Linux挂载OneDrive
- boos里的AHCI RAID_如何用Intel主板集成的RAID控制器(Intel RST)实现硬盘提速与硬盘数据恢复...
- Unity 调用DLL
- var foo = 1; (function foo() { foo = 100; console.log(foo); }()) console.log(foo);
- 实例011阳阳买苹果
- win10注册mysql到windows服务报错:Install/Remove of the Service Denied