Axis2调用webservice报错 AxisFault: Timeout waiting for connection

记录一次正式环境服务报错排查记录。

某日被通知线上服务告警，错误日志全是 Timeout waiting for connection

首先梳理项目架构，项目很简单，就是一个使用axis2构建的webserice的客户端， Axis2版本为1.5.5

开始从此段报错入手排查，定位到 MultiThreadedHttpConnectionManager 这个类的 doGetConnection 方法

private HttpConnection doGetConnection(HostConfiguration hostConfiguration, long timeout) throws ConnectionPoolTimeoutException {HttpConnection connection = null;int maxHostConnections = this.params.getMaxConnectionsPerHost(hostConfiguration);int maxTotalConnections = this.params.getMaxTotalConnections();synchronized (connectionPool) {// we clone the hostConfiguration// so that it cannot be changed once the connection has been retrievedhostConfiguration = new HostConfiguration(hostConfiguration);HostConnectionPool hostPool = connectionPool.getHostPool(hostConfiguration, true);WaitingThread waitingThread = null;boolean useTimeout = (timeout > 0);long timeToWait = timeout;long startWait = 0;long endWait = 0;while (connection == null) {if (shutdown) {throw new IllegalStateException("Connection factory has been shutdown.");}// happen to have a free connection with the right specs//if (hostPool.freeConnections.size() > 0) {connection = connectionPool.getFreeConnection(hostConfiguration);// have room to make more//} else if ((hostPool.numConnections < maxHostConnections) && (connectionPool.numConnections < maxTotalConnections)) {connection = connectionPool.createConnection(hostConfiguration);// have room to add host connection, and there is at least one free// connection that can be liberated to make overall room//} else if ((hostPool.numConnections < maxHostConnections) && (connectionPool.freeConnections.size() > 0)) {connectionPool.deleteLeastUsedConnection();connection = connectionPool.createConnection(hostConfiguration);// otherwise, we have to wait for one of the above conditions to// become true//} else {// TODO: keep track of which hostConfigurations have waiting// threads, so they avoid being sacrificed before necessarytry {if (useTimeout && timeToWait <= 0) {throw new ConnectionPoolTimeoutException("Timeout waiting for connection");}if (LOG.isDebugEnabled()) {LOG.debug("Unable to get a connection, waiting..., hostConfig=" + hostConfiguration);}if (waitingThread == null) {waitingThread = new WaitingThread();waitingThread.hostConnectionPool = hostPool;waitingThread.thread = Thread.currentThread();} else {waitingThread.interruptedByConnectionPool = false;}if (useTimeout) {startWait = System.currentTimeMillis();}hostPool.waitingThreads.addLast(waitingThread);connectionPool.waitingThreads.addLast(waitingThread);connectionPool.wait(timeToWait);} catch (InterruptedException e) {if (!waitingThread.interruptedByConnectionPool) {LOG.debug("Interrupted while waiting for connection", e);throw new IllegalThreadStateException("Interrupted while waiting in MultiThreadedHttpConnectionManager");}// Else, do nothing, we were interrupted by the connection pool// and should now have a connection waiting for us, continue// in the loop and let's get it.} finally {if (!waitingThread.interruptedByConnectionPool) {// Either we timed out, experienced a "spurious wakeup", or were// interrupted by an external thread.  Regardless we need to // cleanup for ourselves in the wait queue.hostPool.waitingThreads.remove(waitingThread);connectionPool.waitingThreads.remove(waitingThread);}if (useTimeout) {endWait = System.currentTimeMillis();timeToWait -= (endWait - startWait);}}}}}return connection;}

得到几个关键变量：

连接池最大连接数 maxTotalConnections=20；

当前连接主机（或相同主机）最大连接数 maxHostConnections=2；

连接池总空闲连接数 connectionPool.freeConnections；

当前连接主机空闲连接数 hostPool.freeConnections；

得到结论一：

当前主机连接数达到最大值，且当前连接主机空闲连接数为0时，获取连接超时时会抛出 Timeout waiting for connection 异常。

到此，初步怀疑某次请求时未正常释放占用的连接，或未回收到空闲连接池中。

继续排查日志，一直找到第一次抛出 Timeout waiting for connection 异常的时候再往上查看还有没有其他错误日志。于是发现了 Transport error: 503 Error: Service Unavailable 这个异常

直接进入源码

/*** Used to handle the HTTP Response** @param msgContext - The MessageContext of the message* @param method     - The HTTP method used* @throws IOException - Thrown in case an exception occurs*/private void handleResponse(MessageContext msgContext,HttpMethodBase method) throws IOException {int statusCode = method.getStatusCode();log.trace("Handling response - " + statusCode);if (statusCode == HttpStatus.SC_OK) {// Save the HttpMethod so that we can release the connection when cleaning upmsgContext.setProperty(HTTPConstants.HTTP_METHOD, method);processResponse(method, msgContext);} else if (statusCode == HttpStatus.SC_ACCEPTED) {// Since we don't expect any content with a 202 response, we must release the connectionmethod.releaseConnection();} else if (statusCode == HttpStatus.SC_INTERNAL_SERVER_ERROR ||statusCode == HttpStatus.SC_BAD_REQUEST) {// Save the HttpMethod so that we can release the connection when cleaning upmsgContext.setProperty(HTTPConstants.HTTP_METHOD, method);Header contenttypeHeader =method.getResponseHeader(HTTPConstants.HEADER_CONTENT_TYPE);String value = null;if (contenttypeHeader != null) {value = contenttypeHeader.getValue();}OperationContext opContext = msgContext.getOperationContext();if(opContext!=null){MessageContext inMessageContext =opContext.getMessageContext(WSDLConstants.MESSAGE_LABEL_IN_VALUE);if(inMessageContext!=null){inMessageContext.setProcessingFault(true);}}if (value != null) {processResponse(method, msgContext);}Object isTransportNonBlocking = msgContext.getProperty(MessageContext.TRANSPORT_NON_BLOCKING);if (isTransportNonBlocking != null && (Boolean)isTransportNonBlocking) {throw new AxisFault(Messages.getMessage("transportError",String.valueOf(statusCode),method.getStatusText()));}} else {throw new AxisFault(Messages.getMessage("transportError",String.valueOf(statusCode),method.getStatusText()));}}

根据我之前读的源码我知道 method.releaseConnection(); 这个方法将会最终调用 connectionPool.freeConnection(conn); 即生成空闲连接放入总连接池中。

而 msgContext.setProperty(HTTPConstants.HTTP_METHOD, method); 这个方法将由其他地方获取调用并最终调用的仍是 method.releaseConnection();

综上可知返回码为503或者任意其他上述代码中未处理的返回码时都不会将占用连接重置为空闲连接放入总连接池中。

到此可知服务告警原因：

因为服务端发生了我们未知的操作，导致客户端获取请求时的返回码为503而占用了所有的当前主机连接数（2个）而不会释放重置为空闲连接放入连接池

于是后续连接从连接池获取连接时均超时抛出异常 Timeout waiting for connection

Axis2调用webservice报错 AxisFault: Timeout waiting for connection相关推荐

【AXIS2 调用WebService报错】The given SCOPAction ..... does not math an operation
场景: 与第三方使用webserver进行数据对接,本地使用axis2生成的客户端程序调用远程服务问题描述说明:axis2 版本 1.7.9 调用报错: The given SCOPAction ...
org.apache.axis2.AxisFault: Timeout waiting for connection
问题背景: RPCServiceClient在调用webService接口调用的时候,如果一段时间内访问量过多,就会出现 org.apache.axis2.AxisFault: Timeout wai ...
php调用webservice报错Class 'SoapClient' not found
php调用webservice报错Class 'SoapClient' not found 原文:php调用webservice报错Class 'SoapClient' not found php在调 ...
调用webservice报错。
自己部署了webservice, 以前调用一直没问题, 前两日服务器中毒,重装系统,. 网站什么的都正常,但是发现webservice不能调用,报错!: 错误: System.InvalidOpera ...
利用JaxWsDynamicClientFactory 调用webservice 报错解决
2019独角兽企业重金招聘Python工程师标准>>> 1.Spring启动跑错,不能存在多个实例cfx,经百度后发现为配置文件出错改成 <jaxws:endpoint id ...
多次使用axis2调用webservice后，报连接超时错误
使用Axis2d的RPCServiceClient调用webservice,连续调用几次web服务后,后台报错,如下所示: org.apache.commons.httpclient.Conn ...
java使用axis2调用webservice接口实例
说明:我目前的项目环境是struts2+spring+mybatis+oracle,以下代码是java使用axis2调用webservice接口实例. import javax.xml.namespa ...
java中web错误返回码,关于在java程序里调用webservice报500返回码的有关问题
关于在java程序里调用webservice报500返回码的问题我现在写了个程序,是调用webservice的,执行后我打印返回码是500 错误信息是 java.io.IOException: S ...
python调用dll报错：ValueError: Procedure called with not enough arguments (4 bytes missing) or wrong call
python调用dll报错:ValueError: Procedure called with not enough arguments (4 bytes missing) or wrong call ...

Axis2调用webservice报错 AxisFault: Timeout waiting for connection

Axis2调用webservice报错 AxisFault: Timeout waiting for connection相关推荐

最新文章

热门文章