记一次jdbc泄漏与解决方案

大家好，我是入错行的bug猫。（http://blog.csdn.net/qq_41399429，谢绝转载）

这是一次非常非常非常脑蛋疼的debug经历

上周五bug猫请假外出。在路上收到公司的紧急电话，说是系统访问慢，各种地方报错。
初步判断是连接池满了，无法获取到Jdbc链接。已经重启过几次了，让bug猫抓紧解决！

当时bug猫身边仅一部手机，赶紧去网吧连远程公司电脑，然后上服务器一看系统的监控，Jdbc的使用方式长这样：

刚开始最大连接数是100，后来改成250。图中Jdbc数量垂直下降原因是，系统重启…
每隔8个小时左右，Jdbc就满了，只能重启系统。电话都被业务部门打爆了…

放大点看，是一个波度平滑的直角三角形！等等，这不是积分电路么？！（暴露专业

毫无疑问，这是Jdbc外泄露！系统Jdbc采用的是阿里的德鲁伊，按照druid官方文档，设置强制回收超时未释放的链接

  <bean id="dataSource" class="com.alibaba.druid.pool.DruidDataSource" destroy-method="close">... ...<property name="removeAbandoned" value="true" /> <!-- 打开removeAbandoned功能 --><property name="removeAbandonedTimeout" value="1800" /> <!-- 1800秒，也就是30分钟 --><property name="logAbandoned" value="true" /> <!-- 关闭abanded连接时输出错误日志 -->... ...</bean>

没卯月！

添加druid自带的监控，

  <!-- 德鲁伊监控 --><servlet><servlet-name>DruidStatView</servlet-name><servlet-class>com.alibaba.druid.support.http.StatViewServlet</servlet-class><init-param><!-- 允许清空统计数据 --><param-name>resetEnable</param-name><param-value>true</param-value></init-param><init-param><!-- 用户名 --><param-name>loginUsername</param-name><param-value>bugcat</param-value></init-param><init-param><!-- 密码 --><param-name>loginPassword</param-name><param-value>bugcat972245132</param-value></init-param></servlet><servlet-mapping><servlet-name>DruidStatView</servlet-name><url-pattern>/druid/*</url-pattern></servlet-mapping>

监控上显示一切正常！

猫了个咪 (╯‵□′)╯︵┻━┻

据说第一次挂的时候，系统已经一个星期没有更新和升级，就是突然发生这种情况。

检查了这段时间内所有提交的代码，有关操作数据库的地方，貌似都没有问题。

观察Jdbc泄露的频率，近似是一条直线，猜测是定时任务造成的，把所有的定时任务全部停止，还是泄露！

系统庞大，到处都是诗山。可能是以前写的功能，之前没人用，现在又用起来了，或者第三方突然发疯猛调接口，或者是以前埋下的暗雷，现在起爆了，全盘检查代码肯定不现实！

QAQ

然后把druid换成了据说目前最牛皮的追光者HikariCP！
依旧是泄露！

根据Jdbc每小时泄露的次数，反向推算一天的访问量，再看访问日志，确定大概是哪个接口造成。等等一系列方法，各种手段都试过，就是不行！！

于是只能手写一个监控了

CatHikariDataSource，自定义的一个dataSource，继承HikariCP，getConnection()返回增强后的CatConnection

import com.zaxxer.hikari.HikariDataSource;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;import java.sql.Connection;
import java.sql.SQLException;
import java.util.Iterator;
import java.util.Map;
import java.util.Timer;
import java.util.TimerTask;public class CatHikariDataSource extends HikariDataSource {private static Logger log = LoggerFactory.getLogger(CatHikariDataSource.class);private static final long max_time = 1200000;/*** 获取 jdbc 链接* @return 增强后的Connection* */@Overridepublic Connection getConnection() throws SQLException {Connection conn = super.getConnection();CatConnection catConn = new CatConnection(conn);return catConn;}/*** 每隔20分钟检查一次未关闭的链接，记录到日志中* */public void init(){Map<String, CatConnection> connMap = CatConnection.getConnMap();Timer timer = new Timer();timer.schedule(new TimerTask(){@Overridepublic void run() {long now = System.currentTimeMillis();Iterator<CatConnection> iter = connMap.values().iterator();while ( iter.hasNext() ) {CatConnection conn = iter.next();if( conn.getActiveTime() + max_time < now ){    //超过20分钟没有提交事务、回滚事务log.warn(DateUtil.getStringDate() + ">>未释放的jdbc链接：lastSql=" + conn.getLastSql(), conn.getEx());}}}}, max_time , max_time);}
}

CatConnection 使用装饰模式，增强Connection


import java.sql.*;
import java.util.Map;
import java.util.Properties;
import java.util.UUID;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.Executor;public class CatConnection implements Connection {/*** 所有活跃的的jdbc链接* */private static Map<String, CatConnection> connMap = new ConcurrentHashMap<>(300);private String id;private long activeTime;    //最后活跃时间private String lastSql;     //该链接执行的最后一个sqlprivate Throwable ex;       //调用的堆栈信息private Connection conn;    //被装饰的原始Connection对象public CatConnection(Connection conn) {this.conn = conn;this.id = UUID.randomUUID().toString();this.activeTime = System.currentTimeMillis();ex = new Throwable();               //记录调用者堆栈信息，重要connMap.put(this.id, this);}@Overridepublic Statement createStatement() throws SQLException {return conn.createStatement();}@Overridepublic PreparedStatement prepareStatement(String sql) throws SQLException {this.lastSql = sql;return conn.prepareStatement(sql);}@Overridepublic CallableStatement prepareCall(String sql) throws SQLException {this.lastSql = sql;return conn.prepareCall(sql);}@Overridepublic String nativeSQL(String sql) throws SQLException {this.lastSql = sql;return conn.nativeSQL(sql);}@Overridepublic void setAutoCommit(boolean autoCommit) throws SQLException {conn.setAutoCommit(autoCommit);}@Overridepublic boolean getAutoCommit() throws SQLException {return conn.getAutoCommit();}@Overridepublic void commit() throws SQLException {conn.commit();this.activeTime = System.currentTimeMillis();}@Overridepublic void rollback() throws SQLException {conn.rollback();this.activeTime = System.currentTimeMillis();}@Overridepublic void close() throws SQLException {conn.close();connMap.remove(this.id);}@Overridepublic boolean isClosed() throws SQLException {return conn.isClosed();}@Overridepublic DatabaseMetaData getMetaData() throws SQLException {return conn.getMetaData();}@Overridepublic void setReadOnly(boolean readOnly) throws SQLException {conn.setReadOnly(readOnly);}@Overridepublic boolean isReadOnly() throws SQLException {return conn.isReadOnly();}@Overridepublic void setCatalog(String catalog) throws SQLException {conn.setCatalog(catalog);}@Overridepublic String getCatalog() throws SQLException {return conn.getCatalog();}@Overridepublic void setTransactionIsolation(int level) throws SQLException {conn.setTransactionIsolation(level);}@Overridepublic int getTransactionIsolation() throws SQLException {return conn.getTransactionIsolation();}@Overridepublic SQLWarning getWarnings() throws SQLException {return conn.getWarnings();}@Overridepublic void clearWarnings() throws SQLException {conn.clearWarnings();}@Overridepublic Statement createStatement(int resultSetType, int resultSetConcurrency) throws SQLException {return conn.createStatement(resultSetType, resultSetConcurrency);}@Overridepublic PreparedStatement prepareStatement(String sql, int resultSetType, int resultSetConcurrency) throws SQLException {return conn.prepareStatement(sql, resultSetType, resultSetConcurrency);}@Overridepublic CallableStatement prepareCall(String sql, int resultSetType, int resultSetConcurrency) throws SQLException {return conn.prepareCall(sql, resultSetType, resultSetConcurrency);}@Overridepublic Map<String, Class<?>> getTypeMap() throws SQLException {return conn.getTypeMap();}@Overridepublic void setTypeMap(Map<String, Class<?>> map) throws SQLException {conn.setTypeMap(map);}@Overridepublic void setHoldability(int holdability) throws SQLException {conn.setHoldability(holdability);}@Overridepublic int getHoldability() throws SQLException {return conn.getHoldability();}@Overridepublic Savepoint setSavepoint() throws SQLException {return conn.setSavepoint();}@Overridepublic Savepoint setSavepoint(String name) throws SQLException {return conn.setSavepoint(name);}@Overridepublic void rollback(Savepoint savepoint) throws SQLException {conn.rollback(savepoint);}@Overridepublic void releaseSavepoint(Savepoint savepoint) throws SQLException {conn.releaseSavepoint(savepoint);}@Overridepublic Statement createStatement(int resultSetType, int resultSetConcurrency, int resultSetHoldability) throws SQLException {return conn.createStatement(resultSetType, resultSetConcurrency, resultSetHoldability);}@Overridepublic PreparedStatement prepareStatement(String sql, int resultSetType, int resultSetConcurrency, int resultSetHoldability) throws SQLException {return conn.prepareStatement(sql, resultSetType, resultSetConcurrency, resultSetHoldability);}@Overridepublic CallableStatement prepareCall(String sql, int resultSetType, int resultSetConcurrency, int resultSetHoldability) throws SQLException {return conn.prepareCall(sql, resultSetType, resultSetConcurrency, resultSetHoldability);}@Overridepublic PreparedStatement prepareStatement(String sql, int autoGeneratedKeys) throws SQLException {return conn.prepareStatement(sql, autoGeneratedKeys);}@Overridepublic PreparedStatement prepareStatement(String sql, int[] columnIndexes) throws SQLException {return conn.prepareStatement(sql, columnIndexes);}@Overridepublic PreparedStatement prepareStatement(String sql, String[] columnNames) throws SQLException {return conn.prepareStatement(sql, columnNames);}@Overridepublic Clob createClob() throws SQLException {return conn.createClob();}@Overridepublic Blob createBlob() throws SQLException {return conn.createBlob();}@Overridepublic NClob createNClob() throws SQLException {return conn.createNClob();}@Overridepublic SQLXML createSQLXML() throws SQLException {return conn.createSQLXML();}@Overridepublic boolean isValid(int timeout) throws SQLException {return conn.isValid(timeout);}@Overridepublic void setClientInfo(String name, String value) throws SQLClientInfoException {conn.setClientInfo(name, value);}@Overridepublic void setClientInfo(Properties properties) throws SQLClientInfoException {conn.setClientInfo(properties);}@Overridepublic String getClientInfo(String name) throws SQLException {return conn.getClientInfo(name);}@Overridepublic Properties getClientInfo() throws SQLException {return conn.getClientInfo();}@Overridepublic Array createArrayOf(String typeName, Object[] elements) throws SQLException {return conn.createArrayOf(typeName, elements);}@Overridepublic Struct createStruct(String typeName, Object[] attributes) throws SQLException {return conn.createStruct(typeName, attributes);}@Overridepublic void setSchema(String schema) throws SQLException {conn.setSchema(schema);}@Overridepublic String getSchema() throws SQLException {return conn.getSchema();}@Overridepublic void abort(Executor executor) throws SQLException {conn.abort(executor);}@Overridepublic void setNetworkTimeout(Executor executor, int milliseconds) throws SQLException {conn.setNetworkTimeout(executor, milliseconds);}@Overridepublic int getNetworkTimeout() throws SQLException {return conn.getNetworkTimeout();}@Overridepublic <T> T unwrap(Class<T> iface) throws SQLException {return conn.unwrap(iface);}@Overridepublic boolean isWrapperFor(Class<?> iface) throws SQLException {return conn.isWrapperFor(iface);}public static Map<String, CatConnection> getConnMap() {return connMap;}public String getLastSql() {return lastSql;}public long getActiveTime() {return activeTime;}public Throwable getEx() {return ex;}
}

然后配置DataSource的地方，指向CatHikariDataSource，并且配置初始方法为init

 <bean id="dataSource" class="com.bugcat.jdbcWatch.CatHikariDataSource" init-method="init" destroy-method="close"><property name="driverClassName" value="${mysql.driver}" />...</bean>

然后坐等半个小时，看日志，就可知道是哪个坑爹的类、哪个sql出错了！

最终泄露原因是，有一个类手动开启了事务，后面紧跟着一个超级大的try/catch代码块，在try最后执行commit，catch块里面执行rollback。
bug猫有几次都检查到这个地方，看似没问题是吧，哼哼~尼玛在try/catch代码块中，竟然有一个return ！！！

心中有十万个草泥马扭着屁股甩着尾巴奔腾而过…

为什么说蛋疼呢，其实druid早就已经给出了问题所在，德鲁伊的监控显示一切正常，说明泄露的Jdbc不归它管！
全局一搜，就2处是手动开启事务…

最终手动开启事务并且直接return的同事被领导和bug猫叼了个爽

记一次jdbc泄漏与解决方案相关推荐

c3p0数据源配置抛出Could not load driverClass com.mysql.jdbc.Driver的解决方案
c3p0数据源配置抛出Could not load driverClass com.mysql.jdbc.Driver的解决方案参考文章: (1)c3p0数据源配置抛出Could not load ...
记一次查询超时的解决方案The timeout period elapsed......
记一次查询超时的解决方案The timeout period elapsed...... 参考文章: (1)记一次查询超时的解决方案The timeout period elapsed...... ( ...
开发过程中程序员非常烦恼的问题，3种内存泄漏的解决方案
程序员非常蛋疼问题,3种内存泄漏的解决方案 1. 内存池的实现以及原理 2. 为内存加上hook,精准内存定位泄露 3. gc的实现方案视频讲解如下,点击观看: 开发过程中程序员非常烦恼的问题,3种 ...
linux服务器突然有大量连接,记一次大量CLOSE_WAIT的解决方案
记一次大量CLOSE_WAIT的解决方案问题: Cannot send, channel has already failed: tcp://ip:61616 Javax.jms.JMSExcept ...
java内部类内存泄漏,Android中常见的内存泄漏和解决方案
什么是内存泄漏? 简单点说,就是指一个对象不再使用,本应该被回收,但由于某些原因导致对象无法回收,仍然占用着内存,这就是内存泄漏. 为什么会产生内存泄漏,内存泄漏会导致什么问题? 相比C++需要手动去 ...
记一次内存泄漏问题的排查经历
源宝导读:随着系统越来越庞大,越来越复杂,疑难杂症问题也越来越多.本文将介绍明源研发协同平台团队针对一次内存泄露问题的排查过程和技术探索. 一.背景内存泄漏,一个说大不大说下不小的瑕疵.作为开发者, ...
ReactiveCocoa中潜在的内存泄漏及解决方案
ReactiveCocoa是GitHub开源的一个函数响应式编程框架,目前在美团App中大量使用.用过它的人都知道很好用,也确实为我们的生活带来了很多便利,特别是跟MVVM模式结合使用,更是如鱼得水. ...
Android Native 内存泄漏系统化解决方案
导读:C++内存泄漏问题的分析.定位一直是Android平台上困扰开发人员的难题.因为地图渲染.导航等核心功能对性能要求很高,高德地图APP中存在大量的C++代码.解决这个问题对于产品质量尤为重要和关 ...
记一次连接泄漏GetConnectionTimeoutException: wait millis 60000, active 20, maxActive 20, creating 0
com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 60000, active 20, maxActive 20, cr ...

记一次jdbc泄漏与解决方案

记一次jdbc泄漏与解决方案相关推荐

最新文章

热门文章