1.背景

通过公司监控工具监控，发现公司某个应用cpu利用率达到120%，也就是说这个应用自己单独占用一个cpu使用，为何占用这么高？让我们一起排查一下

2.开始排查

(1) 第一步获取cpu过高应用的进程信息 (top 命令监控)

(2)通过进程获取占用cpu过高的线程（Top -H -p <pid>）

(3)通过jstack命令导出线程的堆栈信息

"http-bio-7051-exec-28" #33168 daemon prio=5 os_prio=0 tid=0x00007fe343f77000 nid=0xaa1 runnable [0x00007fe329287000]java.lang.Thread.State: RUNNABLEat java.lang.StringBuilder.append(StringBuilder.java:214)at sun.util.calendar.CalendarUtils.sprintf0d(CalendarUtils.java:171)at java.util.Date.toString(Date.java:1040)at java.lang.String.valueOf(String.java:2994)at java.lang.StringBuilder.append(StringBuilder.java:131)at com.jiuyescm.oms.wm.odo.action.service.impl.CommonOmsOrderSendAction.buildAppointKey(CommonOmsOrderSendAction.java:72)at com.jiuyescm.oms.wm.odo.controller.OmsDo2BController.queryStorage(OmsDo2BController.java:1007)at sun.reflect.GeneratedMethodAccessor2011.invoke(Unknown Source)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at com.bstek.dorado.data.method.MethodAutoMatchingUtils.invokeMethod(MethodAutoMatchingUtils.java:602)at com.bstek.dorado.data.method.MethodAutoMatchingUtils.invokeMethod(MethodAutoMatchingUtils.java:695)at com.bstek.dorado.data.provider.manager.DataProviderInterceptorInvoker.invokeInterceptorByParamType(DataProviderInterceptorInvoker.java:542)at com.bstek.dorado.data.provider.manager.DataProviderInterceptorInvoker.invoke(DataProviderInterceptorInvoker.java:157)at com.bstek.dorado.util.proxy.ChainedMethodInvocation.proceed(ChainedMethodInvocation.java:73)at com.bstek.dorado.data.provider.DataProviderGetResultMethodInterceptor.invokeGetResult(DataProviderGetResultMethodInterceptor.java:29)at com.bstek.dorado.data.provider.AbstractDataProviderGetResultMethodInterceptor.invoke(AbstractDataProviderGetResultMethodInterceptor.java:55)at com.bstek.dorado.util.proxy.MethodInterceptorDispatcher.invoke(MethodInterceptorDispatcher.java:241)at com.bstek.dorado.data.provider.DirectDataProvider_$$_jvst349_17.getResult(DirectDataProvider_$$_jvst349_17.java)at com.bstek.dorado.view.service.LoadDataServiceProcessor.doExecute(LoadDataServiceProcessor.java:258)at com.bstek.dorado.view.service.DataServiceProcessorSupport.execute(DataServiceProcessorSupport.java:133)at com.bstek.dorado.view.resolver.ViewServiceInvoker.invoke(ViewServiceInvoker.java:32)at com.bstek.dorado.view.resolver.ViewServiceInvoker_$$_jvst349_16._d5invoke(ViewServiceInvoker_$$_jvst349_16.java)at sun.reflect.GeneratedMethodAccessor216.invoke(Unknown Source)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at com.bstek.dorado.util.proxy.MethodInterceptorDispatcher$FinalJavassistMethodInterceptor.invoke(MethodInterceptorDispatcher.java:96)at com.bstek.dorado.util.proxy.ChainedMethodInvocation.proceed(ChainedMethodInvocation.java:73)at com.bstek.dorado.util.proxy.MethodInterceptorDispatcher$FinalAopallianceMethodInterceptor.invoke(MethodInterceptorDispatcher.java:67)at com.bstek.dorado.util.proxy.ChainedMethodInvocation.proceed(ChainedMethodInvocation.java:73)at com.jiuyescm.cfm.dorado.interceptor.SessionValidationRemoteServiceMethodInterceptor.invoke(SessionValidationRemoteServiceMethodInterceptor.java:33)at com.bstek.dorado.view.service.AbstractRemoteServiceMethodInterceptor.invoke(AbstractRemoteServiceMethodInterceptor.java:30)at com.bstek.dorado.util.proxy.ChainedMethodInvocation.proceed(ChainedMethodInvocation.java:73)at com.bstek.bdf2.core.exception.interceptor.AjaxMethodInterceptor.invoke(AjaxMethodInterceptor.java:26)at com.bstek.dorado.util.proxy.MethodInterceptorDispatcher.invoke(MethodInterceptorDispatcher.java:170)at com.bstek.dorado.util.proxy.MethodInterceptorDispatcher.invoke(MethodInterceptorDispatcher.java:241)at com.bstek.dorado.view.resolver.ViewServiceInvoker_$$_jvst349_16.invoke(ViewServiceInvoker_$$_jvst349_16.java)at com.bstek.dorado.view.resolver.ViewServiceResolver.processTask(ViewServiceResolver.java:154)at com.bstek.dorado.view.resolver.ViewServiceResolver.execute(ViewServiceResolver.java:244)at com.bstek.dorado.web.resolver.AbstractTextualResolver.doHandleRequest(AbstractTextualResolver.java:128)at com.bstek.dorado.web.resolver.WebContextSupportedController.handleRequestInternal(WebContextSupportedController.java:70)at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:154)at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:945)at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:876)at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:931)at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:833)at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:807)at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:123)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:108)at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)at com.bstek.bdf2.core.security.filter.ControllerFilter.doFilter(ControllerFilter.java:64)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:183)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:183)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)at org.jasig.cas.client.session.SingleSignOutFilter.doFilter(SingleSignOutFilter.java:100)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)at com.bstek.bdf2.core.security.filter.PreAuthenticatedProcessingFilter.doFilter(PreAuthenticatedProcessingFilter.java:41)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)at com.bstek.bdf2.core.security.filter.ContextFilter.doFilter(ContextFilter.java:36)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)at com.r.tomcat.session.management.RequestSessionHandlerValve.invoke(RequestSessionHandlerValve.java:30)at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1115)at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)- locked <0x0000000724439528> (a org.apache.tomcat.util.net.SocketWrapper)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)at java.lang.Thread.run(Thread.java:748)

从以上日志中找出最魁祸首

at java.lang.StringBuilder.append(StringBuilder.java:214)at sun.util.calendar.CalendarUtils.sprintf0d(CalendarUtils.java:171)at java.util.Date.toString(Date.java:1040)at java.lang.String.valueOf(String.java:2994)at java.lang.StringBuilder.append(StringBuilder.java:131)at com.jiuyescm.oms.wm.odo.action.service.impl.CommonOmsOrderSendAction.buildAppointKey(CommonOmsOrderSendAction.java:72)at com.jiuyescm.oms.wm.odo.controller.OmsDo2BController.queryStorage(OmsDo2BController.java:1007)

(4)结合代码寻找真相

3.总结

跟我们的堆栈完全可以对应的上，性能耗费在循环调用字符串拼接这里。在这里要说一下字符串使用操作符"+"拼接的问题，如果你用”+”来连接固定长度的字符串，在性能上会有问题，但是如果你是在循环中来”+”多个串的话，性能将指数倍的下降。假设有一个字符串，我们将对这个字符串做大量循环拼接操作，使用”+”的话将得到最低的性能。网上有很多对比的文章随便一找一大把，如果不信自己可以写个demo调用1000次然后看自己cpu的情况。

记一次CPU飙升的问题排查相关推荐

cpu飙升死循环_记一次CPU飙升BUG
一．前言上线后,CPU飙升到100%,怎么办?马上重启?大错特错,马上重启只会让这个雷石沉大海,治标不不治本,等待你的下次的历史重演! 二．现象监控告警,某机器的CPU飙升到100% 三．分析第 ...
记一次CPU飙升的问题分析解决思路(转)
一．前言上线后,CPU飙升到100%,怎么办?马上重启?大错特错,马上重启只会让这个雷石沉大海,治标不不治本,等待你的下次的历史重演! 二．现象监控告警,某机器的CPU飙升到100% 三．分析第 ...
记一次线上cpu飙升100%的排查过程
大家好,我是烤鸭: 最近没怎么写技术文章,还是得回归下初心,正好前几天出现个线上问题,记录下排查过程. 问题描述某个时间点,接收到接口响应慢报警. 过一会收到服务器cpu可用率低(<10%)报 ...
cpu飙升死循环_java排查一个线上死循环cpu暴涨的过程分析
问题,打一个页面cpu暴涨,打开一次就涨100%,一会系统就卡的不行了. 排查方法,因为是线上的linux,没有用jvm监控工具rim链接上去. 只好用命令排查: top cpu排序,一个java进程 ...
Java线上CPU内存冲高问题排查步骤
1 引言作为一名从事Java开发快一年的程序员,在线上经常碰到某个模块的Pod发出CPU与内存告警的问题,而这些问题会导致系统响应缓慢甚至是服务不可用.一般情况下可以通过重启或者调高Pod的资源量或 ...
记一次CPU突然飙升到 100% 问题排查
线上 CPU 飚高问题该如何定位问题呢? 是因为线程太多,导致上下文切换? 还是因为应用代码中出现了死循环? 还是GC频繁导致 CPU 突然飙升? 该如何入手呢? 首先要知道那些情况会导致 CPU 的 ...
记一次cpu指标异常的跟踪排查
记一次cpu指标异常的跟踪排查参考文章: (1)记一次cpu指标异常的跟踪排查 (2)https://www.cnblogs.com/Zfc-Cjk/p/10973932.html (3)https ...
线上服务导致cpu飙升问题排查
一.故障说明昨晚突然收到线上服务器cpu报警,登录监控平台看了下发现cpu瞬间飙升到60%.第一反应就是使用top命令去查看,发现是一个java进程.于是立刻使用jps -l命令定位到该java进程 ...
Java程序CPU飙升排查，找出死循环代码
windows环境下cpu飙升问题线上某台runtime机器(windows Server)cpu报警,这种情况初步就是代码里面死循环了,先把机器下线了保证不再有新的任务分配进来,然而cpu使用依然 ...

记一次CPU飙升的问题排查

1.背景

2.开始排查

3.总结

记一次CPU飙升的问题排查相关推荐

最新文章

热门文章