一款分析linux上应用程序分析的工具----0x.tools
2024-05-15 23:44:17
0x.tools是一个用于分析Linux上应用程序性能的开源实用程序。部署简单且依赖性小。无需升级操作系统、安装内核模块、繁重的监控框架、Java代理或数据库。这款工具还适用于十多年前的Linux内核，如14年前的2.6.18版本。
包含的工具：
安装和使用：
$ git clone https://github.com/tanelpoder/0xtools
$ make
$ make install
具体使用示例请参考：
https://0x.tools/#installation–usage
$ xcapture0xTools xcapture v1.0 by Tanel Poder [https://0x.tools]Sampling /proc...DATE TIME PID TID USERNAME ST COMMAND SYSCALL WCHAN
2020-10-17 12:01:50.583 6404 7524 mysql R (mysqld) fsync wait_on_page_bit
2020-10-17 12:01:50.583 6404 8944 mysql D (mysqld) fsync wait_on_page_bit
2020-10-17 12:01:50.583 6404 8946 mysql D (mysqld) fsync wait_on_page_bit
2020-10-17 12:01:50.583 6404 76046 mysql D (mysqld) fsync wait_on_page_bit
2020-10-17 12:01:50.583 6404 76811 mysql D (mysqld) fdatasync xfs_log_force_lsn
2020-10-17 12:01:50.583 6404 76815 mysql D (mysqld) fsync blkdev_issue_flush
2020-10-17 12:01:50.583 8803 8803 root R (md10_resync) [running] 0
采样所有线程状态并写到csv文件
$ xcapture -a -o /data/xcap &$ head 2020-10-16.21.csv
TS,PID,TID,USERNAME,ST,COMMAND,SYSCALL,WCHAN,EXE,CMDLINE,KSTACK
2020-10-16 21:00:00.001,5335,5335,root,R,(collectl),[running],0,perl,/usr/bin/perl,
2020-10-16 21:00:00.001,8803,8803,root,D,(md10_resync),[no_syscall],msleep,-,-,->ret_from_fork_nospec_begin()->kthread()->md_thread()->md_do_sync()->msleep()
2020-10-16 21:00:01.038,8803,8803,root,R,(md10_resync),[no_syscall],md_do_sync,-,-,->ret_from_fork_nospec_begin()->kthread()->md_thread()->md_do_sync()
2020-10-16 21:00:02.075,8803,8803,root,D,(md10_resync),[no_syscall],md_do_sync,-,-,->ret_from_fork_nospec_begin()->kthread()->md_thread()->md_do_sync()
2020-10-16 21:00:02.075,16762,16762,oracle,R,(ora_m000_lin19c),[running],0,oracle,ora_m000_LIN19C,->do_blockdev_direct_IO()->dio_complete()
2020-10-16 21:00:03.112,8803,8803,root,R,(md10_resync),[no_syscall],md_do_sync,-,-,->ret_from_fork_nospec_begin()->kthread()->md_thread()->md_do_sync()
2020-10-16 21:00:04.149,8803,8803,root,D,(md10_resync),[no_syscall],msleep,-,-,->ret_from_fork_nospec_begin()->kthread()->md_thread()->md_do_sync()->msleep()
2020-10-16 21:00:05.186,8803,8803,root,D,(md10_resync),[no_syscall],md_do_sync,-,-,->ret_from_fork_nospec_begin()->kthread()->md_thread()->md_do_sync()
2020-10-16 21:00:05.186,65913,65913,oracle,D,(ora_ckpt_lin122),pwrite64,blkdev_issue_flush,oracle,ora_ckpt_LIN122,->system_call_fastpath()->SyS_pwrite64()->vfs_write()->do_sync_write()->xfs_file_aio_write()->generic_write_sync()->xfs_file_fsync()->xfs_blkdev_issue_flush()->blkdev_issue_flush()
使用linux文本处理工具过滤查看
$ cat 2020-10-13.01.csv | awk -F, '{ printf("%2s %-20s %-20s %s\n",$5,$4,$7,$10) }' | sort | uniq -c | sort -nbr | head -202303 D root read -1761 R tanel [running] stress1384 D postgres pread64 postgres: tanel pgbench [local] UPDATE894 R root [running] -229 R root read -229 D mysql fsync /usr/sbin/mysqld144 R tanel [running] -115 - - - -110 D oracle io_submit ora_ckpt_LINPRD101 D root [running] -73 D root read dd58 R root [running] /opt/oracle.ahf/jre/bin/java55 R mysql [running] /usr/sbin/mysqld52 D tanel [no_syscall] stress51 R oracle [running] oracleLIN19C50 R root [running] dd35 R oracle [running] xe_mz01_XE32 R tanel [running] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre/bin/java29 R oracle [running] pidstat27 D oracle pread64 oracleLIN19C
xcapture使用：
$ xcapture0x.Tools xcapture v1.0 by Tanel Poder [https://0x.tools]Usage:xcapture [options]By default, sample all /proc tasks in states R, D every second and print to stdoutOptions:-a capture tasks in additional states, even the ones Sleeping (S)-A capture tasks in All states, including Zombie (Z), Exiting (X), Idle (I)-c <c1,c2> print additional columns (for example: -c exe,cmdline,kstack)-d <N> seconds to sleep between samples (default: 1)-E <string> custom task state Exclusion filter (default: XZIS)-h display this help message-o <dirname> write wide output into hourly CSV files in this directory instead of stdout$ xcapture -c exe,kstack
$ xcapture -o .
$ xcapture -o /data/perf_archive/xcappsn是使用python编写的脚本，可以查看当前相关状态
$ psn -p 18286 -G syscall,filenameLinux Process Snapper v0.14 by Tanel Poder [https://0x.tools]
Sampling /proc/stat, syscall for 5 seconds... finished.=== Active Threads ==================================================================================samples | avg_threads | comm | state | syscall | filename
-----------------------------------------------------------------------------------------------------79 | 0.79 | (dd) | Disk (Uninterruptible) | write | /backup/tanel/test (stdout)7 | 0.07 | (dd) | Disk (Uninterruptible) | [running] | 5 | 0.05 | (dd) | Running (ON CPU) | write | /backup/tanel/test (stdout) 4 | 0.04 | (dd) | Disk (Uninterruptible) | read | /reco/fio/mmapfile.0.0 (stdin) 3 | 0.03 | (dd) | Running (ON CPU) | [running] | 2 | 0.02 | (dd) | Running (ON CPU) | read | /reco/fio/mmapfile.0.0 (stdin) ```
$ sudo psn -p "mysqld|kwork" -G syscall,wchanLinux Process Snapper v0.14 by Tanel Poder [https://0x.tools]
Sampling /proc/syscall, stat, wchan for 5 seconds... finished.=== Active Threads ========================================================================================samples | avg_threads | comm | state | syscall | wchan
-----------------------------------------------------------------------------------------------------------25 | 3.12 | (mysqld) | Disk (Uninterruptible) | fsync | _xfs_log_force_lsn16 | 2.00 | (mysqld) | Running (ON CPU) | [running] | 0 14 | 1.75 | (mysqld) | Disk (Uninterruptible) | pwrite64 | call_rwsem_down_write_failed8 | 1.00 | (mysqld) | Disk (Uninterruptible) | fsync | submit_bio_wait 4 | 0.50 | (mysqld) | Disk (Uninterruptible) | pread64 | io_schedule 4 | 0.50 | (mysqld) | Disk (Uninterruptible) | pwrite64 | io_schedule 3 | 0.38 | (mysqld) | Disk (Uninterruptible) | pread64 | 0 3 | 0.38 | (mysqld) | Running (ON CPU) | [running] | io_schedule 3 | 0.38 | (mysqld) | Running (ON CPU) | pread64 | 0 2 | 0.25 | (mysqld) | Disk (Uninterruptible) | [running] | 0 1 | 0.12 | (kworker/*:*) | Running (ON CPU) | read | worker_thread 1 | 0.12 | (mysqld) | Disk (Uninterruptible) | fsync | io_schedule 1 | 0.12 | (mysqld) | Disk (Uninterruptible) | futex | call_rwsem_down_write_failed 1 | 0.12 | (mysqld) | Disk (Uninterruptible) | poll | 0 1 | 0.12 | (mysqld) | Disk (Uninterruptible) | pwrite64 | _xfs_log_force_lsn 1 | 0.12 | (mysqld) | Running (ON CPU) | fsync | submit_bio_wait 1 | 0.12 | (mysqld) | Running (ON CPU) | futex | futex_wait_queue_me
更多相关示例可以参考Linux性能诊断
SchedLat
$ ./schedlat.py 29801
SchedLat by Tanel Poder [https://0x.tools]PID=29801 COMM=oracle_29801_liTIMESTAMP %CPU %LAT %SLP
2020-02-26 23:17:35 100.0 0.0 0.0 <<-- no CPU shortage, process 100% on CPU
2020-02-26 23:17:36 100.0 0.0 0.0
2020-02-26 23:17:37 100.0 0.0 0.0
2020-02-26 23:17:38 100.0 0.0 0.0 <<-- %SLP = 100-(%CPU+%LAT), when Linux reports slightly
2020-02-26 23:17:39 98.0 0.0 2.0 more than "100%" of CPU+LAT, then the derived
2020-02-26 23:17:40 0.0 0.0 100.0 "remaining time" SLP% may show a negative value
2020-02-26 23:17:41 0.0 0.0 100.0
2020-02-26 23:17:42 0.0 0.0 100.0 <<-- no CPU shortage, process sleeping
2020-02-26 23:17:43 0.4 0.0 99.6
2020-02-26 23:17:44 33.5 0.2 66.3 <<-- no CPU shortage, process doing synchronous I/Os
2020-02-26 23:17:45 55.5 0.2 44.2 in a loop (thus taken off CPU frequently by scheduler)
2020-02-26 23:17:46 53.9 0.2 45.9
2020-02-26 23:17:47 54.5 0.2 45.3
2020-02-26 23:17:48 59.1 0.2 40.7
2020-02-26 23:17:49 4.4 0.0 95.6
2020-02-26 23:17:50 58.5 0.1 41.4
2020-02-26 23:17:51 95.7 0.0 4.3
2020-02-26 23:17:52 0.3 0.0 99.7
2020-02-26 23:17:53 0.1 0.0 99.9
2020-02-26 23:17:54 0.1 0.0 99.9
2020-02-26 23:17:55 0.3 1.1 98.6
2020-02-26 23:17:56 0.1 6.0 93.9
2020-02-26 23:17:57 0.1 15.0 84.9
2020-02-26 23:17:58 0.1 13.8 86.1
2020-02-26 23:17:59 9.6 61.4 29.0 <<-- CPU shortage + process doing synchronous I/Os in a loop
2020-02-26 23:18:00 14.6 83.9 1.5 <<-- and spending more time in CPU runqueue after every I/O
2020-02-26 23:18:01 31.4 59.7 8.9
2020-02-26 23:18:02 13.0 13.9 73.1
2020-02-26 23:18:03 0.3 5.3 94.4
更多示例请参考度量cpu调度延迟
CPU profiling
实际使用perf record命令生成相关数据
#cat run_xcpu.shFREQUENCY=1 # 1 Hz sampling
SUDO=sudo # change to empty string if running without sudo
NICE=-5
SLEEP=60
PERF=/usr/bin/perfif [ $# -ne 1 ]; thenecho "Usage: $0 output_dir"exit 1
filogger "$0 Starting up outdir=$1 nice=$NICE"while true ; do$SUDO nice -n $NICE $PERF record -g -F $FREQUENCY -a \--switch-output=1m \--timestamp-filename \--timestamp \-o $1/xcpu# we only get here should perf be terminated, try to restartlogger "$0 terminated with $?, attempting to restart in $SLEEP seconds"sleep $SLEEP
done
使用以上命令进行cpu栈跟踪，然后通过以下命令查看
$ perf report -s sym,dso -i xcpu.2020101619323791
一款分析linux上应用程序分析的工具----0x.tools相关推荐
- linux桌面壁纸目录,酷炫！推荐一款适用于Linux的壁纸程序
原标题:酷炫!推荐一款适用于Linux的壁纸程序 Wonderwall是一个用于Unity和GNOME桌面环境的酷炫壁纸程序.其简单的用户界面允许用户在他的Ubuntu桌面上轻松浏览,下载和安装壁纸. ...
- linux上开发应用程序_如何在Linux上安装软件应用程序
linux上开发应用程序 如何在Linux上安装应用程序? 与许多操作系统一样,该问题不仅有一个答案. 应用程序可以来自许多来源-几乎无法计数-每个开发团队都可以以自己认为最佳的方式交付软件. 知道如 ...
- 你要了解的11款面向Linux系统的一流备份实用工具
个人计算机或者服务器上的备份对于防止数据丢失很重要.系统管理员每天需要处理大量的企业级数据,甚至个人计算机上的数据,所以要深入了解不同的备份工具. 经常备份计算机上的数据是个好的做法,它可以手动完成, ...
- ssh开启图形界面_分享|3 个 Linux 上的 SSH 图形界面工具
了解一下这三个用于 Linux 上的 SSH 图形界面工具. 在你担任 Linux 管理员的职业生涯中,你会使用 Secure Shell(SSH)远程连接到 Linux 服务器或桌面.可能你曾经在某 ...
- 介绍linux上两种rootkits检测工具: Rootkit Hunter和Chkrootkit
原贴:http://blog.csdn.net/linkboy2004/archive/2007/03/22/1537890.aspx 介绍linux上两种rootkits检测工具: Rootkit ...
- linux命令行恢复回收站,Trash-Cli：Linux 上的命令行回收站工具
原标题:Trash-Cli:Linux 上的命令行回收站工具 编译自: https://www.2daygeek.com/trash-cli-command-line-trashcan-linux-s ...
- linux操作系统源码分析,linux操作系统源代码详细分析.doc
linux操作系统源代码详细分析.doc LINUX操作系统源代码详细分析内容简介LINUX拥有现代操作系统所有的功能,如真正的抢先式多任务处理.支持多用户,内存保护,虚拟内存,支持SMP.UP,符合 ...
- c语言报告程序分析报告,2012C语言程序分析报告.doc
2012C语言程序分析报告 C语言程序设计专周 专 周 报 告 班级:10611 学号:20 姓名: 设计时间:2011-5-30至2011-6-3 一.设计题目:职工工资管理小软件 二.实习目的 1 ...
- 基因表达分析（上）- 差异表达分析
基因表达 什么是基因表达,如下是来自于维基百科的解释: Gene expression is the process by which information from a gene is used ...
最新文章
- SAP RETAIL 补货类型RF之初探 II
- 查询选修c语言课程的学生学号和姓名,数据库实验(学生信息表)
- 实时流处理框架Storm、Spark Streaming、Samza、Flink，孰优孰劣？！
- Spring Security系列教程解决Spring Security环境中的跨域问题
- java 四种内存_不可访问内存 Java四种引用包括强引用，软引用，弱引用，虚引用...
- P3577-[POI2014]TUR-Tourism【状压dp】
- eclipse占用内存过大_Java性能调优学习(三)-jmap+mat分析内存溢出问题实战
- 家用计算机注意哪些参数,电脑小白买内存条要注意哪些？主要看哪些参数？这些知识要掌握...
- java应用挂死故障排查
- SpringBoot（四）：mybatis之通用mapper、分页插件PageHelper
- poj 1656 Counting Black
- c语言经典100例对考研有用吗,考研数学经验，没用算这学长输
- Calendar计算两个日期相差几个月
- 学习笔记之Python的六种内置对象
- BZOJ2794 [Poi2012]Cloakroom [离线][DP]
- Vue教程(四十七)Promise基本介绍与使用
- Social Recommendation with Strong and Weak Ties 学习笔记
- 了解资本与公司年报、财报
- Android 10 手机端关闭蓝牙再打开，蓝牙不会自动回连设备蓝牙
- 2021-09-03 【N02】小程序开发学习笔记