注意：因为测试使用的python notebook + pandas, 所有 %使用的是%%
在SQL的末尾，可以增加返回数据的格式

FORMAT CSVWithNames
FORMAT TabSeparatedWithNamesAndTypes

1. 执行SQL查询

1.1. 查看正在执行的查询语句

--查询
SELECT query_id, user, address, elapsed, query
FROM system.processes
ORDER BY query_id ASC--杀死执行慢的SQL
KILL QUERY WHERE query_id='query_id';

1.2. 查看正在执行的更新语句

--查询
SELECT database,table,mutation_id,command,create_time,parts_to_do_names,parts_to_do,latest_fail_reason
FROM system.mutations
where is_done<>1--杀死执行慢的SQL
KILL MUTATION WHERE mutation_id = 'mutation_id';

1.3. 查询今天top 10 最慢的SQL

SELECT user,
formatDateTime(query_start_time, '%%Y%%m%%d %%T') AS start_time,
query_duration_ms / 1000 AS query_duration_s,
formatReadableSize(memory_usage ) AS memory_usage,
result_rows ,
formatReadableSize(result_bytes) AS result_bytes,
read_rows ,
formatReadableSize(read_bytes) AS read_bytes,
written_rows ,
formatReadableSize(written_bytes) AS written_bytes,
query
FROM system.query_log WHERE type = 2 and query_start_time>=today()
ORDER BY query_duration_s DESC LIMIT 10--直接查询
select type,concat(substr(query,1,100),'...') as query,read_rows,query_duration_ms,memory_usage,read_bytes,written_bytes from system.query_log limit 10--统计执行频繁的SQL
select concat(substr(query,1,100),'...') as sql,count(*) as total from system.query_log
where event_time>'2021-12-01 00:00:00' and event_time<'2021-12-02 00:00:00' and is_initial_query=1 and lower(query) like '%%select%%'
group by sql order by total desc

如何开启查询日志：
query_log记录了所有clickhouse服务中所有已经执行的查询记录

     <!--全局定义--><!-- Query log. Used only for queries with setting log_queries = 1. --><query_log><database>system</database><table>query_log</table><partition_by>toYYYYMM(event_date)</partition_by><!-- Interval of flushing data. --><flush_interval_milliseconds>7500</flush_interval_milliseconds></query_log><!--如果只需要为某些用户单独开启query_log,在users.xml的profile中配置--><log_queries>1</log_queries>

1.4. 基于remote统计集群慢SQL

remote(‘addresses_expr’, db, table[, ‘user’[, ‘password’]]) 允许访问远程服务器而不创建分布式表。

按节点按天统计慢SQL数
查询耗时大于指定时间的慢SQL
按节点统计正在运行的SQL数
查询正在运行的SQL

select * from
(
select 'node1' as server,address,toStartOfDay(event_time) as event_day,count(1) as selectCount from remote('x.x.x.x1','system','query_log') where event_time>'2021-12-01 10:00:00' and event_time<'2021-12-01 11:00:00' and query_duration_ms>10000 and is_initial_query=1 and lower(query) like '%%select%%'  group by address,toStartOfDay(event_time)
union all select 'node2' as server,address,toStartOfDay(event_time) as event_day,count(1) as selectCount from remote('x.x.x.x2','system','query_log') where event_time>'2021-12-01 10:00:00' and event_time<'2021-12-01 11:00:00' and  query_duration_ms>10000 and is_initial_query=1 and lower(query) like '%%select%%'  group by address,toStartOfDay(event_time)
union all select 'node3' as server,address,toStartOfDay(event_time) as event_day,count(1) as selectCount from remote('x.x.x.x3','system','query_log') where event_time>'2021-12-01 10:00:00' and event_time<'2021-12-01 11:00:00' and query_duration_ms>10000 and is_initial_query=1 and lower(query) like '%%select%%'  group by address,toStartOfDay(event_time)
) t order by server,event_dayselect 'node1' as server,address,event_time,type,query_duration_ms,query from remote('x.x.x.x1','system','query_log','default') where event_time>'2021-12-01 10:00:00' and event_time<'2021-12-01 11:00:00' and query_duration_ms>10000 and is_initial_query=1
union all select 'yjdsj03' as server,address,event_time,type,query_duration_ms,query from remote('x.x.x.x2','system','query_log','default') where event_time>'2021-12-01 10:00:00' and event_time<'2021-12-01 11:00:00' and  query_duration_ms>10000 and is_initial_query=1
union all select 'yjdsj04' as server,address,event_time,type,query_duration_ms,query from remote('x.x.x.x3','system','query_log','default') where event_time>'2021-12-01 10:00:00' and event_time<'2021-12-01 11:00:00' and query_duration_ms>10000 and is_initial_query=1 select * from
(select 'node1' as server,count(1) as cc from remote('x.x.x.x1','system','processes')
union all select 'node2' as server,count(1) as cc from remote('x.x.x.x2','system','processes')
union all select 'node3' as server,count(1) as cc from remote('x.x.x.x3','system','processes')
) t order by server;select * from
(select 'node1' as server,query_id, user, address, elapsed, query from remote('x.x.x.x1','system','processes')
union all select 'node2' as server,query_id, user, address, elapsed, query from remote('x.x.x.x2','system','processes')
union all select 'node3' as server,query_id, user, address, elapsed, query from remote('x.x.x.x3','system','processes')
) t order by server;

2. 查看表信息

2.1. 查看表占用空间大小

SELECT table,partition,formatReadableSize(sum(data_compressed_bytes)) AS compressed_size ,
formatReadableSize(sum(data_uncompressed_bytes)) AS uncompressed_bytes
FROM system.parts
WHERE active AND (table LIKE 'vehicle_warning_%%')
GROUP BY table,partition
order by partition desc

2.2. 查看列占用空间大小

select column as colName,any(type) as colType,
sum(column_data_compressed_bytes) compressed_size ,
sum(column_data_uncompressed_bytes) uncompressed_bytes,
sum(rows) as rowNum
from system.parts_columns
where active AND table like 'vehicle_warning_LOCAL'
GROUP BY column
ORDER BY uncompressed_bytes desc ;

2.3. 表的分区信息统计

统计指定时间范围内，表的分区、分区文件数及占用空间大小

SELECT partition, count() AS number_of_parts, formatReadableSize(sum(bytes)) AS sum_size
FROM system.parts
WHERE active AND (table = 'vehicle_warning_new_LOCAL') and partition between '2021-11-01 00:00:00' and '2021-11-30 00:00:00'
GROUP BY partition
ORDER BY partition ASC

2.4. 查看表的副本情况

SELECT database, table, is_leader, total_replicas, active_replicas, zookeeper_exception
is_session_expired,future_parts, parts_to_check,queue_size,inserts_in_queue,log_max_index,log_pointer
FROM system.replicas where table = 'vehicle_warning_LOCAL'--查看异常的副本,各个预警的变量可以根据自身情况调整。
SELECT database, table, is_leader, total_replicas, active_replicas, zookeeper_exception
FROM system.replicas
WHERE is_readonly OR is_session_expired OR future_parts > 20 OR parts_to_check > 10 OR queue_size > 20
OR inserts_in_queue > 10 OR log_max_index - log_pointer > 10 OR total_replicas < 2 OR active_replicas < total_replicas

3. 其它

3.1. 查看总连接数

SELECT * FROM system.metrics WHERE metric LIKE '%%Connection';

3.2. 查看磁盘空间

SELECT name,path,formatReadableSize(free_space) AS free_space,
formatReadableSize(total_space) AS total_space, type
FROM system.disks

3.3. 查看集群信息

select cluster,shard_num,shard_weight,replica_num,host_name,host_address,port,user,errors_count,estimated_recovery_time  from system.clusters where replica_num=1

3.5. 查看正在处理MergeTree族表的合并和分区变化

查看目前正在处理MergeTree族表的合并和分区变化的信息

select database ,table,elapsed ,progress,num_parts ,result_part_name ,is_mutation ,total_size_bytes_compressed ,rows_read ,rows_written
from system.merges

附录

clickhouse系统表

系统表介绍详见官网

使用的函数

import time
def yieldDF(df):for index, row in df.iteritems():for i in range(len(row)):yield row[i],indexdef printDF(df,pos=slice(1,2)):for row in df.itertuples():print(row[pos])
def executeSQL(sql):start =time.process_time()df = pd.read_sql(sql,con=engine)end = time.process_time()print('Running time: %s Seconds'%(end-start))return dfsql='''select cluster,shard_num,shard_weight,replica_num,host_name,host_address,port,user,errors_count,estimated_recovery_time
from system.clusters where replica_num=1
'''
executeSQL(sql)for d,i in yieldDF(executeSQL(sql).head(1)):print(i,'==>',d)printDF(executeSQL(sql),slice(1,3))

clickhouse MPPDB数据库运维实用SQL总结相关推荐

美团数据库运维自动化系统构建之路
本文整理自美团点评技术沙龙第10期:数据库技术架构与实践. 美团点评技术沙龙由美团点评技术团队主办,每月一期.每期沙龙邀请美团点评及其它互联网公司的技术专家分享来自一线的实践经验,覆盖各主要技术领域. ...
【clickhouse】ClickHouse之DBA运维宝典
1.概述转载:ClickHouse之DBA运维宝典这里仅仅是积累知识.建议大家去看原来的. 最近有位网友与我聊天,他是一名 DBA,问我在 ClickHouse 中有没有一些能够 "安家 ...
SpringBoot运维实用篇
SpringBoot2零基础到项目实战-基础篇 SpringBoot运维实用篇从此刻开始,咱们就要进入到实用篇的学习了.实用篇是在基础篇的根基之上,补全SpringBoot的知识图谱.比如在基础篇中 ...
解密京东618大促数据库运维的攻守之道
来自:DBAplus社群本文根据高新刚老师在[2019 DAMS中国数据智能管理峰会]现场演讲内容整理而成. 讲师介绍高新刚,京东数科数据库团队负责人,负责京东数科数据库平台的管理维护工作,带领团 ...
oracle 数据库运维技术,Oracle数据库智能运维标准化工艺研究-orastar-DIY数据库运维工具-第1期导论篇...
1.编制目的为了提升运维质量,增强数据库运维标准化.规范化,保障信息系统的安全.稳定.高效运行,助力实现智能运维,star同学在此抛砖引玉,特编制该文档. 2.维度定义根据运维经验,现将数据库日常 ...
微博热点事件背后数据库运维的“功守道”
作者 | 张冬洪责编 | 仲培艺 [导语] 微博拥有超过3.76亿月活用户,是当前社会热点事件传播的主要平台.而热点事件往往具有不可预测性和突发性,较短时间内可能带来流量的翻倍增长,甚至更大.如何快 ...
2020，分布式架构会给传统数据库运维带来哪些变化？
摘要:分布式架构可能是近几年最火的话题.从集中式.SOA到分布式架构,本文回顾了这些年金融行业经历的架构演变:结合当下一些较典型的分布式数据库的实现原理,分析了分布式数据库的三个发展阶段.分布式数据库 ...
顺丰android架构师,顺丰数据库运维架构.pdf
GOPS 全球运维大会 2018 2018.4.13-4.14 中国·广东·深圳·南山区圣淘沙大酒店 (翡翠店 ) G O P S 全球运维大会 2 0 1 8 · 深圳站负重前行- ...
与“十“俱进阿里数据库运维10年演进之路
与"十"俱进阿里数据库运维10年演进之路原文:与"十"俱进阿里数据库运维10年演进之路阿里巴巴集团拥有超大的数据库实例规模,在快速发展的过程中我们在运维 ...

clickhouse MPPDB数据库运维实用SQL总结

1. 执行SQL查询

1.1. 查看正在执行的查询语句

1.2. 查看正在执行的更新语句

1.3. 查询今天top 10 最慢的SQL

1.4. 基于remote统计集群慢SQL

2. 查看表信息

2.1. 查看表占用空间大小

2.2. 查看列占用空间大小

2.3. 表的分区信息统计

2.4. 查看表的副本情况

3. 其它

3.1. 查看总连接数

3.2. 查看磁盘空间

3.3. 查看集群信息

3.5. 查看正在处理MergeTree族表的合并和分区变化

附录

clickhouse系统表

使用的函数

clickhouse MPPDB数据库运维实用SQL总结相关推荐

最新文章

热门文章

clickhouse MPPDB数据库 运维实用SQL总结

1. 执行SQL查询

1.1. 查看正在执行的查询语句

1.2. 查看正在执行的更新语句

1.3. 查询今天top 10 最慢的SQL

1.4. 基于remote统计集群慢SQL

2. 查看表信息

2.1. 查看表占用空间大小

2.2. 查看列占用空间大小

2.3. 表的分区信息统计

2.4. 查看表的副本情况

3. 其它

3.1. 查看总连接数

3.2. 查看磁盘空间

3.3. 查看集群信息

3.5. 查看正在处理MergeTree族表的合并和分区变化

附录

clickhouse系统表

使用的函数

clickhouse MPPDB数据库 运维实用SQL总结相关推荐

最新文章

热门文章

clickhouse MPPDB数据库运维实用SQL总结

clickhouse MPPDB数据库运维实用SQL总结相关推荐