基于pg_qualstats和hypopg的自动索引调优

pg-qualstats的安装和配置

1.安装pg-qualstats

sudo apt install postgresql-10-pg-qualstats

2.将pg_qualstats和pg_stat_statements添加到shared_preload_libraries，使得postgresql .conf文件中具有以下设置：

shared_preload_libraries = 'pg_stat_statements,pg_qualstats' # (change requires restart)

postgresql.conf文件在/etc/postgresql/10/main/目录下

执行grep 'shared_preload' postgresql.conf查看

3.重新启动PG。

service postgresql restart

4.进入PG

sudo su - postgrespsql

5.查询shared_preload_libraries

show shared_preload_libraries ;

Hypopg的安装和配置

1.安装 hypopg

apt install postgresql-server-dev-10 apt install postgresql-10-hypopg

自动索引调优

1.进入PG

sudo su - postgres

2.设置采样率 pg_qualstats .sample_rate1，保证调参涉及到所有的query

psql -d postgres -c "ALTER SYSTEM SET pg_qualstats.sample_rate TO 1"

验证

psql -c "select pg_reload_conf()"

3.进入PG

psql

4.加载extension

CREATE EXTENSION hypopg;CREATE EXTENSION pg_stat_statements ;CREATE EXTENSION pg_qualstats;

5.查看配置

\dxshow shared_preload_libraries ;

6.建立测试数据库

create database testdb owner postgres;

7.复现样例测试

建表

CREATE TABLE test (id int, dept int, id2 int, id3 int, id4 int, id5 int,id6 int,id7 int,details text, zipcode int);

插入数据

INSERT INTO test SELECT (random() * 1000000)::int, (random() * 1000000)::int, (random() * 1000000)::int,(random() * 1000000)::int,(random() * 1000000)::int,(random() * 1000000)::int,(random() * 1000000)::int,(random() * 1000000)::int,md5(g::text), floor(random()* (20000-9999 + 1) + 9999)FROM generate_series(1,1*1e6) g;

执行workload

select * from test where id2 = 1 and id4 = 3;select * from test where id3 = 3;select * from test where id3 = 3 and id4 = 2;select * from test where id4 = 2 and id2 = 3;

建立函数 find_usable_indexes

CREATE OR REPLACE FUNCTION find_usable_indexes()RETURNS VOID AS$$DECLAREl_queries     record;l_querytext     text;l_idx_def       text;l_bef_exp       text;l_after_exp     text;hypo_idx      record;l_attr        record;/* l_err       int; */BEGINCREATE TABLE IF NOT EXISTS public.idx_recommendations (queryid bigint,query text, current_plan jsonb, recmnded_index text, hypo_plan jsonb);FOR l_queries INSELECT t.relid, t.relname, t.queryid, t.attnames, t.attnums,pg_qualstats_example_query(t.queryid) as queryFROM(SELECT qs.relid::regclass AS relname, qs.relid AS relid, qs.queryid,string_agg(DISTINCT attnames.attnames,',') AS attnames, qs.attnumsFROM pg_qualstats_all qsJOIN pg_qualstats q ON q.queryid = qs.queryidJOIN pg_stat_statements ps ON q.queryid = ps.queryidJOIN pg_amop amop ON amop.amopopr = qs.opnoJOIN pg_am ON amop.amopmethod = pg_am.oid,LATERAL(SELECT pg_attribute.attname AS attnamesFROM pg_attributeJOIN unnest(qs.attnums) a(a) ON a.a = pg_attribute.attnumAND pg_attribute.attrelid = qs.relidORDER BY pg_attribute.attnum) attnames,     LATERAL unnest(qs.attnums) attnum(attnum)WHERE NOT(EXISTS(SELECT 1FROM pg_index iWHERE i.indrelid = qs.relid AND(arraycontains((i.indkey::integer[])[0:array_length(qs.attnums, 1) - 1],qs.attnums::integer[]) OR arraycontains(qs.attnums::integer[],(i.indkey::integer[])[0:array_length(i.indkey, 1) + 1]) AND i.indisunique)))GROUP BY qs.relid, qs.queryid, qs.qualnodeid, qs.attnums) tGROUP BY t.relid, t.relname, t.queryid, t.attnames, t.attnums                   LOOP/* RAISE NOTICE '% : is queryid',l_queries.queryid; */execute 'explain (FORMAT JSON) '||l_queries.query INTO l_bef_exp;execute 'select hypopg_reset()';execute 'SELECT indexrelid,indexname FROM hypopg_create_index(''CREATE INDEX on '||l_queries.relname||'('||l_queries.attnames||')'')' INTO hypo_idx;      execute 'explain (FORMAT JSON) '||l_queries.query INTO l_after_exp;execute 'select hypopg_get_indexdef('||hypo_idx.indexrelid||')' INTO l_idx_def;INSERT INTO public.idx_recommendations (queryid,query,current_plan,recmnded_index,hypo_plan)VALUES (l_queries.queryid,l_querytext,l_bef_exp::jsonb,l_idx_def,l_after_exp::jsonb);        END LOOP;    execute 'select hypopg_reset()';END;$$ LANGUAGE plpgsql;

执行函数find_usable_indexes

select find_usable_indexes();

查找索引

select queryid, current_plan->0->'Plan'->>'Total Cost' as "cost_without_index",hypo_plan->0->'Plan'->>'Total Cost' as "cost_with_index",round((((current_plan->0->'Plan'->>'Total Cost')::numeric-(hypo_plan->0->'Plan'->>'Total Cost')::numeric)*100/(current_plan->0->'Plan'->>'Total Cost')::numeric),2) as percent_improvdFROM idx_recommendations order by 4 desc;

select b.query, a.recmnded_index,round((((a.current_plan->0->'Plan'->>'Total Cost')::numeric-(hypo_plan->0->'Plan'->>'Total Cost')::numeric)*100/(a.current_plan->0->'Plan'->>'Total Cost')::numeric),2) as percent_improvd FROM idx_recommendations a JOIN pg_stat_statements b ON a.queryid = b.queryid WHERE round((((current_plan->0->'Plan'->>'Total Cost')::numeric-(hypo_plan->0->'Plan'->>'Total Cost')::numeric)*100/(current_plan->0->'Plan'->>'Total Cost')::numeric),2) > 0 order by 3 desc ;

基于pg_qualstats和hypopg的自动索引调优相关推荐

面向关系数据库的智能索引调优方法
面向关系数据库的智能索引调优方法人工智能技术与咨询来源:<软件学报> ,作者邱涛等摘要:数据库索引是关系数据库系统实现快速查询的有效方式之一.智能索引调优技术可以有效地对数据库实 ...
11g新特性-自动sql调优(Automatic SQL Tuning)
11g新特性-自动sql调优(Automatic SQL Tuning) 在Oracle 10g中,引进了自动sql调优特性.此外,ADDM也会监控捕获高负载的sql语句. 在Oracle 11g中, ...
sql索引调优_使用内置索引利用率指标SQL Server索引性能调优
sql索引调优描述 (Description) Indexing is key to efficient query execution. Knowing what indexes are unne ...
mysql数据库索引调优
一.mysql索引 1.磁盘文件结构 innodb引擎:frm格式文件存储表结构,ibd格式文件存储索引和数据. MyISAM引擎:frm格式文件存储表结构,MYI格式文件存储索引,MYD格式文件存储 ...
【华为云技术分享】[HDC.Cloud]基于鲲鹏平台的Ceph深度性能调优
随着IOT.大数据.移动互联等应用的暴涨,产生的数据也越来越多,整个存储市场总量也逐年增长,预计到2021年分布式存储会占到整个存储市场的50%,到2027年,分布式存储会占到整个市场的70%.Cep ...
专题-子项-1-mysqlf-Explain-执行计划-详解--索引--调优实践
目录使用EXPLAIN关键字可以模拟优化器执行SQL语句,从而知道MySQL是如何处理你的SQL语句的.分析你的查询语句或是结构的性能瓶颈 ------------1. id列---------- ...
艺赛旗（RPA）Mysql 数据查询慢 - 索引调优 Explain 用法
艺赛旗 RPA8.0全新首发免费下载点击下载 http://www.i-search.com.cn/index.html?from=line1 在日常工作中,我们会有时会开慢查询去记录一些执行时间比 ...
mysql double 索引_MySQL架构优化实战系列1：数据类型与索引调优全解析
一.数据类型优化数据类型整数数字类型:整数和实数 tinyint(8).smallint(16).mediuint(24).int(32).bigint(64) 数字表示对应最大存储位数,如 t ...
mysql 字符串索引优化_MySQL性能优化之索引调优实战
索引失效场景或使用注意事项 a.索引无法存储null值,所以建议都给默认值 b.如果条件中有or,即使使用了索引条件也不起作用,所以尽量少用or 如果想使用or,又让索引生效,只能将or的每个列上加上 ...
Mysql索引的原理、调优及其相关基础知识
索引的本质 MySQL官方对索引的定义为:索引(Index)是帮助MySQL高效获取数据的数据结构.提取句子主干,就可以得到索引的本质:索引是一种数据结构. 数据库查询是数据库的主要功能之一,最基本的 ...

基于pg_qualstats和hypopg的自动索引调优

基于pg_qualstats和hypopg的自动索引调优相关推荐

最新文章

热门文章