Pipeline aggregations管道聚合-Sibling-1

Pipeline aggregations管道聚合

管道聚合工作于其他聚合产生的输出结果而不是文档集，用于向输出树添加信息
按管道聚集运算来源分类，管道聚集可以分为基于父聚集结果和基于兄弟聚集结果两类。
前者使用父聚集的结果并将运算结果添加到父聚集结果中，后者则使用兄弟聚集的结果并且结果会展示在自己的聚集结果中。

(1)基于父聚集

基于父聚集的管道聚集包括moving_avg、moving_fn、bucket_script、bucket_selector、bucket_sort、derivative、cumulative_sum、serial_diff八种。

(2)基于兄弟聚集

基于兄弟聚集的管道聚集包括avg_bucket、max_bucket、min_bucket、sum_bucket、stats_bucket、extended_ stats_ bucket、percentiles_bucket七种。如果将它们名称中的bucket去除，它们就与之前介绍的部分指标聚集同名了。事实上，它们不仅在名称上接近，而且在功能上也类似，只是聚集运算的范围由整个文档变成了另一个聚集结果。

buckets_path参数

buckets_path参数用于指定访问其他桶中指标值的路径
buckets_ path参数的值由三部分组成，即聚集名称、指标名称和分隔符。聚集名称与聚集名称之间的分隔符是“>”，而聚集名称与指标名称之间的分隔符使用“.”。
buckets_path 语法

buckets_path	Syntax 语法
AGG_SEPARATOR	`>`
METRIC_SEPARATOR	`.`
AGG_NAME
METRIC
MULTIBUCKET_KEY	`[<KEY_NAME>]`
PATH	<AGG_NAME><MULTIBUCKET_KEY>? (<AGG_SEPARATOR>, <AGG_NAME> )* ( <METRIC_SEPARATOR>, )

1.Average bucket aggregation 平均桶聚合 ====================================================

同级管道聚合
它计算同级聚合中指定度量的平均值。同级聚合必须是多桶聚合，针对的是度量聚合(metric Aggregation)。
参数
1.buckets_path 指定聚合的名称，支持多级嵌套聚合。
2.gap_policy 当管道聚合遇到不存在的值，有点类似于term等聚合的(missing)时所采取的策略，可选择值为：skip、insert_zeros。
– skip：此选项将丢失的数据视为bucket不存在。它将跳过桶并使用下一个可用值继续计算。
– insert_zeros：默认使用0代替。
3.format 用于格式化聚合桶的输出(key)。
Response body 反应小组
value
value_as_string

计算每月平均销售额:

第一级聚合：sales_per_month，（按月）直方图聚合。
第二级聚合：sales，在按月聚合的基础上，对每个月的文档求sum。
第三级聚合：avg_monthly_sales，对上面的聚合求平均值
“buckets_path”: “sales_per_month>sales”
avg_bucket聚合是想要得到sales_per_month日期直方图聚合中sales聚合值的平均值。
1.对sales_per_month分组聚合的sales指标作为pipeline aggregations的输入源。
2.将"sales"中的sum值作为sales_per_month桶聚合的输入
3.将指向 sales指标值，该值包含在 “sales_per_month” 存储桶聚合中。


POST /sales/_search?size=0
{"size": 0,"aggs": {"sales_per_month": {"date_histogram": {"field": "date","calendar_interval": "month","format": "yyyy-MM"},"aggs": {"sales": {"sum": {"field": "price"}}}},"avg_monthly_sales": {"avg_bucket": {"buckets_path": "sales_per_month>sales","gap_policy": "skip","format": "#,##0.00;(#,##0.00)"}}}
}

响应结果：

aggregations由sales_per_month + avg_monthly_sales组成
1.sales_per_month部分：
按月份聚合bucket共3个。
key_as_string 月份
doc_count 当月的doc数量
sales.value 当月price的sum值
2.avg_monthly_sales部分：
value：三个月平均值

{"took" : 1,"timed_out" : false,"_shards" : {"total" : 1,"successful" : 1,"skipped" : 0,"failed" : 0},"hits" : {"total" : {"value" : 16,"relation" : "eq"},"max_score" : null,"hits" : [ ]},"aggregations" : {"sales_per_month" : {"buckets" : [{"key_as_string" : "2022-04","key" : 1648771200000,"doc_count" : 2,"sales" : {"value" : 140.0}},{"key_as_string" : "2022-05","key" : 1651363200000,"doc_count" : 2,"sales" : {"value" : 30.0}},{"key_as_string" : "2022-06","key" : 1654041600000,"doc_count" : 12,"sales" : {"value" : 2110.0}}]},"avg_monthly_sales" : {"value" : 760.0,"value_as_string" : "760.00"}}
}

2.Max Bucket Aggregation====================================================

最大值桶聚合所定义的桶包含一组聚合指定度量的最大值，并且同时输出桶的键和值。指定的度量必须是数字并且这个聚合必须是多桶聚合。

计算每月销售总额的最大值:

第一级聚合：sales_per_month，（按月）直方图聚合。
第二级聚合：sales，在按月聚合的基础上，对每个月的文档求sum。
第三级聚合：max_monthly_sales，对上面的聚合求最大值
“buckets_path”: “sales_per_month>sales”
max_bucket聚合想要计算最大值的桶路径，要得到sales_per_month日期直方图中sales聚合的最大值
1.对sales_per_month分组聚合的sales指标作为pipeline aggregations的输入源。
2.将"sales"中的sum值作为sales_per_month桶聚合的输入
3.将指向 sales指标值，该值包含在 “sales_per_month” 存储桶聚合中。

POST /sales/_search?size=0
{"aggs": {"sales_per_month": {"date_histogram": {"field": "date","calendar_interval": "month","format": "yyyy-MM"},"aggs": {"sales": {"sum": {"field": "price"}}}},"max_monthly_sales": {"max_bucket": {"buckets_path": "sales_per_month>sales","gap_policy": "skip"}}}
}

响应结果：

aggregations 由 sales_per_month + max_monthly_sales 组成
1.sales_per_month部分：
按月份聚合bucket共3个。
key_as_string 月份
doc_count 当月的doc数量
sales.value 当月price的sum值
2.max_monthly_sales 部分：
value：三个月总价的最高值

{"took" : 0,"timed_out" : false,"_shards" : {"total" : 1,"successful" : 1,"skipped" : 0,"failed" : 0},"hits" : {"total" : {"value" : 16,"relation" : "eq"},"max_score" : null,"hits" : [ ]},"aggregations" : {"sales_per_month" : {"buckets" : [{"key_as_string" : "2022-04","key" : 1648771200000,"doc_count" : 2,"sales" : {"value" : 140.0}},{"key_as_string" : "2022-05","key" : 1651363200000,"doc_count" : 2,"sales" : {"value" : 30.0}},{"key_as_string" : "2022-06","key" : 1654041600000,"doc_count" : 12,"sales" : {"value" : 2110.0}}]},"max_monthly_sales" : {"value" : 2110.0,"keys" : ["2022-06"]}}
}

3.Min Bucket Aggregation ====================================================

最小分组聚合所定义的桶包含一组聚合指定度量的最小值，并且同时输出桶的键和值。指定的度量必须是数字型而且这个组聚合必须是多桶聚合。

计算每月销售总额的最小值:

第一级聚合：sales_per_month，（按月）直方图聚合。
第二级聚合：sales，在按月聚合的基础上，对每个月的文档求sum。
第三级聚合：min_monthly_sales，对上面的聚合求最小值
“buckets_path”: “sales_per_month>sales”
min_bucket聚合是要得到sales_per_month日期直方图中sales聚合的最小值。
1.对sales_per_month分组聚合的sales指标作为pipeline aggregations的输入源。
2.将"sales"中的sum值作为sales_per_month桶聚合的输入
3.将指向 sales指标值，该值包含在 “sales_per_month” 存储桶聚合中。

POST /sales/_search?size=0
{"aggs": {"sales_per_month": {"date_histogram": {"field": "date","calendar_interval": "month","format": "yyyy-MM"},"aggs": {"sales": {"sum": {"field": "price"}}}},"min_monthly_sales": {"min_bucket": {"buckets_path": "sales_per_month>sales" }}}
}

响应结果：

aggregations 由 sales_per_month + min_monthly_sales 组成
1.sales_per_month部分：
按月份聚合bucket共3个。
key_as_string 月份
doc_count 当月的doc数量
sales.value 当月price的sum值
2.min_monthly_sales 部分：
value：三个月总价的最小值

{"took" : 1,"timed_out" : false,"_shards" : {"total" : 1,"successful" : 1,"skipped" : 0,"failed" : 0},"hits" : {"total" : {"value" : 16,"relation" : "eq"},"max_score" : null,"hits" : [ ]},"aggregations" : {"sales_per_month" : {"buckets" : [{"key_as_string" : "2022-04","key" : 1648771200000,"doc_count" : 2,"sales" : {"value" : 140.0}},{"key_as_string" : "2022-05","key" : 1651363200000,"doc_count" : 2,"sales" : {"value" : 30.0}},{"key_as_string" : "2022-06","key" : 1654041600000,"doc_count" : 12,"sales" : {"value" : 2110.0}}]},"min_monthly_sales" : {"value" : 30.0,"keys" : ["2022-05"]}}
}

4.Sum bucket aggregation====================================================

总和桶聚合用于计算一组聚合创建的所有桶中指定度量的和。指定的度量必须是数字型而且这个组聚合必须是多桶聚合。

计算每月销售总额求和:

第一级聚合：sales_per_month，（按月）直方图聚合。
第二级聚合：sales，在按月聚合的基础上，对每个月的文档求sum。
第三级聚合：sum_monthly_sales，对上面的聚合求和
“buckets_path”: “sales_per_month>sales”
– sum_bucket聚合是要得到sales_per_month日期直方图中sales聚合的总和。
– 1.对sales_per_month分组聚合的sales指标作为pipeline aggregations的输入源。
– 2.将"sales"中的sum值作为sales_per_month桶聚合的输入
– 3.将指向 sales指标值，该值包含在 “sales_per_month” 存储桶聚合中。


POST /sales/_search?size=0
{"aggs": {"sales_per_month": {"date_histogram": {"field": "date","interval": "month","format": "yyyy-MM"},"aggs": {"sales": {"sum": {"field": "price"}}}},"sum_monthly_sales": {"sum_bucket": {"buckets_path": "sales_per_month>sales"}}}
}

响应结果：

aggregations 由 sales_per_month + sum_monthly_sales 组成
1.sales_per_month部分：
按月份聚合bucket共3个。
key_as_string 月份
doc_count 当月的doc数量
sales.value 当月price的sum值
2.sum_monthly_sales 部分：
value：三个月总价的总和

{"took" : 1,"timed_out" : false,"_shards" : {"total" : 1,"successful" : 1,"skipped" : 0,"failed" : 0},"hits" : {"total" : {"value" : 16,"relation" : "eq"},"max_score" : null,"hits" : [ ]},"aggregations" : {"sales_per_month" : {"buckets" : [{"key_as_string" : "2022-04","key" : 1648771200000,"doc_count" : 2,"sales" : {"value" : 140.0}},{"key_as_string" : "2022-05","key" : 1651363200000,"doc_count" : 2,"sales" : {"value" : 30.0}},{"key_as_string" : "2022-06","key" : 1654041600000,"doc_count" : 12,"sales" : {"value" : 2110.0}}]},"sum_monthly_sales" : {"value" : 2280.0}}
}

5.Stats Bucket Aggregation统计桶聚合====================================================

统计桶聚合，在一组聚合的所有桶中对一个指定的度量计算各种统计值。指定的度量必须是数字型而且这个桶聚合必须是多桶聚合。

计算每月销售总额各种统计状态:

count 总数
min 最小值
max 最大值
avg 平均值
sum 总和

第一级聚合：sales_per_month，（按月）直方图聚合。
第二级聚合：sales，在按月聚合的基础上，对每个月的文档求sum。
第三级聚合：stats_monthly_sales，对上面的聚合统计状态
“buckets_path”: “sales_per_month>sales”
stats_bucket聚合是要得到sales_per_month日期直方图中的sales聚合的统计信息。
1.对sales_per_month分组聚合的sales指标作为pipeline aggregations的输入源。
2.将"sales"中的sum值作为sales_per_month桶聚合的输入
3.将指向 sales指标值，该值包含在 “sales_per_month” 存储桶聚合中。

POST /sales/_search?size=0
{"aggs": {"sales_per_month": {"date_histogram": {"field": "date","calendar_interval": "month","format": "yyyy-MM"},"aggs": {"sales": {"sum": {"field": "price"}}}},"stats_monthly_sales": {"stats_bucket": {"buckets_path": "sales_per_month>sales"}}}
}

响应结果：

aggregations 由 sales_per_month + stats_monthly_sales 组成
1.sales_per_month部分：
按月份聚合bucket共3个。
key_as_string 月份
doc_count 当月的doc数量
sales.value 当月price的sum值
2.stats_monthly_sales 部分：
value：三个月总价的统计状态

{"took" : 1,"timed_out" : false,"_shards" : {"total" : 1,"successful" : 1,"skipped" : 0,"failed" : 0},"hits" : {"total" : {"value" : 16,"relation" : "eq"},"max_score" : null,"hits" : [ ]},"aggregations" : {"sales_per_month" : {"buckets" : [{"key_as_string" : "2022-04","key" : 1648771200000,"doc_count" : 2,"sales" : {"value" : 140.0}},{"key_as_string" : "2022-05","key" : 1651363200000,"doc_count" : 2,"sales" : {"value" : 30.0}},{"key_as_string" : "2022-06","key" : 1654041600000,"doc_count" : 12,"sales" : {"value" : 2110.0}}]},"stats_monthly_sales" : {"count" : 3,"min" : 30.0,"max" : 2110.0,"avg" : 760.0,"sum" : 2280.0}}
}

6. Extended stats bucket aggregation扩展stats====================================================

同级管道聚合，它计算同级集合中指定度量的所有桶的各种统计信息。指定的度量必须是数字，同级聚合必须是多桶聚合。
与stats_bucket聚合相比，此聚合提供了更多的统计信息（平方和，标准偏差等）。
参数
gap_policy 在数据中找到差异时应用的策略
format 应用于此聚合的输出值的格式
sigma 要显示的平均值之上/之下的标准偏差数

计算每月销售总额扩展统计状态:

count 总数
min 最小值
max 最大值
avg 平均值
sum 总和
sum_of_squares
variance
variance_population
variance_sampling
std_deviation
std_deviation_population
std_deviation_sampling
std_deviation_bounds

第一级聚合：sales_per_month，（按月）直方图聚合。
第二级聚合：sales，在按月聚合的基础上，对每个月的文档求sum。
第三级聚合：extended_stats_bucket，对上面的聚合统计状态
“buckets_path”: “sales_per_month>sales”
extendes_stats_bucket聚合要得到在sales_per_month日期直方图中计算sales聚合的统计信息。
1.对sales_per_month分组聚合的sales指标作为pipeline aggregations的输入源。
2.将"sales"中的sum值作为sales_per_month桶聚合的输入
3.将指向 sales指标值，该值包含在 “sales_per_month” 存储桶聚合中。

POST /sales/_search?size=0
{"aggs": {"sales_per_month": {"date_histogram": {"field": "date","calendar_interval": "month","format": "yyyy-MM"},"aggs": {"sales": {"sum": {"field": "price"}}}},"stats_monthly_sales": {"extended_stats_bucket": {"buckets_path": "sales_per_month>sales"}}}
}

响应结果：

aggregations 由 sales_per_month + extended_stats_bucket 组成
1.sales_per_month部分：
按月份聚合bucket共3个。
key_as_string 月份
doc_count 当月的doc数量
sales.value 当月price的sum值
2.extended_stats_bucket 部分：
value：三个月的统计状态

{"took" : 2,"timed_out" : false,"_shards" : {"total" : 1,"successful" : 1,"skipped" : 0,"failed" : 0},"hits" : {"total" : {"value" : 16,"relation" : "eq"},"max_score" : null,"hits" : [ ]},"aggregations" : {"sales_per_month" : {"buckets" : [{"key_as_string" : "2022-04","key" : 1648771200000,"doc_count" : 2,"sales" : {"value" : 140.0}},{"key_as_string" : "2022-05","key" : 1651363200000,"doc_count" : 2,"sales" : {"value" : 30.0}},{"key_as_string" : "2022-06","key" : 1654041600000,"doc_count" : 12,"sales" : {"value" : 2110.0}}]},"stats_monthly_sales" : {"count" : 3,"min" : 30.0,"max" : 2110.0,"avg" : 760.0,"sum" : 2280.0,"sum_of_squares" : 4472600.0,"variance" : 913266.6666666666,"variance_population" : 913266.6666666666,"variance_sampling" : 1369900.0,"std_deviation" : 955.6498661469412,"std_deviation_population" : 955.6498661469412,"std_deviation_sampling" : 1170.427272409525,"std_deviation_bounds" : {"upper" : 2671.2997322938827,"lower" : -1151.2997322938825,"upper_population" : 2671.2997322938827,"lower_population" : -1151.2997322938825,"upper_sampling" : 3100.85454481905,"lower_sampling" : -1580.8545448190498}}}
}

7.Percentiles Bucket Aggregation（百分数桶聚合） ====================================================

同级管道聚合，它计算同级聚合中指定度量的所有桶的百分位数。指定度量必须是数字，并且同级聚合必须是多桶聚合。
百分数桶返回最近输入的数据点，该数据点不大于所请求的百分数；它不会在数据点之间插值。
百分数是精确计算的，不是近似值（与百分数指标不同）。这意味着在丢弃数据之前，实现会在内存中维护一个有序的数据列表来计算百分数。如果你尝试在数百万的数据点中计算一个百分数的percentiles_bucket，可能会遇到内存压力问题。
参数
gap_policy 在数据中找到差异时应用的策略
format 应用于此聚合的输出值的格式
sigma 要显示的平均值之上/之下的标准偏差数
percents 要计算的百分数列表

计算每月sales桶总数的百分数:

第一级聚合：sales_per_month，（按月）直方图聚合。
第二级聚合：sales，在按月聚合的基础上，对每个月的文档求sum。
第三级聚合：percentiles_monthly_sales，对上面的聚合统计状态
“buckets_path”: “sales_per_month>sales”
percentiles_bucket聚合需要得到sales_per_month数据直方图中计算sales聚合的统计数据
1.对sales_per_month分组聚合的sales指标作为pipeline aggregations的输入源。
2.将"sales"中的sum值作为sales_per_month桶聚合的输入
3.将指向 sales指标值，该值包含在 “sales_per_month” 存储桶聚合中。
percents指定了我们希望计算的百分数，在这里，是第25，第50和第75百分数。

POST /sales/_search?size=0
{"aggs": {"sales_per_month": {"date_histogram": {"field": "date","calendar_interval": "month","format": "yyyy-MM"},"aggs": {"sales": {"sum": {"field": "price"}}}},"percentiles_monthly_sales": {"percentiles_bucket": {"buckets_path": "sales_per_month>sales", "percents": [ 25.0, 50.0, 75.0 ]         }}}
}

响应结果：

aggregations 由 sales_per_month + percentiles_monthly_sales 组成
1.sales_per_month部分：
按月份聚合bucket共3个。
key_as_string 月份
doc_count 当月的doc数量
sales.value 当月price的sum值
2.percentiles_monthly_sales 部分：
value：三个月的统计状态
第25百分数
第50百分数
第75百分数

{"took" : 3,"timed_out" : false,"_shards" : {"total" : 1,"successful" : 1,"skipped" : 0,"failed" : 0},"hits" : {"total" : {"value" : 16,"relation" : "eq"},"max_score" : null,"hits" : [ ]},"aggregations" : {"sales_per_month" : {"buckets" : [{"key_as_string" : "2022-04","key" : 1648771200000,"doc_count" : 2,"sales" : {"value" : 140.0}},{"key_as_string" : "2022-05","key" : 1651363200000,"doc_count" : 2,"sales" : {"value" : 30.0}},{"key_as_string" : "2022-06","key" : 1654041600000,"doc_count" : 12,"sales" : {"value" : 2110.0}}]},"percentiles_monthly_sales" : {"values" : {"25.0" : 140.0,"50.0" : 140.0,"75.0" : 2110.0}}}
}

数据源

DELETE salesPUT sales
{"mappings": {"properties": {"name": {"type": "keyword"},"type": {"type": "keyword"},"price": {"type": "integer"},"country": {"type": "keyword"},"tags": {"type": "keyword"},"date": {"type": "date","format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis","fields": {"keyword": {"type": "keyword","ignore_above": 20}}}}}
}POST sales/_bulk
{"index":{"_id":1}}
{"name":"yellow pepper","type":"pepper","price":20,"date":"2022-06-08 21:41:27","timestamp":1654695687000,"promoted":0,"country":"CN","tags":["vegetable","pepper","fresh","kick","yellow"]}
{"index":{"_id":2}}
{"name":"red pepper","type":"pepper","price":10,"date":"2022-06-12 21:41:27","timestamp":1655041287000,"promoted":1,"country":"EN","tags":["vegetable","pepper","fresh","hot","red"]}
{"index":{"_id":3}}
{"name":"green pepper","type":"pepper","price":20,"date":"2022-05-31 21:41:27","timestamp":1654004487000,"promoted":1,"country":"TH","tags":["vegetable","pepper","fresh","fiery","green"]}
{"index":{"_id":4}}
{"name":"bule pepper","type":"pepper","price":10,"date":"2022-05-31 21:41:27","timestamp":1654004487000,"promoted":1,"country":"CN","tags":["vegetable","pepper","pecial discount","hot","blue"]}
{"index":{"_id":5}}
{"name":"balck t-shirt","type":"t-shirt","price":60,"date":"2022-04-01 12:41:27","timestamp":1648788087000,"promoted":1,"country":"TH","tags":["clothes","t-shirt","grandad shirt","balck","hot","sport","cotton"]}
{"index":{"_id":6}}
{"name":"green t-shirt","type":"t-shirt","price":80,"date":"2022-04-01 12:41:27","timestamp":1648788087000,"promoted":1,"country":"CN","tags":["clothes","t-shirt","green","v-neck","blouse","pure cotton"]}
{"index":{"_id":7}}
{"name":"blue t-shirt","type":"t-shirt","price":100,"date":"2022-06-12 21:41:27","timestamp":1655041287000,"promoted":1,"country":"EN","tags":["clothes","t-shirt","blue","navy collar","100% cotton"]}
{"index":{"_id":8}}
{"name":"red t-shirt","type":"t-shirt","price":80,"date":"2022-06-12 21:41:27","timestamp":1655041287000,"promoted":0,"country":"CN","tags":["clothes","t-shirt","red","sweetheart (heart shaped) neckline","pure cotton"]}
{"index":{"_id":9}}
{"name":"balck hat","type":"hat","price":260,"date":"2022-06-12 21:41:27","timestamp":1655041287000,"promoted":1,"country":"CN","tags":["clothes","hat","balck","streetwear","sunscreen","Bucket hat","Screen Printing","polyester 190t","Kaitlyn Bristow"]}
{"index":{"_id":10}}
{"name":"red hat","type":"hat","price":200,"date":"2022-06-12 21:41:27","timestamp":1655041287000,"promoted":1,"country":"EN","tags":["clothes","hat","red","streetwear","sunscreen","straw hat","90% straw"]}
{"index":{"_id":11}}
{"name":"white hat","type":"hat","price":180,"date":"2022-06-12 21:41:27","timestamp":1655041287000,"promoted":1,"country":"CN","tags":["clothes","hat","white","streetwear","sunscreen","cowboy cap","felt","Bollman Hat","80% wool,20% N"]}
{"index":{"_id":12}}
{"name":"balck hat","type":"hat","price":160,"date":"2022-06-12 21:41:27","timestamp":1655041287000,"promoted":1,"country":"TH","tags":["clothes","hat","balck","streetwear","Bucket hat","Baseball cap","hot","sport","30% cotton"]}
{"index":{"_id":13}}
{"name":"blue hat","type":"hat","price":160,"date":"2022-06-12 21:41:27","timestamp":1655041287000,"promoted":1,"country":"TH"}
{"index":{"_id":14}}
{"name":"red bag","type":"bag","price":300,"date":"2022-06-12 21:41:27","timestamp":1655041287000,"promoted":1,"country":"EN","tags":["bag","red","sport","90% straw"]}
{"index":{"_id":15}}
{"name":"white bag","type":"bag","price":380,"date":"2022-06-12 21:41:27","timestamp":1655041287000,"promoted":1,"country":"CN","tags":["bag","white","sport","80% wool,20% N"]}
{"index":{"_id":16}}
{"name":"balck bag","type":"bag","price":260,"date":"2022-06-12 21:41:27","timestamp":1655041287000,"promoted":1,"country":"TH","tags":["clothes","bag","balck","hot","sport","30% cotton"]}