关于spark的sample()算子参数详解

sample(withReplacement : scala.Boolean, fraction : scala.Double，seed scala.Long)

sample算子时用来抽样用的，其有3个参数

withReplacement：表示抽出样本后是否在放回去，true表示会放回去，这也就意味着抽出的样本可能有重复

fraction ：抽出多少，这是一个double类型的参数,0-1之间，eg:0.3表示抽出30%

seed：表示一个种子，根据这个seed随机抽取，一般情况下只用前两个参数就可以，那么这个参数是干嘛的呢，这个参数一般用于调试，有时候不知道是程序出问题还是数据出了问题，就可以将这个参数设置为定值

================================================================================

下面是代码:

大概思路是：通过抽样取出一部分样本，在对样本做wordCount并排序最后取出出现次数最多的key，这个key就是导致数据倾斜的key

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import scala.Tuple2;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;public class Day05 {public static void main(String[] args) {SparkConf conf = new SparkConf().setMaster("local[2]").setAppName("Day05");JavaSparkContext jsc = new JavaSparkContext(conf);List<String> keys = getKeyBySample(jsc);System.out.println("导致数据倾斜的key是:"+keys);jsc.stop();}/*** 通过Sample算子进行抽样并把导致数据倾斜的key找出来* 然后可以做对计算做针对性的优化* @param jsc*/public static List<String> getKeyBySample(JavaSparkContext jsc){List<String> data = Arrays.asList("A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","B","B","B","B","B","B","B","B","C","D","E","F","G");JavaRDD<String> rdd =  jsc.parallelize(data,2);List<Tuple2> item =rdd.mapToPair(x->new Tuple2<String,Integer>(x,1)).sample(true,0.4).reduceByKey((x,y)->x+y).map(x->new Tuple2(x._2,x._1)).sortBy(x->x._1,false,2).take(3);List<String> keys = new ArrayList<>();System.out.println("keys="+item);for(int i=0;i<item.size();i++){if(i == item.size()-1)break;Tuple2 current = item.get(i);Tuple2 next = item.get(i+1);Integer v1 = Integer.parseInt(current._1.toString());Integer v2 = Integer.parseInt(next._1.toString());System.out.println(v1+"   "+v2);/*** 这儿的逻辑有问题,找出导致数据倾斜的key的方式和具体的业务也有关系* 这里只是给了一个简单的判断方法，很有局限性*/if(v1/v2 >= 3){System.out.println("===");keys.add(current._2.toString());}}return keys;}
}

---------------------

原文：https://blog.csdn.net/lyzx_in_csdn/article/details/79948799

关于spark的sample()算子参数详解相关推荐

Spark SQL原理及常用方法详解（二）
Spark SQL 一.Spark SQL基础知识 1.Spark SQL简介 (1)简单介绍 (2)Datasets & DataFrames (3)Spark SQL架构 (4)Spark ...
Lesson 8.3Lesson 8.4 ID3、C4.5决策树的建模流程CART回归树的建模流程与sklearn参数详解
Lesson 8.3 ID3.C4.5决策树的建模流程 ID3和C4.5作为的经典决策树算法,尽管无法通过sklearn来进行建模,但其基本原理仍然值得讨论与学习.接下来我们详细介绍关于ID3和C4. ...
oracle imp 1403,Oracle中用exp/imp命令参数详解【转】
Oracle中用exp/imp命令参数详解 [用 exp 数据导出]: 1 将数据库TEST完全导出,用户名system 密码manager 导出到D:daochu.dmp中 expsyste ...
04_Flink-HA高可用、Standalone集群模式、Flink-Standalone集群重要参数详解、集群节点重启及扩容、启动组件、Flink on Yarn、启动命令等
1.4.Flink集群安装部署standalone+yarn 1.4.1.Standalone集群模式 1.4.2.Flink-Standalone集群重要参数详解 1.4.3.集群节点重启及扩容 1 ...
matlab melbankm,Matlab v_melbankm函数参数详解（英文附例）
Matlab v_melbankm函数参数详解(英文附例) 笔者使用的是R2019的matlab,下载了voicebox安装至matlab路径下即可使用.下载voicebox请参看此博客需要注意的是 ...
Spark: sortBy和sortByKey函数详解
在很多应用场景都需要对结果数据进行排序,Spark中有时也不例外.在Spark中存在两种对RDD进行排序的函数,分别是 sortBy和sortByKey函数.sortBy是对标准的RDD进行排序,它是 ...
【学习笔记】sed 命令及参数详解
sed 命令及参数详解标签:Sed 文章目录 sed 命令及参数详解一.简介二.基本用法 2.1. 三种方式 2.2. 常用选项 2.3. 命令体的组成 2.3.1. 定位文本部分 2.3.2. ...
阿里云AMD服务器ECS实例g6a、c6a和r6a性能参数详解
阿里云AMD服务器ECS计算型c6a.通用型g6a和内存型r6a实例,CPU采用2.6GHz主频的AMD EPYCTM ROME处理器,睿频3.3GHz,计算性能稳定,云服务器吧来详细说下阿里云ECS ...
Matlab mfcc函数参数详解（英文附例）
Matlab mfcc函数参数详解其实可以直接打开源代码看哈. %MFCC Extract the mfcc, log-energy, delta, and delta-delta of audio ...

关于spark的sample()算子参数详解

关于spark的sample()算子参数详解相关推荐

最新文章

热门文章