SparkOnHive

package com.shujia.sqlimport org.apache.spark.sql.expressions.Window
import org.apache.spark.sql.{DataFrame, SparkSession}object Demo06SparkOnHive {def main(args: Array[String]): Unit = {/*** 通过enableHiveSupport()可以开启Hive的支持* 需要在pom文件中加入spark-hive的依赖* <dependency>* <groupId>org.apache.spark</groupId>* <artifactId>spark-hive_2.11</artifactId>* <version>2.4.5</version>* </dependency>* 需要启动hive的metastore* hive --service metastore**/val spark: SparkSession = SparkSession.builder().appName("Demo06HiveOnSpark").master("local").enableHiveSupport() // 开启Hive的支持.config("spark.sql.shuffle.partitions", 2) // 默认200.getOrCreate()import org.apache.spark.sql.functions._import spark.implicits._spark.sql("show databases").show()spark.sql("show tables").show()spark.sql("use stu_spark").show()spark.sql("show tables").show()// 统计每个科目排名前十的学生  分组取topNspark.sql("""|select  t1.student_id|        ,t1.cource_id|        ,t1.sco|        ,t1.rn|from (|        select  student_id|                ,cource_id|                ,sco|                ,row_number() over(partition by cource_id order by sco desc) as rn|        from score|) t1 where rn<=10|""".stripMargin).show()// DSLval scoDF: DataFrame = spark.table("score")scoDF.select($"student_id", $"cource_id", $"sco",row_number() over Window.partitionBy($"cource_id").orderBy($"sco".desc) as "rn").where($"rn" <= 10).show()}
}

+------------+
|databaseName|
+------------+
|     default|
|         stu|
|   stu_spark|
|       test1|
|   testsqoop|
+------------++--------+---------------+-----------+
|database|      tableName|isTemporary|
+--------+---------------+-----------+
| default|ods_yiqing_data|      false|
| default|        student|      false|
| default| students_hbase|      false|
+--------+---------------+-----------+++
||
++
+++---------+---------+-----------+
| database|tableName|isTemporary|
+---------+---------+-----------+
|stu_spark|    score|      false|
|stu_spark|  student|      false|
+---------+---------+-----------++----------+---------+---+---+
|student_id|cource_id|sco| rn|
+----------+---------+---+---+
|1500100007|  1000002|149|  1|
|1500100080|  1000002|149|  2|
|1500100154|  1000002|149|  3|
|1500100206|  1000002|149|  4|
|1500100298|  1000002|149|  5|
|1500100300|  1000002|149|  6|
|1500100547|  1000002|149|  7|
|1500100599|  1000002|149|  8|
|1500100641|  1000002|149|  9|
|1500100679|  1000002|149| 10|
|1500100208|  1000003|149|  1|
|1500100572|  1000003|149|  2|
|1500100616|  1000003|149|  3|
|1500100776|  1000003|149|  4|
|1500100867|  1000003|149|  5|
|1500100871|  1000003|149|  6|
|1500100911|  1000003|149|  7|
|1500100072|  1000003|148|  8|
|1500100645|  1000003|148|  9|
|1500100725|  1000003|148| 10|
+----------+---------+---+---+
only showing top 20 rows+----------+---------+---+---+
|student_id|cource_id|sco| rn|
+----------+---------+---+---+
|1500100007|  1000002|149|  1|
|1500100080|  1000002|149|  2|
|1500100154|  1000002|149|  3|
|1500100206|  1000002|149|  4|
|1500100298|  1000002|149|  5|
|1500100300|  1000002|149|  6|
|1500100547|  1000002|149|  7|
|1500100599|  1000002|149|  8|
|1500100641|  1000002|149|  9|
|1500100679|  1000002|149| 10|
|1500100208|  1000003|149|  1|
|1500100572|  1000003|149|  2|
|1500100616|  1000003|149|  3|
|1500100776|  1000003|149|  4|
|1500100867|  1000003|149|  5|
|1500100871|  1000003|149|  6|
|1500100911|  1000003|149|  7|
|1500100072|  1000003|148|  8|
|1500100645|  1000003|148|  9|
|1500100725|  1000003|148| 10|
+----------+---------+---+---+
only showing top 20 rows

SparkOnHive相关推荐

Spark基础知识(个人总结)
声明: 1. 本文为我的个人复习总结, 并非那种从零基础开始普及知识内容详细全面, 言辞官方的文章 2. 由于是个人总结, 所以用最精简的话语来写文章 ...
惊了！10万字的Spark全文！
Hello,大家好,这里是857技术社区,我是社区创始人之一,以后会持续给大家更新大数据各组件的合集内容,路过给个关注吧!!! 今天给大家分享一篇小白易读懂的 Spark万字概念长文,本篇文章追求的是 ...
离线数仓搭建流程以及遇到的问题Hadoop3.3.1-hive3.1.2-spark 3.3.1
目录简言数仓选型前期准备更改三个节点主机名: 新增用户组以及用户: 配置互信 JDK安装 mysql安装 Zookeeper3.8安装 Hadoop3.3.1搭建 Hive3.1.3搭建 Sp ...
hive udf kyroexception unable to find class
H ive-On-Spark配置成功后,准备试用下Hive UDF是否能在Spark-on-Hive环境下正常使用: set hive.execution.engine=spark; add jar ...
基于 Spark 的数据分析香港六合彩开奖号码采集官网实践
引言: Spark是在借鉴了MapReduce之上发展香港六合彩开奖号码采集官网vip7.maltapi.com而来的,继承了其分布式并行计算的优点并改进了MapReduce明显的缺陷.Spark主要 ...

SparkOnHive

SparkOnHive相关推荐

最新文章

热门文章