idea sparksql操作hive遇到的坑CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent ex

sparksql操作hive遇到的坑
第一次写sparksql项目想用sparksql操作hive查询数据就找到了百度代码做参考【idea工具开发】

代码如下：
import org.apache.spark.sql.SparkSession

object aaa {
def main(args: Array[String]): Unit = {

//设置HADOOP_USER_NAME，否则会有权限问题
System.setProperty("HADOOP_USER_NAME", "hadoop")

val spark = SparkSession
.builder()
.appName("SparkHiveDemo")
.master("spark://192.168.43.128:7077")
.enableHiveSupport()
.config("spark.sql.warehouse.dir", "/user/hive/warehouse/")
.getOrCreate()

spark.sql("select * from student").show()
spark.close()
}
}

现象：执行'show tables;'语句完全没有问题执行'select * from student'就会出现以下现象 executor不停分配和移除：

21/02/26 16:11:01 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 5
21/02/26 16:11:01 INFO StandaloneSchedulerBackend: Granted executor ID app-20210226161103-0021/8 on hostPort 192.168.43.127:7079 with 1 core(s), 1024.0 MB RAM
21/02/26 16:11:01 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20210226161103-0021/8 is now RUNNING
21/02/26 16:11:01 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20210226161103-0021/6 is now EXITED (Command exited with code 1)
21/02/26 16:11:01 INFO StandaloneSchedulerBackend: Executor app-20210226161103-0021/6 removed: Command exited with code 1
21/02/26 16:11:01 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20210226161103-0021/9 on worker-20210226101445-192.168.43.130-7079 (192.168.43.130:7079) with 1 core(s)
21/02/26 16:11:01 INFO BlockManagerMasterEndpoint: Trying to remove executor 6 from BlockManagerMaster.

以上日志显示不断循环

查了一下excutor日志原因找报错如下【报错日志目录spark/work/app-20210226153746-0020/98/stderr】
   ... 4 more
Caused by: java.io.IOException: Failed to connect to DESKTOP-HKJLBCB:58243
   at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:245)
   at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:187)
   at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:198)
   at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:194)
   at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:190)
   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: DESKTOP-HKJLBCB
   at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
   at java.net.InetAddress.getAllByName(InetAddress.java:1192)
   at java.net.InetAddress.getAllByName(InetAddress.java:1126)

原因：master指定错误，替换成如下代码即可完成

变更后的代码：
import org.apache.spark.sql.SparkSession

object aaa {
def main(args: Array[String]): Unit = {

//设置HADOOP_USER_NAME，否则会有权限问题
System.setProperty("HADOOP_USER_NAME", "hadoop")

val spark = SparkSession
.builder()
.appName("SparkHiveDemo")
.master("local[*]")
.enableHiveSupport()
.config("spark.sql.warehouse.dir", "/user/hive/warehouse/")
.getOrCreate()

spark.sql("select * from student").show()
spark.close()
}
}

或者可以参考如下连接：

https://www.cnblogs.com/Mr-lin66/p/13519103.html

idea sparksql操作hive遇到的坑CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent ex相关推荐

SparkSQL操作Hive
title: SparkSQL操作Hive date: 2020-05-12 16:12:55 tags: Spark Apache Hive 是 Hadoop 上的 SQL 引擎,Spark SQL ...
python读取oracle数据到hvie parquet_关于sparksql操作hive，读取本地csv文件并以parquet的形式装入hive中...
说明:spark版本:2.2.0 hive版本:1.2.1 需求: 有本地csv格式的一个文件,格式为${当天日期}visit.txt,例如20180707visit.txt,现在需要将其通过spar ...
SparkSQL操作Hive Table
Spark SQL支持对Hive的读写操作.然而因为Hive有很多依赖包,所以这些依赖包没有包含在默认的Spark包里面.如果Hive依赖的包能在classpath找到,Spark将会自动加载它们.需 ...
Spark采坑系列（三）Spark操作Hive的坑
2019独角兽企业重金招聘Python工程师标准>>> 跟着教学试着用Idea编程,实现Spark查询Hive中的表.结果上来就凉了. 捣鼓好久都不行,在网上查有说将hive-sit ...
sparksql对hive操作
操作hive表数据 spark.table(tableName) 写回数据 df.wrtite.saveAsTable(tableName) spark.sql("select deptno ...
sparkSQL操作hiveSQL
sparkSQL操作hiveSQL <dependency><groupId>org.apache.spark</groupId><artifactId> ...
sparksql 操作hive_Spark SQL 物化视图原理与实践
导言物化视图作为一种预计算的优化方式,广泛应用于传统数据库中,如Oracle,MSSQL Server等.随着大数据技术的普及,各类数仓及查询引擎在业务中扮演着越来越重要的数据分析角色,而物化视图作 ...
python操作hive和hive_sql语句
Hive是一个数据仓库基础的应用工具,在Hadoop中用来处理结构化数据,通过类SQL语言对数据进行操作.Hive将sql语句通过解析器转换成MapReduce作业提交到Hadoop集群上,Hadoo ...
spark2.2读写操作hive和mysql数据库
2019独角兽企业重金招聘Python工程师标准>>> sparksession操作hive数据库 1.需要将core-site.xml,hdfs-site.xml,hive-sit ...

idea sparksql操作hive遇到的坑CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent ex

idea sparksql操作hive遇到的坑CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent ex相关推荐

最新文章

热门文章