Spark-submit 提交 报错 org.apache.spark.sql.execution.datasources.orc.OrcFileFormat could not be instant
2024-05-09 17:59:24
错误场景
如下代码：
spark.sql("select e.empno,e.ename,e.job,e.mgr,e.comm from emp e join dept d on e.deptno = d.deptno").filter("comm is not null").write.parquet("/demp");spark-shell 可以跑的通，spark-submit 提交的时候，报错如下：
Exception in thread "main" java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.execution.datasources.orc.OrcFileFormat could not be instantiatedat java.util.ServiceLoader.fail(ServiceLoader.java:232)at java.util.ServiceLoader.access$100(ServiceLoader.java:185)at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)at java.util.ServiceLoader$1.next(ServiceLoader.java:480)at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)at scala.collection.Iterator$class.foreach(Iterator.scala:893)at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)at scala.collection.AbstractIterable.foreach(Iterable.scala:54)at scala.collection.TraversableLike$class.filterImpl(TraversableLike.scala:247)at scala.collection.TraversableLike$class.filter(TraversableLike.scala:259)at scala.collection.AbstractTraversable.filter(Traversable.scala:104)at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:575)at org.apache.spark.sql.execution.datasources.DataSource.providingClass$lzycompute(DataSource.scala:86)at org.apache.spark.sql.execution.datasources.DataSource.providingClass(DataSource.scala:86)at org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:516)at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:215)at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:198)at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:494)at sparkSql.SparkSQLHiveDemo$.main(SparkSQLHiveDemo.scala:17)at sparkSql.SparkSQLHiveDemo.main(SparkSQLHiveDemo.scala)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:743)at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.FileFormat.$init$(Lorg/apache/spark/sql/execution/datasources/FileFormat;)Vat org.apache.spark.sql.execution.datasources.orc.OrcFileFormat.<init>(OrcFileFormat.scala:81)at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at java.lang.Class.newInstance(Class.java:442)at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)... 28 more
错误原因
定位到错误是由于 .write.parquet("/demp") 写入到 hdfs 报错。
错误原因是因为本地开发所用的 spark jar 和 服务器 Spark 的版本不一致导致的。小伙伴们一定要注意这个问题哦，不然错误莫名其妙。
解决方案
修改 pom 文件，将依赖修改为 provided（由于 Spark 服务器上有这些 jar 所以打包的时候不用将本地的打进去也能运行） ，再次打包提交，运行通过
<dependency><groupId>org.apache.spark</groupId><artifactId>spark-core_2.12</artifactId><version>${spark.version}</version><!-- provider如果存在，那么运行时该Jar包不存在，也不会打包到最终的发布版本中，只是编译器有效 --><scope>provided</scope></dependency><dependency><groupId>org.apache.spark</groupId><artifactId>spark-sql_2.12</artifactId><version>${spark.version}</version><scope>provided</scope></dependency><dependency><groupId>org.apache.spark</groupId><artifactId>spark-streaming_2.12</artifactId><version>${spark.version}</version><scope>provided</scope></dependency><dependency><groupId>org.apache.spark</groupId><artifactId>spark-hive_2.12 </artifactId><version>${spark.version}</version><scope>provided</scope></dependency>Spark-submit 提交 报错 org.apache.spark.sql.execution.datasources.orc.OrcFileFormat could not be instant相关推荐
- 提交spark任务偶尔报错 org.apache.spark.SparkException: A master URL must be set in your configuration
问题一:pox.xml里外部以来库没有加入<scope>compile</scope>导致无法初始化 问题二:main函数以外的方法有时也无法初始化 错误信息: 20/11/2 ...
- spark 序列化错误 集群提交时_【问题解决】本地提交任务到Spark集群报错：Initial job has not accepted any resources...
本地提交任务到Spark集群报错:Initial job has not accepted any resources 错误信息如下: 18/04/17 18:18:14 INFO TaskSched ...
- Spark SQL入门：创建SparkSession时import spark.implicits._ 报错： error: value implicits is not a member of...
Spark SQL入门:创建SparkSession时import spark.implicits._ 报错: error: value implicits is not a member of... ...
- Spark操作sparkSql报错：metastore.ObjectStore: Version information found in metastore differs 2.3.0 from e
Spark操作sparkSql报错:metastore.ObjectStore: Version information found in metastore differs 2.3.0 from e ...
- Spark连接oracle报错requirement failed: Decimal precision 1 exceeds max precision 0
Spark连接oracle报错requirement failed: Decimal precision 1 exceeds max precision 0 报错显示Decimal精度问题,spark ...
- 解决gitte提交报错 error: GE007: Your push would publish a private email address.
错误情况: gitte提交报错,无法提交上去,报错如下: remote: Powered by GITEE.COM [GNK-6.0] remote: error: GE007: Your push ...
- 【错误记录】GitHub 提交报错 ( OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to github.com:443 )
文章目录 一.报错信息 二.解决方案 一.报错信息 GitHub 提交报错 : 16:37:19.781: [ClassLoader_Demo] git -c credential.helper= - ...
- spring boot一个模块加载不到引用另一个模块的mapper.xml报错org.apache.ibatis.binding.BindingException: Invalid bound sta
场景:parent项目有两个子模块,分别是shiro和server,两个子模块各自有各自的实体类.mapper,然后server需要引用shiro中的实体类和mapper.已经在启动类添加注解配置扫描 ...
- @webservice报错org.apache.cxf.common.i18n.UncheckedException: No operation was found with
文章目录 1. 现象 2. 解决办法1 3. 解决办法2 1. 现象 整合spring+cxf的webservice,成功发布了wsdl,但在调用的时候报错 org.apache.cxf.common ...
最新文章
- Asp.net中的AJAX学习记录之一 AJAX技术的崛起与Asp.net AJAX的诞生
- MySQL 为什么表的数据删除一般，表文件大小不变？
- 判断sem信号量为零_Linux系统编程——进程同步与互斥：System V 信号量
- 存储控制器_SDRAM详解
- windows xp https页面找不到_Windows 提权快速查找 Exp
- python程序员怎么面试_Python程序员面试，这些问题你必须提前准备！
- Character，String相关方法，Int，double互相转换
- python变量的理解_如何理解Python中的变量
- 如何监测土壤pH值和水分含量
- 网络编程-在线英英词典项目
- Day11 Dear Japanese English Learners
- c语言由天数求日期,C语言-由日期求天数由天数求日期
- [附源码]JAVA+ssm计算机毕业设计餐饮管理系统（程序+Lw)
- springboot集成flink-cdc
- 麒麟服务器系统磁盘管理之lvm管理
- WY的Java学习笔记（1）基础
- win7桌面右下角有计算机图标,win7系统电脑任务栏右下角图标空白的解决方法
- 如何区别API、REST API、RESTful API和Web Service之间的异同
- 学习笔记 | 极简英语 | 三大变身、虚拟语气、独立主格结构
- 大学生计算机学科竞赛a类,专业学科竞赛（A类）