在Hadoop Yarn上运行pyspark-summit程序

出现以下错误：
19/02/22 18:24:13 ERROR SparkContext: Error initializing SparkContext.
java.net.ConnectException: Call From master/192.168.56.100 to 0.0.0.0:8032 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:731)
at org.apache.hadoop.ipc.Client.call(Client.java:1473)
at org.apache.hadoop.ipc.Client.call(Client.java:1400)
at org.apache.hadoop.ipc.ProtobufRpcEngineInvoker.invoke(ProtobufRpcEngine.java:232)atcom.sun.proxy.Invoker.invoke(ProtobufRpcEngine.java:232) at com.sun.proxy.Invoker.invoke(ProtobufRpcEngine.java:232)atcom.sun.proxy.Proxy9.getNewApplication(Unknown Source)
at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getNewApplication(ApplicationClientProtocolPBClientImpl.java:217)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.Proxy10.getNewApplication(UnknownSource)atorg.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getNewApplication(YarnClientImpl.java:206)atorg.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createApplication(YarnClientImpl.java:214)atorg.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:157)atorg.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)atorg.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:149)atorg.apache.spark.SparkContext.<init>(SparkContext.scala:500)atorg.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)atsun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeMethod)atsun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)atsun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)atjava.lang.reflect.Constructor.newInstance(Constructor.java:423)atpy4j.reflection.MethodInvoker.invoke(MethodInvoker.java:240)atpy4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)atpy4j.Gateway.invoke(Gateway.java:236)atpy4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)atpy4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)atpy4j.GatewayConnection.run(GatewayConnection.java:211)atjava.lang.Thread.run(Thread.java:748)Causedby:java.net.ConnectException:拒绝连接atsun.nio.ch.SocketChannelImpl.checkConnect(NativeMethod)atsun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)atorg.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)atorg.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)atorg.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)atorg.apache.hadoop.ipc.ClientProxy10.getNewApplication(Unknown Source) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getNewApplication(YarnClientImpl.java:206) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createApplication(YarnClientImpl.java:214) at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:157) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56) at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:149) at org.apache.spark.SparkContext.<init>(SparkContext.scala:500) at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:240) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:236) at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80) at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69) at py4j.GatewayConnection.run(GatewayConnection.java:211) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.ConnectException: 拒绝连接 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494) at org.apache.hadoop.ipc.ClientProxy10.getNewApplication(UnknownSource)atorg.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getNewApplication(YarnClientImpl.java:206)atorg.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createApplication(YarnClientImpl.java:214)atorg.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:157)atorg.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)atorg.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:149)atorg.apache.spark.SparkContext.<init>(SparkContext.scala:500)atorg.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)atsun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeMethod)atsun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)atsun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)atjava.lang.reflect.Constructor.newInstance(Constructor.java:423)atpy4j.reflection.MethodInvoker.invoke(MethodInvoker.java:240)atpy4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)atpy4j.Gateway.invoke(Gateway.java:236)atpy4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)atpy4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)atpy4j.GatewayConnection.run(GatewayConnection.java:211)atjava.lang.Thread.run(Thread.java:748)Causedby:java.net.ConnectException:拒绝连接atsun.nio.ch.SocketChannelImpl.checkConnect(NativeMethod)atsun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)atorg.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)atorg.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)atorg.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)atorg.apache.hadoop.ipc.ClientConnection.setupConnection(Client.java:608)
at org.apache.hadoop.ipc.ClientConnection.setupIOstreams(Client.java:706)atorg.apache.hadoop.ipc.ClientConnection.setupIOstreams(Client.java:706) at org.apache.hadoop.ipc.ClientConnection.setupIOstreams(Client.java:706)atorg.apache.hadoop.ipc.ClientConnection.access2800(Client.java:369)atorg.apache.hadoop.ipc.Client.getConnection(Client.java:1522)atorg.apache.hadoop.ipc.Client.call(Client.java:1439)...28more19/02/2218:24:13WARNYarnSchedulerBackend2800(Client.java:369) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1522) at org.apache.hadoop.ipc.Client.call(Client.java:1439) ... 28 more 19/02/22 18:24:13 WARN YarnSchedulerBackend2800(Client.java:369)atorg.apache.hadoop.ipc.Client.getConnection(Client.java:1522)atorg.apache.hadoop.ipc.Client.call(Client.java:1439)...28more19/02/2218:24:13WARNYarnSchedulerBackendYarnSchedulerEndpoint: Attempted to request executors before the AM has registered!
19/02/22 18:24:13 WARN MetricsSystem: Stopping a MetricsSystem that is not running
ERROR - failed to write data to stream: <open file ‘’, mode ‘w’ at 0x7f847e83d150>

请求大佬看一下问题出在哪里，谢谢大佬们！

在Hadoop Yarn上运行pyspark-summit程序相关推荐

pythonspark集群模式运行_有关python numpy pandas scipy 等能在YARN集群上运行PySpark
有关这个问题,似乎这个在某些时候,用python写好,且spark没有响应的算法支持, 能否能在YARN集群上运行PySpark方式, 将python分析程序提交上去? Spark Applicat ...
Spark在Yarn上运行Wordcount程序
过往记忆专注于Hadoop.Spark.Hive.Flume.Hbase,QQ交流群:138615359 首页 Hadoop Spark Hive Hbase Flume 面试题面试题数据结构算 ...
java spark yarn_《Spark官方文档》在YARN上运行Spark
原文链接 Spark在 0.6.0版本后支持在YARN(hadoop NextGen)上运行,并且在后续版本中不断改进. 在YARN上启动Spark 首先,确认 HADOOP_CONF_DIR或YAR ...
Spark Standalone -- 独立集群模式、Spark 提交任务的两种模式、spark在yarn上运行的环境搭建、自己写的spark代码如何提交到yarn上并运行...
目录 Spark Standalone -- 独立集群模式 Standalone 架构图 Standalone 的搭建 1.上传.解压.重命名 2.配置环境变量 3.修改配置文件 conf 4.同步到 ...
Spark 在YARN上运行
在YARN上运行Spark 安全在YARN上启动Spark 添加其他JAR 准备工作组态调试您的应用程序 Spark特性重要笔记的Kerberos YARN特定的Kerberos配置 Ker ...
Spark利用（idea+maven+scala）创建wordcount打包jar并在spark on yarn上运行——Spark的开发
今天的你不付昨天的辛苦,今天的辛苦定会拥抱明日的幸福, 每一次的成长,都要给自己以鼓励,每一次的突破,都给自己以信心,万花丛中我不是最美,但我有我的自信 ------------送给一直努力的你今天 ...
linux里运行windows,在Linux上运行Windows应用程序
当前位置:我的异常网» Linux/Unix » 在Linux上运行Windows应用程序在Linux上运行Windows应用程序 www.myexceptions.net 网友分享于:2015- ...
linux如何运行windows游戏,如何在Linux上运行Windows应用程序和游戏互联网技术圈互联网技术圈...
在本教程中,我将向您介绍如何在Linux中运行Windows应用程序和游戏.您可能知道,Microsoft Windows使用DirectX来运行游戏.因此,在Windows平台上玩一些需要Direc ...
android phone win10下载,微软Your Phone新功能：可在Win 10 PC 上运行 Android 应用程序...
今年 8 月,微软发布了一项 Your Phone 重大功能,可以在 Windows 10 PC 上运行 Android 应用程序.该功能主要通过 Your Phone 应用将 Android 应用投 ...

在Hadoop Yarn上运行pyspark-summit程序

在Hadoop Yarn上运行pyspark-summit程序

在Hadoop Yarn上运行pyspark-summit程序相关推荐

最新文章

热门文章