文章转自：https://blog.csdn.net/lizongti/article/details/99674153

目录

环境准备

依赖

安装Docker

Local模式（without Docker）

安装JDK

安装Scala

安装Spark

测试

Standalone 模式（without Docker）

更改hostname

更改配置

ssh免密登录

关闭防火墙

启动Spark集群

访问集群web

测试

Local模式（with Docker）

Docker 安装

测试

Standalone模式（with Docker）

创建overlay网络

启动集群

测试

Docker Stack部署

Docker Stack测试

环境准备

依赖

CentOS7.6

安装Docker

参照安装（点击）

Local模式（without Docker）

写这篇文章的时候，spark的最新版本是2.4.3，而此时的2.4.3的一些配套库并没有更新（比如mongodb-spark-connector），这里参照BigDataEurope选择Spark2.4.1 + Scala2.11 + JDK1.8。

安装JDK

去官网上下载1.8版本的tar.gz ，如果使用yum安装或者下载rpm包安装，则会缺少Scala2.11需要的部分文件。

tar xf jdk-8u221-linux-x64.tar -C /usr/lib/jvm
rm -rf /usr/bin/java
ln -s /usr/lib/jvm/jdk1.8.0_221/bin/java /usr/bin/java

编辑文件

vim /etc/profile.d/java.sh

添加

export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_221
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=${JAVA_HOME}/lib:${JRE_HOME}/lib:$CLASSPATH
export PATH=${JAVA_HOME}/bin:$PATH

然后使环境变量生效

source /etc/profile

执行以下命令检查环境变量

[root@vm1 bin]# echo $JAVA_HOME
/usr/lib/jvm/jdk1.8.0_221
[root@vm1 bin]# echo $JAVA_HOME
/usr/lib/jvm/jdk1.8.0_221

安装Scala

在官网拷贝rpm下载链接

wget --no-check-certificate https://downloads.lightbend.com/scala/2.11.12/scala-2.11.12.rpm

rpm安装

rpm -ivh scala-2.11.12.rpm

寻找SCALA_HOME

[root@vm1 /]# ls -lrt `which scala`
lrwxrwxrwx. 1 root root 24 Aug 16 01:11 /usr/bin/scala -> ../share/scala/bin/scala

编辑文件

vim /etc/profile.d/scala.sh

添加

export SCALA_HOME=/usr/share/scala
export PATH=${SCALA_HOME}/bin:$PATH

然后使环境变量生效

source /etc/profile

执行以下命令检查环境变量

[root@vm1 bin]# echo $SCALA_HOME
/usr/share/scala
[root@vm1 bin]# scala -version
Scala code runner version 2.11.12 -- Copyright 2002-2017, LAMP/EPFL

安装Spark

在官网下载页根据自己的需求寻找对应的版本，这里选择spark-2.4.1-bin-hadoop2.7.tgz ，默认使用Scala2.11且为hadoop2.7版本进行预构建（pre-build），spark启动需要依赖hadoop的一些库（ org/slf4j/Logger）。

wget --no-check-certificate https://archive.apache.org/dist/spark/spark-2.4.1/spark-2.4.1-bin-hadoop2.7.tgz
tar xf spark-2.4.1-bin-hadoop2.7.tgz -C /usr/share
ln -s /usr/share/spark-2.4.1-bin-hadoop2.7 /usr/share/spark

编辑文件

vim /etc/profile.d/spark.sh

添加

export SPARK_HOME=/usr/share/spark
export PATH=${SPARK_HOME}/bin:$PATH

然后使环境变量生效

source /etc/profile

执行以下命令检查环境变量

[root@vm1 bin]# echo $SPARK_HOME
/usr/share/spark
[root@vm1 bin]# spark-shell -version
19/08/16 03:46:59 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://vm1:4040
Spark context available as 'sc' (master = local[*], app id = local-1565941626509).
Spark session available as 'spark'.
Welcome to____              __/ __/__  ___ _____/ /___\ \/ _ \/ _ `/ __/  '_//___/ .__/\_,_/_/ /_/\_\   version 2.4.1/_/Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_221)
Type in expressions to have them evaluated.
Type :help for more information.scala>

这里出现WARN NativeCodeLoader并不影响运行，原因是系统glibc（2.17）的版本和hadoop（2.14）期望的版本不一致。

测试

local 类型分为 local(等同于local[1])、local[k](k为具体值)，local[*](*为cpu core的数量)

spark-submit --class org.apache.spark.examples.SparkPi --master local[*] /usr/share/spark/examples/jars/spark-examples_2.11-2.4.1.jar

运行结果

19/08/16 04:07:06 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/08/16 04:07:06 INFO SparkContext: Running Spark version 2.4.1
19/08/16 04:07:06 INFO SparkContext: Submitted application: Spark Pi
19/08/16 04:07:06 INFO SecurityManager: Changing view acls to: root
19/08/16 04:07:06 INFO SecurityManager: Changing modify acls to: root
19/08/16 04:07:06 INFO SecurityManager: Changing view acls groups to:
19/08/16 04:07:06 INFO SecurityManager: Changing modify acls groups to:
19/08/16 04:07:06 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
19/08/16 04:07:06 INFO Utils: Successfully started service 'sparkDriver' on port 43859.
19/08/16 04:07:06 INFO SparkEnv: Registering MapOutputTracker
19/08/16 04:07:07 INFO SparkEnv: Registering BlockManagerMaster
19/08/16 04:07:07 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
19/08/16 04:07:07 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
19/08/16 04:07:07 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-2dc555a4-5f5a-4dea-99fd-8e791402392d
19/08/16 04:07:07 INFO MemoryStore: MemoryStore started with capacity 366.3 MB
19/08/16 04:07:07 INFO SparkEnv: Registering OutputCommitCoordinator
19/08/16 04:07:07 INFO Utils: Successfully started service 'SparkUI' on port 4040.
19/08/16 04:07:07 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://vm1:4040
19/08/16 04:07:07 INFO SparkContext: Added JAR file:/usr/share/spark/examples/jars/spark-examples_2.11-2.4.1.jar at spark://vm1:43859/jars/spark-examples_2.11-2.4.1.jar with timestamp 1565942827297
19/08/16 04:07:07 INFO Executor: Starting executor ID driver on host localhost
19/08/16 04:07:07 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 46740.
19/08/16 04:07:07 INFO NettyBlockTransferService: Server created on vm1:46740
19/08/16 04:07:07 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
19/08/16 04:07:07 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, vm1, 46740, None)
19/08/16 04:07:07 INFO BlockManagerMasterEndpoint: Registering block manager vm1:46740 with 366.3 MB RAM, BlockManagerId(driver, vm1, 46740, None)
19/08/16 04:07:07 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, vm1, 46740, None)
19/08/16 04:07:07 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, vm1, 46740, None)
19/08/16 04:07:07 INFO SparkContext: Starting job: reduce at SparkPi.scala:38
19/08/16 04:07:08 INFO DAGScheduler: Got job 0 (reduce at SparkPi.scala:38) with 2 output partitions
19/08/16 04:07:08 INFO DAGScheduler: Final stage: ResultStage 0 (reduce at SparkPi.scala:38)
19/08/16 04:07:08 INFO DAGScheduler: Parents of final stage: List()
19/08/16 04:07:08 INFO DAGScheduler: Missing parents: List()
19/08/16 04:07:08 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has no missing parents
19/08/16 04:07:08 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 1936.0 B, free 366.3 MB)
19/08/16 04:07:08 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1256.0 B, free 366.3 MB)
19/08/16 04:07:08 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on vm1:46740 (size: 1256.0 B, free: 366.3 MB)
19/08/16 04:07:08 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1161
19/08/16 04:07:08 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1))
19/08/16 04:07:08 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
19/08/16 04:07:08 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 7866 bytes)
19/08/16 04:07:08 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, executor driver, partition 1, PROCESS_LOCAL, 7866 bytes)
19/08/16 04:07:08 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
19/08/16 04:07:08 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
19/08/16 04:07:08 INFO Executor: Fetching spark://vm1:43859/jars/spark-examples_2.11-2.4.1.jar with timestamp 1565942827297
19/08/16 04:07:08 INFO TransportClientFactory: Successfully created connection to vm1/172.169.18.28:43859 after 31 ms (0 ms spent in bootstraps)
19/08/16 04:07:08 INFO Utils: Fetching spark://vm1:43859/jars/spark-examples_2.11-2.4.1.jar to /tmp/spark-b22cc6ec-b16c-4c1c-8528-0643c065dd39/userFiles-46cb83a1-095a-4860-bea0-538b424b7608/fetchFileTemp3334060304603096780.tmp
19/08/16 04:07:08 INFO Executor: Adding file:/tmp/spark-b22cc6ec-b16c-4c1c-8528-0643c065dd39/userFiles-46cb83a1-095a-4860-bea0-538b424b7608/spark-examples_2.11-2.4.1.jar to class loader
19/08/16 04:07:08 INFO Executor: Finished task 1.0 in stage 0.0 (TID 1). 824 bytes result sent to driver
19/08/16 04:07:08 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 824 bytes result sent to driver
19/08/16 04:07:08 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 250 ms on localhost (executor driver) (1/2)
19/08/16 04:07:08 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 236 ms on localhost (executor driver) (2/2)
19/08/16 04:07:08 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
19/08/16 04:07:08 INFO DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) finished in 0.463 s
19/08/16 04:07:08 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 0.572216 s
Pi is roughly 3.143915719578598
19/08/16 04:07:08 INFO SparkUI: Stopped Spark web UI at http://vm1:4040
19/08/16 04:07:08 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
19/08/16 04:07:08 INFO MemoryStore: MemoryStore cleared
19/08/16 04:07:08 INFO BlockManager: BlockManager stopped
19/08/16 04:07:08 INFO BlockManagerMaster: BlockManagerMaster stopped
19/08/16 04:07:08 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
19/08/16 04:07:08 INFO SparkContext: Successfully stopped SparkContext
19/08/16 04:07:08 INFO ShutdownHookManager: Shutdown hook called
19/08/16 04:07:08 INFO ShutdownHookManager: Deleting directory /tmp/spark-b22cc6ec-b16c-4c1c-8528-0643c065dd39
19/08/16 04:07:08 INFO ShutdownHookManager: Deleting directory /tmp/spark-07babbac-1aff-4900-968f-596d57454e1b

Standalone 模式（without Docker）

部署三台机器vm1, vm2，vm3，基础环境和Local 模式一样，安装JDK1.8 + Scala2.11 + Spark2.7。

更改hostname

三台机器上分别设置hostname

#vm1
hostnamectl set-hostname vm1
#vm2
hostnamectl set-hostname vm2
#vm3
hostnamectl set-hostname vm3

更改配置

在vm1上修改配置
新增文件spark-env.sh,设置主节点及配置。

vi /usr/share/spark/conf/spark-env.sh

增加以下内容

#!/usr/bin/env bash
export SPARK_MASTER_HOST=vm1

新增文件slaves，增加从节点。

vi /usr/share/spark/conf/slaves

增加以下内容

vm2
vm3

拷贝配置到vm2和vm3上

yum install -y sshpass
sshpass -p xxxxx scp /usr/share/spark/conf/* root@vm3:/usr/share/spark/conf/

ssh免密登录

slave从节点需要从主节点上启动，为了简化启动过程，最好设置vm1到vm2和vm3的免密登录。
参照这里

关闭防火墙

systemctl stop firewalld
systemctl stop iptables

也可以不关闭防火墙，根据自己的实际配置情况来设置防火墙策略。

启动Spark集群

先启动主节点

sh /usr/share/spark/sbin/start-master.sh

显示

starting org.apache.spark.deploy.master.Master, logging to /usr/share/spark/logs/spark-root-org.apache.spark.deploy.master.Master-1-vm1.out

再启动从节点

sh /usr/share/spark/sbin/start-slaves.sh

显示

vm2: starting org.apache.spark.deploy.worker.Worker, logging to /usr/share/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-vm2.out
vm3: starting org.apache.spark.deploy.worker.Worker, logging to /usr/share/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-vm3.out

检查master启动日志

[root@vm1 logs]# tail -f /usr/share/spark/logs/spark-root-org.apache.spark.deploy.master.Master-1-vm1.out
19/08/17 04:30:07 INFO SecurityManager: Changing modify acls to: root
19/08/17 04:30:07 INFO SecurityManager: Changing view acls groups to:
19/08/17 04:30:07 INFO SecurityManager: Changing modify acls groups to:
19/08/17 04:30:07 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
19/08/17 04:30:07 INFO Utils: Successfully started service 'sparkMaster' on port 7077.
19/08/17 04:30:07 INFO Master: Starting Spark master at spark://vm1:7077
19/08/17 04:30:07 INFO Master: Running Spark version 2.4.1
19/08/17 04:30:08 INFO Utils: Successfully started service 'MasterUI' on port 8080.
19/08/17 04:30:08 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://vm1:8080
19/08/17 04:30:08 INFO Master: I have been elected leader! New state: ALIVE

检查slave vm2启动日志

[root@vm2 logs]# tail -f /usr/share/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-vm2.out
19/08/17 05:18:48 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
19/08/17 05:18:49 INFO Utils: Successfully started service 'sparkWorker' on port 44660.
19/08/17 05:18:49 INFO Worker: Starting Spark worker 172.169.18.60:44660 with 2 cores, 2.7 GB RAM
19/08/17 05:18:49 INFO Worker: Running Spark version 2.4.1
19/08/17 05:18:49 INFO Worker: Spark home: /usr/share/spark
19/08/17 05:18:49 INFO Utils: Successfully started service 'WorkerUI' on port 8081.
19/08/17 05:18:49 INFO WorkerWebUI: Bound WorkerWebUI to 0.0.0.0, and started at http://vm2:8081
19/08/17 05:18:49 INFO Worker: Connecting to master vm1:7077...
19/08/17 05:18:49 INFO TransportClientFactory: Successfully created connection to vm1/172.169.18.28:7077 after 44 ms (0 ms spent in bootstraps)
19/08/17 05:18:49 INFO Worker: Successfully registered with master spark://vm1:7077

检查slave vm3启动日志

[root@vm3 logs]# tail -f /usr/share/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-vm3.out
19/08/17 05:18:48 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
19/08/17 05:18:49 INFO Utils: Successfully started service 'sparkWorker' on port 42186.
19/08/17 05:18:49 INFO Worker: Starting Spark worker 172.169.18.2:42186 with 2 cores, 2.7 GB RAM
19/08/17 05:18:49 INFO Worker: Running Spark version 2.4.1
19/08/17 05:18:49 INFO Worker: Spark home: /usr/share/spark
19/08/17 05:18:49 INFO Utils: Successfully started service 'WorkerUI' on port 8081.
19/08/17 05:18:49 INFO WorkerWebUI: Bound WorkerWebUI to 0.0.0.0, and started at http://vm3:8081
19/08/17 05:18:49 INFO Worker: Connecting to master vm1:7077...
19/08/17 05:18:49 INFO TransportClientFactory: Successfully created connection to vm1/172.169.18.28:7077 after 44 ms (0 ms spent in bootstraps)
19/08/17 05:18:49 INFO Worker: Successfully registered with master spark://vm1:7077

访问集群web

默认是8080端口

测试

standalone分为client和cluster两种模式，具体原理另篇再说明。在client模式下，一台机器提交多个application,比如100个,会造成网卡激增问题。在cluster模式下,启动多个application,driver会分散在worker中,解决了网卡激增问题。
不填写deploy-mode情况默认使用client模式。
在vm1上执行

spark-submit --class org.apache.spark.examples.SparkPi --master spark://vm1:7077 --deploy-mode client /usr/share/spark/examples/jars/spark-examples_2.11-2.4.1.jar 10

执行结果

19/08/17 05:34:15 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/08/17 05:34:15 INFO SparkContext: Running Spark version 2.4.1
19/08/17 05:34:15 INFO SparkContext: Submitted application: Spark Pi
19/08/17 05:34:15 INFO SecurityManager: Changing view acls to: root
19/08/17 05:34:15 INFO SecurityManager: Changing modify acls to: root
19/08/17 05:34:15 INFO SecurityManager: Changing view acls groups to:
19/08/17 05:34:15 INFO SecurityManager: Changing modify acls groups to:
19/08/17 05:34:15 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
19/08/17 05:34:15 INFO Utils: Successfully started service 'sparkDriver' on port 44103.
19/08/17 05:34:15 INFO SparkEnv: Registering MapOutputTracker
19/08/17 05:34:15 INFO SparkEnv: Registering BlockManagerMaster
19/08/17 05:34:15 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
19/08/17 05:34:15 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
19/08/17 05:34:15 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-f54c2372-e6e3-43bd-bc49-3b94206691f1
19/08/17 05:34:15 INFO MemoryStore: MemoryStore started with capacity 366.3 MB
19/08/17 05:34:15 INFO SparkEnv: Registering OutputCommitCoordinator
19/08/17 05:34:16 INFO Utils: Successfully started service 'SparkUI' on port 4040.
19/08/17 05:34:16 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://vm1:4040
19/08/17 05:34:16 INFO SparkContext: Added JAR file:/usr/share/spark/examples/jars/spark-examples_2.11-2.4.1.jar at spark://vm1:44103/jars/spark-examples_2.11-2.4.1.jar with timestamp 1566034456184
19/08/17 05:34:16 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://vm1:7077...
19/08/17 05:34:16 INFO TransportClientFactory: Successfully created connection to vm1/172.169.18.28:7077 after 43 ms (0 ms spent in bootstraps)
19/08/17 05:34:16 INFO StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20190817053416-0002
19/08/17 05:34:16 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190817053416-0002/0 on worker-20190817051849-172.169.18.2-42186 (172.169.18.2:42186) with 2 core(s)
19/08/17 05:34:16 INFO StandaloneSchedulerBackend: Granted executor ID app-20190817053416-0002/0 on hostPort 172.169.18.2:42186 with 2 core(s), 1024.0 MB RAM
19/08/17 05:34:16 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190817053416-0002/1 on worker-20190817051849-172.169.18.60-44660 (172.169.18.60:44660) with 2 core(s)
19/08/17 05:34:16 INFO StandaloneSchedulerBackend: Granted executor ID app-20190817053416-0002/1 on hostPort 172.169.18.60:44660 with 2 core(s), 1024.0 MB RAM
19/08/17 05:34:16 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190817053416-0002/0 is now RUNNING
19/08/17 05:34:16 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190817053416-0002/1 is now RUNNING
19/08/17 05:34:16 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 36144.
19/08/17 05:34:16 INFO NettyBlockTransferService: Server created on vm1:36144
19/08/17 05:34:16 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
19/08/17 05:34:16 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, vm1, 36144, None)
19/08/17 05:34:16 INFO BlockManagerMasterEndpoint: Registering block manager vm1:36144 with 366.3 MB RAM, BlockManagerId(driver, vm1, 36144, None)
19/08/17 05:34:16 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, vm1, 36144, None)
19/08/17 05:34:16 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, vm1, 36144, None)
19/08/17 05:34:16 INFO StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
19/08/17 05:34:17 INFO SparkContext: Starting job: reduce at SparkPi.scala:38
19/08/17 05:34:17 INFO DAGScheduler: Got job 0 (reduce at SparkPi.scala:38) with 3 output partitions
19/08/17 05:34:17 INFO DAGScheduler: Final stage: ResultStage 0 (reduce at SparkPi.scala:38)
19/08/17 05:34:17 INFO DAGScheduler: Parents of final stage: List()
19/08/17 05:34:17 INFO DAGScheduler: Missing parents: List()
19/08/17 05:34:17 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has no missing parents
19/08/17 05:34:17 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 1936.0 B, free 366.3 MB)
19/08/17 05:34:17 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1256.0 B, free 366.3 MB)
19/08/17 05:34:17 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on vm1:36144 (size: 1256.0 B, free: 366.3 MB)
19/08/17 05:34:17 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1161
19/08/17 05:34:17 INFO DAGScheduler: Submitting 3 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1, 2))
19/08/17 05:34:17 INFO TaskSchedulerImpl: Adding task set 0.0 with 3 tasks
19/08/17 05:34:19 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.169.18.60:34014) with ID 1
19/08/17 05:34:19 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, 172.169.18.60, executor 1, partition 0, PROCESS_LOCAL, 7870 bytes)
19/08/17 05:34:19 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, 172.169.18.60, executor 1, partition 1, PROCESS_LOCAL, 7870 bytes)
19/08/17 05:34:19 INFO BlockManagerMasterEndpoint: Registering block manager 172.169.18.60:37821 with 366.3 MB RAM, BlockManagerId(1, 172.169.18.60, 37821, None)
19/08/17 05:34:19 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.169.18.2:52382) with ID 0
19/08/17 05:34:19 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, 172.169.18.2, executor 0, partition 2, PROCESS_LOCAL, 7870 bytes)
19/08/17 05:34:19 INFO BlockManagerMasterEndpoint: Registering block manager 172.169.18.2:40495 with 366.3 MB RAM, BlockManagerId(0, 172.169.18.2, 40495, None)
19/08/17 05:34:19 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 172.169.18.60:37821 (size: 1256.0 B, free: 366.3 MB)
19/08/17 05:34:19 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 680 ms on 172.169.18.60 (executor 1) (1/3)
19/08/17 05:34:19 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 720 ms on 172.169.18.60 (executor 1) (2/3)
19/08/17 05:34:21 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 172.169.18.2:40495 (size: 1256.0 B, free: 366.3 MB)
19/08/17 05:34:21 INFO TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 2299 ms on 172.169.18.2 (executor 0) (3/3)
19/08/17 05:34:21 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
19/08/17 05:34:21 INFO DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) finished in 4.343 s
19/08/17 05:34:21 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 4.512565 s
Pi is roughly 3.1425838086126956
19/08/17 05:34:21 INFO SparkUI: Stopped Spark web UI at http://vm1:4040
19/08/17 05:34:21 INFO StandaloneSchedulerBackend: Shutting down all executors
19/08/17 05:34:21 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
19/08/17 05:34:21 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
19/08/17 05:34:21 INFO MemoryStore: MemoryStore cleared
19/08/17 05:34:21 INFO BlockManager: BlockManager stopped
19/08/17 05:34:21 INFO BlockManagerMaster: BlockManagerMaster stopped
19/08/17 05:34:21 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
19/08/17 05:34:21 INFO SparkContext: Successfully stopped SparkContext
19/08/17 05:34:21 INFO ShutdownHookManager: Shutdown hook called
19/08/17 05:34:21 INFO ShutdownHookManager: Deleting directory /tmp/spark-d7196560-2b3d-4766-8b71-3e47260f4d0b
19/08/17 05:34:21 INFO ShutdownHookManager: Deleting directory /tmp/spark-f804e08d-7449-4c29-94db-55c98673cb2f
[root@vm1 logs]# spark-submit --class org.apache.spark.examples.SparkPi --master spark://vm1:7077 /usr/share/spark/examples/jars/spark-examples_2.11-2.4.1.jar 10
19/08/17 05:34:28 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/08/17 05:34:28 INFO SparkContext: Running Spark version 2.4.1
19/08/17 05:34:28 INFO SparkContext: Submitted application: Spark Pi
19/08/17 05:34:28 INFO SecurityManager: Changing view acls to: root
19/08/17 05:34:28 INFO SecurityManager: Changing modify acls to: root
19/08/17 05:34:28 INFO SecurityManager: Changing view acls groups to:
19/08/17 05:34:28 INFO SecurityManager: Changing modify acls groups to:
19/08/17 05:34:28 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
19/08/17 05:34:29 INFO Utils: Successfully started service 'sparkDriver' on port 33704.
19/08/17 05:34:29 INFO SparkEnv: Registering MapOutputTracker
19/08/17 05:34:29 INFO SparkEnv: Registering BlockManagerMaster
19/08/17 05:34:29 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
19/08/17 05:34:29 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
19/08/17 05:34:29 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-162dd9ae-7a06-4b3a-b353-fe42a9d6c7a0
19/08/17 05:34:29 INFO MemoryStore: MemoryStore started with capacity 366.3 MB
19/08/17 05:34:29 INFO SparkEnv: Registering OutputCommitCoordinator
19/08/17 05:34:29 INFO Utils: Successfully started service 'SparkUI' on port 4040.
19/08/17 05:34:29 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://vm1:4040
19/08/17 05:34:29 INFO SparkContext: Added JAR file:/usr/share/spark/examples/jars/spark-examples_2.11-2.4.1.jar at spark://vm1:33704/jars/spark-examples_2.11-2.4.1.jar with timestamp 1566034469382
19/08/17 05:34:29 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://vm1:7077...
19/08/17 05:34:29 INFO TransportClientFactory: Successfully created connection to vm1/172.169.18.28:7077 after 41 ms (0 ms spent in bootstraps)
19/08/17 05:34:29 INFO StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20190817053429-0003
19/08/17 05:34:29 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 45611.
19/08/17 05:34:29 INFO NettyBlockTransferService: Server created on vm1:45611
19/08/17 05:34:29 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
19/08/17 05:34:29 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190817053429-0003/0 on worker-20190817051849-172.169.18.2-42186 (172.169.18.2:42186) with 2 core(s)
19/08/17 05:34:29 INFO StandaloneSchedulerBackend: Granted executor ID app-20190817053429-0003/0 on hostPort 172.169.18.2:42186 with 2 core(s), 1024.0 MB RAM
19/08/17 05:34:29 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190817053429-0003/1 on worker-20190817051849-172.169.18.60-44660 (172.169.18.60:44660) with 2 core(s)
19/08/17 05:34:29 INFO StandaloneSchedulerBackend: Granted executor ID app-20190817053429-0003/1 on hostPort 172.169.18.60:44660 with 2 core(s), 1024.0 MB RAM
19/08/17 05:34:29 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190817053429-0003/1 is now RUNNING
19/08/17 05:34:29 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190817053429-0003/0 is now RUNNING
19/08/17 05:34:29 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, vm1, 45611, None)
19/08/17 05:34:29 INFO BlockManagerMasterEndpoint: Registering block manager vm1:45611 with 366.3 MB RAM, BlockManagerId(driver, vm1, 45611, None)
19/08/17 05:34:29 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, vm1, 45611, None)
19/08/17 05:34:29 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, vm1, 45611, None)
19/08/17 05:34:30 INFO StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
19/08/17 05:34:30 INFO SparkContext: Starting job: reduce at SparkPi.scala:38
19/08/17 05:34:30 INFO DAGScheduler: Got job 0 (reduce at SparkPi.scala:38) with 10 output partitions
19/08/17 05:34:30 INFO DAGScheduler: Final stage: ResultStage 0 (reduce at SparkPi.scala:38)
19/08/17 05:34:30 INFO DAGScheduler: Parents of final stage: List()
19/08/17 05:34:30 INFO DAGScheduler: Missing parents: List()
19/08/17 05:34:30 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has no missing parents
19/08/17 05:34:30 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 1936.0 B, free 366.3 MB)
19/08/17 05:34:30 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1256.0 B, free 366.3 MB)
19/08/17 05:34:30 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on vm1:45611 (size: 1256.0 B, free: 366.3 MB)
19/08/17 05:34:30 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1161
19/08/17 05:34:30 INFO DAGScheduler: Submitting 10 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9))
19/08/17 05:34:30 INFO TaskSchedulerImpl: Adding task set 0.0 with 10 tasks
19/08/17 05:34:32 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.169.18.60:57348) with ID 1
19/08/17 05:34:32 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, 172.169.18.60, executor 1, partition 0, PROCESS_LOCAL, 7870 bytes)
19/08/17 05:34:32 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, 172.169.18.60, executor 1, partition 1, PROCESS_LOCAL, 7870 bytes)
19/08/17 05:34:32 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.169.18.2:40198) with ID 0
19/08/17 05:34:32 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, 172.169.18.2, executor 0, partition 2, PROCESS_LOCAL, 7870 bytes)
19/08/17 05:34:32 INFO TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, 172.169.18.2, executor 0, partition 3, PROCESS_LOCAL, 7870 bytes)
19/08/17 05:34:32 INFO BlockManagerMasterEndpoint: Registering block manager 172.169.18.2:36636 with 366.3 MB RAM, BlockManagerId(0, 172.169.18.2, 36636, None)
19/08/17 05:34:32 INFO BlockManagerMasterEndpoint: Registering block manager 172.169.18.60:34541 with 366.3 MB RAM, BlockManagerId(1, 172.169.18.60, 34541, None)
19/08/17 05:34:33 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 172.169.18.60:34541 (size: 1256.0 B, free: 366.3 MB)
19/08/17 05:34:33 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 172.169.18.2:36636 (size: 1256.0 B, free: 366.3 MB)
19/08/17 05:34:33 INFO TaskSetManager: Starting task 4.0 in stage 0.0 (TID 4, 172.169.18.2, executor 0, partition 4, PROCESS_LOCAL, 7870 bytes)
19/08/17 05:34:33 INFO TaskSetManager: Starting task 5.0 in stage 0.0 (TID 5, 172.169.18.2, executor 0, partition 5, PROCESS_LOCAL, 7870 bytes)
19/08/17 05:34:33 INFO TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 991 ms on 172.169.18.2 (executor 0) (1/10)
19/08/17 05:34:33 INFO TaskSetManager: Starting task 6.0 in stage 0.0 (TID 6, 172.169.18.60, executor 1, partition 6, PROCESS_LOCAL, 7870 bytes)
19/08/17 05:34:33 INFO TaskSetManager: Starting task 7.0 in stage 0.0 (TID 7, 172.169.18.60, executor 1, partition 7, PROCESS_LOCAL, 7870 bytes)
19/08/17 05:34:33 INFO TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 1005 ms on 172.169.18.2 (executor 0) (2/10)
19/08/17 05:34:33 INFO TaskSetManager: Starting task 8.0 in stage 0.0 (TID 8, 172.169.18.2, executor 0, partition 8, PROCESS_LOCAL, 7870 bytes)
19/08/17 05:34:33 INFO TaskSetManager: Finished task 4.0 in stage 0.0 (TID 4) in 34 ms on 172.169.18.2 (executor 0) (3/10)
19/08/17 05:34:33 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 1055 ms on 172.169.18.60 (executor 1) (4/10)
19/08/17 05:34:33 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 1025 ms on 172.169.18.60 (executor 1) (5/10)
19/08/17 05:34:33 INFO TaskSetManager: Starting task 9.0 in stage 0.0 (TID 9, 172.169.18.2, executor 0, partition 9, PROCESS_LOCAL, 7870 bytes)
19/08/17 05:34:33 INFO TaskSetManager: Finished task 5.0 in stage 0.0 (TID 5) in 70 ms on 172.169.18.2 (executor 0) (6/10)
19/08/17 05:34:33 INFO TaskSetManager: Finished task 7.0 in stage 0.0 (TID 7) in 72 ms on 172.169.18.60 (executor 1) (7/10)
19/08/17 05:34:33 INFO TaskSetManager: Finished task 6.0 in stage 0.0 (TID 6) in 76 ms on 172.169.18.60 (executor 1) (8/10)
19/08/17 05:34:33 INFO TaskSetManager: Finished task 8.0 in stage 0.0 (TID 8) in 73 ms on 172.169.18.2 (executor 0) (9/10)
19/08/17 05:34:33 INFO TaskSetManager: Finished task 9.0 in stage 0.0 (TID 9) in 39 ms on 172.169.18.2 (executor 0) (10/10)
19/08/17 05:34:33 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
19/08/17 05:34:33 INFO DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) finished in 2.803 s
19/08/17 05:34:33 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 2.965956 s
Pi is roughly 3.141123141123141
19/08/17 05:34:33 INFO SparkUI: Stopped Spark web UI at http://vm1:4040
19/08/17 05:34:33 INFO StandaloneSchedulerBackend: Shutting down all executors
19/08/17 05:34:33 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
19/08/17 05:34:33 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
19/08/17 05:34:33 INFO MemoryStore: MemoryStore cleared
19/08/17 05:34:33 INFO BlockManager: BlockManager stopped
19/08/17 05:34:33 INFO BlockManagerMaster: BlockManagerMaster stopped
19/08/17 05:34:33 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
19/08/17 05:34:33 INFO SparkContext: Successfully stopped SparkContext
19/08/17 05:34:33 INFO ShutdownHookManager: Shutdown hook called
19/08/17 05:34:33 INFO ShutdownHookManager: Deleting directory /tmp/spark-790e464a-289b-4d6c-8607-20f65e783937
19/08/17 05:34:33 INFO ShutdownHookManager: Deleting directory /tmp/spark-e66f306d-4c1f-44a8-92e7-85251aba5fa1

使用cluster模式运行

spark-submit --class org.apache.spark.examples.SparkPi --master spark://vm1:7077 --deploy-mode cluster /usr/share/spark/examples/jars/spark-examples_2.11-2.4.1.jar 10

显示

log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/08/17 05:48:53 INFO SecurityManager: Changing view acls to: root
19/08/17 05:48:53 INFO SecurityManager: Changing modify acls to: root
19/08/17 05:48:53 INFO SecurityManager: Changing view acls groups to:
19/08/17 05:48:53 INFO SecurityManager: Changing modify acls groups to:
19/08/17 05:48:53 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
19/08/17 05:48:53 INFO Utils: Successfully started service 'driverClient' on port 46251.
19/08/17 05:48:53 INFO TransportClientFactory: Successfully created connection to vm1/172.169.18.28:7077 after 28 ms (0 ms spent in bootstraps)
19/08/17 05:48:53 INFO ClientEndpoint: Driver successfully submitted as driver-20190817054853-0001
19/08/17 05:48:53 INFO ClientEndpoint: ... waiting before polling master for driver state
19/08/17 05:48:58 INFO ClientEndpoint: ... polling master for driver state
19/08/17 05:48:58 INFO ClientEndpoint: State of driver-20190817054853-0001 is RUNNING
19/08/17 05:48:58 INFO ClientEndpoint: Driver running on 172.169.18.2:42186 (worker-20190817051849-172.169.18.2-42186)
19/08/17 05:48:58 INFO ShutdownHookManager: Shutdown hook called
19/08/17 05:48:58 INFO ShutdownHookManager: Deleting directory /tmp/spark-9a6e7757-b29f-48b7-bab2-1c3f89eda79a

可以看到结果很简短，具体操作日志可以在spark web端进行查看。

Local模式（with Docker）

Docker 安装

参照这里（点击）

测试

Big Data Europe是一个大数据的社区组织，重点讨论欧洲大数据项目产生的技术问题。我们使用它在Docker Hub上的发布的公共镜像。

docker run -it --rm bde2020/spark-base:2.4.1-hadoop2.7 /spark/bin/spark-submit --class org.apache.spark.examples.SparkPi --master local[*] /spark/examples/jars/spark-examples_2.11-2.4.1.jar 10

测试结果和非docker 模式下一致

Standalone模式（with Docker）

创建overlay网络

参照这里（点击）

docker network create overlay --driver=overlay --attachable

启动集群

再vm1，vm2，vm3上分别执行命令

#vm1
docker run --network overlay --name spark-master -h spark-master -e ENABLE_INIT_DAEMON=false -d bde2020/spark-master:2.4.1-hadoop2.7
#vm2
docker run --network overlay --name spark-worker-1 -h spark-worker-1 -e ENABLE_INIT_DAEMON=false -d bde2020/spark-worker:2.4.1-hadoop2.7
#vm3
docker run --network overlay --name spark-worker-2 -h spark-worker-2 -e ENABLE_INIT_DAEMON=false -d bde2020/spark-worker:2.4.1-hadoop2.7

spark-master 启动日志

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/08/20 09:32:25 INFO Master: Started daemon with process name: 12@spark-master
19/08/20 09:32:25 INFO SignalUtils: Registered signal handler for TERM
19/08/20 09:32:25 INFO SignalUtils: Registered signal handler for HUP
19/08/20 09:32:25 INFO SignalUtils: Registered signal handler for INT
19/08/20 09:32:26 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/08/20 09:32:26 INFO SecurityManager: Changing view acls to: root
19/08/20 09:32:26 INFO SecurityManager: Changing modify acls to: root
19/08/20 09:32:26 INFO SecurityManager: Changing view acls groups to:
19/08/20 09:32:26 INFO SecurityManager: Changing modify acls groups to:
19/08/20 09:32:26 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
19/08/20 09:32:26 INFO Utils: Successfully started service 'sparkMaster' on port 7077.
19/08/20 09:32:26 INFO Master: Starting Spark master at spark://spark-master:7077
19/08/20 09:32:26 INFO Master: Running Spark version 2.4.1
19/08/20 09:32:26 INFO Utils: Successfully started service 'MasterUI' on port 8080.
19/08/20 09:32:26 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://spark-master:8080
19/08/20 09:32:27 INFO Master: I have been elected leader! New state: ALIVE
19/08/20 09:32:33 INFO Master: Registering worker 10.0.0.4:35308 with 2 cores, 2.7 GB RAM
19/08/20 09:32:37 INFO Master: Registering worker 10.0.0.6:45378 with 2 cores, 2.7 GB RAM

可以看见两个worker加入了集群
spark-worker-1启动日志

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/08/20 09:32:31 INFO Worker: Started daemon with process name: 11@spark-worker-1
19/08/20 09:32:31 INFO SignalUtils: Registered signal handler for TERM
19/08/20 09:32:31 INFO SignalUtils: Registered signal handler for HUP
19/08/20 09:32:31 INFO SignalUtils: Registered signal handler for INT
19/08/20 09:32:32 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/08/20 09:32:32 INFO SecurityManager: Changing view acls to: root
19/08/20 09:32:32 INFO SecurityManager: Changing modify acls to: root
19/08/20 09:32:32 INFO SecurityManager: Changing view acls groups to:
19/08/20 09:32:32 INFO SecurityManager: Changing modify acls groups to:
19/08/20 09:32:32 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
19/08/20 09:32:32 INFO Utils: Successfully started service 'sparkWorker' on port 35308.
19/08/20 09:32:33 INFO Worker: Starting Spark worker 10.0.0.4:35308 with 2 cores, 2.7 GB RAM
19/08/20 09:32:33 INFO Worker: Running Spark version 2.4.1
19/08/20 09:32:33 INFO Worker: Spark home: /spark
19/08/20 09:32:33 INFO Utils: Successfully started service 'WorkerUI' on port 8081.
19/08/20 09:32:33 INFO WorkerWebUI: Bound WorkerWebUI to 0.0.0.0, and started at http://spark-worker-1:8081
19/08/20 09:32:33 INFO Worker: Connecting to master spark-master:7077...
19/08/20 09:32:33 INFO TransportClientFactory: Successfully created connection to spark-master/10.0.0.2:7077 after 47 ms (0 ms spent in bootstraps)
19/08/20 09:32:33 INFO Worker: Successfully registered with master spark://spark-master:7077

spark-worker-2启动日志

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/08/20 09:32:36 INFO Worker: Started daemon with process name: 11@spark-worker-2
19/08/20 09:32:36 INFO SignalUtils: Registered signal handler for TERM
19/08/20 09:32:36 INFO SignalUtils: Registered signal handler for HUP
19/08/20 09:32:36 INFO SignalUtils: Registered signal handler for INT
19/08/20 09:32:36 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/08/20 09:32:36 INFO SecurityManager: Changing view acls to: root
19/08/20 09:32:36 INFO SecurityManager: Changing modify acls to: root
19/08/20 09:32:36 INFO SecurityManager: Changing view acls groups to:
19/08/20 09:32:36 INFO SecurityManager: Changing modify acls groups to:
19/08/20 09:32:36 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
19/08/20 09:32:37 INFO Utils: Successfully started service 'sparkWorker' on port 45378.
19/08/20 09:32:37 INFO Worker: Starting Spark worker 10.0.0.6:45378 with 2 cores, 2.7 GB RAM
19/08/20 09:32:37 INFO Worker: Running Spark version 2.4.1
19/08/20 09:32:37 INFO Worker: Spark home: /spark
19/08/20 09:32:37 INFO Utils: Successfully started service 'WorkerUI' on port 8081.
19/08/20 09:32:37 INFO WorkerWebUI: Bound WorkerWebUI to 0.0.0.0, and started at http://spark-worker-2:8081
19/08/20 09:32:37 INFO Worker: Connecting to master spark-master:7077...
19/08/20 09:32:37 INFO TransportClientFactory: Successfully created connection to spark-master/10.0.0.2:7077 after 47 ms (0 ms spent in bootstraps)
19/08/20 09:32:38 INFO Worker: Successfully registered with master spark://spark-master:7077

测试

获取spark-submit的镜像，先看submit具体实现步骤

docker run  -it --rm bde2020/spark-submit:2.4.1-hadoop2.7 cat submit.sh

显示

#!/bin/bashexport SPARK_MASTER_URL=spark://${SPARK_MASTER_NAME}:${SPARK_MASTER_PORT}
export SPARK_HOME=/spark/wait-for-step.sh/execute-step.sh
if [ -f "${SPARK_APPLICATION_JAR_LOCATION}" ]; thenecho "Submit application ${SPARK_APPLICATION_JAR_LOCATION} with main class ${SPARK_APPLICATION_MAIN_CLASS} to Spark master ${SPARK_MASTER_URL}"echo "Passing arguments ${SPARK_APPLICATION_ARGS}"/spark/bin/spark-submit \--class ${SPARK_APPLICATION_MAIN_CLASS} \--master ${SPARK_MASTER_URL} \${SPARK_SUBMIT_ARGS} \${SPARK_APPLICATION_JAR_LOCATION} ${SPARK_APPLICATION_ARGS}
elseif [ -f "${SPARK_APPLICATION_PYTHON_LOCATION}" ]; thenecho "Submit application ${SPARK_APPLICATION_PYTHON_LOCATION} to Spark master ${SPARK_MASTER_URL}"echo "Passing arguments ${SPARK_APPLICATION_ARGS}"PYSPARK_PYTHON=python3 /spark/bin/spark-submit \--master ${SPARK_MASTER_URL} \${SPARK_SUBMIT_ARGS} \${SPARK_APPLICATION_PYTHON_LOCATION} ${SPARK_APPLICATION_ARGS}elseecho "Not recognized application."fi
fi
/finish-step.sh

可以看出 SPARK_APPLICATION_JAR_LOCATION 、 SPARK_APPLICATION_MAIN_CLASS、SPARK_MASTER_URL为必填参数，SPARK_SUBMIT_ARGS、SPARK_APPLICATION_ARGS为选填参数。

docker run --network overlay \
-e ENABLE_INIT_DAEMON=false \
-e SPARK_APPLICATION_JAR_LOCATION="/spark/examples/jars/spark-examples_2.11-2.4.1.jar" \
-e SPARK_APPLICATION_MAIN_CLASS="org.apache.spark.examples.SparkPi" \
-e SPARK_MASTER_URL="spark://vm1:7077" \
-e SPARK_SUBMIT_ARGS="--deploy-mode client" \
-e SPARK_APPLICATION_ARGS="10" \
-it --rm bde2020/spark-submit:2.4.1-hadoop2.7

测试日志类似Standalone模式。

Docker Stack部署

编写配置文件

version: '3.7'
services:spark-master:image: bde2020/spark-master:2.4.1-hadoop2.7hostname: spark-masternetworks:- overlayports:- 8080:8080- 7077:7077environment:- ENABLE_INIT_DAEMON=false- SPARK_MASTER_HOST=0.0.0.0- SPARK_MASTER_PORT=7077- SPARK_MASTER_WEBUI_PORT=8080volumes:- /etc/localtime:/etc/localtimedeploy:restart_policy:condition: on-failureplacement:constraints:- node.hostname==vm1spark-worker-1:image: bde2020/spark-worker:2.4.1-hadoop2.7hostname: spark-worker-1networks:- overlayports:- 8081:8081environment:- ENABLE_INIT_DAEMON=false- SPARK_WORKER_WEBUI_PORT=8081- SPARK_MASTER=spark://spark-master:7077volumes:- /etc/localtime:/etc/localtimedepends_on:- spark-masterdeploy:restart_policy:condition: on-failureplacement:constraints:- node.hostname==vm2spark-worker-2:image: bde2020/spark-worker:2.4.1-hadoop2.7hostname: spark-worker-2networks:- overlayports:- 8082:8082environment:- ENABLE_INIT_DAEMON=false- SPARK_WORKER_WEBUI_PORT=8082- SPARK_MASTER=spark://spark-master:7077volumes:- /etc/localtime:/etc/localtimedepends_on:- spark-masterdeploy:restart_policy:condition: on-failureplacement:constraints:- node.hostname==vm3
networks:overlay:driver: overlayattachable: true

启动

docker stack deploy -c spark.yaml spark

启动日志跟之前的集群一样

Docker Stack测试

docker run --network spark_overlay \
-e ENABLE_INIT_DAEMON=false \
-e SPARK_APPLICATION_JAR_LOCATION="/spark/examples/jars/spark-examples_2.11-2.4.1.jar" \
-e SPARK_APPLICATION_MAIN_CLASS="org.apache.spark.examples.SparkPi" \
-e SPARK_MASTER_URL="spark://vm1:7077" \
-e SPARK_SUBMIT_ARGS="--deploy-mode client" \
-e SPARK_APPLICATION_ARGS="10" \
-it --rm bde2020/spark-submit:2.4.1-hadoop2.7

Docker 搭建Spark集群相关推荐

  1. docker下，极速搭建spark集群(含hdfs集群)

    搭建spark和hdfs的集群环境会消耗一些时间和精力,处于学习和开发阶段的同学关注的是spark应用的开发,他们希望整个环境能快速搭建好,从而尽快投入编码和调试,今天咱们就借助docker,极速搭建 ...

  2. 使用Docker安装Spark集群(带有HDFS)

    本实验在CentOS 7中完成 第一部分:安装Docker 这一部分是安装Docker,如果机器中已经安装过Docker,可以直接跳过 [root@VM-48-22-centos ~]# system ...

  3. 如何基于Jupyter notebook搭建Spark集群开发环境

    摘要:本文介绍如何基于Jupyter notebook搭建Spark集群开发环境. 本文分享自华为云社区<基于Jupyter Notebook 搭建Spark集群开发环境>,作者:apr鹏 ...

  4. Mac 使用 docker 搭建 kafka 集群 + Zookeeper + kafka-manager

    Kafka 搭建: 建立Zookeeper容器: 这里我们用最简单的方式创建一个独立的Zookeeper节点,如果要考虑zookeeper的高可用,可以将其做成一个集群,最好是能有多台机器. $ do ...

  5. Docker搭建hadoop集群

    参考https://github.com/kiwenlau/hadoop-cluster-docker/blob/master/start-container.sh 因为之前在VMware上操作Had ...

  6. 用Docker搭建Elasticsearch集群

    用Docker搭建Elasticsearch集群 对于用Docker搭建分布式Elasticsearhc集群的一个介绍,以及一些实施中遇到问题的总结 搜索服务简述 结合业务的场景,在目前的商品体系需要 ...

  7. docker 添加端口映射_苟且偷生的程序猿没法摸鱼了，从docker搭建elasticsearch集群开始学习...

    最近刚换了公司,然后公司刚好使用的docker.而本人作为一个石锤的搬砖员,之前只知道搬砖们,对于这些东西确实没学过.今天在本地使用docker搭建elasticsearch集群,记录下过程,而且el ...

  8. Docker学习七：使用docker搭建Hadoop集群

    本博客简单分享了如何在Docker上搭建Hadoop集群,我的电脑是Ubuntu20,听同学说wsl2有些命令不对,所以建议在虚拟机里按照Ubuntu或者直接安装双系统吧 Docker学习一:Dock ...

  9. docker搭建pxc集群

    前言 随着mysql存储的数据量越来越大,mysql查询单表时的响应速度也会随之变慢,尤其是当单节点承载的数据量超出一定的范围后,比如单表超过2000万之后,查询响应速度会下降的很快,因此,一方面可以 ...

最新文章

  1. Silverlight3 Tools Download link
  2. 小程序获取StorageSync时候的坑
  3. rman学习笔记部分
  4. JS(JavaScript）的初了解3（更新中···）
  5. python语言语块句的标记_《自然语言处理理论与实战》
  6. php支持二级域名共享,PHP二级域名session共享方案
  7. HTML+CSS+JS实现 ❤️酷炫彩虹旋转隧道特效❤️
  8. 解决uploadify在Firefox下丢失session的问题
  9. fread函数 linux在哪,fread函数返回值
  10. 【Android】Android签名:pem和pk8转化为keystore,jks
  11. 10款超好用插件，助你玩转Chrome浏览器
  12. android获取悬浮窗权限,Android 悬浮窗权限校验
  13. 阿里云服务器部署（保姆级）
  14. ORA-20011: Approximate NDV failed: ORA-06564: object EXPDDDP does not exist
  15. 前端进行身份证验证（详细）
  16. NIVIDIA 硬解码学习4
  17. 验证手机号码是否正确
  18. ios图锁HQ应用项目源码
  19. RHCA回忆录---CL210介绍
  20. Java面试题中高级，nasdocker有啥好玩的

热门文章

  1. RecyclerView花样布局
  2. mysql mdl 锁_MySQL锁系列3 MDL锁
  3. 第6课：Scratchjr重复指令
  4. 【手拉手 带你准备电赛】原来你是这样的触摸屏（电阻触摸屏）
  5. 用 Python 分析了 6000 款 App，看看哪些神器你还没用过？
  6. java商城毕业设计 JavaWeb家具家居购物商城毕业设计（8）商品评论列表
  7. 四种常见的GPS北斗定位模块问题及解决办法
  8. Unity3D知识体系
  9. 如何获取MIPS汇编对应的机器码
  10. 攻破黑市之拿下吃鸡，DNF钓鱼站群（结局彩蛋）