Apache Hadoop 2.9.2 的YARN High Available 模式部署

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。

一.环境准备

1>.官方文档（http://hadoop.apache.org/docs/r2.9.2/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html）

2>.实验环境

　　环境在上一篇博客基础之上，参考链接：https://www.cnblogs.com/yinzhengjie/p/10720586.html。

二.修改核心配置

1>.修改yarn-site.xml

[root@node101.yinzhengjie.org.cn ~]# cat /yinzhengjie/softwares/hadoop-2.9.2/etc/hadoop/yarn-site.xml
<configuration><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property><!--启用resourcemanager ha--><property><name>yarn.resourcemanager.ha.enabled</name><value>true</value></property><!--声明两台resourcemanager的地址--><property><name>yarn.resourcemanager.cluster-id</name><value>yinzhengjie-yarn</value></property><property><name>yarn.resourcemanager.ha.rm-ids</name><value>resourcemanager1,resourcemanager2</value></property><property><name>yarn.resourcemanager.hostname.resourcemanager1</name><value>node101.yinzhengjie.org.cn</value></property><property><name>yarn.resourcemanager.hostname.resourcemanager2</name><value>node103.yinzhengjie.org.cn</value></property><property><name>yarn.resourcemanager.webapp.address.resourcemanager1</name><value>node101.yinzhengjie.org.cn:8088</value></property><property><name>yarn.resourcemanager.webapp.address.resourcemanager2</name><value>node103.yinzhengjie.org.cn:8088</value></property><!--指定zookeeper集群的地址--> <property><name>yarn.resourcemanager.zk-address</name><value>node101.yinzhengjie.org.cn:2181,node102.yinzhengjie.org.cn:2181,node101.yinzhengjie.org.cn:2181</value></property><!--启用自动恢复--> <property><name>yarn.resourcemanager.recovery.enabled</name><value>true</value></property><!--指定resourcemanager的状态信息存储在zookeeper集群--> <property><name>yarn.resourcemanager.store.class</name>     <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
</property></configuration>
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# cat /yinzhengjie/softwares/hadoop-2.9.2/etc/hadoop/yarn-site.xml

2>.将修改的配置文件同步到其他节点中

[root@node101.yinzhengjie.org.cn ~]# scp /yinzhengjie/softwares/hadoop-2.9.2/etc/hadoop/yarn-site.xml  node102.yinzhengjie.org.cn:/yinzhengjie/softwares/hadoop-2.9.2/etc/hadoop/
yarn-site.xml                                                                                                                                                                                                             100% 1566     2.8MB/s   00:00
[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# scp /yinzhengjie/softwares/hadoop-2.9.2/etc/hadoop/yarn-site.xml node102.yinzhengjie.org.cn:/yinzhengjie/softwares/hadoop-2.9.2/etc/hadoop/

[root@node101.yinzhengjie.org.cn ~]# scp /yinzhengjie/softwares/hadoop-2.9.2/etc/hadoop/yarn-site.xml  node103.yinzhengjie.org.cn:/yinzhengjie/softwares/hadoop-2.9.2/etc/hadoop/
yarn-site.xml                                                                                                                                                                                                             100% 1566     3.0MB/s   00:00
[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# scp /yinzhengjie/softwares/hadoop-2.9.2/etc/hadoop/yarn-site.xml node103.yinzhengjie.org.cn:/yinzhengjie/softwares/hadoop-2.9.2/etc/hadoop/

3>.启动hdfs集群

[root@node101.yinzhengjie.org.cn ~]# start-dfs.sh
Starting namenodes on [node101.yinzhengjie.org.cn node103.yinzhengjie.org.cn]
node101.yinzhengjie.org.cn: starting namenode, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/hadoop-root-namenode-node101.yinzhengjie.org.cn.out
node103.yinzhengjie.org.cn: starting namenode, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/hadoop-root-namenode-node103.yinzhengjie.org.cn.out
node103.yinzhengjie.org.cn: starting datanode, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/hadoop-root-datanode-node103.yinzhengjie.org.cn.out
node102.yinzhengjie.org.cn: starting datanode, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/hadoop-root-datanode-node102.yinzhengjie.org.cn.out
node101.yinzhengjie.org.cn: starting datanode, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/hadoop-root-datanode-node101.yinzhengjie.org.cn.out
Starting journal nodes [node103.yinzhengjie.org.cn node102.yinzhengjie.org.cn node101.yinzhengjie.org.cn]
node103.yinzhengjie.org.cn: starting journalnode, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/hadoop-root-journalnode-node103.yinzhengjie.org.cn.out
node102.yinzhengjie.org.cn: starting journalnode, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/hadoop-root-journalnode-node102.yinzhengjie.org.cn.out
node101.yinzhengjie.org.cn: starting journalnode, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/hadoop-root-journalnode-node101.yinzhengjie.org.cn.out
Starting ZK Failover Controllers on NN hosts [node101.yinzhengjie.org.cn node103.yinzhengjie.org.cn]
node101.yinzhengjie.org.cn: starting zkfc, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/hadoop-root-zkfc-node101.yinzhengjie.org.cn.out
node103.yinzhengjie.org.cn: starting zkfc, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/hadoop-root-zkfc-node103.yinzhengjie.org.cn.out
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# hdfs haadmin -getServiceState namenode1　　　　　　　　　　　　#查看当前节点的状态！很显然，和下面的Web UI展示的结果一致！
active
[root@node101.yinzhengjie.org.cn ~]#

4>.启动yarn集群

[root@node101.yinzhengjie.org.cn ~]# start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/yarn-root-resourcemanager-node101.yinzhengjie.org.cn.out
node102.yinzhengjie.org.cn: starting nodemanager, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/yarn-root-nodemanager-node102.yinzhengjie.org.cn.out
node103.yinzhengjie.org.cn: starting nodemanager, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/yarn-root-nodemanager-node103.yinzhengjie.org.cn.out
node101.yinzhengjie.org.cn: starting nodemanager, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/yarn-root-nodemanager-node101.yinzhengjie.org.cn.out
[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# start-yarn.sh　　　　　　　　　　　　　　　　　　　　　　　　#在resourcemanager1节点上启动YARN

[root@node101.yinzhengjie.org.cn ~]# jps
7024 ZooKeeperMain
2273 Jps
499 QuorumPeerMain
1414 DFSZKFailoverController
1192 JournalNode
824 NameNode
1994 NodeManager
1867 ResourceManager
957 DataNode
[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# jps

[root@node103.yinzhengjie.org.cn ~]# yarn-daemon.sh start resourcemanager
starting resourcemanager, logging to /yinzhengjie/softwares/hadoop-2.9.2/logs/yarn-root-resourcemanager-node103.yinzhengjie.org.cn.out
[root@node103.yinzhengjie.org.cn ~]#
[root@node103.yinzhengjie.org.cn ~]# jps
7905 Jps
7842 ResourceManager
6984 JournalNode
7545 NodeManager
6554 QuorumPeerMain
6764 NameNode
7132 DFSZKFailoverController
6863 DataNode
[root@node103.yinzhengjie.org.cn ~]#

[root@node103.yinzhengjie.org.cn ~]# yarn-daemon.sh start resourcemanager　　　　　　　　　　#在resourcemanager2节点上启动rm进程

[root@node103.yinzhengjie.org.cn ~]# jps
7842 ResourceManager
6984 JournalNode
7545 NodeManager
6554 QuorumPeerMain
6764 NameNode
7132 DFSZKFailoverController
8653 Jps
6863 DataNode
[root@node103.yinzhengjie.org.cn ~]#

[root@node103.yinzhengjie.org.cn ~]# jps

[root@node103.yinzhengjie.org.cn ~]# yarn rmadmin -getServiceState resourcemanager1　　　　　　　　#很显然，我们先启动的是resourcemanager1节点，因此它优先被选为Active状态！
active
[root@node103.yinzhengjie.org.cn ~]#
[root@node103.yinzhengjie.org.cn ~]# yarn rmadmin -getServiceState resourcemanager2
standby
[root@node103.yinzhengjie.org.cn ~]#

三.验证YARN的高可用性

1>.将resourcemanager1节点进程杀掉，查看服务是否可以正常提供服务

[root@node101.yinzhengjie.org.cn ~]# jps
7024 ZooKeeperMain
499 QuorumPeerMain
1414 DFSZKFailoverController
1192 JournalNode
824 NameNode
1994 NodeManager
1867 ResourceManager
957 DataNode
4191 Jps
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# kill -9 `jps | grep ResourceManager | awk '{print $1}' `
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# jps
7024 ZooKeeperMain
499 QuorumPeerMain
4276 Jps
1414 DFSZKFailoverController
1192 JournalNode
824 NameNode
1994 NodeManager
957 DataNode
[root@node101.yinzhengjie.org.cn ~]#

2>.启动yarn集群后，注意观察zookeeper集群中生成的目录结构的变化哟！

[root@node101.yinzhengjie.org.cn ~]# zkCli.sh -server node103.yinzhengjie.org.cn:2181
Connecting to node103.yinzhengjie.org.cn:2181
2019-04-19 10:22:52,906 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
2019-04-19 10:22:52,909 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=node101.yinzhengjie.org.cn
2019-04-19 10:22:52,909 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_201
2019-04-19 10:22:52,911 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2019-04-19 10:22:52,911 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-2.el7_6.x86_64/jre
2019-04-19 10:22:52,911 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/yinzhengjie/softwares/zookeeper-3.4.14/bin/../zookeeper-server/target/classes:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../build/classes:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../zookeeper-server/target/lib/*.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../build/lib/*.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/slf4j-log4j12-1.7.25.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/slf4j-api-1.7.25.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/netty-3.10.6.Final.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/log4j-1.2.17.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/jline-0.9.94.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/audience-annotations-0.5.0.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../zookeeper-3.4.14.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../zookeeper-server/src/main/resources/lib/*.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../conf:
2019-04-19 10:22:52,912 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-04-19 10:22:52,912 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2019-04-19 10:22:52,912 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
2019-04-19 10:22:52,912 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
2019-04-19 10:22:52,912 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2019-04-19 10:22:52,912 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=3.10.0-957.el7.x86_64
2019-04-19 10:22:52,912 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=root
2019-04-19 10:22:52,912 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/root
2019-04-19 10:22:52,912 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/root
2019-04-19 10:22:52,913 [myid:] - INFO  [main:ZooKeeper@442] - Initiating client connection, connectString=node103.yinzhengjie.org.cn:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@5ce65a89
Welcome to ZooKeeper!
2019-04-19 10:22:52,938 [myid:] - INFO  [main-SendThread(node103.yinzhengjie.org.cn:2181):ClientCnxn$SendThread@1025] - Opening socket connection to server node103.yinzhengjie.org.cn/172.30.1.103:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2019-04-19 10:22:52,998 [myid:] - INFO  [main-SendThread(node103.yinzhengjie.org.cn:2181):ClientCnxn$SendThread@879] - Socket connection established to node103.yinzhengjie.org.cn/172.30.1.103:2181, initiating session
[zk: node103.yinzhengjie.org.cn:2181(CONNECTING) 0] 2019-04-19 10:22:53,026 [myid:] - INFO  [main-SendThread(node103.yinzhengjie.org.cn:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server node103.yinzhengjie.org.cn/172.30.1.103:2181, sessionid = 0x6700035fda3a0000, negotiated timeout = 30000WATCHER::WatchedEvent state:SyncConnected type:None path:null[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 0] ls /
[zookeeper,hadoop-ha]
[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 1]
[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 1] ls /
[zookeeper, yarn-leader-election, hadoop-ha, rmstore]
[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 2]

[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 2] get /yarn-leader-electioncZxid = 0x300000008
ctime = Fri Apr 19 10:09:04 CST 2019
mZxid = 0x300000008
mtime = Fri Apr 19 10:09:04 CST 2019
pZxid = 0x300000009
cversion = 1
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 1
[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 3]

[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 2] get /yarn-leader-election

[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 3] ls /yarn-leader-election
[yinzhengjie-yarn]
[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 4]

[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 3] ls /yarn-leader-election

[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 4] ls /yarn-leader-election/yinzhengjie-yarn
[ActiveBreadCrumb, ActiveStandbyElectorLock]
[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 5]

[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 4] ls /yarn-leader-election/yinzhengjie-yarn

[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 7] get /yarn-leader-election/yinzhengjie-yarn/ActiveBreadCrumb       yinzhengjie-yarnresourcemanager2
cZxid = 0x30000000b
ctime = Fri Apr 19 10:09:08 CST 2019
mZxid = 0x30000002c
mtime = Fri Apr 19 10:18:50 CST 2019
pZxid = 0x30000000b
cversion = 0
dataVersion = 1
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 36
numChildren = 0
[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 8]

[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 7] get /yarn-leader-election/yinzhengjie-yarn/ActiveBreadCrumb

[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 8] get /yarn-leader-election/yinzhengjie-yarn/ActiveStandbyElectorLockyinzhengjie-yarnresourcemanager2
cZxid = 0x30000002b
ctime = Fri Apr 19 10:18:50 CST 2019
mZxid = 0x30000002b
mtime = Fri Apr 19 10:18:50 CST 2019
pZxid = 0x30000002b
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x660003eeeeb10001
dataLength = 36
numChildren = 0
[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 9]

[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 8] get /yarn-leader-election/yinzhengjie-yarn/ActiveStandbyElectorLock

转载于:https://www.cnblogs.com/yinzhengjie/p/10726455.html

Apache Hadoop 2.9.2 的YARN High Available 模式部署相关推荐

  1. Apache Hadoop 3.0.0 GA版正式发布，可以部署到线上

    今天凌晨 Apache Hadoop 3.0.0 GA 版本正式发布,这意味着我们就可以正式在线上使用 Hadoop 3.0.0 了!这个版本是 Apache Hadoop 3.0.0 的第一个稳定版 ...

  2. Apache Hadoop下一代MapReduce框架（YARN）简介 （Apache Hadoop NextGen MapReduce (YARN)）

    英文看着头大,先试着翻译一下. E文原文:http://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-yarn/hadoop-yarn-site/YARN ...

  3. Apache Hadoop YARN：另一个资源协调者

    文章目录 摘要 1. 引言 2. 历史和基本原理 2.1 专用集群的时代 2.2 Hadoop on Demand的缺点 2.3 共享集群 3. 架构 3.1 概述 3.2 Resource Mana ...

  4. hive提交命令org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error parsing application ID:

    org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error parsing application ID: 怎么说?我说我重新提交一次行 ...

  5. java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/exceptions/YarnException

    flink提交任务卡死,cancel job以后,在$FLINK_HOME/log/flink-appleyuchi-client-Desktop.log 发现该报错: java.lang.NoCla ...

  6. hadoop错误org.apache.hadoop.yarn.exceptions.YarnException Unauthorized request to start container

    错误: 17/11/22 15:17:15 INFO client.RMProxy: Connecting to ResourceManager at Master/192.168.136.100:8 ...

  7. hadoop错误: 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMaster

    错误: 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMaster 原创hongxiao2016 最后发布于2019-03-30 21:20:5 ...

  8. apache hadoop 2.4.0 64bit 在windows8.1下直接安装指南（无需虚拟机和cygwin）

    工作须要.要開始搞hadoop了,又是大数据,自己感觉大数据.云.仅仅是ERP.SOAP风潮之后与智能地球一起诞生的概念炒作. 只是Apache是个奇妙的组织.Java假设没有它也不会如今如火中天.言 ...

  9. Apache Hadoop版本详解

    由doop版本混乱多变,因此,Hadoop的版本选择问题一直令很多初级用户苦恼.本文总结了ApacheHadoop和Cloudera Hadoop的版本衍化过程,并给出了选择Hadoop版本的一些建议 ...

最新文章

  1. 荧光皮肤有哪些_荧光增白剂真的致癌吗？
  2. Qt 读取XML文档的简单示例
  3. Dev-GridView-对于gridview的列值的合计
  4. 解决 No module named ‘PyQt5.QtWebEngineWidgets‘
  5. 曲线积分与曲面积分总结_高数下册||知识点总结
  6. Redis入门（三）Redis的基本数据类型
  7. class CT where T : new() 泛型类约束
  8. 蓝牙(Android使用)基本原理
  9. Java—计算球体积
  10. 劝学诗整理：安居不用架高堂，书中自有黄金屋。
  11. 微信小程序自定义页面标题
  12. matlab一维图像程序,求大神指教啊！基于图像处理的一维条码识别（MATLAB程序）...
  13. 揭开HPC应用的神秘面纱
  14. GUI编程入门到游戏实战
  15. C1能力认证训练题解析 _ 第二部分 _ Web基础
  16. 【JDBC】——JDBC概述
  17. 0056-在OpenCV环境下使用混合高斯背景建模提取前景目标
  18. 从零开始学习Java设计模式 | 设计模式入门篇：设计模式概述
  19. 软硬件协同设计的系统级开发环境~BPS软件介绍
  20. 计算机组成原理小白教程

热门文章

  1. 企业品牌官网如何设计与制作
  2. 获取Android手机SD卡容量大小
  3. sublime插件安装，配色方案
  4. C++内存机制中内存泄露、内存溢出、内存越界和内存泄漏原因分析
  5. 面试经之给刚毕业的大学生和刚入门的程序员的忠告及图书推荐
  6. 【室内乐】Orchestral Suite No.2 in B minor, BWV 1067, Badinerie
  7. oracle 2003,Oracle + Windows 2003 群集手记
  8. 射频电路原理及主要应用
  9. iOS 开发问与答(85-102)
  10. php加密js函数输出,PHP加密函数 Javascript/Js 解密函数的实现方法