hadoop非HA安装，解决defalutFS（eg：beh）问题

报错如下：

（1）修改core-site.xml :

<property>
<name>fs.defaultFS</name>
<value>hdfs://beh/</value>
</property>

为如下：（因为不修改，hdfs dfs -ls / 会无法查看，报错UnknownHostException: beh）

<property>
<name>fs.defaultFS</name>
<value>hdfs://****:9000/</value>
</property>

修改为如上之后（hdfs dfs -ls / 可以查看，但是运行yarn 测试样例 wordcount无法得到yarn资源，且任务所在节点的nodemanager直接挂掉，查看nodemanager日志，发现如下报错）

注：wordcount测试样例：hadoop jar /opt/beh/core/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.13.1.jar wordcount /wkz/in/hadooptest /wkz/out

FATAL org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
java.lang.IllegalArgumentException: java.net.UnknownHostException: beh
at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:406)
at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:310)
at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:735)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:158)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2816)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:98)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2853)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2835)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:387)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.g
etFileSystem(LogAggregationService.java:181)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.v
erifyAndCreateRemoteLogDir(LogAggregationService.java:188)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.i
nitApp(LogAggregationService.java:336)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.h
andle(LogAggregationService.java:463)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.h
andle(LogAggregationService.java:68)
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:182)
at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:109)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: beh
... 20 more

（2）出现如上错误后，还原core-site.xml 为：

<property>
<name>fs.defaultFS</name>
<value>hdfs://beh/</value>
</property>

运行wordcount测试样例：

hadoop jar /opt/beh/core/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.13.1.jar wordcount hdfs://****:9000/wkz/in/hadooptest hdfs://****:9000/wkz/out

直接报错，报错找不到截图了，只能大概描述下：

第一：还是有java.net.UnknownHostException: beh

第二：mapreduce.framework.name错误

后来做了如下修改：（去掉beh有关的所有，东西，eg：dfs.namenode.rpc-address.beh变为dfs.namenode.rpc-address）

hdfs-site.xml

<configuration>

<property>
<name>dfs.namenode.rpc-address</name>
<value>*****9000</value>
</property>

<property>
<name>dfs.namenode.http-address</name>
<value>******:50070</value>
</property>

<property>
<name>dfs.namenode.name.dir</name>
<value>/data01/dsjdev/testname</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/data01/dsjdev/testdn</value>
</property>
<property>
<name>dfs.block.size</name>
<value>134217728</value>
</property>

<property>
<name>dfs.datanode.handler.count</name>
<value>20</value>
</property>

<property>
<name>dfs.namenode.handler.count</name>
<value>20</value>
</property>

<property>
<name>dfs.datanode.max.xcievers</name>
<value>131072</value>
</property>
<property>
<name>dfs.datanode.socket.write.timeout</name>
<value>0</value>
</property>

<property>
<name>dfs.socket.timeout</name>
<value>180000</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>

<property>
<name>mapred.child.env</name>
<value>JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native</value>
</property>
</configuration>

core-site.xml

<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://******:9000/</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/data01/dsjdev/testnn</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
<property>
<name>io.compression.codecs</name>
<value>org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
<property>
<name>fs.trash.interval</name>
<value>10080</value>
<description>Number of minutes between trash checkpoints.
If zero, the trash feature is disabled.
</description>
</property>
</configuration>

yarn-site.xml

<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>

<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/data01/dsjdev/testnn_local_dir</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/data01/dsjdev/testnn_log_dir</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<description>Where to aggregate logs</description>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>hdfs://******:9000/var/log/hadoop-yarn/apps</value>
</property>
<property>
<name>yarn.resourcemanager.connect.retry-interval.ms</name>
<value>2000</value>
</property>
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>beh</value>
</property>

<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
</property>

<property>
<name>yarn.app.mapreduce.am.scheduler.connection.wait.interval-ms</name>
<value>5000</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>******:23140</value>
</property>

<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>******:23130</value>
</property>

<property>
<name>yarn.resourcemanager.webapp.https.address</name>
<value>******:23189</value>
</property>

<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>******:23188</value>
</property>

<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>******:23125</value>
</property>

<property>
<name>yarn.resourcemanager.admin.address</name>
<value>******:23141</value>
</property>

<property>
<description>Address where the localizer IPC is.</description>
<name>yarn.nodemanager.localizer.address</name>
<value>0.0.0.0:23344</value>
</property>

<property>
<description>NM Webapp address.</description>
<name>yarn.nodemanager.webapp.address</name>
<value>0.0.0.0:23999</value>
</property>

<property>
<name>mapreduce.shuffle.port</name>
<value>23080</value>
</property>

<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>10240</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>6144</value>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>2</value>
<description>number of all the cpu cores</description>
</property>
</configuration>

mapred-site.xml

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>*****:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>*****:19888</value>
</property>
<property>
<name>yarn.app.mapreduce.am.staging-dir</name>
<value>/data01/dsjdev/hadoop-yarn/staging</value>
</property>
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx2g</value>
</property>

<property>
<name>io.sort.factor</name>
<value>20</value>
</property>
<property>
<name>mapred.job.reuse.jvm.num.tasks</name>
<value>-1</value>
</property>
<property>
<name>mapreduce.reduce.shuffle.parallelcopies</name>
<value>20</value>
</property>
<property>
<name>mapred.output.compress</name>
<value>true</value>
</property>
<property>
<name>mapred.output.compression.codec</name>
<value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
<property>
<name>mapreduce.map.output.compress</name>
<value>true</value>
</property>
<property>
<name>mapreduce.map.output.compress.codec</name>
<value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
<property>
<name>mapred.output.compression.type</name>
<value>BLOCK</value>
</property>
<property>
<name>mapreduce.cluster.local.dir</name>
<value>/data01/dsjdev/testmapred/local</value>
</property>
</configuration>

修改后重启hadoop，一切正常

hadoop非HA安装，解决defalutFS（eg：beh）问题相关推荐

hadoop 2.x安装：不能加载本地库 - 解决libc.so.6 version GLIBC_2.14 not found问题
hadoop 2.x安装:不能加载本地库 - 解决libc.so.6 version GLIBC_2.14 not found问题参考文章: (1)hadoop 2.x安装:不能加载本地库 - 解决 ...
解决RStudio（非conda安装）在使用Anaconda中的R环境时，缺失“ libbz2-1.dll ”而不能正常启动问题
解决RStudio(非conda安装)在使用Anaconda中的R环境时,缺失" libbz2-1.dll "而不能正常启动问题参考文章: (1)解决RStudio(非conda ...
vatic标注工具安装步骤（非docker安装）以及错误解决办法
Vatic标注工具安装步骤(非docker安装)以及错误解决 1.安装参考 2.安装步骤 2.1.环境介绍 2.2.下载Vatic 2.3.配置Apache 2.4.配置MySQL 2.5.配置/安装 ...
非chrome应用市场插件安装解决办法
非chrome应用市场插件安装解决办法 Chrome 离线加装扩展的新方法打开 chrome://flags/#extensions-on-chrome-urls,找到( Ctrl + F )#ex ...
Hive、Hadoop完全分布式安装，基本SQL，基本SQL，基本表结构，数据类型，函数，窗口函数，jion,查询和排序，beeline,文件格式及基本架构汇总
目录 Hive简介一.概述二.Hive和数据库的比较三.特点 Hadoop完全分布式安装一.配置二.安装步骤 Hive安装一.概述二.安装步骤其他一.Hive运行日志二.Hive的 ...
Hadoop的HA机制原理解析，与HA高可用集群环境搭建
2019独角兽企业重金招聘Python工程师标准>>> Hadoop的HA工作机制示意图下面我们开始搭建这一套高可用集群环境 hadoop2.0已经发布了稳定版本了,增加了很多特性 ...
HUE配置文件hue.ini 的database模块详解（包含qlite、mysql、 psql、和oracle）（图文详解）（分HA集群和非HA集群）...
不多说,直接上干货! Hue配置文件里,提及到,提供有postgresql_psycopg2, mysql, sqlite3 or oracle. 注意:Hue本身用到的是sqlite3. 在哪里呢, ...
HUE配置文件hue.ini 的hbase模块详解（图文详解）（分HA集群和非HA集群）
不多说,直接上干货! 我的集群机器情况是 bigdatamaster(192.168.80.10).bigdataslave1(192.168.80.11)和bigdataslave2(192.168 ...
Hadoop的HA原理
一.概述 Hadoop的HA应该分为HDFS 的 HA 和 YARN 的 HA,主要是解决NameNode和ResourceManager的单点故障问题,所以HA就是通过配置 Active/Stand ...

hadoop非HA安装，解决defalutFS（eg：beh）问题

hadoop非HA安装，解决defalutFS（eg：beh）问题相关推荐

最新文章

热门文章