一.问题描述

如下图所示，刚安装的CDH,提示存在丢失块，也存在副本不足的块

二.解决方案

2.1 丢失块查找

sudo -u hdfs hadoop fsck / -files -blocks

测试记录:

FSCK started by hdfs (auth:SIMPLE) from /10.31.1.123 for path / at Fri Nov 27 09:55:15 CST 2020
/ <dir>
/tmp <dir>
/tmp/.cloudera_health_monitoring_canary_files <dir>
/tmp/.cloudera_health_monitoring_canary_files/.canary_file_2020_11_19-14_47_30.2b73c248cbe340c5 0 bytes, replicated: replication=3, 0 block(s):  OK/tmp/.cloudera_health_monitoring_canary_files/.canary_file_2020_11_19-14_48_30.f8637315419b1c8e 0 bytes, replicated: replication=3, 0 block(s):  OK/tmp/hive <dir>
/tmp/hive/anonymous <dir>
/tmp/hive/anonymous/5245f213-70eb-45d2-970f-1db033fbafad <dir>
/tmp/hive/anonymous/5245f213-70eb-45d2-970f-1db033fbafad/_tmp_space.db <dir>
/tmp/hive/anonymous/83ac2b81-64c8-4a4e-832f-c1f933cea7eb <dir>
/tmp/hive/anonymous/83ac2b81-64c8-4a4e-832f-c1f933cea7eb/_tmp_space.db <dir>
/tmp/hive/anonymous/83ac2b81-64c8-4a4e-832f-c1f933cea7eb/_tmp_space.db/Values__Tmp__Table__1 <dir>
/tmp/hive/anonymous/83ac2b81-64c8-4a4e-832f-c1f933cea7eb/_tmp_space.db/Values__Tmp__Table__1/data_file 22 bytes, replicated: replication=3, 1 block(s):  OK
0. BP-157751563-10.31.1.123-1605413961809:blk_1073749568_8744 len=22 Live_repl=3/tmp/hive/hive <dir>
/tmp/hive/hive/0988ee1b-25eb-43a4-b8cd-d0d7e43a457e <dir>
/tmp/hive/hive/0988ee1b-25eb-43a4-b8cd-d0d7e43a457e/_tmp_space.db <dir>
/tmp/hive/hive/11709f3e-a78b-4f6c-9d85-214474b0a566 <dir>
/tmp/hive/hive/11709f3e-a78b-4f6c-9d85-214474b0a566/_tmp_space.db <dir>
/tmp/hive/hive/190fef6f-a879-4fe3-8861-510701ce0c62 <dir>
/tmp/hive/hive/190fef6f-a879-4fe3-8861-510701ce0c62/_tmp_space.db <dir>
/tmp/hive/hive/1d83563c-8b79-4fea-9dec-22a3d1e0454b <dir>
/tmp/hive/hive/1d83563c-8b79-4fea-9dec-22a3d1e0454b/_tmp_space.db <dir>
/tmp/hive/hive/4f781ef4-b9da-4515-84f4-ff8037dd6e23 <dir>
/tmp/hive/hive/4f781ef4-b9da-4515-84f4-ff8037dd6e23/_tmp_space.db <dir>
**中间省略N多输出**
/user/yarn <dir>
/user/yarn/mapreduce <dir>
/user/yarn/mapreduce/mr-framework <dir>
/user/yarn/mapreduce/mr-framework/3.0.0-cdh6.3.1-mr-framework.tar.gz 235053931 bytes, replicated: replication=3, 2 block(s):  OK
0. BP-157751563-10.31.1.123-1605413961809:blk_1073766370_25546 len=134217728 Live_repl=3
1. BP-157751563-10.31.1.123-1605413961809:blk_1073766371_25547 len=100836203 Live_repl=3Status: CORRUPTNumber of data-nodes:  4Number of racks:               1Total dirs:                    2108Total symlinks:                0Replicated Blocks:Total size:    128560763438 BTotal files:   2187Total blocks (validated):      3112 (avg. block size 41311299 B)********************************UNDER MIN REPL'D BLOCKS:      1814 (58.29049 %)dfs.namenode.replication.min: 1CORRUPT FILES:        1814MISSING BLOCKS:       1814MISSING SIZE:         1519384904 B********************************Minimally replicated blocks:   1298 (41.70951 %)Over-replicated blocks:        0 (0.0 %)Under-replicated blocks:       0 (0.0 %)Mis-replicated blocks:         0 (0.0 %)Default replication factor:    3Average block replication:     1.2512853Missing blocks:                1814Corrupt blocks:                0Missing replicas:              0 (0.0 %)Blocks queued for replication: 0Erasure Coded Block Groups:Total size:    0 BTotal files:   0Total block groups (validated):        0Minimally erasure-coded block groups:  0Over-erasure-coded block groups:       0Under-erasure-coded block groups:      0Unsatisfactory placement block groups: 0Average block group size:      0.0Missing block groups:          0Corrupt block groups:          0Missing internal blocks:       0Blocks queued for replication: 0
FSCK ended at Fri Nov 27 09:55:15 CST 2020 in 47 millisecondsThe filesystem under path '/' is CORRUPT

过滤出 MISSING的信息
可以看到都是oozie这个空间丢失的块

/user/oozie/share/lib/lib_20201115122055/distcp/hadoop-distcp.jar 4038448 bytes, replicated: replication=3, 1 block(s):  MISSING 1 blocks of total size 4038448 B
0. BP-157751563-10.31.1.123-1605413961809:blk_1073741831_1007 len=4038448 MISSING!
/user/oozie/share/lib/lib_20201115122055/distcp/netty-all-4.1.17.Final.jar 3780056 bytes, replicated: replication=3, 1 block(s):  MISSING 1 blocks of total size 3780056 B
0. BP-157751563-10.31.1.123-1605413961809:blk_1073741829_1005 len=3780056 MISSING!
/user/oozie/share/lib/lib_20201115122055/distcp/oozie-sharelib-distcp-5.1.0-cdh6.3.1.jar 12759 bytes, replicated: replication=3, 1 block(s):  MISSING 1 blocks of total size 12759 B
0. BP-157751563-10.31.1.123-1605413961809:blk_1073741826_1002 len=12759 MISSING!
/user/oozie/share/lib/lib_20201115122055/distcp/oozie-sharelib-distcp.jar 12759 bytes, replicated: replication=3, 1 block(s):  MISSING 1 blocks of total size 12759 B
0. BP-157751563-10.31.1.123-1605413961809:blk_1073741830_1006 len=12759 MISSING!
/user/oozie/share/lib/lib_20201115122055/git/HikariCP-2.6.1.jar 133942 bytes, replicated: replication=3, 1 block(s):  MISSING 1 blocks of total size 133942 B
0. BP-157751563-10.31.1.123-1605413961809:blk_1073741832_1008 len=133942 MISSING!
**中间省略部分输出**
0. BP-157751563-10.31.1.123-1605413961809:blk_1073743627_2803 len=18161 MISSING!
/user/oozie/share/lib/lib_20201115122055/sqoop/xz-1.6.jar 103131 bytes, replicated: replication=3, 1 block(s):  MISSING 1 blocks of total size 103131 B
0. BP-157751563-10.31.1.123-1605413961809:blk_1073743633_2809 len=103131 MISSING!
/user/oozie/share/lib/lib_20201115122055/sqoop/zookeeper.jar 1543701 bytes, replicated: replication=3, 1 block(s):  MISSING 1 blocks of total size 1543701 B
0. BP-157751563-10.31.1.123-1605413961809:blk_1073743638_2814 len=1543701 MISSING!MISSING BLOCKS:       1814MISSING SIZE:         1519384904 B

2.2 解决oozie副本块不足的问题

查看丢失的文件块信息

[root@hp1 ~]# hadoop dfs -ls /user/oozie/share/lib/lib_20201115122055/hive2/jcodings-1.0.18.jar
WARNING: Use of this script to execute dfs is deprecated.
WARNING: Attempting to execute replacement "hdfs dfs" instead.-rwxrwxr-x   3 oo

导致这个问题产生时是由于初始化集群后，只有两个节点的datanode，所以将副本调整为2 、dfs.replication=2，在切换后对hdfs做了抑制，导致告警出现；现在重新加入一个datanode，重新恢复3个datanode节点3副本模式。

2.2.1 设置3个副本模式

sudo -u hdfs hadoop fs -setrep -R 3 /

2.2.2 删除坏的块:

-- 删除损坏的块
sudo -u hdfs hadoop fsck / -delete
-- 列出损坏的块
sudo -u hdfs hadoop fsck -list-corruptfileblocks

测试记录:

[root@hp1 ~]# sudo -u hdfs hadoop fsck -list-corruptfileblocks
WARNING: Use of this script to execute fsck is deprecated.
WARNING: Attempting to execute replacement "hdfs fsck" instead.Connecting to namenode via http://hp1:9870/fsck?ugi=hdfs&listcorruptfileblocks=1&path=%2F
The filesystem under path '/' has 0 CORRUPT files

现在看丢失块的问题解决:

参考

1.https://www.cnblogs.com/hqt0731/articles/8804924.html

CDH 丢失块及副本不足的块相关推荐

块代码编程---开始使用块代码
http://blog.csdn.net/nicktang/article/details/6906352 本章的几节使用例子代码来开始块代码的学习. 定义和使用块代码插入符^有两个作用,一个是声明 ...
Python将图像分割成小块然后将所有的块重新拼接在一起
Python将图像分割成小块然后将所有的块重新拼接在一起 #效果示例 #使用patchify包将图像分割成小块然后将所有的块重新拼接在一起 #二维图像切分及合并 import numpy as np ...
python代码块-Python中的代码块和非代码块是什么
Python中的代码块和非代码块是什么发布时间:2020-06-26 13:53:30 来源:亿速云阅读:150 今天就跟大家聊聊有关Python中的代码块和非代码块是什么,可能很多人都不太了解, ...
java自由块_JAVA 静态的自由块和非静态的自由块
java中的自由块分为静态的自由块和非静态的自由块,这两种的执行是有区别的. 非静态自由块的执行时间是:在执行构造函数之前. 静态自由块的执行时间是:class文件加载时执行. 执行的时间不同,造成的 ...
java中static代码块与非静态代码块
static代码块与非静态代码块的区别: static代是码块是在类加载时就加载的,而非静态代码块只有在对象实例化时才执行! 转载于:https://www.cnblogs.com/dazhuzhu/ ...
oracle protocol=beq 不可用,学习笔记:Oracle数据库坏块深入研究obj$坏块导致exp/expdp不能执行原因...
天萃荷净深入研究Oracle坏块obj$导致exp/expdp不能执行导出的原因上篇(案例:Oracle出现obj$坏块exp/expdp导出不能导出的解决办法ORA-01578 ORA-0111 ...
java synchronized块_Java多线程同步代码块Synchronized
Java多线程同步代码块Synchronized Java中的每个对象都有一个与之关联的内部锁(Intrinsic lock). 这种锁也称为监视器(Monitor), 这种内部锁是一种排他锁,可以保 ...
块元素、行内块和内联元素_如何删除内联块元素之间的空间？
块元素.行内块和内联元素 Introduction: 介绍: This question has become rather popular. How does one remove whitespa ...
静态代码块、非静态代码块、构造函数执行顺序
静态代码块.非静态代码块.构造函数执行顺序 /*** 类的实例化顺序,静态代码块.非静态代码块.构造函数.* @author MING*/ class Person {static String na ...

CDH 丢失块及副本不足的块

文章目录