CDH6.3.2

一、基本介绍

基于最新CDH发布版本编写安装文档，CDH6安装过程基本与CDH5一致，主要包括以下几步：

1、安装前置准备：包括安装操作系统、关闭防火墙、同步服务器时钟等；

2、mysql安装部署；

3、安装Cloudera Manager；

4、安装CDH集群；

软件要求：

CDH 集群搭建需要的安装包包括 JDK、Cloudera-Manager RPM 包（3 个），CDH parcel⽂件（3 个）、MySQL、MySQL 驱动 jar 包。

组件	版本	类型	备注
JDK	8u202	tar	jdk-8u202-linux-x64.tar.gz
CM daemons	6.3.1	rpm	cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
CM agent	6.3.1	rpm	cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
CM server	6.3.1	rpm	cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
CDH 各组件⼆进制服务包	6.3.2	parcel	CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel、CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1、manifest.json
MySQL	8.0.18	tar	mysql-8.0.18-el7-x86_64.tar.gz
MySQL驱动包	8.0.18	jar	mysql-connector-java-8.0.18.jar

节点分布：

CM Server 和 Agent 分配 Cloudera Manager 由 Server 和 Agent 组成，安装 Cloudera Manager Server 的节点称为 Server 节点，安装 Cloudera Manager Agent 的节点称为 Agent 节点。

3个节点的情形

节点	安装的服务
node1	CM daemon，CM server，CM agent
node2	CM daemon，CM agent
node3	CM daemon，CM agent

二、安装前置准备

没有出现其他节点主机名的话，说明每个节点都是需要执行的。

2.1 hosts及hostname

rehl7修改主机名使用hostnamectl命令

#节点一
[root@hadoop-1 ~]# hostnamectl set-hostname hadoop-1
#节点二
[root@hadoop-2 ~]# hostnamectl set-hostname hadoop-2
#节点三
[root@hadoop-3 ~]# hostnamectl set-hostname hadoop-3

修改/etc/hosts（每个节点都执行）

[root@hadoop-1 ~]# vim /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
#集群节点地址 集群主机名
172.15.10.145 hadoop-1
172.15.10.146 hadoop-2
172.15.10.147 hadoop-3

2.2 禁用selinux并关闭防火墙

[root@hadoop-1 ~]# vim /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted
-------------------------------------------------------------------------------------------------------------------
[root@hadoop-1 ~]# systemctl stop firewalld.service
[root@hadoop-1 ~]# systemctl disable firewalld.service

2.3 系统参数修改

swap空间设置

[root@hadoop-1 ~]# echo "vm.swappiness = 0" >> /etc/sysctl.conf
[root@hadoop-1 ~]# sysctl -p

设置透明大页面

[root@hadoop-1 ~]# echo never > /sys/kernel/mm/transparent_hugepage/defrag
[root@hadoop-1 ~]# echo never > /sys/kernel/mm/transparent_hugepage/enabled

增⼤系统打开⽂件数

[root@hadoop-1 ~]# vim /etc/security/limits.conf
#末尾加上
* - nofile 100000
* - nproc 100000
hadoop soft memlock unlimited
hadoop hard memlock unlimited

2.4 ssh免密认证

[root@hadoop-1 ~]# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
[root@hadoop-1 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@hadoop-1
[root@hadoop-1 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@hadoop-2
[root@hadoop-1 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@hadoop-3
#验证方式，ssh hostname,直接连接上了说明配置成功
[root@hadoop-1 ~]# ssh hadoop-2

2.5 安装并启动Apache http

在主节点执行

#安装Apache http
[root@hadoop-1 ~]# yum install -y httpd
#启动Apache http
[root@hadoop-1 cdh]# systemctl start httpd
#设置开机启动Apache http
[root@hadoop-1 cdh]# systemctl enable httpd
#创建安装文件http根目录
[root@hadoop-1 cdh]# mkdir -p /var/www/html/cm6
[root@hadoop-1 cdh]# mkdir -p /var/www/html/cdh6
[root@hadoop-1 cdh]# cd /usr/local/src/cdh/
#上传安装文件到http根目录
[root@hadoop-1 cdh]# cp allkeys.asc *.rpm /var/www/html/cm6
[root@hadoop-1 cdh]# ll /var/www/html/cm6
total 1199784
-rw-r--r-- 1 root root      14041 Jan 22 00:06 allkeys.asc
-rw-r--r-- 1 root root   10483568 Jan 22 00:06 cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
-rw-r--r-- 1 root root 1203832464 Jan 22 00:06 cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
-rw-r--r-- 1 root root      11488 Jan 22 00:06 cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
-rw-r--r-- 1 root root      10996 Jan 22 00:06 cloudera-manager-server-db-2-6.3.1-1466458.el7.x86_64.rpm
-rw-r--r-- 1 root root   14209868 Jan 22 00:06 enterprise-debuginfo-6.3.1-1466458.el7.x86_64.rpm
#创建yum仓库
[root@hadoop-1 cdh]# cd /var/www/html/cm6
[root@hadoop-1 cm6]# yum install -y createrepo
[root@hadoop-1 cm6]# createrepo .

2.6 yum源设置

yum源设置有多种方式，无网络可以采用镜像挂载，有网络可以直接用清华源、阿里源等等，此处不做展开，自行百度。这里介绍下本地挂载的方式。

[root@hadoop-1 ~]# vim /etc/yum.repos.d/rhel7.repo
[base]
name=rhel7
baseurl=file:///media/cdrom
enabled=1
gpgcheck=0
[root@hadoop-1 ~]# mkdir -p /media/cdrom
[root@hadoop-1 ~]# mount /dev/cdrom /media/cdrom
mount: /media/cdrom: WARNING: device write-protected, mounted read-only.
-------------------------------------------------------------------------------------------------------------------
#挂载自建镜像
[root@hadoop-1 cm6]# vim /etc/yum.repos.d/cloudera-manager.repo
[cloudera-manager]
name=Cloudera Manager 6.3.1
baseurl=http://hadoop-1/cm6
gpgcheck=0
enabled=1
autorefresh=0
type=rpm-md
[root@hadoop-1 cm6]# yum clean all
#查看是否挂载成功
[root@hadoop-1 cm6]# yum repolist

网页测试能否正常访问

2.7 时钟同步设置

检查是否安装ntp，如果没有则使用yum安装

[root@hadoop-1 ~]# rpm -qa | grep ntp
[root@hadoop-1 ~]# yum install -y ntp

主节点hadoop-1配置（设置为标准时间服务器）

[root@hadoop-1 ~]# vim /etc/ntp.conf
# For more information about this file, see the man pages
# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).driftfile /var/lib/ntp/drift
#新增:日志目录.
logfile /var/log/ntpd.log# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default nomodify notrap nopeer noquery# Permit all access over the loopback interface.  This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict ::1# Hosts on local network are less restricted.
#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
#新增：授权172.15.10.0网段上的所有机器可以从这台机器上查询和同步时间
restrict 172.15.10.0 mask 255.255.255.0 nomodify# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#修改:注释外部时钟服务器配置，实际生产部署需要外部时钟服务器，否则会有告警
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst#broadcast 192.168.1.255 autokey        # broadcast server
#broadcastclient                        # broadcast client
#broadcast 224.0.1.1 autokey            # multicast server
#multicastclient 224.0.1.1              # multicast client
#manycastserver 239.255.254.254         # manycast server
#manycastclient 239.255.254.254 autokey # manycast client
#新增:当外部时间不可用时，使用本地时间.
server 172.15.10.145 iburst
fudge 127.0.0.1 stratum 10# Enable public key cryptography.
#crypto
-------------------------------------------------------------------------------------------------------------------
[root@hadoop-1 ~]# systemctl enable ntpd
[root@hadoop-1 ~]# systemctl start ntpd

其他节点配置，同步主节点时间

[root@hadoop-2 ~]# echo "server 172.15.10.145" >> /etc/ntp.conf
[root@hadoop-2 ~]# systemctl enable ntpd
[root@hadoop-2 ~]# systemctl start ntpd
#检查方式
[root@hadoop-2 ~]# ntpq -premote           refid      st t when poll reach   delay   offset  jitter
==============================================================================hadoop-1        .INIT.          16 u   32   64    0    0.000    0.000   0.000

2.8 jdk安装

1、卸载默认安装的jdk

[root@hadoop-1 ~]# rpm -qa|grep jdk
[root@hadoop-1 ~]# rpm -e --nodeps #上一步查询出来的jdk名字

2、安装jdk

目录必须严格要求文中设置，cdh启动的时候强制找/usr/java下面的jdk

[root@hadoop-1 src]# mkdir -p /usr/java
[root@hadoop-1 ~]# cd /usr/local/src
#个人习惯将包上传到/usr/local/src目录
[root@hadoop-1 src]# tar -xzvf jdk-8u202-linux-x64.tar.gz -C /usr/java
[root@hadoop-1 src]# cd /usr/java
[root@hadoop-1 java]# ln -s jdk1.8.0_202/ default
[root@hadoop-1 java]# chmod -R 755 /usr/java
#设置环境变量
[root@hadoop-1 ~]# vim /etc/profile
##新增JAVA环境变量
export JAVA_HOME=/usr/java/default
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export PATH=$JAVA_HOME/bin:$PATH
-------------------------------------------------------------------------------------------------------------------
[root@hadoop-1 java]# source /etc/profile

三、mysql安装部署

mysql只需要在server节点部署，以下操作只针对hadoop-1节点

3.1 卸载原有mysql

#查询mysql相关rpm包
[root@hadoop-1 ~]# rpm -qa | grep -i mysql
qt-mysql-4.8.7-2.el7.x86_64
akonadi-mysql-1.9.2-4.el7.x86_64
perl-DBD-MySQL-4.023-6.el7.x86_64
#将查询到的包卸载
[root@hadoop-1 ~]# rpm  --nodeps -ev qt-mysql-4.8.7-2.el7.x86_64 akonadi-mysql-1.9.2-4.el7.x86_64 perl-DBD-MySQL-4.023-6.el7.x86_64
#查询mysql相关文件
[root@hadoop-1 ~]# find / -name mysql
#删除mysql相关文件
[root@hadoop-1 ~]# rm -rf /var/lib/mysql
[root@hadoop-1 ~]# rm -rf /usr/lib64/mysql
[root@hadoop-1 ~]# rm -rf /usr/bin/mysql
[root@hadoop-1 ~]# rm -rf /usr/share/mysql
[root@hadoop-1 ~]# rm -rf /var/lib/pcp/config/pmlogconf/mysql
[root@hadoop-1 ~]# rm -rf /etc/selinux/targeted/active/modules/100/mysql
#删除mysql默认读取配置（使用tar包安装情况）
[root@hadoop-1 ~]# rm -rf /etc/my.cnf /etc/my.cnf.d/

3.2 安装mysql

官网地址：mysql8下载

[root@hadoop-1 ~]# cd /usr/local/src
[root@hadoop-1 src]# wget https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-8.0.18-el7-x86_64.tar.gz

解压压缩包，将解压出的程序目录改名并移动到/usr/local目录下，对其进行初始化操作后便可使用。：

[root@hadoop-1 src]# tar -zxvf mysql-8.0.18-el7-x86_64.tar.gz
[root@hadoop-1 src]# cp -r  mysql-8.0.18-el7-x86_64/ /usr/local/mysql
[root@hadoop-1 src]# cd /usr/local/mysql/
[root@hadoop-1 mysql]# mkdir data
#创建mysql用户
[root@hadoop-1 mysql]# useradd mysql -M -s /sbin/nologin

修改my.cnf配置，典型的mysql8 my.cnf配置：

[root@hadoop-1 ~]# vim /etc/my.cnf
[client]
port = 3306
#根据实际情况调整mysql.sock配置
socket = /tmp/mysql.sock[mysqld]
#Mysql服务的唯一编号 每个mysql服务Id需唯一
server-id = 1
#服务端口号 默认3306
port = 3306
#mysql安装根目录
basedir = /usr/local/mysql
#mysql数据文件所在位置
datadir = /usr/local/mysql/data
#pid
pid-file = /usr/local/mysql/data/mysql.pid
#设置socke文件所在目录
socket = /tmp/mysql.sock
#设置临时目录
tmpdir = /tmp
# 用户
user = mysql
# 允许访问的IP网段
bind-address = 0.0.0.0
# 跳过密码登录
#skip-grant-tables
#主要用于MyISAM存储引擎,如果多台服务器连接一个数据库则建议注释下面内容
skip-external-locking
#只能用IP地址检查客户端的登录，不用主机名
skip_name_resolve = 1
#事务隔离级别，默认为可重复读，mysql默认可重复读级别（此级别下可能参数很多间隙锁，影响性能）
transaction_isolation = READ-COMMITTED
#数据库默认字符集,主流字符集支持一些特殊表情符号（特殊表情符占用4个字节）
character-set-server = utf8mb4
#数据库字符集对应一些排序等规则，注意要和character-set-server对应
collation-server = utf8mb4_general_ci
#设置client连接mysql时的字符集,防止乱码
init_connect='SET NAMES utf8mb4'
#是否对sql语句大小写敏感，1表示不敏感
lower_case_table_names = 1
#最大连接数
max_connections = 400
#最大错误连接数
max_connect_errors = 1000
#TIMESTAMP如果没有显示声明NOT NULL，允许NULL值
explicit_defaults_for_timestamp = true
#SQL数据包发送的大小，如果有BLOB对象建议修改成1G
max_allowed_packet = 128M
#MySQL连接闲置超过一定时间后(单位：秒)将会被强行关闭
#MySQL默认的wait_timeout  值为8个小时, interactive_timeout参数需要同时配置才能生效
interactive_timeout = 1800
wait_timeout = 1800
#内部内存临时表的最大值 ，设置成128M。
#比如大数据量的group by ,order by时可能用到临时表，
#超过了这个值将写入磁盘，系统IO压力增大
tmp_table_size = 134217728
max_heap_table_size = 134217728
#数据库错误日志文件
log_error = error.log
#慢查询sql日志设置
slow_query_log = 1
slow_query_log_file = slow.log
#检查未使用到索引的sql
log_queries_not_using_indexes = 1
#针对log_queries_not_using_indexes开启后，记录慢sql的频次、每分钟记录的条数
log_throttle_queries_not_using_indexes = 5
#作为从库时生效,从库复制中如何有慢sql也将被记录
log_slow_slave_statements = 1
#慢查询执行的秒数，必须达到此值可被记录
long_query_time = 8
#检索的行数必须达到此值才可被记为慢查询
min_examined_row_limit = 100
#mysql binlog日志文件保存的过期时间，过期后自动删除
expire_logs_days = 5

3.3 初始化MySQL服务程序

对目录进行授权，保证数据能够被mysql系统用户读取。在初始化阶段，应使用mysqld命令确认管理MySQL数据库服务的用户名称、数据保存目录及编码信息。在信息确认无误后开始进行初始化。在初始化的最后阶段，系统会给用户分配一个初始化的临时密码。mysql8版本只有在初始化的时候设置 lower_case_table_names=1才有效，切记。

[root@hadoop-1 mysql]# chown -R mysql:mysql /usr/local/mysql
[root@hadoop-1 mysql]# cd bin
[root@hadoop-1 bin]# ./mysqld --initialize --user=mysql --basedir=/usr/local/mysql --datadir=/usr/local/mysql/data --lower-case-table-names=1
#如果使用了上面的配置文件，会生成日志在data目录下，里面会有初始化生成的密码
[root@hadoop-1 bin]# vim /usr/local/mysql/data/error.log
2022-01-20T15:45:19.609453Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2022-01-20T15:45:19.609551Z 0 [System] [MY-013169] [Server] /usr/local/mysql/bin/mysqld (mysqld 8.0.18) initializing of server in progress as process 3281
2022-01-20T15:45:20.506473Z 0 [Warning] [MY-010161] [Server] You need to use --log-bin to make --expire_logs_days work.
2022-01-20T15:45:21.612476Z 5 [Note] [MY-010454] [Server] A temporary password is generated for root@localhost: ww-=dht_A0uj

添加环境变量，方便调用

[root@hadoop-1 bin]# vim ~/.bash_profile
# .bash_profile# Get the aliases and functions
if [ -f ~/.bashrc ]; then. ~/.bashrc
fi# User specific environment and startup programsPATH=$PATH:$HOME/bin:/usr/local/mysql/binexport PATH
[root@hadoop-1 bin]# source ~/.bash_profile

3.4 配置自启动并启动MySQL

libtinfo.so.5文件是MySQL数据库在8.0版本后新添加的重要的函数库文件，但部分情况不存在，需要将libtinfo.so.6.1文件复制过来或者作为链接文件才能正常启动：

[root@hadoop-1 bin]# cd /usr/local/mysql
[root@hadoop-1 mysql]# cp -a support-files/mysql.server /etc/init.d/
[root@hadoop-1 mysql]# chmod a+x /etc/init.d/mysql.server
[root@hadoop-1 mysql]# ln -s /usr/lib64/libtinfo.so.6.1 /usr/lib64/libtinfo.so.5
[root@hadoop-1 mysql]# /etc/init.d/mysql.server start

登录MySQL并修改密码

[root@hadoop-1 mysql]# mysql -u root -p
Enter password: 初始化的时候生成的密码
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 8.0.27Copyright (c) 2000, 2021, Oracle and/or its affiliates.Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql> alter user 'root'@'localhost' identified by 'onaslowboattochina';
#mysql8之后需要额外执行此步才会将密码修改生效
mysql> use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -ADatabase changed
mysql> ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'onaslowboattochina';
#设置远程访问权限,注意Mysql8.0不支持grant all privileges on *.* to root@“%“ identified by “.“；
mysql> CREATE USER 'root'@'%' IDENTIFIED BY 'onaslowboattochina';
Query OK, 0 rows affected (0.00 sec)mysql> grant all privileges on *.* to 'root'@'%' ;
Query OK, 0 rows affected (0.00 sec)mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

3.5 创建数据库

根据实际需要部署的服务，增加或者减少创建的数据库用户数

create database metastore default character set utf8;
CREATE USER 'hive'@'%' IDENTIFIED BY 'onaslowboattochina';
GRANT ALL PRIVILEGES ON metastore. * TO 'hive'@'%';
FLUSH PRIVILEGES;create database rm default character set utf8;
CREATE USER 'rm'@'%' IDENTIFIED BY 'onaslowboattochina';
GRANT ALL PRIVILEGES ON rpm.* TO 'rm'@'%';
FLUSH PRIVILEGES;create database amon default character set utf8;
CREATE USER 'amon'@'%' IDENTIFIED BY 'onaslowboattochina';
GRANT ALL PRIVILEGES ON amon.* TO 'amon'@'%';
FLUSH PRIVILEGES;

四、CDH安装

4.1 Cloudera Manager Server 安装(hadoop-1节点)

[root@hadoop-1 ~]# yum install -y cloudera-manager-server

cloudera-manager-server安装完成后会自动生成/opt/cloudera/parcel-repo目录，将准备好的 CDH parcel 包 copy⾄/opt/cloudera/parcel-repo，拷贝后将 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 以 mv 的⽅式重命名为CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha

[root@hadoop-1 ~]# cd /usr/local/src/cdh/
[root@hadoop-1 cdh]# cp CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1  manifest.json /opt/cloudera/parcel-repo/
[root@hadoop-1 cdh]# cd /opt/cloudera/parcel-repo/
[root@hadoop-1 parcel-repo]# mv CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha

安装mysql驱动

[root@hadoop-1 parcel-repo]# cd /usr/local/src/cdh/
[root@hadoop-1 cdh]# mkdir -p /usr/share/java
[root@hadoop-1 cdh]# cp mysql-connector-java-8.0.18.jar /usr/share/java/mysql-connector-java.jar

执行CM初始化脚本

[root@hadoop-1 ~]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql  scm scm onaslowboattochina
#释义                                                                库  用户 密码
JAVA_HOME=/usr/java
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing:  /usr/java/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
[                          main] DbCommandExecutor              INFO  Successfully connected to database.
All done, your SCM database is configured correctly!

启动CMserver

[root@hadoop-1 java]# systemctl start cloudera-scm-server.service
[root@hadoop-1 java]# systemctl status cloudera-scm-server.service
● cloudera-scm-server.service - Cloudera CM Server ServiceLoaded: loaded (/usr/lib/systemd/system/cloudera-scm-server.service; enabled; vendor preset: disabled)Active: active (running) since Fri 2022-01-21 23:14:05 CST; 4s agoProcess: 32326 ExecStartPre=/opt/cloudera/cm/bin/cm-server-pre (code=exited, status=0/SUCCESS)Main PID: 32332 (java)Tasks: 44CGroup: /system.slice/cloudera-scm-server.service└─32332 /usr/java/jdk1.8.0_202/bin/java -cp .:/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:lib/* -server -Dlog4...Jan 21 23:14:05 hadoop-1 systemd[1]: Starting Cloudera CM Server Service...
Jan 21 23:14:05 hadoop-1 systemd[1]: Started Cloudera CM Server Service.
Jan 21 23:14:05 hadoop-1 cm-server[32332]: JAVA_HOME=/usr/java/jdk1.8.0_202
Jan 21 23:14:05 hadoop-1 cm-server[32332]: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
Jan 21 23:14:07 hadoop-1 cm-server[32332]: ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'org.apac...ation logging.
Jan 21 23:14:09 hadoop-1 cm-server[32332]: Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered...y unnecessary.
Hint: Some lines were ellipsized, use -l to show in full.

4.2 Cloudera Manager agent 安装（所有节点）

[root@hadoop-2 ~]# yum install -y cloudera-manager-agent
[root@hadoop-2 ~]# systemctl start cloudera-scm-agent.service
[root@hadoop-2 ~]# systemctl status cloudera-scm-agent.service
● cloudera-scm-agent.service - Cloudera Manager Agent ServiceLoaded: loaded (/usr/lib/systemd/system/cloudera-scm-agent.service; enabled; vendor preset: disabled)Active: active (running) since Fri 2022-01-21 17:03:11 CST; 12s agoMain PID: 17500 (cmagent)Tasks: 21CGroup: /system.slice/cloudera-scm-agent.service└─17500 /usr/bin/python2 /opt/cloudera/cm-agent/bin/cm agentJan 21 17:03:11 hadoop-2 cm[17500]: [21/Jan/2022 17:03:11 +0000] 17500 MainThread agent        INFO     Re-using pre-existing directory: /var/run/cloudera-scm-agent/flood
Jan 21 17:03:11 hadoop-2 cm[17500]: [21/Jan/2022 17:03:11 +0000] 17500 MainThread agent        INFO     Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor/include
Jan 21 17:03:11 hadoop-2 cm[17500]: [21/Jan/2022 17:03:11 +0000] 17500 MainThread agent        INFO     Re-using pre-existing directory: /var/run/cloudera-scm-agent/cgroups
Jan 21 17:03:11 hadoop-2 cm[17500]: [21/Jan/2022 17:03:11 +0000] 17500 MainThread agent        INFO     Re-using pre-existing directory: /var/run/cloudera-scm-agent/process
Jan 21 17:03:11 hadoop-2 cm[17500]: [21/Jan/2022 17:03:11 +0000] 17500 MainThread tmpfs        INFO     Successfully mounted tmpfs at /var/run/cloudera-scm-agent/process
Jan 21 17:03:11 hadoop-2 cm[17500]: [21/Jan/2022 17:03:11 +0000] 17500 MainThread logging      INFO     Logging to /var/log/cloudera-scm-agent/cloudera-scm-agent.log
Jan 21 17:03:18 hadoop-2 cm[17500]: status_server: added process group
Jan 21 17:03:18 hadoop-2 cm[17500]: flood: added process group
Jan 21 17:03:18 hadoop-2 cm[17500]: /opt/cloudera/cm-agent/lib/python2.7/site-packages/psutil/_pslinux.py:477: RuntimeWarning: dirty, writeback, mapped, commit_limit memory stats couldn't be ...d were set to 0
Jan 21 17:03:18 hadoop-2 cm[17500]: warnings.warn(msg, RuntimeWarning)
Hint: Some lines were ellipsized, use -l to show in full.

4.3 集群界面安装

输入server节点的地址加7180默认端口，进入登录界面，默认账号密码为admin;admin

登陆完成后进入欢迎界面，点击继续即可

用户许可条款和条件界面，勾选后，点击继续

测试环境使用的三个节点，目前没有使用到高级功能，点击免费版本即可，选择后点击继续

进入集群创建环境界面，点击继续

输入集群名称，自行定义，完成后点击继续

输入主机名称，完成后点击搜索，系统将查询出来的主机显示到下方，勾选需要部署的主机后点击继续按钮

进入选择存储库界面，cloudera manager agent栏选择自定义存储库，库地址为之前创建的保存rpm包的地址，CDH other software栏选择使用parcel，下方会展示你放在server节点/opt/cloudera/parcel-repo/目录下的包。确认都勾选完毕后，点击继续。

进入JDK 安装选项，如果提前每台机器都部署好了jdk，这里可以不用勾选，勾选的话，/var/www/html目录下面要有java的rpm包。根据实际情况设置，完成后点击继续。（按本文档部署无需勾选）

进入提供ssh登录凭证界面，输入密码后，修改ssh端口为实际的端口（生产环境可能端口不是22），完成后点击继续

进入 install agents 界面，此处会自动完成agents的安装，等待完成后，点击继续

进入install parcels界面，此处也是全自动化安装，等待完成即可。

inspect network performance:检测网络性能

inspect hosts：检测主机隐患

按照本文档部署，理论上此处检测都是一次性通过，如果结果如下图所示，则点击继续即可。

4.4集群界面设置

进入部署服务选择界面，根据需要添加需要部署的服务。

为勾选的服务分配主机，这个需要提前根据自己部署的服务器的性能分配好，哪些节点部署哪些服务，下图只是参考，选择完毕后点击继续。

进入数据库设置界面，将之前创建的库，用户名，密码分别配置到对应分服务下面，点击测试连接，提示都是successful后，点击继续。

进入审核更改界面，这里的datanode配置的地址的硬盘需要越大越好。其他无需变更，点击继续即可。

进入命令详细信息界面，此处为自动化部署，等待即可，出现下图界面，点击继续即可。

4.5 HDFS及YARN高可用

4. 5.1 HDFS高可用

点击集群下面的HDFS，进入HDFS管理界面

点击操作，然后选择下面的启用High Acailability(PS:这边是部署完成后写的，所以展示的是禁用High Acailability)

进入入门界面，输入需要取的集群名称，点击继续即可。

根据实际情况选择需要部署的主机，完成后点击继续

进入审核更改界面，目录配置同之前说的一样，配置在容量足够大的目录下。

进入命令详细信息界面，此处为全自动化部署，下图报错可以忽略，完成后点击继续。

根据提示完成操作，然后点击完成。

4.5.2 YARN 高可用

点击操作，然后选择下面的启用High Acailability

进入入门界面，选择好备机后，点击继续

进入命令详细信息界面，此处为全程自动化部署，等待即可，出现下图界面说明部署完成，点击完成即可。

4.6 部分报错解决

4.6.1 Erasure Coding Policy Verification Test

问题分析
纠删码策略警告是由于当前的HDFS集群DataNode节点数不满足当前所配置的纠删码策略需要的最小主机数
纠删码策略是HDFS集群提供的数据恢复的安全机制
其中
- RS-3-2-1024k策略需要至少5台DataNode节点
- RS-6-3-1024k策略需要至少9台DataNode节点
- RS-10-4-1024k策略需要至少14台DataNode节点
问题解决
这里我们的集群无法满足最低的5台DataNode节点要求，所以可以选择关闭纠删策略

点击报错Erasure Coding Policy Verification Test

进入Erasure Coding Policy Verification Test界面，点击为此服务更改Fallback Etasure Coding Policy

重启HSDF生效

CDH6.3.2安装（rhel7.6)相关推荐

cdh6.2离线安装（傻瓜式安装教程）
CDH6.2离线安装流程 1 安装准备 1.1 cdh6.2 下载方式一: 官网下载 https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x ...
转载：CDH6.2离线安装（整理版）
引用地址:https://www.cnblogs.com/swordfall/p/10816797.html CDH6.2离线安装(整理版) 1.概述 CDH,全称Cloudera's Distrib ...
联想服务器linux系统raid驱动,ThinkSystem服务器RAID 530/930系列阵列卡驱动及安装RHEL7.3要点说明...
知识点分析: 本文介绍在配置为RAID 530/930系列阵列卡的 ThinkSystem ST550 ThinkSystem ST558 ThinkSystem SR530 ThinkSystem ...
CDH6.3.1安装
CDH6.3.1安装遇到很多问题,我想主要是由于条件有限,毕竟自己的电脑内存不如专业集群的内存大(如果是内存和硬盘充足,有些是可以避免的,甚至不会出现报错的情况),这里就介绍一下我用VMware安装的 ...
运维之红帽管理员篇-----1. Linux系统简介、安装RHEL7系统、 RHEL7基本操作
ONE DAY 总结: 1970-1-1 Linus Torwalds clone-vm7 cat /etc/redhat-release name -r 挂载点: 在Linux访问设备资源的访问 ...
ThinkSystem 服务器RAID 530/930系列阵列卡驱动及安装RHEL7.3
在配置为RAID 530/930系列阵列卡的 ThinkSystem ST550 ThinkSystem ST558 ThinkSystem SR530 ThinkSystem SR550 Think ...
CDH6.2.0安装、踩坑实录
写在前面官方资料 ClouderaManager下载地址 https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/ manage ...
CDH6.2离线安装（整理版）
1.概述 CDH,全称Cloudera's Distribution, including Apache Hadoop.是Hadoop众多分支中对应中的一种,由Cloudera维护,基于稳定版本的Ap ...
Cloudera Manager 和CDH6.0.1安装，卸载，各步骤截图（此博文为笔者辛苦劳作最终生成的，使用了3个熬到凌晨2~4点的夜晚，外加一个周末完成，请转载时记录转载之处，谢谢）
1 Cloudera 文档总地址 https://www.cloudera.com/documentation/enterprise/6/6.0.html 2 Cloudera Manager 和CD ...

CDH6.3.2安装（rhel7.6)