使用ansible部署CDH 5.15.1大数据集群
使用ansible离线部署CDH 5.15.1大数据集群
作者:尹正杰
版权声明:原创作品,谢绝转载!否则将追究法律责任。
在此之前,我之前分享过使用shell自定义脚本部署大数据集群,不管是部署CDH集群还是部署HDP集群基本上套路都一样,调优的方式也都大同小异。只是操作界面上可能存在一定的差异。最近赶上公司搞自动化运维项目,我们用到了ansible的一些功能,本篇博客主要记录ansible部署CDH服务。
一.安装ansible
1>.操作环境介绍
注意,请检查你的虚拟机是否可以正常联网,如果不能联网的话,本篇博客估计也就只能成为你的一个参考部署方案了!由于Mac内存有限,我只开了3台虚拟机,使用的操作系统是CentOS 7.2发行版的,内存分配每个节点4G,大家根据自己的机器适当调大或调小内存。详细信息如下:
[root@node101 ~]# hostname node101.yinzhengjie.org.cn #这是本篇博客所说的主节点,该节点配置必须高于其他节点配置或者配置不能低于2G内存!否则你在后期安装服务时,可能会遇到各种坑!(我测试时,三台主机分配过4,1,1,或者3,3,3基本上都可以正常运行!) [root@node101 ~]# [root@node101 ~]# hostname -i 172.30.1.101 [root@node101 ~]# [root@node101 ~]# cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) [root@node101 ~]# [root@node101 ~]# [root@node101 ~]# free -h total used free shared buff/cache available Mem: 5.9G 120M 5.5G 6.7M 213M 713M Swap: 2.0G 0B 2.0G [root@node101 ~]# [root@node101 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup-lv_root 50G 3.8G 43G 9% / devtmpfs 480M 0 480M 0% /dev tmpfs 495M 0 495M 0% /dev/shm tmpfs 495M 6.7M 488M 2% /run tmpfs 495M 0 495M 0% /sys/fs/cgroup /dev/sda1 477M 133M 315M 30% /boot /dev/mapper/VolGroup-lv_home 12G 41M 11G 1% /home Home 234G 156G 79G 67% /media/psf/Home tmpfs 99M 0 99M 0% /run/user/0 [root@node101 ~]# [root@node101 ~]# cat /etc/hosts | grep yinzhengjie 172.30.1.101 node101.yinzhengjie.org.cn 172.30.1.102 node102.yinzhengjie.org.cn 172.30.1.103 node103.yinzhengjie.org.cn [root@node101 ~]# [root@node101 ~]#
2>.在主节点(node101.yinzhengjie.org.cn)配置至各个节点的免密钥登录
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa Generating public/private rsa key pair. Created directory '/root/.ssh'. Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: 7b:9a:3a:cd:0b:7b:b5:5e:da:ad:b8:dd:aa:0a:b1:fb root@node101.yinzhengjie.org.cn The key's randomart image is: +--[ RSA 2048]----+ | | | | | | | | | . S | | o .. | | +o..... | | .=++.* o | | +=E+*o=oo | +-----------------+ [root@node101 ~]#
生成密钥([root@node101 ~]# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa)
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# ssh-copy-id root@node101.yinzhengjie.org.cn The authenticity of host 'node101.yinzhengjie.org.cn (172.30.1.101)' can't be established. ECDSA key fingerprint is 72:2c:15:15:05:ba:4a:73:6d:5b:88:c4:bd:ab:c2:66. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@node101.yinzhengjie.org.cn's password: Number of key(s) added: 1Now try logging into the machine, with: "ssh 'root@node101.yinzhengjie.org.cn'" and check to make sure that only the key(s) you wanted were added.[root@node101 ~]# [root@node101 ~]# who root pts/0 2019-01-10 16:19 (172.30.1.2) [root@node101 ~]# [root@node101 ~]# ssh node101.yinzhengjie.org.cn Last login: Thu Jan 10 16:19:18 2019 from 172.30.1.2 [root@node101 ~]# [root@node101 ~]# who root pts/0 2019-01-10 16:19 (172.30.1.2) root pts/1 2019-01-10 16:30 (node101.yinzhengjie.org.cn) [root@node101 ~]# [root@node101 ~]# exit logout Connection to node101.yinzhengjie.org.cn closed. [root@node101 ~]# [root@node101 ~]# who root pts/0 2019-01-10 16:19 (172.30.1.2) [root@node101 ~]# [root@node101 ~]#
与本机配置免密登录([root@node101 ~]# ssh-copy-id root@node101.yinzhengjie.org.cn)
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# ssh-copy-id root@node102.yinzhengjie.org.cn The authenticity of host 'node102.yinzhengjie.org.cn (172.30.1.102)' can't be established. ECDSA key fingerprint is 85:f4:84:e1:7a:13:45:31:22:7b:12:49:5d:1b:56:ec. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@node102.yinzhengjie.org.cn's password: Number of key(s) added: 1Now try logging into the machine, with: "ssh 'root@node102.yinzhengjie.org.cn'" and check to make sure that only the key(s) you wanted were added.[root@node101 ~]# [root@node101 ~]# ssh node102.yinzhengjie.org.cn Last login: Thu Jan 10 15:46:07 2019 from 172.30.1.2 [root@node102 ~]# [root@node102 ~]# who root pts/0 2019-01-10 15:46 (172.30.1.2) root pts/1 2019-01-10 16:31 (172.30.1.101) [root@node102 ~]# [root@node102 ~]# exit logout Connection to node102.yinzhengjie.org.cn closed. [root@node101 ~]# [root@node101 ~]#
与node102.yinzhengjie.org.cn节点配置免密登录([root@node101 ~]# ssh-copy-id root@node102.yinzhengjie.org.cn)
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# ssh-copy-id root@node103.yinzhengjie.org.cn The authenticity of host 'node103.yinzhengjie.org.cn (172.30.1.103)' can't be established. ECDSA key fingerprint is 62:e7:04:30:ce:10:0d:55:9d:e5:24:9f:19:ab:4c:e7. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@node103.yinzhengjie.org.cn's password: Number of key(s) added: 1Now try logging into the machine, with: "ssh 'root@node103.yinzhengjie.org.cn'" and check to make sure that only the key(s) you wanted were added.[root@node101 ~]# [root@node101 ~]# ssh node103.yinzhengjie.org.cn Last login: Thu Jan 10 15:46:08 2019 from 172.30.1.2 [root@node103 ~]# [root@node103 ~]# who root pts/0 2019-01-10 15:46 (172.30.1.2) root pts/1 2019-01-10 16:32 (172.30.1.101) [root@node103 ~]# [root@node103 ~]# exit logout Connection to node103.yinzhengjie.org.cn closed. [root@node101 ~]# [root@node101 ~]#
与node103.yinzhengjie.org.cn节点配置免密登录([root@node101 ~]# ssh-copy-id root@node103.yinzhengjie.org.cn)
3>.使用yum工具包安装ansible
安装ansible依赖python2-pyasn1,PyYAML,python-jinja2等模块,使用yum方式安装,他会自动帮我们处理依赖关系,我们配置好yum源即可。
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# yum -y install wget Loaded plugins: fastestmirror base | 3.6 kB 00:00:00 extras | 3.4 kB 00:00:00 updates | 3.4 kB 00:00:00 updates/7/x86_64/primary_db FAILED http://mirrors.cqu.edu.cn/CentOS/7.6.1810/updates/x86_64/repodata/1c8b4fadfe3ee3539b33616064daaae4a96b2d659235607042e44c7cc389e0dc-primary.sqlite.bz2: [Errno 14] curl#56 - "Recv failure: Connection reset by peer" Trying other mirror. (1/4): extras/7/x86_64/primary_db | 156 kB 00:00:00 (2/4): base/7/x86_64/group_gz | 166 kB 00:00:02 (3/4): updates/7/x86_64/primary_db | 2.4 MB 00:00:03 (4/4): base/7/x86_64/primary_db | 6.0 MB 00:00:08 Determining fastest mirrors* base: mirrors.tuna.tsinghua.edu.cn* extras: mirrors.aliyun.com* updates: mirrors.tuna.tsinghua.edu.cn Resolving Dependencies --> Running transaction check ---> Package wget.x86_64 0:1.14-18.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved===================================================================================================================================================================Package Arch Version Repository Size =================================================================================================================================================================== Installing:wget x86_64 1.14-18.el7 base 547 kTransaction Summary =================================================================================================================================================================== Install 1 PackageTotal download size: 547 k Installed size: 2.0 M Downloading packages: warning: /var/cache/yum/x86_64/7/base/packages/wget-1.14-18.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY Public key for wget-1.14-18.el7.x86_64.rpm is not installed wget-1.14-18.el7.x86_64.rpm | 547 kB 00:00:00 Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 Importing GPG key 0xF4A80EB5:Userid : "CentOS-7 Key (CentOS 7 Official Signing Key) <security@centos.org>"Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5Package : centos-release-7-2.1511.el7.centos.2.10.x86_64 (@anaconda)From : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 Running transaction check Running transaction test Transaction test succeeded Running transactionInstalling : wget-1.14-18.el7.x86_64 1/1 Verifying : wget-1.14-18.el7.x86_64 1/1 Installed:wget.x86_64 0:1.14-18.el7 Complete! [root@node101 ~]#
安装下载工具([root@node101 ~]# yum -y install wget)
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# wget http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm --2019-01-10 17:14:45-- http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm Resolving dl.fedoraproject.org (dl.fedoraproject.org)... 209.132.181.23, 209.132.181.24 Connecting to dl.fedoraproject.org (dl.fedoraproject.org)|209.132.181.23|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 15080 (15K) [application/x-rpm] Saving to: ‘epel-release-latest-7.noarch.rpm’100%[================================================================================================================================>] 15,080 78.2KB/s in 0.2s 2019-01-10 17:14:46 (78.2 KB/s) - ‘epel-release-latest-7.noarch.rpm’ saved [15080/15080][root@node101 ~]#
下载epel源([root@node101 ~]# wget http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm)
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# ll total 16 -rw-r--r--. 1 root root 15080 Jan 10 17:00 epel-release-latest-7.noarch.rpm [root@node101 ~]# [root@node101 ~]# rpm -ivh epel-release-latest-7.noarch.rpm warning: epel-release-latest-7.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY Preparing... ################################# [100%] Updating / installing...1:epel-release-7-11 ################################# [100%] [root@node101 ~]# [root@node101 ~]# ll /etc/yum.repos.d/ | grep epel -rw-r--r--. 1 root root 951 Oct 3 2017 epel.repo -rw-r--r--. 1 root root 1050 Oct 3 2017 epel-testing.repo [root@node101 ~]#
安装epel源([root@node101 ~]# rpm -ivh epel-release-latest-7.noarch.rpm )
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# cat /etc/yum.conf | grep -v ^# |grep -v ^$ [main] cachedir=/var/cache/yum/$basearch/$releasever keepcache=0 debuglevel=2 logfile=/var/log/yum.log exactarch=1 obsoletes=1 gpgcheck=1 plugins=1 installonly_limit=5 bugtracker_url=http://bugs.centos.org/set_project.php?project_id=23&ref=http://bugs.centos.org/bug_report_page.php?category=yum distroverpkg=centos-release [root@node101 ~]# [root@node101 ~]# sed -i 's#keepcache=0#keepcache=1#' /etc/yum.conf [root@node101 ~]# [root@node101 ~]# cat /etc/yum.conf | grep -v ^# |grep -v ^$ [main] cachedir=/var/cache/yum/$basearch/$releasever keepcache=1 debuglevel=2 logfile=/var/log/yum.log exactarch=1 obsoletes=1 gpgcheck=1 plugins=1 installonly_limit=5 bugtracker_url=http://bugs.centos.org/set_project.php?project_id=23&ref=http://bugs.centos.org/bug_report_page.php?category=yum distroverpkg=centos-release [root@node101 ~]#
修改yum的配置文件,使得下载rpm安装包后不删除,方便咱们以后自建内网的yum仓库([root@node101 ~]# sed -i 's#keepcache=0#keepcache=1#' /etc/yum.conf)
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# yum clean all Loaded plugins: fastestmirror, langpacks Cleaning repos: epel Cleaning up everything [root@node101 ~]#
清空之前的缓存([root@node101 ~]# yum clean all)
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# yum makecache Loaded plugins: fastestmirror, langpacks base | 3.6 kB 00:00:00 epel/x86_64/metalink | 8.1 kB 00:00:00 epel | 3.2 kB 00:00:00 extras | 3.4 kB 00:00:00 updates | 3.4 kB 00:00:00 (1/18): epel/x86_64/group_gz | 88 kB 00:00:00 (2/18): base/7/x86_64/other_db | 2.6 MB 00:00:06 (3/18): base/7/x86_64/group_gz | 166 kB 00:00:08 (4/18): epel/x86_64/prestodelta | 803 B 00:00:11 (5/18): epel/x86_64/updateinfo | 948 kB 00:00:13 (6/18): epel/x86_64/filelists | 10 MB 00:00:20 (7/18): epel/x86_64/other | 2.3 MB 00:00:06 (8/18): base/7/x86_64/filelists_db | 7.1 MB 00:00:27 (9/18): extras/7/x86_64/primary_db | 156 kB 00:00:01 (10/18): extras/7/x86_64/other_db | 107 kB 00:00:01 (11/18): base/7/x86_64/primary_db | 6.0 MB 00:00:33 (12/18): updates/7/x86_64/prestodelta | 183 kB 00:00:03 (13/18): extras/7/x86_64/filelists_db | 189 kB 00:00:22 (14/18): updates/7/x86_64/filelists_db | 1.4 MB 00:00:14 (15/18): updates/7/x86_64/primary_db | 1.3 MB 00:00:14 extras/7/x86_64/prestodelta FAILED ========================================================-] 200 kB/s | 36 MB 00:00:01 ETA http://mirrors.njupt.edu.cn/centos/7.6.1810/extras/x86_64/repodata/9718255a5fa57675311c5ac73719bee470f94b10d38b5aa988511b9ddd860025-prestodelta.xml.gz: [Errno 12] Timeout on http://10.10.254.10/cache/4/02/mirrors.njupt.edu.cn/4f9b5de52f938c8378fba16c668a0a55/9718255a5fa57675311c5ac73719bee470f94b10d38b5aa988511b9ddd860025-prestodelta.xml.gz: (28, 'Connection timed out after 30884 milliseconds') Trying other mirror. (16/18): extras/7/x86_64/prestodelta | 36 kB 00:00:00 (17/18): epel/x86_64/primary | 3.6 MB 00:00:38 updates/7/x86_64/other_db FAILED http://mirrors.njupt.edu.cn/centos/7.6.1810/updates/x86_64/repodata/1fab468ed066d5469823091845179165040deb4951923d854a025e3b5c903051-other.sqlite.bz2: [Errno 12] Timeout on http://10.10.254.10/cache/5/02/mirrors.njupt.edu.cn/08d3e890295133ac7140ae59fc9dee3f/1fab468ed066d5469823091845179165040deb4951923d854a025e3b5c903051-other.sqlite.bz2: (28, 'Connection timed out after 30866 milliseconds') Trying other mirror. (18/18): updates/7/x86_64/other_db | 191 kB 00:00:00 Determining fastest mirrors* base: mirror.jdcloud.com* epel: mirrors.yun-idc.com* extras: mirror.jdcloud.com* updates: mirrors.tuna.tsinghua.edu.cn epel 12852/12852 epel 12852/12852 epel 12852/12852 Metadata Cache Created [root@node101 ~]#
创建yum缓存([root@node101 ~]# yum makecache)
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# yum -y install ansible Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile* base: mirror.jdcloud.com* epel: mirrors.yun-idc.com* extras: mirror.jdcloud.com* updates: mirrors.tuna.tsinghua.edu.cn Resolving Dependencies --> Running transaction check ---> Package ansible.noarch 0:2.7.5-1.el7 will be installed --> Processing Dependency: sshpass for package: ansible-2.7.5-1.el7.noarch --> Processing Dependency: python2-jmespath for package: ansible-2.7.5-1.el7.noarch --> Processing Dependency: python-paramiko for package: ansible-2.7.5-1.el7.noarch --> Processing Dependency: python-keyczar for package: ansible-2.7.5-1.el7.noarch --> Processing Dependency: python-jinja2 for package: ansible-2.7.5-1.el7.noarch --> Processing Dependency: python-httplib2 for package: ansible-2.7.5-1.el7.noarch --> Processing Dependency: python-crypto for package: ansible-2.7.5-1.el7.noarch --> Processing Dependency: PyYAML for package: ansible-2.7.5-1.el7.noarch --> Running transaction check ---> Package PyYAML.x86_64 0:3.10-11.el7 will be installed --> Processing Dependency: libyaml-0.so.2()(64bit) for package: PyYAML-3.10-11.el7.x86_64 ---> Package python-httplib2.noarch 0:0.9.2-1.el7 will be installed ---> Package python-jinja2.noarch 0:2.7.2-2.el7 will be installed --> Processing Dependency: python-babel >= 0.8 for package: python-jinja2-2.7.2-2.el7.noarch --> Processing Dependency: python-markupsafe for package: python-jinja2-2.7.2-2.el7.noarch ---> Package python-keyczar.noarch 0:0.71c-2.el7 will be installed ---> Package python-paramiko.noarch 0:2.1.1-9.el7 will be installed --> Processing Dependency: python2-pyasn1 for package: python-paramiko-2.1.1-9.el7.noarch ---> Package python2-crypto.x86_64 0:2.6.1-15.el7 will be installed --> Processing Dependency: libtomcrypt.so.0()(64bit) for package: python2-crypto-2.6.1-15.el7.x86_64 ---> Package python2-jmespath.noarch 0:0.9.0-3.el7 will be installed ---> Package sshpass.x86_64 0:1.06-2.el7 will be installed --> Running transaction check ---> Package libtomcrypt.x86_64 0:1.17-26.el7 will be installed --> Processing Dependency: libtommath >= 0.42.0 for package: libtomcrypt-1.17-26.el7.x86_64 --> Processing Dependency: libtommath.so.0()(64bit) for package: libtomcrypt-1.17-26.el7.x86_64 ---> Package libyaml.x86_64 0:0.1.4-11.el7_0 will be installed ---> Package python-babel.noarch 0:0.9.6-8.el7 will be installed ---> Package python-markupsafe.x86_64 0:0.11-10.el7 will be installed ---> Package python-pyasn1.noarch 0:0.1.6-2.el7 will be obsoleted ---> Package python2-pyasn1.noarch 0:0.1.9-7.el7 will be obsoleting --> Running transaction check ---> Package libtommath.x86_64 0:0.42.0-6.el7 will be installed --> Finished Dependency ResolutionDependencies Resolved==========================================================================================================================================================================Package Arch Version Repository Size ========================================================================================================================================================================== Installing:ansible noarch 2.7.5-1.el7 epel 11 Mpython2-pyasn1 noarch 0.1.9-7.el7 base 100 kreplacing python-pyasn1.noarch 0.1.6-2.el7 Installing for dependencies:PyYAML x86_64 3.10-11.el7 base 153 klibtomcrypt x86_64 1.17-26.el7 extras 224 klibtommath x86_64 0.42.0-6.el7 extras 36 klibyaml x86_64 0.1.4-11.el7_0 base 55 kpython-babel noarch 0.9.6-8.el7 base 1.4 Mpython-httplib2 noarch 0.9.2-1.el7 extras 115 kpython-jinja2 noarch 2.7.2-2.el7 base 515 kpython-keyczar noarch 0.71c-2.el7 epel 218 kpython-markupsafe x86_64 0.11-10.el7 base 25 kpython-paramiko noarch 2.1.1-9.el7 updates 269 kpython2-crypto x86_64 2.6.1-15.el7 extras 477 kpython2-jmespath noarch 0.9.0-3.el7 extras 39 ksshpass x86_64 1.06-2.el7 extras 21 kTransaction Summary ========================================================================================================================================================================== Install 2 Packages (+13 Dependent packages)Total download size: 15 M Downloading packages: (1/15): PyYAML-3.10-11.el7.x86_64.rpm | 153 kB 00:00:00 (2/15): libtomcrypt-1.17-26.el7.x86_64.rpm | 224 kB 00:00:00 (3/15): python-httplib2-0.9.2-1.el7.noarch.rpm | 115 kB 00:00:00 (4/15): python-jinja2-2.7.2-2.el7.noarch.rpm | 515 kB 00:00:00 (5/15): libtommath-0.42.0-6.el7.x86_64.rpm | 36 kB 00:00:01 (6/15): python-markupsafe-0.11-10.el7.x86_64.rpm | 25 kB 00:00:00 (7/15): python-paramiko-2.1.1-9.el7.noarch.rpm | 269 kB 00:00:00 (8/15): python2-crypto-2.6.1-15.el7.x86_64.rpm | 477 kB 00:00:00 (9/15): python2-jmespath-0.9.0-3.el7.noarch.rpm | 39 kB 00:00:00 (10/15): python2-pyasn1-0.1.9-7.el7.noarch.rpm | 100 kB 00:00:00 (11/15): sshpass-1.06-2.el7.x86_64.rpm | 21 kB 00:00:00 (12/15): python-babel-0.9.6-8.el7.noarch.rpm | 1.4 MB 00:00:04 (13/15): libyaml-0.1.4-11.el7_0.x86_64.rpm | 55 kB 00:00:05 warning: /var/cache/yum/x86_64/7/epel/packages/python-keyczar-0.71c-2.el7.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY7 kB/s | 9.4 MB 00:00:06 ETA Public key for python-keyczar-0.71c-2.el7.noarch.rpm is not installed (14/15): python-keyczar-0.71c-2.el7.noarch.rpm | 218 kB 00:00:09 (15/15): ansible-2.7.5-1.el7.noarch.rpm | 11 MB 00:00:17 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 870 kB/s | 15 MB 00:00:17 Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 Importing GPG key 0x352C64E5:Userid : "Fedora EPEL (7) <epel@fedoraproject.org>"Fingerprint: 91e9 7d7c 4a5e 96f1 7f3e 888f 6a2f aea2 352c 64e5Package : epel-release-7-11.noarch (installed)From : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 Running transaction check Running transaction test Transaction test succeeded Running transaction Warning: RPMDB altered outside of yum.Installing : python2-pyasn1-0.1.9-7.el7.noarch 1/16 Installing : python-paramiko-2.1.1-9.el7.noarch 2/16 Installing : python-babel-0.9.6-8.el7.noarch 3/16 Installing : python-httplib2-0.9.2-1.el7.noarch 4/16 Installing : sshpass-1.06-2.el7.x86_64 5/16 Installing : libtommath-0.42.0-6.el7.x86_64 6/16 Installing : libtomcrypt-1.17-26.el7.x86_64 7/16 Installing : python2-crypto-2.6.1-15.el7.x86_64 8/16 Installing : python-keyczar-0.71c-2.el7.noarch 9/16 Installing : python-markupsafe-0.11-10.el7.x86_64 10/16 Installing : python-jinja2-2.7.2-2.el7.noarch 11/16 Installing : python2-jmespath-0.9.0-3.el7.noarch 12/16 Installing : libyaml-0.1.4-11.el7_0.x86_64 13/16 Installing : PyYAML-3.10-11.el7.x86_64 14/16 Installing : ansible-2.7.5-1.el7.noarch 15/16 Erasing : python-pyasn1-0.1.6-2.el7.noarch 16/16 Verifying : python-keyczar-0.71c-2.el7.noarch 1/16 Verifying : libyaml-0.1.4-11.el7_0.x86_64 2/16 Verifying : python-jinja2-2.7.2-2.el7.noarch 3/16 Verifying : python2-jmespath-0.9.0-3.el7.noarch 4/16 Verifying : python-paramiko-2.1.1-9.el7.noarch 5/16 Verifying : python-markupsafe-0.11-10.el7.x86_64 6/16 Verifying : python2-crypto-2.6.1-15.el7.x86_64 7/16 Verifying : libtommath-0.42.0-6.el7.x86_64 8/16 Verifying : sshpass-1.06-2.el7.x86_64 9/16 Verifying : python-httplib2-0.9.2-1.el7.noarch 10/16 Verifying : ansible-2.7.5-1.el7.noarch 11/16 Verifying : PyYAML-3.10-11.el7.x86_64 12/16 Verifying : python-babel-0.9.6-8.el7.noarch 13/16 Verifying : libtomcrypt-1.17-26.el7.x86_64 14/16 Verifying : python2-pyasn1-0.1.9-7.el7.noarch 15/16 Verifying : python-pyasn1-0.1.6-2.el7.noarch 16/16 Installed:ansible.noarch 0:2.7.5-1.el7 python2-pyasn1.noarch 0:0.1.9-7.el7 Dependency Installed:PyYAML.x86_64 0:3.10-11.el7 libtomcrypt.x86_64 0:1.17-26.el7 libtommath.x86_64 0:0.42.0-6.el7 libyaml.x86_64 0:0.1.4-11.el7_0 python-babel.noarch 0:0.9.6-8.el7 python-httplib2.noarch 0:0.9.2-1.el7 python-jinja2.noarch 0:2.7.2-2.el7 python-keyczar.noarch 0:0.71c-2.el7 python-markupsafe.x86_64 0:0.11-10.el7 python-paramiko.noarch 0:2.1.1-9.el7 python2-crypto.x86_64 0:2.6.1-15.el7 python2-jmespath.noarch 0:0.9.0-3.el7 sshpass.x86_64 0:1.06-2.el7 Replaced:python-pyasn1.noarch 0:0.1.6-2.el7 Complete! [root@node101 ~]#
安装ansible([root@node101 ~]# yum -y install ansible)
4>.配置ansible的配置文件
我根据我的需求,将主机分为2个角色,大家都知道,hdfs集群又NameNode和DataNode之分,因此我将配置相对较高的那台机器配置为NameNode节点,然后把DataNode服务安置在所有的节点上。
[root@node101 ~]# tail -8 /etc/ansible/hosts #Add by yinzhengjie [namenode] node101.yinzhengjie.org.cn[datanode] node101.yinzhengjie.org.cn node102.yinzhengjie.org.cn node103.yinzhengjie.org.cn [root@node101 ~]# [root@node101 ~]#
5>.安装相关依赖插件,解决cryptography 0.8.2 版本兼容性(如果该步骤不做的话,在你执行“ansible all -m ping”命令时可能会报错!)
5.1>.安装pip
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# wget https://files.pythonhosted.org/packages/45/ae/8a0ad77defb7cc903f09e551d88b443304a9bd6e6f124e75c0fbbf6de8f7/pip-18.1.tar.gz --2019-01-10 18:11:31-- https://files.pythonhosted.org/packages/45/ae/8a0ad77defb7cc903f09e551d88b443304a9bd6e6f124e75c0fbbf6de8f7/pip-18.1.tar.gz Resolving files.pythonhosted.org (files.pythonhosted.org)... 151.101.109.63, 2a04:4e42:36::319 Connecting to files.pythonhosted.org (files.pythonhosted.org)|151.101.109.63|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 1259370 (1.2M) [binary/octet-stream] Saving to: ‘pip-18.1.tar.gz’100%[================================================================================================================================>] 1,259,370 64.7KB/s in 22s 2019-01-10 18:11:55 (55.4 KB/s) - ‘pip-18.1.tar.gz’ saved [1259370/1259370][root@node101 ~]#
下载pip([root@node101 ~]# wget https://files.pythonhosted.org/packages/45/ae/8a0ad77defb7cc903f09e551d88b443304a9bd6e6f124e75c0fbbf6de8f7/pip-18.1.tar.gz)
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# tar -zxf pip-18.1.tar.gz [root@node101 ~]# ll total 1252 -rw-r--r--. 1 root root 15080 Oct 3 2017 epel-release-latest-7.noarch.rpm drwxr-xr-x. 6 501 games 4096 Jan 10 18:13 pip-18.1 -rw-r--r--. 1 root root 1259370 Oct 5 19:20 pip-18.1.tar.gz [root@node101 ~]# [root@node101 ~]# cd pip-18.1/ [root@node101 pip-18.1]# [root@node101 pip-18.1]# python setup.py install .........(内容输出过多,我这里就统一复制了!) Installing pip2.7 script to /usr/bin Installing pip2 script to /usr/binInstalled /usr/lib/python2.7/site-packages/pip-18.1-py2.7.egg Processing dependencies for pip==18.1 Finished processing dependencies for pip==18.1 [root@node101 pip-18.1]#
安装pip工具([root@node101 pip-18.1]# python setup.py install)
5.2>.安装wheel
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# wget https://files.pythonhosted.org/packages/d8/55/221a530d66bf78e72996453d1e2dedef526063546e131d70bed548d80588/wheel-0.32.3.tar.gz --2019-01-10 18:16:37-- https://files.pythonhosted.org/packages/d8/55/221a530d66bf78e72996453d1e2dedef526063546e131d70bed548d80588/wheel-0.32.3.tar.gz Resolving files.pythonhosted.org (files.pythonhosted.org)... 151.101.109.63, 2a04:4e42:1a::319 Connecting to files.pythonhosted.org (files.pythonhosted.org)|151.101.109.63|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 18585 (18K) [binary/octet-stream] Saving to: ‘wheel-0.32.3.tar.gz’100%[================================================================================================================================>] 18,585 --.-K/s in 0.1s 2019-01-10 18:16:38 (162 KB/s) - ‘wheel-0.32.3.tar.gz’ saved [18585/18585][root@node101 ~]# ll total 1272 -rw-r--r--. 1 root root 15080 Oct 3 2017 epel-release-latest-7.noarch.rpm drwxr-xr-x. 6 501 games 4096 Jan 10 18:13 pip-18.1 -rw-r--r--. 1 root root 1259370 Oct 5 19:20 pip-18.1.tar.gz -rw-r--r--. 1 root root 18585 Nov 19 08:26 wheel-0.32.3.tar.gz [root@node101 ~]# [root@node101 ~]#
下载wheel([root@node101 ~]# wget https://files.pythonhosted.org/packages/d8/55/221a530d66bf78e72996453d1e2dedef526063546e131d70bed548d80588/wheel-0.32.3.tar.gz)
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# ll total 1272 -rw-r--r--. 1 root root 15080 Oct 3 2017 epel-release-latest-7.noarch.rpm drwxr-xr-x. 6 501 games 4096 Jan 10 18:13 pip-18.1 -rw-r--r--. 1 root root 1259370 Oct 5 19:20 pip-18.1.tar.gz -rw-r--r--. 1 root root 18585 Nov 19 08:26 wheel-0.32.3.tar.gz [root@node101 ~]# [root@node101 ~]# tar -zxf wheel-0.32.3.tar.gz [root@node101 ~]# [root@node101 ~]# ll total 1276 -rw-r--r--. 1 root root 15080 Oct 3 2017 epel-release-latest-7.noarch.rpm drwxr-xr-x. 6 501 games 4096 Jan 10 18:13 pip-18.1 -rw-r--r--. 1 root root 1259370 Oct 5 19:20 pip-18.1.tar.gz drwxrwxr-x. 4 2000 2000 4096 Nov 19 08:25 wheel-0.32.3 -rw-r--r--. 1 root root 18585 Nov 19 08:26 wheel-0.32.3.tar.gz [root@node101 ~]# [root@node101 ~]# cd wheel-0.32.3/ [root@node101 wheel-0.32.3]# ll total 32 -rw-rw-r--. 1 2000 2000 1125 Nov 19 08:25 LICENSE.txt -rw-rw-r--. 1 2000 2000 244 Nov 19 08:25 MANIFEST.in -rw-rw-r--. 1 2000 2000 2207 Nov 19 08:25 PKG-INFO -rw-rw-r--. 1 2000 2000 799 Nov 19 08:25 README.rst -rw-rw-r--. 1 2000 2000 276 Nov 19 08:25 setup.cfg -rw-rw-r--. 1 2000 2000 2133 Nov 19 08:25 setup.py drwxrwxr-x. 3 2000 2000 4096 Nov 19 08:25 wheel drwxrwxr-x. 2 2000 2000 4096 Nov 19 08:25 wheel.egg-info [root@node101 wheel-0.32.3]# [root@node101 wheel-0.32.3]# python setup.py install /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'project_urls'warnings.warn(msg) /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'warnings.warn(msg) running install running bdist_egg running egg_info writing requirements to wheel.egg-info/requires.txt writing wheel.egg-info/PKG-INFO writing top-level names to wheel.egg-info/top_level.txt writing dependency_links to wheel.egg-info/dependency_links.txt writing entry points to wheel.egg-info/entry_points.txt reading manifest file 'wheel.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no files found matching 'wheel/*.txt' warning: no files found matching '*.sh' warning: no files found matching '*.py' under directory 'wheel/test' warning: no files found matching 'wheel/test/test-1.0-py2.py3-none-win32.whl' warning: no files found matching 'wheel/test/headers.dist/header.h' warning: no files found matching 'wheel/test/pydist-schema.json' no previously-included directories found matching 'wheel/test/*/dist' no previously-included directories found matching 'wheel/test/*/build' writing manifest file 'wheel.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py creating build creating build/lib creating build/lib/wheel copying wheel/wheelfile.py -> build/lib/wheel copying wheel/pep425tags.py -> build/lib/wheel copying wheel/__init__.py -> build/lib/wheel copying wheel/util.py -> build/lib/wheel copying wheel/metadata.py -> build/lib/wheel copying wheel/__main__.py -> build/lib/wheel copying wheel/bdist_wheel.py -> build/lib/wheel copying wheel/pkginfo.py -> build/lib/wheel creating build/lib/wheel/cli copying wheel/cli/install.py -> build/lib/wheel/cli copying wheel/cli/pack.py -> build/lib/wheel/cli copying wheel/cli/__init__.py -> build/lib/wheel/cli copying wheel/cli/convert.py -> build/lib/wheel/cli copying wheel/cli/unpack.py -> build/lib/wheel/cli creating build/bdist.linux-x86_64 creating build/bdist.linux-x86_64/egg creating build/bdist.linux-x86_64/egg/wheel copying build/lib/wheel/wheelfile.py -> build/bdist.linux-x86_64/egg/wheel copying build/lib/wheel/pep425tags.py -> build/bdist.linux-x86_64/egg/wheel copying build/lib/wheel/__init__.py -> build/bdist.linux-x86_64/egg/wheel copying build/lib/wheel/util.py -> build/bdist.linux-x86_64/egg/wheel copying build/lib/wheel/metadata.py -> build/bdist.linux-x86_64/egg/wheel copying build/lib/wheel/__main__.py -> build/bdist.linux-x86_64/egg/wheel copying build/lib/wheel/bdist_wheel.py -> build/bdist.linux-x86_64/egg/wheel copying build/lib/wheel/pkginfo.py -> build/bdist.linux-x86_64/egg/wheel creating build/bdist.linux-x86_64/egg/wheel/cli copying build/lib/wheel/cli/install.py -> build/bdist.linux-x86_64/egg/wheel/cli copying build/lib/wheel/cli/pack.py -> build/bdist.linux-x86_64/egg/wheel/cli copying build/lib/wheel/cli/__init__.py -> build/bdist.linux-x86_64/egg/wheel/cli copying build/lib/wheel/cli/convert.py -> build/bdist.linux-x86_64/egg/wheel/cli copying build/lib/wheel/cli/unpack.py -> build/bdist.linux-x86_64/egg/wheel/cli byte-compiling build/bdist.linux-x86_64/egg/wheel/wheelfile.py to wheelfile.pyc byte-compiling build/bdist.linux-x86_64/egg/wheel/pep425tags.py to pep425tags.pyc byte-compiling build/bdist.linux-x86_64/egg/wheel/__init__.py to __init__.pyc byte-compiling build/bdist.linux-x86_64/egg/wheel/util.py to util.pyc byte-compiling build/bdist.linux-x86_64/egg/wheel/metadata.py to metadata.pyc byte-compiling build/bdist.linux-x86_64/egg/wheel/__main__.py to __main__.pyc byte-compiling build/bdist.linux-x86_64/egg/wheel/bdist_wheel.py to bdist_wheel.pyc byte-compiling build/bdist.linux-x86_64/egg/wheel/pkginfo.py to pkginfo.pyc byte-compiling build/bdist.linux-x86_64/egg/wheel/cli/install.py to install.pyc byte-compiling build/bdist.linux-x86_64/egg/wheel/cli/pack.py to pack.pyc byte-compiling build/bdist.linux-x86_64/egg/wheel/cli/__init__.py to __init__.pyc byte-compiling build/bdist.linux-x86_64/egg/wheel/cli/convert.py to convert.pyc byte-compiling build/bdist.linux-x86_64/egg/wheel/cli/unpack.py to unpack.pyc creating build/bdist.linux-x86_64/egg/EGG-INFO copying wheel.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO copying wheel.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying wheel.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying wheel.egg-info/entry_points.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying wheel.egg-info/not-zip-safe -> build/bdist.linux-x86_64/egg/EGG-INFO copying wheel.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying wheel.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO creating dist creating 'dist/wheel-0.32.3-py2.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it removing 'build/bdist.linux-x86_64/egg' (and everything under it) Processing wheel-0.32.3-py2.7.egg creating /usr/lib/python2.7/site-packages/wheel-0.32.3-py2.7.egg Extracting wheel-0.32.3-py2.7.egg to /usr/lib/python2.7/site-packages Adding wheel 0.32.3 to easy-install.pth file Installing wheel script to /usr/binInstalled /usr/lib/python2.7/site-packages/wheel-0.32.3-py2.7.egg Processing dependencies for wheel==0.32.3 Finished processing dependencies for wheel==0.32.3 [root@node101 wheel-0.32.3]#
安装wheel工具([root@node101 wheel-0.32.3]# python setup.py install)
5.3>.安装cryptography
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# wget https://files.pythonhosted.org/packages/7f/ba/383b51cc26e3141c689ce988814385c7659f5ba01c4b5f2de38233010b5f/cryptography-2.4.2-cp27-cp27mu-manylinux1_x86_64.whl --2019-01-10 18:23:46-- https://files.pythonhosted.org/packages/7f/ba/383b51cc26e3141c689ce988814385c7659f5ba01c4b5f2de38233010b5f/cryptography-2.4.2-cp27-cp27mu-manylinux1_x86_64.whl Resolving files.pythonhosted.org (files.pythonhosted.org)... 151.101.73.63, 2a04:4e42:11::319 Connecting to files.pythonhosted.org (files.pythonhosted.org)|151.101.73.63|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 2138565 (2.0M) [binary/octet-stream] Saving to: ‘cryptography-2.4.2-cp27-cp27mu-manylinux1_x86_64.whl’100%[================================================================================================================================>] 2,138,565 47.2KB/s in 24s 2019-01-10 18:24:11 (85.8 KB/s) - ‘cryptography-2.4.2-cp27-cp27mu-manylinux1_x86_64.whl’ saved [2138565/2138565][root@node101 ~]#
下载cryptography([root@node101 ~]# wget https://files.pythonhosted.org/packages/7f/ba/383b51cc26e3141c689ce988814385c7659f5ba01c4b5f2de38233010b5f/cryptography-2.4.2-cp27-cp27mu-manylinux1_x86_64.whl)
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# pip install cryptography-2.4.2-cp27-cp27mu-manylinux1_x86_64.whl Processing ./cryptography-2.4.2-cp27-cp27mu-manylinux1_x86_64.whl Collecting idna>=2.1 (from cryptography==2.4.2)Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)100% |████████████████████████████████| 61kB 248kB/s Requirement already satisfied: enum34; python_version < "3" in /usr/lib/python2.7/site-packages (from cryptography==2.4.2) (1.0.4) Collecting cffi!=1.11.3,>=1.7 (from cryptography==2.4.2)Downloading https://files.pythonhosted.org/packages/14/dd/3e7a1e1280e7d767bd3fa15791759c91ec19058ebe31217fe66f3e9a8c49/cffi-1.11.5-cp27-cp27mu-manylinux1_x86_64.whl (407kB)100% |████████████████████████████████| 409kB 147kB/s Requirement already satisfied: six>=1.4.1 in /usr/lib/python2.7/site-packages (from cryptography==2.4.2) (1.9.0) Collecting ipaddress; python_version < "3" (from cryptography==2.4.2)Downloading https://files.pythonhosted.org/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl Collecting asn1crypto>=0.21.0 (from cryptography==2.4.2)Downloading https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl (101kB)100% |████████████████████████████████| 102kB 292kB/s Requirement already satisfied: pycparser in /usr/lib/python2.7/site-packages (from cffi!=1.11.3,>=1.7->cryptography==2.4.2) (2.14) Installing collected packages: idna, cffi, ipaddress, asn1crypto, cryptographyFound existing installation: cffi 0.8.6Uninstalling cffi-0.8.6:Successfully uninstalled cffi-0.8.6Found existing installation: cryptography 0.8.2Uninstalling cryptography-0.8.2:Successfully uninstalled cryptography-0.8.2 Successfully installed asn1crypto-0.24.0 cffi-1.11.5 cryptography-2.4.2 idna-2.8 ipaddress-1.0.22 [root@node101 ~]#
安装cryptography([root@node101 ~]# pip install cryptography-2.4.2-cp27-cp27mu-manylinux1_x86_64.whl )
6>.使用ansible同步文件到其他节点
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# ansible all -m ping node102.yinzhengjie.org.cn | SUCCESS => {"changed": false, "ping": "pong" } node103.yinzhengjie.org.cn | SUCCESS => {"changed": false, "ping": "pong" } node101.yinzhengjie.org.cn | SUCCESS => {"changed": false, "ping": "pong" } [root@node101 ~]#
检查所有节点是否在线([root@node101 ~]# ansible all -m ping )
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# ansible datanode -m copy -a 'src=/etc/hosts dest=/etc/hosts' node101.yinzhengjie.org.cn | SUCCESS => {"changed": false, "checksum": "6b35ffc3c04d3ec2d3464b74ea0f78c20f27131a", "dest": "/etc/hosts", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/hosts", "secontext": "system_u:object_r:net_conf_t:s0", "size": 279, "state": "file", "uid": 0 } node102.yinzhengjie.org.cn | CHANGED => {"changed": true, "checksum": "6b35ffc3c04d3ec2d3464b74ea0f78c20f27131a", "dest": "/etc/hosts", "gid": 0, "group": "root", "md5sum": "494c828fc321572368ec771a390f90c1", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:net_conf_t:s0", "size": 279, "src": "/root/.ansible/tmp/ansible-tmp-1547116266.5-263369209847793/source", "state": "file", "uid": 0 } node103.yinzhengjie.org.cn | CHANGED => {"changed": true, "checksum": "6b35ffc3c04d3ec2d3464b74ea0f78c20f27131a", "dest": "/etc/hosts", "gid": 0, "group": "root", "md5sum": "494c828fc321572368ec771a390f90c1", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:net_conf_t:s0", "size": 279, "src": "/root/.ansible/tmp/ansible-tmp-1547116266.52-44859007354579/source", "state": "file", "uid": 0 } [root@node101 ~]# [root@node101 ~]# ansible all -m shell -a 'cat /etc/hosts|grep yinzhengjie' node103.yinzhengjie.org.cn | CHANGED | rc=0 >> 172.30.1.101 node101.yinzhengjie.org.cn 172.30.1.102 node102.yinzhengjie.org.cn 172.30.1.103 node103.yinzhengjie.org.cnnode101.yinzhengjie.org.cn | CHANGED | rc=0 >> 172.30.1.101 node101.yinzhengjie.org.cn 172.30.1.102 node102.yinzhengjie.org.cn 172.30.1.103 node103.yinzhengjie.org.cnnode102.yinzhengjie.org.cn | CHANGED | rc=0 >> 172.30.1.101 node101.yinzhengjie.org.cn 172.30.1.102 node102.yinzhengjie.org.cn 172.30.1.103 node103.yinzhengjie.org.cn[root@node101 ~]#
将namenode节点的数据同步到其他节点中([root@node101 ~]# ansible datanode -m copy -a 'src=/etc/hosts dest=/etc/hosts')
二.安装jdk环境
1>.卸载jdk1.7版本
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# ansible all -m shell -a 'rpm -qa | grep java'[WARNING]: Consider using the yum, dnf or zypper module rather than running rpm. If you need to use command because yum, dnf or zypper is insufficient you can add warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message.node102.yinzhengjie.org.cn | CHANGED | rc=0 >> javapackages-tools-3.4.1-11.el7.noarch tzdata-java-2015g-1.el7.noarch java-1.7.0-openjdk-headless-1.7.0.91-2.6.2.3.el7.x86_64 python-javapackages-3.4.1-11.el7.noarch java-1.7.0-openjdk-1.7.0.91-2.6.2.3.el7.x86_64node103.yinzhengjie.org.cn | CHANGED | rc=0 >> javapackages-tools-3.4.1-11.el7.noarch tzdata-java-2015g-1.el7.noarch java-1.7.0-openjdk-headless-1.7.0.91-2.6.2.3.el7.x86_64 python-javapackages-3.4.1-11.el7.noarch java-1.7.0-openjdk-1.7.0.91-2.6.2.3.el7.x86_64node101.yinzhengjie.org.cn | CHANGED | rc=0 >> javapackages-tools-3.4.1-11.el7.noarch tzdata-java-2015g-1.el7.noarch java-1.7.0-openjdk-headless-1.7.0.91-2.6.2.3.el7.x86_64 python-javapackages-3.4.1-11.el7.noarch java-1.7.0-openjdk-1.7.0.91-2.6.2.3.el7.x86_64[root@node101 ~]#
使用ansible检查各个节点是否安装jdk([root@node101 ~]# ansible all -m shell -a 'rpm -qa | grep java')
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# ansible all -m shell -a 'rpm -e --nodeps javapackages-tools-3.4.1-11.el7.noarch'[WARNING]: Consider using the yum, dnf or zypper module rather than running rpm. If you need to use command because yum, dnf or zypper is insufficient you can add warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message.node101.yinzhengjie.org.cn | CHANGED | rc=0 >>node102.yinzhengjie.org.cn | CHANGED | rc=0 >>node103.yinzhengjie.org.cn | CHANGED | rc=0 >>[root@node101 ~]#
卸载javapackages软件包( [root@node101 ~]# ansible all -m shell -a 'rpm -e --nodeps javapackages-tools-3.4.1-11.el7.noarch')
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# ansible all -m shell -a 'rpm -e --nodeps tzdata-java-2015g-1.el7.noarch' [WARNING]: Consider using the yum, dnf or zypper module rather than running rpm. If you need to use command because yum, dnf or zypper is insufficient you can add warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message.node103.yinzhengjie.org.cn | CHANGED | rc=0 >>node101.yinzhengjie.org.cn | CHANGED | rc=0 >>node102.yinzhengjie.org.cn | CHANGED | rc=0 >>[root@node101 ~]#
卸载tzdata软件包([root@node101 ~]# ansible all -m shell -a 'rpm -e --nodeps tzdata-java-2015g-1.el7.noarch' )
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# ansible all -m shell -a 'rpm -e --nodeps java-1.7.0-openjdk-headless-1.7.0.91-2.6.2.3.el7.x86_64'[WARNING]: Consider using the yum, dnf or zypper module rather than running rpm. If you need to use command because yum, dnf or zypper is insufficient you can add warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message.node101.yinzhengjie.org.cn | CHANGED | rc=0 >>node102.yinzhengjie.org.cn | CHANGED | rc=0 >>node103.yinzhengjie.org.cn | CHANGED | rc=0 >>[root@node101 ~]#
卸载openjdk软件包([root@node101 ~]# ansible all -m shell -a 'rpm -e --nodeps java-1.7.0-openjdk-headless-1.7.0.91-2.6.2.3.el7.x86_64')
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# ansible all -m shell -a 'rpm -e --nodeps python-javapackages-3.4.1-11.el7.noarch' [WARNING]: Consider using the yum, dnf or zypper module rather than running rpm. If you need to use command because yum, dnf or zypper is insufficient you can add warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message.node101.yinzhengjie.org.cn | CHANGED | rc=0 >>node102.yinzhengjie.org.cn | CHANGED | rc=0 >>node103.yinzhengjie.org.cn | CHANGED | rc=0 >>[root@node101 ~]#
[root@node101 ~]# ansible all -m shell -a 'rpm -e --nodeps python-javapackages-3.4.1-11.el7.noarch'
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# ansible all -m shell -a 'rpm -e --nodeps java-1.7.0-openjdk-1.7.0.91-2.6.2.3.el7.x86_64'[WARNING]: Consider using the yum, dnf or zypper module rather than running rpm. If you need to use command because yum, dnf or zypper is insufficient you can add warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message.node101.yinzhengjie.org.cn | CHANGED | rc=0 >>node102.yinzhengjie.org.cn | CHANGED | rc=0 >>node103.yinzhengjie.org.cn | CHANGED | rc=0 >>[root@node101 ~]#
[root@node101 ~]# ansible all -m shell -a 'rpm -e --nodeps java-1.7.0-openjdk-1.7.0.91-2.6.2.3.el7.x86_64'
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# ansible all -m shell -a 'rpm -qa | grep java' [WARNING]: Consider using the yum, dnf or zypper module rather than running rpm. If you need to use command because yum, dnf or zypper is insufficient you can add warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message.node103.yinzhengjie.org.cn | FAILED | rc=1 >> non-zero return codenode102.yinzhengjie.org.cn | FAILED | rc=1 >> non-zero return codenode101.yinzhengjie.org.cn | FAILED | rc=1 >> non-zero return code[root@node101 ~]#
检查jdk1.7相关软件包是否卸载完成([root@node101 ~]# ansible all -m shell -a 'rpm -qa | grep java' )
2>.安装jdk1.8版本几以上版本
大家可以自行去Oracle官网下载相应的jdk,我这里给出下载地址:https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html(推荐下载tar.gz的安装包)。
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# ansible all -m shell -a 'mkdir -pv /yinzhengjie/softwares'[WARNING]: Consider using the file module with state=directory rather than running mkdir. If you need to use command because file is insufficient you can add warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message.node103.yinzhengjie.org.cn | CHANGED | rc=0 >> mkdir: created directory ‘/yinzhengjie’ mkdir: created directory ‘/yinzhengjie/softwares’node101.yinzhengjie.org.cn | CHANGED | rc=0 >> mkdir: created directory ‘/yinzhengjie’ mkdir: created directory ‘/yinzhengjie/softwares’node102.yinzhengjie.org.cn | CHANGED | rc=0 >> mkdir: created directory ‘/yinzhengjie’ mkdir: created directory ‘/yinzhengjie/softwares’[root@node101 ~]#
创建指定的软件存放目录,这个目录大家可以自定义创建目录([root@node101 ~]# ansible all -m shell -a 'mkdir -pv /yinzhengjie/softwares')
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# ansible namenode -m shell -a 'tar -zxf ~/jdk-8u131-linux-x64.tar.gz -C /yinzhengjie/softwares'[WARNING]: Consider using the unarchive module rather than running tar. If you need to use command because unarchive is insufficient you can add warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message.node101.yinzhengjie.org.cn | CHANGED | rc=0 >>[root@node101 ~]# ls /yinzhengjie/softwares/ jdk1.8.0_131 [root@node101 ~]# [root@node101 ~]# ll /yinzhengjie/softwares/jdk1.8.0_131/ total 25884 drwxr-xr-x. 2 10 143 4096 Mar 15 2017 bin -r--r--r--. 1 10 143 3244 Mar 15 2017 COPYRIGHT drwxr-xr-x. 4 10 143 4096 Mar 15 2017 db drwxr-xr-x. 3 10 143 4096 Mar 15 2017 include -rwxr-xr-x. 1 10 143 5097105 Mar 15 2017 javafx-src.zip drwxr-xr-x. 5 10 143 4096 Mar 15 2017 jre drwxr-xr-x. 5 10 143 4096 Mar 15 2017 lib -r--r--r--. 1 10 143 40 Mar 15 2017 LICENSE drwxr-xr-x. 4 10 143 4096 Mar 15 2017 man -r--r--r--. 1 10 143 159 Mar 15 2017 README.html -rw-r--r--. 1 10 143 526 Mar 15 2017 release -rw-r--r--. 1 10 143 21115141 Mar 15 2017 src.zip -rwxr-xr-x. 1 10 143 63933 Mar 15 2017 THIRDPARTYLICENSEREADME-JAVAFX.txt -r--r--r--. 1 10 143 177094 Mar 15 2017 THIRDPARTYLICENSEREADME.txt [root@node101 ~]#
解压jdk软件([root@node101 ~]# ansible namenode -m shell -a 'tar -zxf ~/jdk-8u131-linux-x64.tar.gz -C /yinzhengjie/softwares')
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# ansible datanode -m copy -a 'src=/yinzhengjie/softwares/jdk1.8.0_131 dest=/yinzhengjie/softwares' node101.yinzhengjie.org.cn | CHANGED => {"changed": true, "dest": "/yinzhengjie/softwares/", "src": "/yinzhengjie/softwares/jdk1.8.0_131" } node103.yinzhengjie.org.cn | CHANGED => {"changed": true, "dest": "/yinzhengjie/softwares/", "src": "/yinzhengjie/softwares/jdk1.8.0_131" } node102.yinzhengjie.org.cn | CHANGED => {"changed": true, "dest": "/yinzhengjie/softwares/", "src": "/yinzhengjie/softwares/jdk1.8.0_131" } [root@node101 ~]#
将主节点的java目录同步到其他节点([root@node101 ~]# ansible datanode -m copy -a 'src=/yinzhengjie/softwares/jdk1.8.0_131 dest=/yinzhengjie/softwares')
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# ansible datanode -m shell -a 'ln -s /yinzhengjie/softwares/jdk1.8.0_131 /yinzhengjie/softwares/jdk' [WARNING]: Consider using the file module with state=link rather than running ln. If you need to use command because file is insufficient you can add warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message.node101.yinzhengjie.org.cn | CHANGED | rc=0 >>node103.yinzhengjie.org.cn | CHANGED | rc=0 >>node102.yinzhengjie.org.cn | CHANGED | rc=0 >>[root@node101 ~]#
创建符号链接,即软连接([root@node101 ~]# ansible datanode -m shell -a 'ln -s /yinzhengjie/softwares/jdk1.8.0_131 /yinzhengjie/softwares/jdk' )
3>.配置环境java的系统环境变量
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# tail -3 /etc/profile #ADD JAVA_PATH JAVA_HOME=/yinzhengjie/softwares/jdk PATH=$PATH:$JAVA_HOME/bin [root@node101 ~]# [root@node101 ~]# ansible datanode -m copy -a 'src=/etc/profile dest=/etc/profile' node103.yinzhengjie.org.cn | CHANGED => {"changed": true, "checksum": "5b3a42ee16d78583b834dc0dea67d282b5fa5323", "dest": "/etc/profile", "gid": 0, "group": "root", "md5sum": "cd04b3ecdde83a7c250b923167dc0c01", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 1828, "src": "/root/.ansible/tmp/ansible-tmp-1547122009.37-153584791304684/source", "state": "file", "uid": 0 } node102.yinzhengjie.org.cn | CHANGED => {"changed": true, "checksum": "5b3a42ee16d78583b834dc0dea67d282b5fa5323", "dest": "/etc/profile", "gid": 0, "group": "root", "md5sum": "cd04b3ecdde83a7c250b923167dc0c01", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 1828, "src": "/root/.ansible/tmp/ansible-tmp-1547122009.34-130374655280496/source", "state": "file", "uid": 0 } node101.yinzhengjie.org.cn | SUCCESS => {"changed": false, "checksum": "5b3a42ee16d78583b834dc0dea67d282b5fa5323", "dest": "/etc/profile", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/profile", "secontext": "system_u:object_r:etc_t:s0", "size": 1828, "state": "file", "uid": 0 } [root@node101 ~]#
[root@node101 ~]# ansible datanode -m copy -a 'src=/etc/profile dest=/etc/profile'
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# ansible datanode -m shell -a 'chmod -R 777 /yinzhengjie/softwares/jdk/bin'[WARNING]: Consider using the file module with mode rather than running chmod. If you need to use command because file is insufficient you can add warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message.node103.yinzhengjie.org.cn | CHANGED | rc=0 >>node102.yinzhengjie.org.cn | CHANGED | rc=0 >>node101.yinzhengjie.org.cn | CHANGED | rc=0 >>[root@node101 ~]#
取保拷贝到其他节点到java脚本拥有相应的执行权限([root@node101 ~]# ansible datanode -m shell -a 'chmod -R 777 /yinzhengjie/softwares/jdk/bin')
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# ansible datanode -m shell -a 'source /etc/profile;java -version' node101.yinzhengjie.org.cn | CHANGED | rc=0 >> java version "1.8.0_131" Java(TM) SE Runtime Environment (build 1.8.0_131-b11) Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)node103.yinzhengjie.org.cn | CHANGED | rc=0 >> java version "1.8.0_131" Java(TM) SE Runtime Environment (build 1.8.0_131-b11) Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)node102.yinzhengjie.org.cn | CHANGED | rc=0 >> java version "1.8.0_131" Java(TM) SE Runtime Environment (build 1.8.0_131-b11) Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)[root@node101 ~]#
查看每个节点的环境是否可用([root@node101 ~]# ansible datanode -m shell -a 'source /etc/profile;java -version' )
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# ansible datanode -m shell -a 'mkdir /usr/java' [WARNING]: Consider using the file module with state=directory rather than running mkdir. If you need to use command because file is insufficient you can add warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message.node101.yinzhengjie.org.cn | CHANGED | rc=0 >>node103.yinzhengjie.org.cn | CHANGED | rc=0 >>node102.yinzhengjie.org.cn | CHANGED | rc=0 >>[root@node101 ~]#
[root@node101 ~]# ansible datanode -m shell -a 'mkdir /usr/java'
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# ansible datanode -m shell -a 'ln -s /yinzhengjie/softwares/jdk1.8.0_131/ /usr/java/jdk1.8'[WARNING]: Consider using the file module with state=link rather than running ln. If you need to use command because file is insufficient you can add warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message.node101.yinzhengjie.org.cn | CHANGED | rc=0 >>node103.yinzhengjie.org.cn | CHANGED | rc=0 >>node102.yinzhengjie.org.cn | CHANGED | rc=0 >>[root@node101 ~]#
[root@node101 ~]# ansible datanode -m shell -a 'ln -s /yinzhengjie/softwares/jdk1.8.0_131/ /usr/java/jdk1.8'
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# ansible datanode -m shell -a 'ln -s /yinzhengjie/softwares/jdk1.8.0_131/ /usr/java/defalut'[WARNING]: Consider using the file module with state=link rather than running ln. If you need to use command because file is insufficient you can add warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message.node103.yinzhengjie.org.cn | CHANGED | rc=0 >>node101.yinzhengjie.org.cn | CHANGED | rc=0 >>node102.yinzhengjie.org.cn | CHANGED | rc=0 >>[root@node101 ~]#
[root@node101 ~]# ansible datanode -m shell -a 'ln -s /yinzhengjie/softwares/jdk1.8.0_131/ /usr/java/defalut'
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# ansible datanode -m shell -a 'ls -l /usr/java' node101.yinzhengjie.org.cn | CHANGED | rc=0 >> total 0 lrwxrwxrwx. 1 root root 36 Jan 10 20:55 defalut -> /yinzhengjie/softwares/jdk1.8.0_131/ lrwxrwxrwx. 1 root root 36 Jan 10 20:55 jdk1.8 -> /yinzhengjie/softwares/jdk1.8.0_131/node103.yinzhengjie.org.cn | CHANGED | rc=0 >> total 0 lrwxrwxrwx. 1 root root 36 Jan 10 20:55 defalut -> /yinzhengjie/softwares/jdk1.8.0_131/ lrwxrwxrwx. 1 root root 36 Jan 10 20:55 jdk1.8 -> /yinzhengjie/softwares/jdk1.8.0_131/node102.yinzhengjie.org.cn | CHANGED | rc=0 >> total 0 lrwxrwxrwx. 1 root root 36 Jan 10 20:55 defalut -> /yinzhengjie/softwares/jdk1.8.0_131/ lrwxrwxrwx. 1 root root 36 Jan 10 20:55 jdk1.8 -> /yinzhengjie/softwares/jdk1.8.0_131/[root@node101 ~]#
[root@node101 ~]# ansible datanode -m shell -a 'ls -l /usr/java'
三.安装mysql
1>.下载mysql的yum源
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# wget http://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm --2019-01-10 21:08:45-- http://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm Resolving dev.mysql.com (dev.mysql.com)... 137.254.60.11 Connecting to dev.mysql.com (dev.mysql.com)|137.254.60.11|:80... connected. HTTP request sent, awaiting response... 301 Moved Permanently Location: https://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm [following] --2019-01-10 21:08:45-- https://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm Connecting to dev.mysql.com (dev.mysql.com)|137.254.60.11|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://repo.mysql.com//mysql-community-release-el7-5.noarch.rpm [following] --2019-01-10 21:08:48-- https://repo.mysql.com//mysql-community-release-el7-5.noarch.rpm Resolving repo.mysql.com (repo.mysql.com)... 104.76.8.253 Connecting to repo.mysql.com (repo.mysql.com)|104.76.8.253|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 6140 (6.0K) [application/x-redhat-package-manager] Saving to: ‘mysql-community-release-el7-5.noarch.rpm’100%[================================================================================================================================>] 6,140 --.-K/s in 0s 2019-01-10 21:08:50 (529 MB/s) - ‘mysql-community-release-el7-5.noarch.rpm’ saved [6140/6140][root@node101 ~]# [root@node101 ~]# ll total 8 -rw-r--r--. 1 root root 6140 Nov 12 2015 mysql-community-release-el7-5.noarch.rpm [root@node101 ~]#
[root@node101 ~]# wget http://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm
2>.安装mysql的yum源
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# ll total 8 -rw-r--r--. 1 root root 6140 Nov 12 2015 mysql-community-release-el7-5.noarch.rpm [root@node101 ~]# [root@node101 ~]# rpm -ivh mysql-community-release-el7-5.noarch.rpm Preparing... ################################# [100%] Updating / installing...1:mysql-community-release-el7-5 ################################# [100%] [root@node101 ~]# [root@node101 ~]# ll /etc/yum.repos.d/ | grep mysql -rw-r--r--. 1 root root 1209 Jan 29 2014 mysql-community.repo -rw-r--r--. 1 root root 1060 Jan 29 2014 mysql-community-source.repo [root@node101 ~]#
[root@node101 ~]# rpm -ivh mysql-community-release-el7-5.noarch.rpm
3>.安装mysql服务
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# yum -y install mysql-community-server Loaded plugins: fastestmirror, langpacks mysql-connectors-community | 2.5 kB 00:00:00 mysql-tools-community | 2.5 kB 00:00:01 mysql56-community | 2.5 kB 00:00:02 (1/3): mysql-tools-community/x86_64/primary_db | 48 kB 00:00:00 (2/3): mysql-connectors-community/x86_64/primary_db | 29 kB 00:00:00 (3/3): mysql56-community/x86_64/primary_db | 209 kB 00:00:03 Loading mirror speeds from cached hostfile* base: mirror.jdcloud.com* epel: mirrors.yun-idc.com* extras: mirror.jdcloud.com* updates: mirrors.tuna.tsinghua.edu.cn Resolving Dependencies --> Running transaction check ---> Package mysql-community-server.x86_64 0:5.6.42-2.el7 will be installed --> Processing Dependency: mysql-community-common(x86-64) = 5.6.42-2.el7 for package: mysql-community-server-5.6.42-2.el7.x86_64 --> Processing Dependency: mysql-community-client(x86-64) >= 5.6.10 for package: mysql-community-server-5.6.42-2.el7.x86_64 --> Running transaction check ---> Package mysql-community-client.x86_64 0:5.6.42-2.el7 will be installed --> Processing Dependency: mysql-community-libs(x86-64) >= 5.6.10 for package: mysql-community-client-5.6.42-2.el7.x86_64 ---> Package mysql-community-common.x86_64 0:5.6.42-2.el7 will be installed --> Running transaction check ---> Package mariadb-libs.x86_64 1:5.5.44-2.el7.centos will be obsoleted ---> Package mysql-community-libs.x86_64 0:5.6.42-2.el7 will be obsoleting --> Finished Dependency ResolutionDependencies Resolved==========================================================================================================================================================================Package Arch Version Repository Size ========================================================================================================================================================================== Installing:mysql-community-libs x86_64 5.6.42-2.el7 mysql56-community 2.0 Mreplacing mariadb-libs.x86_64 1:5.5.44-2.el7.centosmysql-community-server x86_64 5.6.42-2.el7 mysql56-community 59 M Installing for dependencies:mysql-community-client x86_64 5.6.42-2.el7 mysql56-community 20 Mmysql-community-common x86_64 5.6.42-2.el7 mysql56-community 257 kTransaction Summary ========================================================================================================================================================================== Install 2 Packages (+2 Dependent packages)Total download size: 81 M Downloading packages: warning: /var/cache/yum/x86_64/7/mysql56-community/packages/mysql-community-common-5.6.42-2.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY02:24 ETA Public key for mysql-community-common-5.6.42-2.el7.x86_64.rpm is not installed (1/4): mysql-community-common-5.6.42-2.el7.x86_64.rpm | 257 kB 00:00:01 (2/4): mysql-community-libs-5.6.42-2.el7.x86_64.rpm | 2.0 MB 00:00:10 (3/4): mysql-community-client-5.6.42-2.el7.x86_64.rpm | 20 MB 00:03:29 (4/4): mysql-community-server-5.6.42-2.el7.x86_64.rpm | 59 MB 00:05:01 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 264 kB/s | 81 MB 00:05:14 Retrieving key from file:/etc/pki/rpm-gpg/RPM-GPG-KEY-mysql Importing GPG key 0x5072E1F5:Userid : "MySQL Release Engineering <mysql-build@oss.oracle.com>"Fingerprint: a4a9 4068 76fc bd3c 4567 70c8 8c71 8d3b 5072 e1f5Package : mysql-community-release-el7-5.noarch (installed)From : file:/etc/pki/rpm-gpg/RPM-GPG-KEY-mysql Running transaction check Running transaction test Transaction test succeeded Running transaction Warning: RPMDB altered outside of yum. ** Found 2 pre-existing rpmdb problem(s), 'yum check' output follows: 1:libreoffice-core-4.3.7.2-5.el7.x86_64 has missing requires of java-headless >= ('1', '1.6', None) 1:libreoffice-ure-4.3.7.2-5.el7.x86_64 has missing requires of libjvm.so()(64bit)Installing : mysql-community-common-5.6.42-2.el7.x86_64 1/5 Installing : mysql-community-libs-5.6.42-2.el7.x86_64 2/5 Installing : mysql-community-client-5.6.42-2.el7.x86_64 3/5 Installing : mysql-community-server-5.6.42-2.el7.x86_64 4/5 Erasing : 1:mariadb-libs-5.5.44-2.el7.centos.x86_64 5/5 Verifying : mysql-community-libs-5.6.42-2.el7.x86_64 1/5 Verifying : mysql-community-common-5.6.42-2.el7.x86_64 2/5 Verifying : mysql-community-client-5.6.42-2.el7.x86_64 3/5 Verifying : mysql-community-server-5.6.42-2.el7.x86_64 4/5 Verifying : 1:mariadb-libs-5.5.44-2.el7.centos.x86_64 5/5 Installed:mysql-community-libs.x86_64 0:5.6.42-2.el7 mysql-community-server.x86_64 0:5.6.42-2.el7 Dependency Installed:mysql-community-client.x86_64 0:5.6.42-2.el7 mysql-community-common.x86_64 0:5.6.42-2.el7 Replaced:mariadb-libs.x86_64 1:5.5.44-2.el7.centos Complete! [root@node101 ~]#
[root@node101 ~]# yum -y install mysql-community-server
4>.启动mysql服务
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 ~]# systemctl enable mysqld [root@node101 ~]# [root@node101 ~]# systemctl start mysqld [root@node101 ~]# [root@node101 ~]# [root@node101 ~]# systemctl status mysqld ● mysqld.service - MySQL Community ServerLoaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)Active: active (running) since Thu 2019-01-10 21:40:29 HKT; 8s agoProcess: 10810 ExecStartPost=/usr/bin/mysql-systemd-start post (code=exited, status=0/SUCCESS)Process: 10730 ExecStartPre=/usr/bin/mysql-systemd-start pre (code=exited, status=0/SUCCESS)Main PID: 10809 (mysqld_safe)CGroup: /system.slice/mysqld.service├─10809 /bin/sh /usr/bin/mysqld_safe --basedir=/usr└─10974 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mysqld.log --pid-file=/var/run/my...Jan 10 21:40:28 node101.yinzhengjie.org.cn mysql-systemd-start[10730]: PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER ! Jan 10 21:40:28 node101.yinzhengjie.org.cn mysql-systemd-start[10730]: To do so, start the server, then issue the following commands: Jan 10 21:40:28 node101.yinzhengjie.org.cn mysql-systemd-start[10730]: /usr/bin/mysqladmin -u root password 'new-password' Jan 10 21:40:28 node101.yinzhengjie.org.cn mysql-systemd-start[10730]: /usr/bin/mysqladmin -u root -h node101.yinzhengjie.org.cn password 'new-password' Jan 10 21:40:28 node101.yinzhengjie.org.cn mysql-systemd-start[10730]: Alternatively you can run: Jan 10 21:40:28 node101.yinzhengjie.org.cn mysql-systemd-start[10730]: /usr/bin/mysql_secure_installation Jan 10 21:40:28 node101.yinzhengjie.org.cn mysql-systemd-start[10730]: which will also give you the option of removing the test Jan 10 21:40:28 node101.yinzhengjie.org.cn mysqld_safe[10809]: 190110 21:40:28 mysqld_safe Logging to '/var/log/mysqld.log'. Jan 10 21:40:28 node101.yinzhengjie.org.cn mysqld_safe[10809]: 190110 21:40:28 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql Jan 10 21:40:29 node101.yinzhengjie.org.cn systemd[1]: Started MySQL Community Server. [root@node101 ~]# [root@node101 ~]#
[root@node101 ~]# systemctl start mysqld
5>.为mysql配置管理用户
[root@node101 ~]# mysql -uroot #安装后的操作系统,默认是密钥root密码的,我们直接可以用root用户登陆即可。 Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 2 Server version: 5.6.42 MySQL Community Server (GPL)Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql> set password for 'root'@'localhost' = password('yinzhengjie'); #修改root本地用户的登陆密码为“yinzhengjie” Query OK, 0 rows affected (0.00 sec)mysql> grant all privileges on *.* to 'root'@'172.30.1.10%' identified by 'yinzhengjie'; #我这里值允许172.30.1.100~172.30.1.109这10个服务器IP使用root用户登陆我这台mysql。其他的机器不允许。 Query OK, 0 rows affected (0.00 sec)mysql> FLUSH PRIVILEGES; #别忘记刷新权限哟! Query OK, 0 rows affected (0.00 sec)mysql> quit Bye [root@node101 ~]#
6>.配置一个cdh超级用户权限(这个步骤如果你调过了,在初始化数据库时可能与报错哟!)
[root@node101 mysql]# systemctl start mysqld [root@node101 mysql]# [root@node101 mysql]# systemctl enable mysqld Created symlink from /etc/systemd/system/mysql.service to /usr/lib/systemd/system/mysqld.service. Created symlink from /etc/systemd/system/multi-user.target.wants/mysqld.service to /usr/lib/systemd/system/mysqld.service. [root@node101 mysql]# [root@node101 mysql]# mysql Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 2 Server version: 5.6.38 MySQL Community Server (GPL)Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql> GRANT ALL PRIVILEGES ON *.* TO 'cdh'@'172.30.1.10%' IDENTIFIED BY 'yinzhengjie' WITH GRANT OPTION; Query OK, 0 rows affected (0.00 sec)mysql> GRANT ALL PRIVILEGES ON *.* TO 'cdh'@'node101.yinzhengjie.org.cn' IDENTIFIED BY 'yinzhengjie' WITH GRANT OPTION; Query OK, 0 rows affected (0.00 sec)mysql> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.00 sec)mysql> quit Bye [root@node101 mysql]#
四.部署cloudrea manager软件
1>.安装相应的依赖包(每个节点都需要安装哟~)
[root@node101 ~]# ansible datanode -m shell -a 'yum -y install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb'
2>.安装Cloudera Manager Server,Agent(在所有机器)
2.1>.CDH的选择
下载地址:http://archive.cloudera.com/cdh5/parcels/5.15.1/
2.2>.CM的选择
下载地址:http://archive.cloudera.com/cm5/cm/5/
2.3>.温馨提示:
如果网速较慢的小伙伴可以用不建议使用wget下载,推荐使用迅雷下载再上传到服务器,如果有百度云超级会员的话也可以使用我给大家体现下载好的软件。百度云链接:https://pan.baidu.com/s/148t70GenClGQRtJYX8WjvQ 密码:i2yd
[root@node101 CDH-5.15.1]# ll total 3071760 -rw-r--r--. 1 root root 2120090032 Jan 10 18:55 CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel -rw-r--r--. 1 root root 41 Jan 10 18:54 CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel.sha1 -rw-r--r--. 1 root root 838894986 Jan 10 18:55 cloudera-manager-centos7-cm5.15.1_x86_64.tar.gz -rw-r--r--. 1 root root 185540433 Jan 10 18:53 jdk-8u131-linux-x64.tar.gz -rw-r--r--. 1 root root 73767 Jan 10 18:55 manifest.json -rw-r--r--. 1 root root 855946 Jan 10 18:55 mysql-connector-java-5.1.26.jar [root@node101 CDH-5.15.1]#
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 CDH-5.15.1]# ansible all -m shell -a 'ls /yinzhengjie/softwares' node103.yinzhengjie.org.cn | CHANGED | rc=0 >> jdk jdk1.8.0_131node101.yinzhengjie.org.cn | CHANGED | rc=0 >> CDH-5.15.1 jdk jdk1.8.0_131node102.yinzhengjie.org.cn | CHANGED | rc=0 >> jdk jdk1.8.0_131[root@node101 CDH-5.15.1]# ansible all -m copy -a 'src=/yinzhengjie/softwares/CDH-5.15.1/cloudera-manager-centos7-cm5.15.1_x86_64.tar.gz dest=/yinzhengjie/softwares/' node101.yinzhengjie.org.cn | CHANGED => {"changed": true, "checksum": "4227988280e0f09164bbfc6c59f291c6a814b8be", "dest": "/yinzhengjie/softwares/cloudera-manager-centos7-cm5.15.1_x86_64.tar.gz", "gid": 0, "group": "root", "md5sum": "ca0be616f62668915d9ceeb22bdcdd4b", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:default_t:s0", "size": 838894986, "src": "/root/.ansible/tmp/ansible-tmp-1547172382.51-139484190159118/source", "state": "file", "uid": 0 } node103.yinzhengjie.org.cn | CHANGED => {"changed": true, "checksum": "4227988280e0f09164bbfc6c59f291c6a814b8be", "dest": "/yinzhengjie/softwares/cloudera-manager-centos7-cm5.15.1_x86_64.tar.gz", "gid": 0, "group": "root", "md5sum": "ca0be616f62668915d9ceeb22bdcdd4b", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:default_t:s0", "size": 838894986, "src": "/root/.ansible/tmp/ansible-tmp-1547172382.57-61195539797705/source", "state": "file", "uid": 0 } node102.yinzhengjie.org.cn | CHANGED => {"changed": true, "checksum": "4227988280e0f09164bbfc6c59f291c6a814b8be", "dest": "/yinzhengjie/softwares/cloudera-manager-centos7-cm5.15.1_x86_64.tar.gz", "gid": 0, "group": "root", "md5sum": "ca0be616f62668915d9ceeb22bdcdd4b", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:default_t:s0", "size": 838894986, "src": "/root/.ansible/tmp/ansible-tmp-1547172382.54-248038987426177/source", "state": "file", "uid": 0 } [root@node101 CDH-5.15.1]# [root@node101 CDH-5.15.1]# [root@node101 CDH-5.15.1]# [root@node101 CDH-5.15.1]# ansible all -m shell -a 'ls /yinzhengjie/softwares' node101.yinzhengjie.org.cn | CHANGED | rc=0 >> CDH-5.15.1 cloudera-manager-centos7-cm5.15.1_x86_64.tar.gz jdk jdk1.8.0_131node103.yinzhengjie.org.cn | CHANGED | rc=0 >> cloudera-manager-centos7-cm5.15.1_x86_64.tar.gz jdk jdk1.8.0_131node102.yinzhengjie.org.cn | CHANGED | rc=0 >> cloudera-manager-centos7-cm5.15.1_x86_64.tar.gz jdk jdk1.8.0_131[root@node101 CDH-5.15.1]#
分发Cloudera Manager到各个节点([root@node101 CDH-5.15.1]# ansible all -m copy -a 'src=/yinzhengjie/softwares/CDH-5.15.1/cloudera-manager-centos7-cm5.15.1_x86_64.tar.gz dest=/yinzhengjie/softwares/')
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 CDH-5.15.1]# ansible all -m shell -a 'mkdir /opt/cloudera-manager'[WARNING]: Consider using the file module with state=directory rather than running mkdir. If you need to use command because file is insufficient you can add warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message.node103.yinzhengjie.org.cn | CHANGED | rc=0 >>node101.yinzhengjie.org.cn | CHANGED | rc=0 >>node102.yinzhengjie.org.cn | CHANGED | rc=0 >>[root@node101 CDH-5.15.1]#
创建解压CM默认解压的目录,如果你想要使用别的解压目录别忘了修改配置文件哟([root@node101 CDH-5.15.1]# ansible all -m shell -a 'mkdir /opt/cloudera-manager')
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 softwares]# ansible all -m shell -a 'tar -zxf /yinzhengjie/softwares/cloudera-manager-centos7-cm5.15.1_x86_64.tar.gz -C /opt/cloudera-manager'[WARNING]: Consider using the unarchive module rather than running tar. If you need to use command because unarchive is insufficient you can add warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message.node101.yinzhengjie.org.cn | CHANGED | rc=0 >>node102.yinzhengjie.org.cn | CHANGED | rc=0 >>node103.yinzhengjie.org.cn | CHANGED | rc=0 >>[root@node101 softwares]# [root@node101 softwares]# ansible all -m shell -a 'ls -l /opt/cloudera-manager' node101.yinzhengjie.org.cn | CHANGED | rc=0 >> total 8 drwxr-xr-x. 4 1106 4001 4096 Jul 31 18:28 cloudera drwxr-xr-x. 9 1106 4001 4096 Jul 31 18:28 cm-5.15.1node102.yinzhengjie.org.cn | CHANGED | rc=0 >> total 8 drwxr-xr-x. 4 1106 4001 4096 Jul 31 18:28 cloudera drwxr-xr-x. 9 1106 4001 4096 Jul 31 18:28 cm-5.15.1node103.yinzhengjie.org.cn | CHANGED | rc=0 >> total 8 drwxr-xr-x. 4 1106 4001 4096 Jul 31 18:28 cloudera drwxr-xr-x. 9 1106 4001 4096 Jul 31 18:28 cm-5.15.1[root@node101 softwares]#
将CM解压到我们刚刚创建的目录([root@node101 softwares]# ansible all -m shell -a 'tar -zxf /yinzhengjie/softwares/cloudera-manager-centos7-cm5.15.1_x86_64.tar.gz -C /opt/cloudera-manager')
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 softwares]# ansible all -m shell -a 'useradd --system --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm' node102.yinzhengjie.org.cn | CHANGED | rc=0 >>node101.yinzhengjie.org.cn | CHANGED | rc=0 >>node103.yinzhengjie.org.cn | CHANGED | rc=0 >>[root@node101 softwares]# [root@node101 softwares]# [root@node101 softwares]# ansible all -m shell -a 'id cloudera-scm' node101.yinzhengjie.org.cn | CHANGED | rc=0 >> uid=986(cloudera-scm) gid=981(cloudera-scm) groups=981(cloudera-scm)node102.yinzhengjie.org.cn | CHANGED | rc=0 >> uid=986(cloudera-scm) gid=981(cloudera-scm) groups=981(cloudera-scm)node103.yinzhengjie.org.cn | CHANGED | rc=0 >> uid=986(cloudera-scm) gid=981(cloudera-scm) groups=981(cloudera-scm)[root@node101 softwares]#
在所有的节点上创建clodera-scm用户([root@node101 softwares]# ansible all -m shell -a 'useradd --system --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm')
3>.配置CM的Agent端
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 softwares]# ansible all -m shell -a 'grep server_host /opt/cloudera-manager/cm-5.15.1/etc/cloudera-scm-agent/config.ini' node103.yinzhengjie.org.cn | CHANGED | rc=0 >> server_host=localhostnode102.yinzhengjie.org.cn | CHANGED | rc=0 >> server_host=localhostnode101.yinzhengjie.org.cn | CHANGED | rc=0 >> server_host=localhost[root@node101 softwares]# [root@node101 softwares]# ansible all -m shell -a 'sed -i 's/server_host=localhost/server_host=node101.yinzhengjie.org.cn/g' /opt/cloudera-manager/cm-5.15.1/etc/cloudera-scm-agent/config.ini'[WARNING]: Consider using the replace, lineinfile or template module rather than running sed. If you need to use command because replace, lineinfile or template is insufficient you can add warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message.node103.yinzhengjie.org.cn | CHANGED | rc=0 >>node102.yinzhengjie.org.cn | CHANGED | rc=0 >>node101.yinzhengjie.org.cn | CHANGED | rc=0 >>[root@node101 softwares]# [root@node101 softwares]# ansible all -m shell -a 'grep server_host /opt/cloudera-manager/cm-5.15.1/etc/cloudera-scm-agent/config.ini' node101.yinzhengjie.org.cn | CHANGED | rc=0 >> server_host=node101.yinzhengjie.org.cnnode103.yinzhengjie.org.cn | CHANGED | rc=0 >> server_host=node101.yinzhengjie.org.cnnode102.yinzhengjie.org.cn | CHANGED | rc=0 >> server_host=node101.yinzhengjie.org.cn[root@node101 softwares]#
我们指定CM的Server端([root@node101 softwares]# ansible all -m shell -a 'sed -i 's/server_host=localhost/server_host=node101.yinzhengjie.org.cn/g' /opt/cloudera-manager/cm-5.15.1/etc/cloudera-scm-agent/config.ini')
4>.初始化CM Server的数据库
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 CDH-5.15.1]# ansible namenode -m shell -a 'mkdir /usr/share/java'[WARNING]: Consider using the file module with state=directory rather than running mkdir. If you need to use command because file is insufficient you can add warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message.node101.yinzhengjie.org.cn | CHANGED | rc=0 >>[root@node101 CDH-5.15.1]#
创建java环境目录([root@node101 CDH-5.15.1]# ansible namenode -m shell -a 'mkdir /usr/share/java')
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 CDH-5.15.1]# ansible namenode -m shell -a 'ls -l /usr/share/java' node101.yinzhengjie.org.cn | CHANGED | rc=0 >> total 0[root@node101 CDH-5.15.1]# [root@node101 CDH-5.15.1]# ansible namenode -m shell -a 'mv /yinzhengjie/softwares/CDH-5.15.1/mysql-connector-java-5.1.26.jar /usr/share/java/mysql-connector-java.jar' node101.yinzhengjie.org.cn | CHANGED | rc=0 >>[root@node101 CDH-5.15.1]# ansible namenode -m shell -a 'ls -l /usr/share/java' node101.yinzhengjie.org.cn | CHANGED | rc=0 >> total 836 -rw-r--r--. 1 root root 855946 Jan 10 18:55 mysql-connector-java.jar[root@node101 CDH-5.15.1]#
将mysql的驱动拷贝到上一步创建的java环境目录,CM数据库初始化脚本默认会去该目录查找相应的驱动([root@node101 CDH-5.15.1]# ansible namenode -m shell -a 'mv /yinzhengjie/softwares/CDH-5.15.1/mysql-connector-java-5.1.26.jar /usr/share/java/mysql-connector-java.jar')
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 CDH-5.15.1]# ansible namenode -m shell -a '/opt/cloudera-manager/cm-5.15.1/share/cmf/schema/scm_prepare_database.sh mysql cdh -h node101.yinzhengjie.org.cn -ucdh -pyinzhengjie --scm-host node101.yinzhengjie.org.cn scm scm scm' node101.yinzhengjie.org.cn | CHANGED | rc=0 >> JAVA_HOME=/usr/java/jdk1.8 Verifying that we can write to /opt/cloudera-manager/cm-5.15.1/etc/cloudera-scm-server Creating SCM configuration file in /opt/cloudera-manager/cm-5.15.1/etc/cloudera-scm-server Executing: /usr/java/jdk1.8/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera-manager/cm-5.15.1/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /opt/cloudera-manager/cm-5.15.1/etc/cloudera-scm-server/db.properties com.cloudera.cmf.db. [ main] DbCommandExecutor INFO Successfully connected to database. All done, your SCM database is configured correctly![root@node101 CDH-5.15.1]#
执行数据库初始化脚本([root@node101 CDH-5.15.1]# ansible namenode -m shell -a '/opt/cloudera-manager/cm-5.15.1/share/cmf/schema/scm_prepare_database.sh mysql cdh -h node101.yinzhengjie.org.cn -ucdh -pyinzhengjie --scm-host node101.yinzhengjie.org.cn scm scm scm' #其格式为:数据库类型,数据库,数据库服务器,用户名,密码,CM服务器,后面的三个scm请不要改动!)
5>.制作CDH本地源
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 CDH-5.15.1]# ansible namenode -m shell -a 'mkdir -p /opt/cloudera/parcel-repo'[WARNING]: Consider using the file module with state=directory rather than running mkdir. If you need to use command because file is insufficient you can add warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message.node101.yinzhengjie.org.cn | CHANGED | rc=0 >>[root@node101 CDH-5.15.1]#
Server端创建Parcel目录([root@node101 CDH-5.15.1]# ansible namenode -m shell -a 'mkdir -p /opt/cloudera/parcel-repo')
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 CDH-5.15.1]# ansible namenode -m shell -a 'chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/'[WARNING]: Consider using the file module with owner rather than running chown. If you need to use command because file is insufficient you can add warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message.node101.yinzhengjie.org.cn | CHANGED | rc=0 >>[root@node101 CDH-5.15.1]# [root@node101 CDH-5.15.1]# ansible namenode -m shell -a 'ls -ld /opt/cloudera/parcel-repo/' node101.yinzhengjie.org.cn | CHANGED | rc=0 >> drwxr-xr-x. 2 cloudera-scm cloudera-scm 4096 Jan 11 10:42 /opt/cloudera/parcel-repo/[root@node101 CDH-5.15.1]#
将上一步创建的目录所有者和所属组改为cloudera-scm用户([root@node101 CDH-5.15.1]# ansible namenode -m shell -a 'chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/')
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 CDH-5.15.1]# ansible datanode -m shell -a 'mkdir -p /opt/cloudera/parcels' [WARNING]: Consider using the file module with state=directory rather than running mkdir. If you need to use command because file is insufficient you can add warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message.node101.yinzhengjie.org.cn | CHANGED | rc=0 >>node102.yinzhengjie.org.cn | CHANGED | rc=0 >>node103.yinzhengjie.org.cn | CHANGED | rc=0 >>[root@node101 CDH-5.15.1]# [root@node101 CDH-5.15.1]# ansible datanode -m shell -a 'ls -ld /opt/cloudera/parcels' node102.yinzhengjie.org.cn | CHANGED | rc=0 >> drwxr-xr-x. 2 root root 4096 Jan 11 10:48 /opt/cloudera/parcelsnode103.yinzhengjie.org.cn | CHANGED | rc=0 >> drwxr-xr-x. 2 root root 4096 Jan 11 10:48 /opt/cloudera/parcelsnode101.yinzhengjie.org.cn | CHANGED | rc=0 >> drwxr-xr-x. 2 root root 4096 Jan 11 10:48 /opt/cloudera/parcels[root@node101 CDH-5.15.1]#
agent端创建parcels目录([root@node101 CDH-5.15.1]# ansible datanode -m shell -a 'mkdir -p /opt/cloudera/parcels' )
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 CDH-5.15.1]# ansible datanode -m shell -a 'chown cloudera-scm:cloudera-scm /opt/cloudera/parcels'[WARNING]: Consider using the file module with owner rather than running chown. If you need to use command because file is insufficient you can add warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message.node101.yinzhengjie.org.cn | CHANGED | rc=0 >>node103.yinzhengjie.org.cn | CHANGED | rc=0 >>node102.yinzhengjie.org.cn | CHANGED | rc=0 >>[root@node101 CDH-5.15.1]# [root@node101 CDH-5.15.1]# ansible datanode -m shell -a 'ls -ld /opt/cloudera/parcels' node101.yinzhengjie.org.cn | CHANGED | rc=0 >> drwxr-xr-x. 2 cloudera-scm cloudera-scm 4096 Jan 11 10:48 /opt/cloudera/parcelsnode103.yinzhengjie.org.cn | CHANGED | rc=0 >> drwxr-xr-x. 2 cloudera-scm cloudera-scm 4096 Jan 11 10:48 /opt/cloudera/parcelsnode102.yinzhengjie.org.cn | CHANGED | rc=0 >> drwxr-xr-x. 2 cloudera-scm cloudera-scm 4096 Jan 11 10:48 /opt/cloudera/parcels[root@node101 CDH-5.15.1]#
别忘了把parcels目录权限授权给我们之前创建的cloudera-scm用户!([root@node101 CDH-5.15.1]# ansible datanode -m shell -a 'chown cloudera-scm:cloudera-scm /opt/cloudera/parcels')
[root@node101 CDH-5.15.1]# [root@node101 CDH-5.15.1]# cp CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel /opt/cloudera/parcel-repo/ [root@node101 CDH-5.15.1]# [root@node101 CDH-5.15.1]# cp manifest.json /opt/cloudera/parcel-repo/ #别小瞧这个文件,尽管它不是很大,但是它却记录着CDH和Hadoop生态圈组件的版本依赖关系! [root@node101 CDH-5.15.1]# [root@node101 CDH-5.15.1]# cp CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel.sha1 /opt/cloudera/parcel-repo/CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel.sha #注意,我在拷贝到时候重命名该文件了! [root@node101 CDH-5.15.1]# [root@node101 CDH-5.15.1]# ll /opt/cloudera/parcel-repo/ total 2070484 -rw-r--r-- 1 root root 2120090032 Sep 13 07:40 CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel -rw-r--r-- 1 root root 41 Sep 13 07:41 CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel.sha -rw-r--r-- 1 root root 73767 Sep 13 07:41 manifest.json [root@node101 CDH-5.15.1]# 温馨提示:如果你没有下载到“CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel.sha”文件是,可以找到“manifest.json”文件中"parcelName": "CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel"对应的"hash": "deff00898e410a34cf0a1e66c5dbe87546608f0c"复制到该文件即可。这个方法也适用于其他的版本!
五.启动集群
1>.启动cloudera manager的Server 端
[root@node101 CDH-5.15.1]# ansible namenode -m shell -a '/opt/cloudera-manager/cm-5.15.1/etc/init.d/cloudera-scm-server start' node101.yinzhengjie.org.cn | CHANGED | rc=0 >> Starting cloudera-scm-server: [ OK ][root@node101 CDH-5.15.1]# [root@node101 CDH-5.15.1]# netstat -untalp | grep 7180 tcp 0 0 0.0.0.0:7180 0.0.0.0:* LISTEN 30064/java [root@node101 CDH-5.15.1]#
![](/assets/blank.gif)
![](/assets/blank.gif)
............(信息会有很多,你会看到日志中启动了很多端口,如果7180默认端口启动成功,那么你就可以访问WebUI界面了,当然这个端口咱们是可以修改的!) 2019-01-11 10:57:52,253 INFO WebServerImpl:com.cloudera.server.web.cmon.JobDetailGatekeeper: ActivityMonitor configured to allow job details for all jobs. 2019-01-11 10:57:53,956 INFO ParcelUpdateService:com.cloudera.parcel.components.LocalParcelManagerImpl: Discovered parcel on CM server: CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel 2019-01-11 10:57:53,957 INFO ParcelUpdateService:com.cloudera.parcel.components.LocalParcelManagerImpl: Created torrent file: /opt/cloudera/parcel-repo/CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel.torrent 2019-01-11 10:57:53,963 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Creating single-file torrent for CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel... 2019-01-11 10:57:53,965 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Hashing data from CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel with 2 threads (4044 pieces)... 2019-01-11 10:57:54,522 INFO WebServerImpl:com.cloudera.server.web.cmf.AggregatorController: AggregateSummaryScheduler started. 2019-01-11 10:57:55,568 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: ... 10% complete 2019-01-11 10:57:55,957 ERROR SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: The server storage directory [/var/lib/cloudera-scm-server] doesn't exist. 2019-01-11 10:57:55,965 ERROR SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: No read permission to the server storage directory [/var/lib/cloudera-scm-server] 2019-01-11 10:57:55,965 ERROR SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: No write permission to the server storage directory [/var/lib/cloudera-scm-server] 2019-01-11 10:57:57,025 INFO WebServerImpl:org.mortbay.log: jetty-6.1.26.cloudera.4 2019-01-11 10:57:57,030 INFO WebServerImpl:org.mortbay.log: Started SelectChannelConnector@0.0.0.0:7180 2019-01-11 10:57:57,030 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server. 2019-01-11 10:57:57,297 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: ... 20% complete 2019-01-11 10:57:58,330 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: ... 30% complete 2019-01-11 10:57:59,348 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: ... 40% complete 2019-01-11 10:58:00,082 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: ... 50% complete
服务端启动时,我们应该查看相应的日志信息([root@node101 CDH-5.15.1]# tail -f /opt/cloudera-manager/cm-5.15.1/log/cloudera-scm-server/cloudera-scm-server.log)
2>.启动cloudera manager的Agent 端
[root@node101 CDH-5.15.1]# ansible datanode -m shell -a '/opt/cloudera-manager/cm-5.15.1/etc/init.d/cloudera-scm-agent start' node102.yinzhengjie.org.cn | CHANGED | rc=0 >> Starting cloudera-scm-agent: [ OK ]node101.yinzhengjie.org.cn | CHANGED | rc=0 >> Starting cloudera-scm-agent: [ OK ]node103.yinzhengjie.org.cn | CHANGED | rc=0 >> Starting cloudera-scm-agent: [ OK ][root@node101 CDH-5.15.1]# [root@node101 CDH-5.15.1]# ansible datanode -m shell -a '/opt/cloudera-manager/cm-5.15.1/etc/init.d/cloudera-scm-agent status' node103.yinzhengjie.org.cn | CHANGED | rc=0 >> cloudera-scm-agent (pid 25342) is running...node102.yinzhengjie.org.cn | CHANGED | rc=0 >> cloudera-scm-agent (pid 25506) is running...node101.yinzhengjie.org.cn | CHANGED | rc=0 >> cloudera-scm-agent (pid 31929) is running...[root@node101 CDH-5.15.1]#
3>.关闭防火墙和selinux
温馨提示:
在实际生产环境中,如果是内网环境咱们可以关闭防火墙和selinux,风向毕竟很小。而且很多公司都是有硬件防火墙的,如果咱们在开启防火墙会影响服务器性能。由于我是测试机环境,为了实验方便就直接禁用这2个服务了,实际生产中,根据你的需求选择是否开启防火墙,到时候你只需要定义一些列端口开放规则即可。
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 CDH-5.15.1]# ansible datanode -m shell -a 'systemctl status firewalld' node101.yinzhengjie.org.cn | CHANGED | rc=0 >> ● firewalld.service - firewalld - dynamic firewall daemonLoaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)Active: active (running) since Thu 2019-01-10 17:58:21 HKT; 17h agoMain PID: 937 (firewalld)CGroup: /system.slice/firewalld.service└─937 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopidJan 10 17:58:19 node101.yinzhengjie.org.cn systemd[1]: Starting firewalld - dynamic firewall daemon... Jan 10 17:58:21 node101.yinzhengjie.org.cn systemd[1]: Started firewalld - dynamic firewall daemon.node103.yinzhengjie.org.cn | CHANGED | rc=0 >> ● firewalld.service - firewalld - dynamic firewall daemonLoaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)Active: active (running) since Thu 2019-01-10 17:59:02 HKT; 17h agoMain PID: 952 (firewalld)CGroup: /system.slice/firewalld.service└─952 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopidJan 10 17:59:01 node103.yinzhengjie.org.cn systemd[1]: Starting firewalld - dynamic firewall daemon... Jan 10 17:59:02 node103.yinzhengjie.org.cn systemd[1]: Started firewalld - dynamic firewall daemon.node102.yinzhengjie.org.cn | CHANGED | rc=0 >> ● firewalld.service - firewalld - dynamic firewall daemonLoaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)Active: active (running) since Thu 2019-01-10 17:58:22 HKT; 17h agoMain PID: 913 (firewalld)CGroup: /system.slice/firewalld.service└─913 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopidJan 10 17:58:21 node102.yinzhengjie.org.cn systemd[1]: Starting firewalld - dynamic firewall daemon... Jan 10 17:58:22 node102.yinzhengjie.org.cn systemd[1]: Started firewalld - dynamic firewall daemon.[root@node101 CDH-5.15.1]#
检查集群的防火墙状态([root@node101 CDH-5.15.1]# ansible datanode -m shell -a 'systemctl status firewalld' )
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 CDH-5.15.1]# ansible datanode -m shell -a 'systemctl stop firewalld' node102.yinzhengjie.org.cn | CHANGED | rc=0 >>node103.yinzhengjie.org.cn | CHANGED | rc=0 >>node101.yinzhengjie.org.cn | CHANGED | rc=0 >>[root@node101 CDH-5.15.1]#
停用防火墙([root@node101 CDH-5.15.1]# ansible datanode -m shell -a 'systemctl stop firewalld' )
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 CDH-5.15.1]# ansible datanode -m shell -a 'systemctl disable firewalld' node103.yinzhengjie.org.cn | CHANGED | rc=0 >> Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service. Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service.node102.yinzhengjie.org.cn | CHANGED | rc=0 >> Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service. Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.node101.yinzhengjie.org.cn | CHANGED | rc=0 >> Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service. Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service.[root@node101 CDH-5.15.1]#
禁用防火墙开机自启动([root@node101 CDH-5.15.1]# ansible datanode -m shell -a 'systemctl disable firewalld')
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 CDH-5.15.1]# ansible datanode -m shell -a 'setenforce 0' node103.yinzhengjie.org.cn | CHANGED | rc=0 >>node101.yinzhengjie.org.cn | CHANGED | rc=0 >>node102.yinzhengjie.org.cn | CHANGED | rc=0 >>[root@node101 CDH-5.15.1]#
临时禁用selinux([root@node101 CDH-5.15.1]# ansible datanode -m shell -a 'setenforce 0' )
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 CDH-5.15.1]# ansible datanode -m shell -a 'getenforce' node101.yinzhengjie.org.cn | CHANGED | rc=0 >> Permissivenode102.yinzhengjie.org.cn | CHANGED | rc=0 >> Permissivenode103.yinzhengjie.org.cn | CHANGED | rc=0 >> Permissive[root@node101 CDH-5.15.1]#
查看selinux的状态([root@node101 CDH-5.15.1]# ansible datanode -m shell -a 'getenforce' )
![](/assets/blank.gif)
![](/assets/blank.gif)
[root@node101 CDH-5.15.1]# ansible datanode -m shell -a 'sed -i 's#SELINUX=enforcing#SELINUX=disabled#' /etc/selinux/config'[WARNING]: Consider using the replace, lineinfile or template module rather than running sed. If you need to use command because replace, lineinfile or template is insufficient you can add warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message.node101.yinzhengjie.org.cn | CHANGED | rc=0 >>node103.yinzhengjie.org.cn | CHANGED | rc=0 >>node102.yinzhengjie.org.cn | CHANGED | rc=0 >>[root@node101 CDH-5.15.1]# [root@node101 CDH-5.15.1]# [root@node101 CDH-5.15.1]# ansible datanode -m shell -a 'cat /etc/selinux/config | grep SELINUX= | grep -v ^#' node103.yinzhengjie.org.cn | CHANGED | rc=0 >> SELINUX=disablednode101.yinzhengjie.org.cn | CHANGED | rc=0 >> SELINUX=disablednode102.yinzhengjie.org.cn | CHANGED | rc=0 >> SELINUX=disabled[root@node101 CDH-5.15.1]#
永久禁用selinux,重启操作系统后生效([root@node101 CDH-5.15.1]# ansible datanode -m shell -a 'sed -i 's#SELINUX=enforcing#SELINUX=disabled#' /etc/selinux/config')
4>.如下图所示,访问webUI界面(http://node101.yinzhengjie.org.cn:7180/)
我们安装CM的过程通过WebUI的安装向导来进行安装,推荐使用谷歌浏览器,不要使用容易崩溃的浏览器,这样会影响你安装进度的!如果你也出现了以上界面,恭喜你CM部署成功,关于CDH的部署详情请参考:https://www.cnblogs.com/yinzhengjie/p/9638360.html。
转载于:https://www.cnblogs.com/yinzhengjie/p/10250816.html
使用ansible部署CDH 5.15.1大数据集群相关推荐
- 大数据集群机房搬迁数据迁移
[一.背景] 按照中心总体计划,目前部署在生产区的运营大数据集群需要搬迁至万国机房. 本次采用的搬迁的方案是通过万国机房的72台物理主机上新建运营大数据集群,老集群应用数据同步至新集群的方式,之后应用 ...
- centos7 ambari2.6.1.5+hdp2.6.4.0 大数据集群安装部署
2019独角兽企业重金招聘Python工程师标准>>> 转载请务必注明原创地址为:http://dongkelun.com/2018/04/25/ambariConf/ 前言 本文是 ...
- python自动化部署hadoop集群_大数据集群的自动化运维实现思路
原标题:大数据集群的自动化运维实现思路 | 作者简介 王晓伟 知数堂<大数据实战就业>课程讲师 六年大数据相关工作经验 清华大学软件工程硕士 曾就职于网易.搜狗等互联网企业 从事大数据及数 ...
- 【大数据集群搭建-Apache】Apache版本进行大数据集群各组件环境部署
[大数据集群搭建-Apache]Apache版本进行大数据集群各组件环境部署 1)大数据环境统一 1.1.设置主机名和域名映射 1.2.关闭服务器防火墙和Selinux 1.3.服务器免密登陆 1.4 ...
- 同程旅行大数据集群在 Kubernetes 上的服务化实践
本文将向大家介绍同程旅行大数据集群在 Kubernetes 上服务化建设的一些实践和经验. 同程旅行大数据集群从 2017 年开始容器化改造,经历了自研调度 Docker 容器 ,到现在的云舱平台,采 ...
- 大数据集群搭建全部过程(Vmware虚拟机、hadoop、zookeeper、hive、flume、hbase、spark、yarn)
大数据集群搭建进度及问题总结 所有资料在评论区那里可以得到 第一章: 1.网关配置(参照文档) 注意事项:第一台虚拟机改了,改为centos 101 ,地址为192.168.181.130 网关依然是 ...
- 全网最详细的Hadoop大数据集群搭建并进行项目分析(基于完全分布式)---第二部分
##所有需要的资料全部已上传到百度网盘上,请自行下载## 获取镜像,https://pan.baidu.com/s/1ho4hMrvIu1V6W4wWdH8nIA,提取码:ygyg 获取Xshell, ...
- 全网最详细的大数据集群环境下如何正确安装并配置多个不同版本的Cloudera Hue(图文详解)...
不多说,直接上干货! 为什么要写这么一篇博文呢? 是因为啊,对于Hue不同版本之间,其实,差异还是相对来说有点大的,具体,大家在使用的时候亲身体会就知道了,比如一些提示和界面. 全网最详细的大数据集群 ...
- 阿里云TSDB在大数据集群监控中的方案与实战
目前大部分的互联网企业基本上都有搭建自己的大数据集群,为了能更好让我们的大数据集群更加高效安全的工作,一个优秀的监控方案是必不可少的:所以今天给大家带来的这篇文章就是讲阿里云TSDB在上海某大型互联网 ...
最新文章
- login控件设置居中
- 【面试招聘】一份转ML的面试心得记录
- 快速解决工作中遇到经典的括号匹配问题
- html多级折叠菜单表单,JS实战篇之收缩菜单表单布局
- h5如何上传文件二进制流_HTML5新特性之文件和二进制数据的操作
- iOS开发——高级篇——iOS开发之网络安全密码学
- c++语言iso标准,C++20标准 (ISO/IEC 14882:2020) 正式发布
- Android应用及应用管理
- WPF和Winform中picturebox图片局部放大
- 网络安全与渗透测试工具导航
- Dream(励志英文演讲)
- cookie和session的详解与区别
- PON串口链接详细步骤说明
- CVE-2017-0199漏洞分析
- 编码 unicode 及其在 javascript 中的使用
- 2013年5月上海乌镇游
- set_sql_trace_in_session PLS-00201
- 离散化/线段树 (POJ - 2528 Mayor's posters)
- 儿童周成就任务全攻略
- winmm.lib库