(超详细)安装CDH6.3.2和Cloud Manager

1. CDH概述

        1.1 创作背景

        由于公司目前所用的自动化管理工具使用的是CDH的收费版本,为了刚好对cdh这类软件运维工具有深刻理解,所以自己搭建了一套基于免费开源CDH 6.3.2版本的集群;

        1.2 CDH介绍

Cloudera版本(Cloudera’s Distribution Including Apache Hadoop,简称CDH),基于Web的用户界面,支持大多数Hadoop组件,包括HDFSMapReduceHivePigHBaseZookeeperSqoop,简化了大数据平台的安装、使用简单。

Cloudera Manager的功能:

  • 管理:对集群进行管理,如添加、删除节点等操作。
  • 监控:监控集群的健康情况,对设置的各种指标和系统运行情况进行全面监控。
  • 诊断:对集群出现的问题进行诊断,对出现的问题给出建议解决方案。
  • 集成:多组件进行整合。

       

 1.3 展示成果

图中是我最终部署后的效果,由于规划时硬盘和内存给的都太小了导致上面出现告警的情况,需要在部署前规划好硬盘和内存;

1.4 注意事项

安装过程中一定会存在很多的问题,需要认真排查问题,我解决问题不下几十个,有时候是网络问题,有些事配置问题,需要通过报错一步一步解决,我会尽力将遇到的问题一一排除,让想了解cdh的能快速搭建好这样一个平台;

2. 准备工作

        2.1 前置准备

前提条件基本信息配置备注
CentOSLinux release 7.9.2009centos-7-isos-x86_64安装包下载_开源镜像站-阿里云最小安装

4台虚拟机

(最好5台)

192.168.2.186
192.168.2.187
192.168.2.188
192.168.2.189

最低配置要求

内存:4G

硬盘:40G

注意:要不然后面会存在硬盘不够用,节点硬盘太小导致的不健康提示

JDKoracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
Cloud Managercm6.3.1-redhat7阿里云盘分享
mysqlmysql57-community-release-el7-11.noarch.rpm

    由于资料很大,只能通过云盘分享:阿里云盘分享,部分资料由于太大了网盘设限了;如果有需要单独留言联系

    2.2 网络配置

#所有节点

#关闭防火墙
systemctl stop firewalld
systemctl disable firewalld

#设置ip和域名
vi /etc/hosts

192.168.2.186 hadoop101 hadoop101
192.168.2.187 hadoop102 hadoop102
192.168.2.188 hadoop103 hadoop103
192.168.2.189 hadoop104 hadoop104

#固定ip,防止重启时变化
vi /etc/sysconfig/network-scripts/ifcfg-ens33

IPADDR=192.168.2.186
NETMASK=255.255.255.0
GATEWAY=192.168.2.2
DNS1=8.8.8.8
DNS2=114.114.114.114

#重启网络
systemctl  restart network

        2.3 各个节点之间免密登录

#所有节点都执行
#生成公钥和私钥(敲三个回车,就会生成两个文件id_rsa(私钥)、id_rsa.pub(公钥))
[root@hadoop101 ~]$ ssh-keygen -t rsa

#进入.ssh目录
[root@hadoop101 ~]$ cd .ssh
[root@hadoop101 .ssh]$ ls
[root@hadoop101 .ssh]$ id_rsa id_rsa.pub

#复制公钥到另外一台服务器
[root@hadoop101 .ssh]$ ssh-copy-id hadoop101
[root@hadoop101 .ssh]$ ssh-copy-id hadoop102
[root@hadoop101 .ssh]$ ssh-copy-id hadoop103
[root@hadoop101 .ssh]$ ssh-copy-id hadoop104

        2.4 集群同步脚本 xsync

#在/root目录下创建bin目录,并在bin目录下创建文件xsync,文件内容如下:
[root@hadoop101 ~]$ mkdir bin
[root@hadoop101 ~]$ cd bin/
[root@hadoop101 bin]$ vi xsync

#在该文件中编写如下代码
#!/bin/bash
#1 获取输入参数个数,如果没有参数,直接退出
pcount=$#
if((pcount==0)); then
echo no args;
exit;
fi

#2 获取文件名称
p1=$1
fname=`basename $p1`
echo fname=$fname

#3 获取上级目录到绝对路径
pdir=`cd -P $(dirname $p1); pwd`
echo pdir=$pdir

#4 获取当前用户名称
user=`whoami`

#5 循环
for((host=102; host<106; host++)); do
        echo ------------------- hadoop$host --------------
        rsync -av $pdir/$fname $user@hadoop$host:$pdir
done

#退出后,修改脚本 xsync 具有执行权限
[root@hadoop101 bin]$ chmod 777 xsync

        2.5 集群整体操作脚本 xcall.sh

#在/root/bin目录下创建脚本xcall.sh
[root@hadoop101 bin]$ vim xcall.sh

#在脚本中编写如下内容
#! /bin/bash

for i in hadoop101 hadoop102 hadoop103 hadoop104 hadoop105
do
        echo --------- $i ----------
        ssh $i "$*"
done


#退出后,修改脚本执行权限
[root@hadoop101 bin]$ chmod 777 xcall.sh

3. 安装

3.1 安装JDK

#需要使用官网提供的jdk oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

#在hadoop101的/opt目录下创建module和software文件夹
[root@hadoop101 opt]# mkdir module
[root@hadoop101 opt]# mkdir software

#上传oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm并安装
[root@hadoop101 software]# rpm -ivh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm 
[root@hadoop101 software]#  vim /etc/profile.d/my_env.sh
export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin

[root@hadoop101 software]# source /etc/profile.d/my_env.sh
[root@hadoop101 software]# java -version
java version "1.8.0_181"

#分发,并source
[root@hadoop101 software]# xsync /usr/java/
[root@hadoop101 software]# xsync /etc/profile.d/my_env.sh 
[root@hadoop102 software]# source /etc/profile.d/my_env.sh
[root@hadoop103 software]# source /etc/profile.d/my_env.sh
[root@hadoop104 software]# source /etc/profile.d/my_env.sh
[root@hadoop101 software]# xcall.sh  java -version

3.2 安装MySQL

#安装包准备
[root@hadoop101 ~]# rpm -qa|grep -i mysql
[root@hadoop101 ~]# 
#如果安装了MySQL,就先卸载
[root@hadoop101 ~]# rpm -e --nodeps #mysql-version
#删除阿里云原有MySql依赖
[root@hadoop101 ~]# yum remove mysql-libs

#下载MySql依赖并安装
[root@hadoop101 ~]# yum install libaio
[root@hadoop101 ~]# yum -y install autoconf
[root@hadoop101 ~]# wget https://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm
[root@hadoop101 ~]# rpm -ivh mysql57-community-release-el7-11.noarch.rpm
#缺省会在/etc/yum.repos.d目录下生成两个配置文件
[root@hadoop101 ~]# ll /etc/yum.repos.d | grep mysql
#安装MySQL服务器
[root@hadoop101 ~]# rpm --import https://repo.mysql.com/RPM-GPG-KEY-mysql-2022
[root@hadoop101 ~]# yum -y install mysql-community-server
#启动mysql并查看状态
[root@hadoop101 ~]# systemctl start mysqld
[root@hadoop101 ~]# systemctl status mysqld
#获取临时密码
[root@hadoop101 ~]# grep "password" /var/log/mysqld.log
#设置root密码
[root@hadoop101 ~]# mysql_secure_installation
#设置新的root密码,缺省安全策略是MEDIUM,要求至少包含8个字符,其中至少有一个大小写字母,一个特殊#字符,一个数字,否则报 ... Failed! Error: Your password does not satisfy the current #policy requirements异常。


#设置内容:
Estimated strength of the password: 100 
Do you wish to continue with the password provided?(Press y|Y for Yes, any other key for No) : y
By default, a MySQL installation has an anonymous user,
allowing anyone to log into MySQL without having to have
a user account created for them. This is intended only for
testing, and to make the installation go a bit smoother.
You should remove them before moving into a production
environment.
 
# 移除匿名用户
Remove anonymous users? (Press y|Y for Yes, any other key for No) : y
Success.  
 
 
Normally, root should only be allowed to connect from
'localhost'. This ensures that someone cannot guess at
the root password from the network.
 
# 禁用远程root登录
Disallow root login remotely? (Press y|Y for Yes, any other key for No) : y
Success.
 
By default, MySQL comes with a database named 'test' that
anyone can access. This is also intended only for testing,
and should be removed before moving into a production
environment.
 
# 删除测试数据库及权限
Remove test database and access to it? (Press y|Y for Yes, any other key for No) : y
 - Dropping test database...
Success.
 
 - Removing privileges on test database...
Success.
 
Reloading the privilege tables will ensure that all changes
made so far will take effect immediately.
 
# 重新加载授权表
Reload privilege tables now? (Press y|Y for Yes, any other key for No) : y
Success.
 
All done! 

#修改安全策略
[root@hadoop101 ~]# mysql -uroot -p

# 查看当前安全策略
mysql>show variables like 'validate_password%';
# 最低安全策略(0->LOW ,1->MEDIUM, 2->STRONG)
mysql>set global validate_password_policy=0; 
# 最小密码字符长度是4
mysql>set global validate_password_length=4; 
# 设置本机root密码
mysql>alter user 'root'@'localhost' identified by 'root';

# 远程授权,all表示所有ddl和dml操作权限,*.*表示针对所有数据库所有表,%表示本机和远程主机均能访问
mysql>grant all privileges on *.* to 'root'@'%' identified by 'root' with grant option;
# 刷新权限
mysql>flush privileges;
#退出
mysql>exit

# 查询数据库列表
show databases
# 切换数据库
use mysql
# 查询数据库表 
show tables
# 查询用户信息
select * from user

#设置开机自启动
[root@hadoop101 ~]# systemctl enable mysqld

3.3 CM安装部署

3.3.1 MySQL中建库
#在MySQL中创建各组件需要的数据库
[root@hadoop101 ~]# mysql -uroot -proot
mysql> GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'root';

mysql> CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
mysql> CREATE DATABASE hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
mysql> CREATE DATABASE oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
mysql> CREATE DATABASE hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
mysql> CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
3.3.2 CM安装
#将mysql-connector-java-5.1.27-bin.jar上传到/usr/share/java路径下,并重命名mysql-connector-java.jar,分发出去
[root@hadoop101 mysql-connector-java-5.1.27]# xsync /usr/share/java/
3.3.3 集群规划

节点

hadoop101

hadoop102

hadoop103

Hadoop104

服务

cloudera-scm-server

cloudera-scm-agent

cloudera-scm-agent

cloudera-scm-agent

cloudera-scm-agent

 3.3.4 安装流程

#创建cloudera-manager目录,存放cdh安装文件
[root@hadoop101 mysql-connector-java-5.1.27]# mkdir /opt/cloudera-manager
[root@hadoop101 mysql-connector-java-5.1.27]# cd /opt/software/
[root@hadoop101 software]# tar -zxvf cm6.3.1-redhat7.tar.gz
[root@hadoop101 software]# cd cm6.3.1/RPMS/x86_64/
[root@hadoop101 x86_64]# mv cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm /opt/cloudera-manager/
[root@hadoop101 x86_64]# mv cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm /opt/cloudera-manager/
[root@hadoop101 x86_64]# mv cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm /opt/cloudera-manager/
[root@hadoop101 x86_64]# cd /opt/cloudera-manager/
[root@hadoop101 cloudera-manager]# ll

#安装cloudera-manager-daemons,安装完毕后多出/opt/cloudera目录
[root@hadoop101 cloudera-manager]# rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm 
[root@hadoop101 cloudera-manager]# cd /opt/
cloudera/         cloudera-manager/ 
[root@hadoop101 cloudera-manager]# cd ..
[root@hadoop101 opt]#  xsync /opt/cloudera-manager/
[root@hadoop102 ~]# cd /opt/cloudera-manager/
[root@hadoop102 cloudera-manager]# rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm 
[root@hadoop103 ~]# cd /opt/cloudera-manager/
[root@hadoop103 cloudera-manager]# rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm 
[root@hadoop104 ~]# cd /opt/cloudera-manager/
[root@hadoop104 cloudera-manager]# rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm 
[root@hadoop105 ~]# cd /opt/cloudera-manager/


#安装cloudera-manager-agent(hadoop101-hadoop104都要执行)
[root@hadoop101 cloudera-manager] yum install bind-utils psmisc cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs /lib/lsb/init-functions httpd mod_ssl openssl-devel python-psycopg2 MySQL-python libxslt
#[root@hadoop101 cloudera-manager]# rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm

#安装agent的server节点
[root@hadoop101 cloudera-manager]# vim /etc/cloudera-scm-agent/config.ini
server_host=hadoop101
[root@hadoop102 cloudera-manager]# vim /etc/cloudera-scm-agent/config.ini
server_host=hadoop101
[root@hadoop103 cloudera-manager]# vim /etc/cloudera-scm-agent/config.ini
server_host=hadoop101
[root@hadoop104 cloudera-manager]# vim /etc/cloudera-scm-agent/config.ini
server_host=hadoop101

#内容修改
server_host=hadoop101


#安装cloudera-manager-server
[root@hadoop101 cloudera-manager]# rpm -ivh cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm 

#上传CDH包到parcel-repo
[root@hadoop101 ~]# cd /opt/software/
[root@hadoop101 software]# mv CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel* /opt/cloudera/parcel-repo
[root@hadoop100 software]# mv manifest.json /opt/cloudera/parcel-repo
[root@hadoop101 parcel-repo]# cd /opt/cloudera/parcel-repo
[root@hadoop101 parcel-repo]# mv CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha
[root@hadoop101 parcel-repo]# ll

#修改server的db.properties
[root@hadoop101 parcel-repo]# vim /etc/cloudera-scm-server/db.properties 
com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=hadoop101:3306
com.cloudera.cmf.db.name=scm
com.cloudera.cmf.db.user=scm
com.cloudera.cmf.db.password=root
com.cloudera.cmf.db.setupType=EXTERNAL

#启动server服务
[root@hadoop101 log]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm root
[root@hadoop101 software]# systemctl start cloudera-scm-server 
[root@hadoop101 software]#tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log 

#启动agent节点
[root@hadoop101 software]# systemctl start cloudera-scm-agent
[root@hadoop102 software]# systemctl start cloudera-scm-agent
[root@hadoop103 software]# systemctl start cloudera-scm-agent
[root@hadoop104 software]# systemctl start cloudera-scm-agent

4. 登录 并配置

地址:192.168.2.186:7180/cmf/login

用户名:admin

密码:admin

 由于几个主机我已经弄好了,没有出现“当前管理的主机”的tab页,首次会出现,选中需要管理的主机,并打勾

 

 

修复问题

[root@hadoop101 software]# echo never > /sys/kernel/mm/transparent_hugepage/defrag
[root@hadoop101 software]# echo never > /sys/kernel/mm/transparent_hugepage/enabled
[root@hadoop102 software]# echo never > /sys/kernel/mm/transparent_hugepage/defrag
[root@hadoop102 software]# echo never > /sys/kernel/mm/transparent_hugepage/enabled
[root@hadoop103 software]# echo never > /sys/kernel/mm/transparent_hugepage/defrag
[root@hadoop103 software]# echo never > /sys/kernel/mm/transparent_hugepage/enabled
[root@hadoop104 software]# echo never > /sys/kernel/mm/transparent_hugepage/defrag
[root@hadoop104 software]# echo never > /sys/kernel/mm/transparent_hugepage/enabled

 

评论 9
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值