运维 centos Hadoop与Hive单节点环境安装文档

Hadoop与Hive单节点环境安装文档

软件清单安装步骤上传安装包至服务器基础环境配置改hosts和hostName防火墙关闭免密登录

基础软件安装NTP时间同步安装与配置JDK安装MySQL安装

Hadoop安装环境变量配置hadoop-env.sh 配置core-site.xml 配置hdfs-site.xml 配置mapred-site.xml 配置yarn-site.xml 配置workers 配置格式化hadoop启动hadoop

Hive安装Hive安装Hive环境变量配置hive-env.sh 配置hive-site.xml 配置升级元数据

异常参考

软件清单

名称版本Hadoop3.3.1Hive3.1.2MySQL5.7.21JDKjdk_8u171mysql-connector5.1.47

安装步骤

上传安装包至服务器

!注意不要漏传了

将软件拖入如下页面即可上传

基础环境配置

改hosts和hostName

## 改主机名

vi /etc/hostname

hadoop001

## 改ip 主机名称映射

vi /etc/hosts

防火墙关闭

systemctl stop firewalld.service

systemctl disable firewalld.service

##检查状态

systemctl status firewalld.service

免密登录

ssh-keygen

#一路回车

Generating public/private rsa key pair.

Enter file in which to save the key (/root/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /root/.ssh/id_rsa.

Your public key has been saved in /root/.ssh/id_rsa.pub.

The key fingerprint is:

df:71:f6:3e:bb:bb:6c:38:91:f4:bc:70:a1:dd:86:a9 root@flower1

The key's randomart image is:

+--[ RSA 2048]----+

| |

| |

| |

| . . |

| S o Ooo|

| . . Oo*o|

| . ..=.o|

| Eo.= |

| o*B|

+-----------------+

cd ~/.ssh

# 把生成的 id_rsa.pub 文件内容追加到 authorized_keys 中

cat id_rsa.pub >> authorized_keys

# authorized_keys 授权600，不然ssh node01 还是需要输入密码

chmod 600 authorized_keys

基础软件安装

NTP时间同步安装与配置

##执行bash安装ntp

yum install -y ntp

添加阿里时间同步服务器

# 第一步

vim /etc/ntp.conf

# 在文件中加入阿里时间同步服务器

server ntp.aliyun.com iburst

# 启动ntp服务

service ntpd start

# 设置开机启动

chkconfig ntpd on

JDK安装

##解压到指定目录

tar -zxvf jdk-8u171-linux-x64.tar.gz -C /usr/local/

vi /etc/profile

##添加如下

export JAVA_HOME=/usr/local/jdk1.8.0_171

export PATH=$JAVA_HOME/bin:$PATH:$HOME/bin

##使配置生效

source /etc/profile

##检查java安装是否成功

java -version

若成功显示如下：

MySQL安装

##卸载删除原有的mariadb，否则可能会报异常

rpm -qa|grep mariadb

## mariadb-xxxxxxxxxxxxxx 为rpm -qa|grep mariadb查询的结果

rpm -e --nodeps mariadb-xxxxxxxxxxxxxx

##再次查询，结果为空表示删除干净

rpm -qa|grep mariadb

##建立文件夹

cd ~

mkdir mysql5.7

##解压到该文件夹中

tar -xvf mysql-5.7.21-1.el7.x86_64.rpm-bundle.tar -C mysql5.7

cd mysql5.7

##安装

sudo rpm -ivh mysql*.rpm --nodeps --force

启动MySQL

##查看mysql状态

systemctl status mysqld

#启动Mysql

systemctl start mysqld

获取初始密码

grep 'temporary password' /var/log/mysqld.log

改密码

##初次登录

mysql -uroot -p

##输入grep 'temporary password' /var/log/mysqld.log 步骤得到的密码

# 修改密码复杂度

set global validate_password_policy=0;

# 修改密码长度

set global validate_password_length=1;

# 修改密码为1234

ALTER USER 'root'@'localhost' IDENTIFIED BY '1234';

# 开启远程访问

GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '1234' WITH GRANT OPTION;

##使权限生效

FLUSH PRIVILEGES;

##建立Hive数据库

#这个hive数据库与hive-site.xml中localhost:3306/hive的hive对应，两者名称必须一致，用来保存hive元数据

create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

设置开机自启

systemctl enable mysqld ##打开mysql的开机自启

systemctl disable mysqld ##关闭mysql开机自启

systemctl daemon-reload

重启可正常使用mysql

##重启

systemctl restart mysqld

Hadoop安装

环境变量配置

##创建必要路径 根据实际调整

mkdir -p /opt/bigdata/

mkdir -p /bigdata/hadoop/tmp

mkdir -p /bigdata/hadoop/data/dfs/name

mkdir -p /bigdata/hadoop/data/dfs/data

##解压安装hadoop3

## 以下配置文件均在下述目录

cd /opt/bigdata/hadoop-3.3.1/etc/hadoop

vi /etc/profile

##新增如下

export HADOOP_HOME=/opt/bigdata/hadoop-3.3.1

export PATH=$PATH:%JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

##使变量生效

source /etc/profile

hadoop-env.sh 配置

## 编辑该配置文件

vi hadoop-env.sh

## 增加

export JAVA_HOME=/usr/local/jdk1.8.0_171/

core-site.xml 配置

## 编辑该配置文件

vi core-site.xml

fs.defaultFS

hdfs://hadoop001:9000

hdfs内部通讯访问地址

hadoop.tmp.dir

/bigdata/hadoop/tmp

hadoop.proxyuser.root.hosts

*

hadoop.proxyuser.root.groups

*

hadoop.proxyuser.root.users

*

hdfs-site.xml 配置

## 编辑该配置文件

vi hdfs-site.xml

dfs.namenode.name.dir

/bigdata/hadoop/data/dfs/name

namenode 存放name table(fsimage)本地目录需要修改,如果没有需要自己创建文件目录)

dfs.datanode.data.dir

/bigdata/hadoop/data/dfs/data

datanode存放block本地目录（需要修改,如果没有需要自己创建文件目录）

dfs.replication

1

mapred-site.xml 配置

## 编辑该配置文件

vi mapred-site.xml

mapreduce.framework.name

yarn

yarn.app.mapreduce.am.env

HADOOP_MAPRED_HOME=/opt/bigdata/hadoop-3.3.1/etc/hadoop:/opt/bigdata/hadoop-3.3.1/share/hadoop/common/lib/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/common/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/hdfs:/opt/bigdata/hadoop-3.3.1/share/hadoop/hdfs/lib/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/hdfs/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/mapreduce/lib/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/mapreduce/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/yarn:/opt/bigdata/hadoop-3.3.1/share/hadoop/yarn/lib/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/yarn/*

mapreduce.map.env

HADOOP_MAPRED_HOME=/opt/bigdata/hadoop-3.3.1/etc/hadoop:/opt/bigdata/hadoop-3.3.1/share/hadoop/common/lib/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/common/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/hdfs:/opt/bigdata/hadoop-3.3.1/share/hadoop/hdfs/lib/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/hdfs/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/mapreduce/lib/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/mapreduce/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/yarn:/opt/bigdata/hadoop-3.3.1/share/hadoop/yarn/lib/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/yarn/*

mapreduce.reduce.env

HADOOP_MAPRED_HOME=/opt/bigdata/hadoop-3.3.1/etc/hadoop:/opt/bigdata/hadoop-3.3.1/share/hadoop/common/lib/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/common/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/hdfs:/opt/bigdata/hadoop-3.3.1/share/hadoop/hdfs/lib/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/hdfs/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/mapreduce/lib/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/mapreduce/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/yarn:/opt/bigdata/hadoop-3.3.1/share/hadoop/yarn/lib/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/yarn/*

yarn-site.xml 配置

## 编辑该配置文件

vi yarn-site.xml

yarn.scheduler.minimum-allocation-mb

2048

default value is 1024

yarn.scheduler.maximum-allocation-mb

4096

yarn.nodemanager.aux-services

mapreduce_shuffle

workers 配置

## 编辑该配置文件

vi workers

hadoop001

格式化hadoop

#hadoop namenode格式化，看到INFO common.Storage: Storage directory /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/data/dfs/name has been successfully formatted.即为成功格式化

[hadoop@hadoop001 hadoop]$ hadoop namenode -format

启动hadoop

[hadoop@hadoop001 hadoop] start-all.sh

#启动完之后,可以jps命令查看一下是否启动成功

[hadoop@hadoop001 hadoop] jps

2241 NameNode

2599 NodeManager

2987 Jps

2350 DataNode

2927 ResourceManager

#测试一下hdfs能否正常运行

[hadoop@hadoop001 hadoop]$ hdfs dfs -ls

19/04/09 00:05:03 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

ls: `.': No such file or directory

Hive安装

Hive安装

tar -zxvf apache-hive-3.1.2-bin.tar.gz -C /opt/bigdata

拷贝mysql-connector-java-5.1.47-bin.jar至$Hive_HOME/lib目录下，如果没有jar包的话会报错

Hive环境变量配置

vi /etc/profile

##新增如下

export HIVE_HOME=/opt/bigdata/hive3.1.2

PATH=$HIVE_HOME/bin:$PATH

export PATH

##使变量生效

source /etc/profile

##改名

cd /opt/bigdata/

mv apache-hive-3.1.2-bin hive3.1.2

### 配置文件位置

cd /opt/bigdata/hive3.1.2/conf

hive-env.sh 配置

vi hive-env.sh

export HADOOP_HOME=/opt/bigdata/hadoop-3.3.1/

export HIVE_CONF_DIR=/opt/bigdata/hive3.1.2/conf

export HIVE_AUX_JARS_PATH=/opt/bigdata/hive3.1.2/lib

hive-site.xml 配置

javax.jdo.option.ConnectionURL

jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true&characterEncoding=UTF-8&useSSL=false

javax.jdo.option.ConnectionDriverName

com.mysql.jdbc.Driver

javax.jdo.option.ConnectionUserName

root

javax.jdo.option.ConnectionPassword

1234

hive.metastore.warehouse.dir

/user/hive/warehouse

升级元数据

使用Hive自带的schematool工具升级元数据，也就是把最新的元数据重新写入MySQL数据库中。 可以在终端中执行如下命令（注意，不是在“mysql>”命令提示下执行）：

cd /opt/bigdata/hive3.1.2

./bin/schematool -initSchema -dbType mysql

异常

拷贝mysql-connector至$Hive_HOME/lib目录下，如果没有jar包的话会报错

HDFS_DATANODE_USER=root

HADOOP_SECURE_DN_USER=root

HDFS_NAMENODE_USER=root

HDFS_SECONDARYNAMENODE_USER=root

YARN_RESOURCEMANAGER_USER=root

HADOOP_SECURE_DN_USER=root

YARN_NODEMANAGER_USER=root

参考

Linux安装jdk

Hive3.1.3安装和使用指南

精彩文章

 您阅读本篇文章共花了：