Hadoop on K8s 編排部署進階篇
一、概述
Hadoop 是一個開源的分布式計算框架,用于處理大規(guī)模數(shù)據(jù)集的存儲和處理。它使用了 Hadoop 分布式文件系統(tǒng)(Hadoop Distributed File System,HDFS)來存儲數(shù)據(jù),并通過 MapReduce 編程模型進行數(shù)據(jù)處理。
Kubernetes(通常簡稱為K8s)是一個開源的容器編排平臺,用于自動化部署、擴展和管理容器化應用程序。它提供了一種強大的方式來管理容器化應用程序的資源和生命周期。
將Hadoop部署在Kubernetes上(通常稱為Hadoop on K8s或Hadoop on Kubernetes)是一種將Hadoop與 Kubernetes 結合使用的方式。它將 Hadoop 集群中的各個組件(如 NameNode、DataNode、ResourceManager 和 NodeManager )打包為容器,并使用Kubernetes來自動管理和編排這些容器。
Hadoop on K8s具有以下一些優(yōu)勢:
- 彈性擴展:Kubernetes提供了動態(tài)擴展的能力,可以根據(jù)工作負載的需求自動調整Hadoop集群的規(guī)模。
- 靈活性:通過將Hadoop部署在Kubernetes上,可以更加靈活地管理Hadoop集群的資源分配和調度,以適應不同的工作負載。
- 多租戶支持:Kubernetes的多租戶支持使得可以在同一個Kubernetes集群上運行多個獨立的Hadoop集群,從而更好地隔離不同的應用和用戶。
- 資源利用率:Kubernetes可以更好地管理和利用集群資源,避免資源浪費,提高資源利用率。
- 故障恢復:Kubernetes提供了故障恢復和自愈能力,可以在節(jié)點故障時自動重新調度Hadoop容器,提高集群的可靠性。
要在 Kubernetes上部署 Hadoop 集群,需要使用適當?shù)墓ぞ吆团渲?,例?nbsp;Apache Hadoop Kubernetes 項目(Hadoop K8s)或其他第三方工具。這些工具提供了與 Kubernetes 集成的方式,并簡化了在Kubernetes 上部署和管理 Hadoop 集群的過程。
總之,Hadoop on K8s 提供了一種在Kubernetes上運行Hadoop集群的方式,充分利用了Kubernetes的彈性、靈活性和資源管理功能。它可以簡化Hadoop集群的部署和管理,并提供更好的資源利用率和可靠性。
之前也寫過一篇類似的文章,因操作的步驟比較多,這里將進行改進升級,感興趣的小伙伴請仔細閱讀下文,這里也提供通過docker-compse一鍵部署教程:
二、k8s 環(huán)境部署
k8s 環(huán)境部署這里不重復講解了,重點是 Hadoop on k8s,不知道怎么部署k8s環(huán)境的可以參考我以下幾篇文章:
三、開始編排部署 Hadoop
1)構建鏡像 Dockerfile
FROM registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/centos:7.7.1908
RUN rm -f /etc/localtime && ln -sv /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo "Asia/Shanghai" > /etc/timezone
RUN export LANG=zh_CN.UTF-8
# 創(chuàng)建用戶和用戶組,跟yaml編排里的user: 10000:10000
RUN groupadd --system --gid=10000 hadoop && useradd --system --home-dir /home/hadoop --uid=10000 --gid=hadoop hadoop
# 安裝sudo
RUN yum -y install sudo ; chmod 640 /etc/sudoers
# 給hadoop添加sudo權限
RUN echo "hadoop ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
RUN yum -y install install net-tools telnet wget nc expect which
RUN mkdir /opt/apache/
# 安裝 JDK
ADD jdk-8u212-linux-x64.tar.gz /opt/apache/
ENV JAVA_HOME /opt/apache/jdk1.8.0_212
ENV PATH $JAVA_HOME/bin:$PATH
# 配置 Hadoop
ENV HADOOP_VERSION 3.3.5
ADD hadoop-${HADOOP_VERSION}.tar.gz /opt/apache/
ENV HADOOP_HOME /opt/apache/hadoop
RUN ln -s /opt/apache/hadoop-${HADOOP_VERSION} $HADOOP_HOME
ENV HADOOP_COMMON_HOME=${HADOOP_HOME} \
HADOOP_HDFS_HOME=${HADOOP_HOME} \
HADOOP_MAPRED_HOME=${HADOOP_HOME} \
HADOOP_YARN_HOME=${HADOOP_HOME} \
HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop \
PATH=${PATH}:${HADOOP_HOME}/bin
# 配置Hive
ENV HIVE_VERSION 3.1.3
ADD apache-hive-${HIVE_VERSION}-bin.tar.gz /opt/apache/
ENV HIVE_HOME=/opt/apache/hive
ENV PATH=$HIVE_HOME/bin:$PATH
RUN ln -s /opt/apache/apache-hive-${HIVE_VERSION}-bin ${HIVE_HOME}
# 配置spark
ENV SPARK_VERSION 3.3.2
ADD spark-${SPARK_VERSION}-bin-hadoop3.tgz /opt/apache/
ENV SPARK_HOME=/opt/apache/spark
ENV PATH=$SPARK_HOME/bin:$PATH
RUN ln -s /opt/apache/spark-${SPARK_VERSION}-bin-hadoop3 ${SPARK_HOME}
# 配置 flink
ENV FLINK_VERSION 1.17.0
ADD flink-${FLINK_VERSION}-bin-scala_2.12.tgz /opt/apache/
ENV FLINK_HOME=/opt/apache/flink
ENV PATH=$FLINK_HOME/bin:$PATH
RUN ln -s /opt/apache/flink-${FLINK_VERSION} ${FLINK_HOME}
# 創(chuàng)建namenode、datanode存儲目錄
RUN mkdir -p /opt/apache/hadoop/data/{hdfs,yarn} /opt/apache/hadoop/data/hdfs/namenode /opt/apache/hadoop/data/hdfs/datanode/data{1..3} /opt/apache/hadoop/data/yarn/{local-dirs,log-dirs,apps}
COPY bootstrap.sh /opt/apache/
COPY config/hadoop-config/* ${HADOOP_HOME}/etc/hadoop/
# hive config
COPY hive-config/* ${HIVE_HOME}/conf/
COPY mysql-connector-java-5.1.49/mysql-connector-java-5.1.49-bin.jar ${HIVE_HOME}/lib/
RUN sudo mkdir -p /home/hadoop/ && sudo chown -R hadoop:hadoop /home/hadoop/
#RUN yum -y install which
ENV ll "ls -l"
RUN chown -R hadoop:hadoop /opt/apache
WORKDIR /opt/apache
bootstrap.sh 腳本內容
#!/usr/bin/env sh
source /etc/profile
wait_for() {
if [ -n "$1" -a -z -n "$2" ];then
echo Waiting for $1 to listen on $2...
while ! nc -z $1 $2; do echo waiting...; sleep 1s; done
fi
}
start_hdfs_namenode() {
namenode_dir=`grep -A1 'dfs.namenode.name.dir' ${HADOOP_HOME}/etc/hadoop/hdfs-site.xml |tail -1|sed 's/<value>//'|sed 's/<\/value>//'`
if [ ! -d ${namenode_dir}/current ];then
${HADOOP_HOME}/bin/hdfs namenode -format
fi
${HADOOP_HOME}/bin/hdfs --loglevel INFO --daemon start namenode
tail -f ${HADOOP_HOME}/logs/*namenode*.log
}
start_hdfs_datanode() {
wait_for $1 $2
${HADOOP_HOME}/bin/hdfs --loglevel INFO --daemon start datanode
tail -f ${HADOOP_HOME}/logs/*datanode*.log
}
start_yarn_resourcemanager() {
${HADOOP_HOME}/bin/yarn --loglevel INFO --daemon start resourcemanager
tail -f ${HADOOP_HOME}/logs/*resourcemanager*.log
}
start_yarn_nodemanager() {
wait_for $1 $2
${HADOOP_HOME}/bin/yarn --loglevel INFO --daemon start nodemanager
tail -f ${HADOOP_HOME}/logs/*nodemanager*.log
}
start_yarn_proxyserver() {
wait_for $1 $2
${HADOOP_HOME}/bin/yarn --loglevel INFO --daemon start proxyserver
tail -f ${HADOOP_HOME}/logs/*proxyserver*.log
}
start_mr_historyserver() {
wait_for $1 $2
${HADOOP_HOME}/bin/mapred --loglevel INFO --daemon start historyserver
tail -f ${HADOOP_HOME}/logs/*historyserver*.log
}
start_hive_metastore() {
if [ ! -f ${HIVE_HOME}/formated ];then
schematool -initSchema -dbType mysql --verbose > ${HIVE_HOME}/formated
fi
$HIVE_HOME/bin/hive --service metastore
}
start_hive_hiveserver2() {
$HIVE_HOME/bin/hive --service hiveserver2
}
case $1 in
hadoop-hdfs-nn)
start_hdfs_namenode $2 $3
;;
hadoop-hdfs-dn)
start_hdfs_datanode $2 $3
;;
hadoop-yarn-rm)
start_yarn_resourcemanager $2 $3
;;
hadoop-yarn-nm)
start_yarn_nodemanager $2 $3
;;
hadoop-yarn-proxyserver)
start_yarn_proxyserver $2 $3
;;
hadoop-mr-historyserver)
start_mr_historyserver $2 $3
;;
hive-metastore)
start_hive_metastore $2 $3
;;
hive-hiveserver2)
start_hive_hiveserver2 $2 $3
;;
*)
echo "請輸入正確的服務啟動命令~"
;;
esac
構建鏡像:
docker build -t registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/hadoop_hive:v1 . --no-cache
# 為了方便小伙伴下載即可使用,我這里將鏡像文件推送到阿里云的鏡像倉庫
docker push registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/hadoop_hive:v1
### 參數(shù)解釋
# -t:指定鏡像名稱
# . :當前目錄Dockerfile
# -f:指定Dockerfile路徑
# --no-cache:不緩存
【溫馨提示】如果不更換版本包,就無需再構建鏡像,我已經構建好傳到阿里云鏡像倉庫了。如果需要修改Hadoop版本,可以基于我的鏡像進行修改。
2)values.yaml 文件配置
image:
repository: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/hadoop_hive
tag: v1
pullPolicy: IfNotPresent
# The version of the hadoop libraries being used in the image.
hadoopVersion: 3.3.5
logLevel: INFO
# Select antiAffinity as either hard or soft, default is soft
antiAffinity: "soft"
hdfs:
nameNode:
replicas: 1
pdbMinAvailable: 1
resources:
requests:
memory: "1024Mi"
cpu: "1000m"
limits:
memory: "2048Mi"
cpu: "1000m"
dataNode:
# Will be used as dfs.datanode.hostname
# You still need to set up services + ingress for every DN
# Datanodes will expect to
externalHostname: example.com
externalDataPortRangeStart: 9866
externalHTTPPortRangeStart: 9864
replicas: 1
pdbMinAvailable: 1
resources:
requests:
memory: "1024Mi"
cpu: "1000m"
limits:
memory: "2048Mi"
cpu: "1000m"
webhdfs:
enabled: true
jounralNode:
replicas: 3
pdbMinAvailable: 1
resources:
requests:
memory: "1024Mi"
cpu: "1000m"
limits:
memory: "2048Mi"
cpu: "1000m"
mrHistoryserver:
pdbMinAvailable: 1
replicas: 1
resources:
requests:
memory: "1024Mi"
cpu: "1000m"
limits:
memory: "1024Mi"
cpu: "1000m"
yarn:
resourceManager:
pdbMinAvailable: 1
replicas: 1
resources:
requests:
memory: "1024Mi"
cpu: "1000m"
limits:
memory: "1024Mi"
cpu: "1000m"
nodeManager:
pdbMinAvailable: 1
# The number of YARN NodeManager instances.
replicas: 1
# Create statefulsets in parallel (K8S 1.7+)
parallelCreate: false
# CPU and memory resources allocated to each node manager pod.
# This should be tuned to fit your workload.
resources:
requests:
memory: "1024Mi"
cpu: "1000m"
limits:
memory: "2048Mi"
cpu: "1000m"
proxyServer:
pdbMinAvailable: 1
replicas: 1
resources:
requests:
memory: "1024Mi"
cpu: "1000m"
limits:
memory: "1024Mi"
cpu: "1000m"
hive:
metastore:
replicas: 1
pdbMinAvailable: 1
resources:
requests:
memory: "1024Mi"
cpu: "1000m"
limits:
memory: "2048Mi"
cpu: "1000m"
hiveserver2:
replicas: 1
pdbMinAvailable: 1
resources:
requests:
memory: "1024Mi"
cpu: "1000m"
limits:
memory: "1024Mi"
cpu: "1000m"
persistence:
nameNode:
enabled: true
enabledStorageClass: false
storageClass: "hadoop-nn-local-storage"
accessMode: ReadWriteOnce
size: 1Gi
local:
#- name: hadoop-nn-0
# host: "local-168-182-110"
# path: "/opt/bigdata/servers/hadoop/nn/data/data1"
volumes:
- name: nn1
mountPath: /opt/apache/hadoop/data/hdfs/namenode
hostPath: /opt/bigdata/servers/hadoop/nn/data/data1
dataNode:
enabled: true
enabledStorageClass: false
storageClass: "hadoop-dn-local-storage"
accessMode: ReadWriteOnce
size: 1Gi
#local:
#- name: hadoop-dn-0
# host: "local-168-182-110"
# path: "/opt/bigdata/servers/hadoop/dn/data/data1"
#- name: hadoop-dn-1
# host: "local-168-182-110"
# path: "/opt/bigdata/servers/hadoop/dn/data/data2"
#- name: hadoop-dn-2
# host: "local-168-182-110"
# path: "/opt/bigdata/servers/hadoop/dn/data/data3"
#- name: hadoop-dn-3
# host: "local-168-182-111"
# path: "/opt/bigdata/servers/hadoop/dn/data/data1"
#- name: hadoop-dn-4
# host: "local-168-182-111"
# path: "/opt/bigdata/servers/hadoop/dn/data/data2"
#- name: hadoop-dn-5
# host: "local-168-182-111"
# path: "/opt/bigdata/servers/hadoop/dn/data/data3"
#- name: hadoop-dn-6
# host: "local-168-182-112"
# path: "/opt/bigdata/servers/hadoop/dn/data/data1"
#- name: hadoop-dn-7
# host: "local-168-182-112"
# path: "/opt/bigdata/servers/hadoop/dn/data/data2"
#- name: hadoop-dn-8
# host: "local-168-182-112"
# path: "/opt/bigdata/servers/hadoop/dn/data/data3"
volumes:
- name: dfs1
mountPath: /opt/apache/hdfs/datanode1
hostPath: /opt/bigdata/servers/hadoop/dn/data/data1
- name: dfs2
mountPath: /opt/apache/hdfs/datanode2
hostPath: /opt/bigdata/servers/hadoop/dn/data/data2
- name: dfs3
mountPath: /opt/apache/hdfs/datanode3
hostPath: /opt/bigdata/servers/hadoop/dn/data/data3
service:
nameNode:
type: NodePort
ports:
dfs: 9000
webhdfs: 9870
nodePorts:
dfs: 30900
webhdfs: 30870
dataNode:
type: NodePort
ports:
webhdfs: 9864
nodePorts:
webhdfs: 30864
mrHistoryserver:
type: NodePort
ports:
web: 19888
nodePorts:
web: 30888
resourceManager:
type: NodePort
ports:
web: 8088
nodePorts:
web: 30088
nodeManager:
type: NodePort
ports:
web: 8042
nodePorts:
web: 30042
proxyServer:
type: NodePort
ports:
web: 9111
nodePorts:
web: 30911
hive:
metastore:
type: NodePort
port: 9083
nodePort: 31183
hiveserver2:
type: NodePort
port: 10000
nodePort: 30000
securityContext:
runAsUser: 10000
privileged: true
【溫馨提示】這里的 namenode 和 datanode 存儲目錄使用 hostPath 掛載方式,通過 enabledStorageClass 來控制是選擇宿主機還是PVC掛載,為 false 是 hostPath 掛載方式,反之亦然。
在每個k8s節(jié)點上創(chuàng)建掛載目錄:
# 如果使用pv,pvc掛載方式,就不需要在宿主機上創(chuàng)建目錄了,非高可用可不用創(chuàng)建jn
mkdir -p /opt/bigdata/servers/hadoop/{nn,jn,dn}/data/data{1..3}
chmod 777 -R /opt/bigdata/servers/hadoop/
3)hadoop configmap yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "hadoop.fullname" . }}
labels:
app.kubernetes.io/name: {{ include "hadoop.name" . }}
helm.sh/chart: {{ include "hadoop.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
data:
core-site.xml: |
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://{{ include "hadoop.fullname" . }}-hdfs-nn:9000/</value>
<description>NameNode URI</description>
</property>
<property>
<name>hadoop.proxyuser.root.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.root.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hadoop.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hadoop.groups</name>
<value>*</value>
</property>
</configuration>
hdfs-site.xml: |
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
{{- if .Values.hdfs.webhdfs.enabled -}}
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
{{- end -}}
<property>
<name>dfs.datanode.use.datanode.hostname</name>
<value>false</value>
</property>
<property>
<name>dfs.client.use.datanode.hostname</name>
<value>false</value>
</property>
<!--
<property>
<name>dfs.datanode.hostname</name>
<value>{{ .Values.hdfs.dataNode.externalHostname }}</value>
</property>
-->
<property>
<name>dfs.namenode.datanode.registration.ip-hostname-check</name>
<value>false</value>
</property>
<property>
<name>dfs.datanode.http.address</name>
<value>0.0.0.0:9864</value>
</property>
<property>
<name>dfs.datanode.address</name>
<value>0.0.0.0:9866</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/opt/apache/hadoop/data/hdfs/datanode/data1,/opt/apache/hadoop/data/hdfs/datanode/data2,/opt/apache/hadoop/data/hdfs/datanode/data3</value>
<description>DataNode directory</description>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/opt/apache/hadoop/data/hdfs/namenode</value>
<description>NameNode directory for namespace and transaction logs storage.</description>
</property>
<property>
<name>dfs.namenode.datanode.registration.ip-hostname-check</name>
<value>false</value>
</property>
<!-- Bind to all interfaces -->
<property>
<name>dfs.namenode.rpc-bind-host</name>
<value>0.0.0.0</value>
</property>
<property>
<name>dfs.namenode.servicerpc-bind-host</name>
<value>0.0.0.0</value>
</property>
<!-- /Bind to all interfaces -->
</configuration>
mapred-site.xml: |
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>{{ include "hadoop.fullname" . }}-mr-historyserver-0:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>{{ include "hadoop.fullname" . }}-mr-historyserver-0:{{ .Values.service.mrHistoryserver.ports.web }}</value>
</property>
</configuration>
yarn-site.xml: |
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>{{ include "hadoop.fullname" . }}-yarn-rm-headless</value>
</property>
<!-- Bind to all interfaces -->
<property>
<name>yarn.resourcemanager.bind-host</name>
<value>0.0.0.0</value>
</property>
<property>
<name>yarn.nodemanager.bind-host</name>
<value>0.0.0.0</value>
</property>
<property>
<name>yarn.timeline-service.bind-host</name>
<value>0.0.0.0</value>
</property>
<!-- /Bind to all interfaces -->
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<description>List of directories to store localized files in.</description>
<name>yarn.nodemanager.local-dirs</name>
<value>/opt/apache/hadoop/data/yarn/local-dirs</value>
</property>
<property>
<description>Where to store container logs.</description>
<name>yarn.nodemanager.log-dirs</name>
<value>/opt/apache/hadoop/data/yarn/log-dirs</value>
</property>
<property>
<description>Where to aggregate logs to.</description>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/opt/apache/hadoop/data/yarn/apps</value>
</property>
<property>
<name>yarn.web-proxy.address</name>
<value>{{ include "hadoop.fullname" . }}-yarn-proxyserver-0:{{ .Values.service.proxyServer.ports.web }}</value>
</property>
<property>
<name>yarn.application.classpath</name>
<value>
/opt/apache/hadoop/etc/hadoop,
/opt/apache/hadoop/share/hadoop/common/*,
/opt/apache/hadoop/share/hadoop/common/lib/*,
/opt/apache/hadoop/share/hadoop/hdfs/*,
/opt/apache/hadoop/share/hadoop/hdfs/lib/*,
/opt/apache/hadoop/share/hadoop/mapreduce/*,
/opt/apache/hadoop/share/hadoop/mapreduce/lib/*,
/opt/apache/hadoop/share/hadoop/yarn/*,
/opt/apache/hadoop/share/hadoop/yarn/lib/*
</value>
</property>
</configuration>
dfs-hosts.includes: |
{{ include "hadoop.fullname" . }}-hdfs-dn-0.{{ include "hadoop.fullname" . }}-hdfs-dn.{{ .Release.Namespace }}.svc.cluster.local
{{ include "hadoop.fullname" . }}-hdfs-dn-1.{{ include "hadoop.fullname" . }}-hdfs-dn.{{ .Release.Namespace }}.svc.cluster.local
{{ include "hadoop.fullname" . }}-hdfs-dn-2.{{ include "hadoop.fullname" . }}-hdfs-dn.{{ .Release.Namespace }}.svc.cluster.local
dfs-hosts.excludes: |
yarn-hosts.includes: |
{{ include "hadoop.fullname" . }}-yarn-nm-0.{{ include "hadoop.fullname" . }}-yarn-nm.{{ .Release.Namespace }}.svc.cluster.local
{{ include "hadoop.fullname" . }}-yarn-nm-1.{{ include "hadoop.fullname" . }}-yarn-nm.{{ .Release.Namespace }}.svc.cluster.local
{{ include "hadoop.fullname" . }}-yarn-nm-2.{{ include "hadoop.fullname" . }}-yarn-nm.{{ .Release.Namespace }}.svc.cluster.local
yarn-hosts.excludes: |
4)hive configmap yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "hadoop.fullname" . }}-hive
labels:
app.kubernetes.io/name: {{ include "hadoop.name" . }}
helm.sh/chart: {{ include "hadoop.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}-hive
data:
hive-site.xml: |
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- 配置hdfs存儲目錄 -->
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/hive_remote/warehouse</value>
</property>
<property>
<name>hive.metastore.local</name>
<value>false</value>
</property>
<!-- 所連接的 MySQL 數(shù)據(jù)庫的地址,hive_local是數(shù)據(jù)庫,程序會自動創(chuàng)建,自定義就行 -->
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://192.168.182.110:13306/hive_metastore?createDatabaseIfNotExist=true&useSSL=false&serverTimezone=Asia/Shanghai</value>
</property>
<!-- MySQL 驅動 -->
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<!--<value>com.mysql.cj.jdbc.Driver</value>-->
<value>com.mysql.jdbc.Driver</value>
</property>
<!-- mysql連接用戶 -->
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
</property>
<!-- mysql連接密碼 -->
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>123456</value>
</property>
<!--元數(shù)據(jù)是否校驗-->
<property>
<name>hive.metastore.schema.verification</name>
<value>false</value>
</property>
<property>
<name>system:user.name</name>
<value>root</value>
<description>user name</description>
</property>
<property>
<name>hive.metastore.uris</name>
<value>thrift://{{ include "hadoop.fullname" . }}-hive-metastore-0.{{ include "hadoop.fullname" . }}-hive-metastore:{{ .Values.service.hive.metastore.port }}</value>
</property>
<!-- host -->
<property>
<name>hive.server2.thrift.bind.host</name>
<value>0.0.0.0</value>
<description>Bind host on which to run the HiveServer2 Thrift service.</description>
</property>
<!-- hs2端口 默認是10000-->
<property>
<name>hive.server2.thrift.port</name>
<value>{{ .Values.service.hive.hiveserver2.port }}</value>
</property>
<property>
<name>hive.server2.active.passive.ha.enable</name>
<value>true</value>
</property>
</configuration>
【溫馨提示】這里只是列舉出重要的配置和腳本。文末會提供git 下載地址,下載整個部署包。
5)開始安裝
cd hadoop-on-kubernetes
# 安裝
helm install hadoop ./ -n hadoop --create-namespace
# 更新
helm upgrade hadoop ./ -n hadoop
# 卸載
helm uninstall hadoop -n hadoop
6)測試驗證
hdfs web:http://ip:30870
yarn web:http://ip:
通過 hive 創(chuàng)建庫表和添加數(shù)據(jù)驗證集群可用性
kubectl exec -it hadoop-hadoop-hive-hiveserver2-0 -n hadoop -- bash
beeline -u jdbc:hive2://hadoop-hadoop-hive-hiveserver2:10000 -n hadoop
# 建表
CREATE TABLE mytable (
id INT,
name STRING,
age INT,
address STRING
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n';
# 添加數(shù)據(jù)
INSERT INTO mytable VALUES (1, 'Alice', 25, 'F'), (2, 'Bob', 30, 'M'), (3, 'Charlie', 35, 'M');
hadoop-on-kubernetes下載地址:https://gitee.com/hadoop-bigdata/hadoop-on-kubernetes,后面會單獨拿一篇文章來講解部署時需要修改的地方和注意事項。