Spark on Yarn 单节点集群 on minikube 的 Mac OS部署趟坑指北

一、环境准备

1.1 安装 Docker-ce

一键式下载安装
https://docs.docker.com/docker-for-mac/install/

1.2 安装 MiniKube

官网 https://kubernetes.io/docs/tasks/tools/install-minikube/

以 macOS 为例：

1. 验证是否开启虚拟化，有输入证明已经开启虚拟化

$ sysctl -a | grep -E --color 'machdep.cpu.features|VMX' 

2. 安装 kubernetes-cli，kubectx（可选），kube-ps1（可选）

$ brew install kubernetes-cli kubectx kube-ps1

3. 配置 zsh plugin（可选）

$ vim ~/.zshrc

# 加入
export KUBE_PS1_ENABLED="off"

plugins=(
    git
    brew
    kubectl
    kube-ps1
)

source "/usr/local/opt/kube-ps1/share/kube-ps1.sh"
PS1='$(kube_ps1)'$PS1

之后就可以使用 kubectl 的命令缩写和自动补齐功能

4. 安装 Hypervisor

可以安装一下三个虚拟驱动作为监管系统，如果不安装默认使用HyperKit

• HyperKit
• VirtualBox
• VMware Fusion

5. 安装 minikube

$ brew install minikube

6. 尝试启动 minikube，可能需要梯子

# <driver_name> = 上面所安装的虚拟驱动，注意是全小写，还要记得加上引号
$ minikube start --vm-driver=<driver_name>

# 比如我启动的命令
$ minikube start --vm-driver='virtualbox' --kubernetes-version='v1.14.8' --extra-config=apiserver.service-node-port-range=1-65535 --alsologtostderr

# 因为 Spark 和 Yarn 都很需要计算资源
# 所以这里必须使用 3个以上CPU和尽量大的内存
# 但是 minikube start 我加入 --memory 和 --cpus 无效
# 所以需要打开 VirtualBox 手动修改一下

7. 验证开启成功

$ minikube status

# 输出如下内容证明启动成功
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

8. 使用 Minikube Docker Deamon，避免镜像反复拉取

$ eval $(minikube docker-env)

9. 配置 minikube 网络

# 这里需要每次手动修改 minikube 的 /etc/hosts 文件
# 其实可以用 minikube mount 把文件挂载持久化
# 但是 minikube 的 mount 有些小 Bug，比如
# 1. 需要一直开启命令行保持挂载
# 2. 挂载之后如果关闭就GG了，再也挂载不上了（官网issue已经解决）

# 获得 minikube 虚拟机的 IP
$ minikube ip
192.168.99.102

# 进入minikube 修改 /etc/hosts
$ minikube ssh
$ su
$ vi /etc/hosts
127.0.0.1       localhost
192.168.99.102  minikube

10. 配置本机网络

$ sudo vim /etc/hosts
192.168.99.102  minikube

原文地址 https:////www.cnblogs.com/clockq/p/12190971.html

二、启动 HDSP 集群

按照下列配置逐个启动

如果不更改任何配置，也可以直接 kubectl -f https://github.com/PharbersDeveloper/bp-hadoop-container/blob/minikube-v1.1.0/hdsp-deploy/mandatory.yaml 直接全部启动

2.1 Namespace 配置

apiVersion: v1
kind: Namespace
metadata:
  name: hdsp-ns
  labels:
    app.kubernetes.io/name: hdsp-ns
    app.kubernetes.io/part-of: hdsp-ns

2.2 StorageClass 配置

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-sc
  namespace: hdsp-ns
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
    creationTimestamp: "2019-12-18"
    resourceVersion: "v0.1"
provisioner: kubernetes.io/no-provisioner
reclaimPolicy: Retain # Supported policies: Delete, Retain
allowVolumeExpansion: false
volumeBindingMode: WaitForFirstConsumer

2.3 master PV 配置

需要提前创建 PV 的挂载目录

$ minikube ssh
$ su
$ mkdir -p /tmp/hostpath_pv/k8s-master-hdfs-pv
$ mkdir -p /tmp/hostpath_pv/k8s-worker1-hdfs-pv

apiVersion: v1
kind: PersistentVolume
metadata:
  name: master-hdfs-pv
  namespace: hdsp-ns
  annotations:
    use: &use master-hdfs
    node_name: &node_name minikube
    local_path: &local_path /tmp/hostpath_pv/k8s-master-hdfs-pv
    capacity_storage: &capacity_storage 2Gi
  labels:
    use: *use
    app: hdsp
spec:
  capacity:
    storage: *capacity_storage
  accessModes:
  - ReadWriteOnce
  storageClassName: local-sc
  local:
    path: *local_path
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - *node_name

2.4 master PVC 配置

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: master-hdfs-pvc
  namespace: hdsp-ns
  annotations:
    use: &use master-hdfs
    node_name: &node_name minikube
    requests_storage: &requests_storage 2Gi
  labels:
    use: &use master-hdfs
    app: hdsp
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: local-sc
  resources:
    requests:
      storage: *requests_storage
  selector:
    matchLabels:
      use: *use

2.5 master deamon set 配置

先配置 node 标签

$ kubectl label nodes minikube hdsprole=master

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: hdsp-master
  namespace: hdsp-ns
  labels:
    app: hdsp
    noderole: master
spec:
  selector:
    matchLabels:
      app: hdsp
      noderole: master
  template:
    metadata:
      labels:
        app: hdsp
        noderole: master
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node-role.kubernetes.io/master
                operator: Exists
            - matchExpressions:
              - key: hdsprole
                operator: In
                values:
                - master
      hostNetwork: true
      containers:
      - name: hdsp-master
        image: pharbers/hadoop:minikube
        imagePullPolicy: IfNotPresent
        args:
        - "-d"
        - "-master"
        securityContext:
          privileged: true
        volumeMounts:
        - mountPath: /usr/soft/hadoop/hdfs
          name: hadoop-data
        livenessProbe:
          httpGet:
            path: /cluster
            port: 8088
          initialDelaySeconds: 100
          timeoutSeconds: 20
          periodSeconds: 100
      imagePullSecrets:
      - name: registry-secret
      volumes:
      - name: hadoop-data
        persistentVolumeClaim:
          claimName: master-hdfs-pvc

2.6 测试

$ kubectl -n hdsp-ns describe pod hdsp-master-m75n4

输出如下信息表示启动成功

...
...
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  10m                   default-scheduler  Successfully assigned hdsp-ns/hdsp-master-m75n4 to minikube
  Warning  Unhealthy  4m6s (x3 over 7m26s)  kubelet, minikube  Liveness probe failed: Get http://192.168.99.102:8088/cluster: dial tcp 192.168.99.102:8088: connect: connection refused
  Normal   Killing    4m6s                  kubelet, minikube  Container hdsp-master failed liveness probe, will be restarted
  Normal   Pulled     3m36s (x2 over 10m)   kubelet, minikube  Container image "pharbers/hadoop:minikube" already present on machine
  Normal   Created    3m36s (x2 over 10m)   kubelet, minikube  Created container hdsp-master
  Normal   Started    3m36s (x2 over 10m)   kubelet, minikube  Started container hdsp-master

如果输入结果类似下方，证明 Yarn 集群启动失败，无法查看 Yarn UI 界面，存活探针出错，Pod会自动重启

...
...
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  10m                   default-scheduler  Successfully assigned hdsp-ns/hdsp-master-m75n4 to minikube
  Warning  Unhealthy  4m6s (x3 over 7m26s)  kubelet, minikube  Liveness probe failed: Get http://192.168.99.102:8088/cluster: dial tcp 192.168.99.102:8088: connect: connection refused

2.7 源码

上方教程使用已编好的镜像，如果需要修改配置或查看源码，欢迎查看 https://github.com/PharbersDeveloper/bp-hadoop-container/tree/hdsp-on-minikube

三、minikube 常见问题

3.1 因为环境隔离，镜像不方便运行

1. 使用 Minikube Docker Deamon，避免镜像反复拉取

$ eval $(minikube docker-env)

2. 拷贝本地镜像到 minikube

$ docker save pharbers/hadoop:using | (eval $(minikube docker-env) && docker load)

3.2 使用 NodePort 范围受限

扩大 NodePort 范围

minikube start --extra-config=apiserver.service-node-port-range=1-65535

3.3 如果 Pod 需要回连宿主机

获得与 Pod 同网段宿主 IP

minikube ssh "route -n | grep ^0.0.0.0 | awk '{ print $2 }'"