k8s apiserver高可用方案

目前官方推荐有 2 种方式部署k8s apiserver 高可用

keepalived and haproxy

部署有2种方式，一种是systemd管理的，另一种是pod形式，使用那种可以根据实际情况选择

服务部署

systemd方式

可以通过包管理工具安装，正常启动之后，通过 kubeadm init 安装集群，配置好 apiserver 地址 VIP:${APISERVER_DEST_PORT}，配置参数见下面👇

pod方式

如果 keepalived 和 haproxy 将在控制平面节点上运行，则可以将它们配置作为静态 Pod 运行，这里所需要的只是在引导集群之前将相应的清单文件放置在 /etc/kubernetes/manifests 目录中，在引导过程中，kubelet 将启动这些 pod，这种方式比较优雅

keepalived

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  name: keepalived
  namespace: kube-system
spec:
  containers:
  - image: osixia/keepalived:2.0.20
    name: keepalived
    resources: {}
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
        - NET_BROADCAST
        - NET_RAW
    volumeMounts:
    - mountPath: /usr/local/etc/keepalived/keepalived.conf
      name: config
    - mountPath: /etc/keepalived/check_apiserver.sh
      name: check
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/keepalived/keepalived.conf
    name: config
  - hostPath:
      path: /etc/keepalived/check_apiserver.sh
    name: check
status: {}

apiVersion: v1
kind: Pod
metadata:
  name: haproxy
  namespace: kube-system
spec:
  containers:
  - image: haproxy:2.8
    name: haproxy
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: localhost
        path: /healthz
        port: ${APISERVER_DEST_PORT}
        scheme: HTTPS
    volumeMounts:
    - mountPath: /usr/local/etc/haproxy/haproxy.cfg
      name: haproxyconf
      readOnly: true
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/haproxy/haproxy.cfg
      type: FileOrCreate
    name: haproxyconf
status: {}

${APISERVER_DEST_PORT} 值需要和配置文件一致，配置好后，执行kubeadm init引导集群

服务配置

keepalived

keepalived配置包含服务配置文件和健康检查文件，配置文件放在**/etc/keepalived**目录，以下配置文件适用于2.0.20 and 2.2.4

! /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
}
vrrp_script check_apiserver {
  script "/etc/keepalived/check_apiserver.sh"
  interval 3
  weight -2
  fall 10
  rise 2
}

vrrp_instance VI_1 {
    state ${STATE}
    interface ${INTERFACE}
    virtual_router_id ${ROUTER_ID}
    priority ${PRIORITY}
    authentication {
        auth_type PASS
        auth_pass ${AUTH_PASS}
    }
    virtual_ipaddress {
        ${APISERVER_VIP}
    }
    track_script {
        check_apiserver
    }
}

根据具体情况把上面配置文件变量替换成具体值

• ${STATE} 对于一台主机来说是 MASTER，对于所有其他主机来说是 BACKUP，虚拟 IP 最初将分配给 MASTER。
• ${INTERFACE} VIP 需要绑定的网络接口，例如eth0。
• ${ROUTER_ID} 应该相同，但在同一子网中的所有集群中是唯一的。许多发行版将其值预先配置为 51，需要确认清楚
• ${PRIORITY} MASTER 节点应高于BACKUP节点上的 ${PRIORITY}。因此 101 和 100 分别就足够了。
  对于所有 keepalived 集群主机，
• ${AUTH_PASS} 验证密码，所有配置文件保持一致，例如123456
• ${APISERVER_VIP} 需要使用的 VIP 地址。

keepalived的健康检查脚本放在/etc/keepalived/check_apiserver.sh。

#!/bin/sh
APISERVER_DEST_PORT=6443
errorExit() {
    echo "*** $*" 1>&2
    exit 1
}

curl -sfk --max-time 2 https://localhost:${APISERVER_DEST_PORT}/healthz -o /dev/null || errorExit "Error GET https://localhost:${APISERVER_DEST_PORT}/healthz"

haproxy

配置文件目录/etc/haproxy，适配版本 2.4 and 2.8

# /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    log stdout format raw local0
    daemon

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 1
    timeout http-request    10s
    timeout queue           20s
    timeout connect         5s
    timeout client          35s
    timeout server          35s
    timeout http-keep-alive 10s
    timeout check           10s

#---------------------------------------------------------------------
# apiserver frontend which proxys to the control plane nodes
#---------------------------------------------------------------------
frontend apiserver
    bind *:${APISERVER_DEST_PORT}
    mode tcp
    option tcplog
    default_backend apiserverbackend

#---------------------------------------------------------------------
# round robin balancing for apiserver
#---------------------------------------------------------------------
backend apiserverbackend
    option httpchk

    http-check connect ssl
    http-check send meth GET uri /healthz
    http-check expect status 200

    mode tcp
    balance     roundrobin
    
    server ${HOST1_ID} ${HOST1_ADDRESS}:${APISERVER_SRC_PORT} check verify none
    # [...]

• ${APISERVER_DEST_PORT} haproxy 监听的端口，转发 apiserver 请求
• ${APISERVER_SRC_PORT} API Server 实例使用的端口
• ${HOST1_ID} 第一个负载平衡 API Server 主机名称，随意起，有多个可以再添加多个 server
• ${HOST1_ADDRESS} 第一个负载平衡 API Server 主机的可解析地址（DNS 名称、IP 地址）

kube-vip

kube-vip官方文档