Kubeadm+Containerd部署k8s(v1.28.2)集群（非高可用版）-EW帮帮网

Kubeadm+Containerd部署k8s(v1.28.2)集群（非高可用版）

文章目录

Kubeadm+Containerd部署k8s(v1.28.2)集群（非高可用版）

一.环境准备

1.服务器准备

服务器信息（此处为同一运营商的服务器，并且处于同一网络组中）如本地搭建的服务器则需要保证在同一内网中，且最好关闭防火墙。

角色	主机名称	IP	Kubernetes版本
master	k8s-master1	192.168.0.2	v1.28.2
node1	k8s-node1	192.168.0.3	v1.28.2
node2	k8s-node2	192.168.0.4	v1.28.2

为什么要用Containerd？
1.Docker Engine 没有实现 CRI，而这是容器运行时在 Kubernetes 中工作所需要的。为此，必须安装一个额外的服务 cri-dockerd。 cri-dockerd 是一个基于传统的内置 Docker 引擎支持的项目，它在 1.24 版本从 kubelet 中移除。

2.不直接兼容 CRI：Docker 使用自己的运行时逻辑，与 Kubernetes的 CRI 不完全兼容。

3.性能和复杂性：Docker 包含许多 Kubernetes 不需要的额外功能（如镜像构建、CLI 工具），相比专注的运行时（如 CRI-O 或 Containerd）更重。
4.Containerd 是一个通用容器运行时，是 Docker 的核心组件之一，后来独立成为一个开源项目。它同样支持 Kubernetes，通过 CRI 插件（cri-Containerd）与 Kubernetes 集成.且被被广泛应用于生产环境（如 AWS EKS、Google GKE）

2.环境配置

确保每个节点上的MAC地址和product_uuid的唯一性，因为某些虚拟机的地址可能会重复。

#查看MAC地址命令
ip link
#校验product_uuid
sudo cat /sys/class/dmi/id/product_uuid

如图，eth0网卡下的，link/ether后面的就是Mac地址。

此处输出的就是product_uuid

3.设置主机名

根据机器的规划，分布设置三台主机名为k8s-master1、k8s-node1和k8s-node2。

#修改配置文件方式，
vi /etc/hostname
##文件内容为：k8s-master1或k8s-node1

#命令行方式，临时修改
# 设置 master 主机名
hostnamectl set-hostname k8s-master1 && bash
# 设置 worker 主机名
hostnamectl set-hostname k8s-node1 && bash
hostnamectl set-hostname k8s-node2 && bash

4.修改国内镜像源地址

改为华为的镜像源为例子

备份镜像源文件

sudo cp -a /etc/apt/sources.list /etc/apt/sources.list.bak

修改sources.list文件，将http://archive.ubuntu.com和http://security.ubuntu.com替换成http://repo.huaweicloud.com

sudo sed -i "s@http://.*archive.ubuntu.com@http://repo.huaweicloud.com@g" /etc/apt/sources.list
sudo sed -i "s@http://.*security.ubuntu.com@http://repo.huaweicloud.com@g" /etc/apt/sources.lis

5.配置时间同步

采用ntpdate命令进行时间校对，设置Crontab定时任务进行定时校对。

#安装工具
sudo apt install ntpdate
#配置Crontab定时任务
crontab -e
#添加如下内容： 0 */1 * * * ntpdate time1.aliyun.com

6.配置内核转发及网桥过滤

添加配置文件

cat << EOF | tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

加载模块
```
modprobe overlay
modprobe br_netfilter
```

添加网桥过滤及内核转发配置文件

cat  <<EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

#然后执行
sysctl -p /etc/sysctl.d/k8s.conf

如果上述配置执行之后，执行sysctl net.ipv4.ip_forward输出和下图不一致。则需手动执行sysctl -w net.ipv4.ip_forward=1

安装Ipset及ipvsadm
```
sudo apt -y install ipset ipvsadm
```

配置ipvsadm模块，添加需要添加的模块

cat  << EOF | tee /etc/modules-load.d/ipvs.conf
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
EOF

加载模块脚本

cat << EOF | tee ipvs.sh > /dev/null && chmod +x ipvs.sh
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
EOF

./ipvs.sh

关闭swap分区
```
swapoff -a  #临时关闭
```
编辑/etc/fstab，注释如下类似的内容，重启系统并验证。如果已经关闭那么该命令不会有输出。
```
sudo swapon --show 
```

二.容器运行时Containerd安装（所有节点）

1.apt源安装

#更新系统源
apt update
#安装必须包
apt install -y apt-transport-https ca-certificates curl software-properties-common
#添加 Docker 官方的 GPG 密钥
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
#更新包的索引
sudo apt update
#查看可安装的Containerd版本
apt-cache madison containerd.io

#默认下载最新版
apt-get install containerd.io
#下载指定版本
apt-get install containerd.io=1.7.24-1
#本次安装的是1.7.24-1

为了保证环境稳定，建议锁定containerd的版本

#锁定版本
apt-mark hold containerd.io
#后续升级的解锁命令
apt-mark unhold containerd.io

配置 containerd

mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml
#启动和设置开机自启动
systemctl start containerd && systemctl enable containerd

验证安装

systemctl status containerd

2.二进制安装文件方法

下载地址，获取Containerd的二进制版本。如下图：

通过下面的命令进行解压

 tar -xf cri-containerd-1.7.12-linux-amd64.tar.gz  -C /

执行该命令后，这些文件和目录会被解压到根目录 / 下，并且解压后的文件会按原有的结构放置。

配置 containerd,通过以命令生成Containerd的配置文件

mkdir /etc/containerd
containerd config default > /etc/containerd/config.toml
#启动和设置开机自启动
systemctl start containerd && systemctl enable containerd

3.配置Containerd镜像加速等

修改 /etc/containerd/config.toml

第67行sandbox_image的内容，从原来的registry.k8s.io/pause:3.8修改成registry.aliyuncs.com/google_containers/pause:3.9

第139行 SystemdCgroup的内容，从原来的false改成true。

第162行plugins."io.containerd.grpc.v1.cri".registry下方的config_path,修改为config_path = "/etc/containerd/certs.d"

如图：

创建相应的目录并配置加速文件

#创建目录
mkdir /etc/containerd/certs.d/docker.io -pv
#配置加速文件
cat > /etc/containerd/certs.d/docker.io/hosts.toml << EOF
server = "https://docker.io"
[host."https://x46sxvnb.mirror.aliyuncs.com"]
  capabilities = ["pull", "resolve"]
EOF

#重启containerd
systemctl restart containerd.service

注意：

# 所有节点都操作
cat <<EOF|tee /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
EOF

三.K8S集群部署

1.集群软件的apt源准备

此处使用使用阿里云镜像源：

#添加镜像源
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
#添加密钥
curl -s https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -
#更新源
apt-get update

2.K8S集群软件的安装

查看可以安装的软件列表

apt-cache madison kubeadm

可以看到最新的版本是1.28.2，此处选择安装的版本是1.28.2，可自行选择。

apt-get install -y kubelet=1.28.2-00 kubeadm=1.28.2-00 kubectl=1.28.2-00

node节点上可以不用安装kubectl
apt-get install -y kubelet=1.28.2-00 kubeadm=1.28.2-00

启动kueblet，并设置开机自启动。

systemctl enable --now kubelet

为求稳定，通过以下命令进行锁定定版本。

apt-mark hold kubelet kubeadm kubectl

3.K8S集群初始化

#创建文件夹
mkdir -p /home/k8s/init/
#生成默认配置文件
kubeadm config print init-defaults > /home/k8s/init/kubeadm-init.yaml

修改文件的一些参数：

apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.0.2 #master IP地址
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
  name: k8s-master1 #master节点主机名称
  taints: null
---
apiServer:
  certSANs:
    - 192.168.0.2
    - 101.126.148.74
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {
   }
dns: {
   }
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers #修改镜像源
kind: ClusterConfiguration
kubernetesVersion: 1.28.2 #修版本信息
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.244.0.0/12 #修改Service网段信息
  podSubnet: 10.96.0.0/16 #添加Pod网段信息
scheduler: {
   }

通过以下命令列举出需要拉取的镜像：

kubeadm config images list --config  /home/k8s/init/kubeadm-init.yaml

通过下面命令进行拉取：

kubeadm config images pull --config  /home/k8s/init/kubeadm-init.yaml

然后通过以下命令进行初始化：

kubeadm init --config=/home/k8s/init/kubeadm-init.yaml  --v=6

出现如下信息则证明初始化成功：

Then you can join any number of worker nodes byrunning the following on each as root:
kubeadm join 192.168.0.2:6443 --token abcdef.69123456789abcdef

然后按照上述的提示，进行如下的操作：

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

执行完上述的命令，就可以通过kubectl进行查看k8s集群的状态信息。如下图：

此处需注意，因为一开始没有安装Calico等网络插件，节点状态应该是STATUS： NotReady。

新节点加入到集群中，在节点上通过如下命令，获取加入集群的token值：

kubeadm token create --print-join-command

直接在node节点执行：

kubeadm join 192.168.0.2:6443 --token xqlsab.83s9nr4w0dfmls59 --discovery-token-ca-cert-hash sha256:dc1719de42513ce9f02ad8602932eff9fc0137261553a9d6209873df73b5352b

暂时未写如何添加另一个Master节点，此处涉及高可用配置。

执行后如图：

然后可以通过如下命令来排查节点插件的安装进程：

kubectl get pod -n kube-system -o wide | grep k8s-node1

如果一直未启动成功可以用如下命令来排查：

kubectl describe pod -n kube-system <PodID>
#或者
kubectl logs -f -n kube-system <PodID>

在节点上还可以通过crictl ps -a命令来排查是否有pod启动。同理，在初始化的时候也可以这样排查。

crictl是Containerd的命令行工具。

4.K8S集群网络插件Calico部署

此处选用科学部署方式，拉取的镜像到自己的阿里云仓库。yaml文件内容如下：

---
# Source: calico/templates/calico-kube-controllers.yaml
# This manifest creates a Pod Disruption Budget for Controller to allow K8s Cluster Autoscaler to evict

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: calico-kube-controllers
  namespace: kube-system
  labels:
    k8s-app: calico-kube-controllers
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      k8s-app: calico-kube-controllers
---
# Source: calico/templates/calico-kube-controllers.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: calico-kube-controllers
  namespace: kube-system
---
# Source: calico/templates/calico-node.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: calico-node
  namespace: kube-system
---
# Source: calico/templates/calico-node.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: calico-cni-plugin
  namespace: kube-system
---
# Source: calico/templates/calico-config.yaml
# This ConfigMap is used to configure a self-hosted Calico installation.
kind: ConfigMap
apiVersion: v1
metadata:
  name: calico-config
  namespace: kube-system
data:
  # Typha is disabled.
  typha_service_name: "none"
  # Configure the backend to use.
  calico_backend: "bird"

  # Configure the MTU to use for workload interfaces and tunnels.
  # By default, MTU is auto-detected, and explicitly setting this field should not be required.
  # You can override auto-detection by providing a non-zero value.
  veth_mtu: "0"

  # The CNI network configuration to install on each node. The special
  # values in this config will be automatically populated.
  cni_network_config: |-
    {
   
      "name": "k8s-pod-network",
      "cniVersion": "0.3.1",
      "plugins": [
        {
   
          "type": "calico",
          "log_level": "info",
          "log_file_path": "/var/log/calico/cni/cni.log",
          "datastore_type": "kubernetes",
          "nodename": "__KUBERNETES_NODE_NAME__",
          "mtu": __CNI_MTU__,
          "ipam": {
   
              "type": "calico-ipam"
          },
          "policy": {
   
              "type": "k8s"
          },
          "kubernetes": {
   
              "kubeconfig": "__KUBECONFIG_FILEPATH__"
          }
        },
        {
   
          "type": "portmap",
          "snat": true,
          "capabilities": {
   "portMappings": true}
        },
        {
   
          "type": "bandwidth",
          "capabilities": {
   "bandwidth": true}
        }
      ]
    }
---
# Source: calico/templates/kdd-crds.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: bgpconfigurations.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: BGPConfiguration
    listKind: BGPConfigurationList
    plural: bgpconfigurations
    singular: bgpconfiguration
  preserveUnknownFields: false
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        description: BGPConfiguration contains the configuration for any BGP routing.
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: BGPConfigurationSpec contains the values of the BGP configuration.
            properties:
              asNumber:
                description: 'ASNumber is the default AS number used by a node. [Default:
                  64512]'
                format: int32
                type: integer
              bindMode:
                description: BindMode indicates whether to listen for BGP connections
                  on all addresses (None) or only on the node's canonical IP address
                  Node.Spec.BGP.IPvXAddress (NodeIP). Default behaviour is to listen
                  for BGP connections on all addresses.
                type: string
              communities:
                description: Communities is a list of BGP community values and their
                  arbitrary names for tagging routes.
                items:
                  description: Community contains standard or large community value
                    and its name.
                  properties:
                    name:
                      description: Name given to community value.
                      type: string
                    value:
                      description: Value must be of format `aa:nn` or `aa:nn:mm`.
                        For standard community use `aa:nn` format, where `aa` and
                        `nn` are 16 bit number. For large community use `aa:nn:mm`
                        format, where `aa`, `nn` and `mm` are 32 bit number. Where,
                        `aa` is an AS Number, `nn` and `mm` are per-AS identifier.
                      pattern: ^(\d+):(\d+)$|^(\d+):(\d+):(\d+)$
                      type: string
                  type: object
                type: array
              ignoredInterfaces:
                description: IgnoredInterfaces indicates the network interfaces that
                  needs to be excluded when reading device routes.
                items:
                  type: string
                type: array
              listenPort:
                description: ListenPort is the port where BGP protocol should listen.
                  Defaults to 179
                maximum: 65535
                minimum: 1
                type: integer
              logSeverityScreen:
                description: 'LogSeverityScreen is the log severity above which logs
                  are sent to the stdout. [Default: INFO]'
                type: string
              nodeMeshMaxRestartTime:
                description: Time to allow for software restart for node-to-mesh peerings.  When
                  specified, this is configured as the graceful restart timeout.  When
                  not specified, the BIRD default of 120s is used. This field can
                  only be set on the default BGPConfiguration instance and requires
                  that NodeMesh is enabled
                type: string
              nodeMeshPassword:
                description: Optional BGP password for full node-to-mesh peerings.
                  This field can only be set on the default BGPConfiguration instance
                  and requires that NodeMesh is enabled
                properties:
                  secretKeyRef:
                    description: Selects a key of a secret in the node pod's namespace.
                    properties:
                      key:
                        description: The key of the secret to select from.  Must be
                          a valid secret key.
                        type: string
                      name:
                        description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
                          TODO: Add other useful fields. apiVersion, kind, uid?'
                        type: string
                      optional:
                        description: Specify whether the Secret or its key must be
                          defined
                        type: boolean
                    required:
                    - key
                    type: object
                type: object
              nodeToNodeMeshEnabled:
                description: 'NodeToNodeMeshEnabled sets whether full node to node
                  BGP mesh is enabled. [Default: true]'
                type: boolean
              prefixAdvertisements:
                description: PrefixAdvertisements contains per-prefix advertisement
                  configuration.
                items:
                  description: PrefixAdvertisement configures advertisement properties
                    for the specified CIDR.
                  properties:
                    cidr:
                      description: CIDR for which properties should be advertised.
                      type: string
                    communities:
                      description: Communities can be list of either community names
                        already defined in `Specs.Communities` or community value
                        of format `aa:nn` or `aa:nn:mm`. For standard community use
                        `aa:nn` format, where `aa` and `nn` are 16 bit number. For
                        large community use `aa:nn:mm` format, where `aa`, `nn` and
                        `mm` are 32 bit number. Where,`aa` is an AS Number, `nn` and
                        `mm` are per-AS identifier.
                      items:
                        type: string
                      type: array
                  type: object
                type: array
              serviceClusterIPs:
                description: ServiceClusterIPs are the CIDR blocks from which service
                  cluster IPs are allocated. If specified, Calico will advertise these
                  blocks, as well as any cluster IPs within them.
                items:
                  description: ServiceClusterIPBlock represents a single allowed ClusterIP
                    CIDR block.
                  properties:
                    cidr:
                      type: string
                  type: object
                type: array
              serviceExternalIPs:
                description: ServiceExternalIPs are the CIDR blocks for Kubernetes
                  Service External IPs. Kubernetes Service ExternalIPs will only be
                  advertised if they are within one of these blocks.
                items:
                  description: ServiceExternalIPBlock represents a single allowed
                    External IP CIDR block.
                  properties:
                    cidr:
                      type: string
                  type: object
                type: array
              serviceLoadBalancerIPs:
                description: ServiceLoadBalancerIPs are the CIDR blocks for Kubernetes
                  Service LoadBalancer IPs. Kubernetes Service status.LoadBalancer.Ingress
                  IPs will only be advertised if they are within one of these blocks.
                items:
                  description: ServiceLoadBalancerIPBlock represents a single allowed
                    LoadBalancer IP CIDR block.
                  properties:
                    cidr:
                      type: string
                  type: object
                type: array
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []
---
# Source: calico/templates/kdd-crds.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: (devel)
  creationTimestamp: null
  name: bgpfilters.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: BGPFilter
    listKind: BGPFilterList
    plural: bgpfilters
    singular: bgpfilter
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: BGPFilterSpec contains the IPv4 and IPv6 filter rules of
              the BGP Filter.
            properties:
              exportV4:
                description: The ordered set of IPv4 BGPFilter rules acting on exporting
                  routes to a peer.
                items:
                  description: BGPFilterRuleV4 defines a BGP filter rule consisting
                    a single IPv4 CIDR block and a filter action for this CIDR.
                  properties:
                    action:
                      type: string
                    cidr:
                      type: string
                    interface:
                      type: string
                    matchOperator:
                      type: string
                    source:
                      type: string
                  required:
                  - action
                  type: object
                type: array
              exportV6:
                description: The ordered set of IPv6 BGPFilter rules acting on exporting
                  routes to a peer.
                items:
                  description: BGPFilterRuleV6 defines a BGP filter rule consisting
                    a single IPv6 CIDR block and a filter action for this CIDR.
                  properties:
                    action:
                      type: string
                    cidr:
                      type: string
                    interface:
                      type: string
                    matchOperator:
                      type: string
                    source:
                      type: string
                  required:
                  - action
                  type: object
                type: array
              importV4:
                description: The ordered set of IPv4 BGPFilter rules acting on importing
                  routes from a peer.
                items:
                  description: BGPFilterRuleV4 defines a BGP filter rule consisting
                    a single IPv4 CIDR block and a filter action for this CIDR.
                  properties:
                    action:
                      type: string
                    cidr:
                      type: string
                    interface:
                      type: string
                    matchOperator:
                      type: string
                    source:
                      type: string
                  required:
                  - action
                  type: object
                type: array
              importV6:
                description: The ordered set of IPv6 BGPFilter rules acting on importing
                  routes from a peer.
                items:
                  description: BGPFilterRuleV6 defines a BGP filter rule consisting
                    a single IPv6 CIDR block and a filter action for this CIDR.
                  properties:
                    action:
                      type: string
                    cidr:
                      type: string
                    interface:
                      type: string
                    matchOperator:
                      type: string
                    source:
                      type: string
                  required:
                  - action
                  type: object
                type: array
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []
---
# Source: calico/templates/kdd-crds.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: bgppeers.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: BGPPeer
    listKind: BGPPeerList
    plural: bgppeers
    singular: bgppeer
  preserveUnknownFields: false
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: BGPPeerSpec contains the specification for a BGPPeer resource.
            properties:
              asNumber:
                description: The AS Number of the peer.
                format: int32
                type: integer
              filters:
                description: The ordered set of BGPFilters applied on this BGP peer.
                items:
                  type: string
                type: array
              keepOriginalNextHop:
                description: Option to keep the original nexthop field when routes
                  are sent to a BGP Peer. Setting "true" configures the selected BGP
                  Peers node to use the "next hop keep;" instead of "next hop self;"(default)
                  in the specific branch of the Node on "bird.cfg".
                type: boolean
              maxRestartTime:
                description: Time to allow for software restart.  When specified,
                  this is configured as the graceful restart timeout.  When not specified,
                  the BIRD default of 120s is used.
                type: string
              node:
                description: The node name identifying the Calico node instance that
                  is targeted by this peer. If this is not set, and no nodeSelector
                  is specified, then this BGP peer selects all nodes in the cluster.
                type: string
              nodeSelector:
                description: Selector for the nodes that should have this peering.  When
                  this is set, the Node field must be empty.
                type: string
              numAllowedLocalASNumbers:
                description: Maximum number of local AS numbers that are allowed in
                  the AS path for received routes. This removes BGP loop prevention
                  and should only be used if absolutely necesssary.
                format: int32
                type: integer
              password:
                description: Optional BGP password for the peerings generated by this
                  BGPPeer resource.
                properties:
                  secretKeyRef:
                    description: Selects a key of a secret in the node pod's namespace.
                    properties:
                      key:
                        description: The key of the secret to select from.  Must be
                          a valid secret key.
                        type: string
                      name:
                        description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
                          TODO: Add other useful fields. apiVersion, kind, uid?'
                        type: string
                      optional:
                        description: Specify whether the Secret or its key must be
                          defined
                        type: boolean
                    required:
                    - key
                    type: object
                type: object
              peerIP:
                description: The IP address of the peer followed by an optional port
                  number to peer with. If port number is given, format should be `[<IPv6>]:port`
                  or `<IPv4>:<port>` for IPv4. If optional port number is not set,
                  and this peer IP and ASNumber belongs to a calico/node with ListenPort
                  set in BGPConfiguration, then we use that port to peer.
                type: string
              peerSelector:
                description: Selector for the remote nodes to peer with.  When this
                  is set, the PeerIP and ASNumber fields must be empty.  For each
                  peering between the local node and selected remote nodes, we configure
                  an IPv4 peering if both ends have NodeBGPSpec.IPv4Address specified,
                  and an IPv6 peering if both ends have NodeBGPSpec.IPv6Address specified.  The
                  remote AS number comes from the remote node's NodeBGPSpec.ASNumber,
                  or the global default if that is not set.
                type: string
              reachableBy:
                description: Add an exact, i.e. /32, static route toward peer IP in
                  order to prevent route flapping. ReachableBy contains the address
                  of the gateway which peer can be reached by.
                type: string
              sourceAddress:
                description: Specifies whether and how to configure a source address
                  for the peerings generated by this BGPPeer resource.  Default value
                  "UseNodeIP" means to configure the node IP as the source address.  "None"
                  means not to configure a source address.
                type: string
              ttlSecurity:
                description: TTLSecurity enables the generalized TTL security mechanism
                  (GTSM) which protects against spoofed packets by ignoring received
                  packets with a smaller than expected TTL value. The provided value
                  is the number of hops (edges) between the peers.
                type: integer
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []
---
# Source: calico/templates/kdd-crds.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: blockaffinities.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: BlockAffinity
    listKind: BlockAffinityList
    plural: blockaffinities
    singular: blockaffinity
  preserveUnknownFields: false
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: BlockAffinitySpec contains the specification for a BlockAffinity
              resource.
            properties:
              cidr:
                type: string
              deleted:
                description: Deleted indicates that this block affinity is being deleted.
                  This field is a string for compatibility with older releases that
                  mistakenly treat this field as a string.
                type: string
              node:
                type: string
              state:
                type: string
            required:
            - cidr
            - deleted
            - node
            - state
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []
---
# Source: calico/templates/kdd-crds.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: (devel)
  creationTimestamp: null
  name: caliconodestatuses.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: CalicoNodeStatus
    listKind: CalicoNodeStatusList
    plural: caliconodestatuses
    singular: caliconodestatus
  preserveUnknownFields: false
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: CalicoNodeStatusSpec contains the specification for a CalicoNodeStatus
              resource.
            properties:
              classes:
                description: Classes declares the types of information to monitor
                  for this calico/node, and allows for selective status reporting
                  about certain subsets of information.
                items:
                  type: string
                type: array
              node:
                description: The node name identifies the Calico node instance for
                  node status.
                type: string
              updatePeriodSeconds:
                description: UpdatePeriodSeconds is the period at which CalicoNodeStatus
                  should be updated. Set to 0 to disable CalicoNodeStatus refresh.
                  Maximum update period is one day.
                format: int32
                type: integer
            type: object
          status:
            description: CalicoNodeStatusStatus defines the observed state of CalicoNodeStatus.
              No validation needed for status since it is updated by Calico.
            properties:
              agent:
                description: Agent holds agent status on the node.
                properties:
                  birdV4:
                    description: BIRDV4 represents the latest observed status of bird4.
                    properties:
                      lastBootTime:
                        description: LastBootTime holds the value of lastBootTime
                          from bird.ctl output.
                        type: string
                      lastReconfigurationTime:
                        description: LastReconfigurationTime holds the value of lastReconfigTime
                          from bird.ctl output.
                        type: string
                      routerID:
                        description: Router ID used by bird.
                        type: string
                      state:
                        description: The state of the BGP Daemon.
                        type: string
                      version:
                        description: Version of the BGP daemon
                        type: string
                    type: object
                  birdV6:
                    description: BIRDV6 represents the latest observed status of bird6.
                    properties:
                      lastBootTime:
                        description: LastBootTime holds the value of lastBootTime
                          from bird.ctl output.
                        type: string
                      lastReconfigurationTime:
                        description: LastReconfigurationTime holds the value of lastReconfigTime
                          from bird.ctl output.
                        type: string
                      routerID:
                        description: Router ID used by bird.
                        type: string
                      state:
                        description: The state of the BGP Daemon.
                        type: string
                      version:
                        description: Version of the BGP daemon
                        type: string
                    type: object
                type: object
              bgp:
                description: BGP holds node BGP status.
                properties:
                  numberEstablishedV4:
                    description: The total number of IPv4 established bgp sessions.
                    type: integer
                  numberEstablishedV6:
                    description: The total number of IPv6 established bgp sessions.
                    type: integer
                  numberNotEstablishedV4:
                    description: The total number of IPv4 non-established bgp sessions.
                    type: integer
                  numberNotEstablishedV6:
                    description: The total number of IPv6 non-established bgp sessions.
                    type: integer
                  peersV4:
                    description: PeersV4 represents IPv4 BGP peers status on the node.
                    items:
                      description: CalicoNodePeer contains the status of BGP peers
                        on the node.
                      properties:
                        peerIP:
                          description: IP address of the peer whose condition we are
                            reporting.
                          type: string
                        since:
                          description: Since the state or reason last changed.
                          type: string
                        state:
                          description: State is the BGP session state.
                          type: string
                        type:
                          description: Type indicates whether this peer is configured
                            via the node-to-node mesh, or via en explicit global or
                            per-node BGPPeer object.
                          type: string
                      type: object
                    type: array
                  peersV6:
                    description: PeersV6 represents IPv6 BGP peers status on the node.
                    items:
                      description: CalicoNodePeer contains the status of BGP peers
                        on the node.
                      properties:
                        peerIP:
                          description: IP address of the peer whose condition we are
                            reporting.
                          type: string
                        since:
                          description: Since the state or reason last changed.
                          type: string
                        state:
                          description: State is the BGP session state.
                          type: string
                        type:
                          description: Type indicates whether this peer is configured
                            via the node-to-node mesh, or via en explicit global or
                            per-node BGPPeer object.
                          type: string
                      type: object
                    type: array
                required:
                - numberEstablishedV4
                - numberEstablishedV6
                - numberNotEstablishedV4
                - numberNotEstablishedV6
                type: object
              lastUpdated:
                description: LastUpdated is a timestamp representing the server time
                  when CalicoNodeStatus object last updated. It is represented in
                  RFC3339 form and is in UTC.
                format: date-time
                nullable: true
                type: string
              routes:
                description: Routes reports routes known to the Calico BGP daemon
                  on the node.
                properties:
                  routesV4:
                    description: RoutesV4 represents IPv4 routes on the node.
                    items:
                      description: CalicoNodeRoute contains the status of BGP routes
                        on the node.
                      properties:
                        destination:

Kubeadm+Containerd部署k8s(v1.28.2)集群（非高可用版）