Calico-BGP FullMesh模式与RR模式 Day04-EW帮帮网

1. BGP协议简单介绍

Configure BGP peering | Calico Documentation

1.1 什么是BGP

边界网关协议（BGP）是一种用于在网络中的路由器之间交换路由信息的标准协议。每台运行 BGP 的路由器都有一个或多个 BGP 对等体 —— 也就是与之通过 BGP 进行通信的其他路由器。

你可以将 Calico 网络视为在你的每个节点上都提供了一个虚拟路由器。你可以对 Calico 节点进行配置，使其相互之间、与路由反射器或者与架顶（ToR）路由器建立对等连接。

1.2 Calico中常用的BGP拓扑

1.2.1 Full-mesh（全网状）

当启用边界网关协议（BGP）时，Calico 的默认行为是创建一个内部边界网关协议（iBGP）连接的全网状结构，即每个节点之间都相互建立对等连接。每个Calico节点之间使用179端口进行通信，并通过 BGP 协议与其他节点或外部路由器交换路由信息。

这使得 Calico 能够在任何二层网络（无论是公有云还是私有云网络）上运行，或者如果配置了 IPIP，它能够作为覆盖网络在任何不阻断 IPIP 流量的网络上运行。

Calico 在 VXLAN 覆盖网络中并不使用 BGP 协议。

Full-mesh对于大约 100 个节点或更少节点的中小型部署来说效果很好，但在规模显著扩大时，全网状结构的效率就会变低，建议使用route reflectors。

该拓扑部署后的默认模式为：node-to-node mesh（节点到节点全网状）。

1.2.2 Route reflectors

为构建大型内部边界网关协议（iBGP）集群，可以使用 BGP Route reflectors（路由反射）来减少每个节点上所使用的 BGP 对等连接数量。

在这种模式下，部分节点充当 Route reflectors，并被配置为在它们之间建立全网状连接。然后，其他节点被配置为与这些 Route reflectors中的一部分（通常为两个以实现冗余）建立对等连接，与全网状连接（Full-mesh）相比，这样可减少 BGP 对等连接的总数。

1.2.3 Top of Rack

在本地部署环境中，你可以将Calico配置为直接与物理网络基础设施建立对等连接。通常，这需要禁用Calico默认的全网状连接模式，转而让Calico与三层架顶（ToR）路由器建立对等连接。

构建本地BGP网络有多种方式。如何配置BGP取决于你自己——Calico在内部BGP（iBGP）和外部BGP（eBGP）配置下都能良好运行，并且在网络设计中，你可以像对待其他路由器一样有效地使用Calico。

2. kind部署Full-mesh环境

2.1 清理环境

2.1.1 移除containerlab生成的网卡

root@superadmin-virtual-machine:~/wcni-kind/LabasCode/calico/04-calico-vxlan-crosssubnet# ls
1  1-setup-env.sh  2-setup-clab.sh  3-datapath  calico.yaml  clab-calico-vxlan-crosssubnet  clab.yaml  cni.yaml  startup-conf
root@superadmin-virtual-machine:~/wcni-kind/LabasCode/calico/04-calico-vxlan-crosssubnet#  containerlab destroy --topo clab.yaml
INFO[0000] Parsing & checking topology file: clab.yaml
INFO[0000] Parsing & checking topology file: clab.yaml
INFO[0000] Destroying lab: calico-vxlan-crosssubnet
INFO[0000] Removed container: clab-calico-vxlan-crosssubnet-server2
INFO[0000] Removed container: clab-calico-vxlan-crosssubnet-server4
INFO[0000] Removed container: clab-calico-vxlan-crosssubnet-server3
INFO[0000] Removed container: clab-calico-vxlan-crosssubnet-server1
INFO[0000] Removed container: clab-calico-vxlan-crosssubnet-gw0
INFO[0000] Removing containerlab host entries from /etc/hosts file
INFO[0000] Removing ssh config for containerlab nodes

2.1.2 删除kind创建的k8s集群

root@superadmin-virtual-machine:~# kind get clusters
calico-vxlan-crosssubnet
root@superadmin-virtual-machine:~# kind delete clusters calico-vxlan-crosssubnet
Deleted nodes: ["calico-vxlan-crosssubnet-worker3" "calico-vxlan-crosssubnet-worker" "calico-vxlan-crosssubnet-control-plane" "calico-vxlan-crosssubnet-worker2"]
Deleted clusters: ["calico-vxlan-crosssubnet"]

2.2 修改相关配置

root@superadmin-virtual-machine:~/wcni-kind/LabasCode/calico/05-calico-fullmesh# grep image 1-setup-env.sh
#cat <<EOF | kind create cluster --name=calico-bgp-fullmesh --image=kindest/node:v1.27.3 --config=-
cat <<EOF | kind create cluster --name=calico-bgp-fullmesh --image=registry.cn-beijing.aliyuncs.com/sanhua-k8s/kindest_node:v1.27.3 --config=-


oot@superadmin-virtual-machine:~/wcni-kind/LabasCode/calico/05-calico-fullmesh# grep 'image:' calico.yaml
          #image: 192.168.2.100:5000/calico/cni:v3.23.2
          image: registry.cn-beijing.aliyuncs.com/sanhua-k8s/calico_cni:v3.23.2
          #image: 192.168.2.100:5000/calico/cni:v3.23.2
          image: registry.cn-beijing.aliyuncs.com/sanhua-k8s/calico_cni:v3.23.2
          #image: 192.168.2.100:5000/calico/node:v3.23.2
          image: registry.cn-beijing.aliyuncs.com/sanhua-k8s/calico_node:v3.23.2
          #image: 192.168.2.100:5000/calico/kube-controllers:v3.23.2
          image: registry.cn-beijing.aliyuncs.com/sanhua-k8s/calico_kube-controllers:v3.23.2


root@superadmin-virtual-machine:~/wcni-kind/LabasCode/calico/05-calico-fullmesh# grep 'image:' cni.yaml
      #- image: 192.168.2.100:5000/nettool
      - image: registry.cn-beijing.aliyuncs.com/sanhua-k8s/nettool

2.3 Full-mesh模式的开启方式

把ipip和vxlan关了，就只能通过bgp路由了。

2.4 部署k8s集群

2.4.1 部署集群

root@superadmin-virtual-machine:~/wcni-kind/LabasCode/calico/05-calico-fullmesh# ./1-setup-env.sh

2.4.2 部署测试pod

root@superadmin-virtual-machine:~/wcni-kind/LabasCode/calico/05-calico-fullmesh# cat cni.yaml
apiVersion: apps/v1
kind: DaemonSet
#kind: Deployment
metadata:
  labels:
    app: wluo
  name: wluo
spec:
  #replicas: 2
  selector:
    matchLabels:
      app: wluo
  template:
    metadata:
      labels:
        app: wluo
    spec:
      containers:
      #- image: 192.168.2.100:5000/nettool
      - image: registry.cn-beijing.aliyuncs.com/sanhua-k8s/nettool
        name: nettoolbox
        env:
          - name: NETTOOL_NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
        securityContext:
          privileged: true
---
apiVersion: v1
kind: Service
metadata:
  name: wluo
spec:
  type: NodePort
  selector:
    app: wluo
  ports:
  - name: wluo
    port: 80
    targetPort: 80
    nodePort: 32000

root@superadmin-virtual-machine:~/wcni-kind/LabasCode/calico/05-calico-fullmesh# kubectl apply -f cni.yaml
daemonset.apps/wluo created
service/wluo created

2.4.3 检查当前是否为Full-mesh

需要安装calicoctl命令

内容含义：

Calico process is running.：说明calico进程当前是运行的。
PEER ADDRESS：对等体地址。
○ 表示当前 Calico 节点与之建立 BGP 连接的对等节点的 IP 地址。表格里面是 172.18.0.3、172.18.0.2.意味着当前节点和 IP 为 172.18.0.3、172.18.0.2 的节点建立了 BGP 连接。
PEER TYPE：对等体类型。
○ node-to-node mesh（默认模式），表示这是节点到节点的全网状连接模式，在这种模式下，每个 Calico 节点都会与其他所有节点建立 BGP 连接，以交换路由信息。
○
STATE：连接状态。
○ UP表示连接正常。即当前节点和对等节点之间已经成功建立了 BGP 会话，可以正常交换路由信息。
SINCE：建立连接的时间。
INFO：连接详细信息。
○ Establelished表示BGP连接建立成功并处于稳定状态。

然后这种相关的信息，还可以去calico-node中查询bgp相关的路由信息：

2.5 抓包（同子网）

这里用上图的两个pod进行测试，主节点pod向worker节点pod发起icmp请求。

抓包我是在主节点pod中、主节点宿主机pod对应的cali网卡、主节点eth0等抓的icmp包。

2.5.1 终端一：主节点pod报文

2.5.2 终端二：主节点cali接口报文

2.5.3 终端三：主节点eth0接口报文

2.6 node-to-node mesh总结

总的来说，node to node mesh通信过程，报文没有什么太特殊的地方。

工作模式就是同子网的节点与节点之间，通过179端口来互相交换路由信息。
优点：

网络拓扑简单。

缺点：

过多的节点会增加cpu和内存的消耗，同时也会增加网络流量，降低网络效率。

适用场景：

Full-mesh对于大约 100 个节点或更少节点的中小型部署来说效果很好，但在规模显著扩大时，全网状结构的效率就会变低，建议使用route reflectors。

3. kind部署RR（路由反射器）环境

这个实验在实际工作中，还涉及到路由交换的BGP配置，我选择放弃。

本次实验拓扑

左下角10.1.5.0/24子网，出口网关为10.1.10.1/24。并且AS65005这个设备和AS500、AS800组成了一个EBGP的网络拓扑。

右下角10.1.8.0/24子网，出口网关为10.1.8.1/24。并且AS65008这个设备和AS500、AS800组成了一个EBGP的网络拓扑。

当同子网的节点间通信时，直接走网关就过去了。

如果是跨子网通信，那就要先到网关，然后有两条路可选，AS500和AS800，然后通过目标网关到达目标节点。

3.1 清理环境

3.1.1 删除k8s集群

root@superadmin-virtual-machine:~# kind get clusters
calico-bgp-fullmesh
root@superadmin-virtual-machine:~# kind delete clusters calico-bgp-fullmesh
Deleted nodes: ["calico-bgp-fullmesh-control-plane" "calico-bgp-fullmesh-worker2" "calico-bgp-fullmesh-worker"]
Deleted clusters: ["calico-bgp-fullmesh"]

3.1.2 删除网卡

root@superadmin-virtual-machine:~/wcni-kind/LabasCode/calico/06-calico-bgp-rr# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether fe:fc:fe:f5:dd:98 brd ff:ff:ff:ff:ff:ff
    altname enp0s18
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
    link/ether 02:42:61:d8:64:66 brd ff:ff:ff:ff:ff:ff
4: br-003fe9e9a06f: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
    link/ether 02:42:94:02:02:02 brd ff:ff:ff:ff:ff:ff
11: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
94: br-pool0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether aa:82:88:95:5c:cd brd ff:ff:ff:ff:ff:ff
95: br-pool1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 26:48:c3:7e:20:78 brd ff:ff:ff:ff:ff:ff


root@superadmin-virtual-machine:~/wcni-kind/LabasCode/calico/06-calico-bgp-rr# ip link show br-pool0 && ip link delete br-pool0
root@superadmin-virtual-machine:~/wcni-kind/LabasCode/calico/06-calico-bgp-rr# ip link show br-pool1 && ip link delete br-pool1

3.2 如何启动RR模式

3.2.1 配置文件层面

上图，如果还没安装calico，可以通过添加参数（变量）的方式，来关闭ipip模式和vxlan模式，如果已经安装了，可以编辑ippool资源的yaml。

3.2.2 node节点层面

启动rr模式，除了调整yaml，还需要配置node节点，才能使用BGP RR模式。

3.3 调整配置文件

root@superadmin-virtual-machine:~/wcni-kind/LabasCode/calico/06-calico-bgp-rr# grep image  1-setup-env.sh
#cat <<EOF | kind create cluster --name=calico-bgp-rr --image=kindest/node:v1.27.3 --config=-
cat <<EOF | kind create cluster --name=calico-bgp-rr --image=registry.cn-beijing.aliyuncs.com/sanhua-k8s/kindest_node:v1.27.3 --config=-


root@superadmin-virtual-machine:~/wcni-kind/LabasCode/calico/06-calico-bgp-rr# grep image  2-setup-clab.sh
      image: registry.cn-beijing.aliyuncs.com/sanhua-k8s/vyos:1.4.9
      image: registry.cn-beijing.aliyuncs.com/sanhua-k8s/vyos:1.4.9
      image: registry.cn-beijing.aliyuncs.com/sanhua-k8s/vyos:1.4.9
      image: registry.cn-beijing.aliyuncs.com/sanhua-k8s/vyos:1.4.9
      image: registry.cn-beijing.aliyuncs.com/sanhua-k8s/nettool
      image: registry.cn-beijing.aliyuncs.com/sanhua-k8s/nettool
      image: registry.cn-beijing.aliyuncs.com/sanhua-k8s/nettool
      image: registry.cn-beijing.aliyuncs.com/sanhua-k8s/nettool


root@superadmin-virtual-machine:~/wcni-kind/LabasCode/calico/06-calico-bgp-rr# grep image  calico.yaml
          image: registry.cn-beijing.aliyuncs.com/sanhua-k8s/calico_cni:v3.23.2
          image: registry.cn-beijing.aliyuncs.com/sanhua-k8s/calico_cni:v3.23.2
          image: registry.cn-beijing.aliyuncs.com/sanhua-k8s/calico_node:v3.23.2
          image: registry.cn-beijing.aliyuncs.com/sanhua-k8s/calico_kube-controllers:v3.23.2


root@superadmin-virtual-machine:~/wcni-kind/LabasCode/calico/06-calico-bgp-rr# grep image clab.yaml
      image: registry.cn-beijing.aliyuncs.com/sanhua-k8s/vyos:1.4.9
      image: registry.cn-beijing.aliyuncs.com/sanhua-k8s/vyos:1.4.9
      image: registry.cn-beijing.aliyuncs.com/sanhua-k8s/vyos:1.4.9
      image: registry.cn-beijing.aliyuncs.com/sanhua-k8s/vyos:1.4.9
      image: registry.cn-beijing.aliyuncs.com/sanhua-k8s/nettool
      image: registry.cn-beijing.aliyuncs.com/sanhua-k8s/nettool
      image: registry.cn-beijing.aliyuncs.com/sanhua-k8s/nettool
      image: registry.cn-beijing.aliyuncs.com/sanhua-k8s/nettool


root@superadmin-virtual-machine:~/wcni-kind/LabasCode/calico/06-calico-bgp-rr# grep image cni.yaml
      - image: registry.cn-beijing.aliyuncs.com/sanhua-k8s/nettool

3.4 部署集群

3.4.1 基础环境部署

root@superadmin-virtual-machine:~/wcni-kind/LabasCode/calico/06-calico-bgp-rr# ./1-setup-env.sh

root@superadmin-virtual-machine:~/wcni-kind/LabasCode/calico/06-calico-bgp-rr# ./2-setup-clab.sh

3.4.2 配置rr模式

root@superadmin-virtual-machine:~/wcni-kind/LabasCode/calico/06-calico-bgp-rr# cat 3-prep-calico-bgp.sh
#!/bin/bash
set -v
# 1. Install CNI[Calico v3.23.2]
kubectl apply -f ./calico.yaml

kubectl wait --timeout=100s --for=condition=Ready=true pods --all -A

# 1.2. disable bgp fullmesh
cat <<EOF | calicoctl  --allow-version-mismatch apply -f -
apiVersion: projectcalico.org/v3
items:
- apiVersion: projectcalico.org/v3
  kind: BGPConfiguration
  metadata:
    name: default
  spec:
    logSeverityScreen: Info
    nodeToNodeMeshEnabled: false
kind: BGPConfigurationList
metadata:
EOF

# 1.3. add() bgp configuration for the nodes
cat <<EOF | calicoctl  --allow-version-mismatch apply -f -
apiVersion: projectcalico.org/v3
kind: Node
metadata:
  annotations:
    projectcalico.org/kube-labels: '{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"calico-bgp-rr-control-plane","kubernetes.io/os":"linux","node-role.kubernetes.io/control-plane":"","node-role.kubernetes.io/master":"","node.kubernetes.io/exclude-from-external-load-balancers":"","rack":"rack0"}'
  labels:
    beta.kubernetes.io/arch: amd64
    beta.kubernetes.io/os: linux
    kubernetes.io/arch: amd64
    kubernetes.io/hostname: calico-bgp-rr-control-plane
    kubernetes.io/os: linux
    node-role.kubernetes.io/control-plane: ""
    node-role.kubernetes.io/master: ""
    node.kubernetes.io/exclude-from-external-load-balancers: ""
    rack: rack0
  name: calico-bgp-rr-control-plane
spec:
  addresses:
  - address: 10.1.5.10
    type: InternalIP
  bgp:
    asNumber: 65005
    ipv4Address: 10.1.5.10/24
  orchRefs:
  - nodeName: calico-bgp-rr-control-plane
    orchestrator: k8s
status:
  podCIDRs:
  - 10.244.0.0/24
EOF

cat <<EOF | calicoctl  --allow-version-mismatch apply -f -
apiVersion: projectcalico.org/v3
kind: Node
metadata:
  annotations:
    projectcalico.org/kube-labels: '{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"calico-bgp-rr-worker","kubernetes.io/os":"linux","rack":"rack0"}'
  creationTimestamp: "2022-12-05T08:40:29Z"
  labels:
    beta.kubernetes.io/arch: amd64
    beta.kubernetes.io/os: linux
    kubernetes.io/arch: amd64
    kubernetes.io/hostname: calico-bgp-rr-worker
    kubernetes.io/os: linux
    rack: rack0
  name: calico-bgp-rr-worker
spec:
  addresses:
  - address: 10.1.5.11
    type: InternalIP
  bgp:
    asNumber: 65005
    ipv4Address: 10.1.5.11/24
  orchRefs:
  - nodeName: calico-bgp-rr-worker
    orchestrator: k8s
status:
  podCIDRs:
  - 10.244.1.0/24
EOF


cat <<EOF | calicoctl  --allow-version-mismatch apply -f -
apiVersion: projectcalico.org/v3
kind: Node
metadata:
  annotations:
    projectcalico.org/kube-labels: '{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"calico-bgp-rr-worker2","kubernetes.io/os":"linux","rack":"rack1"}'
  creationTimestamp: "2022-12-05T08:40:29Z"
  labels:
    beta.kubernetes.io/arch: amd64
    beta.kubernetes.io/os: linux
    kubernetes.io/arch: amd64
    kubernetes.io/hostname: calico-bgp-rr-worker2
    kubernetes.io/os: linux
    rack: rack1
  name: calico-bgp-rr-worker2
spec:
  addresses:
  - address: 10.1.8.10
    type: InternalIP
  bgp:
    asNumber: 65008
    ipv4Address: 10.1.8.10/24
  orchRefs:
  - nodeName: calico-bgp-rr-worker2
    orchestrator: k8s
status:
  podCIDRs:
  - 10.244.2.0/24
EOF

cat <<EOF | calicoctl  --allow-version-mismatch apply -f -
apiVersion: projectcalico.org/v3
kind: Node
metadata:
  annotations:
    projectcalico.org/kube-labels: '{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"calico-bgp-rr-worker3","kubernetes.io/os":"linux","rack":"rack1"}'
  creationTimestamp: "2022-12-05T08:40:29Z"
  labels:
    beta.kubernetes.io/arch: amd64
    beta.kubernetes.io/os: linux
    kubernetes.io/arch: amd64
    kubernetes.io/hostname: calico-bgp-rr-worker3
    kubernetes.io/os: linux
    rack: rack1
  name: calico-bgp-rr-worker3
spec:
  addresses:
  - address: 10.1.8.11
    type: InternalIP
  bgp:
    asNumber: 65008
    ipv4Address: 10.1.8.11/24
  orchRefs:
  - nodeName: calico-bgp-rr-worker3
    orchestrator: k8s
status:
  podCIDRs:
  - 10.244.3.0/24
EOF

# 1.4. peer to leaf0 switch
cat <<EOF | calicoctl  --allow-version-mismatch apply -f -
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: rack0-to-leaf0
spec:
  peerIP: 10.1.5.1
  asNumber: 65005
  nodeSelector: rack == 'rack0'
EOF

# 1.5. peer to leaf1 switch
cat <<EOF | calicoctl --allow-version-mismatch  apply -f -
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: rack1-to-leaf1
spec:
  peerIP: 10.1.8.1
  asNumber: 65008
  nodeSelector: rack == 'rack1'
EOF

root@superadmin-virtual-machine:~/wcni-kind/LabasCode/calico/06-calico-bgp-rr# ./3-prep-calico-bgp.sh

Calico-BGP FullMesh模式与RR模式 Day04