1、创建Amazon EFS CSI 驱动程序
亚马逊相关文档
在 Select trusted entity（选择受信任的实体）页面上操作
在 Add permissions（添加权限）页面上筛选AmazonEFSCSIDriverPolicy操作
记得将AmazonEBSVolumePolicy添加到我们创建的AmazonEKSNodeRole上否则创建存储卷pvc时会报错
Waiting for a volume to be created either by the external provisioner 'efs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
Warning ProvisioningFailed 3s (x3 over 7s) efs.csi.aws.com_efs-csi-controller-78dfdd968d-h6xqr_0d7848ac-e3ef-4353-8e46-91aed02078db failed to provision volume with StorageClass "efs-sc": rpc error: code = Unauthenticated desc = Access Denied. Please ensure you have the right AWS permissions: Access denied
与此同时我们还要将AmazonElasticFileSystemClientFullAccess 和AmazonElasticFileSystemFullAccess添加到AmazonEKSNodeRole
在 Name, review, and create（命名、查看和创建）页面操作命名AmazonEKS_EFS_CSI_DriverRole
点击创建角色后，编辑信任关系
用上图的id替换刚才创建时选的id,并在oidc.eks前面添加一行注意添加逗号
将 Condition 运算符从 “StringEquals” 修改为 “StringLike”
最后更新策略
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::XXXXXXXXX:oidc-provider/oidc.eks.ap-east-1.amazonaws.com/id/8C89F95E5FC3511924CD0C937428FF65"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringLike": {
"oidc.eks.ap-east-1.amazonaws.com/id/8C89F95E5FC3511924CD0C937428FF65:sub": "system:serviceaccount:kube-system:efs-csi-*",
"oidc.eks.ap-east-1.amazonaws.com/id/8C89F95E5FC3511924CD0C937428FF65:aud": "sts.amazonaws.com"
}
}
}
]
}
然后将AmazonElasticFileSystemClientFullAccess 或 AmazonElasticFileSystemFullAccess权限策略添加到AmazonEKS_EFS_CSI_DriverRole
2、安装 Amazon EFS CSI 驱动程序
1）查看某个集群版本的可用附加组件名称。将 1.29 替换为您的集群版本
# eksctl utils describe-addon-versions --kubernetes-version 1.29 | grep AddonName
"AddonName": "vpc-cni",
"AddonName": "solo-io_istio-distro",
"AddonName": "snapshot-controller",
"AddonName": "netapp_trident-operator",
"AddonName": "kubecost_kubecost",
"AddonName": "kube-proxy",
"AddonName": "eks-pod-identity-agent",
"AddonName": "coredns",
"AddonName": "aws-mountpoint-s3-csi-driver",
"AddonName": "aws-guardduty-agent",
"AddonName": "aws-efs-csi-driver",
"AddonName": "aws-ebs-csi-driver",
"AddonName": "amazon-cloudwatch-observability",
"AddonName": "adot",
查看要创建的附加组件的可用版本。将 1.29 替换为您的集群版本。将 name-of-addon 替换为您要查看其版本的附加组件的名称。该名称必须是前面步骤中返回的名称之一
# eksctl utils describe-addon-versions --kubernetes-version 1.29 --name aws-efs-csi-driver | grep AddonVersion
"AddonVersions": [
"AddonVersion": "v2.0.1-eksbuild.1",
"AddonVersion": "v2.0.0-eksbuild.1",
"AddonVersion": "v1.7.7-eksbuild.1",
"AddonVersion": "v1.7.6-eksbuild.2",
"AddonVersion": "v1.7.6-eksbuild.1",
"AddonVersion": "v1.7.5-eksbuild.2",
"AddonVersion": "v1.7.5-eksbuild.1",
"AddonVersion": "v1.7.4-eksbuild.1",
"AddonVersion": "v1.7.3-eksbuild.1",
"AddonVersion": "v1.7.2-eksbuild.1",
"AddonVersion": "v1.7.1-eksbuild.1",
"AddonVersion": "v1.7.0-eksbuild.1",
"AddonVersion": "v1.5.9-eksbuild.1",
"AddonVersion": "v1.5.8-eksbuild.1",
确定您要创建的附加组件是 Amazon EKS 还是 AWS Marketplace 附加组件。AWS Marketplace 具有第三方附加组件，需要您完成额外的步骤才能创建附加组件。
# eksctl utils describe-addon-versions --kubernetes-version 1.29 --name name-of-addon | grep ProductUrl
创建 Amazon EKS 附加组件
将
my-cluster替换为您的集群名称。将
name-of-addon替换为您要创建的附加组件的名称。如果您需要早于最新版本的附加组件版本，请将
latest替换为前面步骤的输出中返回的适用版本号。如果附加组件使用了服务账户角色，请将
111122223333替换为您的账户 ID，并将role-name替换为该角色的名称。有关为服务账户创建角色的说明，请参阅您正在创建的附加组件的文档。指定服务账户角色需要您的集群具有 IAM OpenID Connect（OIDC）提供程序。要确定您的集群是否具有此提供程序，或者要创建此提供程序，请参阅为集群创建 IAM OIDC 提供商。如果附加组件不使用服务账户角色，请删除
--service-account-role-arn arn:aws:iam::111122223333:role/role-name。此示例命令将覆盖附加组件的任何现有自行管理版本（如果有的话）的配置。如果您不想覆盖现有的自行管理附加组件的配置，请删除
--force选项。如果您删除此选项，并且 Amazon EKS 附加组件需要覆盖现有的自行管理附加组件的配置，那么创建 Amazon EKS 附加组件将会失败，并显示一条帮助您解决冲突的错误消息。在指定此选项之前，请确保 Amazon EKS 附加组件不会管理您需要管理的设置，因为这些设置会被此选项覆盖。
eksctl create addon --cluster my-cluster --name name-of-addon --version latest \
--service-account-role-arn arn:aws:iam::111122223333:role/role-name --force
我的命令如下
# eksctl create addon --cluster eks-test --name aws-efs-csi-driver --version v2.0.1-eksbuild.1 --service-account-role-arn arn:aws:iam::XXXXXX:role/AmazonEKS_EFS_CSI_DriverRole --force
2024-05-04 16:34:14 [ℹ] Kubernetes version "1.29" in use by cluster "eks-test"
2024-05-04 16:34:14 [ℹ] using provided ServiceAccountRoleARN "arn:aws:iam::XXXXXX:role/AmazonEKS_EFS_CSI_DriverRole"
2024-05-04 16:34:14 [ℹ] creating addon
查询描述驱动
# kubectl describe csidriver efs.csi.aws.com
Name: efs.csi.aws.com
Namespace:
Labels: <none>
Annotations: <none>
API Version: storage.k8s.io/v1
Kind: CSIDriver
Metadata:
Creation Timestamp: 2024-05-04T15:43:15Z
Managed Fields:
API Version: storage.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:spec:
f:attachRequired:
Manager: eks
Operation: Apply
Time: 2024-05-04T16:34:16Z
Resource Version: 8169
UID: 856f7022-dc56-4254-b3ab-916fb235d6f5
Spec:
Attach Required: false
Fs Group Policy: ReadWriteOnceWithFSType
Pod Info On Mount: false
Requires Republish: false
Se Linux Mount: false
Storage Capacity: false
Volume Lifecycle Modes:
Persistent
Events: <none>
删除
# eksctl delete addon --cluster=eks-test --name=aws-efs-csi-driver
2024-05-04 16:45:51 [ℹ] Kubernetes version "1.29" in use by cluster "eks-test"
2024-05-04 16:45:51 [ℹ] deleting addon: aws-efs-csi-driver
2024-05-04 16:45:51 [ℹ] deleted addon: aws-efs-csi-driver
2024-05-04 16:45:51 [ℹ] no associated IAM stacks found
# eksctl create addon --cluster=eks-test --name=aws-efs-csi-driver --force
2024-05-04 16:46:28 [ℹ] Kubernetes version "1.29" in use by cluster "eks-test"
2024-05-04 16:46:28 [ℹ] creating role using recommended policies
2024-05-04 16:46:28 [ℹ] deploying stack "eksctl-eks-test-addon-aws-efs-csi-driver"
2024-05-04 16:46:28 [ℹ] waiting for CloudFormation stack "eksctl-eks-test-addon-aws-efs-csi-driver"
2024-05-04 16:46:58 [ℹ] waiting for CloudFormation stack "eksctl-eks-test-addon-aws-efs-csi-driver"
2024-05-04 16:47:46 [ℹ] waiting for CloudFormation stack "eksctl-eks-test-addon-aws-efs-csi-driver"
2024-05-04 16:47:46 [ℹ] creating addon
修复
eksctl utils associate-iam-oidc-provider --region=ap-east-1 --cluster=eks-test
3、创建存储类
1）创建efs
AWS控制台创建EFS
将eks集群所在VPC、Subnet添加为EFS挂载目标
挂载目标的安全组开通入站端口2049,源地址为eks集群所在cidr
2）下载配置模板
curl -o storageclass.yaml https://raw.githubusercontent.com/kubernetes-sigs/aws-efs-csi-driver/master/examples/kubernetes/dynamic_provisioning/specs/storageclass.yaml
更改fileSystemId值为你的EFS ID
reclaimPolicy-Retain：手动回收类型，当pod资源被删除时被分配的pv不会被自动删除
volumeBindingMode-WaitForFirstConsumer：延迟pod与pv的绑定，直到pod被成功创建出来，pv才会与pod进行绑定
# cat storageclass.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: efs-sc
provisioner: efs.csi.aws.com
parameters:
provisioningMode: efs-ap
fileSystemId: fs-XXXXXXXXXXX
directoryPerms: "700"
gidRangeStart: "1000" # optional
gidRangeEnd: "2000" # optional
basePath: "/dynamic_provisioning" # optional
subPathPattern: "${.PVC.namespace}/${.PVC.name}" # optional
ensureUniqueDirectory: "true" # optional
reuseAccessPoint: "false" # optional
reclaimPolicy: Retain
volumeBindingMode: Immediate
#volumeBindingMode: WaitForFirstConsumer
创建存储类
# kubectl apply -f storageclass.yaml
storageclass.storage.k8s.io/efs-sc created
4、创建pvc
# cat efs-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: efs-pvc
spec:
accessModes:
- ReadWriteMany
volumeMode: Filesystem
resources:
requests:
storage: 2Gi
storageClassName: efs-sc
创建pvc
# kubectl apply -f efs-pvc.yaml
persistentvolumeclaim/efs-pvc created
查看pvc
kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
efs-pvc Bound pvc-b445157e-317c-482b-a9f1-7c61bbf7b101 2Gi RWX efs-sc <unset> 103s
5、nginx pod实例测试
volumeMounts:
- mountPath: /usr/share/nginx/html/
name: volume-xankc
subPath: html
volumes:
- name: volume-xankc
persistentVolumeClaim:
claimName: efs-pvc
测试结果：
#没有文件内容时输出报错
root@nginx-5d9b597d4d-jflrl:/usr/share/nginx/html# curl http://10.0.156.114
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx/1.24.0</center>
</body>
</html>
#生成新的index.html文件时
root@nginx-5d9b597d4d-jflrl:/usr/share/nginx/html# vim index.html
root@nginx-5d9b597d4d-jflrl:/usr/share/nginx/html# cat index.html
123345
root@nginx-5d9b597d4d-jflrl:/usr/share/nginx/html# curl http://10.0.156.114
123345
#新加一个pod节点时
root@nginx-5d9b597d4d-jflrl:/usr/share/nginx/html# curl http://10.0.128.121
123345
#重启pods后两个pod都能重新读到数据
root@nginx-7d8456c4d7-pzjtz:/# curl http://10.0.158.211
123345
root@nginx-7d8456c4d7-pzjtz:/# curl http://10.0.129.104
123345
与此同时挂载零个tomcat的logs路径在efs盘上
volumeMounts:
- mountPath: /usr/local/tomcat/logs
name: volume-zcb56
subPath: htm/tomcat/
volumes:
- name: volume-zcb56
persistentVolumeClaim:
claimName: efs-pvc
验证
#第一次访问日志第一行
# head -1 logs/catalina.2024-05-04.log
04-May-2024 17:10:30.254 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name: Apache Tomcat/10.1.23
#重启tomcat后访问第一行二者一致
# head -1 logs/catalina.2024-05-04.log
04-May-2024 17:10:30.254 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name: Apache Tomcat/10.1.23
整个测试deployment
---
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
k8s.kuboard.cn/displayName: nginx
name: nginx
namespace: default
spec:
replicas: 1
selector:
matchLabels:
k8s.kuboard.cn/name: nginx
template:
metadata:
creationTimestamp: null
spec:
containers:
- image: 'nginx:1.24'
imagePullPolicy: IfNotPresent
name: nginx
ports:
- containerPort: 80
name: sddaf
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /usr/share/nginx/html/
name: volume-xankc
subPath: html
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- name: volume-xankc
persistentVolumeClaim:
claimName: efs-pvc
---
apiVersion: v1
kind: Service
metadata:
annotations: {}
labels:
k8s.kuboard.cn/name: nginx
name: nginx
namespace: default
spec:
ports:
- name: htrmqz
port: 80
protocol: TCP
targetPort: 80
selector:
k8s.kuboard.cn/name: nginx
sessionAffinity: None
type: ClusterIP