1_引入
随着 Kubernetes 生态系统的持续发展，越来越多高层次的对象将会不断涌现。比起目前使用的对象，新对象将更加专业化。
有了它们，开发者将不再需要逐一进行 Deployment、Service、configMap 等步骤，而是创建并管理一些用于表达整个应用程序或者软件服务的对象。
我们能使用自定义控制器观察高阶对象，并在这些高阶对象的基础上创建底层对象。
例如，你想在 Kubernetes 集群中运行一个 messaging 代理，只需要创建一个队列资源实例，而自定义队列控制器将自动完成所需的 Secret、Deployment 和 Service。目前，Kubernetes 已经提供了类似的自定义资源添加方式。
2_CRD 的概念
CustomResourceDefinitions（CRD）允许开发者向 Kubernetes API 服务提交 CRD 对象，即可以定义新的资源类型。在成功提交后，开发者可以通过 API 服务提交 JSON 清单或 YAML 清单来创建自定义资源，以及其他 Kubernetes 资源实例。
注意：在 Kubernetes 1.7 之前的版本中，需要通过 ThirdPartyResource 对象的方式来定义自定义资源，ThirdPartyResource 于 Kubernetes 1.8 中被 CRD 替代。
开发者可以通过创建 CRD 来创建新的对象类型。不过，如果创建的对象无法在集群中解决实际问题，那么它就是一个无效特性。通常，CRD 与所有 Kubernetes 核心资源都有一个基于自定义对象有效实现目标的控制器。
| CRD 的创建流程 |
|---|
 |
3_CRD 安装
CRD 添加
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata: # 资源名复数.组名
name: websites.extensions.example.com
spec: # API 组名
group: extensions.example.com
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema: # 资源的结构模式，用于校验资源字段
type: object # 顶层资源是一个对象
properties: # 定义了资源的 spec 部分，类型为 object
spec:
type: object
properties:
gitRepo:
type: string
required: # 指定 spec 是必须字段
- spec
scope: Namespaced
names:
plural: websites # 定义资源的复数形式，供 API 使用
singular: website # 定义资源的单数形式
kind: Website # 定义资源的 Kubernetes 对象类型（kind 字段值）
代理当前接口，主要为了跳过一些认证
kubectl proxy
发起对当前接口的监听
curl http://localhost:8001/apis/extensions.example.com/v1/websites?watch=true
测试 CRD 是否有效，创建 Website 的资源清单文件
apiVersion: extensions.example.com/v1
kind: Website
metadata:
name: website
namespace: default
spec:
gitRepo: https://gitee.com/efewagtehqwedqw/website.git
虽然资源出来了，但是没有人为其做实际的动作
| 查看结果 |
|---|
 |
Json数据模型创建：
{
"type": "ADDED",
"object": {
"apiVersion": "extensions.example.com/v1",
"kind": "Website",
"metadata": {
"creationTimestamp": "2025-01-14T17:56:32Z",
"generation": 1,
"managedFields": [
{
"apiVersion": "extensions.example.com/v1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:spec": {
".": {},
"f:gitRepo": {}
}
},
"manager": "kubectl-create",
"operation": "Update",
"time": "2025-01-14T17:56:32Z"
}
],
"name": "website",
"namespace": "default",
"resourceVersion": "448560",
"uid": "b0ef73a5-cd1c-4f18-b8d2-08e2bb612bcb"
},
"spec": {
"gitRepo": "https://gitee.com/efewagtehqwedqw/website.git"
}
}
}
数据模型——删除
{
"type": "DELETED",
"object": {
"apiVersion": "extensions.example.com/v1",
"kind": "Website",
"metadata": {
"creationTimestamp": "2025-01-14T18:54:00Z",
"generation": 1,
"managedFields": [
{
"apiVersion": "extensions.example.com/v1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:spec": {
".": {},
"f:gitRepo": {}
}
},
"manager": "kubectl-create",
"operation": "Update",
"time": "2025-01-14T18:54:00Z"
}
],
"name": "website",
"namespace": "default",
"resourceVersion": "456357",
"uid": "e6467fdd-1583-4b1a-84c6-d391f0fb691b"
},
"spec": {
"gitRepo": "https://gitee.com/efewagtehqwedqw/website.git"
}
}
}
我们只需要根据监听到的消息做出动作即可，自定义案例逻辑：只需要提供 Website 类型资源清单文件，我们就直接自动创建好对应的 deployment、Service，还能根据给出的 gitRepo 准备好 index.html
| 可以发现两个容器和保证两个容器数据一致性的 emptyDir |
|---|
 |
创建 website-controller 代码，主要逻辑就是接收到 JSON 格式消息后根据自定义逻辑请求 ApiServer
package org.example.controller;
import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.extern.slf4j.Slf4j;
import org.example.WebsiteControllerApplication;
import org.example.models.WebsiteWatchEvent;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
import org.springframework.web.reactive.function.client.WebClient;
import reactor.core.publisher.Flux;
import java.io.IOException;
import java.io.InputStream;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
@Slf4j
@Service
public class WebsiteController {
private final WebClient webClient;
@Value("${k8s.api.url}") // 用于 Kubernetes API 地址配置
private String k8sApiUrl;
public WebsiteController() {
this.webClient = WebClient.create(); // 使用 WebClient 创建基础客户端
}
private final ObjectMapper objectMapper = new ObjectMapper();
public void startWatching() {
log.info("website-controller started.");
// 开启一个新的线程处理任务
Runnable runnable = this::watchWebsites;
Thread thread = new Thread(runnable);
thread.start();
}
public void watchWebsites() {
// 使用响应式流的方式处理连续数据流
Flux<String> flux = webClient.get()
.uri(k8sApiUrl + "/apis/extensions.example.com/v1/websites?watch=true")
.retrieve()
.bodyToFlux(String.class); // 以 String 流的形式处理返回的每一部分数据
flux.subscribe(eventJson -> {
try {
// 这里是处理每个事件的逻辑
WebsiteWatchEvent eventObj = objectMapper.readValue(eventJson, WebsiteWatchEvent.class);
log.info("Received event: {}: {}: {}: {}", eventObj.type, eventObj.object.getApiVersion(), eventObj.object.metadata.name, eventObj.object.spec.gitRepo);
if ("ADDED".equals(eventObj.type)) {
createWebsite(eventObj.object);
} else if ("DELETED".equals(eventObj.type)) {
deleteWebsite(eventObj.object);
} else {
log.warn("Unexpected event: {}", eventObj.type);
}
} catch (IOException e) {
log.error("Error occurred while watching websites IO", e);
}
});
}
private void createWebsite(WebsiteWatchEvent.Website website) {
createResource(website, "api/v1", "services", "service-template.json");
createResource(website, "apis/apps/v1", "deployments", "deployment-template.json");
}
private void deleteWebsite(WebsiteWatchEvent.Website website) {
deleteResource(website, "api/v1", "services");
deleteResource(website, "apis/apps/v1", "deployments");
}
private void createResource(WebsiteWatchEvent.Website website, String apiGroup, String kind, String filename) {
try {
log.info("Creating {} with name {} in namespace {}", kind, website.metadata.name, website.metadata.namespace);
// 读取模板文件
InputStream inputStream = WebsiteControllerApplication.class.getClassLoader().getResourceAsStream(filename);
if (inputStream == null) {
log.error("Resource file not found: {}", filename);
return;
}
String template = new String(inputStream.readAllBytes(), StandardCharsets.UTF_8);
template = template.replace("[NAME]", website.metadata.name);
template = template.replace("[GIT-REPO]", website.spec.gitRepo);
// 发送 POST 请求
String url = String.format("%s/%s/namespaces/%s/%s", k8sApiUrl, apiGroup, website.metadata.namespace, kind);
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(url))
.POST(HttpRequest.BodyPublishers.ofString(template))
.header("Content-Type", "application/json")
.build();
HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString());
log.info("Resource {} created successfully.", kind);
} catch (IOException | InterruptedException e) {
log.error("Error creating resource", e);
}
}
private void deleteResource(WebsiteWatchEvent.Website website, String apiGroup, String kind) {
try {
log.info("Deleting {} with name {} in namespace {}", kind, website.metadata.name, website.metadata.namespace);
String url = String.format("%s/%s/namespaces/%s/%s/%s", k8sApiUrl, apiGroup, website.metadata.namespace, kind, website.metadata.name);
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(url))
.DELETE()
.build();
HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString());
log.info("Resource {} deleted successfully.", kind);
} catch (IOException | InterruptedException e) {
log.error("Error deleting resource", e);
}
}
}
准备好的资源清单模版，deployment
{
"apiVersion": "apps/v1",
"kind": "Deployment",
"metadata": {
"name": "[NAME]",
"labels": {
"webserver": "[NAME]"
}
},
"spec": {
"replicas": 1,
"selector": {
"matchLabels": {
"webserver": "[NAME]"
}
},
"template": {
"metadata": {
"name": "[NAME]",
"labels": {
"webserver": "[NAME]"
}
},
"spec": {
"containers": [
{
"image": "nginx:1.27.3-alpine",
"name": "main",
"volumeMounts": [
{
"name": "html",
"mountPath": "/usr/share/nginx/html",
"readOnly": true
}
],
"ports": [
{
"containerPort": 80,
"protocol": "TCP"
}
]
},
{
"image": "assigned/website:gitsync",
"name": "git-sync",
"env": [
{
"name": "GIT_SYNC_REPO",
"value": "[GIT-REPO]"
},
{
"name": "GIT_SYNC_DEST",
"value": "/gitrepo"
},
{
"name": "GIT_SYNC_BRANCH",
"value": "master"
},
{
"name": "GIT_SYNC_REV",
"value": "FETCH_HEAD"
},
{
"name": "GIT_SYNC_WAIT",
"value": "10"
}
],
"volumeMounts": [
{
"name": "html",
"mountPath": "/gitrepo"
}
]
}
],
"volumes": [
{
"name": "html",
"emptyDir": {}
}
]
}
}
}
}
Service
{
"apiVersion": "v1",
"kind": "Service",
"metadata": {
"labels": {
"webserver": "[NAME]"
},
"name": "[NAME]"
},
"spec": {
"type": "NodePort",
"ports": [
{
"port": 80,
"protocol": "TCP",
"targetPort": 80
}
],
"selector": {
"webserver": "[NAME]"
}
}
}
打包后封装容器镜像
FROM openjdk:11-jdk
LABEL maintainer="sy<1463476251@qq.com>"
COPY website-controller.jar /usr/local
WORKDIR /usr/local
ENTRYPOINT [ "java","-jar","website-controller.jar" ]
使用docker build命令构建镜像并分发到其他各个节点上，封装镜像已上传至 hub.docker.com，镜像名为assigned/website:controller
docker build -t assigned/website:controller .
配置 kubectl proxy 容器权限
apiVersion: v1
kind: Namespace
metadata:
name: website
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: website-controller
namespace: website
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: website-controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: website-controller
namespace: website
部署 website-controller
apiVersion: apps/v1
kind: Deployment
metadata:
name: website-controller
namespace: website
spec:
replicas: 1
selector:
matchLabels:
app: website-controller
template:
metadata:
labels:
app: website-controller
spec:
containers:
- image: assigned/website:controller
imagePullPolicy: IfNotPresent
name: main
- image: assigned/website:kubectl-proxy
name: proxy
serviceAccount: website-controller
serviceAccountName: website-controller
再次使用 Website 的资源清单文件进行创建，过一段时间后查看结果
[root@k8s-master 3]# kubectl get pod,svc,deploy -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/website-86bc8fb756-pzh2q 2/2 Running 0 4m36s 10.244.85.241 k8s-node01 <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 21d <none>
service/website NodePort 10.2.104.81 <none> 80:30418/TCP 4m36s webserver=website
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.apps/website 1/1 1 1 4m36s main,git-sync nginx:1.27.3-alpine,assigned/website:gitsync webserver=website
在浏览器中访问 masterIP:30418 即可