文章目录
helm方式安装在K8S上
1、下载charts
helm pull gitlab/gitlab-runner
tar -zxvf gitlab-runner-0.27.0.tgz
#解压后内容：
CHANGELOG.md
Chart.yaml #
CONTRIBUTING.md
LICENSE
Makefile
NOTICE
README.md
templates #
values.yaml #
2、修改 values.yaml，templates 等资源
values.yaml
gitlabUrl: https://gitlab.example.com/ #修改为gitlab地址
runnerRegistrationToken: "" #修改为gitlab runner token，可从 /admin/runners 查看
rbac:
create: true
clusterWideAccess: true
serviceAccountName: gitlab-runner-gitlab-runner
runners:
tags: ""
serviceAccountName: gitlab-runner-gitlab-runner
templates/configmap.yaml
主要用于maven，docker绑定本地目录，修改 entrypoint key。增加 config.toml 配置。
#以下一段是增加的内容
cat >>/home/gitlab-runner/.gitlab-runner/config.toml <<EOF
[[runners.kubernetes.volumes.host_path]]
name = "maven"
mount_path = "/root/.m2"
read_only = false
host_path = "/root/.m2"
[[runners.kubernetes.volumes.host_path]]
name = "docker"
mount_path = "/var/run/docker.sock"
read_only = true
host_path = "/var/run/docker.sock"
EOF
# Start the runner
exec /entrypoint run --user=gitlab-runner \
--working-directory=/home/gitlab-runner
新的方式可以通过values.yaml 的 runners 段设置属性。不能同时以上面和下面2种方式，不然会重复。
runners:
config: |
[[runners]]
[runners.kubernetes]
image = "ubuntu:16.04"
[[runners.kubernetes.volumes.host_path]]
name = "maven"
mount_path = "/root/.m2"
read_only = false
host_path = "/root/.m2"
[[runners.kubernetes.volumes.host_path]]
name = "docker"
mount_path = "/var/run/docker.sock"
read_only = true
host_path = "/var/run/docker.sock"
_cache.tpl
里面CACHE_S3_INSECURE 参数 是固定值，导致 values 配置无效。
{{- if .Values.runners.cache.s3CacheInsecure }}
- name: CACHE_S3_INSECURE
value: "true"
{{- end }}
{{ default "" .Values.runners.cache.s3BucketLocation | quote }}
#----- 修改为：
- name: CACHE_S3_INSECURE
value: {{ default "true" .Values.runners.cache.s3CacheInsecure | quote }}
3、添加 helm 仓库
helm repo add gitlab https://charts.gitlab.io
4、创建namespace、等资源
kubectl create ns gitlab
---
apiVersion: v1
data:
accesskey: bWluaW8= #base64 编码
secretkey: #base64 编码
kind: Secret
metadata:
name: minio-secrets
type: Opaque
5、启动 gitlab-runner
# 安装仓库中的chart
$ helm install gitlab-runner --namespace gitlab -f values.yaml gitlab/gitlab-runner
#安装本地的chart
helm install gitlab-runner ./ --namespace gitlab
#更新配置--通过本地chart更新
helm upgrade --install gitlab-runner ./gitlab-runner --namespace gitlab
#卸载
helm uninstall gitlab-runner --namespace gitlab
如果没有修改gitlabUrl，则会提示更新配置
#############################################################################################
## WARNING: You did not specify an gitlabUrl in your 'helm install' call. ##
#############################################################################################
This deployment will be incomplete until you provide the URL that your
GitLab instance is reachable at:
helm upgrade gitlab-runner \
--set gitlabUrl=http://gitlab.your-domain.com,runnerRegistrationToken=your-registration-token \
gitlab/gitlab-runner
#也可以使用命令：helm upgrade
参考
安装：https://docs.gitlab.com/runner/install/
https://docs.gitlab.com/runner/
执行器参数：https://docs.gitlab.com/runner/executors/kubernetes.html
cache secret ： https://blog.csdn.net/xichenguan/article/details/101436883
gitlab runner配置（toml配置项）：https://docs.gitlab.com/runner/configuration/advanced-configuration.html
gitlab CI/CD 文件变量
新的版本支持，比较旧的不支持。
但是可以通过base64 编解码来实现
echo $(cat ~/.kube/config | base64) | tr -d " "
deploy_k8s_job:
image: registry.cn-hangzhou.aliyuncs.com/haoshuwei24/kubectl:1.16.6
stage: deploy_k8s
tags:
- k8s-runner
script:
- mkdir -p /etc/deploy
- echo $kube_config |base64 -d > $KUBECONFIG
- sed -i "s/IMAGE_TAG/$CI_PIPELINE_ID/g" deployment.yaml
- cat deployment.yaml
- kubectl apply -f deployment.yaml
缓存服务器
使用minio作为缓存服务器。配置如下：
cache:
## General settings
## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration and https://docs.gitlab.com/runner/install/kubernetes.html#using-cache-with-configuration-template
cacheType: s3
cachePath: "gitlab_runner"
cacheShared: true
## S3 settings
## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration and https://docs.gitlab.com/runner/install/kubernetes.html#using-cache-with-configuration-template
s3ServerAddress: s3.amazonaws.com
s3BucketName: "gitlabrunner" #Minio bucket
s3BucketLocation: #minio时区。
s3CacheInsecure: false #是否在不安全模式。true：使用http；false使用https，不设置则默认为false。
## S3 the name of the secret.
secretName: minio-secrets #minio 对应的secret
**注意：**很多博客或者什么资料，把s3CacheInsecure解释为是否使用https，正确的解释应该是是否在不安全模式。意思刚好相反。
最终的文件内容可以在
/home/gitlabrunner/.gitlabrunner/config.toml文件查看。值为false时不会出现在config.toml中。
以上方式是废弃的方式，新的方式采用template。对应的template为_cache.yaml
runners:
config: |
[[runners]]
[runners.kubernetes]
image = "ubuntu:16.04"
[runners.cache]
Type = "s3"
Path = "gitlab_runner"
Shared = true
[runners.cache.s3]
ServerAddress = "s3.amazonaws.com"
BucketName = "gitlabrunner"
BucketLocation = "eu-west-1"
Insecure = true
#AccessKey = "access"
#SecretKey = "secret123456"
cache:
secretName: minio-secrets
以上使用到了一个secret。通过以下语句创建secret 或者通过yaml创建。
kubectl create secret generic minio \
--from-literal=accesskey="access" \
--from-literal=secretkey="secret123456" -n gitlab
参考：https://docs.gitlab.com/runner/install/kubernetes.html#using-cache-with-configuration-template
K8S部署
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: minio
finalizers:
- kubernetes.io/pvc-protection
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
storageClassName: rook-cephfs
volumeMode: Filesystem
---
apiVersion: v1
kind: Service
metadata:
labels:
app: minio
name: minio
spec:
ports:
- name: 9000-tcp
port: 9000
protocol: TCP
targetPort: 9000
selector:
app: minio
sessionAffinity: None
type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: minio
name: minio
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: minio
template:
metadata:
labels:
app: minio
spec:
containers:
- image: minio/minio:RELEASE.2019-02-26T19-51-46Z
imagePullPolicy: Always
env:
- name: MINIO_ACCESS_KEY
value: minio
- name: MINIO_SECRET_KEY
value: ssss
command:
- minio
- server
- /data
name: minio
ports:
- containerPort: 9000
protocol: TCP
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /data
name: volume-data
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
imagePullSecrets:
- name: harbor-key #注意docker 仓库 key
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- name: volume-data
persistentVolumeClaim:
claimName: minio
kubectl apply -f minio.single.yaml -n gitlab
docker镜像
maven
maven:3.6.3-openjdk-8： https://registry.hub.docker.com/_/maven
maven的setting.xml 可以通过configmap解决，（没验证过）
[[runners.kubernetes.volumes.config_map]]
name = "gitlab-runner-maven"
mount_path = "/usr/share/maven/configmap/"
也可以通过mount path解决（见前面内容）
[[runners.kubernetes.volumes.host_path]]
docker
docker ：https://registry.hub.docker.com/_/docker 。版本：（20.10.2）
需要在/root/.docker/config.json 中增加auth 凭据。
FROM docker
MAINTAINER lihz
ADD config.json /root/.docker/config.json
config.json
主要是增加访问凭据
{
"auths": {
"192.168.1.X": {
"auth": "?????????????"
},
"docker-registry-default.cloud.com": {
"auth": "YWRtaW46TEpWUUhYX2g3MGFabGYtUlJLdDc1RlBmRW5LeFRXXXXXXXXXXX"
}
}
,
"experimental": true
}
安装docker buildx
如果需要支持多平台打包，则需要安装docker buildx (github.com/docker/buildx v0.10.5 86bdced)，下载
wget -O docker-buildx https://github.com/docker/buildx/releases/download/v0.10.5/buildx-v0.10.5.linux-amd64
mkdir -p /usr/libexec/docker/cli-plugins/docker-buildx
mv docker-buildx /usr/libexec/docker/cli-plugins/docker-buildx
chmod +x /usr/libexec/docker/cli-plugins/docker-buildx
docker buildx version
docker客户端需要开启实验室功能
$ cat ~/.docker/config.json
{
"experimental": "enabled"
}
# 确认实验室性能开启。
$ docker version
构造docker 打包的镜像，包含buildx
FROM docker:20.10.2
MAINTAINER lihz
ADD config.json /root/.docker/config.json
RUN mkdir -p /usr/libexec/docker/cli-plugins/ && mkdir -p /etc/docker
COPY docker-buildx /usr/libexec/docker/cli-plugins/docker-buildx
COPY daemon.json buildkitd.toml /etc/docker/
RUN chmod +x /usr/libexec/docker/cli-plugins/docker-buildx
ENV IMAGE_BUILDKIT=192.168.1.X/GROUP/buildkit:buildx-stable-1
buildkitd.toml
debug = true
# insecure-entitlements allows insecure entitlements, disabled by default.
insecure-entitlements = [ "network.host", "security.insecure" ]
# 如果不加这些，就会默认使用https请求。
# optionally mirror configuration can be done by defining it as a registry.
[registry."192.168.1.XX"]
http = true
insecure = true
- 打包
minio
minio/minio:RELEASE.2019-02-26T19-51-46Z : https://registry.hub.docker.com/r/minio/minio
node
node:14.7.0 : https://registry.hub.docker.com/_/node
FROM node:14.7.0
RUN npm config set registry https://registry.npm.taobao.org
helm
alpine/helm:3.5.0：https://registry.hub.docker.com/r/alpine/helm
Dockerfile：
From 192.168.1.X/GROUP/helm:3.5.0
#增加K8S的凭据
ADD config /etc/deploy/config
config：
K8S的凭据
apiVersion: v1
clusters:
- cluster:
certificate-authority-data:
........
server: https://lb.kubesphere.local:6443
name: cluster.local
contexts:
- context:
cluster: cluster.local
namespace: demo
user: kubernetes-admin
name: ctx-demo
- context:
cluster: cluster.local
user: kubernetes-admin
name: kubernetes-admin@cluster.local
current-context: ctx-demo
kind: Config
preferences: {}
users:
- name: kubernetes-admin
user:
client-certificate-data:
..........
client-key-data:
..........
kubectl
将业务镜像部署到k8s上
sonar-scanner-cli
用于扫描前端代码。参考：https://docs.sonarqube.org/latest/analysis/scan/sonarscanner/
sonarsource/sonar-scanner-cli:4.6：https://registry.hub.docker.com/r/sonarsource/sonar-scanner-cli
Dockerfile：
From sonarsource/sonar-scanner-cli:4.6
#登录凭据
ENV SONAR_HOST_URL=http://192.168.1.XXX:9000 SONAR_LOGIN=a34d8e475e19faa108404fec82cd058493XXXXXX
ENTRYPOINT [""]
绑定目录：
docker run --rm -v $PWD:/usr/src
问题
https://docs.gitlab.com/ee/ci/docker/using_docker_build.html
清除cache
cache是没有过期时间的，而且每一次新的push触发的pipeline，都会重新生成cache，重新生成的cache的名字为“-”，其中num是随着push数量递增的。如果不去清除cache，cache会永久保留在Runner上，日积月累会填满存储空间的，因此最好隔一段时间进行一次清除，清除方法请参考https://docs.gitlab.com/ee/ci/caching/#clearing-the-cache，或者使用clear_volumes.sh 这个简单脚本来处理它, 清除cache的原理是将相关的volume移除，当然，docker也有自带的清除命令，推荐将docker system prune -f --volumes加入到定时任务中。
helm执行时无权限
Executing "step_script" stage of the job script
$ sed -i "s/IMAGE_TAG/$DOCKER_TAG/g;s/CI_PROJECT_NAME/$CI_PROJECT_NAME/g;s/SVC_PORT/${SVC_PORT}/g;" ${MODULE_NAME}/src/main/charts/values.yaml
$ sed -i "s/CI_PROJECT_NAME/$CI_PROJECT_NAME/g" ${MODULE_NAME}/src/main/charts/Chart.yaml
$ helm upgrade --install $CI_PROJECT_NAME ${MODULE_NAME}/src/main/charts -n $K8S_NS
Release "sample" does not exist. Installing it now.
Error: rendered manifests contain a resource that already exists. Unable to continue with install: could not get information about the resource: deployments.apps "sample" is forbidden: User "system:serviceaccount:gitlab:gitlab-runner-gitlab-runner" cannot get resource "deployments" in API group "apps" in the namespace "release"
ERROR: Job failed: command terminated with exit code 1
是由于 gitlab runner的权限问题
执行以下语句：
kubectl create clusterrolebinding gitlab-cluster-admin --clusterrole=cluster-admin --group=system:serviceaccounts
下载镜像失败
Job failed (system failure): prepare environment: image pull failed
临时解决方法，在K8S节点 docker pull <IMAGE> 把镜像下载下来
根本性解决：
打开以下选项，并设置docker仓库的secret。
## Specifying ImagePullSecrets on a Pod （设置在gitlab-runner中）
## Kubernetes supports specifying container image registry keys on a Pod.
## ref: https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
##
imagePullSecrets:
- name: "harbor-key"
## For RBAC support:
rbac:
create: true
## Specify one or more imagePullSecrets used for pulling the runner image
##
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account
##
imagePullSecrets: ["harbor-key"]
## Configuration for the Pods that the runner launches for each new job
##
runners:
## Specify one or more imagePullSecrets （用于拉取image）
##
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
##
## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration
imagePullSecrets: ["harbor-key"]
## Run all containers with the privileged flag enabled
## This will allow the docker:dind image to run if you need to run Docker
## commands. Please read the docs before turning this on:
## ref: https://docs.gitlab.com/runner/executors/kubernetes.html#using-dockerdind
##
## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration
#
privileged: true
下载gitlab-runner镜像失败
在K8S部署环境中，会下载以下镜像，可能会导致失败，最好重新tag在本地仓库
# helm配置（helpers.tpl中）为：
printf "192.168.1.X/GROUP/gitlab-runner:alpine-%s" $appVersion
#tag为：
192.168.1.x/GROUP/gitlab-runner:alpine-v13.8.0
# 最后一部分是 CI_RUNNER_VERSION，对应的版本的 sha256ID，参考：https://gitlab.com/gitlab-org/gitlab-runner/-/tags?sort=updated_desc&search=13.8.0
gitlab/gitlab-runner-helper:x86_64-775dd39d
docker tag gitlab/gitlab-runner-helper:x86_64-775dd39d 192.168.1.X/GROUP/gitlab-runner-helper:x86_64-775dd39d
docker push 192.168.1.X/GROUP/gitlab-runner-helper:x86_64-775dd39d
修改配置：
[[runners]]
[runners.kubernetes]
image = "ubuntu:22.04"
# 由上文可知
helper_image = "192.168.1.X/GROUP/gitlab-runner-helper:x86_64-775dd39d"
Gitlab-ci中使用
java
variables:
DOCKER_TAG: "3.0.0-RELEASE"
MODULE_NAME: "project-biz"
SONAR_PROJECT_KEY: "project"
stages:
- package
- docker_build
mvn_build_job:
image: ${DEPOSITORY}/maven
stage: package
script:
- mvn clean verify sonar:sonar -DskipTests -DskipDocker -Dsonar.projectVersion=master -Dsonar.projectKey=$SONAR_PROJECT_KEY -Dsonar.host.url=${SONAR_URL} -Dsonar.login=${SONAR_TOKEN}
- mvn deploy -B -DskipTests -DskipDocker
artifacts:
paths:
- ${MODULE_NAME}/target/*.jar
only:
- master
- /^.*-dev$/
when: manual
mvn_build_release_job:
image: ${DEPOSITORY}/maven
stage: package
script:
- mvn deploy -B -DskipTests -DskipDocker
artifacts:
paths:
- ${MODULE_NAME}/target/*.jar
only:
- /^.*-RELEASE$/
- /^.*-release/
- /^.*-hotfix$/
docker_build_release_job:
image: ${DEPOSITORY}/docker
stage: docker_build
script:
- cp ${MODULE_NAME}/target/*.jar ${MODULE_NAME}/src/main/docker
- docker build -t ${DEPOSITORY}/${MODULE_NAME}:${DOCKER_TAG} ${MODULE_NAME}/src/main/docker
- docker push ${DEPOSITORY}/${MODULE_NAME}:${DOCKER_TAG}
only:
- /^.*-RELEASE$/
- /^.*-release/
前端
variables:
DOCKER_TAG: "dev"
MODULE_NAME: "biz-web"
stages:
- package
- docker_build
- deploy
npm_build_job:
image: maven:3.6.3-openjdk-8
stage: package
cache:
paths:
- node_modules/
artifacts:
paths:
- dist
script:
- npm install
- npm run build
only:
- master
- /^.*-dev$/
when: manual
docker_build_job:
image: docker
stage: docker_build
script:
- docker build -t ${DEPOSITORY}/${MODULE_NAME}:${DOCKER_TAG} ./
dependencies:
- npm_build_job
only:
- master
- /^.*-dev$/
when: manual
docker_build_release_job:
image: docker
stage: docker_build
script:
- docker build -t ${DEPOSITORY}/${MODULE_NAME}:${DOCKER_TAG} ./
dependencies:
- npm_build_job
only:
- /^.*-RELEASE$/
- /^.*-release/
- /^.*-hotfix$/