Jenkins 基于Kubernetes动态创建pod |
您所在的位置：网站首页 › kubernetes部署jenkins › Jenkins 基于Kubernetes动态创建pod |
Jenkins 基于Kubernetes动态创建pod
|
1、概述 DevOps 是一系列做法和工具，可以使 IT 和软件开发团队之间的流程实现自动化。其中，随着敏捷软件开发日趋流行，持续集成 (CI) 和持续交付 (CD) 已经成为该领域一个理想的解决方案。在 CI/CD 工作流中，每次集成都通过自动化构建来验证，包括编码、发布和测试，从而帮助开发者提前发现集成错误，团队也可以快速、安全、可靠地将内部软件交付到生产环境。 不过，传统的 Jenkins Master-Agent 架构（即多个 Agent 为一个 Master 工作）有以下不足。 如果 Master 宕机，整个 CI/CD 流水线会崩溃。资源分配不均衡，一些 Agent 的流水线任务 (Job) 出现排队等待，而其他 Agent 处于空闲状态。不同的 Agent 可能配置环境不同，并需要使用不同的编码语言。这种差异会给管理和维护带来不便。KubeSphere CI/CD 流水线工作流（和下面实践原理一样） KubeSphere CI/CD 流水线基于底层 Kubernetes Jenkins Agent 而运行。这些 Jenkins Agent 可以动态扩缩，即根据任务状态进行动态供应或释放。Jenkins Master 和 Agent 以 Pod 的形式运行在 KubeSphere 节点上。Master 运行在其中一个节点上，其配置数据存储在一个存储卷 (Volume) 中。Agent 运行在各个节点上，但可能不会一直处于运行状态，而是根据需求动态创建并自动删除。 当 Jenkins Master 收到构建请求，会根据标签动态创建运行在 Pod 中的 Jenkins Agent 并注册到 Master 上。当 Agent 运行完任务后，将会被释放，相关的 Pod 也会被删除。 动态供应 Jenkins Agent 动态供应 Jenkins Agent 有以下优势： 资源分配合理：KubeSphere 动态分配已创建的 Agent 至空闲节点，避免因单个节点资源利用率高而导致任务排队等待。高可扩缩性：当 KubeSphere 集群因资源不足而导致任务长时间排队等待时，您可以向集群新增节点。高可用性：当 Jenkins Master 故障时，KubeSphere 会自动创建一个新的 Jenkins Master 容器，并将存储卷挂载至新创建的容器，保证数据不会丢失，从而实现集群高可用。 2、安装插件 在Kubernetes平台部署Jenkins
在部署Jenkins的时候，也需要注意数据的持久化，存储在远端。 [root@master cicd]# cat jenkins.yml apiVersion: apps/v1 kind: Deployment metadata: name: jenkins namespace: ops labels: name: jenkins spec: replicas: 1 selector: matchLabels: name: jenkins template: metadata: name: jenkins labels: name: jenkins spec: serviceAccountName: jenkins containers: - name: jenkins image: jenkins/jenkins ports: - containerPort: 8080 - containerPort: 50000 resources: limits: cpu: 1 memory: 1.5Gi requests: cpu: 1 memory: 1Gi env: - name: LIMITS_MEMORY valueFrom: resourceFieldRef: resource: limits.memory divisor: 1Mi - name: JAVA_OPTS value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 volumeMounts: - name: jenkins-home mountPath: /var/jenkins_home securityContext: fsGroup: 1000 volumes: - name: jenkins-home persistentVolumeClaim: claimName: jenkins --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: jenkins namespace: ops spec: storageClassName: "managed-nfs-storage" accessModes: - ReadWriteMany resources: requests: storage: 5Gi --- apiVersion: v1 kind: Service metadata: name: jenkins namespace: ops spec: selector: name: jenkins type: NodePort ports: - name: http port: 80 targetPort: 8080 protocol: TCP nodePort: 30008 - name: agent port: 50000 protocol: TCP --- --- apiVersion: v1 kind: ServiceAccount metadata: name: jenkins namespace: ops --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: jenkins namespace: ops rules: - apiGroups: [""] resources: ["pods","events"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/exec"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/log"] verbs: ["get","list","watch"] - apiGroups: [""] resources: ["secrets","events"] verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: jenkins namespace: ops roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: jenkins subjects: - kind: ServiceAccount name: jenkins [root@master secrets]# pwd /ifs/kubernetes/ops-jenkins-pvc-921620d5-6d2c-4aaa-aa46-287fbd2fdb17/secrets [root@master secrets]# ls /ifs/kubernetes/ default-nfs-pvc-pvc-9e696088-2834-43d0-baef-f9d5a2820425 ops-grafana-pvc-c8291750-10d6-4501-961f-c351d72cec8f ops-es-pvc-pvc-d451bfeb-8dab-4c4a-84f9-bdb6bf6ce53f ops-jenkins-pvc-921620d5-6d2c-4aaa-aa46-287fbd2fdb17默认从国外网络下载插件，会比较慢，建议修改国内源： # 进入到nfs共享目录 cd /ifs/kubernetes/ops-jenkins-pvc-ea1e0744-e147-4150-9b05-3e38bc5feaa8/updates sed -i 's/https:\/\/updates.jenkins.io\/download/https:\/\/mirrors.tuna.tsinghua.edu.cn\/jenkins/g' default.json && \ sed -i 's/http:\/\/www.google.com/https:\/\/www.baidu.com/g' default.json # 删除pod重建，pod名称改成你实际的 kubectl delete pod jenkins-dccd449c7-vx6sj -n ops管理Jenkins->系统配置-->管理插件-->分别搜索Git Parameter/Git/Pipeline/kubernetes/Config File Provider，选中点击安装。 Git Parameter：Git参数化构建Git：拉取代码Pipeline：流水线kubernetes：连接Kubernetes动态创建Slave代理Config File Provider：存储kubectl用于连接k8s集群的kubeconfig配置文件 3、添加kubernetes集群管理Jenkins->Manage Nodes and Clouds->configureClouds->Add Kubernetes插件：Jenkins在Kubernetes集群中运行动态代理 插件介绍：https://github.com/jenkinsci/kubernetes-plugin
Kubernetes 地址：可以通过这个service地址去连接到，这样Jenkins pod就能够连接到kubernetes [root@master ~]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE app-log-stdout ClusterIP 10.233.1.6 80/TCP 9d kubernetes ClusterIP 10.233.0.1 443/TCP 11d
Jenkins 地址：这个是slave pod连接Jenkins的地址，就是让slave pod连接过来的地址。用这个地址去连接Jenkins master。 [root@master ~]# kubectl get svc -n ops NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE jenkins NodePort 10.233.60.174 80:30008/TCP,50000:30561/TCP 21h 4、Jenkins在K8s中动态创建代理Jenkins Master/Slave架构：当需要创建agent的时候，Jenkins调用k8s去创建，不用的话就将其销毁，这样就更加自动化了，并且更大化的去利用资源。 需要发布项目就创建一个slave pod，不需要的话就将其销毁，释放资源，动态化。 这个pod也是通过镜像启动，这个镜像也需要一些工具，比如maven和jdk这些，所以先要构建slave镜像。
目录里涉及四个文件： Dockerfile：构建镜像jenkins-slave：shell脚本启动slave.jarsettings.xml：修改maven官方源为阿里云源slave.jar：agent程序，接受master下发的任务构建并推送到镜像仓库： docker build -t reg.ctnrs.com/library/jenkins-slave-jdk:1.8 . docker push reg.ctnrs.com/library/jenkins-slave-jdk:1.8 [root@master jenkins-slave]# ls Dockerfile jenkins-slave jenkins-slave.tar jenkins.tar kubectl settings.xml slave.jar [root@master jenkins-slave]# cat Dockerfile FROM centos:7 LABEL maintainer lizhenliang RUN yum install -y java-1.8.0-openjdk maven curl git libtool-ltdl-devel && \ yum clean all && \ rm -rf /var/cache/yum/* && \ mkdir -p /usr/share/jenkins COPY slave.jar /usr/share/jenkins/slave.jar COPY jenkins-slave /usr/bin/jenkins-slave COPY settings.xml /etc/maven/settings.xml RUN chmod +x /usr/bin/jenkins-slave COPY kubectl /usr/bin/ ENTRYPOINT ["jenkins-slave"] pipeline { agent { kubernetes { label "jenkins-slave" yaml ''' apiVersion: v1 kind: Pod metadata: name: jenkins-slave spec: containers: - name: jnlp image: jenkins-slave-jdk:1.8 imagePullPolicy: Never ''' } } stages { stage('TestAgent') { steps { sh 'hostname' } } } }注意，这个容器名字 - name: jnlp不能写错，只能为jnlp，否者会创建失败 Started by user admin Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] echo [WARNING] label option is deprecated. To use a static pod template, use the 'inheritFrom' option. [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes ops/jenkins-slave-v3zqc-zkrw4 [Normal][ops/jenkins-slave-v3zqc-zkrw4][Scheduled] Successfully assigned ops/jenkins-slave-v3zqc-zkrw4 to node2 [Normal][ops/jenkins-slave-v3zqc-zkrw4][Pulled] Container image "jenkins-slave-jdk:1.8" already present on machine Still waiting to schedule task ‘jenkins-slave-v3zqc-zkrw4’ is offline [Normal][ops/jenkins-slave-v3zqc-zkrw4][Created] Created container jnlp [Normal][ops/jenkins-slave-v3zqc-zkrw4][Started] Started container jnlp Agent jenkins-slave-v3zqc-zkrw4 is provisioned from template jenkins-slave-v3zqc --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.ops/job/tomcat-web/18/" runUrl: "job/tomcat-web/18/" labels: jenkins: "slave" jenkins/label-digest: "03ddc3eddf95d5470d5c7fb6d2937abaeca3b79e" jenkins/label: "jenkins-slave" name: "jenkins-slave-v3zqc-zkrw4" spec: containers: - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_AGENT_NAME" value: "jenkins-slave-v3zqc-zkrw4" - name: "JENKINS_NAME" value: "jenkins-slave-v3zqc-zkrw4" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.ops/" image: "jenkins-slave-jdk:1.8" imagePullPolicy: "Never" name: "jnlp" resources: limits: {} requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false nodeSelector: kubernetes.io/os: "linux" restartPolicy: "Never" volumes: - emptyDir: medium: "" name: "workspace-volume" Running on jenkins-slave-v3zqc-zkrw4 in /home/jenkins/agent/workspace/tomcat-web [Pipeline] { [Pipeline] stage [Pipeline] { (TestAgent) [Pipeline] sh + hostname jenkins-slave-v3zqc-zkrw4 [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] End of Pipeline Finished: SUCCESS |
【本文地址】
公司简介
联系我们
今日新闻 |
点击排行 |
|
推荐新闻 |
图片新闻 |
|
专题文章 |