CCE集群使用Jenkins持续发布
所有文档

          容器引擎 CCE

          CCE集群使用Jenkins持续发布

          持续构建与发布是我们日常工作中必不可少的一个步骤,目前大多公司都采用 Jenkins 集群来搭建符合需求的 CI/CD 流程,Jenkins的持续发布流程可以跟Kubernetes集群更好的对接,更好的发挥它的部署优势,这篇文档可以指定用户把Jenkins发布流程跟CCE集群集成。

          1. 设置 Jenkins 存储目录

          Kubenetes 环境下所起的应用都是一个个Docker镜像,为了保证应用重启的情况下数据安全,所以需要将Jenkins的数据目录持久化到存储中。这里用的是CCE提供的多种持久化存储之一,方便在Kubernetes环境下应用启动节点转义数据一致。当然也可以选择存储到本地,但是为了保证应用数据一致,需要设置Jenkins固定到某一 Kubernetes节点。

          参考 容器引擎CCE-操作指南-存储管理 一节。

          选取任意一种方式部署生成PVC,记录下PVC的名字。

          2. 部署 Jenkins Server 到 Kubernetes

          service-account.yaml

          ---
          apiVersion: v1
          kind: ServiceAccount
          metadata:
            name: jenkins
           
          ---
          kind: Role
          apiVersion: rbac.authorization.k8s.io/v1beta1
          metadata:
            name: jenkins
          rules:
          - apiGroups: [""]
            resources: ["pods"]
            verbs: ["create","delete","get","list","patch","update","watch"]
          - apiGroups: [""]
            resources: ["pods/exec"]
            verbs: ["create","delete","get","list","patch","update","watch"]
          - apiGroups: [""]
            resources: ["pods/log"]
            verbs: ["get","list","watch"]
          - apiGroups: [""]
            resources: ["events"]
            verbs: ["watch"]
          - apiGroups: [""]
            resources: ["secrets"]
            verbs: ["get"]
           
          ---
          apiVersion: rbac.authorization.k8s.io/v1beta1
          kind: RoleBinding
          metadata:
            name: jenkins
          roleRef:
            apiGroup: rbac.authorization.k8s.io
            kind: Role
            name: jenkins
          subjects:
          - kind: ServiceAccount
            name: jenkins

          jenkins.yaml

          # jenkins
           
          ---
          apiVersion: apps/v1
          kind: StatefulSet
          metadata:
            name: jenkins
            labels:
              name: jenkins
          spec:
            selector:
              matchLabels:
                name: jenkins
            serviceName: jenkins
            replicas: 1
            updateStrategy:
              type: RollingUpdate
            template:
              metadata:
                name: jenkins
                labels:
                  name: jenkins
              spec:
                terminationGracePeriodSeconds: 10
                serviceAccountName: jenkins
                containers:
                  - name: jenkins
                    image: hub.baidubce.com/jpaas-public/jenkins-github:v0
                    imagePullPolicy: Always
                    ports:
                      - containerPort: 8080
                      - containerPort: 50000
                    resources:
                      limits:
                        cpu: 1
                        memory: 1Gi
                      requests:
                        cpu: 0.5
                        memory: 500Mi
                    env:
                      - name: LIMITS_MEMORY
                        valueFrom:
                          resourceFieldRef:
                            resource: limits.memory
                            divisor: 1Mi
                      - name: JAVA_OPTS
                        # value: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
                        value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
                    volumeMounts:
                      - name: jenkins-home
                        mountPath: /var/jenkins_home
                    livenessProbe:
                      httpGet:
                        path: /login
                        port: 8080
                      initialDelaySeconds: 60
                      timeoutSeconds: 5
                      failureThreshold: 12 # ~2 minutes
                    readinessProbe:
                      httpGet:
                        path: /login
                        port: 8080
                      initialDelaySeconds: 60
                      timeoutSeconds: 5
                      failureThreshold: 12 # ~2 minutes
                securityContext:
                  fsGroup: 1000
                volumes:
                  - name: jenkins-home
                    persistentVolumeClaim:
                      claimName: myjenkinspvc
           
          ---
          apiVersion: v1
          kind: Service
          metadata:
            name: jenkins
          spec:
            type: NodePort
            selector:
              name: jenkins
            # ensure the client ip is propagated to avoid the invalid crumb issue when using LoadBalancer (k8s >=1.7)
            #externalTrafficPolicy: Local
            ports:
              - name: http
                port: 80
                targetPort: 8080
                protocol: TCP
              - name: agent
                port: 50000
                protocol: TCP

          注意:

          • jenkins.yaml文件中的字段claimName需要改成1、设置Jenkins存储目录中生成的PVC名字

          CCE Kubernetes集群中执行以下命令

          kubectl create -f service-account.yaml
          kubectl create -f jenkins.yaml

          依次生成以下内容代表创建成功

          create.png

          3. 初始化配置 Jenkins

          此时Jenkins Master服务已经部署启动起来了,并且将端口暴露到80:30427,50000:31598,此时可以通过浏览器打开http://<Node_IP>:30427访问 Jenkins页面了。

          在浏览器上完成Jenkins的初始化插件安装过程,并配置管理员账户信息,这里忽略过程,初始化完成后界面如下: init.png

          注意:

          • 初始化过程中,让输入 /var/jenkins_home/secret/initialAdminPassword初始密码时,可以直接到挂载的PVC持久化目录里面读取,或者直接进入到容器的内部进行获取

          kubectl exec -it jenkins-0 cat /var/jenkins_home/secrets/initialAdminPassword

          4. Jenkins 安装Kubernetes Plugin 插件

          管理员账户登录Jenkins Master页面,点击 "系统管理" -> "插件管理" —> "可选插件" —> "Kubernetes" 勾选安装即可。 k8s-plugin.png

          安装完毕后,点击"系统管理" —> "系统设置" —> "新增一个云" —> 选择 "Kubernetes",然后填写KubernetesJenkins配置信息。

          add-cloud.png

          说明一下:

          • 1、Name处默认为kubernetes,也可以修改为其他名称,如果这里修改了,下边在执行Job时指定podTemplate()参数cloud为其对应名称,否则会找不到,cloud默认值取:kubernetes
          • 2、Kubernetes URL处填写了https://kubernetes.default这里填写了 Kubernetes Service对应的DNS记录,通过该DNS记录可以解析成该ServiceCluster IP

          注意: 也可以填写https://kubernetes.default.svc.cluster.local完整DNS记录,因为它要符合<svc_name>.<namespace_name>.svc.cluster.local的命名方式,或者直接填写外部Kubernetes的地址https://<ClusterIP>:<Ports>

          • 3、Jenkins URL处填写了http://jenkins.default,跟上边类似,也是使用 Jenkins Service对应的DNS记录, 同时也可以用http://<ClusterIP>:<Node_Port>方式,例如我这里可以填http://x.x.x.x:30427也是没有问题的,这里的30427就是对外暴露的NodePort
          • 4、配置完毕,可以点击"Test Connection"按钮测试是否能够连接的到 Kubernetes,如果显示Connection test successful则表示连接成功,配置没有问题

          5. 非集群内 Jenkins 连接 Kubernetes

          5.1. 填写 Kubernetes 配置内容

          kube-conf.png

          以一个kubeconfig文件为例

          apiVersion: v1
          clusters:
          - cluster:
              certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURwRENDQW95Z0F3SUJBZ0lVUldSdmNwRkxNaTVaUFZJUVllL2o2WkxsZlJJd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2FqRUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbAphVXBwYm1jeEREQUtCZ05WQkFvVEEyczRjekVVTUJJR0ExVUVDeE1MWTJ4dmRXUnVZWFJwZG1VeEV6QVJCZ05WCkJBTVRDbXQxWW1WeWJtVjBaWE13SGhjTk1qQXdOVEU0TURjeE5qQXdXaGNOTWpVd05URTNNRGN4TmpBd1dqQnEKTVFzd0NRWURWUVFHRXdKRFRqRVFNQTRHQTFVRUNCTUhRbVZwU21sdVp6RVFNQTRHQTFVRUJ4TUhRbVZwU21sdQpaekVNTUFvR0ExVUVDaE1EYXpoek1SUXdFZ1lEVlFRTEV3dGpiRzkxWkc1aGRHbDJaVEVUTUJFR0ExVUVBeE1LCmEzVmlaWEp1WlhSbGN6Q0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUtuNmNNMWYKQzlGMTFVVG1jVFlEQmZJMGlBUnJ2N3RtUEhDNU02NHBQNlViRVhIQ2lMNEFIUEVHK29tdVZKWHZ6MExhNVZRagp2SjZUblc0K2h4UjNuT2pGSmhGbldVZDcrWUtBM1Fic05qUVNybFNLTVhqRVlTSTA2M3NGS1YzNFZNUERXR3ByClVvVlZWZjduVmVkY0FhRTdzeFg3cXFOeERDS3Fjc2cxWCtDNFFrK01zaExKaUdyRnNnRC8rOHNUVkRzVzRoTEMKZ01zZ1R5WlVwRDlmM2hBTXQ0dzduV3RiWURsOFlvZnhEcU9tYndpVEx3VlNyaXAvZHlVeW9BZXhxbFFWaUgrMwpTNXY5cXRnTXVRRjhUMVNPUkwzcldsMUNYR0JPU3k0YTVBenA0dUx6TTlnbjJzdzZIU3gySG1aZlpxUlI1SXdICkFnQU9ydStFZ1lXVmNoOENBd0VBQWFOQ01FQXdEZ1lEVlIwUEFRSC9CQVFEQWdFR01BOEdBMVVkRXdFQi93UUYKTUFNQkFmOHdIUVlEVlIwT0JCWUVGSjBNOXlyc2F5SXEzZnVuYkhWWHJiZUZZL3RBTUEwR0NTcUdTSWIzRFFFQgpDd1VBQTRJQkFRQkE0UzBoVkpyOTBmNjNPU0tRMzVrUGNvTE9ObTg0TVNUbkI1OHE3alJHOHBtRHVvc09TUXUwCjZpUE9CZVBMOE92eEpObzhRcHBXOWJCR1N1ZFRKMG5CUnMyTmFjWU1BM0FqeUM0Y09nOFFJNGxHcUJWQllvVEcKa0I0cVlsYjl4dVJ2bnBJenVUWkd0RkVYQkpkZXFGZzZzSno2THRIZzFiUmM2cGJGZThuVUZZYnRSMzRPeGl2Qgp5WUNUKzNSbEpxTmhXcjlJK2djNDhMeXNheTBnL3BmQ1lsSzdXQm8rSWFZTXF1Z2Zmc2RQS2dISkU0Tm0xei9pCmlsNnhYMWV3R1RYMEFsMENKN2tNeitHa3hZalFHcHY1UG5iNzRpOHZZMm5uODVhUnliZVZjeENMZkhTazBTN2kKL0hjTXRkRjdxMVE5bndqMlhiclMxRmRCTEVSWGFGQTkKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
              server: https://1.1.1.1:6443
            name: kubernetes
          contexts:
          - context:
              cluster: kubernetes
              user: kubernetes-admin
            name: kubernetes-admin@kubernetes
          current-context: kubernetes-admin@kubernetes
          kind: Config
          preferences: {}
          users:
          - name: kubernetes-admin
            user:
              client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUsdafZJQ0FURS0tLS0tCUJBZ0lVQlRMK3lWNXdNbjNvamdkdzRKSmFBa0RpNTZrd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2FqRUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbAphVXBwYm1jeEREQUtCZ05WQkFvVEEyczRjekVVTUJJR0ExVUVDeE1MWTJ4dmRXUnVZWFJwZG1VeEV6QVJCZ05WCkJBTVRDbXQxWW1WeWJtVjBaWE13SGhjTk1qQXdOVEU0TURjeE5qQXdXaGNOTXpBd05URTJNRGN4TmpBd1dqQjcKTVFzd0NRWURWUVFHRXdKRFRqRVFNQTRHQTFVRUNCTUhRbVZwU21sdVp6RVFNQTRHQTFVRUJ4TUhRbVZwU21sdQpaekVYTUJVR0ExVUVDaE1PYzNsemRHVnRPbTFoYzNSbGNuTXhGREFTQmdOVkJBc1RDMk5zYjNWa2JtRjBhWFpsCk1Sa3dGd1lEVlFRREV4QnJkV0psY201bGRHVnpMV0ZrYldsdU1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0MKQVE4QU1JSUJDZ0tDQVFFQTErQjh2aXJGU0ZTQVJEQk4yOWxzRHN2WDlxaHJRTU82WnlCK0E5bWNybUtQZG5NOQozdU80T3FGQ1RjNktxTVNDQStDUHVKdlQwZXFJM29UUWZRMHZmWW40UndVNFhIZ21IbWQ2Nk50SXlQZUdzZGtGClRJc1FxaElzVm1VQS9ZNXBEb2h1Yk9OQWlGbDFOaEliUTNqU3JmMy9MYzVLY3RYM2ZOSGhxOE1hblpYYkVmRUsKY1NsdVI5Um4rQkdET1pubGNMZTkzRWRxME9EeFRsaXV6aFNYektUQUhyZnRLbGlIS3hyVXVkRlFZcGpHaFgrTgpLT2pRTFAzdkUwNjYzbklibGxZb2trYkxpZUVnd3lkcDNWUFpHSXI2aE5kZnd5bEdpU2lHMmw0N045bG1lVVhoCmt6MjZoWG5OeDZXRzdvL2d1aXBLdUpWeHZmT1NXS0tHWmJyZmhRSURBUUFCbzRISE1JSEVNQTRHQTFVZER3RUIKL3dRRUF3SUZvREFkQmdOVkhTVUVGakFVQmdnckJnRUZCUWNEQVFZSUt3WUJCUVVIQXdJd0RBWURWUjBUQVFILwpCQUl3QURBZEJnTlZIUTRFRmdRVVZUQm5ESjJYRUtCMkRRTlJZY2c5SFlsL2VCQXdId1lEVlIwakJCZ3dGb0FVCm5RejNLdXhySWlyZCs2ZHNkVmV0dDRWaiswQXdSUVlEVlIwUkJENHdQSWNFckJJQUFZY0VaRURtSUljRVpFYWsKeUljRVpFRG1INGNFWkVha3dZY0VaRURtSG9jRVpFYWt2b2NFWkVEbUlZY0VhZ3hkSm9jRVpFSUFiREFOQmdrcQpoa2lHOXcwQkFRc0ZBQU9DQVFFQUNFc2JBOWh6cHp6YWdwL2RMemQremx2anJVQ0dqNVFBQjNmQlN0OEN4Z2loCnhKNzBLeWV5TDdVVExOZDRNd09Tcjd0VGxhMTFmZTE4VkRvRUNDV0NzY0FabXZEK014eVJ2alFaL1NjY29xMGwKRkc5RFlBU3dOL2RhMURaeTY2L0FJdllaWEV1WmxQbGtSNnlrQ1F4MHJNYjJtdVdSemdJTS9MUTNwSFVYenJ4cQo3cUlreEtJeWlHZC8yUDdvcFBtSzMvemJIU2wyVnNnc1pFL2tJb1ZNQ3BYc1NEaml2aEpHNUlya1lyamNSa2JzCmV2cDI2b0M0K0lKNkNYNjh0TDROTkFxNzFoRmh1QTJ1TGhVLzFhc3NOY0w4RFpFNnFvTlFpWUZXQjVheE81NEsKZXlxRkM3dDBTbEVNYTFRVnNlbklwTVlTRmRlYTZselhtN3lXSFVCVEdBPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
              client-key-data:LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVkbjRSdlR3Nka0YKVVBL1k1cERvaHViT05BaUZsMU5oSWJRM2pTcmYzL0xjNUtjdFgzZk5IaHE4TWFuWlhiRWZFSwpjU2x1UjlSbitCR0RPWm5sY0xlOTNFZHEwT0R4VGxpdXpoU1h6S1RBSHJmdEtsaUhLeHJVdWRGUVlwakdoWCtOCktPalFMUDN2RTA2NjNuSWJsbFlva2tiTGllRWd3eWRwM1ZQWkdJcjZoTmRmd3lsR2lTaUcybDQ3TjlsbWVVWGgKa3oyNmhYbk54NldHN28vZ3VpcEt1SlZ4dmZPU1dLS0daYnJmaFFJREFRQUJBb0lCQUZRVUhlR2ZIT2xYNkFFbQo0eHd1YVZTMTllNGVtRzlJRERDZ1NoUkx6Q2RyUWI3N2tXeGZPdUN5Ny9VSDdaOWZzZGU2dlo5RUtkTEhTdm1ICnR3QU5nNktjZXZPR1IvWlQ0VnpVSnQzTWttT2JiSDJXTGVjcS9wbU9ySEFWdExZTW9rUkR4T1pwK0RkaXlERUEKQ2xoVUZaSW9yQnQyRGk3OXdQOS9heXFQdWEwSCttK0NpV2JSYzhodkRSZWtIZnd2b2pXTWtzNVNUNkZhSmVzNwpXQW1GNnVTaEs2Y3ovZjV5QjgvNjVmQ0tvYVN2RFlkdk1HVHZ2U3BqRlBoZVlScHRnREVWd2VJMElaRFBXdDQ4CmpnWklpQjVLbmpOdXprc1l6eEUvWnRhV05PS3QxVmtKYTduQ1M2SitOM0JESDFwOE9QVWk0ZVpDVTRzd05kRVMKVDRoV0ZZRUNnWUVBOGthR0Y3aGxmSEhBZkwya3MwNUJNb01mTktOUFFkSkxYV01ZUVk0L0FwYWJlaWhOVzZRSwpabDJ1cjNoRkFZZ3B6ZFp3RkUzbEx5VmhnODZSS05XNVZvSWcwUW5qWEpBc1U5VGt1KzNzVXRqWnFHTkNmdWRHClB1dG1qWFNwR1NvQWQxZDdwQkNkbEdMYWZBd1ArREI3VkFCNHRTQklVd1NHY3hDb2pUOU92T0VDZ1lFQTVCc2gKcnpOTHZJcU81MlpyL2xlZ3BWRC96aFhZWmNhaDhQcFFLNUY1VkJNa3MyalRNL25uVXdvYzl1QXllUlYxbm1lTApNajlXTmI2QkhZNXNRZXZaNUFPQ2pHWGxyL24zd29qSitTTVpyQnlLMVVqdjBVakdXcVlncHBXMUxhM21MREV5CmpGbDBhcnB5MllRSkxoK1VoU3R0c2ZKU25qdVVmTXJVR3pRUTh5VUNnWUIyRDJmSXAxTE5FYUY3Sis3YWNZZlQKMVpHZlZQV0tYYS9jRWkzL3hCRndjWFBTVTFGZkZ0RDZrU3hPMVl6SzhrOXN2dEpmRXBaY0l4c2gzOGRjM3NreQpIcmRmSmpKbEtOeHcvWTE1QnJmaXAwbHBoUFVpWWhFWkdCMGhVWGdWaXlJdkJiSjZnSjVKY09LSEVGbTMxK2hCClJ2bUxTZS8waElBQUVsNFFkb2tvQVFLQmdRQzd6Q3FiVjV3UENmUkZSdW02YU9KMXVJNGlXWkhqbVBsU3NJSzQKbS9oTDQ4YmZmbm9EM01jNmNxVU9DOThDR1V6UXNXYkVZNmpTYnBsV2dCOVkxcGg1UlBxQ0pKSkpvMzc3eGlxaQoxdWNYOEJmTktWTm45b1ozc3paR2NCTE9ITkhYcUZsNWUxeUJVaWVrTlRScHFNNWFKVHNXdWU2VEgzSk1tNkN0CkZOeXZrUUtCZ0VlY05hN2UzNTRJcEtlMGowZ0dzWXZuVXNFOVRKSkRzdmhjYmRhL0E5TEZvV2tRNGFHVmt2SlgKSG5yb29ZWDg2OU1FM2YyaHNxUmVsMng2a1phZWF6ZWNnWld0RzkvR2kzWXRqVWJZVktzU2pWZDVhSkFrM2k1cwpEVWRmMFZUU1QrR0hjRDUzL2NkK3M3WnA1WkFzdWx1czdnQmFPWkdzOUx6VXY5aHIrRzJWCi0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==
          • 序号1:保持不变既可,默认是kubernetes
          • 序号2:填写kubeconfig文件中的clusters.cluster.server的地址
          • 序号3: 获取kubeconfig文件中的certificate-authority-data的内容并转化成base64 encoded文件
          ecbo xxx | base64 -d > /opt/crt/ca.crt

          ca.crt的内容填写到jenkins kubernetesKubernetes服务证书key栏中获取kubeconfigclient-certificate-dataclient-key-data的内容并转化成base64 encoded文件

          echo xxxxx== | base64 -d > /opt/crt/client.crt
          echo xxxxx== | base64 -d > /opt/crt/client.key
           
           
          # 生成Client P12认证文件cert.pfx,并下载至本地
          openssl pkcs12 -export -out /opt/crt/cert.pfx -inkey /opt/crt/client.key -in /opt/crt/client.crt -certfile /opt/crt/ca.crt
          Enter Export Password:
          Verifying - Enter Export Password:
           
          # 注:自定义一个password并牢记
          • 序号4: 在云kubernetes中添加凭证

          cloud01.png

          注意: Upload certificate上次刚生成并下载至本地的cert.pfx文件,Password值添加生成cert.pfx文件时输入的密钥,然后在序号4中选择这个证书

          cloud2.png 最后点击连接测试:出现代表测试成功 cloud3.png

          6. 测试并验证

          好了,通过Kubernetes安装Jenkins Master完毕并且已经配置好了连接,接下来,我们可以配置 一个Job测试一下是否会成功发布

          6.1. pipeline 类型支持

          pipeline.png

          创建一个Pipeline类型Job并命名为my-k8s-jenkins-pipeline,然后Pipeline脚本处填写一个简单的测试脚本如下:

          def label = "mypod-${UUID.randomUUID().toString()}"
          podTemplate(label: label, cloud: 'kubernetes') {
              node(label) {
                  stage('Run shell') {
                      sh 'sleep 130s'
                      sh 'echo hello world.'
                  }
              }
          }

          执行构建,此时去构建队列里面,可以看到有一个构建任务,点击立即构建之后,等初始化成功之后,会成功发布,我们通过kubectl命令行,可以看到整个自动创建和删除过程。

          注意: 示例中使用的镜像,如果直接拉取会有超时的情况,可以使用以下命令在机器上提前执行

          docker pull hub.baidubce.com/jpaas-public/jenkins/jnlp-slave:v0
          
          
          docker tag hub.baidubce.com/jpaas-public/jenkins/jnlp-slave:v0 jenkins/jnlp-slave:4.0.1-1
          上一篇
          CCE容器网络模式选择
          下一篇
          CCE 支持周期、定时伸缩部署最佳实践