CCE基于nginx-ingress实现灰度发布
所有文档

          容器引擎 CCE

          CCE基于nginx-ingress实现灰度发布

          本文介绍如何利用百度云容器服务的Ingress功能,实现蓝绿发布。

          背景信息:

          灰度及蓝绿发布是为新版本创建一个与老版本完全一致的生产环境,在不影响老版本的前提下,按照一定的规则把部分流量切换到新版本,当新版本试运行一段时间没有问题后,将用户的全量流量从老版本迁移至新版本。

          其中A/B测试就是一种灰度发布方式,一部分用户继续使用老版本的服务,将一部分用户的流量切换到新版本,如果新版本运行稳定,则逐步将所有用户迁移到新版本。

          Ingress-Nginx Annotation 简介

          CCE 基于 Nginx Ingress Controller 实现了项目的网关,作为项目对外的流量入口和项目中各个服务的反向代理。而 Ingress-Nginx 支持配置 Ingress Annotations 来实现不同场景下的灰度发布和测试,可以满足金丝雀发布、蓝绿部署与 A/B 测试等业务场景。

          Nginx Annotations 支持以下 4 种 Canary 规则:

          • nginx.ingress.kubernetes.io/canary-by-header:基于 Request Header 的流量切分,适用于灰度发布以及 A/B 测试。当 Request Header 设置为 always时,请求将会被一直发送到 Canary 版本;当 Request Header 设置为 never时,请求不会被发送到 Canary 入口;对于任何其他 Header 值,将忽略 Header,并通过优先级将请求与其他金丝雀规则进行优先级的比较。
          • nginx.ingress.kubernetes.io/canary-by-header-value:要匹配的 Request Header 的值,用于通知 Ingress 将请求路由到 Canary Ingress 中指定的服务。当 Request Header 设置为此值时,它将被路由到 Canary 入口。该规则允许用户自定义 Request Header 的值,必须与上一个 annotation (即:canary-by-header)一起使用。
          • nginx.ingress.kubernetes.io/canary-weight:基于服务权重的流量切分,适用于蓝绿部署,权重范围 0 - 100 按百分比将请求路由到 Canary Ingress 中指定的服务。权重为 0 意味着该金丝雀规则不会向 Canary 入口的服务发送任何请求。权重为 100 意味着所有请求都将被发送到 Canary 入口。
          • nginx.ingress.kubernetes.io/canary-by-cookie:基于 Cookie 的流量切分,适用于灰度发布与 A/B 测试。用于通知 Ingress 将请求路由到 Canary Ingress 中指定的服务的cookie。当 cookie 值设置为 always时,它将被路由到 Canary 入口;当 cookie 值设置为 never时,请求不会被发送到 Canary 入口;对于任何其他值,将忽略 cookie 并将请求与其他金丝雀规则进行优先级的比较。

          canary规则优先级:canary-by-header - > canary-by-cookie - > canary-weight

          安装nginx-ingress-controller

          # yaml文件内容见附录
          kubectl apply -f ingress-nginx.yaml
          kubectl apply -f ingress-nginx-service.yaml

          部署production任务

          1、创建production应用资源

          kubectl apply -f production.yaml -n canary-demo
          apiVersion: apps/v1
          kind: Deployment
          metadata:
            name: production
          spec:
            replicas: 1
            selector:
              matchLabels:
                app: production
            template:
              metadata:
                labels:
                  app: production
              spec:
                containers:
                - name: production
                  image: hub.baidubce.com/jpaas-public/echoserver:1.10
                  ports:
                  - containerPort: 8080
                  env:
                    - name: NODE_NAME
                      valueFrom:
                        fieldRef:
                          fieldPath: spec.nodeName
                    - name: POD_NAME
                      valueFrom:
                        fieldRef:
                          fieldPath: metadata.name
                    - name: POD_NAMESPACE
                      valueFrom:
                        fieldRef:
                          fieldPath: metadata.namespace
                    - name: POD_IP
                      valueFrom:
                        fieldRef:
                          fieldPath: status.podIP
           
          ---
           
          apiVersion: v1
          kind: Service
          metadata:
            name: production
            labels:
              app: production
          spec:
            ports:
            - port: 80
              targetPort: 8080
              protocol: TCP
              name: http
            selector:
              app: production

          2、 创建 Production 版本的应用路由 (Ingress)

          kubectl apply -f production.ingress.yaml -n canary-demo
          apiVersion: extensions/v1beta1
          kind: Ingress
          metadata:
            name: production
            annotations:
              kubernetes.io/ingress.class: nginx
          spec:
            rules:
            - host: cce.canary.io
              http:
                paths:
                - backend:
                    serviceName: production
                    servicePort: 80

          3、本机访问应用:
          绑定hosts:vi /etc/hosts
          106.12.7.210 cce.canary.io
          image2020-3-9_15-7-30.png curl cce.canary.io 如下访问成功
          image2020-3-9_15-8-44.png

          创建canary版本任务

          1、创建canary版本应用资源

          kubectl apply -f canary.yaml -n canary-demo
          apiVersion: apps/v1
          kind: Deployment
          metadata:
            name: canary
          spec:
            replicas: 1
            selector:
              matchLabels:
                app: canary
            template:
              metadata:
                labels:
                  app: canary
              spec:
                containers:
                - name: canary
                  image: hub.baidubce.com/jpaas-public/echoserver:1.10
                  ports:
                  - containerPort: 8080
                  env:
                    - name: NODE_NAME
                      valueFrom:
                        fieldRef:
                          fieldPath: spec.nodeName
                    - name: POD_NAME
                      valueFrom:
                        fieldRef:
                          fieldPath: metadata.name
                    - name: POD_NAMESPACE
                      valueFrom:
                        fieldRef:
                          fieldPath: metadata.namespace
                    - name: POD_IP
                      valueFrom:
                        fieldRef:
                          fieldPath: status.podIP
           
          ---
           
          apiVersion: v1
          kind: Service
          metadata:
            name: canary
            labels:
              app: canary
          spec:
            ports:
            - port: 80
              targetPort: 8080
              protocol: TCP
              name: http
            selector:
              app: canary

          2、 基于权重创建 canary 版本的应用路由 (Ingress)

          kubectl apply -f canary.ingress.yaml -n canary-demo
          apiVersion: extensions/v1beta1
          kind: Ingress
          metadata:
            name: canary
            annotations:
              kubernetes.io/ingress.class: nginx
              nginx.ingress.kubernetes.io/canary: "true"
              nginx.ingress.kubernetes.io/canary-weight: "30"
          spec:
            rules:
            - host: cce.canary.io
              http:
                paths:
                - backend:
                    serviceName: canary
                    servicePort: 80

          3、访问应用域名验证

          for i in $(seq 1 10); do curl cce.canary.io | grep Hostname ; done

          如下图,流量部分打入canary

          应用的 Canary 版本基于权重 (30%) 进行流量切分后,访问到 Canary 版本的概率接近 30%,流量比例可能会有小范围的浮动,属正常现象
          image2020-3-9_15-23-19.png

          附录:ingress-nginx相关yaml文件

          • ingress-nginx.yaml
          apiVersion: v1
          kind: Namespace
          metadata:
            name: ingress-nginx
            labels:
              app.kubernetes.io/name: ingress-nginx
              app.kubernetes.io/part-of: ingress-nginx
           
          ---
           
          kind: ConfigMap
          apiVersion: v1
          metadata:
            name: nginx-configuration
            namespace: ingress-nginx
            labels:
              app.kubernetes.io/name: ingress-nginx
              app.kubernetes.io/part-of: ingress-nginx
           
          ---
          kind: ConfigMap
          apiVersion: v1
          metadata:
            name: tcp-services
            namespace: ingress-nginx
            labels:
              app.kubernetes.io/name: ingress-nginx
              app.kubernetes.io/part-of: ingress-nginx
           
          ---
          kind: ConfigMap
          apiVersion: v1
          metadata:
            name: udp-services
            namespace: ingress-nginx
            labels:
              app.kubernetes.io/name: ingress-nginx
              app.kubernetes.io/part-of: ingress-nginx
           
          ---
          apiVersion: v1
          kind: ServiceAccount
          metadata:
            name: nginx-ingress-serviceaccount
            namespace: ingress-nginx
            labels:
              app.kubernetes.io/name: ingress-nginx
              app.kubernetes.io/part-of: ingress-nginx
           
          ---
          apiVersion: rbac.authorization.k8s.io/v1beta1
          kind: ClusterRole
          metadata:
            name: nginx-ingress-clusterrole
            labels:
              app.kubernetes.io/name: ingress-nginx
              app.kubernetes.io/part-of: ingress-nginx
          rules:
            - apiGroups:
                - ""
              resources:
                - configmaps
                - endpoints
                - nodes
                - pods
                - secrets
              verbs:
                - list
                - watch
            - apiGroups:
                - ""
              resources:
                - nodes
              verbs:
                - get
            - apiGroups:
                - ""
              resources:
                - services
              verbs:
                - get
                - list
                - watch
            - apiGroups:
                - ""
              resources:
                - events
              verbs:
                - create
                - patch
            - apiGroups:
                - "extensions"
                - "networking.k8s.io"
              resources:
                - ingresses
              verbs:
                - get
                - list
                - watch
            - apiGroups:
                - "extensions"
                - "networking.k8s.io"
              resources:
                - ingresses/status
              verbs:
                - update
           
          ---
          apiVersion: rbac.authorization.k8s.io/v1beta1
          kind: Role
          metadata:
            name: nginx-ingress-role
            namespace: ingress-nginx
            labels:
              app.kubernetes.io/name: ingress-nginx
              app.kubernetes.io/part-of: ingress-nginx
          rules:
            - apiGroups:
                - ""
              resources:
                - configmaps
                - pods
                - secrets
                - namespaces
              verbs:
                - get
            - apiGroups:
                - ""
              resources:
                - configmaps
              resourceNames:
                # Defaults to "<election-id>-<ingress-class>"
                # Here: "<ingress-controller-leader>-<nginx>"
                # This has to be adapted if you change either parameter
                # when launching the nginx-ingress-controller.
                - "ingress-controller-leader-nginx"
              verbs:
                - get
                - update
            - apiGroups:
                - ""
              resources:
                - configmaps
              verbs:
                - create
            - apiGroups:
                - ""
              resources:
                - endpoints
              verbs:
                - get
           
          ---
          apiVersion: rbac.authorization.k8s.io/v1beta1
          kind: RoleBinding
          metadata:
            name: nginx-ingress-role-nisa-binding
            namespace: ingress-nginx
            labels:
              app.kubernetes.io/name: ingress-nginx
              app.kubernetes.io/part-of: ingress-nginx
          roleRef:
            apiGroup: rbac.authorization.k8s.io
            kind: Role
            name: nginx-ingress-role
          subjects:
            - kind: ServiceAccount
              name: nginx-ingress-serviceaccount
              namespace: ingress-nginx
           
          ---
          apiVersion: rbac.authorization.k8s.io/v1beta1
          kind: ClusterRoleBinding
          metadata:
            name: nginx-ingress-clusterrole-nisa-binding
            labels:
              app.kubernetes.io/name: ingress-nginx
              app.kubernetes.io/part-of: ingress-nginx
          roleRef:
            apiGroup: rbac.authorization.k8s.io
            kind: ClusterRole
            name: nginx-ingress-clusterrole
          subjects:
            - kind: ServiceAccount
              name: nginx-ingress-serviceaccount
              namespace: ingress-nginx
           
          ---
           
          apiVersion: apps/v1
          kind: Deployment
          metadata:
            name: nginx-ingress-controller
            namespace: ingress-nginx
            labels:
              app.kubernetes.io/name: ingress-nginx
              app.kubernetes.io/part-of: ingress-nginx
          spec:
            replicas: 1
            selector:
              matchLabels:
                app.kubernetes.io/name: ingress-nginx
                app.kubernetes.io/part-of: ingress-nginx
            template:
              metadata:
                labels:
                  app.kubernetes.io/name: ingress-nginx
                  app.kubernetes.io/part-of: ingress-nginx
                annotations:
                  prometheus.io/port: "10254"
                  prometheus.io/scrape: "true"
              spec:
                # wait up to five minutes for the drain of connections
                terminationGracePeriodSeconds: 300
                serviceAccountName: nginx-ingress-serviceaccount
                nodeSelector:
                  kubernetes.io/os: linux
                containers:
                  - name: nginx-ingress-controller
                    image: hub.baidubce.com/jpaas-public/nginx-ingress-controller:0.30.0
                    args:
                      - /nginx-ingress-controller
                      - --configmap=$(POD_NAMESPACE)/nginx-configuration
                      - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
                      - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
                      - --publish-service=$(POD_NAMESPACE)/ingress-nginx
                      - --annotations-prefix=nginx.ingress.kubernetes.io
                    securityContext:
                      allowPrivilegeEscalation: true
                      capabilities:
                        drop:
                          - ALL
                        add:
                          - NET_BIND_SERVICE
                      # www-data -> 101
                      runAsUser: 101
                    env:
                      - name: POD_NAME
                        valueFrom:
                          fieldRef:
                            fieldPath: metadata.name
                      - name: POD_NAMESPACE
                        valueFrom:
                          fieldRef:
                            fieldPath: metadata.namespace
                    ports:
                      - name: http
                        containerPort: 80
                        protocol: TCP
                      - name: https
                        containerPort: 443
                        protocol: TCP
                    livenessProbe:
                      failureThreshold: 3
                      httpGet:
                        path: /healthz
                        port: 10254
                        scheme: HTTP
                      initialDelaySeconds: 10
                      periodSeconds: 10
                      successThreshold: 1
                      timeoutSeconds: 10
                    readinessProbe:
                      failureThreshold: 3
                      httpGet:
                        path: /healthz
                        port: 10254
                        scheme: HTTP
                      periodSeconds: 10
                      successThreshold: 1
                      timeoutSeconds: 10
                    lifecycle:
                      preStop:
                        exec:
                          command:
                            - /wait-shutdown
           
          ---
           
          apiVersion: v1
          kind: LimitRange
          metadata:
            name: ingress-nginx
            namespace: ingress-nginx
            labels:
              app.kubernetes.io/name: ingress-nginx
              app.kubernetes.io/part-of: ingress-nginx
          spec:
            limits:
            - min:
                memory: 90Mi
                cpu: 100m
              type: Container
          • ingress-nginx-service.yaml
          kind: Service
          apiVersion: v1
          metadata:
            name: ingress-nginx
            namespace: ingress-nginx
            labels:
              app.kubernetes.io/name: ingress-nginx
              app.kubernetes.io/part-of: ingress-nginx
          spec:
            externalTrafficPolicy: Cluster
            type: LoadBalancer
            selector:
              app.kubernetes.io/name: ingress-nginx
              app.kubernetes.io/part-of: ingress-nginx
            ports:
              - name: http
                port: 80
                protocol: TCP
                targetPort: http
              - name: https
                port: 443
                protocol: TCP
                targetPort: https
          上一篇
          通过YAML创建CCE_Ingress
          下一篇
          网络管理