使用云盘CDS
所有文档

          容器引擎 CCE

          使用云盘CDS

          当前CDS磁盘以FlexVolumeCSI形式在集群中挂载。FlexVolume可以直接在Pod中挂载,也可以基于CSI创建PV和PVC存储资源后,再在Pod中挂载。两种挂载方案都可以满足容器内特定数据持久化的需求,两者详细的区别请参考kubernetes官方文档:

          准备工作

          用户做好以下准备,才能在容器中挂载cds实例。

          注意: 需要k8s版本>=1.11

          集群节点所处地域和可用区相同的CDS磁盘已创建

          CDS磁盘状态为未挂载

          创建容器集群

          1. 创建一个容器集群,操作步骤参考创建集群
          2. 下载命令行客户端kubectl,并连接集群,操作步骤参考[通过kubectl连接Kubernetes集群](CCE/操作指南/集群管理/通过 kubectl 连接集群.md)。

          操作指南

          通过Volume方式挂载

          注意:

          CDS同一时间只能挂载在一台bcc实例上。 CDS同一时间只能被一个Pod声明使用。 对于Deployment/ReplicaSet等资源,replicas域只能设置为1。 下面的示例 cds-example.yaml 将文件中声明的CDS磁盘挂载到Pod内my-nginx容器的/data路径下。

          apiVersion: apps/v1
          kind: Deployment
          metadata:
            name: my-nginx
          spec:
            replicas: 1
            selector:
              matchLabels:
                app: my-nginx
            template:
              metadata:
                name: my-nginx
                labels:
                  app: my-nginx
              spec:
                containers:
                - name: my-nginx
                  image: nginx
                  volumeMounts:
                  - name: cds-data
                    mountPath: /data
                  ports:
                  - containerPort: 80
                volumes:
                - name: cds-data
                  flexVolume:
                    driver: "baidubce/cds"
                    fsType: "ext4"
                    options:
                      volumeID: "v-hTXAGtin"

          flexVolume字段说明:

          fsType:CDS磁盘的文件系统,如果是新建的CDS磁盘,将会自动格式化该CDS磁盘为该文件系统 volumeID:CDS磁盘ID,您可以从百度智能云控制台的页面获取

          使用 kubectl create -f cds-example.yaml 创建相应的资源,相应的CDS磁盘将会被挂载到my-nginx容器内的/data目录下。Pod在集群内发生漂移时,该CDS磁盘将会跟随Pod一同迁移。

          PV/PVC

          通过PV/PVC方式使用CDS,具体分为两种方式:

          • 静态挂载:需要用户提前在百度智能云中创建好CDS(操作方法参考CDS文档),然后通过CDS的volume id在集群中创建PV和PVC资源。
          • 动态挂载:用户在集群中声明PVC时,自动创建出CDS磁盘并且动态关联至PV。

          静态挂载的方式需要用户提供的CDS磁盘处于未挂载状态,并且确保填入正确的volume id。动态挂载在部分场景下可能会更加方便,但是需要注意动态挂载时将会自动创建CDS磁盘并且计费,动态创建的CDS将会随着用户删除PVC时被联动删除(需要将对应storageClass的reclaimPolicy参数设置为delete),但是如果用户在没有删除PVC的情况下直接删除了CCE集群,则需要手动对这些CDS进行处理。

          静态PV/PVC方式挂载cds

          1.创建CdsPlugin、Attacher

          cds-template.yaml是一个yaml文件模板,包含了需要创建的集群资源信息,使用kubectl apply -f cds-template.yaml进行相关组件部署。

          1.11集群 cds-template.yaml文件内容如下(1.11集群使用):

          apiVersion: apiextensions.k8s.io/v1beta1
          kind: CustomResourceDefinition
          metadata:
            name: csinodeinfos.csi.storage.k8s.io
            labels:
              addonmanager.kubernetes.io/mode: Reconcile
          spec:
            group: csi.storage.k8s.io
            names:
              kind: CSINodeInfo
              plural: csinodeinfos
            scope: Cluster
            validation:
              openAPIV3Schema:
                properties:
                  spec:
                    description: Specification of CSINodeInfo
                    properties:
                      drivers:
                        description: List of CSI drivers running on the node and their specs.
                        type: array
                        items:
                          properties:
                            name:
                              description: The CSI driver that this object refers to.
                              type: string
                            nodeID:
                              description: The node from the driver point of view.
                              type: string
                            topologyKeys:
                              description: List of keys supported by the driver.
                              items:
                                type: string
                              type: array
                  status:
                    description: Status of CSINodeInfo
                    properties:
                      drivers:
                        description: List of CSI drivers running on the node and their statuses.
                        type: array
                        items:
                          properties:
                            name:
                              description: The CSI driver that this object refers to.
                              type: string
                            available:
                              description: Whether the CSI driver is installed.
                              type: boolean
                            volumePluginMechanism:
                              description: Indicates to external components the required mechanism
                                to use for any in-tree plugins replaced by this driver.
                              pattern: in-tree|csi
                              type: string
            version: v1alpha1
          ---
          apiVersion: v1
          kind: ServiceAccount
          metadata:
            name: csi-external-runner
            namespace: kube-system
          ---
          kind: ClusterRole
          apiVersion: rbac.authorization.k8s.io/v1
          metadata:
            name: external-runner
          rules:
            - apiGroups: [""]
              resources: ["events"]
              verbs: ["get", "list", "watch", "update", "create", "patch"]
            - apiGroups: [""]
              resources: ["persistentvolumes"]
              verbs: ["get", "list", "watch", "update", "create", "delete"]
            - apiGroups: [""]
              resources: ["persistentvolumeclaims"]
              verbs: ["get", "list", "watch", "update"]
            - apiGroups: [""]
              resources: ["nodes"]
              verbs: ["get", "list", "watch", "update"]
            - apiGroups: ["storage.k8s.io"]
              resources: ["volumeattachments"]
              verbs: ["get", "list", "watch", "update"]
            - apiGroups: [""]
              resources: ["namespaces"]
              verbs: ["get", "list"]
            - apiGroups: ["storage.k8s.io"]
              resources: ["storageclasses"]
              verbs: ["get", "list", "watch"]
          ---
          kind: ClusterRoleBinding
          apiVersion: rbac.authorization.k8s.io/v1
          metadata:
            name: csi-role-binding
          subjects:
            - kind: ServiceAccount
              name: csi-external-runner
              namespace: kube-system
          roleRef:
            kind: ClusterRole
            name: external-runner
            apiGroup: rbac.authorization.k8s.io
          ---
          kind: Role
          apiVersion: rbac.authorization.k8s.io/v1
          metadata:
            namespace: kube-system
            name: external-runner-cfg
          rules:
          - apiGroups: [""]
            resources: ["configmaps"]
            verbs: ["get", "watch", "list", "delete", "update", "create"]
          ---
          kind: RoleBinding
          apiVersion: rbac.authorization.k8s.io/v1
          metadata:
            name: csi-role-cfg
            namespace: kube-system
          subjects:
            - kind: ServiceAccount
              name: csi-external-runner
              namespace: kube-system
          roleRef:
            kind: Role
            name: external-runner-cfg
            apiGroup: rbac.authorization.k8s.io
          ---
          kind: DaemonSet
          apiVersion: apps/v1
          metadata:
            name: csi-cdsplugin
            namespace: kube-system
          spec:
            selector:
              matchLabels:
                app: csi-cdsplugin
            template:
              metadata:
                labels:
                  app: csi-cdsplugin
              spec:
                serviceAccount: csi-external-runner
                hostNetwork: true
                containers:
                  - name: driver-registrar
                    image: hub.baidubce.com/cce/driver-registrar:latest-1.11
                    args:
                      - "--v=5"
                      - "--csi-address=$(ADDRESS)"
                    env:
                      - name: ADDRESS
                        value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
                      - name: KUBE_NODE_NAME
                        valueFrom:
                          fieldRef:
                            fieldPath: spec.nodeName
                    imagePullPolicy: "Always"
                    volumeMounts:
                      - name: socket-dir
                        mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
                  - name: csi-cdsplugin
                    securityContext:
                      privileged: true
                      capabilities:
                        add: ["SYS_ADMIN"]
                      allowPrivilegeEscalation: true
                    image: hub.baidubce.com/cce/cdsplugin:latest-1.11
                    args :
                      - "--nodeid=$(NODE_ID)"
                      - "--endpoint=$(CSI_ENDPOINT)"
                      - "--v=5"
                      - "--drivername=csi-cdsplugin"
                    env:
                      - name: NODE_ID
                        valueFrom:
                          fieldRef:
                            fieldPath: spec.nodeName
                      - name: CSI_ENDPOINT
                        value: unix://var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
                    imagePullPolicy: "Always"
                    volumeMounts:
                      - name: cloud-config
                        mountPath: /etc/kubernetes
                      - name: plugin-dir
                        mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
                      - name: pods-mount-dir
                        mountPath: /var/lib/kubelet/pods
                        mountPropagation: "Bidirectional"
                      - name: pods-mount-dir-mnt
                        mountPath: /mnt/kubelet/pods
                        mountPropagation: "Bidirectional"
                      - name: pods-mount-dir-data
                        mountPath: /data/kubelet/pods
                        mountPropagation: "Bidirectional"
                      - mountPath: /sys
                        name: host-sys
                      - name: lib-modules
                        mountPath: /lib/modules
                        readOnly: true
                      - name: host-dev
                        mountPath: /dev
                volumes:
                  - name: cloud-config
                    hostPath:
                      path: /etc/kubernetes
                  - name: plugin-dir
                    hostPath:
                      path: /var/lib/kubelet/plugins/csi-cdsplugin
                      type: DirectoryOrCreate
                  - name: pods-mount-dir
                    hostPath:
                      path: /var/lib/kubelet/pods
                      type: DirectoryOrCreate
                  - name: pods-mount-dir-data
                    hostPath:
                      path: /data/kubelet/pods
                      type: DirectoryOrCreate
                  - name: pods-mount-dir-mnt
                    hostPath:
                      path: /mnt/kubelet/pods
                      type: DirectoryOrCreate
                  - name: socket-dir
                    hostPath:
                      path: /var/lib/kubelet/plugins/csi-cdsplugin
                      type: DirectoryOrCreate
                  - name: host-sys
                    hostPath:
                      path: /sys
                  - name: lib-modules
                    hostPath:
                      path: /lib/modules
                  - name: host-dev
                    hostPath:
                      path: /dev
          ---
          kind: Service
          apiVersion: v1
          metadata:
            name: csi-attacher-cds
            namespace: kube-system
            labels:
              app: csi-attacher-cds
          spec:
            selector:
              app: csi-attacher-cds
            clusterIP: None
          ---
          kind: StatefulSet
          apiVersion: apps/v1
          metadata:
            name: csi-attacher-cds
            namespace: kube-system
          spec:
            selector:
              matchLabels:
                app: csi-attacher-cds
            serviceName: "csi-attacher-cds"
            replicas: 2
            template:
              metadata:
                labels:
                  app: csi-attacher-cds
              spec:
                serviceAccount: csi-external-runner
                containers:
                  - name: csi-attacher-cds
                    image: hub.baidubce.com/cce/csi-attacher:latest-1.11
                    args:
                      - "--v=5"
                      - "--csi-address=$(ADDRESS)"
                      - "--leader-election"
                      - "--leader-election-namespace=$(MY_NAMESPACE)"
                      - "--leader-election-identity=$(MY_NAME)"
                    env:
                      - name: MY_NAME
                        valueFrom:
                          fieldRef:
                            fieldPath: metadata.name
                      - name: MY_NAMESPACE
                        valueFrom:
                          fieldRef:
                            fieldPath: metadata.namespace
                      - name: ADDRESS
                        value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
                    imagePullPolicy: "Always"
                    volumeMounts:
                      - name: socket-dir
                        mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
                volumes:
                  - name: socket-dir
                    hostPath:
                      path: /var/lib/kubelet/plugins/csi-cdsplugin
                      type: DirectoryOrCreate

          1.13集群: cds-template.yaml文件内容如下(1.13集群使用):

          apiVersion: apiextensions.k8s.io/v1beta1
          kind: CustomResourceDefinition
          metadata:
            name: csinodeinfos.csi.storage.k8s.io
            labels:
              addonmanager.kubernetes.io/mode: Reconcile
          spec:
            group: csi.storage.k8s.io
            names:
              kind: CSINodeInfo
              plural: csinodeinfos
            scope: Cluster
            validation:
              openAPIV3Schema:
                properties:
                  spec:
                    description: Specification of CSINodeInfo
                    properties:
                      drivers:
                        description: List of CSI drivers running on the node and their specs.
                        type: array
                        items:
                          properties:
                            name:
                              description: The CSI driver that this object refers to.
                              type: string
                            nodeID:
                              description: The node from the driver point of view.
                              type: string
                            topologyKeys:
                              description: List of keys supported by the driver.
                              items:
                                type: string
                              type: array
                  status:
                    description: Status of CSINodeInfo
                    properties:
                      drivers:
                        description: List of CSI drivers running on the node and their statuses.
                        type: array
                        items:
                          properties:
                            name:
                              description: The CSI driver that this object refers to.
                              type: string
                            available:
                              description: Whether the CSI driver is installed.
                              type: boolean
                            volumePluginMechanism:
                              description: Indicates to external components the required mechanism
                                to use for any in-tree plugins replaced by this driver.
                              pattern: in-tree|csi
                              type: string
            version: v1alpha1
          ---
          apiVersion: v1
          kind: ServiceAccount
          metadata:
            name: csi-external-runner
            namespace: kube-system
          ---
          kind: ClusterRole
          apiVersion: rbac.authorization.k8s.io/v1
          metadata:
            name: external-runner
          rules:
            - apiGroups: [""]
              resources: ["events"]
              verbs: ["get", "list", "watch", "update", "create", "patch"]
            - apiGroups: [""]
              resources: ["persistentvolumes"]
              verbs: ["get", "list", "watch", "update", "create", "delete"]
            - apiGroups: [""]
              resources: ["persistentvolumeclaims"]
              verbs: ["get", "list", "watch", "update"]
            - apiGroups: [""]
              resources: ["nodes"]
              verbs: ["get", "list", "watch", "update"]
            - apiGroups: ["storage.k8s.io"]
              resources: ["volumeattachments"]
              verbs: ["get", "list", "watch", "update"]
            - apiGroups: [""]
              resources: ["namespaces"]
              verbs: ["get", "list"]
            - apiGroups: ["storage.k8s.io"]
              resources: ["storageclasses"]
              verbs: ["get", "list", "watch"]
            - apiGroups: ["snapshot.storage.k8s.io"]
              resources: ["*"]
              verbs: ["create", "get", "list", "watch", "update", "patch"]
            - apiGroups: ["apiextensions.k8s.io"]
              resources: ["*"]
              verbs: ["create", "get", "list", "watch"]
            - apiGroups: ["csi.storage.k8s.io"]
              resources: ["csidrivers"]
              verbs: ["create", "delete", "get", "list", "watch"]
            - apiGroups: ["csi.storage.k8s.io"]
              resources: ["csinodeinfos"]
              verbs: ["get", "list", "watch"]
          ---
          kind: ClusterRoleBinding
          apiVersion: rbac.authorization.k8s.io/v1
          metadata:
            name: csi-role-binding
          subjects:
            - kind: ServiceAccount
              name: csi-external-runner
              namespace: kube-system
          roleRef:
            kind: ClusterRole
            name: external-runner
            apiGroup: rbac.authorization.k8s.io
          ---
          kind: Role
          apiVersion: rbac.authorization.k8s.io/v1
          metadata:
            namespace: kube-system
            name: external-runner-cfg
          rules:
          - apiGroups: [""]
            resources: ["configmaps"]
            verbs: ["get", "watch", "list", "delete", "update", "create"]
          ---
          kind: RoleBinding
          apiVersion: rbac.authorization.k8s.io/v1
          metadata:
            name: csi-role-cfg
            namespace: kube-system
          subjects:
            - kind: ServiceAccount
              name: csi-external-runner
              namespace: kube-system
          roleRef:
            kind: Role
            name: external-runner-cfg
            apiGroup: rbac.authorization.k8s.io
          ---
          apiVersion: apps/v1
          kind: DaemonSet
          metadata:
            name: csi-cdsplugin
            namespace: kube-system
          spec:
            selector:
              matchLabels:
                app: csi-cdsplugin
            template:
              metadata:
                labels:
                  app: csi-cdsplugin
              spec:
                containers:
                - args:
                  - --v=5
                  - --csi-address=$(ADDRESS)
                  - --kubelet-registration-path=$(ADDRESS)
                  lifecycle:
                    preStop:
                      exec:
                        command: ["/bin/sh", "-c", "rm -rf /registration/csi-cdsplugin /registration/csi-cdsplugin-reg.sock"]
                  env:
                  - name: ADDRESS
                    value: /var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
                  - name: KUBE_NODE_NAME
                    valueFrom:
                      fieldRef:
                        apiVersion: v1
                        fieldPath: spec.nodeName
                  image: hub.baidubce.com/cce/driver-registrar:latest-1.13
                  imagePullPolicy: Always
                  name: driver-registrar
                  resources: {}
                  terminationMessagePath: /dev/termination-log
                  terminationMessagePolicy: File
                  volumeMounts:
                  - mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
                    name: socket-dir
                  - mountPath: /registration
                    name: reg-dir
                - args:
                  - --nodeid=$(NODE_ID)
                  - --endpoint=$(CSI_ENDPOINT)
                  - --v=5
                  - --drivername=csi-cdsplugin
                  env:
                  - name: NODE_ID
                    valueFrom:
                      fieldRef:
                        apiVersion: v1
                        fieldPath: spec.nodeName
                  - name: CSI_ENDPOINT
                    value: unix://var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
                  image: hub.baidubce.com/cce/cdsplugin:latest-1.13
                  imagePullPolicy: Always
                  name: csi-cdsplugin
                  securityContext:
                    allowPrivilegeEscalation: true
                    capabilities:
                      add:
                      - SYS_ADMIN
                    privileged: true
                  volumeMounts:
                  - mountPath: /etc/kubernetes
                    name: cloud-config
                  - mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
                    name: socket-dir
                  - mountPath: /var/lib/kubelet/pods
                    mountPropagation: Bidirectional
                    name: pods-mount-dir
                  - mountPath: /data/kubelet/pods
                    mountPropagation: Bidirectional
                    name: pods-mount-dir-data
                  - name: pods-mount-dir-mnt
                    mountPath: /mnt/kubelet/pods
                    mountPropagation: "Bidirectional"
                  - mountPath: /sys
                    name: host-sys
                  - mountPath: /lib/modules
                    name: lib-modules
                    readOnly: true
                  - mountPath: /dev
                    name: host-dev
                hostNetwork: true
                restartPolicy: Always
                serviceAccount: csi-external-runner
                serviceAccountName: csi-external-runner
                priorityClassName: system-node-critical
                volumes:
                - hostPath:
                    path: /etc/kubernetes
                    type: ""
                  name: cloud-config
                - hostPath:
                    path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
                    type: DirectoryOrCreate
                  name: reg-dir
                - hostPath:
                    path: /var/lib/kubelet/pods
                    type: DirectoryOrCreate
                  name: pods-mount-dir
                - hostPath:
                    path: /data/kubelet/pods
                    type: DirectoryOrCreate
                  name: pods-mount-dir-data
                - name: pods-mount-dir-mnt
                  hostPath:
                    path: /mnt/kubelet/pods
                    type: DirectoryOrCreate
                - hostPath:
                    path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
                    type: DirectoryOrCreate
                  name: socket-dir
                - hostPath:
                    path: /sys
                    type: ""
                  name: host-sys
                - hostPath:
                    path: /lib/modules
                    type: ""
                  name: lib-modules
                - hostPath:
                    path: /dev
                    type: ""
                  name: host-dev
          ---
          kind: Service
          apiVersion: v1
          metadata:
            name: csi-attacher-cds
            namespace: kube-system
            labels:
              app: csi-attacher-cds
          spec:
            selector:
              app: csi-attacher-cds
            clusterIP: None
          ---
          kind: StatefulSet
          apiVersion: apps/v1
          metadata:
            name: csi-attacher-cds
            namespace: kube-system
          spec:
            serviceName: "csi-attacher-cds"
            replicas: 2
            selector:
              matchLabels:
                app: csi-attacher-cds
            template:
              metadata:
                labels:
                  app: csi-attacher-cds
              spec:
                priorityClassName: system-cluster-critical
                serviceAccount: csi-external-runner
                containers:
                  - name: csi-attacher-cds
                    image: hub.baidubce.com/cce/csi-attacher:latest-1.13
                    args:
                      - "--v=5"
                      - "--csi-address=$(ADDRESS)"
                      - "--leader-election"
                      - "--leader-election-namespace=$(MY_NAMESPACE)"
                      - "--leader-election-identity=$(MY_NAME)"
                    env:
                      - name: MY_NAME
                        valueFrom:
                          fieldRef:
                            fieldPath: metadata.name
                      - name: MY_NAMESPACE
                        valueFrom:
                          fieldRef:
                            fieldPath: metadata.namespace
                      - name: ADDRESS
                        value: /var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
                    imagePullPolicy: "Always"
                    volumeMounts:
                      - name: socket-dir
                        mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
                volumes:
                  - name: socket-dir
                    hostPath:
                      path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
                      type: DirectoryOrCreate

          1.16集群 cds-template.yaml文件内容如下(1.16集群使用):

          apiVersion: v1
          kind: ServiceAccount
          metadata:
            name: csi-external-runner
            namespace: kube-system
          ---
          kind: ClusterRole
          apiVersion: rbac.authorization.k8s.io/v1
          metadata:
            name: external-runner
          rules:
            - apiGroups: [""]
              resources: ["events"]
              verbs: ["get", "list", "watch", "update", "create", "patch"]
            - apiGroups: [""]
              resources: ["persistentvolumes"]
              verbs: ["get", "list", "watch", "update", "create", "delete", "patch"]
            - apiGroups: [""]
              resources: ["persistentvolumeclaims"]
              verbs: ["get", "list", "watch", "update"]
            - apiGroups: [""]
              resources: ["nodes"]
              verbs: ["get", "list", "watch", "update"]
            - apiGroups: ["storage.k8s.io"]
              resources: ["volumeattachments"]
              verbs: ["get", "list", "watch", "update", "patch"]
            - apiGroups: ["storage.k8s.io"]
              resources: ["volumeattachments/status"]
              verbs: ["patch"]
            - apiGroups: [""]
              resources: ["namespaces"]
              verbs: ["get", "list"]
            - apiGroups: ["storage.k8s.io"]
              resources: ["storageclasses"]
              verbs: ["get", "list", "watch"]
            - apiGroups: ["snapshot.storage.k8s.io"]
              resources: ["*"]
              verbs: ["create", "get", "list", "watch", "update", "patch"]
            - apiGroups: ["apiextensions.k8s.io"]
              resources: ["*"]
              verbs: ["create", "get", "list", "watch"]
            - apiGroups: ["csi.storage.k8s.io"]
              resources: ["csidrivers"]
              verbs: ["create", "delete", "get", "list", "watch"]
            - apiGroups: ["csi.storage.k8s.io"]
              resources: ["csinodeinfos"]
              verbs: ["get", "list", "watch"]
            - apiGroups: ["storage.k8s.io"]
              resources: ["csinodes"]
              verbs: ["get", "list", "watch"]
          ---
          kind: ClusterRoleBinding
          apiVersion: rbac.authorization.k8s.io/v1
          metadata:
            name: csi-role-binding
          subjects:
            - kind: ServiceAccount
              name: csi-external-runner
              namespace: kube-system
          roleRef:
            kind: ClusterRole
            name: external-runner
            apiGroup: rbac.authorization.k8s.io
          ---
          kind: Role
          apiVersion: rbac.authorization.k8s.io/v1
          metadata:
            namespace: kube-system
            name: external-runner-cfg
          rules:
          - apiGroups: [""]
            resources: ["configmaps"]
            verbs: ["get", "watch", "list", "delete", "update", "create"]
          - apiGroups: ["coordination.k8s.io"]
            resources: ["leases"]
            verbs: ["get", "watch", "list", "delete", "update", "create"]
          ---
          kind: RoleBinding
          apiVersion: rbac.authorization.k8s.io/v1
          metadata:
            name: csi-role-cfg
            namespace: kube-system
          subjects:
            - kind: ServiceAccount
              name: csi-external-runner
              namespace: kube-system
          roleRef:
            kind: Role
            name: external-runner-cfg
            apiGroup: rbac.authorization.k8s.io
          ---
          apiVersion: apps/v1
          kind: DaemonSet
          metadata:
            name: csi-cdsplugin
            namespace: kube-system
          spec:
            selector:
              matchLabels:
                app: csi-cdsplugin
            template:
              metadata:
                labels:
                  app: csi-cdsplugin
              spec:
                containers:
                - args:
                  - --v=6
                  - --csi-address=$(ADDRESS)
                  - --kubelet-registration-path=$(ADDRESS)
                  lifecycle:
                    preStop:
                      exec:
                        command: ["/bin/sh", "-c", "rm -rf /registration/csi-cdsplugin /registration/csi-cdsplugin-reg.sock"]
                  env:
                  - name: ADDRESS
                    value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
                  - name: KUBE_NODE_NAME
                    valueFrom:
                      fieldRef:
                        apiVersion: v1
                        fieldPath: spec.nodeName
                  image: hub.baidubce.com/cce/csi-node-driver-registrar:latest-1.16
                  imagePullPolicy: Always
                  name: node-driver-registrar
                  resources: {}
                  terminationMessagePath: /dev/termination-log
                  terminationMessagePolicy: File
                  volumeMounts:
                  - mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
                    name: socket-dir
                  - mountPath: /registration
                    name: reg-dir
                - args:
                  - --nodeid=$(NODE_ID)
                  - --endpoint=$(CSI_ENDPOINT)
                  - --v=5
                  - --drivername=csi-cdsplugin
                  env:
                  - name: NODE_ID
                    valueFrom:
                      fieldRef:
                        apiVersion: v1
                        fieldPath: spec.nodeName
                  - name: CSI_ENDPOINT
                    value: unix://var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
                  image: hub.baidubce.com/cce/cdsplugin:latest-1.16
                  imagePullPolicy: Always
                  name: csi-cdsplugin
                  securityContext:
                    allowPrivilegeEscalation: true
                    capabilities:
                      add:
                      - SYS_ADMIN
                    privileged: true
                  volumeMounts:
                  - mountPath: /etc/kubernetes
                    name: cloud-config
                  - mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
                    name: socket-dir
                  - mountPath: /var/lib/kubelet/pods
                    mountPropagation: Bidirectional
                    name: pods-mount-dir
                  - mountPath: /data/kubelet/pods
                    mountPropagation: Bidirectional
                    name: pods-mount-dir-data
                  - name: pods-mount-dir-mnt
                    mountPath: /mnt/kubelet/pods
                    mountPropagation: "Bidirectional"
                  - mountPath: /sys
                    name: host-sys
                  - mountPath: /lib/modules
                    name: lib-modules
                    readOnly: true
                  - mountPath: /dev
                    name: host-dev
                hostNetwork: true
                restartPolicy: Always
                serviceAccount: csi-external-runner
                serviceAccountName: csi-external-runner
                priorityClassName: system-node-critical
                volumes:
                - hostPath:
                    path: /etc/kubernetes
                    type: ""
                  name: cloud-config
                - hostPath:
                    path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
                    type: DirectoryOrCreate
                  name: reg-dir
                - hostPath:
                    path: /var/lib/kubelet/pods
                    type: DirectoryOrCreate
                  name: pods-mount-dir
                - hostPath:
                    path: /data/kubelet/pods
                    type: DirectoryOrCreate
                  name: pods-mount-dir-data
                - name: pods-mount-dir-mnt
                  hostPath:
                    path: /mnt/kubelet/pods
                    type: DirectoryOrCreate
                - hostPath:
                    path: /var/lib/kubelet/plugins/csi-cdsplugin
                    type: DirectoryOrCreate
                  name: socket-dir
                - hostPath:
                    path: /sys
                    type: ""
                  name: host-sys
                - hostPath:
                    path: /lib/modules
                    type: ""
                  name: lib-modules
                - hostPath:
                    path: /dev
                    type: ""
                  name: host-dev
          ---
          kind: Service
          apiVersion: v1
          metadata:
            name: csi-attacher-cds
            namespace: kube-system
            labels:
              app: csi-attacher-cds
          spec:
            selector:
              app: csi-attacher-cds
            clusterIP: None
          ---
          kind: StatefulSet
          apiVersion: apps/v1
          metadata:
            name: csi-attacher-cds
            namespace: kube-system
          spec:
            serviceName: "csi-attacher-cds"
            replicas: 1
            selector:
              matchLabels:
                app: csi-attacher-cds
            template:
              metadata:
                labels:
                  app: csi-attacher-cds
              spec:
                priorityClassName: system-cluster-critical
                serviceAccount: csi-external-runner
                containers:
                  - name: csi-attacher-cds
                    image: hub.baidubce.com/cce/csi-attacher:latest-1.16
                    args:
                      - "--v=5"
                      - "--csi-address=$(ADDRESS)"
                      - "--leader-election"
                    env:
                      - name: MY_NAME
                        valueFrom:
                          fieldRef:
                            fieldPath: metadata.name
                      - name: MY_NAMESPACE
                        valueFrom:
                          fieldRef:
                            fieldPath: metadata.namespace
                      - name: ADDRESS
                        value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
                    imagePullPolicy: "Always"
                    volumeMounts:
                      - name: socket-dir
                        mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
                volumes:
                  - name: socket-dir
                    hostPath:
                      path: /var/lib/kubelet/plugins/csi-cdsplugin
                      type: DirectoryOrCreate
          2.在集群中创建PV和PVC资源

          使用kubectl,执行 kubectl create -f pv-cds.yaml 完成PV的创建

          对应的pv-cds.yaml文件如下所示:

          apiVersion: v1
          kind: PersistentVolume
          metadata:
            name: pv-cds
            namespace: "default"
          spec:
            accessModes:
            - ReadWriteOnce
            capacity:
              storage: 5Gi
            csi:
              driver: "csi-cdsplugin"
              volumeHandle: "v-xxxx"
            persistentVolumeReclaimPolicy: Retain

          注意: yaml中volumeHandle字段对应的是cds volume id(短 id)

          创建PV后,输入kubectl get pv可以看见一个available状态的PV,如下所示:

          $ kubectl get pv
          NAME                   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                     STORAGECLASS   REASON    AGE
          pv-cds                 5Gi        RWO            Retain           Available                                                      22s

          建立一个能够与该PV绑定的PVC

          使用kubectl,执行 kubectl create -f pvc-cds.yaml完成PVC的创建

          对应的pvc-cds.yaml文件如下所示:

          apiVersion: v1
          kind: PersistentVolumeClaim
          metadata:
            name: csi-cds-pvc
          spec:
            accessModes:
            - ReadWriteOnce
            resources:
              requests:
                storage: 5Gi

          绑定前,PVC为pending状态

          $ kubectl get pvc
          NAME      STATUS    VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS   AGE
          cds-pvc   Pending                                                      2s  

          绑定后,PV和PVC状态变为Bound

          $ kubectl get pv
          NAME      CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                  STORAGECLASS   REASON    AGE
          pv-cds    5Gi        RWO            Retain           Bound     default/csi-cds-pvc                             4s
          $ kubectl get pvc
          NAME           STATUS    VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS   AGE
          csi-cds-pvc    Bound     pv-cds    5Gi        RWO                           36s

          有关PV和PVC的更多设置和字段说明,见k8s官方文档

          3.在Pod内挂载PVC

          在Pod spec内指定相应的PVC名称即可,使用kubectl,执行 kubectl create -f demo-cds-rc.yaml 完成rc的创建

          对应的demo-cds-rc.yaml文件如下所示:

          apiVersion: apps/v1
          kind: ReplicaSet
          metadata:
            name: web-server-rs
            labels:
              test: ws
          spec:
            replicas: 1
            selector:
              matchLabels:
                se: ws
            template:
              metadata:
                labels:
                  se: ws
                name: web-server
              spec:
                containers:
                 - name: web-server
                   image: nginx
                   volumeMounts:
                     - mountPath: /var/lib/www/html
                       name: csi-cds-pvc
                volumes:
                 - name: csi-cds-pvc
                   persistentVolumeClaim:
                     claimName: csi-cds-pvc

          Pod创建后,可以读写容器内的/cds-volume路径来访问相应的cds存储上的内容。

          由于创建PV和PVC时只支持accessModes为ReadWriteOnce,该PVC可以被一节点上的Pod挂载读写。

          4.释放PV和PVC资源

          完成存储资源的使用后,可以释放PVC和PV资源

          使用以下命令可以释放PVC

          $ kubectl delete -f  pvc-cds.yaml
          释放PVC后,原来与之绑定的PV状态会变为Release,如下所示:
          
          NAME      CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                  STORAGECLASS   REASON    AGE
          pv-cds    5Gi        RWO            Retain           Released   default/csi-cds-pvc                             4m

          输入以下指令释放PV资源

          $ kubectl delete -f pv-cds.yaml

          动态PV/PVC方式挂载cds

          1.创建StorageClass、Attacher、Provisioner

          dynamic-cds-template.yaml是一个yaml文件模板,包含了需要创建的集群资源信息。

          1.11集群: dynamic-cds-template.yaml文件内容如下(1.11集群):

          apiVersion: apiextensions.k8s.io/v1beta1
          kind: CustomResourceDefinition
          metadata:
            name: csinodeinfos.csi.storage.k8s.io
            labels:
              addonmanager.kubernetes.io/mode: Reconcile
          spec:
            group: csi.storage.k8s.io
            names:
              kind: CSINodeInfo
              plural: csinodeinfos
            scope: Cluster
            validation:
              openAPIV3Schema:
                properties:
                  spec:
                    description: Specification of CSINodeInfo
                    properties:
                      drivers:
                        description: List of CSI drivers running on the node and their specs.
                        type: array
                        items:
                          properties:
                            name:
                              description: The CSI driver that this object refers to.
                              type: string
                            nodeID:
                              description: The node from the driver point of view.
                              type: string
                            topologyKeys:
                              description: List of keys supported by the driver.
                              items:
                                type: string
                              type: array
                  status:
                    description: Status of CSINodeInfo
                    properties:
                      drivers:
                        description: List of CSI drivers running on the node and their statuses.
                        type: array
                        items:
                          properties:
                            name:
                              description: The CSI driver that this object refers to.
                              type: string
                            available:
                              description: Whether the CSI driver is installed.
                              type: boolean
                            volumePluginMechanism:
                              description: Indicates to external components the required mechanism
                                to use for any in-tree plugins replaced by this driver.
                              pattern: in-tree|csi
                              type: string
            version: v1alpha1
          ---
          apiVersion: v1
          kind: ServiceAccount
          metadata:
            name: csi-external-runner
            namespace: kube-system
          ---
          kind: ClusterRole
          apiVersion: rbac.authorization.k8s.io/v1
          metadata:
            name: external-runner
          rules:
            - apiGroups: [""]
              resources: ["events"]
              verbs: ["get", "list", "watch", "update", "create", "patch"]
            - apiGroups: [""]
              resources: ["persistentvolumes"]
              verbs: ["get", "list", "watch", "update", "create", "delete"]
            - apiGroups: [""]
              resources: ["persistentvolumeclaims"]
              verbs: ["get", "list", "watch", "update"]
            - apiGroups: [""]
              resources: ["nodes"]
              verbs: ["get", "list", "watch", "update"]
            - apiGroups: ["storage.k8s.io"]
              resources: ["volumeattachments"]
              verbs: ["get", "list", "watch", "update"]
            - apiGroups: [""]
              resources: ["namespaces"]
              verbs: ["get", "list"]
            - apiGroups: ["storage.k8s.io"]
              resources: ["storageclasses"]
              verbs: ["get", "list", "watch"]
          ---
          kind: ClusterRoleBinding
          apiVersion: rbac.authorization.k8s.io/v1
          metadata:
            name: csi-role-binding
          subjects:
            - kind: ServiceAccount
              name: csi-external-runner
              namespace: kube-system
          roleRef:
            kind: ClusterRole
            name: external-runner
            apiGroup: rbac.authorization.k8s.io
          ---
          kind: Role
          apiVersion: rbac.authorization.k8s.io/v1
          metadata:
            namespace: kube-system
            name: external-runner-cfg
          rules:
          - apiGroups: [""]
            resources: ["configmaps"]
            verbs: ["get", "watch", "list", "delete", "update", "create"]
          ---
          kind: RoleBinding
          apiVersion: rbac.authorization.k8s.io/v1
          metadata:
            name: csi-role-cfg
            namespace: kube-system
          subjects:
            - kind: ServiceAccount
              name: csi-external-runner
              namespace: kube-system
          roleRef:
            kind: Role
            name: external-runner-cfg
            apiGroup: rbac.authorization.k8s.io
          ---
          kind: DaemonSet
          apiVersion: apps/v1
          metadata:
            name: csi-cdsplugin
            namespace: kube-system
          spec:
            selector:
              matchLabels:
                app: csi-cdsplugin
            template:
              metadata:
                labels:
                  app: csi-cdsplugin
              spec:
                serviceAccount: csi-external-runner
                hostNetwork: true
                containers:
                  - name: driver-registrar
                    image: hub.baidubce.com/cce/driver-registrar:latest-1.11
                    args:
                      - "--v=5"
                      - "--csi-address=$(ADDRESS)"
                    env:
                      - name: ADDRESS
                        value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
                      - name: KUBE_NODE_NAME
                        valueFrom:
                          fieldRef:
                            fieldPath: spec.nodeName
                    imagePullPolicy: "Always"
                    volumeMounts:
                      - name: socket-dir
                        mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
                  - name: csi-cdsplugin
                    securityContext:
                      privileged: true
                      capabilities:
                        add: ["SYS_ADMIN"]
                      allowPrivilegeEscalation: true
                    image: hub.baidubce.com/cce/cdsplugin:latest-1.11
                    args :
                      - "--nodeid=$(NODE_ID)"
                      - "--endpoint=$(CSI_ENDPOINT)"
                      - "--v=5"
                      - "--drivername=csi-cdsplugin"
                    env:
                      - name: NODE_ID
                        valueFrom:
                          fieldRef:
                            fieldPath: spec.nodeName
                      - name: CSI_ENDPOINT
                        value: unix://var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
                    imagePullPolicy: "Always"
                    volumeMounts:
                      - name: cloud-config
                        mountPath: /etc/kubernetes
                      - name: plugin-dir
                        mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
                      - name: pods-mount-dir
                        mountPath: /var/lib/kubelet/pods
                        mountPropagation: "Bidirectional"
                      - name: pods-mount-dir-data
                        mountPath: /data/kubelet/pods
                        mountPropagation: "Bidirectional"
                      - mountPath: /sys
                        name: host-sys
                      - name: lib-modules
                        mountPath: /lib/modules
                        readOnly: true
                      - name: host-dev
                        mountPath: /dev
                volumes:
                  - name: cloud-config
                    hostPath:
                      path: /etc/kubernetes
                  - name: plugin-dir
                    hostPath:
                      path: /var/lib/kubelet/plugins/csi-cdsplugin
                      type: DirectoryOrCreate
                  - name: pods-mount-dir
                    hostPath:
                      path: /var/lib/kubelet/pods
                      type: DirectoryOrCreate
                  - name: pods-mount-dir-data
                    hostPath:
                      path: /data/kubelet/pods
                      type: DirectoryOrCreate
                  - name: socket-dir
                    hostPath:
                      path: /var/lib/kubelet/plugins/csi-cdsplugin
                      type: DirectoryOrCreate
                  - name: host-sys
                    hostPath:
                      path: /sys
                  - name: lib-modules
                    hostPath:
                      path: /lib/modules
                  - name: host-dev
                    hostPath:
                      path: /dev
          ---
          kind: Service
          apiVersion: v1
          metadata:
            name: csi-attacher-cds
            namespace: kube-system
            labels:
              app: csi-attacher-cds
          spec:
            selector:
              app: csi-attacher-cds
            clusterIP: None
          ---
          kind: StatefulSet
          apiVersion: apps/v1
          metadata:
            name: csi-attacher-cds
            namespace: kube-system
          spec:
            serviceName: "csi-attacher-cds"
            replicas: 2
            selector:
              matchLabels:
                app: csi-attacher-cds
            template:
              metadata:
                labels:
                  app: csi-attacher-cds
              spec:
                serviceAccount: csi-external-runner
                containers:
                  - name: csi-attacher-cds
                    image: hub.baidubce.com/cce/csi-attacher:latest-1.11
                    args:
                      - "--v=5"
                      - "--csi-address=$(ADDRESS)"
                      - "--leader-election"
                      - "--leader-election-namespace=$(MY_NAMESPACE)"
                      - "--leader-election-identity=$(MY_NAME)"
                    env:
                      - name: MY_NAME
                        valueFrom:
                          fieldRef:
                            fieldPath: metadata.name
                      - name: MY_NAMESPACE
                        valueFrom:
                          fieldRef:
                            fieldPath: metadata.namespace
                      - name: ADDRESS
                        value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
                    imagePullPolicy: "Always"
                    volumeMounts:
                      - name: socket-dir
                        mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
                volumes:
                  - name: socket-dir
                    hostPath:
                      path: /var/lib/kubelet/plugins/csi-cdsplugin
                      type: DirectoryOrCreate
          
          ---
          
          kind: Service
          apiVersion: v1
          metadata:
            name: csi-provisioner-cds
            namespace: kube-system
            labels:
              app: csi-provisioner-cds
          spec:
            selector:
              app: csi-provisioner-cds
            clusterIP: None
          
          ---
          kind: StatefulSet
          apiVersion: apps/v1
          metadata:
            name: csi-provisioner-cds
            namespace: kube-system
          spec:
            serviceName: "csi-provisioner-cds"
            replicas: 1
            selector:
              matchLabels:
                app: csi-provisioner-cds
            template:
              metadata:
                labels:
                  app: csi-provisioner-cds
              spec:
                serviceAccount: csi-external-runner
                containers:
                  - name: csi-provisioner-cds
                    image: hub.baidubce.com/cce/csi-provisioner:latest-1.11
                    args:
                      - "--provisioner=csi-cdsplugin"
                      - "--csi-address=$(ADDRESS)"
                      - "--v=5"
                    env:
                      - name: ADDRESS
                        value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
                    imagePullPolicy: "IfNotPresent"
                    volumeMounts:
                      - name: socket-dir
                        mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
                volumes:
                  - name: socket-dir
                    hostPath:
                      path: /var/lib/kubelet/plugins/csi-cdsplugin
                      type: DirectoryOrCreate
          
          ---
          apiVersion: storage.k8s.io/v1
          kind: StorageClass
          metadata:
            name: hp1               #名字可以自定义
          provisioner: csi-cdsplugin
          parameters:
            dynamicVolume: "true"       #需要为 true才会创建
            cdsSizeInGB: "40"           #对单个 storageClass能创建的 size限制
            paymentTiming: "Postpaid"   #Prepaid or Postpaid
            storageType: "hp1"          #支持std1 hp1 ssd
            reservationLength: "3"      #Prepaid 模式下需要填写
            zone: "xxx"                  #参数可选
          reclaimPolicy: Delete         #支持 Delete、Retain 默认值为 Delete

          1.13集群: dynamic-cds-template.yaml文件内容如下(1.13集群):

          apiVersion: apiextensions.k8s.io/v1beta1
          kind: CustomResourceDefinition
          metadata:
            name: csinodeinfos.csi.storage.k8s.io
            labels:
              addonmanager.kubernetes.io/mode: Reconcile
          spec:
            group: csi.storage.k8s.io
            names:
              kind: CSINodeInfo
              plural: csinodeinfos
            scope: Cluster
            validation:
              openAPIV3Schema:
                properties:
                  spec:
                    description: Specification of CSINodeInfo
                    properties:
                      drivers:
                        description: List of CSI drivers running on the node and their specs.
                        type: array
                        items:
                          properties:
                            name:
                              description: The CSI driver that this object refers to.
                              type: string
                            nodeID:
                              description: The node from the driver point of view.
                              type: string
                            topologyKeys:
                              description: List of keys supported by the driver.
                              items:
                                type: string
                              type: array
                  status:
                    description: Status of CSINodeInfo
                    properties:
                      drivers:
                        description: List of CSI drivers running on the node and their statuses.
                        type: array
                        items:
                          properties:
                            name:
                              description: The CSI driver that this object refers to.
                              type: string
                            available:
                              description: Whether the CSI driver is installed.
                              type: boolean
                            volumePluginMechanism:
                              description: Indicates to external components the required mechanism
                                to use for any in-tree plugins replaced by this driver.
                              pattern: in-tree|csi
                              type: string
            version: v1alpha1
          ---
          apiVersion: v1
          kind: ServiceAccount
          metadata:
            name: csi-external-runner
            namespace: kube-system
          ---
          kind: ClusterRole
          apiVersion: rbac.authorization.k8s.io/v1
          metadata:
            name: external-runner
          rules:
            - apiGroups: [""]
              resources: ["events"]
              verbs: ["get", "list", "watch", "update", "create", "patch"]
            - apiGroups: [""]
              resources: ["persistentvolumes"]
              verbs: ["get", "list", "watch", "update", "create", "delete"]
            - apiGroups: [""]
              resources: ["persistentvolumeclaims"]
              verbs: ["get", "list", "watch", "update"]
            - apiGroups: [""]
              resources: ["nodes"]
              verbs: ["get", "list", "watch", "update"]
            - apiGroups: ["storage.k8s.io"]
              resources: ["volumeattachments"]
              verbs: ["get", "list", "watch", "update"]
            - apiGroups: [""]
              resources: ["namespaces"]
              verbs: ["get", "list"]
            - apiGroups: ["storage.k8s.io"]
              resources: ["storageclasses"]
              verbs: ["get", "list", "watch"]
            - apiGroups: ["snapshot.storage.k8s.io"]
              resources: ["*"]
              verbs: ["create", "get", "list", "watch", "update", "patch"]
            - apiGroups: ["apiextensions.k8s.io"]
              resources: ["*"]
              verbs: ["create", "get", "list", "watch"]
            - apiGroups: ["csi.storage.k8s.io"]
              resources: ["csidrivers"]
              verbs: ["create", "delete", "get", "list", "watch"]
            - apiGroups: ["csi.storage.k8s.io"]
              resources: ["csinodeinfos"]
              verbs: ["get", "list", "watch"]
          ---
          kind: ClusterRoleBinding
          apiVersion: rbac.authorization.k8s.io/v1
          metadata:
            name: csi-role-binding
          subjects:
            - kind: ServiceAccount
              name: csi-external-runner
              namespace: kube-system
          roleRef:
            kind: ClusterRole
            name: external-runner
            apiGroup: rbac.authorization.k8s.io
          ---
          kind: Role
          apiVersion: rbac.authorization.k8s.io/v1
          metadata:
            namespace: kube-system
            name: external-runner-cfg
          rules:
          - apiGroups: [""]
            resources: ["configmaps"]
            verbs: ["get", "watch", "list", "delete", "update", "create"]
          ---
          kind: RoleBinding
          apiVersion: rbac.authorization.k8s.io/v1
          metadata:
            name: csi-role-cfg
            namespace: kube-system
          subjects:
            - kind: ServiceAccount
              name: csi-external-runner
              namespace: kube-system
          roleRef:
            kind: Role
            name: external-runner-cfg
            apiGroup: rbac.authorization.k8s.io
          ---
          apiVersion: apps/v1
          kind: DaemonSet
          metadata:
            name: csi-cdsplugin
            namespace: kube-system
          spec:
            selector:
              matchLabels:
                app: csi-cdsplugin
            template:
              metadata:
                labels:
                  app: csi-cdsplugin
              spec:
                containers:
                - args:
                  - --v=5
                  - --csi-address=$(ADDRESS)
                  - --kubelet-registration-path=$(ADDRESS)
                  lifecycle:
                    preStop:
                      exec:
                        command: ["/bin/sh", "-c", "rm -rf /registration/csi-cdsplugin /registration/csi-cdsplugin-reg.sock"]
                  env:
                  - name: ADDRESS
                    value: /var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
                  - name: KUBE_NODE_NAME
                    valueFrom:
                      fieldRef:
                        apiVersion: v1
                        fieldPath: spec.nodeName
                  image: hub.baidubce.com/cce/driver-registrar:latest-1.13
                  imagePullPolicy: Always
                  name: driver-registrar
                  resources: {}
                  terminationMessagePath: /dev/termination-log
                  terminationMessagePolicy: File
                  volumeMounts:
                  - mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
                    name: socket-dir
                  - mountPath: /registration
                    name: reg-dir
                - args:
                  - --nodeid=$(NODE_ID)
                  - --endpoint=$(CSI_ENDPOINT)
                  - --v=5
                  - --drivername=csi-cdsplugin
                  env:
                  - name: NODE_ID
                    valueFrom:
                      fieldRef:
                        apiVersion: v1
                        fieldPath: spec.nodeName
                  - name: CSI_ENDPOINT
                    value: unix://var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
                  image: hub.baidubce.com/cce/cdsplugin:latest-1.13
                  imagePullPolicy: Always
                  name: csi-cdsplugin
                  securityContext:
                    allowPrivilegeEscalation: true
                    capabilities:
                      add:
                      - SYS_ADMIN
                    privileged: true
                  volumeMounts:
                  - mountPath: /etc/kubernetes
                    name: cloud-config
                  - mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
                    name: socket-dir
                  - mountPath: /var/lib/kubelet/pods
                    mountPropagation: Bidirectional
                    name: pods-mount-dir
                  - mountPath: /data/kubelet/pods
                    mountPropagation: Bidirectional
                    name: pods-mount-dir-data
                  - mountPath: /sys
                    name: host-sys
                  - mountPath: /lib/modules
                    name: lib-modules
                    readOnly: true
                  - mountPath: /dev
                    name: host-dev
                hostNetwork: true
                restartPolicy: Always
                serviceAccount: csi-external-runner
                serviceAccountName: csi-external-runner
                priorityClassName: system-node-critical
                volumes:
                - hostPath:
                    path: /etc/kubernetes
                    type: ""
                  name: cloud-config
                - hostPath:
                    path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
                    type: DirectoryOrCreate
                  name: reg-dir
                - hostPath:
                    path: /var/lib/kubelet/pods
                    type: DirectoryOrCreate
                  name: pods-mount-dir
                - hostPath:
                    path: /data/kubelet/pods
                    type: DirectoryOrCreate
                  name: pods-mount-dir-data
                - hostPath:
                    path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
                    type: DirectoryOrCreate
                  name: socket-dir
                - hostPath:
                    path: /sys
                    type: ""
                  name: host-sys
                - hostPath:
                    path: /lib/modules
                    type: ""
                  name: lib-modules
                - hostPath:
                    path: /dev
                    type: ""
                  name: host-dev
          ---
          kind: Service
          apiVersion: v1
          metadata:
            name: csi-attacher-cds
            namespace: kube-system
            labels:
              app: csi-attacher-cds
          spec:
            selector:
              app: csi-attacher-cds
            clusterIP: None
          ---
          kind: StatefulSet
          apiVersion: apps/v1
          metadata:
            name: csi-attacher-cds
            namespace: kube-system
          spec:
            serviceName: "csi-attacher-cds"
            replicas: 2
            selector:
              matchLabels:
                app: csi-attacher-cds
            template:
              metadata:
                labels:
                  app: csi-attacher-cds
              spec:
                priorityClassName: system-cluster-critical
                serviceAccount: csi-external-runner
                containers:
                  - name: csi-attacher-cds
                    image: hub.baidubce.com/cce/csi-attacher:latest-1.13
                    args:
                      - "--v=5"
                      - "--csi-address=$(ADDRESS)"
                      - "--leader-election"
                      - "--leader-election-namespace=$(MY_NAMESPACE)"
                      - "--leader-election-identity=$(MY_NAME)"
                    env:
                      - name: MY_NAME
                        valueFrom:
                          fieldRef:
                            fieldPath: metadata.name
                      - name: MY_NAMESPACE
                        valueFrom:
                          fieldRef:
                            fieldPath: metadata.namespace
                      - name: ADDRESS
                        value: /var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
                    imagePullPolicy: "Always"
                    volumeMounts:
                      - name: socket-dir
                        mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
                volumes:
                  - name: socket-dir
                    hostPath:
                      path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
                      type: DirectoryOrCreate
          ---
          kind: Service
          apiVersion: v1
          metadata:
            name: csi-provisioner-cds
            namespace: kube-system
            labels:
              app: csi-provisioner-cds
          spec:
            selector:
              app: csi-provisioner-cds
            clusterIP: None
          ---
          kind: StatefulSet
          apiVersion: apps/v1
          metadata:
            name: csi-provisioner-cds
            namespace: kube-system
          spec:
            serviceName: "csi-provisioner-cds"
            replicas: 1
            selector:
              matchLabels:
                app: csi-provisioner-cds
            template:
              metadata:
                labels:
                  app: csi-provisioner-cds
              spec:
                priorityClassName: system-cluster-critical
                serviceAccount: csi-external-runner
                containers:
                  - name: csi-provisioner-cds
                    image: hub.baidubce.com/cce/csi-provisioner:latest-1.13
                    args:
                      - "--provisioner=csi-cdsplugin"
                      - "--csi-address=$(ADDRESS)"
                      - "--v=5"
                      - "--feature-gates=Topology=true"
                      - "--volume-name-prefix=pv"
                    env:
                      - name: ADDRESS
                        value: /var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
                    imagePullPolicy: "Always"
                    volumeMounts:
                      - name: socket-dir
                        mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
                volumes:
                  - name: socket-dir
                    hostPath:
                      path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
                      type: DirectoryOrCreate
          ---
          apiVersion: storage.k8s.io/v1
          kind: StorageClass
          metadata:
            name: hp1               #名字可以自定义
          provisioner: csi-cdsplugin
          parameters:
            dynamicVolume: "true"       #需要为 true才会创建
            cdsSizeInGB: "40"           #对单个 storageClass能创建的 size限制
            paymentTiming: "Postpaid"   #Prepaid or Postpaid
            storageType: "hp1"          #支持std1 hp1 ssd
            reservationLength: "3"      #Prepaid 模式下需要填写
            zone: "xxx"                  #参数可选
          reclaimPolicy: Delete         #支持 Delete、Retain 默认值为 Delete

          1.16集群: dynamic-cds-template.yaml文件内容如下(1.16集群):

          apiVersion: v1
          kind: ServiceAccount
          metadata:
            name: csi-external-runner
            namespace: kube-system
          ---
          kind: ClusterRole
          apiVersion: rbac.authorization.k8s.io/v1
          metadata:
            name: external-runner
          rules:
            - apiGroups: [""]
              resources: ["events"]
              verbs: ["get", "list", "watch", "update", "create", "patch"]
            - apiGroups: [""]
              resources: ["persistentvolumes"]
              verbs: ["get", "list", "watch", "update", "create", "delete", "patch"]
            - apiGroups: [""]
              resources: ["persistentvolumeclaims"]
              verbs: ["get", "list", "watch", "update"]
            - apiGroups: [""]
              resources: ["nodes"]
              verbs: ["get", "list", "watch", "update"]
            - apiGroups: ["storage.k8s.io"]
              resources: ["volumeattachments"]
              verbs: ["get", "list", "watch", "update", "patch"]
            - apiGroups: ["storage.k8s.io"]
              resources: ["volumeattachments/status"]
              verbs: ["patch"]
            - apiGroups: [""]
              resources: ["namespaces"]
              verbs: ["get", "list"]
            - apiGroups: ["storage.k8s.io"]
              resources: ["storageclasses"]
              verbs: ["get", "list", "watch"]
            - apiGroups: ["snapshot.storage.k8s.io"]
              resources: ["*"]
              verbs: ["create", "get", "list", "watch", "update", "patch"]
            - apiGroups: ["apiextensions.k8s.io"]
              resources: ["*"]
              verbs: ["create", "get", "list", "watch"]
            - apiGroups: ["csi.storage.k8s.io"]
              resources: ["csidrivers"]
              verbs: ["create", "delete", "get", "list", "watch"]
            - apiGroups: ["csi.storage.k8s.io"]
              resources: ["csinodeinfos"]
              verbs: ["get", "list", "watch"]
            - apiGroups: ["storage.k8s.io"]
              resources: ["csinodes"]
              verbs: ["get", "list", "watch"]
          ---
          kind: ClusterRoleBinding
          apiVersion: rbac.authorization.k8s.io/v1
          metadata:
            name: csi-role-binding
          subjects:
            - kind: ServiceAccount
              name: csi-external-runner
              namespace: kube-system
          roleRef:
            kind: ClusterRole
            name: external-runner
            apiGroup: rbac.authorization.k8s.io
          ---
          kind: Role
          apiVersion: rbac.authorization.k8s.io/v1
          metadata:
            namespace: kube-system
            name: external-runner-cfg
          rules:
          - apiGroups: [""]
            resources: ["configmaps"]
            verbs: ["get", "watch", "list", "delete", "update", "create"]
          - apiGroups: ["coordination.k8s.io"]
            resources: ["leases"]
            verbs: ["get", "watch", "list", "delete", "update", "create"]
          ---
          kind: RoleBinding
          apiVersion: rbac.authorization.k8s.io/v1
          metadata:
            name: csi-role-cfg
            namespace: kube-system
          subjects:
            - kind: ServiceAccount
              name: csi-external-runner
              namespace: kube-system
          roleRef:
            kind: Role
            name: external-runner-cfg
            apiGroup: rbac.authorization.k8s.io
          ---
          apiVersion: apps/v1
          kind: DaemonSet
          metadata:
            name: csi-cdsplugin
            namespace: kube-system
          spec:
            selector:
              matchLabels:
                app: csi-cdsplugin
            template:
              metadata:
                labels:
                  app: csi-cdsplugin
              spec:
                containers:
                - args:
                  - --v=6
                  - --csi-address=$(ADDRESS)
                  - --kubelet-registration-path=$(ADDRESS)
                  lifecycle:
                    preStop:
                      exec:
                        command: ["/bin/sh", "-c", "rm -rf /registration/csi-cdsplugin /registration/csi-cdsplugin-reg.sock"]
                  env:
                  - name: ADDRESS
                    value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
                  - name: KUBE_NODE_NAME
                    valueFrom:
                      fieldRef:
                        apiVersion: v1
                        fieldPath: spec.nodeName
                  image: hub.baidubce.com/cce/csi-node-driver-registrar:latest-1.16
                  imagePullPolicy: Always
                  name: node-driver-registrar
                  resources: {}
                  terminationMessagePath: /dev/termination-log
                  terminationMessagePolicy: File
                  volumeMounts:
                  - mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
                    name: socket-dir
                  - mountPath: /registration
                    name: reg-dir
                - args:
                  - --nodeid=$(NODE_ID)
                  - --endpoint=$(CSI_ENDPOINT)
                  - --v=5
                  - --drivername=csi-cdsplugin
                  env:
                  - name: NODE_ID
                    valueFrom:
                      fieldRef:
                        apiVersion: v1
                        fieldPath: spec.nodeName
                  - name: CSI_ENDPOINT
                    value: unix://var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
                  image: hub.baidubce.com/cce/cdsplugin:latest-1.16
                  imagePullPolicy: Always
                  name: csi-cdsplugin
                  securityContext:
                    allowPrivilegeEscalation: true
                    capabilities:
                      add:
                      - SYS_ADMIN
                    privileged: true
                  volumeMounts:
                  - mountPath: /etc/kubernetes
                    name: cloud-config
                  - mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
                    name: socket-dir
                  - mountPath: /var/lib/kubelet/pods
                    mountPropagation: Bidirectional
                    name: pods-mount-dir
                  - mountPath: /data/kubelet/pods
                    mountPropagation: Bidirectional
                    name: pods-mount-dir-data
                  - mountPath: /sys
                    name: host-sys
                  - mountPath: /lib/modules
                    name: lib-modules
                    readOnly: true
                  - mountPath: /dev
                    name: host-dev
                hostNetwork: true
                restartPolicy: Always
                serviceAccount: csi-external-runner
                serviceAccountName: csi-external-runner
                priorityClassName: system-node-critical
                volumes:
                - hostPath:
                    path: /etc/kubernetes
                    type: ""
                  name: cloud-config
                - hostPath:
                    path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
                    type: DirectoryOrCreate
                  name: reg-dir
                - hostPath:
                    path: /var/lib/kubelet/pods
                    type: DirectoryOrCreate
                  name: pods-mount-dir
                - hostPath:
                    path: /data/kubelet/pods
                    type: DirectoryOrCreate
                  name: pods-mount-dir-data
                - hostPath:
                    path: /var/lib/kubelet/plugins/csi-cdsplugin
                    type: DirectoryOrCreate
                  name: socket-dir
                - hostPath:
                    path: /sys
                    type: ""
                  name: host-sys
                - hostPath:
                    path: /lib/modules
                    type: ""
                  name: lib-modules
                - hostPath:
                    path: /dev
                    type: ""
                  name: host-dev
          ---
          kind: Service
          apiVersion: v1
          metadata:
            name: csi-attacher-cds
            namespace: kube-system
            labels:
              app: csi-attacher-cds
          spec:
            selector:
              app: csi-attacher-cds
            clusterIP: None
          ---
          kind: StatefulSet
          apiVersion: apps/v1
          metadata:
            name: csi-attacher-cds
            namespace: kube-system
          spec:
            serviceName: "csi-attacher-cds"
            replicas: 1
            selector:
              matchLabels:
                app: csi-attacher-cds
            template:
              metadata:
                labels:
                  app: csi-attacher-cds
              spec:
                priorityClassName: system-cluster-critical
                serviceAccount: csi-external-runner
                containers:
                  - name: csi-attacher-cds
                    image: hub.baidubce.com/cce/csi-attacher:latest-1.16
                    args:
                      - "--v=5"
                      - "--csi-address=$(ADDRESS)"
                      - "--leader-election"
                    env:
                      - name: MY_NAME
                        valueFrom:
                          fieldRef:
                            fieldPath: metadata.name
                      - name: MY_NAMESPACE
                        valueFrom:
                          fieldRef:
                            fieldPath: metadata.namespace
                      - name: ADDRESS
                        value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
                    imagePullPolicy: "Always"
                    volumeMounts:
                      - name: socket-dir
                        mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
                volumes:
                  - name: socket-dir
                    hostPath:
                      path: /var/lib/kubelet/plugins/csi-cdsplugin
                      type: DirectoryOrCreate
          ---
          kind: Service
          apiVersion: v1
          metadata:
            name: csi-provisioner-cds
            namespace: kube-system
            labels:
              app: csi-provisioner-cds
          spec:
            selector:
              app: csi-provisioner-cds
            clusterIP: None
          ---
          kind: StatefulSet
          apiVersion: apps/v1
          metadata:
            name: csi-provisioner-cds
            namespace: kube-system
          spec:
            serviceName: "csi-provisioner-cds"
            replicas: 1
            selector:
              matchLabels:
                app: csi-provisioner-cds
            template:
              metadata:
                labels:
                  app: csi-provisioner-cds
              spec:
                priorityClassName: system-cluster-critical
                serviceAccount: csi-external-runner
                containers:
                  - name: csi-provisioner-cds
                    image: hub.baidubce.com/cce/csi-provisioner:latest-1.16
                    args:
                      - "--provisioner=csi-cdsplugin"
                      - "--csi-address=$(ADDRESS)"
                      - "--v=5"
                      - "--feature-gates=Topology=true"
                      - "--volume-name-prefix=pv"
                    env:
                      - name: ADDRESS
                        value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
                    imagePullPolicy: "Always"
                    volumeMounts:
                      - name: socket-dir
                        mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
                volumes:
                  - name: socket-dir
                    hostPath:
                      path: /var/lib/kubelet/plugins/csi-cdsplugin
                      type: DirectoryOrCreate
          ---
          apiVersion: storage.k8s.io/v1
          kind: StorageClass
          metadata:
            name: hp1               #名字可以自定义
          provisioner: csi-cdsplugin
          parameters:
            dynamicVolume: "true"       #需要为 true才会创建
            cdsSizeInGB: "40"           #对单个 storageClass能创建的 size限制
            paymentTiming: "Postpaid"   #Prepaid or Postpaid
            storageType: "hp1"          #支持std1 hp1 ssd
            reservationLength: "3"      #Prepaid 模式下需要填写
            zone: "xxx"                  #参数可选
          reclaimPolicy: Delete         #支持 Delete、Retain 默认值为 Delete

          StorageClass 参数说明:

          • cdsSizeInGB

            cds 创建磁盘有限制: hp1: 5~32765GB std1: 5~32765GB ssd:50~32765GB cdsSizeInGB用于设置 StorageClass创建的cds 的最大值, PVC中设置值需要小于这个值

          • zone

            对于机器都在同一个 Zone 的情况,可以通过指定 StorageClass里的 Zone,保证cds 在指定的 Zone创建。 集群中节点所在可用区可以通过kubectl get nodes --show-labels查看failure-domain.beta.kubernetes.io/zone标签的值确定。 如: 可用区 A 的机器对应为 zoneA 可用区 B 的机器对应为 zoneB

          • paymentTiming

            用于设置付费方式是:预付费 or 后付费

          • reservationLength

            磁盘的保留时间,预付费模式下需要填写,单位为月

          bash
          $ kubectl apply -f dynamic-cds-template.yaml
          serviceaccount/csi-external-runner created
          clusterrole.rbac.authorization.k8s.io/external-runner configured
          clusterrolebinding.rbac.authorization.k8s.io/csi-role-binding configured
          role.rbac.authorization.k8s.io/external-runner-cfg configured
          rolebinding.rbac.authorization.k8s.io/csi-role-cfg configured
          daemonset.apps/csi-cdsplugin created
          service/csi-attacher-cds created
          statefulset.apps/csi-attacher-cds created
          service/csi-provisioner-cds created
          statefulset.apps/csi-provisioner-cds created
          storageclass.storage.k8s.io/csi-cds configured
          
          $ kubectl get pod --namespace kube-system  | grep provisioner
          kubectl get pod --namespace kube-system  | grep provisioner
          csi-provisioner-cds-0                    1/1       Running             0          10s
          
          $ kubectl get pod --namespace kube-system  | grep attacher
          csi-attacher-cds-0                       1/1       Running             0          20s
          csi-attacher-cds-1                       1/1       Running             0          20s

          如果相应的Pod进入Running状态,则动态绑定PV所需的资源已经建立成功。

          2.创建PVC时动态生成PV并绑定

          在PVC Spec中指定上面创建的StorageClass名称,则在创建PVC时,会自动调用相应StorageClass绑定的的Provisioner生成相应的PV进行绑定。

          使用kubectl,执行 kubectl create -f csi-pvc-cds.yaml 完成PVC的创建。

          假设创建的StorageClass名称为hp1,对应的 csi-pvc-cds.yaml 文件如下所示

          kind: PersistentVolumeClaim
          apiVersion: v1
          metadata:
            name: csi-pvc-cds
          spec:
            accessModes:
              - ReadWriteOnce
            storageClassName: hp1
            resources:
              requests:
                storage: 5Gi

          PVC参数说明: PVC 中 resource request的 storge,对于不同StorageClass中不同storageType有相关限制。 std1: 5~32765GB hp1: 5~32765GB ssd:50~32765GB 同时 PVC中申请的磁盘大小应该<= StorageClass 中cdsSizeInGB

          创建PVC后,可以看见相应的PV自动创建,PVC状态变为Bound,即PVC已经与新创建的PV绑定。

          $ kubectl create -f csi-pvc-cds.yaml
          persistentvolumeclaim "csi-pvc-cds" created
          $ kubectl get pvc
          NAME              STATUS    VOLUME                 CAPACITY   ACCESS MODES   STORAGECLASS   AGE
          csi-pvc-cds       Bound     pvc-1ab36e4d1d2711e9   5Gi        RWX            hp1            4s
          $ kubectl get pv
          NAME                   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                     STORAGECLASS   REASON    AGE
          pvc-1ab36e4d1d2711e9   5Gi        RWX            Delete           Bound     default/csi-pvc-cds       hp1                      7s
          3.在Pod内挂载PVC

          在Pod spec内指定相应的PVC名称即可,使用kubectl,执行 kubectl create -f dynamic-cds-pod.yaml 完成资源的创建。

          对应的dynamic-cds-pod.yaml文件如下所示:

          apiVersion: v1
          kind: Pod
          metadata:
            name: test-pvc-pod
            labels:
              app: test-pvc-pod
          spec:
            containers:
            - name: test-pvc-pod
              image: nginx
              volumeMounts:
                - name: cds-pvc
                  mountPath: "/cds-volume"
            volumes:
              - name: cds-pvc
                persistentVolumeClaim:
                  claimName: csi-pvc-cds

          Pod创建后,可以读写容器内的/cds-volume路径来访问相应的cds存储上的内容。

          4.释放PVC时动态销毁绑定PV

          删除PVC时,与之绑定的动态PV是否会被自动删除由自定义的 StorageClass 中reclaimPolicy决定;reclaimPolicy 支持 Delete、Retain 默认值为 Delete。

          该值设置为 Delete 时:删除 PVC 会自动删除PV,并且动态申请的 cds 会被自动释放 该值设置为 Retain 时:删除 PVC时PV状态会变成 Released,不会自动删除PV,对应自动创建的 cds 需要到 cds console上手动释放

          $ kubectl delete pvc csi-pvc-cds
          
          persistentvolumeclaim "csi-pvc-cds" deleted  
          
          $ kubectl get pv  
          
          NAME                   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM        
          
          pvc-1ab36e4d1d2711e9   40Gi       RWO            Retain           Released   default/csi-pvc-cds   hp1                     3m
          5.在 StatefuleSet里通过 claimTemple 批量使用 pvc的场景,需要前置步骤为步骤1

          以 sts-claim.yaml为例

          apiVersion: apps/v1
          kind: StatefulSet
          metadata:
            name: web
          spec:
            serviceName: "nginx"
            replicas: 2
            selector:
              matchLabels:
                app: nginx
            template:
              metadata:
                labels:
                  app: nginx
              spec:
                containers:
                - name: nginx
                  image: nginx
                  ports:
                  - containerPort: 80
                    name: web
                  volumeMounts:
                  - name: www
                    mountPath: /usr/share/nginx/html
            volumeClaimTemplates:
            - metadata:
                name: www
              spec:
                accessModes: [ "ReadWriteOnce" ]
                resources:
                  requests:
                    storage: 5Gi
                storageClassName: hp1

          创建 sts-claim.yaml,同时会自动创建 PVC

          $ kubectl apply -f sts-claim.yaml
          
          $ kubectl get pod
          NAME         READY     STATUS              RESTARTS   AGE
          web-0        1/1       Running             0          3m
          web-1        1/1       Running             0          2m
          
          $ kubectl get pvc
          NAME           STATUS    VOLUME                 CAPACITY   ACCESS MODES   STORAGECLASS   AGE
          www-web-0      Bound     pvc-a1e885701d2f11e9   5Gi        RWO            hp1            6m
          www-web-1      Bound     pvc-c91edb891d2f11e9   5Gi        RWO            hp1            5m
          6.多可用区集群使用使用cds

          (1)StorageClass指定zone方式

          目前 cds不支持跨可用区,如果cce集群中有多个可用区的机器,动态创建cds需要通过设置StorageClass中zone字段和NodeSelector等指定可用区实现;使用已有cds,指定NodeSelector需要使用的可用区可实现。

          CCE集群默认会为集群节点打上可用区标签:

          $ kubectl get nodes --show-labels
          NAME            STATUS    ROLES     AGE       VERSION   LABELS
          192.168.80.15   Ready     <none>    13d       v1.8.12   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=BCC,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=bj,failure-domain.beta.kubernetes.io/zone=zoneC,kubernetes.io/hostname=192.168.80.15

          其中 failure-domain.beta.kubernetes.io/zone=zoneC 标签说明该集群节点处于C可用区。

          1)多可用区集群使用 cds例子

          StorageClass-Multi-Zone-Temple.yaml

          apiVersion: storage.k8s.io/v1
          kind: StorageClass
          metadata:
            name: csi-cds-ssd-new
          provisioner: csi-cdsplugin
          parameters:
            dynamicVolume: "true"
            cdsSizeInGB: "40" 
            paymentTiming: "Postpaid"
            storageType: "hp1"
            reservationLength: "3"
            zone: "zoneA"
          reclaimPolicy: Delete

          Sts-Multi-Zone-Temple.yaml

          apiVersion: apps/v1
          kind: StatefulSet
          metadata:
            name: sts-multi-zone
          spec:
            serviceName: "nginx"
            replicas: 1
            selector:
              matchLabels:
                app: nginx
            template:
              metadata:
                labels:
                  app: nginx
              spec:
                nodeSelector:
                  failure-domain.beta.kubernetes.io/zone: "zoneA"
                containers:
                - name: nginx
                  image: nginx
                  ports:
                  - containerPort: 80
                    name: web
                  volumeMounts:
                  - name: www
                    mountPath: /usr/share/nginx/html
            volumeClaimTemplates:
            - metadata:
                name: www
              spec:
                accessModes: [ "ReadWriteOnce" ]
                resources:
                  requests:
                    storage: 40Gi
                storageClassName: csi-cds-ssd-new

          (2)设置 StorageClass的Volume Binding Mode 通过指定WaitForFirstConsumer和配置PersistentVolume 的模式来解决此问题,指定绑定模式为WaitForFirstConsumer,K8S会先进行调度 Pod,随后创建 CDS。

          1)使用示例

          StorageClass-WaitForFirstConsumer-Temple.yaml

          apiVersion: storage.k8s.io/v1
          kind: StorageClass
          metadata:
            name: csi-cds
          provisioner: csi-cdsplugin
          volumeBindingMode: WaitForFirstConsumer
          parameters:
            dynamicVolume: "true"
            cdsSizeInGB: "40"
            paymentTiming: "Postpaid"
            storageType: "hp1"
          reclaimPolicy: Delete

          Sts-WaitForFirstConsumer-Temple.yaml

          apiVersion: apps/v1
          kind: StatefulSet
          metadata:
            name: web
          spec:
            serviceName: "nginx"
            replicas: 1
            selector:
              matchLabels:
                app: nginx
            template:
              metadata:
                labels:
                  app: nginx
              spec:
                containers:
                - name: nginx
                  image: nginx
                  ports:
                  - containerPort: 80
                    name: web
                  volumeMounts:
                  - name: www
                    mountPath: /usr/share/nginx/html
            volumeClaimTemplates:
            - metadata:
                name: www
              spec:
                accessModes: [ "ReadWriteOnce" ]
                resources:
                  requests:
                    storage: 5Gi
                storageClassName: csi-cds

          具体详情请看相关文档 StorageClass

          注意:

          1. 如果需要销毁 cce集群并且使用了本文中的 csi-provisioner 同时 storageClass中reclaimPolicy设置为 Delete,请先 delete pvc,否则可能会存在cds 无法自动释放。
          2. 多可用区集群使用cds,如果同时设置了WaitForFirstConsumer和 zone,zone 将会被覆盖。

          在单个Pod中挂载多块CDS

          注意:

          • 在Pod Spec的volumes字段中,每块CDS只能被声明一次。
          • 在Pod Spec的volumeMounts字段中,每块CDS对应的volume可以被引用多次。
          • 对于Deployment/ReplicaSet等资源,replicas域只能设置为1。

          下面的示例 cds-multi-mount-example.yaml 将文件中声明的四块CDS磁盘分别挂载到Pod mysql-nginx内的my-nginx和my-mysql两个容器的相应路径下,其中test-data和test-data-2两个volume对应的CDS为两个容器共享,而nginx-data和db-data两个volume对应的CDS分别只挂载在各自容器中。

          apiVersion: apps/v1
          kind: Deployment
          metadata:
            name: mysql-nginx
          spec:
            replicas: 1
            selector:
              matchLabels:
                app: mysql-nginx
            template:
              metadata:
                name: mysql-nginx
                labels:
                  app: mysql-nginx
              spec:
                containers:
                - name: my-nginx
                  image: nginx
                  volumeMounts:
                  - name: nginx-data
                    mountPath: /nginx-data
                  - name: test-data
                    mountPath: /test-data
                  - name: test-data-2
                    mountPath: /test-data-2
                  ports:
                  - containerPort: 80
                - name: my-mysql
                  image: mysql:5.7
                  volumeMounts:
                  - name: db-data
                    mountPath: /db-data
                  - name: test-data
                    mountPath: /test-data
                  - name: test-data-2
                    mountPath: /test-data-2
                  ports:
                  - containerPort: 3306
                  env:
                  - name: MYSQL_ROOT_PASSWORD
                    value: M6jshva8hTXAGnin
                volumes:
                - name: db-data
                  flexVolume:
                    driver: "baidubce/cds"
                    fsType: "ext4"
                    options:
                      volumeID: "v-Hd8MUXA7"
                - name: nginx-data
                  flexVolume:
                    driver: "baidubce/cds"
                    fsType: "ext4"
                    options:
                      volumeID: "v-OWEHtlSI"
                - name: test-data
                  flexVolume:
                    driver: "baidubce/cds"
                    fsType: "ext4"
                    options:
                      volumeID: "v-9PhGVoMu"
                - name: test-data-2
                  flexVolume:
                    driver: "baidubce/cds"
                    fsType: "ext4"
                    options:
                      volumeID: "v-SxjD482V"

          flexVolume字段说明:

          • fsType:CDS磁盘的文件系统,如果是新建的CDS磁盘,将会自动格式化该CDS磁盘为该文件系统
          • volumeID:CDS磁盘ID,您可以从百度智能云控制台的页面获取

          使用 kubectl create -f cds-multi-mount-example.yaml 创建相应的资源,相应的CDS磁盘将会被挂载到文件中声明的各个目录下。Pod在集群内发生漂移时,这些CDS磁盘将会跟随Pod一同迁移。

          1.13集群PVC快照功能

          使用此功能需要更新provsioner对应的yaml,加入对snapshotter功能

          1.新建snapshotter

          # provisioner + snapshotter
          kind: Service
          apiVersion: v1
          metadata:
            name: csi-provisioner-cds
            namespace: kube-system
            labels:
              app: csi-provisioner-cds
          spec:
            selector:
              app: csi-provisioner-cds
            clusterIP: None
          ---
          kind: StatefulSet
          apiVersion: apps/v1
          metadata:
            name: csi-provisioner-cds
            namespace: kube-system
          spec:
            serviceName: "csi-provisioner-cds"
            replicas: 1
            selector:
              matchLabels:
                app: csi-provisioner-cds
            template:
              metadata:
                labels:
                  app: csi-provisioner-cds
              spec:
                priorityClassName: system-cluster-critical
                serviceAccount: csi-external-runner
                containers:
                  - name: csi-provisioner-cds
                    image: hub.baidubce.com/cce/csi-provisioner:latest-1.13
                    args:
                      - "--provisioner=csi-cdsplugin"
                      - "--csi-address=$(ADDRESS)"
                      - "--v=5"
                      - "--feature-gates=Topology=true"
                      - "--volume-name-prefix=pv"
                    env:
                      - name: ADDRESS
                        value: /var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
                    imagePullPolicy: "Always"
                    volumeMounts:
                      - name: socket-dir
                        mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
                  - name: csi-snapshotter
                    image: hub.baidubce.com/cce/csi-snapshotter:latest-1.13
                    args:
                      - --csi-address=$(ADDRESS)
                      - --connection-timeout=15s
                      - --v=5
                    env:
                      - name: ADDRESS
                        value: /var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
                    imagePullPolicy: "Always"
                    volumeMounts:
                      - name: socket-dir
                        mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
                volumes:
                  - name: socket-dir
                    hostPath:
                      path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
                      type: DirectoryOrCreate

          2.新建snapshotclass

          # snapshot class
          apiVersion: snapshot.storage.k8s.io/v1alpha1
          kind: VolumeSnapshotClass
          metadata:
            name: csi-cdsplugin-snapclass #名称可变
          snapshotter: csi-cdsplugin

          3.创建snapshot

          # create shanpshot.yaml
          apiVersion: snapshot.storage.k8s.io/v1alpha1
          kind: VolumeSnapshot
          metadata:
            name: new-snapshot-demo
          spec:
            snapshotClassName: csi-cdsplugin-snapclass #需已创建
            source:
              name: csi-cds-pvc-test #pvc 必须存在,且必须为bound
              kind: PersistentVolumeClaim

          执行 kubectl apply -f shanpshot.yaml

          4.从snapshot创建pvc

          #restore
          apiVersion: v1
          kind: PersistentVolumeClaim
          metadata:
            name: cds-restore
          spec:
            storageClassName: csi-cds
            dataSource:
              name: new-snapshot-demo 
              kind: VolumeSnapshot
              apiGroup: snapshot.storage.k8s.io
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 30Gi
                

          5.使用从snapshot恢复的pod

          # pod
          apiVersion: v1
          kind: Pod
          metadata:
            name: web-server-restore
            namespace: default
          spec:
            containers:
            - image: nginx
              imagePullPolicy: Always
              name: web-server
              volumeMounts:
              - mountPath: /var/lib/www/html
                name: mypvc
            volumes:
            - name: mypvc
              persistentVolumeClaim:
                claimName: cds-restore

          6.删除shapshot

          执行 kubectl delele volumesnapshots new-snapshot-demo 如果用 pvc 在使用该 snapshot, delete会导致terminating

          上一篇
          使用本地存储
          下一篇
          使用文件存储CFS