容器引擎CCE

    使用云盘CDS

    当前CDS磁盘以FlexVolumeCSI形式在集群中挂载。FlexVolume可以直接在Pod中挂载,也可以基于CSI创建PV和PVC存储资源后,再在Pod中挂载。两种挂载方案都可以满足容器内特定数据持久化的需求,两者详细的区别请参考kubernetes官方文档:

    准备工作

    用户做好以下准备,才能在容器中挂载cds实例。

    注意: 需要k8s版本>=1.11

    集群节点所处地域和可用区相同的CDS磁盘已创建

    CDS磁盘状态为未挂载

    创建容器集群

    1. 创建一个容器集群,操作步骤参考创建集群
    2. 下载命令行客户端kubectl,并连接集群,操作步骤参考通过kubectl连接Kubernetes集群

    操作指南

    通过Volume方式挂载

    注意:

    CDS同一时间只能挂载在一台bcc实例上。 CDS同一时间只能被一个Pod声明使用。 对于Deployment/ReplicaSet等资源,replicas域只能设置为1。 下面的示例 cds-example.yaml 将文件中声明的CDS磁盘挂载到Pod内my-nginx容器的/data路径下。

    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      name: my-nginx
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: my-nginx
      template:
        metadata:
          name: my-nginx
          labels:
            app: my-nginx
        spec:
          containers:
          - name: my-nginx
            image: nginx
            volumeMounts:
            - name: cds-data
              mountPath: /data
            ports:
            - containerPort: 80
          volumes:
          - name: cds-data
            flexVolume:
              driver: "baidubce/cds"
              fsType: "ext4"
              options:
                volumeID: "v-hTXAGtin"

    flexVolume字段说明:

    fsType:CDS磁盘的文件系统,如果是新建的CDS磁盘,将会自动格式化该CDS磁盘为该文件系统 volumeID:CDS磁盘ID,您可以从百度智能云控制台的页面获取

    使用 kubectl create -f cds-example.yaml 创建相应的资源,相应的CDS磁盘将会被挂载到my-nginx容器内的/data目录下。Pod在集群内发生漂移时,该CDS磁盘将会跟随Pod一同迁移。

    PV/PVC

    通过PV/PVC方式使用CDS,具体分为两种方式:

    • 静态挂载:需要用户提前在百度智能云中创建好CDS(操作方法参考CDS文档),然后通过CDS的volume id在集群中创建PV和PVC资源。
    • 动态挂载:用户在集群中声明PVC时,自动创建出CDS磁盘并且动态关联至PV。

    静态挂载的方式需要用户提供的CDS磁盘处于未挂载状态,并且确保填入正确的volume id。动态挂载在部分场景下可能会更加方便,但是需要注意动态挂载时将会自动创建CDS磁盘并且计费,动态创建的CDS将会随着用户删除PVC时被联动删除(需要将对应storageClass的reclaimPolicy参数设置为delete),但是如果用户在没有删除PVC的情况下直接删除了CCE集群,则需要手动对这些CDS进行处理。

    静态PV/PVC方式挂载cds

    1.创建CdsPlugin、Attacher

    cds-template.yaml是一个yaml文件模板,包含了需要创建的集群资源信息,使用kubectl apply -f cds-template.yaml进行相关组件部署。

    1.11集群 cds-template.yaml文件内容如下(1.11集群使用):

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: csi-external-runner
      namespace: kube-system
    ---
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: external-runner
    rules:
      - apiGroups: [""]
        resources: ["events"]
        verbs: ["get", "list", "watch", "update", "create", "patch"]
      - apiGroups: [""]
        resources: ["persistentvolumes"]
        verbs: ["get", "list", "watch", "update", "create", "delete"]
      - apiGroups: [""]
        resources: ["persistentvolumeclaims"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: [""]
        resources: ["nodes"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["volumeattachments"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: [""]
        resources: ["namespaces"]
        verbs: ["get", "list"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["storageclasses"]
        verbs: ["get", "list", "watch"]
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: csi-role-binding
    subjects:
      - kind: ServiceAccount
        name: csi-external-runner
        namespace: kube-system
    roleRef:
      kind: ClusterRole
      name: external-runner
      apiGroup: rbac.authorization.k8s.io
    ---
    kind: Role
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      namespace: kube-system
      name: external-runner-cfg
    rules:
    - apiGroups: [""]
      resources: ["configmaps"]
      verbs: ["get", "watch", "list", "delete", "update", "create"]
    ---
    kind: RoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: csi-role-cfg
      namespace: kube-system
    subjects:
      - kind: ServiceAccount
        name: csi-external-runner
        namespace: kube-system
    roleRef:
      kind: Role
      name: external-runner-cfg
      apiGroup: rbac.authorization.k8s.io
    ---
    kind: DaemonSet
    apiVersion: apps/v1beta2
    metadata:
      name: csi-cdsplugin
      namespace: kube-system
    spec:
      selector:
        matchLabels:
          app: csi-cdsplugin
      template:
        metadata:
          labels:
            app: csi-cdsplugin
        spec:
          serviceAccount: csi-external-runner
          hostNetwork: true
          containers:
            - name: driver-registrar
              image: hub.baidubce.com/cce/driver-registrar:latest-1.11
              args:
                - "--v=5"
                - "--csi-address=$(ADDRESS)"
              env:
                - name: ADDRESS
                  value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
                - name: KUBE_NODE_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: spec.nodeName
              imagePullPolicy: "Always"
              volumeMounts:
                - name: socket-dir
                  mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
            - name: csi-cdsplugin
              securityContext:
                privileged: true
                capabilities:
                  add: ["SYS_ADMIN"]
                allowPrivilegeEscalation: true
              image: hub.baidubce.com/cce/cdsplugin:latest-1.11
              args :
                - "--nodeid=$(NODE_ID)"
                - "--endpoint=$(CSI_ENDPOINT)"
                - "--v=5"
                - "--drivername=csi-cdsplugin"
              env:
                - name: NODE_ID
                  valueFrom:
                    fieldRef:
                      fieldPath: spec.nodeName
                - name: CSI_ENDPOINT
                  value: unix://var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
              imagePullPolicy: "Always"
              volumeMounts:
                - name: cloud-config
                  mountPath: /etc/kubernetes
                - name: plugin-dir
                  mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
                - name: pods-mount-dir
                  mountPath: /var/lib/kubelet/pods
                  mountPropagation: "Bidirectional"
                - name: pods-mount-dir-mnt
                  mountPath: /mnt/kubelet/pods
                  mountPropagation: "Bidirectional"
                - name: pods-mount-dir-data
                  mountPath: /data/kubelet/pods
                  mountPropagation: "Bidirectional"
                - mountPath: /sys
                  name: host-sys
                - name: lib-modules
                  mountPath: /lib/modules
                  readOnly: true
                - name: host-dev
                  mountPath: /dev
          volumes:
            - name: cloud-config
              hostPath:
                path: /etc/kubernetes
            - name: plugin-dir
              hostPath:
                path: /var/lib/kubelet/plugins/csi-cdsplugin
                type: DirectoryOrCreate
            - name: pods-mount-dir
              hostPath:
                path: /var/lib/kubelet/pods
                type: Directory
            - name: pods-mount-dir-data
              hostPath:
                path: /data/kubelet/pods
                type: DirectoryOrCreate
            - name: pods-mount-dir-mnt
              hostPath:
                path: /mnt/kubelet/pods
                type: DirectoryOrCreate
            - name: socket-dir
              hostPath:
                path: /var/lib/kubelet/plugins/csi-cdsplugin
                type: DirectoryOrCreate
            - name: host-sys
              hostPath:
                path: /sys
            - name: lib-modules
              hostPath:
                path: /lib/modules
            - name: host-dev
              hostPath:
                path: /dev
    ---
    kind: Service
    apiVersion: v1
    metadata:
      name: csi-attacher-cds
      namespace: kube-system
      labels:
        app: csi-attacher-cds
    spec:
      selector:
        app: csi-attacher-cds
      clusterIP: None
    ---
    kind: StatefulSet
    apiVersion: apps/v1beta1
    metadata:
      name: csi-attacher-cds
      namespace: kube-system
    spec:
      serviceName: "csi-attacher-cds"
      replicas: 2
      template:
        metadata:
          labels:
            app: csi-attacher-cds
        spec:
          serviceAccount: csi-external-runner
          containers:
            - name: csi-attacher-cds
              image: hub.baidubce.com/cce/csi-attacher:latest-1.11
              args:
                - "--v=5"
                - "--csi-address=$(ADDRESS)"
                - "--leader-election"
                - "--leader-election-namespace=$(MY_NAMESPACE)"
                - "--leader-election-identity=$(MY_NAME)"
              env:
                - name: MY_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.name
                - name: MY_NAMESPACE
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.namespace
                - name: ADDRESS
                  value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
              imagePullPolicy: "Always"
              volumeMounts:
                - name: socket-dir
                  mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
          volumes:
            - name: socket-dir
              hostPath:
                path: /var/lib/kubelet/plugins/csi-cdsplugin
                type: DirectoryOrCreate

    1.13集群: cds-template.yaml文件内容如下(1.13集群使用):

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: csi-external-runner
      namespace: kube-system
    ---
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: external-runner
    rules:
      - apiGroups: [""]
        resources: ["events"]
        verbs: ["get", "list", "watch", "update", "create", "patch"]
      - apiGroups: [""]
        resources: ["persistentvolumes"]
        verbs: ["get", "list", "watch", "update", "create", "delete"]
      - apiGroups: [""]
        resources: ["persistentvolumeclaims"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: [""]
        resources: ["nodes"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["volumeattachments"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: [""]
        resources: ["namespaces"]
        verbs: ["get", "list"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["storageclasses"]
        verbs: ["get", "list", "watch"]
      - apiGroups: ["snapshot.storage.k8s.io"]
        resources: ["*"]
        verbs: ["create", "get", "list", "watch", "update", "patch"]
      - apiGroups: ["apiextensions.k8s.io"]
        resources: ["*"]
        verbs: ["create", "get", "list", "watch"]
      - apiGroups: ["csi.storage.k8s.io"]
        resources: ["csidrivers"]
        verbs: ["create", "delete", "get", "list", "watch"]
      - apiGroups: ["csi.storage.k8s.io"]
        resources: ["csinodeinfos"]
        verbs: ["get", "list", "watch"]
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: csi-role-binding
    subjects:
      - kind: ServiceAccount
        name: csi-external-runner
        namespace: kube-system
    roleRef:
      kind: ClusterRole
      name: external-runner
      apiGroup: rbac.authorization.k8s.io
    ---
    kind: Role
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      namespace: kube-system
      name: external-runner-cfg
    rules:
    - apiGroups: [""]
      resources: ["configmaps"]
      verbs: ["get", "watch", "list", "delete", "update", "create"]
    ---
    kind: RoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: csi-role-cfg
      namespace: kube-system
    subjects:
      - kind: ServiceAccount
        name: csi-external-runner
        namespace: kube-system
    roleRef:
      kind: Role
      name: external-runner-cfg
      apiGroup: rbac.authorization.k8s.io
    ---
    apiVersion: extensions/v1beta1
    kind: DaemonSet
    metadata:
      name: csi-cdsplugin
      namespace: kube-system
    spec:
      selector:
        matchLabels:
          app: csi-cdsplugin
      template:
        metadata:
          labels:
            app: csi-cdsplugin
        spec:
          containers:
          - args:
            - --v=5
            - --csi-address=$(ADDRESS)
            - --kubelet-registration-path=$(ADDRESS)
            lifecycle:
              preStop:
                exec:
                  command: ["/bin/sh", "-c", "rm -rf /registration/csi-cdsplugin /registration/csi-cdsplugin-reg.sock"]
            env:
            - name: ADDRESS
              value: /var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
            image: hub.baidubce.com/cce/driver-registrar:latest-1.13
            imagePullPolicy: Always
            name: driver-registrar
            resources: {}
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            volumeMounts:
            - mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
              name: socket-dir
            - mountPath: /registration
              name: reg-dir
          - args:
            - --nodeid=$(NODE_ID)
            - --endpoint=$(CSI_ENDPOINT)
            - --v=5
            - --drivername=csi-cdsplugin
            env:
            - name: NODE_ID
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
            - name: CSI_ENDPOINT
              value: unix://var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
            image: hub.baidubce.com/cce/cdsplugin:latest-1.13
            imagePullPolicy: Always
            name: csi-cdsplugin
            securityContext:
              allowPrivilegeEscalation: true
              capabilities:
                add:
                - SYS_ADMIN
              privileged: true
            volumeMounts:
            - mountPath: /etc/kubernetes
              name: cloud-config
            - mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
              name: socket-dir
            - mountPath: /var/lib/kubelet/pods
              mountPropagation: Bidirectional
              name: pods-mount-dir
            - mountPath: /data/kubelet/pods
              mountPropagation: Bidirectional
              name: pods-mount-dir-data
            - name: pods-mount-dir-mnt
              mountPath: /mnt/kubelet/pods
              mountPropagation: "Bidirectional"
            - mountPath: /sys
              name: host-sys
            - mountPath: /lib/modules
              name: lib-modules
              readOnly: true
            - mountPath: /dev
              name: host-dev
          hostNetwork: true
          restartPolicy: Always
          serviceAccount: csi-external-runner
          serviceAccountName: csi-external-runner
          priorityClassName: system-node-critical
          volumes:
          - hostPath:
              path: /etc/kubernetes
              type: ""
            name: cloud-config
          - hostPath:
              path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
              type: DirectoryOrCreate
            name: reg-dir
          - hostPath:
              path: /var/lib/kubelet/pods
              type: Directory
            name: pods-mount-dir
          - hostPath:
              path: /data/kubelet/pods
              type: Directory
            name: pods-mount-dir-data
          - name: pods-mount-dir-mnt
            hostPath:
              path: /mnt/kubelet/pods
              type: DirectoryOrCreate
          - hostPath:
              path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
              type: DirectoryOrCreate
            name: socket-dir
          - hostPath:
              path: /sys
              type: ""
            name: host-sys
          - hostPath:
              path: /lib/modules
              type: ""
            name: lib-modules
          - hostPath:
              path: /dev
              type: ""
            name: host-dev
    ---
    kind: Service
    apiVersion: v1
    metadata:
      name: csi-attacher-cds
      namespace: kube-system
      labels:
        app: csi-attacher-cds
    spec:
      selector:
        app: csi-attacher-cds
      clusterIP: None
    ---
    kind: StatefulSet
    apiVersion: apps/v1beta1
    metadata:
      name: csi-attacher-cds
      namespace: kube-system
    spec:
      serviceName: "csi-attacher-cds"
      replicas: 2
      template:
        metadata:
          labels:
            app: csi-attacher-cds
        spec:
          priorityClassName: system-cluster-critical
          serviceAccount: csi-external-runner
          containers:
            - name: csi-attacher-cds
              image: hub.baidubce.com/cce/csi-attacher:latest-1.13
              args:
                - "--v=5"
                - "--csi-address=$(ADDRESS)"
                - "--leader-election"
                - "--leader-election-namespace=$(MY_NAMESPACE)"
                - "--leader-election-identity=$(MY_NAME)"
              env:
                - name: MY_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.name
                - name: MY_NAMESPACE
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.namespace
                - name: ADDRESS
                  value: /var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
              imagePullPolicy: "Always"
              volumeMounts:
                - name: socket-dir
                  mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
          volumes:
            - name: socket-dir
              hostPath:
                path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
                type: DirectoryOrCreate

    1.16集群 cds-template.yaml文件内容如下(1.16集群使用):

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: csi-external-runner
      namespace: kube-system
    ---
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: external-runner
    rules:
      - apiGroups: [""]
        resources: ["events"]
        verbs: ["get", "list", "watch", "update", "create", "patch"]
      - apiGroups: [""]
        resources: ["persistentvolumes"]
        verbs: ["get", "list", "watch", "update", "create", "delete", "patch"]
      - apiGroups: [""]
        resources: ["persistentvolumeclaims"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: [""]
        resources: ["nodes"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["volumeattachments"]
        verbs: ["get", "list", "watch", "update", "patch"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["volumeattachments/status"]
        verbs: ["patch"]
      - apiGroups: [""]
        resources: ["namespaces"]
        verbs: ["get", "list"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["storageclasses"]
        verbs: ["get", "list", "watch"]
      - apiGroups: ["snapshot.storage.k8s.io"]
        resources: ["*"]
        verbs: ["create", "get", "list", "watch", "update", "patch"]
      - apiGroups: ["apiextensions.k8s.io"]
        resources: ["*"]
        verbs: ["create", "get", "list", "watch"]
      - apiGroups: ["csi.storage.k8s.io"]
        resources: ["csidrivers"]
        verbs: ["create", "delete", "get", "list", "watch"]
      - apiGroups: ["csi.storage.k8s.io"]
        resources: ["csinodeinfos"]
        verbs: ["get", "list", "watch"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["csinodes"]
        verbs: ["get", "list", "watch"]
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: csi-role-binding
    subjects:
      - kind: ServiceAccount
        name: csi-external-runner
        namespace: kube-system
    roleRef:
      kind: ClusterRole
      name: external-runner
      apiGroup: rbac.authorization.k8s.io
    ---
    kind: Role
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      namespace: kube-system
      name: external-runner-cfg
    rules:
    - apiGroups: [""]
      resources: ["configmaps"]
      verbs: ["get", "watch", "list", "delete", "update", "create"]
    - apiGroups: ["coordination.k8s.io"]
      resources: ["leases"]
      verbs: ["get", "watch", "list", "delete", "update", "create"]
    ---
    kind: RoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: csi-role-cfg
      namespace: kube-system
    subjects:
      - kind: ServiceAccount
        name: csi-external-runner
        namespace: kube-system
    roleRef:
      kind: Role
      name: external-runner-cfg
      apiGroup: rbac.authorization.k8s.io
    ---
    apiVersion: extensions/v1beta1
    kind: DaemonSet
    metadata:
      name: csi-cdsplugin
      namespace: kube-system
    spec:
      selector:
        matchLabels:
          app: csi-cdsplugin
      template:
        metadata:
          labels:
            app: csi-cdsplugin
        spec:
          containers:
          - args:
            - --v=6
            - --csi-address=$(ADDRESS)
            - --kubelet-registration-path=$(ADDRESS)
            lifecycle:
              preStop:
                exec:
                  command: ["/bin/sh", "-c", "rm -rf /registration/csi-cdsplugin /registration/csi-cdsplugin-reg.sock"]
            env:
            - name: ADDRESS
              value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
            image: hub.baidubce.com/cce/csi-node-driver-registrar:latest-1.16
            imagePullPolicy: Always
            name: node-driver-registrar
            resources: {}
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            volumeMounts:
            - mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
              name: socket-dir
            - mountPath: /registration
              name: reg-dir
          - args:
            - --nodeid=$(NODE_ID)
            - --endpoint=$(CSI_ENDPOINT)
            - --v=5
            - --drivername=csi-cdsplugin
            env:
            - name: NODE_ID
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
            - name: CSI_ENDPOINT
              value: unix://var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
            image: hub.baidubce.com/cce/cdsplugin:latest-1.16
            imagePullPolicy: Always
            name: csi-cdsplugin
            securityContext:
              allowPrivilegeEscalation: true
              capabilities:
                add:
                - SYS_ADMIN
              privileged: true
            volumeMounts:
            - mountPath: /etc/kubernetes
              name: cloud-config
            - mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
              name: socket-dir
            - mountPath: /var/lib/kubelet/pods
              mountPropagation: Bidirectional
              name: pods-mount-dir
            - mountPath: /data/kubelet/pods
              mountPropagation: Bidirectional
              name: pods-mount-dir-data
            - name: pods-mount-dir-mnt
              mountPath: /mnt/kubelet/pods
              mountPropagation: "Bidirectional"
            - mountPath: /sys
              name: host-sys
            - mountPath: /lib/modules
              name: lib-modules
              readOnly: true
            - mountPath: /dev
              name: host-dev
          hostNetwork: true
          restartPolicy: Always
          serviceAccount: csi-external-runner
          serviceAccountName: csi-external-runner
          priorityClassName: system-node-critical
          volumes:
          - hostPath:
              path: /etc/kubernetes
              type: ""
            name: cloud-config
          - hostPath:
              path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
              type: DirectoryOrCreate
            name: reg-dir
          - hostPath:
              path: /var/lib/kubelet/pods
              type: Directory
            name: pods-mount-dir
          - hostPath:
              path: /data/kubelet/pods
              type: DirectoryOrCreate
            name: pods-mount-dir-data
          - name: pods-mount-dir-mnt
            hostPath:
              path: /mnt/kubelet/pods
              type: DirectoryOrCreate
          - hostPath:
              path: /var/lib/kubelet/plugins/csi-cdsplugin
              type: DirectoryOrCreate
            name: socket-dir
          - hostPath:
              path: /sys
              type: ""
            name: host-sys
          - hostPath:
              path: /lib/modules
              type: ""
            name: lib-modules
          - hostPath:
              path: /dev
              type: ""
            name: host-dev
    ---
    kind: Service
    apiVersion: v1
    metadata:
      name: csi-attacher-cds
      namespace: kube-system
      labels:
        app: csi-attacher-cds
    spec:
      selector:
        app: csi-attacher-cds
      clusterIP: None
    ---
    kind: StatefulSet
    apiVersion: apps/v1beta1
    metadata:
      name: csi-attacher-cds
      namespace: kube-system
    spec:
      serviceName: "csi-attacher-cds"
      replicas: 1
      template:
        metadata:
          labels:
            app: csi-attacher-cds
        spec:
          priorityClassName: system-cluster-critical
          serviceAccount: csi-external-runner
          containers:
            - name: csi-attacher-cds
              image: hub.baidubce.com/cce/csi-attacher:latest-1.16
              args:
                - "--v=5"
                - "--csi-address=$(ADDRESS)"
                - "--leader-election"
              env:
                - name: MY_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.name
                - name: MY_NAMESPACE
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.namespace
                - name: ADDRESS
                  value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
              imagePullPolicy: "Always"
              volumeMounts:
                - name: socket-dir
                  mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
          volumes:
            - name: socket-dir
              hostPath:
                path: /var/lib/kubelet/plugins/csi-cdsplugin
                type: DirectoryOrCreate
    2.在集群中创建PV和PVC资源

    使用kubectl,执行 kubectl create -f pv-cds.yaml 完成PV的创建

    对应的pv-cds.yaml文件如下所示:

    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: pv-cds
      namespace: "default"
    spec:
      accessModes:
      - ReadWriteOnce
      capacity:
        storage: 5Gi
      csi:
        driver: "csi-cdsplugin"
        volumeHandle: "v-xxxx"
      persistentVolumeReclaimPolicy: Retain

    注意: yaml中volumeHandle字段对应的是cds volume id(短 id)

    创建PV后,输入kubectl get pv可以看见一个available状态的PV,如下所示:

    $ kubectl get pv
    NAME                   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                     STORAGECLASS   REASON    AGE
    pv-cds                 5Gi        RWO            Retain           Available                                                      22s

    建立一个能够与该PV绑定的PVC

    使用kubectl,执行 kubectl create -f pvc-cds.yaml完成PVC的创建

    对应的pvc-cds.yaml文件如下所示:

    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: csi-cds-pvc
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi

    绑定前,PVC为pending状态

    $ kubectl get pvc
    NAME      STATUS    VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    cds-pvc   Pending                                                      2s  

    绑定后,PV和PVC状态变为Bound

    $ kubectl get pv
    NAME      CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                  STORAGECLASS   REASON    AGE
    pv-cds    5Gi        RWO            Retain           Bound     default/csi-cds-pvc                             4s
    $ kubectl get pvc
    NAME           STATUS    VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    csi-cds-pvc    Bound     pv-cds    5Gi        RWO                           36s

    有关PV和PVC的更多设置和字段说明,见k8s官方文档

    3.在Pod内挂载PVC

    在Pod spec内指定相应的PVC名称即可,使用kubectl,执行 kubectl create -f demo-cds-rc.yaml 完成rc的创建

    对应的demo-cds-rc.yaml文件如下所示:

    apiVersion: apps/v1
    kind: ReplicaSet
    metadata:
      name: web-server-rs
      labels:
        test: ws
    spec:
      replicas: 1
      selector:
        matchLabels:
          se: ws
      template:
        metadata:
          labels:
            se: ws
          name: web-server
        spec:
          containers:
           - name: web-server
             image: nginx
             volumeMounts:
               - mountPath: /var/lib/www/html
                 name: csi-cds-pvc
          volumes:
           - name: csi-cds-pvc
             persistentVolumeClaim:
               claimName: csi-cds-pvc

    Pod创建后,可以读写容器内的/cds-volume路径来访问相应的cds存储上的内容。

    由于创建PV和PVC时只支持accessModes为ReadWriteOnce,该PVC可以被一节点上的Pod挂载读写。

    4.释放PV和PVC资源

    完成存储资源的使用后,可以释放PVC和PV资源

    使用以下命令可以释放PVC

    $ kubectl delete -f  pvc-cds.yaml
    释放PVC后,原来与之绑定的PV状态会变为Release,如下所示:
    
    NAME      CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                  STORAGECLASS   REASON    AGE
    pv-cds    5Gi        RWO            Retain           Released   default/csi-cds-pvc                             4m

    输入以下指令释放PV资源

    $ kubectl delete -f pv-cds.yaml

    动态PV/PVC方式挂载cds

    1.创建StorageClass、Attacher、Provisioner

    dynamic-cds-template.yaml是一个yaml文件模板,包含了需要创建的集群资源信息。

    1.11集群: dynamic-cds-template.yaml文件内容如下(1.11集群):

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: csi-external-runner
      namespace: kube-system
    ---
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: external-runner
    rules:
      - apiGroups: [""]
        resources: ["events"]
        verbs: ["get", "list", "watch", "update", "create", "patch"]
      - apiGroups: [""]
        resources: ["persistentvolumes"]
        verbs: ["get", "list", "watch", "update", "create", "delete"]
      - apiGroups: [""]
        resources: ["persistentvolumeclaims"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: [""]
        resources: ["nodes"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["volumeattachments"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: [""]
        resources: ["namespaces"]
        verbs: ["get", "list"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["storageclasses"]
        verbs: ["get", "list", "watch"]
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: csi-role-binding
    subjects:
      - kind: ServiceAccount
        name: csi-external-runner
        namespace: kube-system
    roleRef:
      kind: ClusterRole
      name: external-runner
      apiGroup: rbac.authorization.k8s.io
    ---
    kind: Role
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      namespace: kube-system
      name: external-runner-cfg
    rules:
    - apiGroups: [""]
      resources: ["configmaps"]
      verbs: ["get", "watch", "list", "delete", "update", "create"]
    ---
    kind: RoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: csi-role-cfg
      namespace: kube-system
    subjects:
      - kind: ServiceAccount
        name: csi-external-runner
        namespace: kube-system
    roleRef:
      kind: Role
      name: external-runner-cfg
      apiGroup: rbac.authorization.k8s.io
    ---
    kind: DaemonSet
    apiVersion: apps/v1beta2
    metadata:
      name: csi-cdsplugin
      namespace: kube-system
    spec:
      selector:
        matchLabels:
          app: csi-cdsplugin
      template:
        metadata:
          labels:
            app: csi-cdsplugin
        spec:
          serviceAccount: csi-external-runner
          hostNetwork: true
          containers:
            - name: driver-registrar
              image: hub.baidubce.com/cce/driver-registrar:latest-1.11
              args:
                - "--v=5"
                - "--csi-address=$(ADDRESS)"
              env:
                - name: ADDRESS
                  value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
                - name: KUBE_NODE_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: spec.nodeName
              imagePullPolicy: "Always"
              volumeMounts:
                - name: socket-dir
                  mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
            - name: csi-cdsplugin
              securityContext:
                privileged: true
                capabilities:
                  add: ["SYS_ADMIN"]
                allowPrivilegeEscalation: true
              image: hub.baidubce.com/cce/cdsplugin:latest-1.11
              args :
                - "--nodeid=$(NODE_ID)"
                - "--endpoint=$(CSI_ENDPOINT)"
                - "--v=5"
                - "--drivername=csi-cdsplugin"
              env:
                - name: NODE_ID
                  valueFrom:
                    fieldRef:
                      fieldPath: spec.nodeName
                - name: CSI_ENDPOINT
                  value: unix://var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
              imagePullPolicy: "Always"
              volumeMounts:
                - name: cloud-config
                  mountPath: /etc/kubernetes
                - name: plugin-dir
                  mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
                - name: pods-mount-dir
                  mountPath: /var/lib/kubelet/pods
                  mountPropagation: "Bidirectional"
                - name: pods-mount-dir-data
                  mountPath: /data/kubelet/pods
                  mountPropagation: "Bidirectional"
                - mountPath: /sys
                  name: host-sys
                - name: lib-modules
                  mountPath: /lib/modules
                  readOnly: true
                - name: host-dev
                  mountPath: /dev
          volumes:
            - name: cloud-config
              hostPath:
                path: /etc/kubernetes
            - name: plugin-dir
              hostPath:
                path: /var/lib/kubelet/plugins/csi-cdsplugin
                type: DirectoryOrCreate
            - name: pods-mount-dir
              hostPath:
                path: /var/lib/kubelet/pods
                type: Directory
            - name: pods-mount-dir-data
              hostPath:
                path: /data/kubelet/pods
                type: DirectoryOrCreate
            - name: socket-dir
              hostPath:
                path: /var/lib/kubelet/plugins/csi-cdsplugin
                type: DirectoryOrCreate
            - name: host-sys
              hostPath:
                path: /sys
            - name: lib-modules
              hostPath:
                path: /lib/modules
            - name: host-dev
              hostPath:
                path: /dev
    ---
    kind: Service
    apiVersion: v1
    metadata:
      name: csi-attacher-cds
      namespace: kube-system
      labels:
        app: csi-attacher-cds
    spec:
      selector:
        app: csi-attacher-cds
      clusterIP: None
    ---
    kind: StatefulSet
    apiVersion: apps/v1beta1
    metadata:
      name: csi-attacher-cds
      namespace: kube-system
    spec:
      serviceName: "csi-attacher-cds"
      replicas: 2
      template:
        metadata:
          labels:
            app: csi-attacher-cds
        spec:
          serviceAccount: csi-external-runner
          containers:
            - name: csi-attacher-cds
              image: hub.baidubce.com/cce/csi-attacher:latest-1.11
              args:
                - "--v=5"
                - "--csi-address=$(ADDRESS)"
                - "--leader-election"
                - "--leader-election-namespace=$(MY_NAMESPACE)"
                - "--leader-election-identity=$(MY_NAME)"
              env:
                - name: MY_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.name
                - name: MY_NAMESPACE
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.namespace
                - name: ADDRESS
                  value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
              imagePullPolicy: "Always"
              volumeMounts:
                - name: socket-dir
                  mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
          volumes:
            - name: socket-dir
              hostPath:
                path: /var/lib/kubelet/plugins/csi-cdsplugin
                type: DirectoryOrCreate
    
    ---
    
    kind: Service
    apiVersion: v1
    metadata:
      name: csi-provisioner-cds
      namespace: kube-system
      labels:
        app: csi-provisioner-cds
    spec:
      selector:
        app: csi-provisioner-cds
      clusterIP: None
    
    ---
    kind: StatefulSet
    apiVersion: apps/v1beta1
    metadata:
      name: csi-provisioner-cds
      namespace: kube-system
    spec:
      serviceName: "csi-provisioner-cds"
      replicas: 1
      template:
        metadata:
          labels:
            app: csi-provisioner-cds
        spec:
          serviceAccount: csi-external-runner
          containers:
            - name: csi-provisioner-cds
              image: hub.baidubce.com/cce/csi-provisioner:latest-1.11
              args:
                - "--provisioner=csi-cdsplugin"
                - "--csi-address=$(ADDRESS)"
                - "--v=5"
              env:
                - name: ADDRESS
                  value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
              imagePullPolicy: "IfNotPresent"
              volumeMounts:
                - name: socket-dir
                  mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
          volumes:
            - name: socket-dir
              hostPath:
                path: /var/lib/kubelet/plugins/csi-cdsplugin
                type: DirectoryOrCreate
    
    ---
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: hp1               #名字可以自定义
    provisioner: csi-cdsplugin
    parameters:
      dynamicVolume: "true"       #需要为 true才会创建
      cdsSizeInGB: "40"           #对单个 storageClass能创建的 size限制
      paymentTiming: "Postpaid"   #Prepaid or Postpaid
      storageType: "hp1"          #支持std1 hp1 ssd
      reservationLength: "3"      #Prepaid 模式下需要填写
      zone: "xxx"                  #参数可选
    reclaimPolicy: Delete         #支持 Delete、Retain 默认值为 Delete

    1.13集群: dynamic-cds-template.yaml文件内容如下(1.13集群):

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: csi-external-runner
      namespace: kube-system
    ---
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: external-runner
    rules:
      - apiGroups: [""]
        resources: ["events"]
        verbs: ["get", "list", "watch", "update", "create", "patch"]
      - apiGroups: [""]
        resources: ["persistentvolumes"]
        verbs: ["get", "list", "watch", "update", "create", "delete"]
      - apiGroups: [""]
        resources: ["persistentvolumeclaims"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: [""]
        resources: ["nodes"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["volumeattachments"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: [""]
        resources: ["namespaces"]
        verbs: ["get", "list"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["storageclasses"]
        verbs: ["get", "list", "watch"]
      - apiGroups: ["snapshot.storage.k8s.io"]
        resources: ["*"]
        verbs: ["create", "get", "list", "watch", "update", "patch"]
      - apiGroups: ["apiextensions.k8s.io"]
        resources: ["*"]
        verbs: ["create", "get", "list", "watch"]
      - apiGroups: ["csi.storage.k8s.io"]
        resources: ["csidrivers"]
        verbs: ["create", "delete", "get", "list", "watch"]
      - apiGroups: ["csi.storage.k8s.io"]
        resources: ["csinodeinfos"]
        verbs: ["get", "list", "watch"]
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: csi-role-binding
    subjects:
      - kind: ServiceAccount
        name: csi-external-runner
        namespace: kube-system
    roleRef:
      kind: ClusterRole
      name: external-runner
      apiGroup: rbac.authorization.k8s.io
    ---
    kind: Role
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      namespace: kube-system
      name: external-runner-cfg
    rules:
    - apiGroups: [""]
      resources: ["configmaps"]
      verbs: ["get", "watch", "list", "delete", "update", "create"]
    ---
    kind: RoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: csi-role-cfg
      namespace: kube-system
    subjects:
      - kind: ServiceAccount
        name: csi-external-runner
        namespace: kube-system
    roleRef:
      kind: Role
      name: external-runner-cfg
      apiGroup: rbac.authorization.k8s.io
    ---
    apiVersion: extensions/v1beta1
    kind: DaemonSet
    metadata:
      name: csi-cdsplugin
      namespace: kube-system
    spec:
      selector:
        matchLabels:
          app: csi-cdsplugin
      template:
        metadata:
          labels:
            app: csi-cdsplugin
        spec:
          containers:
          - args:
            - --v=5
            - --csi-address=$(ADDRESS)
            - --kubelet-registration-path=$(ADDRESS)
            lifecycle:
              preStop:
                exec:
                  command: ["/bin/sh", "-c", "rm -rf /registration/csi-cdsplugin /registration/csi-cdsplugin-reg.sock"]
            env:
            - name: ADDRESS
              value: /var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
            image: hub.baidubce.com/cce/driver-registrar:latest-1.13
            imagePullPolicy: Always
            name: driver-registrar
            resources: {}
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            volumeMounts:
            - mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
              name: socket-dir
            - mountPath: /registration
              name: reg-dir
          - args:
            - --nodeid=$(NODE_ID)
            - --endpoint=$(CSI_ENDPOINT)
            - --v=5
            - --drivername=csi-cdsplugin
            env:
            - name: NODE_ID
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
            - name: CSI_ENDPOINT
              value: unix://var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
            image: hub.baidubce.com/cce/cdsplugin:latest-1.13
            imagePullPolicy: Always
            name: csi-cdsplugin
            securityContext:
              allowPrivilegeEscalation: true
              capabilities:
                add:
                - SYS_ADMIN
              privileged: true
            volumeMounts:
            - mountPath: /etc/kubernetes
              name: cloud-config
            - mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
              name: socket-dir
            - mountPath: /var/lib/kubelet/pods
              mountPropagation: Bidirectional
              name: pods-mount-dir
            - mountPath: /data/kubelet/pods
              mountPropagation: Bidirectional
              name: pods-mount-dir-data
            - mountPath: /sys
              name: host-sys
            - mountPath: /lib/modules
              name: lib-modules
              readOnly: true
            - mountPath: /dev
              name: host-dev
          hostNetwork: true
          restartPolicy: Always
          serviceAccount: csi-external-runner
          serviceAccountName: csi-external-runner
          priorityClassName: system-node-critical
          volumes:
          - hostPath:
              path: /etc/kubernetes
              type: ""
            name: cloud-config
          - hostPath:
              path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
              type: DirectoryOrCreate
            name: reg-dir
          - hostPath:
              path: /var/lib/kubelet/pods
              type: Directory
            name: pods-mount-dir
          - hostPath:
              path: /data/kubelet/pods
              type: Directory
            name: pods-mount-dir-data
          - hostPath:
              path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
              type: DirectoryOrCreate
            name: socket-dir
          - hostPath:
              path: /sys
              type: ""
            name: host-sys
          - hostPath:
              path: /lib/modules
              type: ""
            name: lib-modules
          - hostPath:
              path: /dev
              type: ""
            name: host-dev
    ---
    kind: Service
    apiVersion: v1
    metadata:
      name: csi-attacher-cds
      namespace: kube-system
      labels:
        app: csi-attacher-cds
    spec:
      selector:
        app: csi-attacher-cds
      clusterIP: None
    ---
    kind: StatefulSet
    apiVersion: apps/v1beta1
    metadata:
      name: csi-attacher-cds
      namespace: kube-system
    spec:
      serviceName: "csi-attacher-cds"
      replicas: 2
      template:
        metadata:
          labels:
            app: csi-attacher-cds
        spec:
          priorityClassName: system-cluster-critical
          serviceAccount: csi-external-runner
          containers:
            - name: csi-attacher-cds
              image: hub.baidubce.com/cce/csi-attacher:latest-1.13
              args:
                - "--v=5"
                - "--csi-address=$(ADDRESS)"
                - "--leader-election"
                - "--leader-election-namespace=$(MY_NAMESPACE)"
                - "--leader-election-identity=$(MY_NAME)"
              env:
                - name: MY_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.name
                - name: MY_NAMESPACE
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.namespace
                - name: ADDRESS
                  value: /var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
              imagePullPolicy: "Always"
              volumeMounts:
                - name: socket-dir
                  mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
          volumes:
            - name: socket-dir
              hostPath:
                path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
                type: DirectoryOrCreate
    ---
    kind: Service
    apiVersion: v1
    metadata:
      name: csi-provisioner-cds
      namespace: kube-system
      labels:
        app: csi-provisioner-cds
    spec:
      selector:
        app: csi-provisioner-cds
      clusterIP: None
    ---
    kind: StatefulSet
    apiVersion: apps/v1beta1
    metadata:
      name: csi-provisioner-cds
      namespace: kube-system
    spec:
      serviceName: "csi-provisioner-cds"
      replicas: 1
      template:
        metadata:
          labels:
            app: csi-provisioner-cds
        spec:
          priorityClassName: system-cluster-critical
          serviceAccount: csi-external-runner
          containers:
            - name: csi-provisioner-cds
              image: hub.baidubce.com/cce/csi-provisioner:latest-1.13
              args:
                - "--provisioner=csi-cdsplugin"
                - "--csi-address=$(ADDRESS)"
                - "--v=5"
                - "--feature-gates=Topology=true"
                - "--volume-name-prefix=pv"
              env:
                - name: ADDRESS
                  value: /var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
              imagePullPolicy: "Always"
              volumeMounts:
                - name: socket-dir
                  mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
          volumes:
            - name: socket-dir
              hostPath:
                path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
                type: DirectoryOrCreate
    ---
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: hp1               #名字可以自定义
    provisioner: csi-cdsplugin
    parameters:
      dynamicVolume: "true"       #需要为 true才会创建
      cdsSizeInGB: "40"           #对单个 storageClass能创建的 size限制
      paymentTiming: "Postpaid"   #Prepaid or Postpaid
      storageType: "hp1"          #支持std1 hp1 ssd
      reservationLength: "3"      #Prepaid 模式下需要填写
      zone: "xxx"                  #参数可选
    reclaimPolicy: Delete         #支持 Delete、Retain 默认值为 Delete

    1.16集群: dynamic-cds-template.yaml文件内容如下(1.16集群):

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: csi-external-runner
      namespace: kube-system
    ---
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: external-runner
    rules:
      - apiGroups: [""]
        resources: ["events"]
        verbs: ["get", "list", "watch", "update", "create", "patch"]
      - apiGroups: [""]
        resources: ["persistentvolumes"]
        verbs: ["get", "list", "watch", "update", "create", "delete", "patch"]
      - apiGroups: [""]
        resources: ["persistentvolumeclaims"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: [""]
        resources: ["nodes"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["volumeattachments"]
        verbs: ["get", "list", "watch", "update", "patch"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["volumeattachments/status"]
        verbs: ["patch"]
      - apiGroups: [""]
        resources: ["namespaces"]
        verbs: ["get", "list"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["storageclasses"]
        verbs: ["get", "list", "watch"]
      - apiGroups: ["snapshot.storage.k8s.io"]
        resources: ["*"]
        verbs: ["create", "get", "list", "watch", "update", "patch"]
      - apiGroups: ["apiextensions.k8s.io"]
        resources: ["*"]
        verbs: ["create", "get", "list", "watch"]
      - apiGroups: ["csi.storage.k8s.io"]
        resources: ["csidrivers"]
        verbs: ["create", "delete", "get", "list", "watch"]
      - apiGroups: ["csi.storage.k8s.io"]
        resources: ["csinodeinfos"]
        verbs: ["get", "list", "watch"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["csinodes"]
        verbs: ["get", "list", "watch"]
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: csi-role-binding
    subjects:
      - kind: ServiceAccount
        name: csi-external-runner
        namespace: kube-system
    roleRef:
      kind: ClusterRole
      name: external-runner
      apiGroup: rbac.authorization.k8s.io
    ---
    kind: Role
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      namespace: kube-system
      name: external-runner-cfg
    rules:
    - apiGroups: [""]
      resources: ["configmaps"]
      verbs: ["get", "watch", "list", "delete", "update", "create"]
    - apiGroups: ["coordination.k8s.io"]
      resources: ["leases"]
      verbs: ["get", "watch", "list", "delete", "update", "create"]
    ---
    kind: RoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: csi-role-cfg
      namespace: kube-system
    subjects:
      - kind: ServiceAccount
        name: csi-external-runner
        namespace: kube-system
    roleRef:
      kind: Role
      name: external-runner-cfg
      apiGroup: rbac.authorization.k8s.io
    ---
    apiVersion: extensions/v1beta1
    kind: DaemonSet
    metadata:
      name: csi-cdsplugin
      namespace: kube-system
    spec:
      selector:
        matchLabels:
          app: csi-cdsplugin
      template:
        metadata:
          labels:
            app: csi-cdsplugin
        spec:
          containers:
          - args:
            - --v=6
            - --csi-address=$(ADDRESS)
            - --kubelet-registration-path=$(ADDRESS)
            lifecycle:
              preStop:
                exec:
                  command: ["/bin/sh", "-c", "rm -rf /registration/csi-cdsplugin /registration/csi-cdsplugin-reg.sock"]
            env:
            - name: ADDRESS
              value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
            image: hub.baidubce.com/cce/csi-node-driver-registrar:latest-1.16
            imagePullPolicy: Always
            name: node-driver-registrar
            resources: {}
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            volumeMounts:
            - mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
              name: socket-dir
            - mountPath: /registration
              name: reg-dir
          - args:
            - --nodeid=$(NODE_ID)
            - --endpoint=$(CSI_ENDPOINT)
            - --v=5
            - --drivername=csi-cdsplugin
            env:
            - name: NODE_ID
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
            - name: CSI_ENDPOINT
              value: unix://var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
            image: hub.baidubce.com/cce/cdsplugin:latest-1.16
            imagePullPolicy: Always
            name: csi-cdsplugin
            securityContext:
              allowPrivilegeEscalation: true
              capabilities:
                add:
                - SYS_ADMIN
              privileged: true
            volumeMounts:
            - mountPath: /etc/kubernetes
              name: cloud-config
            - mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
              name: socket-dir
            - mountPath: /var/lib/kubelet/pods
              mountPropagation: Bidirectional
              name: pods-mount-dir
            - mountPath: /data/kubelet/pods
              mountPropagation: Bidirectional
              name: pods-mount-dir-data
            - mountPath: /sys
              name: host-sys
            - mountPath: /lib/modules
              name: lib-modules
              readOnly: true
            - mountPath: /dev
              name: host-dev
          hostNetwork: true
          restartPolicy: Always
          serviceAccount: csi-external-runner
          serviceAccountName: csi-external-runner
          priorityClassName: system-node-critical
          volumes:
          - hostPath:
              path: /etc/kubernetes
              type: ""
            name: cloud-config
          - hostPath:
              path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
              type: DirectoryOrCreate
            name: reg-dir
          - hostPath:
              path: /var/lib/kubelet/pods
              type: Directory
            name: pods-mount-dir
          - hostPath:
              path: /data/kubelet/pods
              type: Directory
            name: pods-mount-dir-data
          - hostPath:
              path: /var/lib/kubelet/plugins/csi-cdsplugin
              type: DirectoryOrCreate
            name: socket-dir
          - hostPath:
              path: /sys
              type: ""
            name: host-sys
          - hostPath:
              path: /lib/modules
              type: ""
            name: lib-modules
          - hostPath:
              path: /dev
              type: ""
            name: host-dev
    ---
    kind: Service
    apiVersion: v1
    metadata:
      name: csi-attacher-cds
      namespace: kube-system
      labels:
        app: csi-attacher-cds
    spec:
      selector:
        app: csi-attacher-cds
      clusterIP: None
    ---
    kind: StatefulSet
    apiVersion: apps/v1beta1
    metadata:
      name: csi-attacher-cds
      namespace: kube-system
    spec:
      serviceName: "csi-attacher-cds"
      replicas: 1
      template:
        metadata:
          labels:
            app: csi-attacher-cds
        spec:
          priorityClassName: system-cluster-critical
          serviceAccount: csi-external-runner
          containers:
            - name: csi-attacher-cds
              image: hub.baidubce.com/cce/csi-attacher:latest-1.16
              args:
                - "--v=5"
                - "--csi-address=$(ADDRESS)"
                - "--leader-election"
              env:
                - name: MY_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.name
                - name: MY_NAMESPACE
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.namespace
                - name: ADDRESS
                  value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
              imagePullPolicy: "Always"
              volumeMounts:
                - name: socket-dir
                  mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
          volumes:
            - name: socket-dir
              hostPath:
                path: /var/lib/kubelet/plugins/csi-cdsplugin
                type: DirectoryOrCreate
    ---
    kind: Service
    apiVersion: v1
    metadata:
      name: csi-provisioner-cds
      namespace: kube-system
      labels:
        app: csi-provisioner-cds
    spec:
      selector:
        app: csi-provisioner-cds
      clusterIP: None
    ---
    kind: StatefulSet
    apiVersion: apps/v1beta1
    metadata:
      name: csi-provisioner-cds
      namespace: kube-system
    spec:
      serviceName: "csi-provisioner-cds"
      replicas: 1
      template:
        metadata:
          labels:
            app: csi-provisioner-cds
        spec:
          priorityClassName: system-cluster-critical
          serviceAccount: csi-external-runner
          containers:
            - name: csi-provisioner-cds
              image: hub.baidubce.com/cce/csi-provisioner:latest-1.16
              args:
                - "--provisioner=csi-cdsplugin"
                - "--csi-address=$(ADDRESS)"
                - "--v=5"
                - "--feature-gates=Topology=true"
                - "--volume-name-prefix=pv"
              env:
                - name: ADDRESS
                  value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
              imagePullPolicy: "Always"
              volumeMounts:
                - name: socket-dir
                  mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
          volumes:
            - name: socket-dir
              hostPath:
                path: /var/lib/kubelet/plugins/csi-cdsplugin
                type: DirectoryOrCreate
    ---
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: hp1               #名字可以自定义
    provisioner: csi-cdsplugin
    parameters:
      dynamicVolume: "true"       #需要为 true才会创建
      cdsSizeInGB: "40"           #对单个 storageClass能创建的 size限制
      paymentTiming: "Postpaid"   #Prepaid or Postpaid
      storageType: "hp1"          #支持std1 hp1 ssd
      reservationLength: "3"      #Prepaid 模式下需要填写
      zone: "xxx"                  #参数可选
    reclaimPolicy: Delete         #支持 Delete、Retain 默认值为 Delete

    StorageClass 参数说明:

    • cdsSizeInGB

      cds 创建磁盘有限制: hp1: 5~32765GB std1: 5~32765GB ssd:50~32765GB cdsSizeInGB用于设置 StorageClass创建的cds 的最大值, PVC中设置值需要小于这个值

    • zone

      对于机器都在同一个 Zone 的情况,可以通过指定 StorageClass里的 Zone,保证cds 在指定的 Zone创建。 集群中节点所在可用区可以通过kubectl get nodes --show-labels查看failure-domain.beta.kubernetes.io/zone标签的值确定。 如: 可用区 A 的机器对应为 zoneA 可用区 B 的机器对应为 zoneB

    • paymentTiming

      用于设置付费方式是:预付费 or 后付费

    • reservationLength

      磁盘的保留时间,预付费模式下需要填写,单位为月

    bash
    $ kubectl apply -f dynamic-cds-template.yaml
    serviceaccount/csi-external-runner created
    clusterrole.rbac.authorization.k8s.io/external-runner configured
    clusterrolebinding.rbac.authorization.k8s.io/csi-role-binding configured
    role.rbac.authorization.k8s.io/external-runner-cfg configured
    rolebinding.rbac.authorization.k8s.io/csi-role-cfg configured
    daemonset.apps/csi-cdsplugin created
    service/csi-attacher-cds created
    statefulset.apps/csi-attacher-cds created
    service/csi-provisioner-cds created
    statefulset.apps/csi-provisioner-cds created
    storageclass.storage.k8s.io/csi-cds configured
    
    $ kubectl get pod --namespace kube-system  | grep provisioner
    kubectl get pod --namespace kube-system  | grep provisioner
    csi-provisioner-cds-0                    1/1       Running             0          10s
    
    $ kubectl get pod --namespace kube-system  | grep attacher
    csi-attacher-cds-0                       1/1       Running             0          20s
    csi-attacher-cds-1                       1/1       Running             0          20s

    如果相应的Pod进入Running状态,则动态绑定PV所需的资源已经建立成功。

    2.创建PVC时动态生成PV并绑定

    在PVC Spec中指定上面创建的StorageClass名称,则在创建PVC时,会自动调用相应StorageClass绑定的的Provisioner生成相应的PV进行绑定。

    使用kubectl,执行 kubectl create -f csi-pvc-cds.yaml 完成PVC的创建。

    假设创建的StorageClass名称为hp1,对应的 csi-pvc-cds.yaml 文件如下所示

    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      name: csi-pvc-cds
    spec:
      accessModes:
        - ReadWriteOnce
      storageClassName: hp1
      resources:
        requests:
          storage: 5Gi

    PVC参数说明: PVC 中 resource request的 storge,对于不同StorageClass中不同storageType有相关限制。 std1: 5~32765GB hp1: 5~32765GB ssd:50~32765GB 同时 PVC中申请的磁盘大小应该<= StorageClass 中cdsSizeInGB

    创建PVC后,可以看见相应的PV自动创建,PVC状态变为Bound,即PVC已经与新创建的PV绑定。

    $ kubectl create -f csi-pvc-cds.yaml
    persistentvolumeclaim "csi-pvc-cds" created
    $ kubectl get pvc
    NAME              STATUS    VOLUME                 CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    csi-pvc-cds       Bound     pvc-1ab36e4d1d2711e9   5Gi        RWX            hp1            4s
    $ kubectl get pv
    NAME                   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                     STORAGECLASS   REASON    AGE
    pvc-1ab36e4d1d2711e9   5Gi        RWX            Delete           Bound     default/csi-pvc-cds       hp1                      7s
    3.在Pod内挂载PVC

    在Pod spec内指定相应的PVC名称即可,使用kubectl,执行 kubectl create -f dynamic-cds-pod.yaml 完成资源的创建。

    对应的dynamic-cds-pod.yaml文件如下所示:

    apiVersion: v1
    kind: Pod
    metadata:
      name: test-pvc-pod
      labels:
        app: test-pvc-pod
    spec:
      containers:
      - name: test-pvc-pod
        image: nginx
        volumeMounts:
          - name: cds-pvc
            mountPath: "/cds-volume"
      volumes:
        - name: cds-pvc
          persistentVolumeClaim:
            claimName: csi-pvc-cds

    Pod创建后,可以读写容器内的/cds-volume路径来访问相应的cds存储上的内容。

    4.释放PVC时动态销毁绑定PV

    删除PVC时,与之绑定的动态PV是否会被自动删除由自定义的 StorageClass 中reclaimPolicy决定;reclaimPolicy 支持 Delete、Retain 默认值为 Delete。

    该值设置为 Delete 时:删除 PVC 会自动删除PV,并且动态申请的 cds 会被自动释放 该值设置为 Retain 时:删除 PVC时PV状态会变成 Released,不会自动删除PV,对应自动创建的 cds 需要到 cds console上手动释放

    $ kubectl delete pvc csi-pvc-cds
    
    persistentvolumeclaim "csi-pvc-cds" deleted  
    
    $ kubectl get pv  
    
    NAME                   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM        
    
    pvc-1ab36e4d1d2711e9   40Gi       RWO            Retain           Released   default/csi-pvc-cds   hp1                     3m
    5.在 StatefuleSet里通过 claimTemple 批量使用 pvc的场景,需要前置步骤为步骤1

    以 sts-claim.yaml为例

    apiVersion: apps/v1beta1
    kind: StatefulSet
    metadata:
      name: web
    spec:
      serviceName: "nginx"
      replicas: 2
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - name: nginx
            image: nginx
            ports:
            - containerPort: 80
              name: web
            volumeMounts:
            - name: www
              mountPath: /usr/share/nginx/html
      volumeClaimTemplates:
      - metadata:
          name: www
        spec:
          accessModes: [ "ReadWriteOnce" ]
          resources:
            requests:
              storage: 5Gi
          storageClassName: hp1

    创建 sts-claim.yaml,同时会自动创建 PVC

    $ kubectl apply -f sts-claim.yaml
    
    $ kubectl get pod
    NAME         READY     STATUS              RESTARTS   AGE
    web-0        1/1       Running             0          3m
    web-1        1/1       Running             0          2m
    
    $ kubectl get pvc
    NAME           STATUS    VOLUME                 CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    www-web-0      Bound     pvc-a1e885701d2f11e9   5Gi        RWO            hp1            6m
    www-web-1      Bound     pvc-c91edb891d2f11e9   5Gi        RWO            hp1            5m
    6.多可用区集群使用使用cds

    (1)StorageClass指定zone方式

    目前 cds不支持跨可用区,如果cce集群中有多个可用区的机器,动态创建cds需要通过设置StorageClass中zone字段和NodeSelector等指定可用区实现;使用已有cds,指定NodeSelector需要使用的可用区可实现。

    CCE集群默认会为集群节点打上可用区标签:

    $ kubectl get nodes --show-labels
    NAME            STATUS    ROLES     AGE       VERSION   LABELS
    192.168.80.15   Ready     <none>    13d       v1.8.12   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=BCC,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=bj,failure-domain.beta.kubernetes.io/zone=zoneC,kubernetes.io/hostname=192.168.80.15

    其中 failure-domain.beta.kubernetes.io/zone=zoneC 标签说明该集群节点处于C可用区。

    1)多可用区集群使用 cds例子

    StorageClass-Multi-Zone-Temple.yaml

    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: csi-cds-ssd-new
    provisioner: csi-cdsplugin
    parameters:
      dynamicVolume: "true"
      cdsSizeInGB: "40" 
      paymentTiming: "Postpaid"
      storageType: "hp1"
      reservationLength: "3"
      zone: "zoneA"
    reclaimPolicy: Delete

    Sts-Multi-Zone-Temple.yaml

    apiVersion: apps/v1beta1
    kind: StatefulSet
    metadata:
      name: sts-multi-zone
    spec:
      serviceName: "nginx"
      replicas: 1
      template:
        metadata:
          labels:
            app: nginx
        spec:
          nodeSelector:
            failure-domain.beta.kubernetes.io/zone: "zoneA"
          containers:
          - name: nginx
            image: nginx
            ports:
            - containerPort: 80
              name: web
            volumeMounts:
            - name: www
              mountPath: /usr/share/nginx/html
      volumeClaimTemplates:
      - metadata:
          name: www
        spec:
          accessModes: [ "ReadWriteOnce" ]
          resources:
            requests:
              storage: 40Gi
          storageClassName: csi-cds-ssd-new

    (2)设置 StorageClass的Volume Binding Mode 通过指定WaitForFirstConsumer和配置PersistentVolume 的模式来解决此问题,指定绑定模式为WaitForFirstConsumer,K8S会先进行调度 Pod,随后创建 CDS。

    1)使用示例

    StorageClass-WaitForFirstConsumer-Temple.yaml

    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: csi-cds
    provisioner: csi-cdsplugin
    volumeBindingMode: WaitForFirstConsumer
    parameters:
      dynamicVolume: "true"
      cdsSizeInGB: "40"
      paymentTiming: "Postpaid"
      storageType: "hp1"
    reclaimPolicy: Delete

    Sts-WaitForFirstConsumer-Temple.yaml

    apiVersion: apps/v1beta1
    kind: StatefulSet
    metadata:
      name: web
    spec:
      serviceName: "nginx"
      replicas: 1
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - name: nginx
            image: nginx
            ports:
            - containerPort: 80
              name: web
            volumeMounts:
            - name: www
              mountPath: /usr/share/nginx/html
      volumeClaimTemplates:
      - metadata:
          name: www
        spec:
          accessModes: [ "ReadWriteOnce" ]
          resources:
            requests:
              storage: 5Gi
          storageClassName: csi-cds

    具体详情请看相关文档 StorageClass

    注意:

    1. 如果需要销毁 cce集群并且使用了本文中的 csi-provisioner 同时 storageClass中reclaimPolicy设置为 Delete,请先 delete pvc,否则可能会存在cds 无法自动释放。
    2. 多可用区集群使用cds,如果同时设置了WaitForFirstConsumer和 zone,zone 将会被覆盖。

    在单个Pod中挂载多块CDS

    注意:

    • 在Pod Spec的volumes字段中,每块CDS只能被声明一次。
    • 在Pod Spec的volumeMounts字段中,每块CDS对应的volume可以被引用多次。
    • 对于Deployment/ReplicaSet等资源,replicas域只能设置为1。

    下面的示例 cds-multi-mount-example.yaml 将文件中声明的四块CDS磁盘分别挂载到Pod mysql-nginx内的my-nginx和my-mysql两个容器的相应路径下,其中test-data和test-data-2两个volume对应的CDS为两个容器共享,而nginx-data和db-data两个volume对应的CDS分别只挂载在各自容器中。

    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      name: mysql-nginx
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: mysql-nginx
      template:
        metadata:
          name: mysql-nginx
          labels:
            app: mysql-nginx
        spec:
          containers:
          - name: my-nginx
            image: nginx
            volumeMounts:
            - name: nginx-data
              mountPath: /nginx-data
            - name: test-data
              mountPath: /test-data
            - name: test-data-2
              mountPath: /test-data-2
            ports:
            - containerPort: 80
          - name: my-mysql
            image: mysql:5.7
            volumeMounts:
            - name: db-data
              mountPath: /db-data
            - name: test-data
              mountPath: /test-data
            - name: test-data-2
              mountPath: /test-data-2
            ports:
            - containerPort: 3306
            env:
            - name: MYSQL_ROOT_PASSWORD
              value: M6jshva8hTXAGnin
          volumes:
          - name: db-data
            flexVolume:
              driver: "baidubce/cds"
              fsType: "ext4"
              options:
                volumeID: "v-Hd8MUXA7"
          - name: nginx-data
            flexVolume:
              driver: "baidubce/cds"
              fsType: "ext4"
              options:
                volumeID: "v-OWEHtlSI"
          - name: test-data
            flexVolume:
              driver: "baidubce/cds"
              fsType: "ext4"
              options:
                volumeID: "v-9PhGVoMu"
          - name: test-data-2
            flexVolume:
              driver: "baidubce/cds"
              fsType: "ext4"
              options:
                volumeID: "v-SxjD482V"

    flexVolume字段说明:

    • fsType:CDS磁盘的文件系统,如果是新建的CDS磁盘,将会自动格式化该CDS磁盘为该文件系统
    • volumeID:CDS磁盘ID,您可以从百度智能云控制台的页面获取

    使用 kubectl create -f cds-multi-mount-example.yaml 创建相应的资源,相应的CDS磁盘将会被挂载到文件中声明的各个目录下。Pod在集群内发生漂移时,这些CDS磁盘将会跟随Pod一同迁移。

    1.13集群PVC快照功能

    使用此功能需要更新provsioner对应的yaml,加入对snapshotter功能

    1.新建snapshotter

    # provisioner + snapshotter
    kind: Service
    apiVersion: v1
    metadata:
      name: csi-provisioner-cds
      namespace: kube-system
      labels:
        app: csi-provisioner-cds
    spec:
      selector:
        app: csi-provisioner-cds
      clusterIP: None
    ---
    kind: StatefulSet
    apiVersion: apps/v1beta1
    metadata:
      name: csi-provisioner-cds
      namespace: kube-system
    spec:
      serviceName: "csi-provisioner-cds"
      replicas: 1
      template:
        metadata:
          labels:
            app: csi-provisioner-cds
        spec:
          priorityClassName: system-cluster-critical
          serviceAccount: csi-external-runner
          containers:
            - name: csi-provisioner-cds
              image: hub.baidubce.com/cce/csi-provisioner:latest-1.13
              args:
                - "--provisioner=csi-cdsplugin"
                - "--csi-address=$(ADDRESS)"
                - "--v=5"
                - "--feature-gates=Topology=true"
                - "--volume-name-prefix=pv"
              env:
                - name: ADDRESS
                  value: /var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
              imagePullPolicy: "Always"
              volumeMounts:
                - name: socket-dir
                  mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
            - name: csi-snapshotter
              image: hub.baidubce.com/cce/csi-snapshotter:latest-1.13
              args:
                - --csi-address=$(ADDRESS)
                - --connection-timeout=15s
                - --v=5
              env:
                - name: ADDRESS
                  value: /var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
              imagePullPolicy: "Always"
              volumeMounts:
                - name: socket-dir
                  mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
          volumes:
            - name: socket-dir
              hostPath:
                path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
                type: DirectoryOrCreate

    2.新建snapshotclass

    # snapshot class
    apiVersion: snapshot.storage.k8s.io/v1alpha1
    kind: VolumeSnapshotClass
    metadata:
      name: csi-cdsplugin-snapclass #名称可变
    snapshotter: csi-cdsplugin

    3.创建snapshot

    # create shanpshot.yaml
    apiVersion: snapshot.storage.k8s.io/v1alpha1
    kind: VolumeSnapshot
    metadata:
      name: new-snapshot-demo
    spec:
      snapshotClassName: csi-cdsplugin-snapclass #需已创建
      source:
        name: csi-cds-pvc-test #pvc 必须存在,且必须为bound
        kind: PersistentVolumeClaim

    执行 kubectl apply -f shanpshot.yaml

    4.从snapshot创建pvc

    #restore
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: cds-restore
    spec:
      storageClassName: csi-cds
      dataSource:
        name: new-snapshot-demo 
        kind: VolumeSnapshot
        apiGroup: snapshot.storage.k8s.io
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 30Gi
          

    5.使用从snapshot恢复的pod

    # pod
    apiVersion: v1
    kind: Pod
    metadata:
      name: web-server-restore
      namespace: default
    spec:
      containers:
      - image: nginx
        imagePullPolicy: Always
        name: web-server
        volumeMounts:
        - mountPath: /var/lib/www/html
          name: mypvc
      volumes:
      - name: mypvc
        persistentVolumeClaim:
          claimName: cds-restore

    6.删除shapshot

    执行 kubectl delele volumesnapshots new-snapshot-demo 如果用 pvc 在使用该 snapshot, delete会导致terminating

    上一篇
    使用本地存储
    下一篇
    使用文件存储CFS