通过PV/PVC方式使用CDS

概述

容器的特性决定了容器本身是非持久化的,容器被删除,其上的数据也一并删除。cds可以解决容器的数据共享和持久化存储问题,适用于大数据分析、媒体、游戏等很多场景。
而PV(PersistentVolume)和PVC(PersistentVolumeClaim)是K8S提供的用于抽象存储细节的API资源。利用cds在集群内创建PV和PVC资源,用户可以直接将cds作为存储卷挂载到容器中,而无需关注底层的实现细节,从而更加便捷地为容器集群提供持久化存储方案。

准备工作

用户做好以下准备,才能在容器中挂载cds实例。

  • 注册百度智能云账号,并完成实名认证

  • 创建一个可用的容器集群。

注意:
需要k8s版本>=1.11

创建容器集群

  1. 创建一个容器集群,操作步骤参考创建集群
  2. 下载命令行客户端kubectl,并连接集群,操作步骤参考通过kubectl连接Kubernetes集群

操作指南

通过PV/PVC方式使用CDS,具体分为两种方式:

  • 静态挂载:需要用户提前在百度智能云中创建好CDS(操作方法参考CDS文档),然后通过CDS的volume id在集群中创建PV和PVC资源。
  • 动态挂载:用户在集群中声明PVC时,自动创建出CDS磁盘并且动态关联至PV。

静态挂载的方式需要用户提供的CDS磁盘处于未挂载状态,并且确保填入正确的volume id。动态挂载在部分场景下可能会更加方便,但是需要注意动态挂载时将会自动创建CDS磁盘并且计费,动态创建的CDS将会随着用户删除PVC时被联动删除(需要将对应storageClasss的reclaimPolicy参数设置为delete),但是如果用户在没有删除PVC的情况下直接删除了CCE集群,则需要手动对这些CDS进行处理。

静态PV/PVC方式挂载cds

1.创建CdsPlugin、Attacher

cds-template.yaml是一个yaml文件模板,包含了需要创建的集群资源信息,使用kubectl apply -f cds-template.yaml进行相关组件部署。

cds-template.yaml文件内容如下(1.11集群使用):

apiVersion: v1
kind: ServiceAccount
metadata:
  name: csi-external-runner
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: external-runner
rules:
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get", "list", "watch", "update", "create", "patch"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "update", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: [""]
    resources: ["namespaces"]
    verbs: ["get", "list"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-role-binding
subjects:
  - kind: ServiceAccount
    name: csi-external-runner
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: external-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: kube-system
  name: external-runner-cfg
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "watch", "list", "delete", "update", "create"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-role-cfg
  namespace: kube-system
subjects:
  - kind: ServiceAccount
    name: csi-external-runner
    namespace: kube-system
roleRef:
  kind: Role
  name: external-runner-cfg
  apiGroup: rbac.authorization.k8s.io
---
kind: DaemonSet
apiVersion: apps/v1beta2
metadata:
  name: csi-cdsplugin
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: csi-cdsplugin
  template:
    metadata:
      labels:
        app: csi-cdsplugin
    spec:
      serviceAccount: csi-external-runner
      hostNetwork: true
      containers:
        - name: driver-registrar
          image: hub.baidubce.com/jpaas-public/driver-registrar:v0.3.0
          args:
            - "--v=5"
            - "--csi-address=$(ADDRESS)"
          env:
            - name: ADDRESS
              value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          imagePullPolicy: "Always"
          volumeMounts:
            - name: socket-dir
              mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
        - name: csi-cdsplugin
          securityContext:
            privileged: true
            capabilities:
              add: ["SYS_ADMIN"]
            allowPrivilegeEscalation: true
          image: hub.baidubce.com/jpaas-public/cdsplugin:latest
          args :
            - "--nodeid=$(NODE_ID)"
            - "--endpoint=$(CSI_ENDPOINT)"
            - "--v=5"
            - "--drivername=csi-cdsplugin"
          env:
            - name: NODE_ID
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: CSI_ENDPOINT
              value: unix://var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
          imagePullPolicy: "Always"
          volumeMounts:
            - name: cloud-config
              mountPath: /etc/kubernetes
            - name: plugin-dir
              mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
            - name: pods-mount-dir
              mountPath: /var/lib/kubelet/pods
              mountPropagation: "Bidirectional"
            - name: pods-mount-dir-mnt
              mountPath: /mnt/kubelet/pods
              mountPropagation: "Bidirectional"
            - name: pods-mount-dir-data
              mountPath: /data/kubelet/pods
              mountPropagation: "Bidirectional"
            - mountPath: /sys
              name: host-sys
            - name: lib-modules
              mountPath: /lib/modules
              readOnly: true
            - name: host-dev
              mountPath: /dev
      volumes:
        - name: cloud-config
          hostPath:
            path: /etc/kubernetes
        - name: plugin-dir
          hostPath:
            path: /var/lib/kubelet/plugins/csi-cdsplugin
            type: DirectoryOrCreate
        - name: pods-mount-dir
          hostPath:
            path: /var/lib/kubelet/pods
            type: Directory
        - name: pods-mount-dir-data
          hostPath:
            path: /data/kubelet/pods
            type: DirectoryOrCreate
        - name: pods-mount-dir-mnt
          hostPath:
            path: /mnt/kubelet/pods
            type: DirectoryOrCreate
        - name: socket-dir
          hostPath:
            path: /var/lib/kubelet/plugins/csi-cdsplugin
            type: DirectoryOrCreate
        - name: host-sys
          hostPath:
            path: /sys
        - name: lib-modules
          hostPath:
            path: /lib/modules
        - name: host-dev
          hostPath:
            path: /dev
---
kind: Service
apiVersion: v1
metadata:
  name: csi-attacher-cds
  namespace: kube-system
  labels:
    app: csi-attacher-cds
spec:
  selector:
    app: csi-attacher-cds
  clusterIP: None
---
kind: StatefulSet
apiVersion: apps/v1beta1
metadata:
  name: csi-attacher-cds
  namespace: kube-system
spec:
  serviceName: "csi-attacher-cds"
  replicas: 2
  template:
    metadata:
      labels:
        app: csi-attacher-cds
    spec:
      serviceAccount: csi-external-runner
      containers:
        - name: csi-attacher-cds
          image: hub.baidubce.com/jpaas-public/csi-attacher:v0.3.0
          args:
            - "--v=5"
            - "--csi-address=$(ADDRESS)"
            - "--leader-election"
            - "--leader-election-namespace=$(MY_NAMESPACE)"
            - "--leader-election-identity=$(MY_NAME)"
          env:
            - name: MY_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: MY_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: ADDRESS
              value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
          imagePullPolicy: "Always"
          volumeMounts:
            - name: socket-dir
              mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
      volumes:
        - name: socket-dir
          hostPath:
            path: /var/lib/kubelet/plugins/csi-cdsplugin
            type: DirectoryOrCreate

cds-template.yaml文件内容如下(1.13集群使用):

apiVersion: v1
kind: ServiceAccount
metadata:
  name: csi-external-runner
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: external-runner
rules:
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get", "list", "watch", "update", "create", "patch"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "update", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: [""]
    resources: ["namespaces"]
    verbs: ["get", "list"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["*"]
    verbs: ["create", "get", "list", "watch", "update", "patch"]
  - apiGroups: ["apiextensions.k8s.io"]
    resources: ["*"]
    verbs: ["create", "get", "list", "watch"]
  - apiGroups: ["csi.storage.k8s.io"]
    resources: ["csidrivers"]
    verbs: ["create", "delete", "get", "list", "watch"]
  - apiGroups: ["csi.storage.k8s.io"]
    resources: ["csinodeinfos"]
    verbs: ["get", "list", "watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-role-binding
subjects:
  - kind: ServiceAccount
    name: csi-external-runner
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: external-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: kube-system
  name: external-runner-cfg
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "watch", "list", "delete", "update", "create"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-role-cfg
  namespace: kube-system
subjects:
  - kind: ServiceAccount
    name: csi-external-runner
    namespace: kube-system
roleRef:
  kind: Role
  name: external-runner-cfg
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: csi-cdsplugin
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: csi-cdsplugin
  template:
    metadata:
      labels:
        app: csi-cdsplugin
    spec:
      containers:
      - args:
        - --v=5
        - --csi-address=$(ADDRESS)
        - --kubelet-registration-path=$(ADDRESS)
        env:
        - name: ADDRESS
          value: /var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
        - name: KUBE_NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        image: hub.baidubce.com/jpaas-public/driver-registrar:v1.0.1
        imagePullPolicy: Always
        name: driver-registrar
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
          name: socket-dir
        - mountPath: /registration
          name: reg-dir
      - args:
        - --nodeid=$(NODE_ID)
        - --endpoint=$(CSI_ENDPOINT)
        - --v=5
        - --drivername=csi-cdsplugin
        env:
        - name: NODE_ID
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: CSI_ENDPOINT
          value: unix://var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
        image: hub.baidubce.com/jpaas-public/cdsplugin:v1.1.0.1
        imagePullPolicy: Always
        name: csi-cdsplugin
        securityContext:
          allowPrivilegeEscalation: true
          capabilities:
            add:
            - SYS_ADMIN
          privileged: true
        volumeMounts:
        - mountPath: /etc/kubernetes
          name: cloud-config
        - mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
          name: socket-dir
        - mountPath: /var/lib/kubelet/pods
          mountPropagation: Bidirectional
          name: pods-mount-dir
        - mountPath: /data/kubelet/pods
          mountPropagation: Bidirectional
          name: pods-mount-dir-data
        - mountPath: /sys
          name: host-sys
        - mountPath: /lib/modules
          name: lib-modules
          readOnly: true
        - mountPath: /dev
          name: host-dev
      hostNetwork: true
      restartPolicy: Always
      serviceAccount: csi-external-runner
      serviceAccountName: csi-external-runner
      priorityClassName: system-node-critical
      volumes:
      - hostPath:
          path: /etc/kubernetes
          type: ""
        name: cloud-config
      - hostPath:
          path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
          type: DirectoryOrCreate
        name: reg-dir
      - hostPath:
          path: /var/lib/kubelet/pods
          type: Directory
        name: pods-mount-dir
      - hostPath:
          path: /data/kubelet/pods
          type: Directory
        name: pods-mount-dir-data
      - hostPath:
          path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
          type: DirectoryOrCreate
        name: socket-dir
      - hostPath:
          path: /sys
          type: ""
        name: host-sys
      - hostPath:
          path: /lib/modules
          type: ""
        name: lib-modules
      - hostPath:
          path: /dev
          type: ""
        name: host-dev
---
kind: Service
apiVersion: v1
metadata:
  name: csi-attacher-cds
  namespace: kube-system
  labels:
    app: csi-attacher-cds
spec:
  selector:
    app: csi-attacher-cds
  clusterIP: None
---
kind: StatefulSet
apiVersion: apps/v1beta1
metadata:
  name: csi-attacher-cds
  namespace: kube-system
spec:
  serviceName: "csi-attacher-cds"
  replicas: 2
  template:
    metadata:
      labels:
        app: csi-attacher-cds
    spec:
      priorityClassName: system-cluster-critical
      serviceAccount: csi-external-runner
      containers:
        - name: csi-attacher-cds
          image: hub.baidubce.com/jpaas-public/csi-attacher:v1.0.1
          args:
            - "--v=5"
            - "--csi-address=$(ADDRESS)"
            - "--leader-election"
            - "--leader-election-namespace=$(MY_NAMESPACE)"
            - "--leader-election-identity=$(MY_NAME)"
          env:
            - name: MY_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: MY_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: ADDRESS
              value: /var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
          imagePullPolicy: "Always"
          volumeMounts:
            - name: socket-dir
              mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
      volumes:
        - name: socket-dir
          hostPath:
            path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
            type: DirectoryOrCreate

2.在集群中创建PV和PVC资源

使用kubectl,执行 kubectl create -f pv-cds.yaml 完成PV的创建

对应的pv-cds.yaml文件如下所示:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-cds
  namespace: "default"
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 5Gi
  csi:
    driver: "csi-cdsplugin"
    volumeHandle: "v-xxxx"
  persistentVolumeReclaimPolicy: Retain

注意: yaml中volumeHandle字段对应的是cds volume id(短 id)

创建PV后,输入kubectl get pv可以看见一个available状态的PV,如下所示:

$ kubectl get pv
NAME                   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                     STORAGECLASS   REASON    AGE
pv-cds                 5Gi        RWO            Retain           Available                                                      22s

建立一个能够与该PV绑定的PVC

使用kubectl,执行 kubectl create -f pvc-cds.yaml完成PVC的创建

对应的pvc-cds.yaml文件如下所示:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-cds-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

绑定前,PVC为pending状态

$ kubectl get pvc
NAME      STATUS    VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS   AGE
cds-pvc   Pending                                                      2s                                                  2s

绑定后,PV和PVC状态变为Bound

$ kubectl get pv
NAME      CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                  STORAGECLASS   REASON    AGE
pv-cds    5Gi        RWO            Retain           Bound     default/csi-cds-pvc                             4s
$ kubectl get pvc
NAME           STATUS    VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS   AGE
csi-cds-pvc    Bound     pv-cds    5Gi        RWO                           36s

有关PV和PVC的更多设置和字段说明,见k8s官方文档

3.在Pod内挂载PVC

在Pod spec内指定相应的PVC名称即可,使用kubectl,执行 kubectl create -f demo-cds-rc.yaml 完成rc的创建

对应的demo-cds-rs.yaml文件如下所示:

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: web-server-rs
  labels:
    test: ws
spec:
  replicas: 1
  selector:
    matchLabels:
      se: ws
  template:
    metadata:
      labels:
        se: ws
      name: web-server
    spec:
      containers:
       - name: web-server
         image: nginx
         volumeMounts:
           - mountPath: /var/lib/www/html
             name: csi-cds-pvc
      volumes:
       - name: csi-cds-pvc
         persistentVolumeClaim:
           claimName: csi-cds-pvc

Pod创建后,可以读写容器内的/cds-volume路径来访问相应的cds存储上的内容。

由于创建PV和PVC时只支持accessModesReadWriteOnce,该PVC可以被一节点上的Pod挂载读写。

4.释放PV和PVC资源

完成存储资源的使用后,可以释放PVC和PV资源

使用以下命令可以释放PVC

$ kubectl delete -f  pvc-cds.yaml

释放PVC后,原来与之绑定的PV状态会变为Release,如下所示:

NAME      CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                  STORAGECLASS   REASON    AGE
pv-cds    5Gi        RWO            Retain           Released   default/csi-cds-pvc                             4m

输入以下指令释放PV资源

$ kubectl delete -f  pv-cds.yaml

动态PV/PVC方式挂载cds

1.创建StorageClass、Attacher、Provisioner

dynamic-cds-template.yaml是一个yaml文件模板,包含了需要创建的集群资源信息。

dynamic-cds-template.yaml文件内容如下(1.11集群):

apiVersion: v1
kind: ServiceAccount
metadata:
  name: csi-external-runner
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: external-runner
rules:
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get", "list", "watch", "update", "create", "patch"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "update", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: [""]
    resources: ["namespaces"]
    verbs: ["get", "list"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-role-binding
subjects:
  - kind: ServiceAccount
    name: csi-external-runner
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: external-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: kube-system
  name: external-runner-cfg
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "watch", "list", "delete", "update", "create"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-role-cfg
  namespace: kube-system
subjects:
  - kind: ServiceAccount
    name: csi-external-runner
    namespace: kube-system
roleRef:
  kind: Role
  name: external-runner-cfg
  apiGroup: rbac.authorization.k8s.io
---
kind: DaemonSet
apiVersion: apps/v1beta2
metadata:
  name: csi-cdsplugin
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: csi-cdsplugin
  template:
    metadata:
      labels:
        app: csi-cdsplugin
    spec:
      serviceAccount: csi-external-runner
      hostNetwork: true
      containers:
        - name: driver-registrar
          image: hub.baidubce.com/jpaas-public/driver-registrar:v0.3.0
          args:
            - "--v=5"
            - "--csi-address=$(ADDRESS)"
          env:
            - name: ADDRESS
              value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          imagePullPolicy: "Always"
          volumeMounts:
            - name: socket-dir
              mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
        - name: csi-cdsplugin
          securityContext:
            privileged: true
            capabilities:
              add: ["SYS_ADMIN"]
            allowPrivilegeEscalation: true
          image: hub.baidubce.com/jpaas-public/cdsplugin:latest
          args :
            - "--nodeid=$(NODE_ID)"
            - "--endpoint=$(CSI_ENDPOINT)"
            - "--v=5"
            - "--drivername=csi-cdsplugin"
          env:
            - name: NODE_ID
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: CSI_ENDPOINT
              value: unix://var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
          imagePullPolicy: "Always"
          volumeMounts:
            - name: cloud-config
              mountPath: /etc/kubernetes
            - name: plugin-dir
              mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
            - name: pods-mount-dir
              mountPath: /var/lib/kubelet/pods
              mountPropagation: "Bidirectional"
            - name: pods-mount-dir-data
              mountPath: /data/kubelet/pods
              mountPropagation: "Bidirectional"
            - mountPath: /sys
              name: host-sys
            - name: lib-modules
              mountPath: /lib/modules
              readOnly: true
            - name: host-dev
              mountPath: /dev
      volumes:
        - name: cloud-config
          hostPath:
            path: /etc/kubernetes
        - name: plugin-dir
          hostPath:
            path: /var/lib/kubelet/plugins/csi-cdsplugin
            type: DirectoryOrCreate
        - name: pods-mount-dir
          hostPath:
            path: /var/lib/kubelet/pods
            type: Directory
        - name: pods-mount-dir-data
          hostPath:
            path: /data/kubelet/pods
            type: DirectoryOrCreate
        - name: socket-dir
          hostPath:
            path: /var/lib/kubelet/plugins/csi-cdsplugin
            type: DirectoryOrCreate
        - name: host-sys
          hostPath:
            path: /sys
        - name: lib-modules
          hostPath:
            path: /lib/modules
        - name: host-dev
          hostPath:
            path: /dev
---
kind: Service
apiVersion: v1
metadata:
  name: csi-attacher-cds
  namespace: kube-system
  labels:
    app: csi-attacher-cds
spec:
  selector:
    app: csi-attacher-cds
  clusterIP: None
---
kind: StatefulSet
apiVersion: apps/v1beta1
metadata:
  name: csi-attacher-cds
  namespace: kube-system
spec:
  serviceName: "csi-attacher-cds"
  replicas: 2
  template:
    metadata:
      labels:
        app: csi-attacher-cds
    spec:
      serviceAccount: csi-external-runner
      containers:
        - name: csi-attacher-cds
          image: hub.baidubce.com/jpaas-public/csi-attacher:v0.3.0
          args:
            - "--v=5"
            - "--csi-address=$(ADDRESS)"
            - "--leader-election"
            - "--leader-election-namespace=$(MY_NAMESPACE)"
            - "--leader-election-identity=$(MY_NAME)"
          env:
            - name: MY_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: MY_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: ADDRESS
              value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
          imagePullPolicy: "Always"
          volumeMounts:
            - name: socket-dir
              mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
      volumes:
        - name: socket-dir
          hostPath:
            path: /var/lib/kubelet/plugins/csi-cdsplugin
            type: DirectoryOrCreate

---

kind: Service
apiVersion: v1
metadata:
  name: csi-provisioner-cds
  namespace: kube-system
  labels:
    app: csi-provisioner-cds
spec:
  selector:
    app: csi-provisioner-cds
  clusterIP: None

---
kind: StatefulSet
apiVersion: apps/v1beta1
metadata:
  name: csi-provisioner-cds
  namespace: kube-system
spec:
  serviceName: "csi-provisioner-cds"
  replicas: 1
  template:
    metadata:
      labels:
        app: csi-provisioner-cds
    spec:
      serviceAccount: csi-external-runner
      containers:
        - name: csi-provisioner-cds
          image: hub.baidubce.com/jpaas-public/csi-provisioner:v0.3.0
          args:
            - "--provisioner=csi-cdsplugin"
            - "--csi-address=$(ADDRESS)"
            - "--v=5"
          env:
            - name: ADDRESS
              value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
          imagePullPolicy: "IfNotPresent"
          volumeMounts:
            - name: socket-dir
              mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
      volumes:
        - name: socket-dir
          hostPath:
            path: /var/lib/kubelet/plugins/csi-cdsplugin
            type: DirectoryOrCreate

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: hp1               #名字可以自定义
provisioner: csi-cdsplugin
parameters:
  dynamicVolume: "true"       #需要为 true才会创建
  cdsSizeInGB: "40"           #对单个 storageClass能创建的 size限制
  paymentTiming: "Postpaid"   #Prepaid or Postpaid
  storageType: "hp1"          #支持std1 hp1 ssd
  reservationLength: "3"      #Prepaid 模式下需要填写
  zone:"xxx"                  #参数可选
reclaimPolicy: Delete         #支持 Delete、Retain 默认值为 Delete

dynamic-cds-template.yaml文件内容如下(1.13集群):

apiVersion: v1
kind: ServiceAccount
metadata:
  name: csi-external-runner
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: external-runner
rules:
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get", "list", "watch", "update", "create", "patch"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "update", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: [""]
    resources: ["namespaces"]
    verbs: ["get", "list"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["*"]
    verbs: ["create", "get", "list", "watch", "update", "patch"]
  - apiGroups: ["apiextensions.k8s.io"]
    resources: ["*"]
    verbs: ["create", "get", "list", "watch"]
  - apiGroups: ["csi.storage.k8s.io"]
    resources: ["csidrivers"]
    verbs: ["create", "delete", "get", "list", "watch"]
  - apiGroups: ["csi.storage.k8s.io"]
    resources: ["csinodeinfos"]
    verbs: ["get", "list", "watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-role-binding
subjects:
  - kind: ServiceAccount
    name: csi-external-runner
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: external-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: kube-system
  name: external-runner-cfg
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "watch", "list", "delete", "update", "create"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-role-cfg
  namespace: kube-system
subjects:
  - kind: ServiceAccount
    name: csi-external-runner
    namespace: kube-system
roleRef:
  kind: Role
  name: external-runner-cfg
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: csi-cdsplugin
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: csi-cdsplugin
  template:
    metadata:
      labels:
        app: csi-cdsplugin
    spec:
      containers:
      - args:
        - --v=5
        - --csi-address=$(ADDRESS)
        - --kubelet-registration-path=$(ADDRESS)
        env:
        - name: ADDRESS
          value: /var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
        - name: KUBE_NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        image: hub.baidubce.com/jpaas-public/driver-registrar:v1.0.1
        imagePullPolicy: Always
        name: driver-registrar
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
          name: socket-dir
        - mountPath: /registration
          name: reg-dir
      - args:
        - --nodeid=$(NODE_ID)
        - --endpoint=$(CSI_ENDPOINT)
        - --v=5
        - --drivername=csi-cdsplugin
        env:
        - name: NODE_ID
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: CSI_ENDPOINT
          value: unix://var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
        image: hub.baidubce.com/jpaas-public/cdsplugin:v1.1.0.1
        imagePullPolicy: Always
        name: csi-cdsplugin
        securityContext:
          allowPrivilegeEscalation: true
          capabilities:
            add:
            - SYS_ADMIN
          privileged: true
        volumeMounts:
        - mountPath: /etc/kubernetes
          name: cloud-config
        - mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
          name: socket-dir
        - mountPath: /var/lib/kubelet/pods
          mountPropagation: Bidirectional
          name: pods-mount-dir
        - mountPath: /data/kubelet/pods
          mountPropagation: Bidirectional
          name: pods-mount-dir-data
        - mountPath: /sys
          name: host-sys
        - mountPath: /lib/modules
          name: lib-modules
          readOnly: true
        - mountPath: /dev
          name: host-dev
      hostNetwork: true
      restartPolicy: Always
      serviceAccount: csi-external-runner
      serviceAccountName: csi-external-runner
      priorityClassName: system-node-critical
      volumes:
      - hostPath:
          path: /etc/kubernetes
          type: ""
        name: cloud-config
      - hostPath:
          path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
          type: DirectoryOrCreate
        name: reg-dir
      - hostPath:
          path: /var/lib/kubelet/pods
          type: Directory
        name: pods-mount-dir
      - hostPath:
          path: /data/kubelet/pods
          type: Directory
        name: pods-mount-dir-data
      - hostPath:
          path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
          type: DirectoryOrCreate
        name: socket-dir
      - hostPath:
          path: /sys
          type: ""
        name: host-sys
      - hostPath:
          path: /lib/modules
          type: ""
        name: lib-modules
      - hostPath:
          path: /dev
          type: ""
        name: host-dev
---
kind: Service
apiVersion: v1
metadata:
  name: csi-attacher-cds
  namespace: kube-system
  labels:
    app: csi-attacher-cds
spec:
  selector:
    app: csi-attacher-cds
  clusterIP: None
---
kind: StatefulSet
apiVersion: apps/v1beta1
metadata:
  name: csi-attacher-cds
  namespace: kube-system
spec:
  serviceName: "csi-attacher-cds"
  replicas: 2
  template:
    metadata:
      labels:
        app: csi-attacher-cds
    spec:
      priorityClassName: system-cluster-critical
      serviceAccount: csi-external-runner
      containers:
        - name: csi-attacher-cds
          image: hub.baidubce.com/jpaas-public/csi-attacher:v1.0.1
          args:
            - "--v=5"
            - "--csi-address=$(ADDRESS)"
            - "--leader-election"
            - "--leader-election-namespace=$(MY_NAMESPACE)"
            - "--leader-election-identity=$(MY_NAME)"
          env:
            - name: MY_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: MY_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: ADDRESS
              value: /var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
          imagePullPolicy: "Always"
          volumeMounts:
            - name: socket-dir
              mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
      volumes:
        - name: socket-dir
          hostPath:
            path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
            type: DirectoryOrCreate
---
kind: Service
apiVersion: v1
metadata:
  name: csi-provisioner-cds
  namespace: kube-system
  labels:
    app: csi-provisioner-cds
spec:
  selector:
    app: csi-provisioner-cds
  clusterIP: None
---
kind: StatefulSet
apiVersion: apps/v1beta1
metadata:
  name: csi-provisioner-cds
  namespace: kube-system
spec:
  serviceName: "csi-provisioner-cds"
  replicas: 1
  template:
    metadata:
      labels:
        app: csi-provisioner-cds
    spec:
      priorityClassName: system-cluster-critical
      serviceAccount: csi-external-runner
      containers:
        - name: csi-provisioner-cds
          image: hub.baidubce.com/jpaas-public/csi-provisioner:v1.0.1
          args:
            - "--provisioner=csi-cdsplugin"
            - "--csi-address=$(ADDRESS)"
            - "--v=5"
            - "--feature-gates=Topology=true"
            - "--volume-name-prefix=pv"
          env:
            - name: ADDRESS
              value: /var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
          imagePullPolicy: "Always"
          volumeMounts:
            - name: socket-dir
              mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
      volumes:
        - name: socket-dir
          hostPath:
            path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
            type: DirectoryOrCreate
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: hp1               #名字可以自定义
provisioner: csi-cdsplugin
parameters:
  dynamicVolume: "true"       #需要为 true才会创建
  cdsSizeInGB: "40"           #对单个 storageClass能创建的 size限制
  paymentTiming: "Postpaid"   #Prepaid or Postpaid
  storageType: "hp1"          #支持std1 hp1 ssd
  reservationLength: "3"      #Prepaid 模式下需要填写
  zone:"xxx"                  #参数可选
reclaimPolicy: Delete         #支持 Delete、Retain 默认值为 Delete

StorageClass 参数说明:

  1. cdsSizeInGB
    cds 创建磁盘有限制:
    hp1: 5~32765GB
    std1: 5~32765GB
    ssd:50~32765GB
    cdsSizeInGB用于设置 StorageClass创建的cds 的最大值, PVC中设置值需要小于这个值

  2. zone
    对于机器都在同一个 Zone 的情况,可以通过指定 StorageClass里的 Zone,保证cds 在指定的 Zone创建。
    集群中节点所在可用区可以通过kubectl get nodes --show-labels查看failure-domain.beta.kubernetes.io/zone标签的值确定。
    如:
    可用区 A 的机器对应为 zoneA
    可用区 B 的机器对应为 zoneB

  3. paymentTiming
    用于设置付费方式是:预付费 or 后付费

  4. reservationLength
    磁盘的保留时间,预付费模式下需要填写,单位为月
bash
$ kubectl apply -f dynamic-cds-template.yaml
serviceaccount/csi-external-runner created
clusterrole.rbac.authorization.k8s.io/external-runner configured
clusterrolebinding.rbac.authorization.k8s.io/csi-role-binding configured
role.rbac.authorization.k8s.io/external-runner-cfg configured
rolebinding.rbac.authorization.k8s.io/csi-role-cfg configured
daemonset.apps/csi-cdsplugin created
service/csi-attacher-cds created
statefulset.apps/csi-attacher-cds created
service/csi-provisioner-cds created
statefulset.apps/csi-provisioner-cds created
storageclass.storage.k8s.io/csi-cds configured

$ kubectl get pod --namespace kube-system  | grep provisioner
kubectl get pod --namespace kube-system  | grep provisioner
csi-provisioner-cds-0                    1/1       Running             0          10s

$ kubectl get pod --namespace kube-system  | grep attacher
csi-attacher-cds-0                       1/1       Running             0          20s
csi-attacher-cds-1                       1/1       Running             0          20s

如果相应的Pod进入Running状态,则动态绑定PV所需的资源已经建立成功。

2.创建PVC时动态生成PV并绑定

在PVC Spec中指定上面创建的StorageClass名称,则在创建PVC时,会自动调用相应StorageClass绑定的的Provisioner生成相应的PV进行绑定。

使用kubectl,执行 kubectl create -f csi-pvc-cds.yaml 完成PVC的创建。

假设创建的StorageClass名称为hp1,对应的 csi-pvc-cds.yaml 文件如下所示

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: csi-pvc-cds
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: hp1
  resources:
    requests:
      storage: 5Gi

PVC参数说明:
PVC 中 resource request的 storge,对于不同StorageClass中不同storageType有相关限制。
std1: 5~32765GB
hp1: 5~32765GB
ssd:50~32765GB
同时 PVC中申请的磁盘大小应该<= StorageClass 中cdsSizeInGB

创建PVC后,可以看见相应的PV自动创建,PVC状态变为Bound,即PVC已经与新创建的PV绑定。

$ kubectl create -f csi-pvc-cds.yaml
persistentvolumeclaim "csi-pvc-cds" created
$ kubectl get pvc
NAME              STATUS    VOLUME                 CAPACITY   ACCESS MODES   STORAGECLASS   AGE
csi-pvc-cds       Bound     pvc-1ab36e4d1d2711e9   5Gi        RWX            hp1            4s
$ kubectl get pv
NAME                   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                     STORAGECLASS   REASON    AGE
pvc-1ab36e4d1d2711e9   5Gi        RWX            Delete           Bound     default/csi-pvc-cds       hp1                      7s

3.在Pod内挂载PVC

在Pod spec内指定相应的PVC名称即可,使用kubectl,执行 kubectl create -f dynamic-cds-pod.yaml 完成资源的创建。

对应的dynamic-cds-pod.yaml文件如下所示:

apiVersion: v1
kind: Pod
metadata:
  name: test-pvc-pod
  labels:
    app: test-pvc-pod
spec:
  containers:
  - name: test-pvc-pod
    image: nginx
    volumeMounts:
      - name: cds-pvc
        mountPath: "/cds-volume"
  volumes:
    - name: cds-pvc
      persistentVolumeClaim:
        claimName: csi-pvc-cds

Pod创建后,可以读写容器内的/cds-volume路径来访问相应的cds存储上的内容。

4.释放PVC时动态销毁绑定PV

删除PVC时,与之绑定的动态PV是否会被自动删除由自定义的 StorageClass 中reclaimPolicy决定;reclaimPolicy 支持 Delete、Retain 默认值为 Delete。

该值设置为 Delete 时:删除 PVC 会自动删除PV,并且动态申请的 cds 会被自动释放
该值设置为 Retain 时:删除 PVC时PV状态会变成 Released,不会自动删除PV,对应自动创建的 cds 需要到 cds console上手动释放

$ kubectl delete pvc csi-pvc-cds

persistentvolumeclaim "csi-pvc-cds" deleted  

$ kubectl get pv  

NAME                   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM        

pvc-1ab36e4d1d2711e9   40Gi       RWO            Retain           Released   default/csi-pvc-cds   hp1                     3m

5.在 StatefuleSet里通过 claimTemple 批量使用 pvc的场景,需要前置步骤为步骤1
sts-claim.yaml为例

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: "nginx"
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi
      storageClassName: hp1

创建 sts-claim.yaml,同时会自动创建 PVC

$ kubectl apply -f sts-claim.yaml

$ kubectl get pod
NAME         READY     STATUS              RESTARTS   AGE
web-0        1/1       Running             0          3m
web-1        1/1       Running             0          2m

$ kubectl get pvc
NAME           STATUS    VOLUME                 CAPACITY   ACCESS MODES   STORAGECLASS   AGE
www-web-0      Bound     pvc-a1e885701d2f11e9   5Gi        RWO            hp1            6m
www-web-1      Bound     pvc-c91edb891d2f11e9   5Gi        RWO            hp1            5m

6.多可用区集群使用使用cds

(1)StorageClass指定zone方式

目前 cds不支持跨可用区,如果cce集群中有多个可用区的机器,动态创建cds需要通过设置StorageClass中zone字段和NodeSelector等指定可用区实现;使用已有cds,指定NodeSelector需要使用的可用区可实现。

1)多可用区集群使用 cds例子

StorageClass-Multi-Zone-Temple.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-cds-ssd-new
provisioner: csi-cdsplugin
parameters:
  dynamicVolume: "true"
  cdsSizeInGB: "40" 
  paymentTiming: "Postpaid"
  storageType: "hp1"
  reservationLength: "3"
  zone: "zoneA"
reclaimPolicy: Delete

Sts-Multi-Zone-Temple.yaml

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: sts-multi-zone
spec:
  serviceName: "nginx"
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      nodeSelector:
        failure-domain.beta.kubernetes.io/zone: "zoneA"
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 40Gi
      storageClassName: csi-cds-ssd-new

(2)设置 StorageClass的Volume Binding Mode
通过指定WaitForFirstConsumer和配置PersistentVolume 的模式来解决此问题,指定绑定模式为WaitForFirstConsumer,K8S会先进行调度 Pod,随后创建 CDS。

1)使用示例

StorageClass-WaitForFirstConsumer-Temple.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-cds
provisioner: csi-cdsplugin
volumeBindingMode: WaitForFirstConsumer
parameters:
  dynamicVolume: "true"
  cdsSizeInGB: "40"
  paymentTiming: "Postpaid"
  storageType: "hp1"
reclaimPolicy: Delete

Sts-WaitForFirstConsumer-Temple.yaml

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: "nginx"
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi
      storageClassName: csi-cds

具体详情请看相关文档 StorageClass

注意:
1 如果需要销毁 cce集群并且使用了本文中的 csi-provisioner 同时 storageClass中reclaimPolicy设置为 Delete,请先 delete pvc,否则可能会存在cds 无法自动释放
2 多可用区集群使用cds,如果同时设置了WaitForFirstConsumer和 zone,zone 将会被覆盖