使用云盘CDS
更新时间：2020-09-24
当前CDS磁盘以FlexVolume和CSI形式在集群中挂载。FlexVolume可以直接在Pod中挂载，也可以基于CSI创建PV和PVC存储资源后，再在Pod中挂载。两种挂载方案都可以满足容器内特定数据持久化的需求，两者详细的区别请参考kubernetes官方文档：
准备工作
用户做好以下准备，才能在容器中挂载cds实例。
注意： 需要k8s版本>=1.11
集群节点所处地域和可用区相同的CDS磁盘已创建
CDS磁盘状态为未挂载
创建容器集群
- 创建一个容器集群，操作步骤参考创建集群。
- 下载命令行客户端kubectl，并连接集群，操作步骤参考通过kubectl连接Kubernetes集群。
操作指南
通过Volume方式挂载
注意：
CDS同一时间只能挂载在一台bcc实例上。 CDS同一时间只能被一个Pod声明使用。 对于Deployment/ReplicaSet等资源，replicas域只能设置为1。 下面的示例 cds-example.yaml 将文件中声明的CDS磁盘挂载到Pod内my-nginx容器的/data路径下。
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: my-nginx
spec:
replicas: 1
selector:
matchLabels:
app: my-nginx
template:
metadata:
name: my-nginx
labels:
app: my-nginx
spec:
containers:
- name: my-nginx
image: nginx
volumeMounts:
- name: cds-data
mountPath: /data
ports:
- containerPort: 80
volumes:
- name: cds-data
flexVolume:
driver: "baidubce/cds"
fsType: "ext4"
options:
volumeID: "v-hTXAGtin"flexVolume字段说明：
fsType：CDS磁盘的文件系统，如果是新建的CDS磁盘，将会自动格式化该CDS磁盘为该文件系统 volumeID：CDS磁盘ID，您可以从百度智能云控制台的页面获取
使用 kubectl create -f cds-example.yaml 创建相应的资源，相应的CDS磁盘将会被挂载到my-nginx容器内的/data目录下。Pod在集群内发生漂移时，该CDS磁盘将会跟随Pod一同迁移。
PV/PVC
通过PV/PVC方式使用CDS，具体分为两种方式：
- 静态挂载：需要用户提前在百度智能云中创建好CDS（操作方法参考CDS文档），然后通过CDS的volume id在集群中创建PV和PVC资源。
- 动态挂载：用户在集群中声明PVC时，自动创建出CDS磁盘并且动态关联至PV。
静态挂载的方式需要用户提供的CDS磁盘处于未挂载状态，并且确保填入正确的volume id。动态挂载在部分场景下可能会更加方便，但是需要注意动态挂载时将会自动创建CDS磁盘并且计费，动态创建的CDS将会随着用户删除PVC时被联动删除（需要将对应storageClass的reclaimPolicy参数设置为delete），但是如果用户在没有删除PVC的情况下直接删除了CCE集群，则需要手动对这些CDS进行处理。
静态PV/PVC方式挂载cds
1.创建CdsPlugin、Attacher
cds-template.yaml是一个yaml文件模板，包含了需要创建的集群资源信息，使用kubectl apply -f cds-template.yaml进行相关组件部署。
1.11集群 cds-template.yaml文件内容如下(1.11集群使用)：
apiVersion: v1
kind: ServiceAccount
metadata:
name: csi-external-runner
namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: external-runner
rules:
- apiGroups: [""]
resources: ["events"]
verbs: ["get", "list", "watch", "update", "create", "patch"]
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "update", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: csi-role-binding
subjects:
- kind: ServiceAccount
name: csi-external-runner
namespace: kube-system
roleRef:
kind: ClusterRole
name: external-runner
apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: kube-system
name: external-runner-cfg
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "watch", "list", "delete", "update", "create"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: csi-role-cfg
namespace: kube-system
subjects:
- kind: ServiceAccount
name: csi-external-runner
namespace: kube-system
roleRef:
kind: Role
name: external-runner-cfg
apiGroup: rbac.authorization.k8s.io
---
kind: DaemonSet
apiVersion: apps/v1beta2
metadata:
name: csi-cdsplugin
namespace: kube-system
spec:
selector:
matchLabels:
app: csi-cdsplugin
template:
metadata:
labels:
app: csi-cdsplugin
spec:
serviceAccount: csi-external-runner
hostNetwork: true
containers:
- name: driver-registrar
image: hub.baidubce.com/cce/driver-registrar:latest-1.11
args:
- "--v=5"
- "--csi-address=$(ADDRESS)"
env:
- name: ADDRESS
value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
- name: KUBE_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
imagePullPolicy: "Always"
volumeMounts:
- name: socket-dir
mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
- name: csi-cdsplugin
securityContext:
privileged: true
capabilities:
add: ["SYS_ADMIN"]
allowPrivilegeEscalation: true
image: hub.baidubce.com/cce/cdsplugin:latest-1.11
args :
- "--nodeid=$(NODE_ID)"
- "--endpoint=$(CSI_ENDPOINT)"
- "--v=5"
- "--drivername=csi-cdsplugin"
env:
- name: NODE_ID
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: CSI_ENDPOINT
value: unix://var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
imagePullPolicy: "Always"
volumeMounts:
- name: cloud-config
mountPath: /etc/kubernetes
- name: plugin-dir
mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
- name: pods-mount-dir
mountPath: /var/lib/kubelet/pods
mountPropagation: "Bidirectional"
- name: pods-mount-dir-mnt
mountPath: /mnt/kubelet/pods
mountPropagation: "Bidirectional"
- name: pods-mount-dir-data
mountPath: /data/kubelet/pods
mountPropagation: "Bidirectional"
- mountPath: /sys
name: host-sys
- name: lib-modules
mountPath: /lib/modules
readOnly: true
- name: host-dev
mountPath: /dev
volumes:
- name: cloud-config
hostPath:
path: /etc/kubernetes
- name: plugin-dir
hostPath:
path: /var/lib/kubelet/plugins/csi-cdsplugin
type: DirectoryOrCreate
- name: pods-mount-dir
hostPath:
path: /var/lib/kubelet/pods
type: Directory
- name: pods-mount-dir-data
hostPath:
path: /data/kubelet/pods
type: DirectoryOrCreate
- name: pods-mount-dir-mnt
hostPath:
path: /mnt/kubelet/pods
type: DirectoryOrCreate
- name: socket-dir
hostPath:
path: /var/lib/kubelet/plugins/csi-cdsplugin
type: DirectoryOrCreate
- name: host-sys
hostPath:
path: /sys
- name: lib-modules
hostPath:
path: /lib/modules
- name: host-dev
hostPath:
path: /dev
---
kind: Service
apiVersion: v1
metadata:
name: csi-attacher-cds
namespace: kube-system
labels:
app: csi-attacher-cds
spec:
selector:
app: csi-attacher-cds
clusterIP: None
---
kind: StatefulSet
apiVersion: apps/v1beta1
metadata:
name: csi-attacher-cds
namespace: kube-system
spec:
serviceName: "csi-attacher-cds"
replicas: 2
template:
metadata:
labels:
app: csi-attacher-cds
spec:
serviceAccount: csi-external-runner
containers:
- name: csi-attacher-cds
image: hub.baidubce.com/cce/csi-attacher:latest-1.11
args:
- "--v=5"
- "--csi-address=$(ADDRESS)"
- "--leader-election"
- "--leader-election-namespace=$(MY_NAMESPACE)"
- "--leader-election-identity=$(MY_NAME)"
env:
- name: MY_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: MY_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: ADDRESS
value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
imagePullPolicy: "Always"
volumeMounts:
- name: socket-dir
mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
volumes:
- name: socket-dir
hostPath:
path: /var/lib/kubelet/plugins/csi-cdsplugin
type: DirectoryOrCreate1.13集群： cds-template.yaml文件内容如下(1.13集群使用)：
apiVersion: v1
kind: ServiceAccount
metadata:
name: csi-external-runner
namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: external-runner
rules:
- apiGroups: [""]
resources: ["events"]
verbs: ["get", "list", "watch", "update", "create", "patch"]
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "update", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["*"]
verbs: ["create", "get", "list", "watch", "update", "patch"]
- apiGroups: ["apiextensions.k8s.io"]
resources: ["*"]
verbs: ["create", "get", "list", "watch"]
- apiGroups: ["csi.storage.k8s.io"]
resources: ["csidrivers"]
verbs: ["create", "delete", "get", "list", "watch"]
- apiGroups: ["csi.storage.k8s.io"]
resources: ["csinodeinfos"]
verbs: ["get", "list", "watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: csi-role-binding
subjects:
- kind: ServiceAccount
name: csi-external-runner
namespace: kube-system
roleRef:
kind: ClusterRole
name: external-runner
apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: kube-system
name: external-runner-cfg
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "watch", "list", "delete", "update", "create"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: csi-role-cfg
namespace: kube-system
subjects:
- kind: ServiceAccount
name: csi-external-runner
namespace: kube-system
roleRef:
kind: Role
name: external-runner-cfg
apiGroup: rbac.authorization.k8s.io
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: csi-cdsplugin
namespace: kube-system
spec:
selector:
matchLabels:
app: csi-cdsplugin
template:
metadata:
labels:
app: csi-cdsplugin
spec:
containers:
- args:
- --v=5
- --csi-address=$(ADDRESS)
- --kubelet-registration-path=$(ADDRESS)
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "rm -rf /registration/csi-cdsplugin /registration/csi-cdsplugin-reg.sock"]
env:
- name: ADDRESS
value: /var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
- name: KUBE_NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
image: hub.baidubce.com/cce/driver-registrar:latest-1.13
imagePullPolicy: Always
name: driver-registrar
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
name: socket-dir
- mountPath: /registration
name: reg-dir
- args:
- --nodeid=$(NODE_ID)
- --endpoint=$(CSI_ENDPOINT)
- --v=5
- --drivername=csi-cdsplugin
env:
- name: NODE_ID
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: CSI_ENDPOINT
value: unix://var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
image: hub.baidubce.com/cce/cdsplugin:latest-1.13
imagePullPolicy: Always
name: csi-cdsplugin
securityContext:
allowPrivilegeEscalation: true
capabilities:
add:
- SYS_ADMIN
privileged: true
volumeMounts:
- mountPath: /etc/kubernetes
name: cloud-config
- mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
name: socket-dir
- mountPath: /var/lib/kubelet/pods
mountPropagation: Bidirectional
name: pods-mount-dir
- mountPath: /data/kubelet/pods
mountPropagation: Bidirectional
name: pods-mount-dir-data
- name: pods-mount-dir-mnt
mountPath: /mnt/kubelet/pods
mountPropagation: "Bidirectional"
- mountPath: /sys
name: host-sys
- mountPath: /lib/modules
name: lib-modules
readOnly: true
- mountPath: /dev
name: host-dev
hostNetwork: true
restartPolicy: Always
serviceAccount: csi-external-runner
serviceAccountName: csi-external-runner
priorityClassName: system-node-critical
volumes:
- hostPath:
path: /etc/kubernetes
type: ""
name: cloud-config
- hostPath:
path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
type: DirectoryOrCreate
name: reg-dir
- hostPath:
path: /var/lib/kubelet/pods
type: Directory
name: pods-mount-dir
- hostPath:
path: /data/kubelet/pods
type: Directory
name: pods-mount-dir-data
- name: pods-mount-dir-mnt
hostPath:
path: /mnt/kubelet/pods
type: DirectoryOrCreate
- hostPath:
path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
type: DirectoryOrCreate
name: socket-dir
- hostPath:
path: /sys
type: ""
name: host-sys
- hostPath:
path: /lib/modules
type: ""
name: lib-modules
- hostPath:
path: /dev
type: ""
name: host-dev
---
kind: Service
apiVersion: v1
metadata:
name: csi-attacher-cds
namespace: kube-system
labels:
app: csi-attacher-cds
spec:
selector:
app: csi-attacher-cds
clusterIP: None
---
kind: StatefulSet
apiVersion: apps/v1beta1
metadata:
name: csi-attacher-cds
namespace: kube-system
spec:
serviceName: "csi-attacher-cds"
replicas: 2
template:
metadata:
labels:
app: csi-attacher-cds
spec:
priorityClassName: system-cluster-critical
serviceAccount: csi-external-runner
containers:
- name: csi-attacher-cds
image: hub.baidubce.com/cce/csi-attacher:latest-1.13
args:
- "--v=5"
- "--csi-address=$(ADDRESS)"
- "--leader-election"
- "--leader-election-namespace=$(MY_NAMESPACE)"
- "--leader-election-identity=$(MY_NAME)"
env:
- name: MY_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: MY_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: ADDRESS
value: /var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
imagePullPolicy: "Always"
volumeMounts:
- name: socket-dir
mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
volumes:
- name: socket-dir
hostPath:
path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
type: DirectoryOrCreate1.16集群 cds-template.yaml文件内容如下(1.16集群使用)：
apiVersion: v1
kind: ServiceAccount
metadata:
name: csi-external-runner
namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: external-runner
rules:
- apiGroups: [""]
resources: ["events"]
verbs: ["get", "list", "watch", "update", "create", "patch"]
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "update", "create", "delete", "patch"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments"]
verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments/status"]
verbs: ["patch"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["*"]
verbs: ["create", "get", "list", "watch", "update", "patch"]
- apiGroups: ["apiextensions.k8s.io"]
resources: ["*"]
verbs: ["create", "get", "list", "watch"]
- apiGroups: ["csi.storage.k8s.io"]
resources: ["csidrivers"]
verbs: ["create", "delete", "get", "list", "watch"]
- apiGroups: ["csi.storage.k8s.io"]
resources: ["csinodeinfos"]
verbs: ["get", "list", "watch"]
- apiGroups: ["storage.k8s.io"]
resources: ["csinodes"]
verbs: ["get", "list", "watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: csi-role-binding
subjects:
- kind: ServiceAccount
name: csi-external-runner
namespace: kube-system
roleRef:
kind: ClusterRole
name: external-runner
apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: kube-system
name: external-runner-cfg
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "watch", "list", "delete", "update", "create"]
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["get", "watch", "list", "delete", "update", "create"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: csi-role-cfg
namespace: kube-system
subjects:
- kind: ServiceAccount
name: csi-external-runner
namespace: kube-system
roleRef:
kind: Role
name: external-runner-cfg
apiGroup: rbac.authorization.k8s.io
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: csi-cdsplugin
namespace: kube-system
spec:
selector:
matchLabels:
app: csi-cdsplugin
template:
metadata:
labels:
app: csi-cdsplugin
spec:
containers:
- args:
- --v=6
- --csi-address=$(ADDRESS)
- --kubelet-registration-path=$(ADDRESS)
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "rm -rf /registration/csi-cdsplugin /registration/csi-cdsplugin-reg.sock"]
env:
- name: ADDRESS
value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
- name: KUBE_NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
image: hub.baidubce.com/cce/csi-node-driver-registrar:latest-1.16
imagePullPolicy: Always
name: node-driver-registrar
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
name: socket-dir
- mountPath: /registration
name: reg-dir
- args:
- --nodeid=$(NODE_ID)
- --endpoint=$(CSI_ENDPOINT)
- --v=5
- --drivername=csi-cdsplugin
env:
- name: NODE_ID
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: CSI_ENDPOINT
value: unix://var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
image: hub.baidubce.com/cce/cdsplugin:latest-1.16
imagePullPolicy: Always
name: csi-cdsplugin
securityContext:
allowPrivilegeEscalation: true
capabilities:
add:
- SYS_ADMIN
privileged: true
volumeMounts:
- mountPath: /etc/kubernetes
name: cloud-config
- mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
name: socket-dir
- mountPath: /var/lib/kubelet/pods
mountPropagation: Bidirectional
name: pods-mount-dir
- mountPath: /data/kubelet/pods
mountPropagation: Bidirectional
name: pods-mount-dir-data
- name: pods-mount-dir-mnt
mountPath: /mnt/kubelet/pods
mountPropagation: "Bidirectional"
- mountPath: /sys
name: host-sys
- mountPath: /lib/modules
name: lib-modules
readOnly: true
- mountPath: /dev
name: host-dev
hostNetwork: true
restartPolicy: Always
serviceAccount: csi-external-runner
serviceAccountName: csi-external-runner
priorityClassName: system-node-critical
volumes:
- hostPath:
path: /etc/kubernetes
type: ""
name: cloud-config
- hostPath:
path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
type: DirectoryOrCreate
name: reg-dir
- hostPath:
path: /var/lib/kubelet/pods
type: Directory
name: pods-mount-dir
- hostPath:
path: /data/kubelet/pods
type: DirectoryOrCreate
name: pods-mount-dir-data
- name: pods-mount-dir-mnt
hostPath:
path: /mnt/kubelet/pods
type: DirectoryOrCreate
- hostPath:
path: /var/lib/kubelet/plugins/csi-cdsplugin
type: DirectoryOrCreate
name: socket-dir
- hostPath:
path: /sys
type: ""
name: host-sys
- hostPath:
path: /lib/modules
type: ""
name: lib-modules
- hostPath:
path: /dev
type: ""
name: host-dev
---
kind: Service
apiVersion: v1
metadata:
name: csi-attacher-cds
namespace: kube-system
labels:
app: csi-attacher-cds
spec:
selector:
app: csi-attacher-cds
clusterIP: None
---
kind: StatefulSet
apiVersion: apps/v1beta1
metadata:
name: csi-attacher-cds
namespace: kube-system
spec:
serviceName: "csi-attacher-cds"
replicas: 1
template:
metadata:
labels:
app: csi-attacher-cds
spec:
priorityClassName: system-cluster-critical
serviceAccount: csi-external-runner
containers:
- name: csi-attacher-cds
image: hub.baidubce.com/cce/csi-attacher:latest-1.16
args:
- "--v=5"
- "--csi-address=$(ADDRESS)"
- "--leader-election"
env:
- name: MY_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: MY_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: ADDRESS
value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
imagePullPolicy: "Always"
volumeMounts:
- name: socket-dir
mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
volumes:
- name: socket-dir
hostPath:
path: /var/lib/kubelet/plugins/csi-cdsplugin
type: DirectoryOrCreate2.在集群中创建PV和PVC资源
使用kubectl，执行 kubectl create -f pv-cds.yaml 完成PV的创建
对应的pv-cds.yaml文件如下所示：
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-cds
namespace: "default"
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 5Gi
csi:
driver: "csi-cdsplugin"
volumeHandle: "v-xxxx"
persistentVolumeReclaimPolicy: Retain注意： yaml中volumeHandle字段对应的是cds volume id(短 id)
创建PV后，输入kubectl get pv可以看见一个available状态的PV，如下所示：
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pv-cds 5Gi RWO Retain Available 22s建立一个能够与该PV绑定的PVC
使用kubectl，执行 kubectl create -f pvc-cds.yaml完成PVC的创建
对应的pvc-cds.yaml文件如下所示：
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: csi-cds-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi绑定前，PVC为pending状态
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
cds-pvc Pending 2s 绑定后，PV和PVC状态变为Bound
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pv-cds 5Gi RWO Retain Bound default/csi-cds-pvc 4s
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
csi-cds-pvc Bound pv-cds 5Gi RWO 36s有关PV和PVC的更多设置和字段说明，见k8s官方文档
3.在Pod内挂载PVC
在Pod spec内指定相应的PVC名称即可，使用kubectl，执行 kubectl create -f demo-cds-rc.yaml 完成rc的创建
对应的demo-cds-rc.yaml文件如下所示：
apiVersion: apps/v1
kind: ReplicaSet
metadata:
name: web-server-rs
labels:
test: ws
spec:
replicas: 1
selector:
matchLabels:
se: ws
template:
metadata:
labels:
se: ws
name: web-server
spec:
containers:
- name: web-server
image: nginx
volumeMounts:
- mountPath: /var/lib/www/html
name: csi-cds-pvc
volumes:
- name: csi-cds-pvc
persistentVolumeClaim:
claimName: csi-cds-pvcPod创建后，可以读写容器内的/cds-volume路径来访问相应的cds存储上的内容。
由于创建PV和PVC时只支持accessModes为ReadWriteOnce，该PVC可以被一节点上的Pod挂载读写。
4.释放PV和PVC资源
完成存储资源的使用后，可以释放PVC和PV资源
使用以下命令可以释放PVC
$ kubectl delete -f pvc-cds.yaml
释放PVC后，原来与之绑定的PV状态会变为Release，如下所示：
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pv-cds 5Gi RWO Retain Released default/csi-cds-pvc 4m输入以下指令释放PV资源
$ kubectl delete -f pv-cds.yaml
动态PV/PVC方式挂载cds
1.创建StorageClass、Attacher、Provisioner
dynamic-cds-template.yaml是一个yaml文件模板，包含了需要创建的集群资源信息。
1.11集群： dynamic-cds-template.yaml文件内容如下（1.11集群）：
apiVersion: v1
kind: ServiceAccount
metadata:
name: csi-external-runner
namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: external-runner
rules:
- apiGroups: [""]
resources: ["events"]
verbs: ["get", "list", "watch", "update", "create", "patch"]
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "update", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: csi-role-binding
subjects:
- kind: ServiceAccount
name: csi-external-runner
namespace: kube-system
roleRef:
kind: ClusterRole
name: external-runner
apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: kube-system
name: external-runner-cfg
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "watch", "list", "delete", "update", "create"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: csi-role-cfg
namespace: kube-system
subjects:
- kind: ServiceAccount
name: csi-external-runner
namespace: kube-system
roleRef:
kind: Role
name: external-runner-cfg
apiGroup: rbac.authorization.k8s.io
---
kind: DaemonSet
apiVersion: apps/v1beta2
metadata:
name: csi-cdsplugin
namespace: kube-system
spec:
selector:
matchLabels:
app: csi-cdsplugin
template:
metadata:
labels:
app: csi-cdsplugin
spec:
serviceAccount: csi-external-runner
hostNetwork: true
containers:
- name: driver-registrar
image: hub.baidubce.com/cce/driver-registrar:latest-1.11
args:
- "--v=5"
- "--csi-address=$(ADDRESS)"
env:
- name: ADDRESS
value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
- name: KUBE_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
imagePullPolicy: "Always"
volumeMounts:
- name: socket-dir
mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
- name: csi-cdsplugin
securityContext:
privileged: true
capabilities:
add: ["SYS_ADMIN"]
allowPrivilegeEscalation: true
image: hub.baidubce.com/cce/cdsplugin:latest-1.11
args :
- "--nodeid=$(NODE_ID)"
- "--endpoint=$(CSI_ENDPOINT)"
- "--v=5"
- "--drivername=csi-cdsplugin"
env:
- name: NODE_ID
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: CSI_ENDPOINT
value: unix://var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
imagePullPolicy: "Always"
volumeMounts:
- name: cloud-config
mountPath: /etc/kubernetes
- name: plugin-dir
mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
- name: pods-mount-dir
mountPath: /var/lib/kubelet/pods
mountPropagation: "Bidirectional"
- name: pods-mount-dir-data
mountPath: /data/kubelet/pods
mountPropagation: "Bidirectional"
- mountPath: /sys
name: host-sys
- name: lib-modules
mountPath: /lib/modules
readOnly: true
- name: host-dev
mountPath: /dev
volumes:
- name: cloud-config
hostPath:
path: /etc/kubernetes
- name: plugin-dir
hostPath:
path: /var/lib/kubelet/plugins/csi-cdsplugin
type: DirectoryOrCreate
- name: pods-mount-dir
hostPath:
path: /var/lib/kubelet/pods
type: Directory
- name: pods-mount-dir-data
hostPath:
path: /data/kubelet/pods
type: DirectoryOrCreate
- name: socket-dir
hostPath:
path: /var/lib/kubelet/plugins/csi-cdsplugin
type: DirectoryOrCreate
- name: host-sys
hostPath:
path: /sys
- name: lib-modules
hostPath:
path: /lib/modules
- name: host-dev
hostPath:
path: /dev
---
kind: Service
apiVersion: v1
metadata:
name: csi-attacher-cds
namespace: kube-system
labels:
app: csi-attacher-cds
spec:
selector:
app: csi-attacher-cds
clusterIP: None
---
kind: StatefulSet
apiVersion: apps/v1beta1
metadata:
name: csi-attacher-cds
namespace: kube-system
spec:
serviceName: "csi-attacher-cds"
replicas: 2
template:
metadata:
labels:
app: csi-attacher-cds
spec:
serviceAccount: csi-external-runner
containers:
- name: csi-attacher-cds
image: hub.baidubce.com/cce/csi-attacher:latest-1.11
args:
- "--v=5"
- "--csi-address=$(ADDRESS)"
- "--leader-election"
- "--leader-election-namespace=$(MY_NAMESPACE)"
- "--leader-election-identity=$(MY_NAME)"
env:
- name: MY_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: MY_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: ADDRESS
value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
imagePullPolicy: "Always"
volumeMounts:
- name: socket-dir
mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
volumes:
- name: socket-dir
hostPath:
path: /var/lib/kubelet/plugins/csi-cdsplugin
type: DirectoryOrCreate
---
kind: Service
apiVersion: v1
metadata:
name: csi-provisioner-cds
namespace: kube-system
labels:
app: csi-provisioner-cds
spec:
selector:
app: csi-provisioner-cds
clusterIP: None
---
kind: StatefulSet
apiVersion: apps/v1beta1
metadata:
name: csi-provisioner-cds
namespace: kube-system
spec:
serviceName: "csi-provisioner-cds"
replicas: 1
template:
metadata:
labels:
app: csi-provisioner-cds
spec:
serviceAccount: csi-external-runner
containers:
- name: csi-provisioner-cds
image: hub.baidubce.com/cce/csi-provisioner:latest-1.11
args:
- "--provisioner=csi-cdsplugin"
- "--csi-address=$(ADDRESS)"
- "--v=5"
env:
- name: ADDRESS
value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: socket-dir
mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
volumes:
- name: socket-dir
hostPath:
path: /var/lib/kubelet/plugins/csi-cdsplugin
type: DirectoryOrCreate
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: hp1 #名字可以自定义
provisioner: csi-cdsplugin
parameters:
dynamicVolume: "true" #需要为 true才会创建
cdsSizeInGB: "40" #对单个 storageClass能创建的 size限制
paymentTiming: "Postpaid" #Prepaid or Postpaid
storageType: "hp1" #支持std1 hp1 ssd
reservationLength: "3" #Prepaid 模式下需要填写
zone: "xxx" #参数可选
reclaimPolicy: Delete #支持 Delete、Retain 默认值为 Delete1.13集群： dynamic-cds-template.yaml文件内容如下（1.13集群）：
apiVersion: v1
kind: ServiceAccount
metadata:
name: csi-external-runner
namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: external-runner
rules:
- apiGroups: [""]
resources: ["events"]
verbs: ["get", "list", "watch", "update", "create", "patch"]
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "update", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["*"]
verbs: ["create", "get", "list", "watch", "update", "patch"]
- apiGroups: ["apiextensions.k8s.io"]
resources: ["*"]
verbs: ["create", "get", "list", "watch"]
- apiGroups: ["csi.storage.k8s.io"]
resources: ["csidrivers"]
verbs: ["create", "delete", "get", "list", "watch"]
- apiGroups: ["csi.storage.k8s.io"]
resources: ["csinodeinfos"]
verbs: ["get", "list", "watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: csi-role-binding
subjects:
- kind: ServiceAccount
name: csi-external-runner
namespace: kube-system
roleRef:
kind: ClusterRole
name: external-runner
apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: kube-system
name: external-runner-cfg
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "watch", "list", "delete", "update", "create"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: csi-role-cfg
namespace: kube-system
subjects:
- kind: ServiceAccount
name: csi-external-runner
namespace: kube-system
roleRef:
kind: Role
name: external-runner-cfg
apiGroup: rbac.authorization.k8s.io
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: csi-cdsplugin
namespace: kube-system
spec:
selector:
matchLabels:
app: csi-cdsplugin
template:
metadata:
labels:
app: csi-cdsplugin
spec:
containers:
- args:
- --v=5
- --csi-address=$(ADDRESS)
- --kubelet-registration-path=$(ADDRESS)
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "rm -rf /registration/csi-cdsplugin /registration/csi-cdsplugin-reg.sock"]
env:
- name: ADDRESS
value: /var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
- name: KUBE_NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
image: hub.baidubce.com/cce/driver-registrar:latest-1.13
imagePullPolicy: Always
name: driver-registrar
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
name: socket-dir
- mountPath: /registration
name: reg-dir
- args:
- --nodeid=$(NODE_ID)
- --endpoint=$(CSI_ENDPOINT)
- --v=5
- --drivername=csi-cdsplugin
env:
- name: NODE_ID
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: CSI_ENDPOINT
value: unix://var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
image: hub.baidubce.com/cce/cdsplugin:latest-1.13
imagePullPolicy: Always
name: csi-cdsplugin
securityContext:
allowPrivilegeEscalation: true
capabilities:
add:
- SYS_ADMIN
privileged: true
volumeMounts:
- mountPath: /etc/kubernetes
name: cloud-config
- mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
name: socket-dir
- mountPath: /var/lib/kubelet/pods
mountPropagation: Bidirectional
name: pods-mount-dir
- mountPath: /data/kubelet/pods
mountPropagation: Bidirectional
name: pods-mount-dir-data
- mountPath: /sys
name: host-sys
- mountPath: /lib/modules
name: lib-modules
readOnly: true
- mountPath: /dev
name: host-dev
hostNetwork: true
restartPolicy: Always
serviceAccount: csi-external-runner
serviceAccountName: csi-external-runner
priorityClassName: system-node-critical
volumes:
- hostPath:
path: /etc/kubernetes
type: ""
name: cloud-config
- hostPath:
path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
type: DirectoryOrCreate
name: reg-dir
- hostPath:
path: /var/lib/kubelet/pods
type: Directory
name: pods-mount-dir
- hostPath:
path: /data/kubelet/pods
type: Directory
name: pods-mount-dir-data
- hostPath:
path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
type: DirectoryOrCreate
name: socket-dir
- hostPath:
path: /sys
type: ""
name: host-sys
- hostPath:
path: /lib/modules
type: ""
name: lib-modules
- hostPath:
path: /dev
type: ""
name: host-dev
---
kind: Service
apiVersion: v1
metadata:
name: csi-attacher-cds
namespace: kube-system
labels:
app: csi-attacher-cds
spec:
selector:
app: csi-attacher-cds
clusterIP: None
---
kind: StatefulSet
apiVersion: apps/v1beta1
metadata:
name: csi-attacher-cds
namespace: kube-system
spec:
serviceName: "csi-attacher-cds"
replicas: 2
template:
metadata:
labels:
app: csi-attacher-cds
spec:
priorityClassName: system-cluster-critical
serviceAccount: csi-external-runner
containers:
- name: csi-attacher-cds
image: hub.baidubce.com/cce/csi-attacher:latest-1.13
args:
- "--v=5"
- "--csi-address=$(ADDRESS)"
- "--leader-election"
- "--leader-election-namespace=$(MY_NAMESPACE)"
- "--leader-election-identity=$(MY_NAME)"
env:
- name: MY_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: MY_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: ADDRESS
value: /var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
imagePullPolicy: "Always"
volumeMounts:
- name: socket-dir
mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
volumes:
- name: socket-dir
hostPath:
path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
type: DirectoryOrCreate
---
kind: Service
apiVersion: v1
metadata:
name: csi-provisioner-cds
namespace: kube-system
labels:
app: csi-provisioner-cds
spec:
selector:
app: csi-provisioner-cds
clusterIP: None
---
kind: StatefulSet
apiVersion: apps/v1beta1
metadata:
name: csi-provisioner-cds
namespace: kube-system
spec:
serviceName: "csi-provisioner-cds"
replicas: 1
template:
metadata:
labels:
app: csi-provisioner-cds
spec:
priorityClassName: system-cluster-critical
serviceAccount: csi-external-runner
containers:
- name: csi-provisioner-cds
image: hub.baidubce.com/cce/csi-provisioner:latest-1.13
args:
- "--provisioner=csi-cdsplugin"
- "--csi-address=$(ADDRESS)"
- "--v=5"
- "--feature-gates=Topology=true"
- "--volume-name-prefix=pv"
env:
- name: ADDRESS
value: /var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
imagePullPolicy: "Always"
volumeMounts:
- name: socket-dir
mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
volumes:
- name: socket-dir
hostPath:
path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
type: DirectoryOrCreate
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: hp1 #名字可以自定义
provisioner: csi-cdsplugin
parameters:
dynamicVolume: "true" #需要为 true才会创建
cdsSizeInGB: "40" #对单个 storageClass能创建的 size限制
paymentTiming: "Postpaid" #Prepaid or Postpaid
storageType: "hp1" #支持std1 hp1 ssd
reservationLength: "3" #Prepaid 模式下需要填写
zone: "xxx" #参数可选
reclaimPolicy: Delete #支持 Delete、Retain 默认值为 Delete1.16集群： dynamic-cds-template.yaml文件内容如下（1.16集群）：
apiVersion: v1
kind: ServiceAccount
metadata:
name: csi-external-runner
namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: external-runner
rules:
- apiGroups: [""]
resources: ["events"]
verbs: ["get", "list", "watch", "update", "create", "patch"]
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "update", "create", "delete", "patch"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments"]
verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments/status"]
verbs: ["patch"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["*"]
verbs: ["create", "get", "list", "watch", "update", "patch"]
- apiGroups: ["apiextensions.k8s.io"]
resources: ["*"]
verbs: ["create", "get", "list", "watch"]
- apiGroups: ["csi.storage.k8s.io"]
resources: ["csidrivers"]
verbs: ["create", "delete", "get", "list", "watch"]
- apiGroups: ["csi.storage.k8s.io"]
resources: ["csinodeinfos"]
verbs: ["get", "list", "watch"]
- apiGroups: ["storage.k8s.io"]
resources: ["csinodes"]
verbs: ["get", "list", "watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: csi-role-binding
subjects:
- kind: ServiceAccount
name: csi-external-runner
namespace: kube-system
roleRef:
kind: ClusterRole
name: external-runner
apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: kube-system
name: external-runner-cfg
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "watch", "list", "delete", "update", "create"]
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["get", "watch", "list", "delete", "update", "create"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: csi-role-cfg
namespace: kube-system
subjects:
- kind: ServiceAccount
name: csi-external-runner
namespace: kube-system
roleRef:
kind: Role
name: external-runner-cfg
apiGroup: rbac.authorization.k8s.io
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: csi-cdsplugin
namespace: kube-system
spec:
selector:
matchLabels:
app: csi-cdsplugin
template:
metadata:
labels:
app: csi-cdsplugin
spec:
containers:
- args:
- --v=6
- --csi-address=$(ADDRESS)
- --kubelet-registration-path=$(ADDRESS)
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "rm -rf /registration/csi-cdsplugin /registration/csi-cdsplugin-reg.sock"]
env:
- name: ADDRESS
value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
- name: KUBE_NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
image: hub.baidubce.com/cce/csi-node-driver-registrar:latest-1.16
imagePullPolicy: Always
name: node-driver-registrar
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
name: socket-dir
- mountPath: /registration
name: reg-dir
- args:
- --nodeid=$(NODE_ID)
- --endpoint=$(CSI_ENDPOINT)
- --v=5
- --drivername=csi-cdsplugin
env:
- name: NODE_ID
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: CSI_ENDPOINT
value: unix://var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
image: hub.baidubce.com/cce/cdsplugin:latest-1.16
imagePullPolicy: Always
name: csi-cdsplugin
securityContext:
allowPrivilegeEscalation: true
capabilities:
add:
- SYS_ADMIN
privileged: true
volumeMounts:
- mountPath: /etc/kubernetes
name: cloud-config
- mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
name: socket-dir
- mountPath: /var/lib/kubelet/pods
mountPropagation: Bidirectional
name: pods-mount-dir
- mountPath: /data/kubelet/pods
mountPropagation: Bidirectional
name: pods-mount-dir-data
- mountPath: /sys
name: host-sys
- mountPath: /lib/modules
name: lib-modules
readOnly: true
- mountPath: /dev
name: host-dev
hostNetwork: true
restartPolicy: Always
serviceAccount: csi-external-runner
serviceAccountName: csi-external-runner
priorityClassName: system-node-critical
volumes:
- hostPath:
path: /etc/kubernetes
type: ""
name: cloud-config
- hostPath:
path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
type: DirectoryOrCreate
name: reg-dir
- hostPath:
path: /var/lib/kubelet/pods
type: Directory
name: pods-mount-dir
- hostPath:
path: /data/kubelet/pods
type: Directory
name: pods-mount-dir-data
- hostPath:
path: /var/lib/kubelet/plugins/csi-cdsplugin
type: DirectoryOrCreate
name: socket-dir
- hostPath:
path: /sys
type: ""
name: host-sys
- hostPath:
path: /lib/modules
type: ""
name: lib-modules
- hostPath:
path: /dev
type: ""
name: host-dev
---
kind: Service
apiVersion: v1
metadata:
name: csi-attacher-cds
namespace: kube-system
labels:
app: csi-attacher-cds
spec:
selector:
app: csi-attacher-cds
clusterIP: None
---
kind: StatefulSet
apiVersion: apps/v1beta1
metadata:
name: csi-attacher-cds
namespace: kube-system
spec:
serviceName: "csi-attacher-cds"
replicas: 1
template:
metadata:
labels:
app: csi-attacher-cds
spec:
priorityClassName: system-cluster-critical
serviceAccount: csi-external-runner
containers:
- name: csi-attacher-cds
image: hub.baidubce.com/cce/csi-attacher:latest-1.16
args:
- "--v=5"
- "--csi-address=$(ADDRESS)"
- "--leader-election"
env:
- name: MY_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: MY_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: ADDRESS
value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
imagePullPolicy: "Always"
volumeMounts:
- name: socket-dir
mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
volumes:
- name: socket-dir
hostPath:
path: /var/lib/kubelet/plugins/csi-cdsplugin
type: DirectoryOrCreate
---
kind: Service
apiVersion: v1
metadata:
name: csi-provisioner-cds
namespace: kube-system
labels:
app: csi-provisioner-cds
spec:
selector:
app: csi-provisioner-cds
clusterIP: None
---
kind: StatefulSet
apiVersion: apps/v1beta1
metadata:
name: csi-provisioner-cds
namespace: kube-system
spec:
serviceName: "csi-provisioner-cds"
replicas: 1
template:
metadata:
labels:
app: csi-provisioner-cds
spec:
priorityClassName: system-cluster-critical
serviceAccount: csi-external-runner
containers:
- name: csi-provisioner-cds
image: hub.baidubce.com/cce/csi-provisioner:latest-1.16
args:
- "--provisioner=csi-cdsplugin"
- "--csi-address=$(ADDRESS)"
- "--v=5"
- "--feature-gates=Topology=true"
- "--volume-name-prefix=pv"
env:
- name: ADDRESS
value: /var/lib/kubelet/plugins/csi-cdsplugin/csi.sock
imagePullPolicy: "Always"
volumeMounts:
- name: socket-dir
mountPath: /var/lib/kubelet/plugins/csi-cdsplugin
volumes:
- name: socket-dir
hostPath:
path: /var/lib/kubelet/plugins/csi-cdsplugin
type: DirectoryOrCreate
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: hp1 #名字可以自定义
provisioner: csi-cdsplugin
parameters:
dynamicVolume: "true" #需要为 true才会创建
cdsSizeInGB: "40" #对单个 storageClass能创建的 size限制
paymentTiming: "Postpaid" #Prepaid or Postpaid
storageType: "hp1" #支持std1 hp1 ssd
reservationLength: "3" #Prepaid 模式下需要填写
zone: "xxx" #参数可选
reclaimPolicy: Delete #支持 Delete、Retain 默认值为 DeleteStorageClass 参数说明：
cdsSizeInGB
cds 创建磁盘有限制: hp1: 5~32765GB std1: 5~32765GB ssd:50~32765GB cdsSizeInGB用于设置 StorageClass创建的cds 的最大值, PVC中设置值需要小于这个值
zone
对于机器都在同一个 Zone 的情况，可以通过指定 StorageClass里的 Zone，保证cds 在指定的 Zone创建。 集群中节点所在可用区可以通过kubectl get nodes --show-labels查看failure-domain.beta.kubernetes.io/zone标签的值确定。 如： 可用区 A 的机器对应为 zoneA 可用区 B 的机器对应为 zoneB
paymentTiming
用于设置付费方式是：预付费 or 后付费
reservationLength
磁盘的保留时间，预付费模式下需要填写，单位为月
bash
$ kubectl apply -f dynamic-cds-template.yaml
serviceaccount/csi-external-runner created
clusterrole.rbac.authorization.k8s.io/external-runner configured
clusterrolebinding.rbac.authorization.k8s.io/csi-role-binding configured
role.rbac.authorization.k8s.io/external-runner-cfg configured
rolebinding.rbac.authorization.k8s.io/csi-role-cfg configured
daemonset.apps/csi-cdsplugin created
service/csi-attacher-cds created
statefulset.apps/csi-attacher-cds created
service/csi-provisioner-cds created
statefulset.apps/csi-provisioner-cds created
storageclass.storage.k8s.io/csi-cds configured
$ kubectl get pod --namespace kube-system | grep provisioner
kubectl get pod --namespace kube-system | grep provisioner
csi-provisioner-cds-0 1/1 Running 0 10s
$ kubectl get pod --namespace kube-system | grep attacher
csi-attacher-cds-0 1/1 Running 0 20s
csi-attacher-cds-1 1/1 Running 0 20s如果相应的Pod进入Running状态，则动态绑定PV所需的资源已经建立成功。
2.创建PVC时动态生成PV并绑定
在PVC Spec中指定上面创建的StorageClass名称，则在创建PVC时，会自动调用相应StorageClass绑定的的Provisioner生成相应的PV进行绑定。
使用kubectl，执行 kubectl create -f csi-pvc-cds.yaml 完成PVC的创建。
假设创建的StorageClass名称为hp1，对应的 csi-pvc-cds.yaml 文件如下所示
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: csi-pvc-cds
spec:
accessModes:
- ReadWriteOnce
storageClassName: hp1
resources:
requests:
storage: 5GiPVC参数说明： PVC 中 resource request的 storge，对于不同StorageClass中不同storageType有相关限制。 std1: 5~32765GB hp1: 5~32765GB ssd:50~32765GB 同时 PVC中申请的磁盘大小应该<= StorageClass 中cdsSizeInGB
创建PVC后，可以看见相应的PV自动创建，PVC状态变为Bound，即PVC已经与新创建的PV绑定。
$ kubectl create -f csi-pvc-cds.yaml
persistentvolumeclaim "csi-pvc-cds" created
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
csi-pvc-cds Bound pvc-1ab36e4d1d2711e9 5Gi RWX hp1 4s
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-1ab36e4d1d2711e9 5Gi RWX Delete Bound default/csi-pvc-cds hp1 7s3.在Pod内挂载PVC
在Pod spec内指定相应的PVC名称即可，使用kubectl，执行 kubectl create -f dynamic-cds-pod.yaml 完成资源的创建。
对应的dynamic-cds-pod.yaml文件如下所示：
apiVersion: v1
kind: Pod
metadata:
name: test-pvc-pod
labels:
app: test-pvc-pod
spec:
containers:
- name: test-pvc-pod
image: nginx
volumeMounts:
- name: cds-pvc
mountPath: "/cds-volume"
volumes:
- name: cds-pvc
persistentVolumeClaim:
claimName: csi-pvc-cdsPod创建后，可以读写容器内的/cds-volume路径来访问相应的cds存储上的内容。
4.释放PVC时动态销毁绑定PV
删除PVC时，与之绑定的动态PV是否会被自动删除由自定义的 StorageClass 中reclaimPolicy决定；reclaimPolicy 支持 Delete、Retain 默认值为 Delete。
该值设置为 Delete 时：删除 PVC 会自动删除PV，并且动态申请的 cds 会被自动释放 该值设置为 Retain 时：删除 PVC时PV状态会变成 Released，不会自动删除PV,对应自动创建的 cds 需要到 cds console上手动释放
$ kubectl delete pvc csi-pvc-cds
persistentvolumeclaim "csi-pvc-cds" deleted
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM
pvc-1ab36e4d1d2711e9 40Gi RWO Retain Released default/csi-pvc-cds hp1 3m5.在 StatefuleSet里通过 claimTemple 批量使用 pvc的场景，需要前置步骤为步骤1
以 sts-claim.yaml为例
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: web
spec:
serviceName: "nginx"
replicas: 2
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: www
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 5Gi
storageClassName: hp1创建 sts-claim.yaml，同时会自动创建 PVC
$ kubectl apply -f sts-claim.yaml
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
web-0 1/1 Running 0 3m
web-1 1/1 Running 0 2m
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
www-web-0 Bound pvc-a1e885701d2f11e9 5Gi RWO hp1 6m
www-web-1 Bound pvc-c91edb891d2f11e9 5Gi RWO hp1 5m6.多可用区集群使用使用cds
(1)StorageClass指定zone方式
目前 cds不支持跨可用区，如果cce集群中有多个可用区的机器，动态创建cds需要通过设置StorageClass中zone字段和NodeSelector等指定可用区实现；使用已有cds，指定NodeSelector需要使用的可用区可实现。
CCE集群默认会为集群节点打上可用区标签：
$ kubectl get nodes --show-labels
NAME STATUS ROLES AGE VERSION LABELS
192.168.80.15 Ready <none> 13d v1.8.12 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=BCC,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=bj,failure-domain.beta.kubernetes.io/zone=zoneC,kubernetes.io/hostname=192.168.80.15其中 failure-domain.beta.kubernetes.io/zone=zoneC 标签说明该集群节点处于C可用区。
1）多可用区集群使用 cds例子
StorageClass-Multi-Zone-Temple.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-cds-ssd-new
provisioner: csi-cdsplugin
parameters:
dynamicVolume: "true"
cdsSizeInGB: "40"
paymentTiming: "Postpaid"
storageType: "hp1"
reservationLength: "3"
zone: "zoneA"
reclaimPolicy: DeleteSts-Multi-Zone-Temple.yaml
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: sts-multi-zone
spec:
serviceName: "nginx"
replicas: 1
template:
metadata:
labels:
app: nginx
spec:
nodeSelector:
failure-domain.beta.kubernetes.io/zone: "zoneA"
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: www
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 40Gi
storageClassName: csi-cds-ssd-new(2)设置 StorageClass的Volume Binding Mode 通过指定WaitForFirstConsumer和配置PersistentVolume 的模式来解决此问题，指定绑定模式为WaitForFirstConsumer，K8S会先进行调度 Pod，随后创建 CDS。
1）使用示例
StorageClass-WaitForFirstConsumer-Temple.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-cds
provisioner: csi-cdsplugin
volumeBindingMode: WaitForFirstConsumer
parameters:
dynamicVolume: "true"
cdsSizeInGB: "40"
paymentTiming: "Postpaid"
storageType: "hp1"
reclaimPolicy: DeleteSts-WaitForFirstConsumer-Temple.yaml
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: web
spec:
serviceName: "nginx"
replicas: 1
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: www
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 5Gi
storageClassName: csi-cds具体详情请看相关文档 StorageClass
注意：
- 如果需要销毁 cce集群并且使用了本文中的 csi-provisioner 同时 storageClass中reclaimPolicy设置为 Delete，请先 delete pvc，否则可能会存在cds 无法自动释放。
- 多可用区集群使用cds，如果同时设置了WaitForFirstConsumer和 zone，zone 将会被覆盖。
在单个Pod中挂载多块CDS
注意：
- 在Pod Spec的volumes字段中，每块CDS只能被声明一次。
- 在Pod Spec的volumeMounts字段中，每块CDS对应的volume可以被引用多次。
- 对于Deployment/ReplicaSet等资源，replicas域只能设置为1。
下面的示例 cds-multi-mount-example.yaml 将文件中声明的四块CDS磁盘分别挂载到Pod mysql-nginx内的my-nginx和my-mysql两个容器的相应路径下，其中test-data和test-data-2两个volume对应的CDS为两个容器共享，而nginx-data和db-data两个volume对应的CDS分别只挂载在各自容器中。
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: mysql-nginx
spec:
replicas: 1
selector:
matchLabels:
app: mysql-nginx
template:
metadata:
name: mysql-nginx
labels:
app: mysql-nginx
spec:
containers:
- name: my-nginx
image: nginx
volumeMounts:
- name: nginx-data
mountPath: /nginx-data
- name: test-data
mountPath: /test-data
- name: test-data-2
mountPath: /test-data-2
ports:
- containerPort: 80
- name: my-mysql
image: mysql:5.7
volumeMounts:
- name: db-data
mountPath: /db-data
- name: test-data
mountPath: /test-data
- name: test-data-2
mountPath: /test-data-2
ports:
- containerPort: 3306
env:
- name: MYSQL_ROOT_PASSWORD
value: M6jshva8hTXAGnin
volumes:
- name: db-data
flexVolume:
driver: "baidubce/cds"
fsType: "ext4"
options:
volumeID: "v-Hd8MUXA7"
- name: nginx-data
flexVolume:
driver: "baidubce/cds"
fsType: "ext4"
options:
volumeID: "v-OWEHtlSI"
- name: test-data
flexVolume:
driver: "baidubce/cds"
fsType: "ext4"
options:
volumeID: "v-9PhGVoMu"
- name: test-data-2
flexVolume:
driver: "baidubce/cds"
fsType: "ext4"
options:
volumeID: "v-SxjD482V"flexVolume字段说明：
- fsType：CDS磁盘的文件系统，如果是新建的CDS磁盘，将会自动格式化该CDS磁盘为该文件系统
- volumeID：CDS磁盘ID，您可以从百度智能云控制台的页面获取
使用 kubectl create -f cds-multi-mount-example.yaml 创建相应的资源，相应的CDS磁盘将会被挂载到文件中声明的各个目录下。Pod在集群内发生漂移时，这些CDS磁盘将会跟随Pod一同迁移。
1.13集群PVC快照功能
使用此功能需要更新provsioner对应的yaml，加入对snapshotter功能
1.新建snapshotter
# provisioner + snapshotter
kind: Service
apiVersion: v1
metadata:
name: csi-provisioner-cds
namespace: kube-system
labels:
app: csi-provisioner-cds
spec:
selector:
app: csi-provisioner-cds
clusterIP: None
---
kind: StatefulSet
apiVersion: apps/v1beta1
metadata:
name: csi-provisioner-cds
namespace: kube-system
spec:
serviceName: "csi-provisioner-cds"
replicas: 1
template:
metadata:
labels:
app: csi-provisioner-cds
spec:
priorityClassName: system-cluster-critical
serviceAccount: csi-external-runner
containers:
- name: csi-provisioner-cds
image: hub.baidubce.com/cce/csi-provisioner:latest-1.13
args:
- "--provisioner=csi-cdsplugin"
- "--csi-address=$(ADDRESS)"
- "--v=5"
- "--feature-gates=Topology=true"
- "--volume-name-prefix=pv"
env:
- name: ADDRESS
value: /var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
imagePullPolicy: "Always"
volumeMounts:
- name: socket-dir
mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
- name: csi-snapshotter
image: hub.baidubce.com/cce/csi-snapshotter:latest-1.13
args:
- --csi-address=$(ADDRESS)
- --connection-timeout=15s
- --v=5
env:
- name: ADDRESS
value: /var/lib/kubelet/plugins_registry/csi-cdsplugin/csi.sock
imagePullPolicy: "Always"
volumeMounts:
- name: socket-dir
mountPath: /var/lib/kubelet/plugins_registry/csi-cdsplugin
volumes:
- name: socket-dir
hostPath:
path: /var/lib/kubelet/plugins_registry/csi-cdsplugin
type: DirectoryOrCreate2.新建snapshotclass
# snapshot class
apiVersion: snapshot.storage.k8s.io/v1alpha1
kind: VolumeSnapshotClass
metadata:
name: csi-cdsplugin-snapclass #名称可变
snapshotter: csi-cdsplugin3.创建snapshot
# create shanpshot.yaml
apiVersion: snapshot.storage.k8s.io/v1alpha1
kind: VolumeSnapshot
metadata:
name: new-snapshot-demo
spec:
snapshotClassName: csi-cdsplugin-snapclass #需已创建
source:
name: csi-cds-pvc-test #pvc 必须存在，且必须为bound
kind: PersistentVolumeClaim执行 kubectl apply -f shanpshot.yaml
4.从snapshot创建pvc
#restore
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: cds-restore
spec:
storageClassName: csi-cds
dataSource:
name: new-snapshot-demo
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 30Gi
5.使用从snapshot恢复的pod
# pod
apiVersion: v1
kind: Pod
metadata:
name: web-server-restore
namespace: default
spec:
containers:
- image: nginx
imagePullPolicy: Always
name: web-server
volumeMounts:
- mountPath: /var/lib/www/html
name: mypvc
volumes:
- name: mypvc
persistentVolumeClaim:
claimName: cds-restore6.删除shapshot
执行 kubectl delele volumesnapshots new-snapshot-demo
如果用 pvc 在使用该 snapshot, delete会导致terminating