在CCE集群中使用NetworkPolicy

更新时间：2021-07-09

NetworkPolicy 是 K8S 提供的一种资源，用于定义基于 Pod 的网络隔离策略。它描述了一组 Pod 能否与其它组 Pod 及其它 Endpoints 进行通信。本文主要演示如何使用开源工具 kube-router 在 CCE 上实现 NetworkPolicy 功能.

注意： kube-router 仅能搭配 kubenet 网络模式使用（详见 “VPC 网络”模式高级选项）

kube-router

kube-router 是一个 kubernetes 的容器网络解决方案，它的官网和代码地址如下：

官网：https://www.kube-router.io
项目：https://github.com/cloudnativelabs/kube-router

kube-router 有三大功能：

Pod Networking；
IPVS/LVS based service proxy；
Network Policy Controller.

CCE 有自己的容器网络实现方案，本文主要使用 kube-router 的 Network Policy Controller 的功能.

部署 kube-router

在 CCE K8S 集群上部署 kube-router ，YAML 如下：

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-router
  namespace: kube-system

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kube-router
  namespace: kube-system
rules:
  - apiGroups:
    - ""
    resources:
      - namespaces
      - pods
      - services
      - nodes
      - endpoints
    verbs:
      - list
      - get
      - watch
  - apiGroups:
    - "networking.k8s.io"
    resources:
      - networkpolicies
    verbs:
      - list
      - get
      - watch
  - apiGroups:
    - extensions
    resources:
      - networkpolicies
    verbs:
      - get
      - list
      - watch

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kube-router
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-router
subjects:
- kind: ServiceAccount
  name: kube-router
  namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-router-cfg
  namespace: kube-system
  labels:
    tier: node
    k8s-app: kube-router
data:
  cni-conf.json: |
    {
      "name":"kubernetes",
      "type":"bridge",
      "bridge":"kube-bridge",
      "isDefaultGateway":true,
      "ipam": {
        "type":"host-local"
      }
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-router
  namespace: kube-system
  labels:
    k8s-app: kube-router
spec:
  selector:
    matchLabels:
      k8s-app: kube-router
  template:
    metadata:
      labels:
        k8s-app: kube-router
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      serviceAccountName: kube-router
      containers:
      - name: kube-router
        image: registry.baidubce.com/cce-plugin-pro/kube-router:latest
        args: ["--run-router=false", "--run-firewall=true", "--run-service-proxy=false"]
        securityContext:
          privileged: true
        imagePullPolicy: Always
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        livenessProbe:
          httpGet:
            path: /healthz
            port: 20244
          initialDelaySeconds: 10
          periodSeconds: 3
        volumeMounts:
        - name: lib-modules
          mountPath: /lib/modules
          readOnly: true
        - name: cni-conf-dir
          mountPath: /etc/cni/net.d
      initContainers:
      - name: install-cni
        image: registry.baidubce.com/cce-plugin-pro/kube-router-busybox:latest
        imagePullPolicy: Always
        command:
        - /bin/sh
        - -c
        - set -e -x;
          if [ ! -f /etc/cni/net.d/10-kuberouter.conf ]; then
            TMP=/etc/cni/net.d/.tmp-kuberouter-cfg;
            cp /etc/kube-router/cni-conf.json ${TMP};
            mv ${TMP} /etc/cni/net.d/10-kuberouter.conf;
          fi
        volumeMounts:
        - name: cni-conf-dir
          mountPath: /etc/cni/net.d
        - name: kube-router-cfg
          mountPath: /etc/kube-router
      hostNetwork: true
      tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
        operator: Exists
      - effect: NoSchedule
        key: node.kubernetes.io/not-ready
        operator: Exists
      volumes:
      - name: lib-modules
        hostPath:
          path: /lib/modules
      - name: cni-conf-dir
        hostPath:
          path: /etc/cni/net.d
      - name: kube-router-cfg
        configMap:
          name: kube-router-cfg

例子说明

1 创建namespaces

$kubectl create namespace production
$kubectl create namespace staging

2 启动 nginx 服务

在不同的 namespace 中创建 nginx deployment.

$kubectl apply -f nginx.yaml --namespace=production
$kubectl apply -f nginx.yaml --namespace=staging

nginx.yaml 的 YAML 如下：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: hub.baidubce.com/cce/nginx-alpine-go:latest
        ports:
        - containerPort: 80

验证 Pod 启动成功：

# staging 环境
$kubectl get pods -n staging
NAME                                READY     STATUS    RESTARTS   AGE
nginx-deployment-7fbd5f4c55-2xgd4   1/1       Running   0          45s
nginx-deployment-7fbd5f4c55-5xr75   1/1       Running   0          45s
nginx-deployment-7fbd5f4c55-fn6lr   1/1       Running   0          20m

# productionn 环境
$kubectl get pods -n production
NAME                                READY     STATUS    RESTARTS   AGE
nginx-deployment-7fbd5f4c55-m764f   1/1       Running   0          10s
nginx-deployment-7fbd5f4c55-pdhhz   1/1       Running   0          10s
nginx-deployment-7fbd5f4c55-r98w5   1/1       Running   0          20m

没有设置 NetworkPolicy 的时候，所有的 Pod 是可以相互访问的，可以直接 ping PodIP.

Network Policy 策略测试

1. Default deny all ingress traffic

禁止 namespace=staging 中 pod 被访问.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: staging
spec:
  podSelector: {}
  policyTypes:
  - Ingress

各个字段含义说明：

PodSelector：选中需要隔离的 Pod；
policyTypes：策略类型，NetworkPolicy 将流量分为 ingress 和 egress，即入方向和出方向。如果没有指定则表示不闲置；
ingress：入方向，白名单，需要指定 from、ports，即来源、目的端口号，from有三种类型，ipBlock/namespaceSelector/podSelector；
egress：出方向，白名单，类似 ingress，egress 需要指定 to、ports，即目的、目的端口号。

上述 NetworkPolicy 创建完成后，可以在任意 Pod 中访问 namespace=staging 下的 PodIP，发现是无法访问，比如从 production 中的 pod 进行访问：

$kubectl exec -it nginx-deployment-7fbd5f4c55-m764f /bin/sh -n production
/ # ping 172.16.0.92
PING 172.16.0.92 (172.16.0.92): 56 data bytes

2. Default allow all ingress traffic

允许 namespace=staging 中 pod 被访问.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all
  namespace: staging
spec:
  podSelector: {}
  ingress:
  - {}
  policyTypes:
  - Ingress

3. Default deny all egress traffic

禁止 namespace=production 中 pod 对外访问.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Egress

4. Default allow all egress traffic

允许 namespace=production 中 pod 对外访问.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all
  namespace: production
spec:
  podSelector: {}
  egress:
  - {}
  policyTypes:
  - Egress

5. Default deny all ingress and all egress traffic

禁止所有 pod 的入和出的流量：

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

流量接入

创建VPC-CNI模式集群

容器引擎 CCE