Istio多集群系列-单控制面跨AZ集群区域负载均衡

kbsonlong 收录于云原生

2023-08-17 约 2300 字预计阅读 5 分钟 - 次阅读

基于安全容灾考虑，生产K8S集群是多可用区。多可用区集群必然涉及跨可用区调用，由于是海外业务，使用的是AWS云，AWS内部跨可用区流量不像国内云厂商跨可用区流量不收费，是需要收费而且特贵。

基于成本和容灾考虑使用Istio的地域负载均衡功能，优先调度到同可用区，出现问题时调度到另一可用区。

现使用 kind 在本地创建一套多节点集群，通过标签模拟多可用区场景。
由于办公电脑是M1，而Istio1.13并不支持M1,所以选用Istio1.16，K8S1.24版本

创建多节点集群模拟跨AZ

kind配置

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
mkdir -p multicluster
cd multicluster
cat << EOF > kind-cluster1.yaml
kind: Cluster
apiVersion: "kind.x-k8s.io/v1alpha4"
networking:
  apiServerAddress: "172.26.128.224" # 本机IP，可不设置。
  podSubnet: "10.10.0.0/16"
  serviceSubnet: "10.11.0.0/16"
nodes:
  - role: control-plane
    image: registry.cn-hangzhou.aliyuncs.com/seam/node:v1.24.15
  - role: worker
    image: registry.cn-hangzhou.aliyuncs.com/seam/node:v1.24.15
    kubeadmConfigPatches:
    - |
      kind: JoinConfiguration
      nodeRegistration:
        kubeletExtraArgs:
          node-labels: "topology.kubernetes.io/region=az01,topology.kubernetes.io/zone=az01"
  - role: worker
    image: registry.cn-hangzhou.aliyuncs.com/seam/node:v1.24.15
    kubeadmConfigPatches:
    - |
      kind: JoinConfiguration
      nodeRegistration:
        kubeletExtraArgs:
          node-labels: "topology.kubernetes.io/region=az02,topology.kubernetes.io/zone=az02"
EOF

创建集群

1
2
3
kind create cluster --name cluster1 --kubeconfig=istio-kubeconfig --config=kind-cluster1.yaml
kubectl config rename-context kind-cluster1  cluster1 --kubeconfig istio-kubeconfig
export KUBECONFIG=`pwd`/istio-kubeconfig

部署Istio

1
2
3
4
5
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.16.2 sh -
cd istio-1.16.2
export PATH="$PATH:`pwd`/bin"

istioctl install --set profile=demo -y

部署服务

sleep发起调用，访问helloworld

创建 `sleep` 部署

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: sample
  labels:
    istio-injection: enabled
EOF


cat << EOF | kubectl apply -f - -n sample
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sleep
---
apiVersion: v1
kind: Service
metadata:
  name: sleep
  labels:
    app: sleep
    service: sleep
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: sleep
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      terminationGracePeriodSeconds: 0
      serviceAccountName: sleep
      containers:
      - name: sleep
        image: curlimages/curl
        command: ["/bin/sleep", "infinity"]
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: /etc/sleep/tls
          name: secret-volume
      volumes:
      - name: secret-volume
        secret:
          secretName: sleep-secret
          optional: true
---
EOF

创建 `helloworld` 部署

生产环境目前只有部分业务开启mesh，所以为尽量模拟生产一致，关闭被调用端 helloworld 注入。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
cat << EOF | kubectl apply -f - -n sample
apiVersion: v1
kind: Service
metadata:
  name: helloworld
  labels:
    app: helloworld
    service: helloworld
spec:
  ports:
  - port: 5000
    name: http
  selector:
    app: helloworld
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloworld-az01
  labels:
    app: helloworld
    version: az01
spec:
  replicas: 1
  selector:
    matchLabels:
      app: helloworld
      version: az01
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
      labels:
        app: helloworld
        version: az01
    spec:
      nodeSelector:
        topology.kubernetes.io/zone: az01
      containers:
      - name: helloworld
        env:
        - name: SERVICE_VERSION
          value: az01
        image: docker.io/istio/examples-helloworld-v1
        resources:
          requests:
            cpu: "100m"
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 5000
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloworld-az02
  labels:
    app: helloworld
    version: az02
spec:
  replicas: 1
  selector:
    matchLabels:
      app: helloworld
      version: az02
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "false" # 被调用端关闭mesh
      labels:
        app: helloworld
        version: az02
    spec:
      nodeSelector:  # 通过nodeSelector将服务调度到不同节点模拟不同可用区
        topology.kubernetes.io/zone: az02
      containers:
      - name: helloworld
        env:
        - name: SERVICE_VERSION
          value: az02
        image: docker.io/istio/examples-helloworld-v1
        resources:
          requests:
            cpu: "100m"
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 5000
EOF

验证流量

可以看到请求默认是轮询

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
~ $ while true
> do
> curl -sSL helloworld.sample:5000/hello
> done
Hello version: az02, instance: helloworld-az02-c874d5486-srjnj
Hello version: az01, instance: helloworld-az01-5dd7bfb647-4zrpg
Hello version: az01, instance: helloworld-az01-5dd7bfb647-4zrpg
Hello version: az02, instance: helloworld-az02-c874d5486-srjnj
Hello version: az02, instance: helloworld-az02-c874d5486-srjnj
Hello version: az01, instance: helloworld-az01-5dd7bfb647-4zrpg
Hello version: az01, instance: helloworld-az01-5dd7bfb647-4zrpg
Hello version: az02, instance: helloworld-az02-c874d5486-srjnj
Hello version: az01, instance: helloworld-az01-5dd7bfb647-4zrpg
Hello version: az02, instance: helloworld-az02-c874d5486-srjnj
Hello version: az01, instance: helloworld-az01-5dd7bfb647-4zrpg
Hello version: az02, instance: helloworld-az02-c874d5486-srjnj
Hello version: az01, instance: helloworld-az01-5dd7bfb647-4zrpg
Hello version: az02, instance: helloworld-az02-c874d5486-srjnj
Hello version: az01, instance: helloworld-az01-5dd7bfb647-4zrpg
Hello version: az02, instance: helloworld-az02-c874d5486-srjnj

服务分布

配置规则

通过 DestinationRule 设置确保全部流量都在跟发起端在同一个AZ

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
cat << EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: helloworld
  namespace: sample
spec:
  host: helloworld.sample.svc.cluster.local
  trafficPolicy:
    connectionPool:
      http:
        maxRequestsPerConnection: 1
    loadBalancer:
      localityLbSetting:
        enabled: true
        failover:
        - from: az01
          to: az02
      simple: ROUND_ROBIN
    outlierDetection:
      baseEjectionTime: 1m
      consecutive5xxErrors: 1
      interval: 1s
EOF

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
~ $ while true
> do
> curl -sSL helloworld.sample:5000/hello
> done
Hello version: az01, instance: helloworld-az01-5dd7bfb647-4zrpg
Hello version: az01, instance: helloworld-az01-5dd7bfb647-4zrpg
Hello version: az01, instance: helloworld-az01-5dd7bfb647-4zrpg
Hello version: az01, instance: helloworld-az01-5dd7bfb647-4zrpg
Hello version: az01, instance: helloworld-az01-5dd7bfb647-4zrpg
Hello version: az01, instance: helloworld-az01-5dd7bfb647-4zrpg
Hello version: az01, instance: helloworld-az01-5dd7bfb647-4zrpg
Hello version: az01, instance: helloworld-az01-5dd7bfb647-4zrpg
Hello version: az01, instance: helloworld-az01-5dd7bfb647-4zrpg
Hello version: az01, instance: helloworld-az01-5dd7bfb647-4zrpg
Hello version: az01, instance: helloworld-az01-5dd7bfb647-4zrpg
Hello version: az01, instance: helloworld-az01-5dd7bfb647-4zrpg
Hello version: az01, instance: helloworld-az01-5dd7bfb647-4zrpg
Hello version: az01, instance: helloworld-az01-5dd7bfb647-4zrpg
Hello version: az01, instance: helloworld-az01-5dd7bfb647-4zrpg

驱逐 Envoy Sidecar 代理

1
2
3
4
5
# kubectl get pod -n sample -l app=helloworld,version=az01
NAME                               READY   STATUS    RESTARTS   AGE
helloworld-az01-5dd7bfb647-4zrpg   2/2     Running   0          4m5s
# kubectl exec -ti -n sample helloworld-az01-5dd7bfb647-4zrpg -c istio-proxy -- curl -sSL -X POST 127.0.0.1:15000/drain_listeners
OK

20230817105320

20230817112032

20230817111955

按地域设置权重

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
kubectl  apply -n sample -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: helloworld
spec:
  host: helloworld.sample.svc.cluster.local
  trafficPolicy:
    connectionPool:
      http:
        maxRequestsPerConnection: 1  # 最大连接数量
    loadBalancer:
      localityLbSetting:
        enabled: true
        distribute:
        - from: sg/az01/*
          to:
            "sg/az01/*": 100
        - from: sg/az02/*
          to:
            "sg/az02/*": 100
    outlierDetection:
      consecutive5xxErrors: 100 # 连续错误数量
      interval: 1s  # 移除检测的时间间隔
      baseEjectionTime: 1m  # 最小的移除时间长度
EOF

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
# Pod 分布情况
# kubectl get pod -n sample -o wide
NAME                               READY   STATUS    RESTARTS   AGE    IP          NODE               NOMINATED NODE   READINESS GATES
helloworld-az01-5dd7bfb647-dvf5r   1/1     Running   0          12m    10.10.1.4   cluster1-worker    <none>           <none>
helloworld-az02-c874d5486-lthh8    1/1     Running   0          12m    10.10.2.4   cluster1-worker2   <none>           <none>
sleep-69cfb4968f-7q5vx             2/2     Running   0          13m    10.10.1.3   cluster1-worker    <none>           <none>
sleep-69cfb4968f-pg9qg             2/2     Running   0          109s   10.10.2.5   cluster1-worker2   <none>           <none>
# istioctl ps
NAME                                                   CLUSTER        CDS        LDS        EDS        RDS          ECDS         ISTIOD                      VERSION
istio-egressgateway-77f4f66bf5-kv8bl.istio-system      Kubernetes     SYNCED     SYNCED     SYNCED     NOT SENT     NOT SENT     istiod-7dfb57994d-cst8p     1.16.2
istio-ingressgateway-5b7797564c-4gfzh.istio-system     Kubernetes     SYNCED     SYNCED     SYNCED     NOT SENT     NOT SENT     istiod-7dfb57994d-cst8p     1.16.2
sleep-69cfb4968f-7q5vx.sample                          Kubernetes     SYNCED     SYNCED     SYNCED     SYNCED       NOT SENT     istiod-7dfb57994d-cst8p     1.16.2
sleep-69cfb4968f-pg9qg.sample                          Kubernetes     SYNCED     SYNCED     SYNCED     SYNCED       NOT SENT     istiod-7dfb57994d-cst8p     1.16.2

# AZ01区域的sleep服务
# kubectl get pod -n sample sleep-69cfb4968f-7q5vx -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP          NODE              NOMINATED NODE   READINESS GATES
sleep-69cfb4968f-7q5vx   2/2     Running   0          14m   10.10.1.3   cluster1-worker   <none>           <none>
# kubectl get node cluster1-worker --show-labels
NAME              STATUS   ROLES    AGE   VERSION    LABELS
cluster1-worker   Ready    <none>   23m   v1.24.15   beta.kubernetes.io/arch=arm64,beta.kubernetes.io/os=linux,kubernetes.io/arch=arm64,kubernetes.io/hostname=cluster1-worker,kubernetes.io/os=linux,topology.kubernetes.io/region=sg,topology.kubernetes.io/zone=az01
# istioctl pc endpoints sleep-69cfb4968f-7q5vx.sample --cluster "outbound|5000||helloworld.sample.svc.cluster.local"
ENDPOINT           STATUS      OUTLIER CHECK     CLUSTER
10.10.1.4:5000     HEALTHY     OK                outbound|5000||helloworld.sample.svc.cluster.local
# kubectl get pod -n sample -o wide |grep 10.10.1.4
helloworld-az01-5dd7bfb647-dvf5r   1/1     Running   0          13m   10.10.1.4   cluster1-worker    <none>           <none>

# AZ02区域sleep服务
# kubectl get pod -n sample sleep-69cfb4968f-pg9qg -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP          NODE               NOMINATED NODE   READINESS GATES
sleep-69cfb4968f-pg9qg   2/2     Running   0          18m   10.10.2.5   cluster1-worker2   <none>           <none>
# istioctl pc endpoints sleep-69cfb4968f-pg9qg.sample --cluster "outbound|5000||helloworld.sample.svc.cluster.local"
ENDPOINT           STATUS      OUTLIER CHECK     CLUSTER
10.10.2.4:5000     HEALTHY     OK                outbound|5000||helloworld.sample.svc.cluster.local

可以看到AZ01区域的 sleep-69cfb4968f-7q5vx.sample 只能查看到 10.10.1.4:5000 endpoint
AZ02区域的 sleep-69cfb4968f-pg9qg 只能发现 10.10.1.4:5000 endpoint

地域感知失效可能原因

DestinationRule 未配置 outlierDetection

地域感知默认开启，但还需要配置 DestinationRule，且指定 outlierDetection 才可以生效，指定这个配置的作用主要是让 istio 感知 endpoints 是否异常，当前 locality 的 endpoints 发生异常时会 failover 到其它地方的 endpoints。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: nginx
spec:
  host: nginx
  trafficPolicy:
    outlierDetection:
      consecutive5xxErrors: 3
      interval: 30s
      baseEjectionTime: 30s

client 没配置 service

istio 控制面会为每个数据面单独下发 EDS，不同数据面实例(Envoy)的locality可能不一样，生成的 EDS 也就可能不一样。istio会获取数据面的locality信息，获取方式主要是找到数据面对应的 endpoint 上保存的 region、zone 等信息，如果 client 没有任何 service，也就不会有 endpoint，控制面也就无法获取 client 的 locality 信息，也就无法实现地域感知。

解决方案: 为 client 配置 service，selector 选中 client 的 label。如果 client 本身不对外提供服务，service 的 ports 也可以随便定义一个。

使用了 headless service

如果是访问 headless service，本身是不支持地域感知的，因为 istio 会对 headless service 请求直接 passthrough，不做负载均衡，客户端会直接访问到 dns 解析出来的 pod ip。

解决方案: 单独再创建一个 service (非 headless)