Karmada 多集群服务发现

kbsonlong 收录于 Karmada

2023-06-29 约 1200 字预计阅读 3 分钟 - 次阅读

安装 Karmada

可以通过参考快速开始来安装Karmada

成员集群网络

确保至少有两个集群被添加到 Karmada，并且成员集群之间的容器网络可相互连接。

如果你使用 hack/local-up-karmada.sh 脚本来部署 Karmada，Karmada 将有三个成员集群，member1 和 member2 的容器网络将被连接。
你可以使用 Submariner 或其他相关的开源项目来连接成员集群之间的网络。

注意：为了防止路由冲突，集群之间的Pod和Service CIDR需要满足不重叠。

安装 ServiceExport 和 ServiceImport CRD

1
2
export KUBECONFIG=~/.kube/karmada.config
kubectl config use-context karmada-apiserver

Karmada 控制面分发到成员集群

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
cat << EOF | kubectl apply -f -
# propagate ServiceExport CRD
apiVersion: policy.karmada.io/v1alpha1
kind: ClusterPropagationPolicy
metadata:
  name: serviceexport-policy
spec:
  resourceSelectors:
    - apiVersion: apiextensions.k8s.io/v1
      kind: CustomResourceDefinition
      name: serviceexports.multicluster.x-k8s.io
  placement:
    clusterAffinity:
      clusterNames:
        - member1
        - member2
---
# propagate ServiceImport CRD
apiVersion: policy.karmada.io/v1alpha1
kind: ClusterPropagationPolicy
metadata:
  name: serviceimport-policy
spec:
  resourceSelectors:
    - apiVersion: apiextensions.k8s.io/v1
      kind: CustomResourceDefinition
      name: serviceimports.multicluster.x-k8s.io
  placement:
    clusterAffinity:
      clusterNames:
        - member1
        - member2
---
# propagate propagationpolicies CRD
apiVersion: policy.karmada.io/v1alpha1
kind: ClusterPropagationPolicy
metadata:
  name: propagationpolicies-policy
spec:
  resourceSelectors:
    - apiVersion: apiextensions.k8s.io/v1
      kind: CustomResourceDefinition
      name: propagationpolicies.policy.karmada.io
  placement:
    clusterAffinity:
      clusterNames:
        - member1
        - member2

EOF

查看成员集群crd

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
➜  demo git:(v1.6.0) ✗  kubectl --kubeconfig ~/.kube/members.config --context member2 get crd
NAME                                    CREATED AT
propagationpolicies.policy.karmada.io   2023-06-29T09:14:38Z
serviceexports.multicluster.x-k8s.io    2023-06-29T08:56:26Z
serviceimports.multicluster.x-k8s.io    2023-06-29T08:56:26Z
➜  demo git:(v1.6.0) ✗  kubectl --kubeconfig ~/.kube/members.config --context member1 get crd
NAME                                    CREATED AT
propagationpolicies.policy.karmada.io   2023-06-29T09:14:38Z
serviceexports.multicluster.x-k8s.io    2023-06-29T08:56:26Z
serviceimports.multicluster.x-k8s.io    2023-06-29T08:56:26Z

示例

在member1集群上部署服务

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
cat << EOF | kubectl --kubeconfig ~/.kube/karmada.config --context karmada-apiserver  apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: serve
spec:
  replicas: 1
  selector:
    matchLabels:
      app: serve
  template:
    metadata:
      labels:
        app: serve
    spec:
      containers:
      - name: serve
        image: jeremyot/serve:0a40de8
        args:
        - "--message='hello from cluster member1 (Node: {{env \"NODE_NAME\"}} Pod: {{env \"POD_NAME\"}} Address: {{addr}})'"
        env:
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
---
apiVersion: v1
kind: Service
metadata:
  name: serve
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: serve
---
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: mcs-workload
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: serve
    - apiVersion: v1
      kind: Service
      name: serve
  placement:
    clusterAffinity:
      clusterNames:
        - member1
EOF

查看Pod信息

1
2
3
 kubectl --kubeconfig ~/.kube/members.config --context member1 get pod -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP          NODE                    NOMINATED NODE   READINESS GATES
serve-5899cfd5cd-lj6ls   1/1     Running   0          26m   10.10.0.5   member1-control-plane   <none>           <none>

导出服务到 member2 集群

member1 导出服务

在karmada控制平面上创建一个 ServiceExport 对象，然后创建一个 PropagationPolicy ，将 ServiceExport 对象分发到 member1 集群。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
cat << EOF | kubectl --kubeconfig ~/.kube/karmada.config --context karmada-apiserver apply -f -
apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ServiceExport
metadata:
  name: serve
---
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: serve-export-policy
spec:
  resourceSelectors:
    - apiVersion: multicluster.x-k8s.io/v1alpha1
      kind: ServiceExport
      name: serve
  placement:
    clusterAffinity:
      clusterNames:
        - member1
EOF

member2 导入服务

在karmada控制平面上创建一个 ServiceImport 对象，然后创建一个 PropagationPlicy 来分发 ServiceImport 对象到 member2 集群。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
cat << EOF | kubectl --kubeconfig ~/.kube/karmada.config --context karmada-apiserver apply -f -
apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ServiceImport
metadata:
  name: serve
spec:
  type: ClusterSetIP
  ports:
  - port: 80
    protocol: TCP
---
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: serve-import-policy
spec:
  resourceSelectors:
    - apiVersion: multicluster.x-k8s.io/v1alpha1
      kind: ServiceImport
      name: serve
  placement:
    clusterAffinity:
      clusterNames:
        - member2
EOF

从 member2 集群获取服务

在 member2 集群上启动一个Pod request来访问派生服务的ClusterIP

1
2
3
4
5
# 我们可以在member2集群中找到对应的派生服务。
$ kubectl --kubeconfig ~/.kube/members.config --context member2 get svc
NAME            TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
derived-serve   ClusterIP   10.13.74.91   <none>        80/TCP    31m
kubernetes      ClusterIP   10.13.0.1     <none>        443/TCP   75m

例如，如果我们使用派生的服务名来持续访问该服务10s，将会得到如下输出:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
kubectl --kubeconfig ~/.kube/members.config --context member2 run -i --rm --restart=Never --image=jeremyot/request:0a40de8 request -- --duration=10s --address=derived-serve
If you don't see a command prompt, try pressing enter.
2023/06/29 10:16:47 'hello from cluster member1 (Node: member1-control-plane Pod: serve-5899cfd5cd-lj6ls Address: 10.10.0.5)'
2023/06/29 10:16:48 'hello from cluster member1 (Node: member1-control-plane Pod: serve-5899cfd5cd-lj6ls Address: 10.10.0.5)'
2023/06/29 10:16:49 'hello from cluster member1 (Node: member1-control-plane Pod: serve-5899cfd5cd-lj6ls Address: 10.10.0.5)'
2023/06/29 10:16:50 'hello from cluster member1 (Node: member1-control-plane Pod: serve-5899cfd5cd-lj6ls Address: 10.10.0.5)'
2023/06/29 10:16:51 'hello from cluster member1 (Node: member1-control-plane Pod: serve-5899cfd5cd-lj6ls Address: 10.10.0.5)'
2023/06/29 10:16:52 'hello from cluster member1 (Node: member1-control-plane Pod: serve-5899cfd5cd-lj6ls Address: 10.10.0.5)'
2023/06/29 10:16:53 'hello from cluster member1 (Node: member1-control-plane Pod: serve-5899cfd5cd-lj6ls Address: 10.10.0.5)'
2023/06/29 10:16:54 'hello from cluster member1 (Node: member1-control-plane Pod: serve-5899cfd5cd-lj6ls Address: 10.10.0.5)'
2023/06/29 10:16:55 'hello from cluster member1 (Node: member1-control-plane Pod: serve-5899cfd5cd-lj6ls Address: 10.10.0.5)'

可以看到请求返回信息跟 member1 部署的 Pod 信息是一致的