使用Shell实现Operaror

kbsonlong 收录于 Operator

2023-06-19 约 1400 字预计阅读 3 分钟 - 次阅读

shell-operator架构

Shell Operator 是个冷僻又有点用的东西。方便运维监听kubernetes事件，并基于这些事件做一些简单任务处理；并且shell语言基本上大部分运维人员都懂，而不需要太高的学习成本。

运行原理

shell-operator 部署在 Pod 中。在 Pod 中有一个 /hooks 的一个子目录，其中存储了可执行文件，它们可以用 Bash、Python、Ruby等编写的，我们称这些可执行文件为hooks。Shell-opeator 订阅 Kubernetes 事件并执行这些钩子来响应我们感兴趣的事件。

监听 `Pod` 新增

配置脚本

1
2
mkdir -p shell-operator-demo/hooks
vim shell-operator-demo/hooks/pod-hooks.sh

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
#!/usr/bin/env bash

if [[ $1 == "--config" ]] ; then
  cat <<EOF
configVersion: v1
kubernetes:
- apiVersion: v1
  kind: Pod
  executeHookOnEvent:
  - Added
EOF
else
  type=$(jq -r '.[0].type' $BINDING_CONTEXT_PATH)
  if [[ $type == "Event" ]] ; then
    podName=$(jq -r '.[0].object.metadata.name' $BINDING_CONTEXT_PATH)
    echo "Pod '${podName}' added"
  fi
fi

创建 `Dockerfile`

1
2
FROM ghcr.io/flant/shell-operator:latest
ADD hooks /hooks

构建镜像

1
2
docker build -t registry.cn-hangzhou.aliyuncs.com/seam/shell-operator:monitor-pods .
docker push  registry.cn-hangzhou.aliyuncs.com/seam/shell-operator:monitor-pods

配置权限 (RBAC)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: monitor-pods-acc

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: monitor-pods
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: monitor-pods
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: monitor-pods
subjects:
  - kind: ServiceAccount
    name: monitor-pods-acc
    namespace: example-monitor-pods

运行 `Pod`

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
---
apiVersion: v1
kind: Pod
metadata:
  name: shell-operator
spec:
  containers:
  - name: shell-operator
    image: registry.cn-hangzhou.aliyuncs.com/seam/shell-operator:monitor-pods
    imagePullPolicy: Always
  serviceAccountName: monitor-pods-acc

扩容 `kbernetes-dashboard`

1
kubectl -n kube-system scale --replicas=1 deploy/kubernetes-dashboard

查看 `shell-operator` 的日志

监听自定义 `CRD`

自定义 users.loki.alongparty.cn 资源，包含 name 和 password 两个字段，监听 User 的事件变更生成 htpasswd,并生成 secret 配置

自定义资源(CRD)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  # name must match the spec fields below, and be in the form: <plural>.<group>
  name: users.loki.alongparty.cn
spec:
  # group name to use for REST API: /apis/<group>/<version>
  group: loki.alongparty.cn
  # list of versions supported by this CustomResourceDefinition
  versions:
    - name: v1
      # Each version can be enabled/disabled by Served flag.
      served: true
      # One and only one version must be marked as the storage version.
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                name:
                  type: string
                password:
                  type: string
  # either Namespaced or Cluster
  scope: Namespaced
  names:
    # plural name to be used in the URL: /apis/<group>/<version>/<plural>
    plural: users
    # singular name to be used as an alias on the CLI and for display
    singular: user
    # kind is normally the CamelCased singular type. Your resource manifests use this.
    kind: User
    # shortNames allow shorter string to match your resource on the CLI
    shortNames:
    - user

监听 `CRD`

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
#!/usr/bin/env bash

if [[ $1 == "--config" ]] ; then
  cat <<EOF
{
    "configVersion": "v1",
    "kubernetes": [
        {
            "name":"OnCreateDelete",
            "apiVersion": "loki.alongparty.cn/v1",
            "kind": "User",
            "executeHookOnEvent": [
                "Added",
                "Deleted"
            ]
        },
        {
            "name":"OnModified",
            "apiVersion": "loki.alongparty.cn/v1",
            "kind": "User",
            "executeHookOnEvent": [
                "Modified"
            ]
        }
    ]
}
EOF
else
  bindingName=$(jq -r '.[0].binding' $BINDING_CONTEXT_PATH)

  if [[ $bindingName == "onStartup" ]] ; then
    echo "namespace-hook is triggered on startup."
    exit 0
  fi


  type=$(jq -r '.[0].type' ${BINDING_CONTEXT_PATH})
  if [[ $type == "Synchronization" ]] ; then
    : handle existing objects
    : jq '.[0].objects | ... '
    exit 0
  fi

  resourceEvent=$(jq -r '.[0].watchEvent' $BINDING_CONTEXT_PATH)
  resourceName=$(jq -r '.[0].object.metadata.name' $BINDING_CONTEXT_PATH)
  kind=$(jq -r '.[0].object.kind' ${BINDING_CONTEXT_PATH})
  name=$(jq -r '.[0].object.spec.name' $BINDING_CONTEXT_PATH)
  password=$(jq -r '.[0].object.spec.password' $BINDING_CONTEXT_PATH)

  kubectl get secret  loki-gateway
  IsSecret=$?
  if [[ "${IsSecret}" != 0 ]] ; then
    echo ${IsSecret}
    kubectl create secret generic loki-gateway
  fi

  kubectl get secret  loki-gateway -o=jsonpath="{.data.\.htpasswd}" | base64 -d > /tmp/htpasswd

  if [[ $bindingName == "OnModified" ]] ; then
    echo "${kind}/${resourceName} object were modified"
    echo "${name} ${password}"
    htpasswd -D /tmp/htpasswd "${name}" >>/dev/null
    htpasswd -b /tmp/htpasswd "${name}" "${password}"

  else
    if [[ $resourceEvent == "Added" ]] ; then
      echo "${kind}/${resourceName} object were created"
      echo "${name} ${password}"
      htpasswd -b /tmp/htpasswd "${name}" "${password}"

    else
      echo "${kind}/${resourceName} object were deleted"
      echo "${name} ${password}"
      htpasswd -D /tmp/htpasswd "${name}"

    fi
  fi

  kubectl patch   secret loki-gateway  --patch="{\"data\": { \".htpasswd\": \"$(cat /tmp/htpasswd | base64 -i -)\"}}"
  echo "secret patched successfully"
fi

构建镜像

1
2
3
FROM ghcr.io/flant/shell-operator:latest
ADD hooks /hooks
RUN apk add apache2-utils

1
2
docker build -t registry.cn-hangzhou.aliyuncs.com/seam/shell-operator:crds .
docker push  registry.cn-hangzhou.aliyuncs.com/seam/shell-operator:crds

配置权限 (RBAC)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: crd-simple-acc

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: crd-simple
rules:
- apiGroups: ["loki.alongparty.cn"]
  resources: ["users"]
  verbs: ["get", "watch", "list"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: crd-simple
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: crd-simple
subjects:
  - kind: ServiceAccount
    name: crd-simple-acc
    namespace: default

运行 Operator

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
---
apiVersion: v1
kind: Pod
metadata:
  name: shell-operator
spec:
  containers:
  - name: shell-operator
    image: registry.cn-hangzhou.aliyuncs.com/seam/shell-operator:crds
    imagePullPolicy: Always
  serviceAccountName: crd-simple-acc

配置资源对象

1
2
3
4
5
6
7
8
9
apiVersion: "loki.alongparty.cn/v1"
kind: User
metadata:
  name: user01
  labels:
    name: user01
spec:
  name: "user01"
  password: "password01"

查看 operator 日志

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
...
...
...
INFO[2023-06-20T12:06:38+08:00] queue task HookRun:main:kubernetes:crd.sh:OnModified  binding=kubernetes event.id=eaf0bf72-b246-4418-b9ff-76652c8355f0 queue=main
INFO[2023-06-20T12:06:38+08:00] Execute hook                                  binding=OnModified event=kubernetes hook=crd.sh queue=main task=HookRun
INFO[2023-06-20T12:06:39+08:00] NAME           TYPE     DATA   AGE            binding=OnModified event=kubernetes hook=crd.sh output=stdout queue=main task=HookRun
INFO[2023-06-20T12:06:39+08:00] loki-gateway   Opaque   1      8m51s          binding=OnModified event=kubernetes hook=crd.sh output=stdout queue=main task=HookRun
INFO[2023-06-20T12:06:39+08:00] User/user02 object were modified              binding=OnModified event=kubernetes hook=crd.sh output=stdout queue=main task=HookRun
INFO[2023-06-20T12:06:39+08:00] user05 password002                            binding=OnModified event=kubernetes hook=crd.sh output=stdout queue=main task=HookRun
INFO[2023-06-20T12:06:39+08:00] User user05 not found                         binding=OnModified event=kubernetes hook=crd.sh output=stderr queue=main task=HookRun
INFO[2023-06-20T12:06:39+08:00] Adding password for user user05               binding=OnModified event=kubernetes hook=crd.sh output=stderr queue=main task=HookRun
INFO[2023-06-20T12:06:39+08:00] secret/loki-gateway patched                   binding=OnModified event=kubernetes hook=crd.sh output=stdout queue=main task=HookRun
INFO[2023-06-20T12:06:39+08:00] secret patched successfully                   binding=OnModified event=kubernetes hook=crd.sh output=stdout queue=main task=HookRun
INFO[2023-06-20T12:06:39+08:00] Hook executed successfully                    binding=OnModified event=kubernetes hook=crd.sh queue=main task=HookRun

查看secret

1
2
3
4
5
6
kubectl get secret  loki-gateway -o=jsonpath="{.data.\.htpasswd}" | base64 -d
user02:$apr1$j/FmH28w$DfBvoRMyVoqURgSc24W8..
user01:$apr1$LNyYXi/1$RFBEW/Rko6fbYVhuwGxwF/
user03:$apr1$F7r6YH7R$dvW5yQWAliuPWrlunaFRu/
user04:$apr1$Up7gY4fi$MqgqclnAdITB0AABJMg6q0
user05:$apr1$FW9XnAcP$M/at.hOvIiRDjT60Luu040

参考资料

使用shell-operator实现Operator

shell-operator文档

使用Shell实现Operaror

运行原理

监听 Pod 新增

配置脚本

创建 Dockerfile

构建镜像

配置权限 (RBAC)

运行 Pod

扩容 kbernetes-dashboard

查看 shell-operator 的日志

监听自定义 CRD

自定义资源(CRD)

监听 CRD

构建镜像

配置权限 (RBAC)

运行 Operator

配置资源对象

查看 operator 日志

查看secret

参考资料

监听 `Pod` 新增

创建 `Dockerfile`

运行 `Pod`

扩容 `kbernetes-dashboard`

查看 `shell-operator` 的日志

监听自定义 `CRD`

监听 `CRD`