概述
在 Kubernetes 集群中，单一 CNI 插件往往无法满足复杂的网络需求。本文详细介绍如何使用 Flannel 作为主 CNI 插件，配合 Multus-CNI 实现多网段隔离，为不同类型的工作负载提供专用的网络平面。
架构优势
网络隔离 ：不同业务使用独立的网络平面，提高安全性性能优化 ：关键业务可使用高性能网络接口灵活配置 ：支持多种网络插件组合渐进式迁移 ：可在现有 Flannel 集群基础上扩展架构设计
整体架构
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
┌─────────────────────────────────────────────────────────────┐
│ Kubernetes Cluster │
├─────────────────────────────────────────────────────────────┤
│ Pod Network Planes │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Default │ │ Business │ │ Storage │ │
│ │ Network │ │ Network │ │ Network │ │
│ │ (Flannel) │ │ (MacVLAN) │ │ (SR-IOV) │ │
│ │10.244.0.0/16│ │192.168.0.0/16│ │172.16.0.0/16│ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
├─────────────────────────────────────────────────────────────┤
│ CNI Layer │
│ ┌─────────────┐ ┌─────────────────────────────────────┐ │
│ │ Flannel │ │ Multus-CNI │ │
│ │ (Primary) │ │ (Meta Plugin) │ │
│ └─────────────┘ └─────────────────────────────────────┘ │
├─────────────────────────────────────────────────────────────┤
│ Physical Network Infrastructure │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ eth0 │ │ eth1 │ │ eth2 │ │
│ │ (Management)│ │ (Business) │ │ (Storage) │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
└─────────────────────────────────────────────────────────────┘
网络平面规划
网络平面 CNI 插件 CIDR 范围 用途 接口 默认网络 Flannel 10.244.0.0/16 集群内部通信 eth0 业务网络 MacVLAN 192.168.0.0/16 业务应用通信 eth1 存储网络 SR-IOV 172.16.0.0/16 高性能存储 eth2 管理网络 Host-Device 10.0.0.0/24 运维管理 eth0
环境准备
1. 集群要求
1
2
3
4
5
6
7
8
9
# 检查 Kubernetes 版本
kubectl version --short
# 检查节点状态
kubectl get nodes -o wide
# 检查现有 CNI 配置
ls -la /etc/cni/net.d/
cat /etc/cni/net.d/10-flannel.conflist
2. 网络接口准备
1
2
3
4
5
6
7
8
9
# 检查网络接口
ip link show
# 配置额外网络接口（每个节点）
sudo ip link add link eth0 name eth0.100 type vlan id 100
sudo ip link set eth0.100 up
# 验证接口状态
ip addr show eth0.100
3. 创建命名空间
1
2
3
4
5
6
# 创建 Multus 系统命名空间
kubectl create namespace multus-system
# 创建业务命名空间
kubectl create namespace business-apps
kubectl create namespace storage-apps
Multus-CNI 部署
1. 部署 Multus-CNI
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
# multus-daemonset.yaml
apiVersion : apps/v1
kind : DaemonSet
metadata :
name : kube-multus-ds
namespace : multus-system
labels :
tier : node
app : multus
name : multus
spec :
selector :
matchLabels :
name : multus
updateStrategy :
type : RollingUpdate
template :
metadata :
labels :
tier : node
app : multus
name : multus
spec :
hostNetwork : true
tolerations :
- operator : Exists
effect : NoSchedule
- operator : Exists
effect : NoExecute
serviceAccountName : multus
containers :
- name : kube-multus
image : ghcr.io/k8snetworkplumbingwg/multus-cni:v4.0.2
command : [ "/entrypoint.sh" ]
args :
- "--multus-conf-file=auto"
- "--multus-autoconfig-dir=/host/etc/cni/net.d"
- "--cni-conf-dir=/host/etc/cni/net.d"
- "--multus-kubeconfig-file-host=/etc/cni/net.d/multus.d/multus.kubeconfig"
resources :
requests :
cpu : "100m"
memory : "50Mi"
limits :
cpu : "100m"
memory : "50Mi"
securityContext :
privileged : true
volumeMounts :
- name : cni
mountPath : /host/etc/cni/net.d
- name : cnibin
mountPath : /host/opt/cni/bin
- name : multus-cfg
mountPath : /tmp/multus-conf
terminationGracePeriodSeconds : 10
volumes :
- name : cni
hostPath :
path : /etc/cni/net.d
- name : cnibin
hostPath :
path : /opt/cni/bin
- name : multus-cfg
configMap :
name : multus-cni-config
items :
- key : cni-conf.json
path : 70 -multus.conf
---
apiVersion : v1
kind : ServiceAccount
metadata :
name : multus
namespace : multus-system
---
apiVersion : rbac.authorization.k8s.io/v1
kind : ClusterRole
metadata :
name : multus
rules :
apiGroups : [ "k8s.cni.cncf.io" ]
resources :
- '*'
verbs :
- '*'
apiGroups :
- ""
resources :
- pods
- pods/status
verbs :
- get
- update
apiGroups :
- ""
- events.k8s.io
resources :
- events
verbs :
- create
- patch
- update
---
apiVersion : rbac.authorization.k8s.io/v1
kind : ClusterRoleBinding
metadata :
name : multus
roleRef :
apiGroup : rbac.authorization.k8s.io
kind : ClusterRole
name : multus
subjects :
kind : ServiceAccount
name : multus
namespace : multus-system
2. Multus 配置
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
# multus-config.yaml
apiVersion : v1
kind : ConfigMap
metadata :
name : multus-cni-config
namespace : multus-system
labels :
tier : node
app : multus
data :
cni-conf.json : |
{
"cniVersion": "0.3.1",
"name": "multus-cni-network",
"type": "multus",
"capabilities": {
"portMappings": true
},
"delegates": [
{
"cniVersion": "0.3.1",
"name": "default-cni-network",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
],
"kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig"
}
3. 部署命令
1
2
3
4
5
6
7
# 应用配置
kubectl apply -f multus-config.yaml
kubectl apply -f multus-daemonset.yaml
# 验证部署
kubectl get pods -n multus-system
kubectl logs -n multus-system -l name = multus
网络附件定义 (NetworkAttachmentDefinition)
1. MacVLAN 业务网络
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# business-network-nad.yaml
apiVersion : "k8s.cni.cncf.io/v1"
kind : NetworkAttachmentDefinition
metadata :
name : business-network
namespace : business-apps
spec :
config : '{
"cniVersion": "0.3.1",
"type": "macvlan",
"master": "eth1",
"mode": "bridge",
"ipam": {
"type": "host-local",
"subnet": "192.168.0.0/16",
"rangeStart": "192.168.100.10",
"rangeEnd": "192.168.100.200",
"gateway": "192.168.100.1",
"routes": [
{
"dst": "0.0.0.0/0",
"gw": "192.168.100.1"
}
]
}
}'
2. SR-IOV 存储网络
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# storage-network-nad.yaml
apiVersion : "k8s.cni.cncf.io/v1"
kind : NetworkAttachmentDefinition
metadata :
name : storage-network
namespace : storage-apps
spec :
config : '{
"cniVersion": "0.3.1",
"type": "sriov",
"deviceID": "0000:03:00.0",
"vf": 0,
"ipam": {
"type": "host-local",
"subnet": "172.16.0.0/16",
"rangeStart": "172.16.1.10",
"rangeEnd": "172.16.1.200",
"gateway": "172.16.1.1"
}
}'
3. VLAN 隔离网络
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# vlan-network-nad.yaml
apiVersion : "k8s.cni.cncf.io/v1"
kind : NetworkAttachmentDefinition
metadata :
name : vlan-network
namespace : default
spec :
config : '{
"cniVersion": "0.3.1",
"type": "vlan",
"master": "eth0",
"vlanId": 100,
"ipam": {
"type": "host-local",
"subnet": "10.100.0.0/24",
"rangeStart": "10.100.0.10",
"rangeEnd": "10.100.0.200",
"gateway": "10.100.0.1"
}
}'
4. Host-Device 管理网络
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# management-network-nad.yaml
apiVersion : "k8s.cni.cncf.io/v1"
kind : NetworkAttachmentDefinition
metadata :
name : management-network
namespace : kube-system
spec :
config : '{
"cniVersion": "0.3.1",
"type": "host-device",
"device": "eth2",
"ipam": {
"type": "host-local",
"subnet": "10.0.0.0/24",
"rangeStart": "10.0.0.10",
"rangeEnd": "10.0.0.200",
"gateway": "10.0.0.1"
}
}'
5. 应用网络附件定义
1
2
3
4
5
6
7
8
9
# 创建所有网络附件定义
kubectl apply -f business-network-nad.yaml
kubectl apply -f storage-network-nad.yaml
kubectl apply -f vlan-network-nad.yaml
kubectl apply -f management-network-nad.yaml
# 验证网络附件定义
kubectl get network-attachment-definitions --all-namespaces
kubectl describe network-attachment-definitions business-network -n business-apps
应用部署示例
1. 多网络业务应用
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
# business-app-deployment.yaml
apiVersion : apps/v1
kind : Deployment
metadata :
name : business-app
namespace : business-apps
labels :
app : business-app
spec :
replicas : 3
selector :
matchLabels :
app : business-app
template :
metadata :
labels :
app : business-app
annotations :
k8s.v1.cni.cncf.io/networks : |
[
{
"name": "business-network",
"interface": "net1",
"ips": ["192.168.100.10/24"]
}
]
spec :
containers :
- name : business-app
image : nginx:1.21
ports :
- containerPort : 80
name : http
- containerPort : 8080
name : business
resources :
requests :
cpu : 100m
memory : 128Mi
limits :
cpu : 500m
memory : 512Mi
env :
- name : NETWORK_MODE
value : "multi-network"
---
apiVersion : v1
kind : Service
metadata :
name : business-app-service
namespace : business-apps
spec :
selector :
app : business-app
ports :
- name : http
port : 80
targetPort : 80
- name : business
port : 8080
targetPort : 8080
type : ClusterIP
2. 高性能存储应用
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
# storage-app-deployment.yaml
apiVersion : apps/v1
kind : Deployment
metadata :
name : storage-app
namespace : storage-apps
labels :
app : storage-app
spec :
replicas : 2
selector :
matchLabels :
app : storage-app
template :
metadata :
labels :
app : storage-app
annotations :
k8s.v1.cni.cncf.io/networks : |
[
{
"name": "storage-network",
"interface": "storage0"
},
{
"name": "business-network",
"interface": "business0"
}
]
spec :
containers :
- name : storage-app
image : minio/minio:latest
command :
- /bin/bash
- -c
args :
- minio server /data --console-address ":9001"
ports :
- containerPort : 9000
name : api
- containerPort : 9001
name : console
resources :
requests :
cpu : 500m
memory : 1Gi
limits :
cpu : 2
memory : 4Gi
volumeMounts :
- name : data
mountPath : /data
volumes :
- name : data
emptyDir : {}
3. 网络策略隔离
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
# network-policy.yaml
apiVersion : networking.k8s.io/v1
kind : NetworkPolicy
metadata :
name : business-network-policy
namespace : business-apps
spec :
podSelector :
matchLabels :
app : business-app
policyTypes :
- Ingress
- Egress
ingress :
- from :
- namespaceSelector :
matchLabels :
name : business-apps
- podSelector :
matchLabels :
network-access : "business"
ports :
- protocol : TCP
port : 80
- protocol : TCP
port : 8080
egress :
- to :
- namespaceSelector :
matchLabels :
name : storage-apps
ports :
- protocol : TCP
port : 9000
- to : []
ports :
- protocol : UDP
port : 53
---
apiVersion : networking.k8s.io/v1
kind : NetworkPolicy
metadata :
name : storage-network-policy
namespace : storage-apps
spec :
podSelector :
matchLabels :
app : storage-app
policyTypes :
- Ingress
- Egress
ingress :
- from :
- namespaceSelector :
matchLabels :
name : business-apps
ports :
- protocol : TCP
port : 9000
egress :
- to : []
ports :
- protocol : UDP
port : 53
验证和测试
1. 网络连通性测试
1
2
3
4
5
6
7
8
9
10
11
# 部署测试应用
kubectl apply -f business-app-deployment.yaml
kubectl apply -f storage-app-deployment.yaml
# 检查 Pod 状态
kubectl get pods -n business-apps -o wide
kubectl get pods -n storage-apps -o wide
# 检查网络接口
kubectl exec -n business-apps business-app-xxx -- ip addr show
kubectl exec -n business-apps business-app-xxx -- ip route show
2. 多网络接口验证
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# 创建网络测试脚本
cat > network-test.sh << 'EOF'
#!/bin/bash
echo "=== 网络接口信息 ==="
ip addr show
echo "\n=== 路由表信息 ==="
ip route show
echo "\n=== 默认网络连通性测试 ==="
ping -c 3 kubernetes.default.svc.cluster.local
echo "\n=== 业务网络连通性测试 ==="
ping -c 3 -I net1 192.168.100.1
echo "\n=== 网络性能测试 ==="
iperf3 -c 192.168.100.1 -t 10 -B 192.168.100.10
EOF
# 在 Pod 中执行测试
kubectl cp network-test.sh business-apps/business-app-xxx:/tmp/
kubectl exec -n business-apps business-app-xxx -- chmod +x /tmp/network-test.sh
kubectl exec -n business-apps business-app-xxx -- /tmp/network-test.sh
3. 网络隔离验证
1
2
3
4
5
6
7
8
9
# 测试网络策略隔离
kubectl run test-pod --image= busybox --rm -it --restart= Never -- /bin/sh
# 在测试 Pod 中执行
wget -qO- http://business-app-service.business-apps.svc.cluster.local
wget -qO- http://storage-app-service.storage-apps.svc.cluster.local:9000
# 测试跨网络访问
kubectl exec -n business-apps business-app-xxx -- curl http://storage-app-service.storage-apps.svc.cluster.local:9000
监控和运维
1. 网络监控配置
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# network-monitoring.yaml
apiVersion : v1
kind : ConfigMap
metadata :
name : network-monitoring-config
namespace : monitoring
data :
prometheus.yml : |
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'multus-metrics'
kubernetes_sd_configs:
- role: pod
namespaces:
names:
- multus-system
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_name]
action: keep
regex: multus
- job_name: 'cni-metrics'
static_configs:
- targets: ['localhost:9090']
metrics_path: /metrics
scrape_interval: 30s
2. 网络故障排查脚本
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# network-troubleshoot.sh
#!/bin/bash
echo "=== Multus-CNI 状态检查 ==="
kubectl get pods -n multus-system
kubectl get network-attachment-definitions --all-namespaces
echo "\n=== CNI 配置检查 ==="
for node in $( kubectl get nodes -o name | cut -d/ -f2) ; do
echo "Node: $node "
kubectl debug node/$node -it --image= busybox -- ls -la /host/etc/cni/net.d/
done
echo "\n=== 网络接口状态 ==="
for pod in $( kubectl get pods --all-namespaces -o jsonpath = '{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}' ) ; do
namespace = $( echo $pod | cut -d' ' -f1)
name = $( echo $pod | cut -d' ' -f2)
echo "Pod: $namespace / $name "
kubectl exec -n $namespace $name -- ip addr show 2>/dev/null || echo "无法访问"
done
echo "\n=== 网络策略检查 ==="
kubectl get networkpolicies --all-namespaces
echo "\n=== 事件日志检查 ==="
kubectl get events --all-namespaces --field-selector type = Warning
3. 性能优化建议
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# network-optimization.sh
#!/bin/bash
echo "=== 网络性能优化配置 ==="
# 1. 调整网络缓冲区
echo 'net.core.rmem_max = 134217728' >> /etc/sysctl.conf
echo 'net.core.wmem_max = 134217728' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_rmem = 4096 87380 134217728' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_wmem = 4096 65536 134217728' >> /etc/sysctl.conf
# 2. 启用网络优化
echo 'net.core.netdev_max_backlog = 5000' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_congestion_control = bbr' >> /etc/sysctl.conf
# 3. 应用配置
sysctl -p
echo "网络优化配置已应用"
最佳实践
1. 网络规划原则
分层设计 ：按业务类型划分网络平面CIDR 规划 ：避免网段冲突，预留扩展空间性能考虑 ：关键业务使用高性能网络接口安全隔离 ：通过网络策略实现访问控制2. 配置管理
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# network-config-template.yaml
apiVersion : v1
kind : ConfigMap
metadata :
name : network-config-template
namespace : kube-system
data :
network-plan.json : |
{
"networks": {
"default": {
"cni": "flannel",
"cidr": "10.244.0.0/16",
"purpose": "cluster-internal"
},
"business": {
"cni": "macvlan",
"cidr": "192.168.0.0/16",
"purpose": "business-traffic",
"interface": "eth1"
},
"storage": {
"cni": "sriov",
"cidr": "172.16.0.0/16",
"purpose": "high-performance-storage",
"interface": "eth2"
}
}
}
3. 运维自动化
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
# network-automation.sh
#!/bin/bash
# 自动化网络配置部署
function deploy_network_config() {
local namespace = $1
local network_type = $2
echo "部署 $network_type 网络配置到 $namespace "
# 生成网络附件定义
envsubst < network-template.yaml | kubectl apply -f -
# 验证部署
kubectl get network-attachment-definitions -n $namespace
}
# 网络健康检查
function network_health_check() {
echo "执行网络健康检查..."
# 检查 Multus 状态
kubectl get pods -n multus-system
# 检查网络附件定义
kubectl get network-attachment-definitions --all-namespaces
# 检查网络策略
kubectl get networkpolicies --all-namespaces
}
# 主函数
main() {
case $1 in
"deploy" )
deploy_network_config $2 $3
;;
"check" )
network_health_check
;;
*)
echo "用法: $0 {deploy|check} [namespace] [network_type]"
exit 1
;;
esac
}
main " $@ "
故障排查
1. 常见问题
问题 症状 解决方案 Multus Pod 启动失败 CrashLoopBackOff 检查 CNI 配置文件权限 网络附件定义无效 Pod 创建失败 验证 JSON 配置语法 多网络接口未创建 只有默认接口 检查网络附件定义和注解 网络隔离失效 跨网络访问成功 检查网络策略配置 性能问题 网络延迟高 优化网络参数和接口类型
2. 诊断命令
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
# 完整的网络诊断脚本
cat > network-diagnosis.sh << 'EOF'
#!/bin/bash
echo "=== 1. 基础环境检查 ==="
kubectl version --short
kubectl get nodes -o wide
echo "\n=== 2. CNI 配置检查 ==="
ls -la /etc/cni/net.d/
cat /etc/cni/net.d/*.conf* 2>/dev/null
echo "\n=== 3. Multus 状态检查 ==="
kubectl get pods -n multus-system
kubectl logs -n multus-system -l name=multus --tail=50
echo "\n=== 4. 网络附件定义检查 ==="
kubectl get network-attachment-definitions --all-namespaces
echo "\n=== 5. Pod 网络状态检查 ==="
for ns in $(kubectl get namespaces -o name | cut -d/ -f2); do
echo "Namespace: $ns"
kubectl get pods -n $ns -o wide
done
echo "\n=== 6. 网络策略检查 ==="
kubectl get networkpolicies --all-namespaces
echo "\n=== 7. 事件检查 ==="
kubectl get events --all-namespaces --sort-by='.lastTimestamp' | tail -20
echo "\n=== 8. 系统网络配置 ==="
ip addr show
ip route show
EOF
chmod +x network-diagnosis.sh
./network-diagnosis.sh
总结
通过 Flannel + Multus-CNI 的组合方案，我们成功实现了 Kubernetes 集群中的多网段隔离：
技术优势
灵活性 ：支持多种 CNI 插件组合隔离性 ：不同业务使用独立网络平面性能 ：关键业务可使用高性能网络接口兼容性 ：与现有 Flannel 集群完全兼容应用场景
多租户环境 ：不同租户使用独立网络混合云部署 ：本地和云端网络隔离高性能计算 ：存储和计算网络分离安全合规 ：满足网络隔离要求后续扩展
集成 Istio 服务网格 实现网络 QoS 控制 添加网络监控和告警 支持动态网络配置 这套方案为 Kubernetes 集群提供了企业级的网络隔离能力，满足了复杂业务场景的网络需求。
