蜷缩的蜗牛
蜷缩的蜗牛

OpenEBS性能测试

kbsonlong 收录于 存储
 约 9900 字 预计阅读 20 分钟 - 次阅读  

依赖说明

准备裸盘 

1
2
3
4
5
6
7

root@ubuntu:~# lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
nvme2n1     259:2    0   50G  0 disk           ### 50G未格式化磁盘
nvme1n1     259:0    0  600G  0 disk
└─nvme1n1p1 259:3    0  600G  0 part /data
nvme0n1     259:1    0   40G  0 disk
└─nvme0n1p1 259:4    0   40G  0 part /

安装iscsi 

1
2
3

sudo apt-get update
sudo apt-get install open-iscsi
sudo systemctl enable --now iscsid

OpenEBS安装 

1
2
3
4
5

helm repo add openebs https://openebs.github.io/charts
helm repo update

### 默认安装
helm install openebs --namespace openebs openebs/openebs  --create-namespace

启用cstor 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64

### 启用cstor,替换镜像地址:k8s.gcr.io/sig-storage
cat << EOF >> custom-values.yaml
USER-SUPPLIED VALUES:
cstor:
  admissionServer:
    nodeSelector:
      biz.type: test
  csiController:
    attacher:
      image:
        registry: registry.cn-hangzhou.aliyuncs.com/
        repository: seam/csi-attacher
    nodeSelector:
      biz.type: test
    provisioner:
      image:
        registry: registry.cn-hangzhou.aliyuncs.com/
        repository: seam/csi-provisioner
    resizer:
      image:
        registry: registry.cn-hangzhou.aliyuncs.com/
        repository: seam/csi-resizer
    snapshotController:
      image:
        registry: registry.cn-hangzhou.aliyuncs.com/
        repository: seam/snapshot-controller
    snapshotter:
      image:
        registry: registry.cn-hangzhou.aliyuncs.com/
        repository: seam/csi-snapshotter
  csiNode:
    driverRegistrar:
      image:
        registry: registry.cn-hangzhou.aliyuncs.com/
        repository: seam/csi-node-driver-registrar
    kubeletDir: /data/k8s/kubelet/
    nodeSelector:
      biz.type: test
  cspcOperator:
    nodeSelector:
      biz.type: test
  cvcOperator:
    nodeSelector:
      biz.type: test
  enabled: true
localprovisioner:
  nodeSelector:
    biz.type: test
ndm:
  nodeSelector:
    biz.type: test
ndmOperator:
  nodeSelector:
    biz.type: test
provisioner:
  nodeSelector:
    biz.type: test
snapshotOperator:
  nodeSelector:
    biz.type: test
webhook:
  nodeSelector:
    biz.type: test
EOF
1

helm upgrade --install openebs --namespace openebs openebs/openebs --version 3.4.1 -f custom-values.yaml

cstor.csiNode.kubeletDir="/data/k8s/kubelet/" 设置为实际集群运行配置

安装后查看磁盘情况 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

root@ubuntu:~# lsblk 
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
nvme2n1     259:2    0   50G  0 disk 
└─nvme2n1p1 259:6    0   50G  0 part           ### 格式化未挂载盘
nvme1n1     259:0    0  600G  0 disk 
└─nvme1n1p1 259:3    0  600G  0 part /data
nvme0n1     259:1    0   40G  0 disk 
└─nvme0n1p1 259:4    0   40G  0 part /

# kubectl get bd -n openebs
NAME                                           NODENAME                                                SIZE          CLAIMSTATE   STATUS   AGE
blockdevice-5cc77e185a944270fa46b7d233320660   dev-vm-8c32g-600g-10-86-183-104   53686025728   Unclaimed    Active   4m56s
blockdevice-b3978ac55b8d0fc6ae9e2bc3b5b92fd6   dev-vm-8c64g-600g-10-86-16-225    53686025728   Unclaimed    Active   4m56s
blockdevice-d93d0a91351a13f4de81fd0a5e6e4540   dev-vm-4c16g-600g-10-86-141-68    53686025728   Unclaimed    Active   4m56s

Local PV device 测试

验证PVC挂载 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

# cat << EOF > local-device-pod.yaml
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: local-device-pvc
spec:
  storageClassName: openebs-device
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5G
---
apiVersion: v1
kind: Pod
metadata:
  name: hello-local-device-pod
spec:
  volumes:
  - name: local-storage
    persistentVolumeClaim:
      claimName: local-device-pvc
  containers:
  - name: hello-container
    image: dockerhub.kubekey.local/library/busybox:latest
    command:
       - sh
       - -c
       - 'while true; do echo "`date` [`hostname`] Hello from OpenEBS Local PV." >> /mnt/store/greet.txt; sleep $(($RANDOM % 5 + 300)); done'
    volumeMounts:
    - mountPath: /mnt/store
      name: local-storage
EOF

# kubectl apply -f local-device-pod.yaml

查看PVC挂载情况 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# kubectl get pvc local-device-pvc
NAME               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     AGE
local-device-pvc   Bound    pvc-b743d6b9-2854-4e9c-aaff-39064605e594   5G         RWO            openebs-device   2m3s

# kubectl get pod hello-local-device-pod  -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP             NODE                                                   NOMINATED NODE   READINESS GATES
hello-local-device-pod   1/1     Running   0          18s   10.86.232.79   dev-vm-4c16g-600g-10-86-141-68   <none>           <none>
# kubectl get bd -n openebs
NAME                                           NODENAME                                                SIZE          CLAIMSTATE   STATUS   AGE
blockdevice-5cc77e185a944270fa46b7d233320660   dev-vm-8c32g-600g-10-86-183-104   53686025728   Unclaimed    Active   11m
blockdevice-b3978ac55b8d0fc6ae9e2bc3b5b92fd6   dev-vm-8c64g-600g-10-86-16-225    53686025728   Unclaimed    Active   11m
blockdevice-d93d0a91351a13f4de81fd0a5e6e4540   dev-vm-4c16g-600g-10-86-141-68    53686025728   Claimed      Active   11m

Pod调度到节点10-86-141-68, 节点10-86-141-68对应的 blockdevice 状态为 Claimed

登录节点10-86-141-68 验证数据是否写入

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

# lsblk 
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
nvme2n1     259:2    0   50G  0 disk 
└─nvme2n1p1 259:5    0   50G  0 part /data/k8s/kubelet/pods/d1f84232-f355-4f9f-850d-129b8fd59864/volumes/kubernetes.io~local-volume/pvc-b743d6b9-2854-4e9c-aaff-39064605e594
nvme1n1     259:0    0  600G  0 disk 
└─nvme1n1p1 259:3    0  600G  0 part /data
nvme0n1     259:1    0   40G  0 disk 
└─nvme0n1p1 259:4    0   40G  0 part /
# ls -l /data/k8s/kubelet/pods/d1f84232-f355-4f9f-850d-129b8fd59864/volumes/kubernetes.io~local-volume/pvc-b743d6b9-2854-4e9c-aaff-39064605e594/
total 20
-rw-r--r-- 1 root root    67 Oct 18 15:33 greet.txt
drwx------ 2 root root 16384 Oct 18 15:28 lost+found
# ls -l /data/k8s/kubelet/pods/d1f84232-f355-4f9f-850d-129b8fd59864/volumes/kubernetes.io~local-volume/pvc-b743d6b9-2854-4e9c-aaff-39064605e594/greet.txt 
-rw-r--r-- 1 root root 67 Oct 18 15:33 /data/k8s/kubelet/pods/d1f84232-f355-4f9f-850d-129b8fd59864/volumes/kubernetes.io~local-volume/pvc-b743d6b9-2854-4e9c-aaff-39064605e594/greet.txt
# cat /data/k8s/kubelet/pods/d1f84232-f355-4f9f-850d-129b8fd59864/volumes/kubernetes.io~local-volume/pvc-b743d6b9-2854-4e9c-aaff-39064605e594/greet.txt 
Tue Oct 18 15:32:27 CST 2022 [ubuntu] Hello from OpenEBS Local PV.

Local PV device 性能测试

创建PVC 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

cat > dbench-pvc.yaml <<EOF
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: dbench
spec:
  storageClassName: openebs-device
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5G
EOF

kubectl apply -f dbench-pvc.yaml

创建压测任务 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93

cat << EOF > fio-job.yaml
# NOTE: For details of params to construct an fio job, refer to this link:
# https://fio.readthedocs.io/en/latest/fio_doc.html

---
apiVersion: batch/v1
kind: Job
metadata:
  generateName: dbench-
spec:
  template:
    spec:
      containers:
      - name: dbench
        image: openebs/perf-test:latest
        imagePullPolicy: IfNotPresent
        env:

          ## storage mount point on which testfiles are created

          - name: DBENCH_MOUNTPOINT
            value: /data

          ##########################################################
          # I/O PROFILE COVERAGE FOR SPECIFIC PERF CHARACTERISTICS #
          ##########################################################

          ## quick: {read, write} iops, {read, write} bw (all random)
          ## detailed: {quick}, {read, write} latency & mixed 75r:25w (all random), {read, write} bw (all sequential)  
          ## custom: a single user-defined job run with params specified in env 'CUSTOM'

          - name: DBENCH_TYPE
            value: detailed

          ####################################################
          # STANDARD TUNABLES FOR DBENCH_TYPE=QUICK/DETAILED #
          ####################################################

          ## active data size for the bench test

          - name: FIO_SIZE
            value: 2G

          ## use un-buffered i/o (usually O_DIRECT)

          - name: FIO_DIRECT
            value: '1'

          ## no of independent threads doing the same i/o

          - name: FIO_NUMJOBS
            value: '1'

          ## space b/w starting offsets on a file in case of parallel file i/o

          - name: FIO_OFFSET_INCREMENT
            value: 250M

          ## nature of i/o to file. commonly supported: libaio, sync, 

          - name: FIO_IOENGINE
            value: libaio

          ## additional runtime options which will be appended to the above params
          ## ensure options used are not mutually exclusive w/ above params
          ## ex: '--group_reporting=1, stonewall, --ramptime=<val> etc..,
          
          - name: OPTIONS
            value: ''

          ####################################################
          # CUSTOM JOB SPEC FOR DBENCH_TYPE=CUSTOM           #
          ####################################################

          ## this will execute a single job run with the params specified 
          ## ex: '--bs=16k --iodepth=64 --ioengine=sync --size=500M --name=custom --readwrite=randrw --rwmixread=80 --random_distribution=pareto' 

          - name: CUSTOM
            value: ''
          
        volumeMounts:
        - name: dbench-pv
          mountPath: /data
      restartPolicy: Never
      volumes:
      - name: dbench-pv
        persistentVolumeClaim:
          claimName: dbench
  backoffLimit: 4
---
EOF

kubectl create -f fio-job.yaml
IOPSBandwidth
随机读2997125MiB/s
随机写2997125MiB/s
顺序读126MiB/s
顺序写128MiB/s
混合读写50%:50%3070/1011

第一次压测日志 

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278

# kubectl logs -f dbench-4jwlb-qjhzg 
Working dir: /data

Testing Read IOPS...
read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.13
Starting 1 process
read_iops: Laying out IO file (1 file / 2048MiB)

read_iops: (groupid=0, jobs=1): err= 0: pid=13: Tue Oct 18 07:55:44 2022
read: IOPS=2997, BW=11.7MiB/s (12.3MB/s)(176MiB/15027msec)
bw (  KiB/s): min=11848, max=12136, per=99.92%, avg=11997.87, stdev=44.56, samples=30
iops        : min= 2962, max= 3034, avg=2999.47, stdev=11.14, samples=30
cpu          : usr=0.24%, sys=1.06%, ctx=13280, majf=0, minf=15
IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=45049,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=11.7MiB/s (12.3MB/s), 11.7MiB/s-11.7MiB/s (12.3MB/s-12.3MB/s), io=176MiB (185MB), run=15027-15027msec

Disk stats (read/write):
nvme2n1: ios=53999/2, merge=0/1, ticks=537596/20, in_queue=537796, util=99.46%


Testing Write IOPS...
write_iops: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.13
Starting 1 process

write_iops: (groupid=0, jobs=1): err= 0: pid=25: Tue Oct 18 07:56:01 2022
write: IOPS=2997, BW=11.7MiB/s (12.3MB/s)(176MiB/15022msec); 0 zone resets
bw (  KiB/s): min=11808, max=12184, per=99.92%, avg=11996.00, stdev=51.79, samples=30
iops        : min= 2952, max= 3046, avg=2999.00, stdev=12.95, samples=30
cpu          : usr=0.40%, sys=1.04%, ctx=13036, majf=0, minf=15
IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,45026,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
WRITE: bw=11.7MiB/s (12.3MB/s), 11.7MiB/s-11.7MiB/s (12.3MB/s-12.3MB/s), io=176MiB (185MB), run=15022-15022msec

Disk stats (read/write):
nvme2n1: ios=0/52619, merge=0/20, ticks=0/540636, in_queue=540776, util=99.43%


Testing Read Bandwidth...
read_bw: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=64
fio-3.13
Starting 1 process

read_bw: (groupid=0, jobs=1): err= 0: pid=37: Tue Oct 18 07:56:19 2022
read: IOPS=999, BW=125MiB/s (132MB/s)(1892MiB/15080msec)
bw (  KiB/s): min=121088, max=134912, per=99.78%, avg=128204.73, stdev=2722.06, samples=30
iops        : min=  946, max= 1054, avg=1001.57, stdev=21.26, samples=30
cpu          : usr=0.16%, sys=0.69%, ctx=4062, majf=0, minf=15
IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=15075,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=125MiB/s (132MB/s), 125MiB/s-125MiB/s (132MB/s-132MB/s), io=1892MiB (1984MB), run=15080-15080msec

Disk stats (read/write):
nvme2n1: ios=18031/3, merge=0/1, ticks=565048/44, in_queue=565480, util=99.42%


Testing Write Bandwidth...
write_bw: (g=0): rw=randwrite, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=64
fio-3.13
Starting 1 process

write_bw: (groupid=0, jobs=1): err= 0: pid=49: Tue Oct 18 07:56:36 2022
write: IOPS=999, BW=125MiB/s (132MB/s)(1887MiB/15038msec); 0 zone resets
bw (  KiB/s): min=122368, max=134642, per=99.78%, avg=128203.90, stdev=3227.58, samples=30
iops        : min=  956, max= 1051, avg=1001.53, stdev=25.10, samples=30
cpu          : usr=0.48%, sys=0.77%, ctx=4129, majf=0, minf=15
IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,15032,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
WRITE: bw=125MiB/s (132MB/s), 125MiB/s-125MiB/s (132MB/s-132MB/s), io=1887MiB (1979MB), run=15038-15038msec

Disk stats (read/write):
nvme2n1: ios=0/17567, merge=0/3, ticks=0/554272, in_queue=554916, util=99.42%


Testing Read Latency...
read_latency: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.13
Starting 1 process

read_latency: (groupid=0, jobs=1): err= 0: pid=61: Tue Oct 18 07:56:54 2022
read: IOPS=2999, BW=11.7MiB/s (12.3MB/s)(176MiB/15001msec)
slat (nsec): min=1239, max=49196, avg=2330.03, stdev=1456.48
clat (usec): min=322, max=6106, avg=1330.33, stdev=194.34
lat (usec): min=330, max=6110, avg=1332.75, stdev=194.25
clat percentiles (usec):
|  1.00th=[  906],  5.00th=[ 1012], 10.00th=[ 1090], 20.00th=[ 1188],
| 30.00th=[ 1254], 40.00th=[ 1303], 50.00th=[ 1336], 60.00th=[ 1352],
| 70.00th=[ 1401], 80.00th=[ 1467], 90.00th=[ 1582], 95.00th=[ 1663],
| 99.00th=[ 1827], 99.50th=[ 1893], 99.90th=[ 2114], 99.95th=[ 2376],
| 99.99th=[ 3064]
bw (  KiB/s): min=11976, max=12008, per=100.00%, avg=11999.20, stdev= 9.48, samples=30
iops        : min= 2994, max= 3002, avg=2999.80, stdev= 2.37, samples=30
lat (usec)   : 500=0.08%, 750=0.22%, 1000=3.81%
lat (msec)   : 2=95.68%, 4=0.21%, 10=0.01%
cpu          : usr=0.77%, sys=1.63%, ctx=44476, majf=0, minf=15
IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=44997,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
READ: bw=11.7MiB/s (12.3MB/s), 11.7MiB/s-11.7MiB/s (12.3MB/s-12.3MB/s), io=176MiB (184MB), run=15001-15001msec

Disk stats (read/write):
nvme2n1: ios=53998/3, merge=0/1, ticks=67516/0, in_queue=67516, util=99.42%


Testing Write Latency...
write_latency: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.13
Starting 1 process

write_latency: (groupid=0, jobs=1): err= 0: pid=73: Tue Oct 18 07:57:12 2022
write: IOPS=2999, BW=11.7MiB/s (12.3MB/s)(176MiB/15001msec); 0 zone resets
slat (nsec): min=1573, max=211898, avg=2906.11, stdev=2062.04
clat (usec): min=535, max=10202, avg=1329.76, stdev=199.07
lat (usec): min=541, max=10204, avg=1332.77, stdev=198.98
clat percentiles (usec):
|  1.00th=[  857],  5.00th=[ 1045], 10.00th=[ 1123], 20.00th=[ 1205],
| 30.00th=[ 1254], 40.00th=[ 1303], 50.00th=[ 1336], 60.00th=[ 1369],
| 70.00th=[ 1401], 80.00th=[ 1450], 90.00th=[ 1516], 95.00th=[ 1598],
| 99.00th=[ 1827], 99.50th=[ 1958], 99.90th=[ 2343], 99.95th=[ 2769],
| 99.99th=[ 5604]
bw (  KiB/s): min=11968, max=12008, per=99.99%, avg=11997.24, stdev=11.37, samples=29
iops        : min= 2992, max= 3002, avg=2999.31, stdev= 2.84, samples=29
lat (usec)   : 750=0.37%, 1000=3.12%
lat (msec)   : 2=96.09%, 4=0.40%, 10=0.02%, 20=0.01%
cpu          : usr=0.88%, sys=1.73%, ctx=44800, majf=0, minf=15
IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,44994,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
WRITE: bw=11.7MiB/s (12.3MB/s), 11.7MiB/s-11.7MiB/s (12.3MB/s-12.3MB/s), io=176MiB (184MB), run=15001-15001msec

Disk stats (read/write):
nvme2n1: ios=0/51872, merge=0/3, ticks=0/66336, in_queue=66328, util=99.35%


Testing Read Sequential Speed...
read_seq: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.13
Starting 1 thread

read_seq: (groupid=0, jobs=1): err= 0: pid=85: Tue Oct 18 07:57:29 2022
read: IOPS=124, BW=126MiB/s (132MB/s)(1929MiB/15350msec)
bw (  KiB/s): min=112640, max=139264, per=99.61%, avg=128178.20, stdev=4948.41, samples=30
iops        : min=  110, max=  136, avg=125.10, stdev= 4.77, samples=30
cpu          : usr=0.00%, sys=0.52%, ctx=2234, majf=0, minf=0
IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, >=64=0.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1914,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
READ: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=1929MiB (2023MB), run=15350-15350msec

Disk stats (read/write):
nvme2n1: ios=9158/3, merge=0/1, ticks=555836/88, in_queue=556884, util=99.45%


Testing Write Sequential Speed...
write_seq: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
...
fio-3.13
Starting 4 threads
write_seq: Laying out IO file (1 file / 2798MiB)

write_seq: (groupid=0, jobs=1): err= 0: pid=97: Tue Oct 18 07:57:48 2022
write: IOPS=31, BW=31.0MiB/s (33.5MB/s)(503MiB/15726msec); 0 zone resets
bw (  KiB/s): min=20480, max=40960, per=24.44%, avg=32039.16, stdev=4774.88, samples=31
iops        : min=   20, max=   40, avg=31.26, stdev= 4.66, samples=31
cpu          : usr=0.13%, sys=0.15%, ctx=820, majf=0, minf=0
IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, >=64=0.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.2%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,488,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=16
write_seq: (groupid=0, jobs=1): err= 0: pid=98: Tue Oct 18 07:57:48 2022
write: IOPS=30, BW=31.3MiB/s (32.8MB/s)(497MiB/15886msec); 0 zone resets
bw (  KiB/s): min=22528, max=43008, per=23.89%, avg=31310.58, stdev=4966.47, samples=31
iops        : min=   22, max=   42, avg=30.52, stdev= 4.86, samples=31
cpu          : usr=0.03%, sys=0.28%, ctx=817, majf=0, minf=0
IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, >=64=0.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.2%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,482,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=16
write_seq: (groupid=0, jobs=1): err= 0: pid=99: Tue Oct 18 07:57:48 2022
write: IOPS=31, BW=31.9MiB/s (33.5MB/s)(515MiB/16128msec); 0 zone resets
bw (  KiB/s): min=26570, max=51200, per=24.36%, avg=31930.19, stdev=5672.31, samples=32
iops        : min=   25, max=   50, avg=31.09, stdev= 5.57, samples=32
cpu          : usr=0.15%, sys=0.15%, ctx=895, majf=0, minf=0
IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, >=64=0.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.2%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,500,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=16
write_seq: (groupid=0, jobs=1): err= 0: pid=100: Tue Oct 18 07:57:48 2022
write: IOPS=33, BW=34.9MiB/s (36.6MB/s)(567MiB/16265msec); 0 zone resets
bw (  KiB/s): min=22528, max=55296, per=25.98%, avg=34048.00, stdev=5902.52, samples=32
iops        : min=   22, max=   54, avg=33.25, stdev= 5.76, samples=32
cpu          : usr=0.12%, sys=0.20%, ctx=946, majf=0, minf=0
IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, >=64=0.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.2%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,552,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
WRITE: bw=128MiB/s (134MB/s), 31.3MiB/s-34.9MiB/s (32.8MB/s-36.6MB/s), io=2082MiB (2183MB), run=15726-16265msec

Disk stats (read/write):
nvme2n1: ios=0/9450, merge=0/88, ticks=0/852524, in_queue=852584, util=99.44%


Testing Read/Write Mixed...
rw_mix: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.13
Starting 1 process

rw_mix: (groupid=0, jobs=1): err= 0: pid=112: Tue Oct 18 07:58:06 2022
read: IOPS=3070, BW=12.0MiB/s (12.6MB/s)(180MiB/15026msec)
bw (  KiB/s): min=11760, max=12808, per=100.00%, avg=12298.87, stdev=262.57, samples=30
iops        : min= 2940, max= 3202, avg=3074.67, stdev=65.64, samples=30
write: IOPS=1011, BW=4054KiB/s (4152kB/s)(59.5MiB/15026msec); 0 zone resets
bw (  KiB/s): min= 3848, max= 4336, per=99.96%, avg=4052.23, stdev=135.92, samples=30
iops        : min=  962, max= 1084, avg=1013.03, stdev=33.97, samples=30
cpu          : usr=0.48%, sys=1.73%, ctx=11406, majf=0, minf=15
IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=46142,15205,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=12.0MiB/s (12.6MB/s), 12.0MiB/s-12.0MiB/s (12.6MB/s-12.6MB/s), io=180MiB (189MB), run=15026-15026msec
WRITE: bw=4054KiB/s (4152kB/s), 4054KiB/s-4054KiB/s (4152kB/s-4152kB/s), io=59.5MiB (62.4MB), run=15026-15026msec

Disk stats (read/write):
nvme2n1: ios=35817/18189, merge=0/171, ticks=370700/192060, in_queue=563056, util=99.43%


All tests complete.

==================
= Dbench Summary =
==================
Random Read/Write IOPS: 2997/2997. BW: 125MiB/s / 125MiB/s
Average Latency (usec) Read/Write: 1332.75/1332.77
Sequential Read/Write: 126MiB/s / 128MiB/s
Mixed Random Read/Write IOPS: 3070/1011

第二次压测日志 

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280

# kubectl logs -f dbench-s7mtt-vqqr4 
Working dir: /data

Testing Read IOPS...
read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.13
Starting 1 process
read_iops: Laying out IO file (1 file / 2048MiB)

read_iops: (groupid=0, jobs=1): err= 0: pid=13: Tue Oct 18 08:13:55 2022
read: IOPS=2997, BW=11.7MiB/s (12.3MB/s)(176MiB/15027msec)
bw (  KiB/s): min=11816, max=12200, per=99.92%, avg=11997.87, stdev=71.61, samples=30
iops        : min= 2954, max= 3050, avg=2999.47, stdev=17.90, samples=30
cpu          : usr=0.27%, sys=1.06%, ctx=13230, majf=0, minf=15
IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=45050,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=11.7MiB/s (12.3MB/s), 11.7MiB/s-11.7MiB/s (12.3MB/s-12.3MB/s), io=176MiB (185MB), run=15027-15027msec

Disk stats (read/write):
nvme2n1: ios=53999/2, merge=0/1, ticks=540308/20, in_queue=540500, util=99.44%


Testing Write IOPS...
write_iops: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.13
Starting 1 process

write_iops: (groupid=0, jobs=1): err= 0: pid=25: Tue Oct 18 08:14:13 2022
write: IOPS=2997, BW=11.7MiB/s (12.3MB/s)(176MiB/15021msec); 0 zone resets
bw (  KiB/s): min=11824, max=12224, per=99.91%, avg=11995.47, stdev=68.68, samples=30
iops        : min= 2956, max= 3056, avg=2998.87, stdev=17.17, samples=30
cpu          : usr=0.32%, sys=1.15%, ctx=12096, majf=0, minf=15
IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,45023,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
WRITE: bw=11.7MiB/s (12.3MB/s), 11.7MiB/s-11.7MiB/s (12.3MB/s-12.3MB/s), io=176MiB (185MB), run=15021-15021msec

Disk stats (read/write):
nvme2n1: ios=0/52621, merge=0/20, ticks=0/554696, in_queue=554804, util=99.42%


Testing Read Bandwidth...
read_bw: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=64
fio-3.13
Starting 1 process

read_bw: (groupid=0, jobs=1): err= 0: pid=37: Tue Oct 18 08:14:31 2022
read: IOPS=999, BW=125MiB/s (132MB/s)(1892MiB/15081msec)
bw (  KiB/s): min=122112, max=134144, per=99.79%, avg=128221.80, stdev=1735.92, samples=30
iops        : min=  954, max= 1048, avg=1001.70, stdev=13.56, samples=30
cpu          : usr=0.05%, sys=0.74%, ctx=4397, majf=0, minf=15
IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=15076,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=125MiB/s (132MB/s), 125MiB/s-125MiB/s (132MB/s-132MB/s), io=1892MiB (1984MB), run=15081-15081msec

Disk stats (read/write):
nvme2n1: ios=18030/2, merge=0/1, ticks=546088/60, in_queue=546624, util=99.44%


Testing Write Bandwidth...
write_bw: (g=0): rw=randwrite, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=64
fio-3.13
Starting 1 process

write_bw: (groupid=0, jobs=1): err= 0: pid=49: Tue Oct 18 08:14:48 2022
write: IOPS=999, BW=125MiB/s (132MB/s)(1893MiB/15086msec); 0 zone resets
bw (  KiB/s): min=123648, max=133376, per=99.76%, avg=128187.70, stdev=1641.17, samples=30
iops        : min=  966, max= 1042, avg=1001.43, stdev=12.82, samples=30
cpu          : usr=0.48%, sys=0.77%, ctx=4275, majf=0, minf=15
IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,15081,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
WRITE: bw=125MiB/s (132MB/s), 125MiB/s-125MiB/s (132MB/s-132MB/s), io=1893MiB (1985MB), run=15086-15086msec

Disk stats (read/write):
nvme2n1: ios=0/17582, merge=0/3, ticks=0/554348, in_queue=554852, util=99.43%


Testing Read Latency...
read_latency: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.13
Starting 1 process

read_latency: (groupid=0, jobs=1): err= 0: pid=61: Tue Oct 18 08:15:06 2022
read: IOPS=2999, BW=11.7MiB/s (12.3MB/s)(176MiB/15001msec)
slat (nsec): min=1239, max=53956, avg=2736.42, stdev=2015.01
clat (usec): min=322, max=4819, avg=1329.74, stdev=196.90
lat (usec): min=331, max=4824, avg=1332.59, stdev=196.79
clat percentiles (usec):
|  1.00th=[  889],  5.00th=[ 1020], 10.00th=[ 1090], 20.00th=[ 1205],
| 30.00th=[ 1254], 40.00th=[ 1303], 50.00th=[ 1319], 60.00th=[ 1352],
| 70.00th=[ 1385], 80.00th=[ 1450], 90.00th=[ 1565], 95.00th=[ 1663],
| 99.00th=[ 1827], 99.50th=[ 1926], 99.90th=[ 2376], 99.95th=[ 2933],
| 99.99th=[ 4178]
bw (  KiB/s): min=11968, max=12040, per=100.00%, avg=11999.47, stdev=14.39, samples=30
iops        : min= 2992, max= 3010, avg=2999.87, stdev= 3.60, samples=30
lat (usec)   : 500=0.14%, 750=0.34%, 1000=3.48%
lat (msec)   : 2=95.75%, 4=0.28%, 10=0.01%
cpu          : usr=0.91%, sys=1.71%, ctx=44388, majf=0, minf=15
IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=44998,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
READ: bw=11.7MiB/s (12.3MB/s), 11.7MiB/s-11.7MiB/s (12.3MB/s-12.3MB/s), io=176MiB (184MB), run=15001-15001msec

Disk stats (read/write):
nvme2n1: ios=53999/2, merge=0/1, ticks=67188/0, in_queue=67204, util=99.32%


Testing Write Latency...
write_latency: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.13
Starting 1 process

write_latency: (groupid=0, jobs=1): err= 0: pid=73: Tue Oct 18 08:15:23 2022
write: IOPS=2999, BW=11.7MiB/s (12.3MB/s)(176MiB/15002msec); 0 zone resets
slat (nsec): min=1540, max=112288, avg=2741.34, stdev=1649.97
clat (usec): min=514, max=9491, avg=1329.98, stdev=221.62
lat (usec): min=517, max=9493, avg=1332.82, stdev=221.54
clat percentiles (usec):
|  1.00th=[  832],  5.00th=[ 1004], 10.00th=[ 1090], 20.00th=[ 1188],
| 30.00th=[ 1237], 40.00th=[ 1287], 50.00th=[ 1336], 60.00th=[ 1369],
| 70.00th=[ 1418], 80.00th=[ 1467], 90.00th=[ 1565], 95.00th=[ 1647],
| 99.00th=[ 1860], 99.50th=[ 1975], 99.90th=[ 2376], 99.95th=[ 2835],
| 99.99th=[ 6718]
bw (  KiB/s): min=11968, max=12008, per=100.00%, avg=11997.07, stdev=10.40, samples=30
iops        : min= 2992, max= 3002, avg=2999.27, stdev= 2.60, samples=30
lat (usec)   : 750=0.42%, 1000=4.52%
lat (msec)   : 2=94.64%, 4=0.40%, 10=0.02%
cpu          : usr=0.83%, sys=1.71%, ctx=44840, majf=0, minf=15
IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,44995,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
WRITE: bw=11.7MiB/s (12.3MB/s), 11.7MiB/s-11.7MiB/s (12.3MB/s-12.3MB/s), io=176MiB (184MB), run=15002-15002msec

Disk stats (read/write):
nvme2n1: ios=0/52717, merge=0/3, ticks=0/67412, in_queue=67408, util=99.42%


Testing Read Sequential Speed...
read_seq: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.13
Starting 1 thread

read_seq: (groupid=0, jobs=1): err= 0: pid=85: Tue Oct 18 08:15:41 2022
read: IOPS=124, BW=126MiB/s (132MB/s)(1934MiB/15388msec)
bw (  KiB/s): min=114688, max=141312, per=99.82%, avg=128469.53, stdev=5409.21, samples=30
iops        : min=  112, max=  138, avg=125.43, stdev= 5.30, samples=30
cpu          : usr=0.00%, sys=0.49%, ctx=2175, majf=0, minf=0
IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, >=64=0.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1919,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
READ: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=1934MiB (2028MB), run=15388-15388msec

Disk stats (read/write):
nvme2n1: ios=9139/2, merge=0/1, ticks=557100/72, in_queue=558068, util=99.44%


Testing Write Sequential Speed...

write_seq: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
...
fio-3.13
Starting 4 threads
write_seq: Laying out IO file (1 file / 2798MiB)

write_seq: (groupid=0, jobs=1): err= 0: pid=97: Tue Oct 18 08:16:00 2022
write: IOPS=33, BW=34.4MiB/s (36.0MB/s)(559MiB/16260msec); 0 zone resets
bw (  KiB/s): min=26624, max=47104, per=25.42%, avg=33339.75, stdev=4773.88, samples=32
iops        : min=   26, max=   46, avg=32.50, stdev= 4.66, samples=32
cpu          : usr=0.15%, sys=0.17%, ctx=907, majf=0, minf=0
IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, >=64=0.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.2%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,544,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=16
write_seq: (groupid=0, jobs=1): err= 0: pid=98: Tue Oct 18 08:16:00 2022
write: IOPS=31, BW=32.1MiB/s (33.6MB/s)(513MiB/16003msec); 0 zone resets
bw (  KiB/s): min=24576, max=40960, per=24.29%, avg=31867.88, stdev=3674.44, samples=32
iops        : min=   24, max=   40, avg=31.06, stdev= 3.58, samples=32
cpu          : usr=0.15%, sys=0.17%, ctx=899, majf=0, minf=0
IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, >=64=0.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.2%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,498,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=16
write_seq: (groupid=0, jobs=1): err= 0: pid=99: Tue Oct 18 08:16:00 2022
write: IOPS=30, BW=31.6MiB/s (33.2MB/s)(504MiB/15935msec); 0 zone resets
bw (  KiB/s): min=22528, max=40960, per=24.07%, avg=31578.84, stdev=3949.13, samples=31
iops        : min=   22, max=   40, avg=30.84, stdev= 3.86, samples=31
cpu          : usr=0.18%, sys=0.13%, ctx=865, majf=0, minf=0
IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, >=64=0.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.2%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,489,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=16
write_seq: (groupid=0, jobs=1): err= 0: pid=100: Tue Oct 18 08:16:00 2022
write: IOPS=30, BW=31.8MiB/s (33.4MB/s)(507MiB/15927msec); 0 zone resets
bw (  KiB/s): min=22528, max=40960, per=24.37%, avg=31970.97, stdev=4316.00, samples=31
iops        : min=   22, max=   40, avg=31.16, stdev= 4.21, samples=31
cpu          : usr=0.10%, sys=0.20%, ctx=884, majf=0, minf=0
IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, >=64=0.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.2%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,492,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
WRITE: bw=128MiB/s (134MB/s), 31.6MiB/s-34.4MiB/s (33.2MB/s-36.0MB/s), io=2083MiB (2184MB), run=15927-16260msec

Disk stats (read/write):
nvme2n1: ios=0/9324, merge=0/50, ticks=0/857300, in_queue=859252, util=99.45%


Testing Read/Write Mixed...
rw_mix: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.13
Starting 1 process

rw_mix: (groupid=0, jobs=1): err= 0: pid=112: Tue Oct 18 08:16:18 2022
read: IOPS=3066, BW=11.0MiB/s (12.6MB/s)(180MiB/15016msec)
bw (  KiB/s): min=11624, max=12944, per=100.00%, avg=12277.60, stdev=324.75, samples=30
iops        : min= 2906, max= 3236, avg=3069.40, stdev=81.19, samples=30
write: IOPS=1029, BW=4121KiB/s (4220kB/s)(60.4MiB/15016msec); 0 zone resets
bw (  KiB/s): min= 3800, max= 4448, per=99.95%, avg=4119.13, stdev=163.04, samples=30
iops        : min=  950, max= 1112, avg=1029.73, stdev=40.77, samples=30
cpu          : usr=0.56%, sys=1.49%, ctx=11815, majf=0, minf=15
IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=46045,15453,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=11.0MiB/s (12.6MB/s), 11.0MiB/s-11.0MiB/s (12.6MB/s-12.6MB/s), io=180MiB (189MB), run=15016-15016msec
WRITE: bw=4121KiB/s (4220kB/s), 4121KiB/s-4121KiB/s (4220kB/s-4220kB/s), io=60.4MiB (63.4MB), run=15016-15016msec

Disk stats (read/write):
nvme2n1: ios=35461/18545, merge=0/225, ticks=371724/198644, in_queue=570628, util=99.41%


All tests complete.

==================
= Dbench Summary =
==================
Random Read/Write IOPS: 2997/2997. BW: 125MiB/s / 125MiB/s
Average Latency (usec) Read/Write: 1332.59/1332.82
Sequential Read/Write: 126MiB/s / 128MiB/s
Mixed Random Read/Write IOPS: 3066/1029
root@k8s-master-7b4d73fd8d:~#

cStor PV 测试

创建存储池(cStor storage pools) 

1
2
3
4
5

# kubectl get bd -n openebs
NAME                                           NODENAME                                                SIZE          CLAIMSTATE   STATUS   AGE
blockdevice-5cc77e185a944270fa46b7d233320660   dev-vm-8c32g-600g-10-86-183-104   53686025728   Unclaimed    Active   84m
blockdevice-b3978ac55b8d0fc6ae9e2bc3b5b92fd6   dev-vm-8c64g-600g-10-86-16-225    53686025728   Unclaimed    Active   84m
blockdevice-d93d0a91351a13f4de81fd0a5e6e4540   dev-vm-4c16g-600g-10-86-141-68    53686025728   Unclaimed    Active   84m
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

cat << EOF > cspc.yaml
apiVersion: cstor.openebs.io/v1
kind: CStorPoolCluster
metadata:
 name: cstor-disk-pool
 namespace: openebs
spec:
 pools:
   - nodeSelector:
       kubernetes.io/hostname: dev-vm-8c32g-600g-10-86-183-104
     dataRaidGroups:
       - blockDevices:
           - blockDeviceName: "blockdevice-5cc77e185a944270fa46b7d233320660"
     poolConfig:
       dataRaidGroupType: "stripe"

   - nodeSelector:
       kubernetes.io/hostname: dev-vm-8c64g-600g-10-86-16-225
     dataRaidGroups:
       - blockDevices:
           - blockDeviceName: "blockdevice-b3978ac55b8d0fc6ae9e2bc3b5b92fd6"
     poolConfig:
       dataRaidGroupType: "stripe"

   - nodeSelector:
       kubernetes.io/hostname: dev-vm-4c16g-600g-10-86-141-68
     dataRaidGroups:
       - blockDevices:
           - blockDeviceName: "blockdevice-d93d0a91351a13f4de81fd0a5e6e4540"
     poolConfig:
       dataRaidGroupType: "stripe"
EOF
kubectl apply -f cspc.yaml

blockdevice的CLAIMSTATE状态为Unclaimed才可加入存储池

1
2
3
4
5
6
7
8

# kubectl get cspc -n openebs
NAME              HEALTHYINSTANCES   PROVISIONEDINSTANCES   DESIREDINSTANCES   AGE
cstor-disk-pool                      3                      3                  72s
# kubectl get cspi -n openebs
NAME                   HOSTNAME                                                FREE   CAPACITY   READONLY   PROVISIONEDREPLICAS   HEALTHYREPLICAS   STATUS   AGE
cstor-disk-pool-2v28   dev-vm-8c32g-600g-10-86-183-104   0      0          false      0                     0                          75s
cstor-disk-pool-6mpx   dev-vm-8c64g-600g-10-86-16-225    0      0          false      0                     0                          75s
cstor-disk-pool-qmz7   dev-vm-4c16g-600g-10-86-141-68    0      0          false      0                     0                          74s

创建存储卷声明(StorageClass) 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

cat << EOF > cstor-csi-disk.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: cstor-csi-disk
provisioner: cstor.csi.openebs.io
allowVolumeExpansion: true
parameters:
  cas-type: cstor
  # cstorPoolCluster should have the name of the CSPC
  cstorPoolCluster: cstor-disk-pool
  # replicaCount should be <= no. of CSPI created in the selected CSPC
  replicaCount: "2"
EOF
kubectl apply -f cstor-csi-disk.yaml

查看存储卷声明

1
2
3

# kubectl get sc cstor-csi-disk
NAME             PROVISIONER            RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
cstor-csi-disk   cstor.csi.openebs.io   Delete          Immediate           true                   8s

创建PVC 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

cat > cstor-pvc.yaml <<EOF
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: cstor-pvc
spec:
  storageClassName: cstor-csi-disk
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5G
EOF

kubectl apply -f cstor-pvc.yaml

验证PVC挂载是否正常 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

cat <<EOF >cstor-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - command:
       - sh
       - -c
       - 'date >> /mnt/openebs-csi/date.txt; hostname >> /mnt/openebs-csi/hostname.txt; sync; sleep 5; sync; tail -f /dev/null;'
    image: busybox
    imagePullPolicy: Always
    name: busybox
    volumeMounts:
    - mountPath: /mnt/openebs-csi
      name: demo-vol
  volumes:
  - name: demo-vol
    persistentVolumeClaim:
      claimName: cstor-pvc
EOF
kubectl apply -f cstor-pod.yaml

创建压测任务 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93

cat << EOF > fio-job.yaml
# NOTE: For details of params to construct an fio job, refer to this link:
# https://fio.readthedocs.io/en/latest/fio_doc.html

---
apiVersion: batch/v1
kind: Job
metadata:
  generateName: dbench-
spec:
  template:
    spec:
      containers:
      - name: dbench
        image: openebs/perf-test:latest
        imagePullPolicy: IfNotPresent
        env:

          ## storage mount point on which testfiles are created

          - name: DBENCH_MOUNTPOINT
            value: /data

          ##########################################################
          # I/O PROFILE COVERAGE FOR SPECIFIC PERF CHARACTERISTICS #
          ##########################################################

          ## quick: {read, write} iops, {read, write} bw (all random)
          ## detailed: {quick}, {read, write} latency & mixed 75r:25w (all random), {read, write} bw (all sequential)  
          ## custom: a single user-defined job run with params specified in env 'CUSTOM'

          - name: DBENCH_TYPE
            value: detailed

          ####################################################
          # STANDARD TUNABLES FOR DBENCH_TYPE=QUICK/DETAILED #
          ####################################################

          ## active data size for the bench test

          - name: FIO_SIZE
            value: 2G

          ## use un-buffered i/o (usually O_DIRECT)

          - name: FIO_DIRECT
            value: '1'

          ## no of independent threads doing the same i/o

          - name: FIO_NUMJOBS
            value: '1'

          ## space b/w starting offsets on a file in case of parallel file i/o

          - name: FIO_OFFSET_INCREMENT
            value: 250M

          ## nature of i/o to file. commonly supported: libaio, sync, 

          - name: FIO_IOENGINE
            value: libaio

          ## additional runtime options which will be appended to the above params
          ## ensure options used are not mutually exclusive w/ above params
          ## ex: '--group_reporting=1, stonewall, --ramptime=<val> etc..,
          
          - name: OPTIONS
            value: ''

          ####################################################
          # CUSTOM JOB SPEC FOR DBENCH_TYPE=CUSTOM           #
          ####################################################

          ## this will execute a single job run with the params specified 
          ## ex: '--bs=16k --iodepth=64 --ioengine=sync --size=500M --name=custom --readwrite=randrw --rwmixread=80 --random_distribution=pareto' 

          - name: CUSTOM
            value: ''
          
        volumeMounts:
        - name: dbench-pv
          mountPath: /data
      restartPolicy: Never
      volumes:
      - name: dbench-pv
        persistentVolumeClaim:
          claimName: cstor-pvc   ## 修改pvc
  backoffLimit: 4
---
EOF

kubectl create -f fio-job.yaml

第一次测试 

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279

root@k8s-master-7b4d73fd8d:~# kubectl logs -f dbench-4j6xm-v8xzk
Working dir: /data

Testing Read IOPS...
read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.13
Starting 1 process
read_iops: Laying out IO file (1 file / 2048MiB)

read_iops: (groupid=0, jobs=1): err= 0: pid=17: Tue Oct 18 10:44:57 2022
  read: IOPS=8859, BW=34.6MiB/s (36.3MB/s)(520MiB/15014msec)
   bw (  KiB/s): min=33912, max=36200, per=99.98%, avg=35446.50, stdev=496.88, samples=30
   iops        : min= 8478, max= 9050, avg=8861.60, stdev=124.22, samples=30
  cpu          : usr=2.42%, sys=9.91%, ctx=127613, majf=0, minf=15
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=133015,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=34.6MiB/s (36.3MB/s), 34.6MiB/s-34.6MiB/s (36.3MB/s-36.3MB/s), io=520MiB (545MB), run=15014-15014msec

Disk stats (read/write):
  sda: ios=149875/8, merge=20/18, ticks=1085532/44, in_queue=1086020, util=99.44%


Testing Write IOPS...
write_iops: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.13
Starting 1 process

write_iops: (groupid=0, jobs=1): err= 0: pid=33: Tue Oct 18 10:45:15 2022
  write: IOPS=2064, BW=8276KiB/s (8474kB/s)(125MiB/15503msec); 0 zone resets
   bw (  KiB/s): min=  152, max=13240, per=99.81%, avg=8259.35, stdev=3710.86, samples=31
   iops        : min=   38, max= 3310, avg=2064.81, stdev=927.72, samples=31
  cpu          : usr=1.01%, sys=2.45%, ctx=30501, majf=0, minf=15
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,32012,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=8276KiB/s (8474kB/s), 8276KiB/s-8276KiB/s (8474kB/s-8474kB/s), io=125MiB (131MB), run=15503-15503msec

Disk stats (read/write):
  sda: ios=0/37817, merge=0/13, ticks=0/1087692, in_queue=1088200, util=99.47%


Testing Read Bandwidth...
read_bw: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=64
fio-3.13
Starting 1 process

read_bw: (groupid=0, jobs=1): err= 0: pid=49: Tue Oct 18 10:45:33 2022
  read: IOPS=3512, BW=440MiB/s (461MB/s)(6617MiB/15055msec)
   bw (  KiB/s): min=386304, max=500480, per=100.00%, avg=450733.17, stdev=29199.52, samples=30
   iops        : min= 3018, max= 3910, avg=3521.30, stdev=228.13, samples=30
  cpu          : usr=1.36%, sys=6.22%, ctx=51800, majf=0, minf=15
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=52874,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=440MiB/s (461MB/s), 440MiB/s-440MiB/s (461MB/s-461MB/s), io=6617MiB (6939MB), run=15055-15055msec

Disk stats (read/write):
  sda: ios=58329/2, merge=1149/1, ticks=1065600/92, in_queue=1066352, util=99.44%


Testing Write Bandwidth...
write_bw: (g=0): rw=randwrite, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=64
fio-3.13
Starting 1 process

write_bw: (groupid=0, jobs=1): err= 0: pid=65: Tue Oct 18 10:45:51 2022
  write: IOPS=587, BW=73.0MiB/s (77.6MB/s)(1119MiB/15121msec); 0 zone resets
   bw (  KiB/s): min=16384, max=155904, per=100.00%, avg=90962.80, stdev=37067.55, samples=25
   iops        : min=  128, max= 1218, avg=710.60, stdev=289.64, samples=25
  cpu          : usr=0.45%, sys=1.24%, ctx=8473, majf=0, minf=15
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,8886,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=73.0MiB/s (77.6MB/s), 73.0MiB/s-73.0MiB/s (77.6MB/s-77.6MB/s), io=1119MiB (1173MB), run=15121-15121msec

Disk stats (read/write):
  sda: ios=0/11066, merge=0/78, ticks=0/1078704, in_queue=1080272, util=99.42%


Testing Read Latency...
read_latency: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.13
Starting 1 process

read_latency: (groupid=0, jobs=1): err= 0: pid=81: Tue Oct 18 10:46:08 2022
  read: IOPS=5914, BW=23.1MiB/s (24.2MB/s)(347MiB/15001msec)
    slat (usec): min=2, max=536, avg= 9.27, stdev= 4.56
    clat (usec): min=285, max=27620, avg=665.58, stdev=609.19
     lat (usec): min=292, max=27630, avg=675.01, stdev=609.48
    clat percentiles (usec):
     |  1.00th=[  334],  5.00th=[  367], 10.00th=[  392], 20.00th=[  457],
     | 30.00th=[  498], 40.00th=[  515], 50.00th=[  529], 60.00th=[  553],
     | 70.00th=[  578], 80.00th=[  652], 90.00th=[ 1172], 95.00th=[ 1303],
     | 99.00th=[ 2040], 99.50th=[ 3392], 99.90th=[ 8848], 99.95th=[11863],
     | 99.99th=[21365]
   bw (  KiB/s): min=13160, max=25904, per=99.87%, avg=23627.59, stdev=2895.10, samples=29
   iops        : min= 3290, max= 6476, avg=5906.90, stdev=723.77, samples=29
  lat (usec)   : 500=31.45%, 750=50.10%, 1000=2.67%
  lat (msec)   : 2=14.75%, 4=0.65%, 10=0.31%, 20=0.06%, 50=0.02%
  cpu          : usr=2.08%, sys=8.48%, ctx=80129, majf=0, minf=15
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=88725,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=23.1MiB/s (24.2MB/s), 23.1MiB/s-23.1MiB/s (24.2MB/s-24.2MB/s), io=347MiB (363MB), run=15001-15001msec

Disk stats (read/write):
  sda: ios=98750/2, merge=0/1, ticks=66048/24, in_queue=66056, util=99.34%


Testing Write Latency...
write_latency: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.13
Starting 1 process

write_latency: (groupid=0, jobs=1): err= 0: pid=97: Tue Oct 18 10:46:26 2022
  write: IOPS=1985, BW=7943KiB/s (8133kB/s)(117MiB/15043msec); 0 zone resets
    slat (usec): min=2, max=1804, avg=10.74, stdev=14.56
    clat (usec): min=171, max=502636, avg=2035.87, stdev=24315.27
     lat (usec): min=354, max=502644, avg=2046.78, stdev=24315.25
    clat percentiles (usec):
     |  1.00th=[   375],  5.00th=[   392], 10.00th=[   404], 20.00th=[   445],
     | 30.00th=[   578], 40.00th=[   611], 50.00th=[   652], 60.00th=[   766],
     | 70.00th=[   816], 80.00th=[   898], 90.00th=[  1057], 95.00th=[  1467],
     | 99.00th=[  5342], 99.50th=[  9372], 99.90th=[501220], 99.95th=[501220],
     | 99.99th=[501220]
   bw (  KiB/s): min= 2696, max=12688, per=100.00%, avg=7963.57, stdev=2832.99, samples=30
   iops        : min=  674, max= 3172, avg=1990.87, stdev=708.27, samples=30
  lat (usec)   : 250=0.01%, 500=24.64%, 750=33.35%, 1000=28.16%
  lat (msec)   : 2=10.30%, 4=2.08%, 10=0.99%, 20=0.20%, 50=0.02%
  lat (msec)   : 100=0.01%, 250=0.02%, 500=0.01%, 750=0.23%
  cpu          : usr=0.90%, sys=2.77%, ctx=23583, majf=0, minf=15
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,29868,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
  WRITE: bw=7943KiB/s (8133kB/s), 7943KiB/s-7943KiB/s (8133kB/s-8133kB/s), io=117MiB (122MB), run=15043-15043msec

Disk stats (read/write):
  sda: ios=0/34133, merge=0/3, ticks=0/67908, in_queue=68364, util=99.44%


Testing Read Sequential Speed...
read_seq: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.13
Starting 1 thread

read_seq: (groupid=0, jobs=1): err= 0: pid=113: Tue Oct 18 10:46:46 2022
  read: IOPS=22, BW=22.9MiB/s (24.0MB/s)(389MiB/16992msec)
   bw (  KiB/s): min=10219, max=124928, per=97.39%, avg=22830.27, stdev=27843.90, samples=33
   iops        : min=    9, max=  122, avg=22.18, stdev=27.16, samples=33
  cpu          : usr=0.00%, sys=0.16%, ctx=330, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.7%, 8=0.0%, 16=0.3%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=374,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=22.9MiB/s (24.0MB/s), 22.9MiB/s-22.9MiB/s (24.0MB/s-24.0MB/s), io=389MiB (408MB), run=16992-16992msec

Disk stats (read/write):
  sda: ios=430/2, merge=0/1, ticks=295480/4548, in_queue=306836, util=99.49%


Testing Write Sequential Speed...
write_seq: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
...
fio-3.13
Starting 4 threads
write_seq: Laying out IO file (1 file / 2798MiB)

write_seq: (groupid=0, jobs=1): err= 0: pid=129: Tue Oct 18 10:47:06 2022
  write: IOPS=16, BW=17.4MiB/s (18.3MB/s)(301MiB/17271msec); 0 zone resets
   bw (  KiB/s): min= 2048, max=65536, per=44.80%, avg=30233.84, stdev=12799.84, samples=19
   iops        : min=    2, max=   64, avg=29.47, stdev=12.47, samples=19
  cpu          : usr=0.05%, sys=0.14%, ctx=266, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.7%, 8=0.0%, 16=0.3%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,286,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
write_seq: (groupid=0, jobs=1): err= 0: pid=130: Tue Oct 18 10:47:06 2022
  write: IOPS=16, BW=17.2MiB/s (18.1MB/s)(300MiB/17421msec); 0 zone resets
   bw (  KiB/s): min= 8192, max=75923, per=46.21%, avg=31184.22, stdev=17161.58, samples=18
   iops        : min=    8, max=   74, avg=30.28, stdev=16.80, samples=18
  cpu          : usr=0.05%, sys=0.14%, ctx=290, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.7%, 8=0.0%, 16=0.3%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,285,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
write_seq: (groupid=0, jobs=1): err= 0: pid=131: Tue Oct 18 10:47:06 2022
  write: IOPS=15, BW=16.4MiB/s (17.2MB/s)(277MiB/16890msec); 0 zone resets
   bw (  KiB/s): min= 4096, max=53248, per=37.14%, avg=25062.81, stdev=12968.89, samples=21
   iops        : min=    4, max=   52, avg=24.43, stdev=12.71, samples=21
  cpu          : usr=0.00%, sys=0.17%, ctx=272, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.6%, 8=0.0%, 16=0.4%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,262,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
write_seq: (groupid=0, jobs=1): err= 0: pid=132: Tue Oct 18 10:47:06 2022
  write: IOPS=15, BW=16.5MiB/s (17.3MB/s)(270MiB/16387msec); 0 zone resets
   bw (  KiB/s): min= 2048, max=65536, per=36.85%, avg=24865.62, stdev=15505.40, samples=21
   iops        : min=    2, max=   64, avg=24.24, stdev=15.13, samples=21
  cpu          : usr=0.05%, sys=0.12%, ctx=242, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.6%, 8=0.0%, 16=0.4%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,255,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=65.9MiB/s (69.1MB/s), 16.4MiB/s-17.4MiB/s (17.2MB/s-18.3MB/s), io=1148MiB (1204MB), run=16387-17421msec

Disk stats (read/write):
  sda: ios=0/1518, merge=0/51, ticks=0/1168892, in_queue=1171940, util=99.50%


Testing Read/Write Mixed...
rw_mix: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.13
Starting 1 process

rw_mix: (groupid=0, jobs=1): err= 0: pid=148: Tue Oct 18 10:47:24 2022
  read: IOPS=4423, BW=17.3MiB/s (18.1MB/s)(268MiB/15513msec)
   bw (  KiB/s): min=  120, max=28384, per=100.00%, avg=17704.81, stdev=8247.77, samples=31
   iops        : min=   30, max= 7096, avg=4426.19, stdev=2061.94, samples=31
  write: IOPS=1451, BW=5816KiB/s (5955kB/s)(88.1MiB/15513msec); 0 zone resets
   bw (  KiB/s): min=   88, max= 8704, per=99.94%, avg=5811.55, stdev=2637.87, samples=31
   iops        : min=   22, max= 2176, avg=1452.87, stdev=659.47, samples=31
  cpu          : usr=1.24%, sys=5.85%, ctx=56649, majf=0, minf=15
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=68624,22523,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=17.3MiB/s (18.1MB/s), 17.3MiB/s-17.3MiB/s (18.1MB/s-18.1MB/s), io=268MiB (281MB), run=15513-15513msec
  WRITE: bw=5816KiB/s (5955kB/s), 5816KiB/s-5816KiB/s (5955kB/s-5955kB/s), io=88.1MiB (92.4MB), run=15513-15513msec

Disk stats (read/write):
  sda: ios=40368/24093, merge=1/225, ticks=531908/556304, in_queue=1088700, util=99.47%


All tests complete.

==================
= Dbench Summary =
==================
Random Read/Write IOPS: 8859/2064. BW: 440MiB/s / 73.0MiB/s
Average Latency (usec) Read/Write: 675.01/2046.78
Sequential Read/Write: 22.9MiB/s / 65.9MiB/s
Mixed Random Read/Write IOPS: 4423/1451

第二次测试 

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280

root@k8s-master-7b4d73fd8d:~# kubectl logs -f dbench-jbhd6-5llrh
Working dir: /data

Testing Read IOPS...
read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.13
Starting 1 process
read_iops: Laying out IO file (1 file / 2048MiB)

read_iops: (groupid=0, jobs=1): err= 0: pid=17: Tue Oct 18 10:49:04 2022
  read: IOPS=8569, BW=33.5MiB/s (35.1MB/s)(503MiB/15013msec)
   bw (  KiB/s): min=30248, max=35544, per=100.00%, avg=34292.93, stdev=1149.85, samples=30
   iops        : min= 7562, max= 8886, avg=8573.20, stdev=287.44, samples=30
  cpu          : usr=2.50%, sys=9.65%, ctx=123769, majf=0, minf=15
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=128654,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=33.5MiB/s (35.1MB/s), 33.5MiB/s-33.5MiB/s (35.1MB/s-35.1MB/s), io=503MiB (527MB), run=15013-15013msec

Disk stats (read/write):
  sda: ios=144442/2, merge=31/1, ticks=1085432/20, in_queue=1086668, util=99.47%


Testing Write IOPS...
write_iops: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.13
Starting 1 process

write_iops: (groupid=0, jobs=1): err= 0: pid=33: Tue Oct 18 10:49:22 2022
  write: IOPS=1916, BW=7680KiB/s (7865kB/s)(116MiB/15498msec); 0 zone resets
   bw (  KiB/s): min=  688, max=13664, per=100.00%, avg=7906.90, stdev=3126.87, samples=30
   iops        : min=  172, max= 3416, avg=1976.67, stdev=781.72, samples=30
  cpu          : usr=0.62%, sys=2.63%, ctx=29085, majf=0, minf=15
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,29695,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=7680KiB/s (7865kB/s), 7680KiB/s-7680KiB/s (7865kB/s-7865kB/s), io=116MiB (122MB), run=15498-15498msec

Disk stats (read/write):
  sda: ios=0/34694, merge=0/21, ticks=0/1089984, in_queue=1091036, util=99.48%


Testing Read Bandwidth...
read_bw: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=64
fio-3.13
Starting 1 process

read_bw: (groupid=0, jobs=1): err= 0: pid=49: Tue Oct 18 10:49:40 2022
  read: IOPS=2690, BW=337MiB/s (353MB/s)(5067MiB/15041msec)
   bw (  KiB/s): min=265984, max=425216, per=100.00%, avg=345130.80, stdev=45757.32, samples=30
   iops        : min= 2078, max= 3322, avg=2696.33, stdev=357.48, samples=30
  cpu          : usr=0.98%, sys=5.16%, ctx=39652, majf=0, minf=15
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=40472,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=337MiB/s (353MB/s), 337MiB/s-337MiB/s (353MB/s-353MB/s), io=5067MiB (5313MB), run=15041-15041msec

Disk stats (read/write):
  sda: ios=44388/3, merge=837/1, ticks=1062960/108, in_queue=1063800, util=99.46%


Testing Write Bandwidth...
write_bw: (g=0): rw=randwrite, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=64
fio-3.13
Starting 1 process

write_bw: (groupid=0, jobs=1): err= 0: pid=65: Tue Oct 18 10:49:57 2022
  write: IOPS=593, BW=74.7MiB/s (78.4MB/s)(1129MiB/15100msec); 0 zone resets
   bw (  KiB/s): min= 9472, max=182528, per=100.00%, avg=91171.60, stdev=44402.78, samples=25
   iops        : min=   74, max= 1426, avg=712.24, stdev=346.86, samples=25
  cpu          : usr=0.42%, sys=1.27%, ctx=8676, majf=0, minf=15
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,8966,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=74.7MiB/s (78.4MB/s), 74.7MiB/s-74.7MiB/s (78.4MB/s-78.4MB/s), io=1129MiB (1183MB), run=15100-15100msec

Disk stats (read/write):
  sda: ios=0/11142, merge=0/70, ticks=0/1080072, in_queue=1082452, util=99.44%


Testing Read Latency...
read_latency: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.13
Starting 1 process

read_latency: (groupid=0, jobs=1): err= 0: pid=81: Tue Oct 18 10:50:15 2022
  read: IOPS=5231, BW=20.4MiB/s (21.4MB/s)(307MiB/15002msec)
    slat (usec): min=2, max=589, avg= 9.94, stdev= 5.75
    clat (usec): min=153, max=65739, avg=753.35, stdev=1012.94
     lat (usec): min=283, max=65749, avg=763.48, stdev=1013.36
    clat percentiles (usec):
     |  1.00th=[  334],  5.00th=[  359], 10.00th=[  383], 20.00th=[  433],
     | 30.00th=[  494], 40.00th=[  523], 50.00th=[  545], 60.00th=[  578],
     | 70.00th=[  627], 80.00th=[ 1074], 90.00th=[ 1237], 95.00th=[ 1369],
     | 99.00th=[ 3064], 99.50th=[ 5276], 99.90th=[14877], 99.95th=[19006],
     | 99.99th=[34341]
   bw (  KiB/s): min= 9208, max=24231, per=99.98%, avg=20921.33, stdev=3941.11, samples=30
   iops        : min= 2302, max= 6057, avg=5230.30, stdev=985.25, samples=30
  lat (usec)   : 250=0.01%, 500=31.87%, 750=41.75%, 1000=3.67%
  lat (msec)   : 2=20.92%, 4=1.08%, 10=0.52%, 20=0.14%, 50=0.04%
  lat (msec)   : 100=0.01%
  cpu          : usr=1.81%, sys=8.05%, ctx=71877, majf=0, minf=15
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=78477,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=20.4MiB/s (21.4MB/s), 20.4MiB/s-20.4MiB/s (21.4MB/s-21.4MB/s), io=307MiB (321MB), run=15002-15002msec

Disk stats (read/write):
  sda: ios=89691/3, merge=0/1, ticks=67024/60, in_queue=67068, util=99.39%


Testing Write Latency...
write_latency: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.13
Starting 1 process

write_latency: (groupid=0, jobs=1): err= 0: pid=97: Tue Oct 18 10:50:33 2022
  write: IOPS=2077, BW=8309KiB/s (8508kB/s)(122MiB/15001msec); 0 zone resets
    slat (usec): min=2, max=1192, avg=11.09, stdev=17.41
    clat (usec): min=70, max=501359, avg=1926.98, stdev=22292.94
     lat (usec): min=357, max=501370, avg=1938.22, stdev=22292.97
    clat percentiles (usec):
     |  1.00th=[   379],  5.00th=[   400], 10.00th=[   424], 20.00th=[   578],
     | 30.00th=[   619], 40.00th=[   668], 50.00th=[   775], 60.00th=[   832],
     | 70.00th=[   914], 80.00th=[  1012], 90.00th=[  1156], 95.00th=[  1893],
     | 99.00th=[  5211], 99.50th=[  7832], 99.90th=[501220], 99.95th=[501220],
     | 99.99th=[501220]
   bw (  KiB/s): min=   80, max=12696, per=99.62%, avg=8277.79, stdev=3387.47, samples=29
   iops        : min=   20, max= 3174, avg=2069.45, stdev=846.87, samples=29
  lat (usec)   : 100=0.01%, 250=0.01%, 500=16.14%, 750=31.30%, 1000=30.92%
  lat (msec)   : 2=17.06%, 4=3.03%, 10=1.20%, 20=0.14%, 50=0.01%
  lat (msec)   : 250=0.01%, 500=0.03%, 750=0.18%
  cpu          : usr=0.99%, sys=2.88%, ctx=24330, majf=0, minf=15
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,31158,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
  WRITE: bw=8309KiB/s (8508kB/s), 8309KiB/s-8309KiB/s (8508kB/s-8508kB/s), io=122MiB (128MB), run=15001-15001msec

Disk stats (read/write):
  sda: ios=0/35335, merge=0/3, ticks=0/67208, in_queue=67884, util=99.41%


Testing Read Sequential Speed...
read_seq: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.13
Starting 1 thread

read_seq: (groupid=0, jobs=1): err= 0: pid=113: Tue Oct 18 10:50:51 2022
  read: IOPS=94, BW=95.3MiB/s (99.9MB/s)(1477MiB/15497msec)
   bw (  KiB/s): min=67584, max=126976, per=99.67%, avg=97273.03, stdev=12945.96, samples=30
   iops        : min=   66, max=  124, avg=94.97, stdev=12.63, samples=30
  cpu          : usr=0.05%, sys=0.52%, ctx=1478, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1462,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=95.3MiB/s (99.9MB/s), 95.3MiB/s-95.3MiB/s (99.9MB/s-99.9MB/s), io=1477MiB (1549MB), run=15497-15497msec

Disk stats (read/write):
  sda: ios=1635/3, merge=0/1, ticks=274836/780, in_queue=276808, util=99.45%


Testing Write Sequential Speed...
write_seq: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
...
fio-3.13
Starting 4 threads
write_seq: Laying out IO file (1 file / 2798MiB)

write_seq: (groupid=0, jobs=1): err= 0: pid=129: Tue Oct 18 10:51:11 2022
  write: IOPS=13, BW=14.8MiB/s (15.5MB/s)(237MiB/15985msec); 0 zone resets
   bw (  KiB/s): min= 2048, max=63488, per=41.99%, avg=28282.31, stdev=15202.07, samples=16
   iops        : min=    2, max=   62, avg=27.50, stdev=14.90, samples=16
  cpu          : usr=0.05%, sys=0.10%, ctx=232, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.6%, 8=0.0%, 16=0.4%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,222,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
write_seq: (groupid=0, jobs=1): err= 0: pid=130: Tue Oct 18 10:51:11 2022
  write: IOPS=16, BW=17.5MiB/s (18.3MB/s)(301MiB/17223msec); 0 zone resets
   bw (  KiB/s): min= 6144, max=63488, per=48.31%, avg=32536.78, stdev=14949.09, samples=18
   iops        : min=    6, max=   62, avg=31.72, stdev=14.60, samples=18
  cpu          : usr=0.07%, sys=0.12%, ctx=273, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.7%, 8=0.0%, 16=0.3%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,286,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
write_seq: (groupid=0, jobs=1): err= 0: pid=131: Tue Oct 18 10:51:11 2022
  write: IOPS=16, BW=17.4MiB/s (18.3MB/s)(301MiB/17260msec); 0 zone resets
   bw (  KiB/s): min= 2048, max=63488, per=43.18%, avg=29081.60, stdev=15146.17, samples=20
   iops        : min=    2, max=   62, avg=28.40, stdev=14.79, samples=20
  cpu          : usr=0.14%, sys=0.05%, ctx=279, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.7%, 8=0.0%, 16=0.3%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,286,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
write_seq: (groupid=0, jobs=1): err= 0: pid=132: Tue Oct 18 10:51:11 2022
  write: IOPS=16, BW=17.3MiB/s (18.2MB/s)(300MiB/17316msec); 0 zone resets
   bw (  KiB/s): min= 2048, max=59392, per=43.66%, avg=29410.53, stdev=13713.83, samples=19
   iops        : min=    2, max=   58, avg=28.58, stdev=13.33, samples=19
  cpu          : usr=0.07%, sys=0.12%, ctx=269, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.7%, 8=0.0%, 16=0.3%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,285,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=65.8MiB/s (68.0MB/s), 14.8MiB/s-17.5MiB/s (15.5MB/s-18.3MB/s), io=1139MiB (1194MB), run=15985-17316msec

Disk stats (read/write):
  sda: ios=0/1453, merge=0/90, ticks=0/1160892, in_queue=1164800, util=99.52%


Testing Read/Write Mixed...
rw_mix: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.13
Starting 1 process

rw_mix: (groupid=0, jobs=1): err= 0: pid=148: Tue Oct 18 10:51:29 2022
  read: IOPS=4549, BW=17.8MiB/s (18.6MB/s)(276MiB/15516msec)
   bw (  KiB/s): min=  304, max=28808, per=100.00%, avg=18217.06, stdev=7313.16, samples=31
   iops        : min=   76, max= 7202, avg=4554.32, stdev=1828.28, samples=31
  write: IOPS=1507, BW=6040KiB/s (6185kB/s)(91.5MiB/15516msec); 0 zone resets
   bw (  KiB/s): min=  152, max= 9304, per=99.93%, avg=6035.55, stdev=2433.19, samples=31
   iops        : min=   38, max= 2326, avg=1508.87, stdev=608.33, samples=31
  cpu          : usr=1.78%, sys=5.44%, ctx=59089, majf=0, minf=15
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=70595,23398,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=17.8MiB/s (18.6MB/s), 17.8MiB/s-17.8MiB/s (18.6MB/s-18.6MB/s), io=276MiB (289MB), run=15516-15516msec
  WRITE: bw=6040KiB/s (6185kB/s), 6040KiB/s-6040KiB/s (6185kB/s-6185kB/s), io=91.5MiB (95.0MB), run=15516-15516msec

Disk stats (read/write):
  sda: ios=42139/25208, merge=5/237, ticks=526796/561544, in_queue=1089088, util=99.48%


All tests complete.

==================
= Dbench Summary =
==================
Random Read/Write IOPS: 8569/1916. BW: 337MiB/s / 74.7MiB/s
Average Latency (usec) Read/Write: 763.48/1938.22
Sequential Read/Write: 95.3MiB/s / 65.8MiB/s
Mixed Random Read/Write IOPS: 4549/1507
IOPSBandwidth
随机读8859440MiB/s——
随机写206473.0MiB/s——
顺序读—22.9MiB/s——
顺序写—65.9MiB/s——
混合读写50%4423/1451———
混合读写70%————
更新于 2022-10-18 
CC BY-NC 4.0
阅读原始文档
LINE Myspace
Openebs
返回 | 主页
0%