zoukankan      html  css  js  c++  java

  • Kafka吞吐量测试案例

     　　　　　　　　　　　　　　　　　　　　　　　　Kafka吞吐量测试案例

    　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

    版权声明：原创作品，谢绝转载！否则将追究法律责任。

    　　领英公司参考连接：https://www.slideshare.net/JiangjieQin/producer-performance-tuning-for-apache-kafka-63147600?qid=84026ff8-243f-49a7-a4d0-69976cf317b7&v=&b=&from_search=9(温馨提示，需要VPN哟～)。

    一.kafka-producer-perf-test.sh 使用案例

    1>.测试kafka写入的吞吐量

    　　我用的kafka版本是0.10.2.1，领英用的貌似是0.10.0，我使用我的版本按照它的方式敲击，发现不太好事，这就有点小尴尬了！于是它表红线的参数我就没有使用。

    [root@node105 ~]# nohup kafka-producer-perf-test.sh --num-records 100000000 --record-size 1000 --topic yinzhengjie-kafka --throughput 1000000 --producer-props bootstrap.servers=1
0.1.2.101:9092,10.1.2.102:9092,10.1.2.103:9092,10.1.2.104:9092,10.1.2.105:9092 > yinzhengjie-kafka.log &
[root@node105 ~]# 

相关参数说明：
—topic             
    指定Kafka集群的topic名称，本例为yinzhengjie-kafka

    
    --num-records       
    
    　　总共需要发送的消息数，本例为100000000
    


    --record-size      
     每个记录的字节数，本例为1000



    --throughput       
     每秒钟发送的记录数，本例为20000
    


    --producer-props    bootstrap.servers=10.1.2.101:9092,10.1.2.102:9092,10.1.2.103:9092,10.1.2.104:9092,10.1.2.105:9092        
    很显然是kafka集群的broker地址

    2>.计算生产者的吞吐量

    [root@node105 ~]# cat yinzhengjie-kafka.log 
485204 records sent, 96673.4 records/sec (92.19 MB/sec), 284.4 ms avg latency, 648.0 max latency.
591703 records sent, 118340.6 records/sec (112.86 MB/sec), 260.7 ms avg latency, 795.0 max latency.
568061 records sent, 113612.2 records/sec (108.35 MB/sec), 268.2 ms avg latency, 870.0 max latency.
576896 records sent, 115034.1 records/sec (109.71 MB/sec), 268.7 ms avg latency, 732.0 max latency.
548140 records sent, 106912.4 records/sec (101.96 MB/sec), 257.9 ms avg latency, 913.0 max latency.
568845 records sent, 112575.7 records/sec (107.36 MB/sec), 294.4 ms avg latency, 1283.0 max latency.
557990 records sent, 111598.0 records/sec (106.43 MB/sec), 279.4 ms avg latency, 828.0 max latency.
464493 records sent, 92898.6 records/sec (88.60 MB/sec), 329.3 ms avg latency, 1213.0 max latency.
488281 records sent, 97656.2 records/sec (93.13 MB/sec), 312.9 ms avg latency, 1020.0 max latency.
517560 records sent, 103512.0 records/sec (98.72 MB/sec), 294.8 ms avg latency, 1062.0 max latency.
209745 records sent, 41949.0 records/sec (40.01 MB/sec), 740.2 ms avg latency, 3722.0 max latency.
394980 records sent, 78980.2 records/sec (75.32 MB/sec), 388.7 ms avg latency, 2079.0 max latency.
445178 records sent, 89035.6 records/sec (84.91 MB/sec), 342.4 ms avg latency, 1274.0 max latency.
693946 records sent, 138789.2 records/sec (132.36 MB/sec), 223.2 ms avg latency, 469.0 max latency.
367182 records sent, 62616.3 records/sec (59.72 MB/sec), 247.3 ms avg latency, 3303.0 max latency.
631412 records sent, 126282.4 records/sec (120.43 MB/sec), 381.9 ms avg latency, 3304.0 max latency.
393343 records sent, 78668.6 records/sec (75.02 MB/sec), 386.0 ms avg latency, 1857.0 max latency.
441035 records sent, 88207.0 records/sec (84.12 MB/sec), 347.2 ms avg latency, 1018.0 max latency.
521952 records sent, 104223.6 records/sec (99.40 MB/sec), 296.4 ms avg latency, 698.0 max latency.
641069 records sent, 127779.4 records/sec (121.86 MB/sec), 238.2 ms avg latency, 560.0 max latency.
514232 records sent, 101747.5 records/sec (97.03 MB/sec), 296.9 ms avg latency, 855.0 max latency.
582718 records sent, 116543.6 records/sec (111.14 MB/sec), 269.8 ms avg latency, 633.0 max latency.
15689 records sent, 2589.8 records/sec (2.47 MB/sec), 262.5 ms avg latency, 6291.0 max latency.
300150 records sent, 34393.3 records/sec (32.80 MB/sec), 954.0 ms avg latency, 6731.0 max latency.
528932 records sent, 105786.4 records/sec (100.89 MB/sec), 595.4 ms avg latency, 5857.0 max latency.
460320 records sent, 91770.3 records/sec (87.52 MB/sec), 335.8 ms avg latency, 1688.0 max latency.
613573 records sent, 119815.1 records/sec (114.26 MB/sec), 245.9 ms avg latency, 931.0 max latency.
316608 records sent, 63321.6 records/sec (60.39 MB/sec), 502.0 ms avg latency, 2900.0 max latency.
581817 records sent, 116084.8 records/sec (110.71 MB/sec), 262.5 ms avg latency, 540.0 max latency.
636679 records sent, 125801.0 records/sec (119.97 MB/sec), 244.7 ms avg latency, 517.0 max latency.
501821 records sent, 97554.6 records/sec (93.04 MB/sec), 298.4 ms avg latency, 1026.0 max latency.
92700 records sent, 18197.9 records/sec (17.35 MB/sec), 1709.4 ms avg latency, 4843.0 max latency.
430113 records sent, 86022.6 records/sec (82.04 MB/sec), 370.9 ms avg latency, 1119.0 max latency.
318603 records sent, 63720.6 records/sec (60.77 MB/sec), 478.7 ms avg latency, 2001.0 max latency.
389034 records sent, 76732.5 records/sec (73.18 MB/sec), 379.7 ms avg latency, 1345.0 max latency.
413297 records sent, 82659.4 records/sec (78.83 MB/sec), 391.5 ms avg latency, 1016.0 max latency.
258853 records sent, 51729.2 records/sec (49.33 MB/sec), 528.6 ms avg latency, 2228.0 max latency.
437625 records sent, 87490.0 records/sec (83.44 MB/sec), 377.5 ms avg latency, 1178.0 max latency.
352050 records sent, 70227.4 records/sec (66.97 MB/sec), 424.4 ms avg latency, 1456.0 max latency.
402187 records sent, 80437.4 records/sec (76.71 MB/sec), 394.4 ms avg latency, 1452.0 max latency.
406540 records sent, 81178.1 records/sec (77.42 MB/sec), 383.9 ms avg latency, 1344.0 max latency.
352573 records sent, 66965.4 records/sec (63.86 MB/sec), 426.7 ms avg latency, 2243.0 max latency.
405849 records sent, 81169.8 records/sec (77.41 MB/sec), 406.3 ms avg latency, 1204.0 max latency.
394551 records sent, 78674.2 records/sec (75.03 MB/sec), 369.5 ms avg latency, 980.0 max latency.
371400 records sent, 74235.5 records/sec (70.80 MB/sec), 376.5 ms avg latency, 1539.0 max latency.
224325 records sent, 44141.1 records/sec (42.10 MB/sec), 730.9 ms avg latency, 2348.0 max latency.
294300 records sent, 56142.7 records/sec (53.54 MB/sec), 537.1 ms avg latency, 1441.0 max latency.
350175 records sent, 69909.2 records/sec (66.67 MB/sec), 480.5 ms avg latency, 1660.0 max latency.
350175 records sent, 66674.6 records/sec (63.59 MB/sec), 437.3 ms avg latency, 1390.0 max latency.
364683 records sent, 72936.6 records/sec (69.56 MB/sec), 440.6 ms avg latency, 1651.0 max latency.
385692 records sent, 76708.8 records/sec (73.16 MB/sec), 326.2 ms avg latency, 1466.0 max latency.
388067 records sent, 76875.4 records/sec (73.31 MB/sec), 468.7 ms avg latency, 1773.0 max latency.
312283 records sent, 62170.6 records/sec (59.29 MB/sec), 492.0 ms avg latency, 1382.0 max latency.
350175 records sent, 65983.6 records/sec (62.93 MB/sec), 432.4 ms avg latency, 1238.0 max latency.
400200 records sent, 79912.1 records/sec (76.21 MB/sec), 420.5 ms avg latency, 1244.0 max latency.
335917 records sent, 66623.8 records/sec (63.54 MB/sec), 445.0 ms avg latency, 1054.0 max latency.
364433 records sent, 70929.0 records/sec (67.64 MB/sec), 413.4 ms avg latency, 1165.0 max latency.
305884 records sent, 61176.8 records/sec (58.34 MB/sec), 536.8 ms avg latency, 1517.0 max latency.
430627 records sent, 85953.5 records/sec (81.97 MB/sec), 354.6 ms avg latency, 1919.0 max latency.
414064 records sent, 80229.4 records/sec (76.51 MB/sec), 363.1 ms avg latency, 1194.0 max latency.
304470 records sent, 60894.0 records/sec (58.07 MB/sec), 509.9 ms avg latency, 1365.0 max latency.
385418 records sent, 77083.6 records/sec (73.51 MB/sec), 417.5 ms avg latency, 1468.0 max latency.
402698 records sent, 80059.2 records/sec (76.35 MB/sec), 343.1 ms avg latency, 1147.0 max latency.
419702 records sent, 83940.4 records/sec (80.05 MB/sec), 402.7 ms avg latency, 1136.0 max latency.
272698 records sent, 54539.6 records/sec (52.01 MB/sec), 554.7 ms avg latency, 2612.0 max latency.
327574 records sent, 65514.8 records/sec (62.48 MB/sec), 452.0 ms avg latency, 1112.0 max latency.
444357 records sent, 88871.4 records/sec (84.75 MB/sec), 304.7 ms avg latency, 1523.0 max latency.
325969 records sent, 65193.8 records/sec (62.17 MB/sec), 546.3 ms avg latency, 2538.0 max latency.
391147 records sent, 78229.4 records/sec (74.61 MB/sec), 380.9 ms avg latency, 1704.0 max latency.
418355 records sent, 83671.0 records/sec (79.79 MB/sec), 380.5 ms avg latency, 1402.0 max latency.
318282 records sent, 63656.4 records/sec (60.71 MB/sec), 465.3 ms avg latency, 2996.0 max latency.
431130 records sent, 85916.7 records/sec (81.94 MB/sec), 366.3 ms avg latency, 1606.0 max latency.
398883 records sent, 79776.6 records/sec (76.08 MB/sec), 380.9 ms avg latency, 1387.0 max latency.
551592 records sent, 109791.4 records/sec (104.71 MB/sec), 270.6 ms avg latency, 1512.0 max latency.
483225 records sent, 96645.0 records/sec (92.17 MB/sec), 317.9 ms avg latency, 988.0 max latency.
317175 records sent, 58519.4 records/sec (55.81 MB/sec), 473.6 ms avg latency, 2136.0 max latency.
339750 records sent, 67665.8 records/sec (64.53 MB/sec), 491.5 ms avg latency, 2083.0 max latency.
421923 records sent, 84384.6 records/sec (80.48 MB/sec), 384.9 ms avg latency, 1260.0 max latency.
365130 records sent, 73026.0 records/sec (69.64 MB/sec), 410.0 ms avg latency, 1084.0 max latency.
346304 records sent, 69260.8 records/sec (66.05 MB/sec), 456.2 ms avg latency, 1526.0 max latency.
347720 records sent, 69544.0 records/sec (66.32 MB/sec), 420.3 ms avg latency, 1192.0 max latency.
374846 records sent, 74969.2 records/sec (71.50 MB/sec), 421.8 ms avg latency, 1478.0 max latency.
332024 records sent, 66404.8 records/sec (63.33 MB/sec), 451.3 ms avg latency, 1840.0 max latency.
413353 records sent, 82079.6 records/sec (78.28 MB/sec), 355.0 ms avg latency, 1328.0 max latency.
456013 records sent, 91202.6 records/sec (86.98 MB/sec), 362.7 ms avg latency, 1106.0 max latency.
443987 records sent, 85744.9 records/sec (81.77 MB/sec), 302.0 ms avg latency, 1187.0 max latency.
401025 records sent, 78694.1 records/sec (75.05 MB/sec), 431.8 ms avg latency, 1545.0 max latency.
339375 records sent, 67820.7 records/sec (64.68 MB/sec), 443.9 ms avg latency, 1530.0 max latency.
460950 records sent, 84655.6 records/sec (80.73 MB/sec), 304.3 ms avg latency, 1762.0 max latency.
339450 records sent, 66952.7 records/sec (63.85 MB/sec), 546.6 ms avg latency, 2097.0 max latency.
394914 records sent, 78982.8 records/sec (75.32 MB/sec), 402.1 ms avg latency, 1448.0 max latency.
291396 records sent, 57679.3 records/sec (55.01 MB/sec), 517.6 ms avg latency, 2158.0 max latency.
372347 records sent, 74469.4 records/sec (71.02 MB/sec), 397.6 ms avg latency, 1819.0 max latency.
408480 records sent, 81696.0 records/sec (77.91 MB/sec), 399.9 ms avg latency, 1226.0 max latency.
367899 records sent, 73579.8 records/sec (70.17 MB/sec), 423.3 ms avg latency, 1575.0 max latency.
270450 records sent, 54090.0 records/sec (51.58 MB/sec), 513.7 ms avg latency, 1724.0 max latency.
395764 records sent, 78276.1 records/sec (74.65 MB/sec), 387.1 ms avg latency, 1762.0 max latency.
437056 records sent, 87411.2 records/sec (83.36 MB/sec), 380.6 ms avg latency, 1763.0 max latency.
363344 records sent, 69182.0 records/sec (65.98 MB/sec), 421.1 ms avg latency, 1361.0 max latency.
450225 records sent, 87202.2 records/sec (83.16 MB/sec), 359.5 ms avg latency, 1487.0 max latency.
400200 records sent, 75623.6 records/sec (72.12 MB/sec), 265.6 ms avg latency, 2355.0 max latency.
335026 records sent, 67005.2 records/sec (63.90 MB/sec), 625.9 ms avg latency, 2568.0 max latency.
376506 records sent, 75301.2 records/sec (71.81 MB/sec), 389.1 ms avg latency, 1071.0 max latency.
281738 records sent, 56347.6 records/sec (53.74 MB/sec), 541.7 ms avg latency, 2047.0 max latency.
346980 records sent, 66255.5 records/sec (63.19 MB/sec), 461.9 ms avg latency, 1747.0 max latency.
351900 records sent, 69244.4 records/sec (66.04 MB/sec), 408.8 ms avg latency, 1318.0 max latency.
265397 records sent, 53079.4 records/sec (50.62 MB/sec), 630.5 ms avg latency, 1979.0 max latency.
424290 records sent, 84722.4 records/sec (80.80 MB/sec), 387.0 ms avg latency, 1130.0 max latency.
302624 records sent, 60500.6 records/sec (57.70 MB/sec), 504.8 ms avg latency, 1415.0 max latency.
356639 records sent, 71213.9 records/sec (67.91 MB/sec), 397.4 ms avg latency, 1366.0 max latency.
400200 records sent, 76887.6 records/sec (73.33 MB/sec), 405.1 ms avg latency, 1269.0 max latency.
435842 records sent, 87168.4 records/sec (83.13 MB/sec), 378.4 ms avg latency, 1388.0 max latency.
364558 records sent, 70801.7 records/sec (67.52 MB/sec), 411.4 ms avg latency, 1323.0 max latency.
449387 records sent, 89877.4 records/sec (85.71 MB/sec), 355.8 ms avg latency, 1039.0 max latency.
403288 records sent, 77376.8 records/sec (73.79 MB/sec), 340.2 ms avg latency, 1702.0 max latency.
417825 records sent, 80242.9 records/sec (76.53 MB/sec), 421.3 ms avg latency, 1226.0 max latency.
369900 records sent, 72686.2 records/sec (69.32 MB/sec), 417.1 ms avg latency, 1447.0 max latency.
437842 records sent, 87568.4 records/sec (83.51 MB/sec), 359.2 ms avg latency, 2027.0 max latency.
450833 records sent, 89986.6 records/sec (85.82 MB/sec), 343.0 ms avg latency, 1238.0 max latency.
468195 records sent, 93639.0 records/sec (89.30 MB/sec), 318.6 ms avg latency, 887.0 max latency.
381300 records sent, 76260.0 records/sec (72.73 MB/sec), 416.3 ms avg latency, 1537.0 max latency.
386025 records sent, 77205.0 records/sec (73.63 MB/sec), 398.5 ms avg latency, 1199.0 max latency.
355139 records sent, 71027.8 records/sec (67.74 MB/sec), 431.3 ms avg latency, 1298.0 max latency.
374749 records sent, 74770.4 records/sec (71.31 MB/sec), 353.8 ms avg latency, 1290.0 max latency.
397407 records sent, 79481.4 records/sec (75.80 MB/sec), 438.1 ms avg latency, 1648.0 max latency.
360720 records sent, 72144.0 records/sec (68.80 MB/sec), 410.4 ms avg latency, 1585.0 max latency.
364249 records sent, 72849.8 records/sec (69.47 MB/sec), 413.8 ms avg latency, 1505.0 max latency.
401363 records sent, 80272.6 records/sec (76.55 MB/sec), 388.3 ms avg latency, 1054.0 max latency.
314103 records sent, 59220.0 records/sec (56.48 MB/sec), 495.4 ms avg latency, 1338.0 max latency.
298134 records sent, 59626.8 records/sec (56.86 MB/sec), 551.7 ms avg latency, 1610.0 max latency.
295673 records sent, 59134.6 records/sec (56.40 MB/sec), 522.3 ms avg latency, 1689.0 max latency.
386488 records sent, 77297.6 records/sec (73.72 MB/sec), 400.0 ms avg latency, 1311.0 max latency.
422477 records sent, 84495.4 records/sec (80.58 MB/sec), 359.9 ms avg latency, 1357.0 max latency.
364903 records sent, 72980.6 records/sec (69.60 MB/sec), 405.4 ms avg latency, 1113.0 max latency.
333375 records sent, 63415.4 records/sec (60.48 MB/sec), 451.5 ms avg latency, 1377.0 max latency.
407003 records sent, 81400.6 records/sec (77.63 MB/sec), 390.1 ms avg latency, 1321.0 max latency.
393397 records sent, 77076.2 records/sec (73.51 MB/sec), 408.1 ms avg latency, 1285.0 max latency.
388801 records sent, 77760.2 records/sec (74.16 MB/sec), 396.4 ms avg latency, 1162.0 max latency.
383069 records sent, 76613.8 records/sec (73.06 MB/sec), 417.4 ms avg latency, 1343.0 max latency.
378705 records sent, 73735.4 records/sec (70.32 MB/sec), 391.5 ms avg latency, 2222.0 max latency.
385890 records sent, 77178.0 records/sec (73.60 MB/sec), 413.3 ms avg latency, 1166.0 max latency.
444551 records sent, 88910.2 records/sec (84.79 MB/sec), 353.1 ms avg latency, 1147.0 max latency.
347314 records sent, 69296.5 records/sec (66.09 MB/sec), 438.3 ms avg latency, 1684.0 max latency.
316545 records sent, 62508.9 records/sec (59.61 MB/sec), 446.3 ms avg latency, 1352.0 max latency.
433528 records sent, 86085.8 records/sec (82.10 MB/sec), 387.1 ms avg latency, 1173.0 max latency.
374452 records sent, 74890.4 records/sec (71.42 MB/sec), 385.6 ms avg latency, 1550.0 max latency.
338620 records sent, 67133.2 records/sec (64.02 MB/sec), 453.3 ms avg latency, 1589.0 max latency.
374775 records sent, 73993.1 records/sec (70.57 MB/sec), 412.4 ms avg latency, 1358.0 max latency.
357450 records sent, 70586.5 records/sec (67.32 MB/sec), 456.5 ms avg latency, 1546.0 max latency.
275232 records sent, 55046.4 records/sec (52.50 MB/sec), 512.0 ms avg latency, 1688.0 max latency.
442490 records sent, 88498.0 records/sec (84.40 MB/sec), 370.3 ms avg latency, 1130.0 max latency.
313110 records sent, 62622.0 records/sec (59.72 MB/sec), 498.3 ms avg latency, 1636.0 max latency.
400308 records sent, 80061.6 records/sec (76.35 MB/sec), 395.6 ms avg latency, 1373.0 max latency.
337885 records sent, 66512.8 records/sec (63.43 MB/sec), 395.0 ms avg latency, 1690.0 max latency.
446967 records sent, 89393.4 records/sec (85.25 MB/sec), 386.0 ms avg latency, 1827.0 max latency.
439940 records sent, 87988.0 records/sec (83.91 MB/sec), 355.3 ms avg latency, 1251.0 max latency.
262322 records sent, 52464.4 records/sec (50.03 MB/sec), 516.0 ms avg latency, 1680.0 max latency.
314593 records sent, 62918.6 records/sec (60.00 MB/sec), 544.6 ms avg latency, 1431.0 max latency.
312654 records sent, 61985.3 records/sec (59.11 MB/sec), 481.0 ms avg latency, 1430.0 max latency.
399757 records sent, 79951.4 records/sec (76.25 MB/sec), 386.0 ms avg latency, 1160.0 max latency.
400965 records sent, 80193.0 records/sec (76.48 MB/sec), 381.1 ms avg latency, 1262.0 max latency.
363852 records sent, 69998.5 records/sec (66.76 MB/sec), 398.8 ms avg latency, 1203.0 max latency.
536223 records sent, 107223.2 records/sec (102.26 MB/sec), 317.9 ms avg latency, 1063.0 max latency.
314202 records sent, 60962.7 records/sec (58.14 MB/sec), 425.6 ms avg latency, 2061.0 max latency.
240900 records sent, 48151.1 records/sec (45.92 MB/sec), 674.7 ms avg latency, 1933.0 max latency.
209325 records sent, 40023.9 records/sec (38.17 MB/sec), 752.9 ms avg latency, 1802.0 max latency.
256875 records sent, 51344.2 records/sec (48.97 MB/sec), 622.6 ms avg latency, 1677.0 max latency.
193350 records sent, 37986.2 records/sec (36.23 MB/sec), 762.4 ms avg latency, 1840.0 max latency.
483000 records sent, 96311.1 records/sec (91.85 MB/sec), 308.9 ms avg latency, 1841.0 max latency.
382314 records sent, 76401.7 records/sec (72.86 MB/sec), 443.2 ms avg latency, 1543.0 max latency.
435336 records sent, 80602.9 records/sec (76.87 MB/sec), 356.3 ms avg latency, 1044.0 max latency.
373638 records sent, 74727.6 records/sec (71.27 MB/sec), 430.1 ms avg latency, 1650.0 max latency.
336477 records sent, 67295.4 records/sec (64.18 MB/sec), 438.0 ms avg latency, 1648.0 max latency.
350867 records sent, 70173.4 records/sec (66.92 MB/sec), 458.6 ms avg latency, 1661.0 max latency.
499218 records sent, 99843.6 records/sec (95.22 MB/sec), 312.3 ms avg latency, 1054.0 max latency.
362143 records sent, 72428.6 records/sec (69.07 MB/sec), 433.5 ms avg latency, 1483.0 max latency.
308457 records sent, 60269.1 records/sec (57.48 MB/sec), 491.1 ms avg latency, 1619.0 max latency.
375692 records sent, 75138.4 records/sec (71.66 MB/sec), 422.2 ms avg latency, 1286.0 max latency.
387466 records sent, 76908.7 records/sec (73.35 MB/sec), 374.8 ms avg latency, 1611.0 max latency.
347292 records sent, 62428.9 records/sec (59.54 MB/sec), 464.3 ms avg latency, 1787.0 max latency.
354210 records sent, 70842.0 records/sec (67.56 MB/sec), 484.1 ms avg latency, 2107.0 max latency.
496215 records sent, 87147.0 records/sec (83.11 MB/sec), 306.1 ms avg latency, 1225.0 max latency.
400200 records sent, 75395.6 records/sec (71.90 MB/sec), 433.9 ms avg latency, 1668.0 max latency.
361743 records sent, 72348.6 records/sec (69.00 MB/sec), 453.9 ms avg latency, 1441.0 max latency.
388632 records sent, 68590.2 records/sec (65.41 MB/sec), 397.4 ms avg latency, 1394.0 max latency.
282975 records sent, 56012.5 records/sec (53.42 MB/sec), 593.2 ms avg latency, 1667.0 max latency.
340549 records sent, 68109.8 records/sec (64.95 MB/sec), 480.5 ms avg latency, 1633.0 max latency.
320276 records sent, 60750.4 records/sec (57.94 MB/sec), 451.4 ms avg latency, 1519.0 max latency.
382376 records sent, 75778.0 records/sec (72.27 MB/sec), 423.8 ms avg latency, 1378.0 max latency.
375394 records sent, 75078.8 records/sec (71.60 MB/sec), 403.5 ms avg latency, 1199.0 max latency.
442290 records sent, 88334.3 records/sec (84.24 MB/sec), 367.1 ms avg latency, 935.0 max latency.
435002 records sent, 87000.4 records/sec (82.97 MB/sec), 360.3 ms avg latency, 1660.0 max latency.
348105 records sent, 69621.0 records/sec (66.40 MB/sec), 429.2 ms avg latency, 1201.0 max latency.
378901 records sent, 75780.2 records/sec (72.27 MB/sec), 403.9 ms avg latency, 1311.0 max latency.
242099 records sent, 48419.8 records/sec (46.18 MB/sec), 587.5 ms avg latency, 2074.0 max latency.
380623 records sent, 75475.5 records/sec (71.98 MB/sec), 441.9 ms avg latency, 1329.0 max latency.
299522 records sent, 59701.4 records/sec (56.94 MB/sec), 494.5 ms avg latency, 1573.0 max latency.
374171 records sent, 74834.2 records/sec (71.37 MB/sec), 375.8 ms avg latency, 1409.0 max latency.
439817 records sent, 84547.7 records/sec (80.63 MB/sec), 371.5 ms avg latency, 1183.0 max latency.
364202 records sent, 72840.4 records/sec (69.47 MB/sec), 456.6 ms avg latency, 1347.0 max latency.
408650 records sent, 81730.0 records/sec (77.94 MB/sec), 348.7 ms avg latency, 1212.0 max latency.
386220 records sent, 77244.0 records/sec (73.67 MB/sec), 430.5 ms avg latency, 1499.0 max latency.
590650 records sent, 118130.0 records/sec (112.66 MB/sec), 257.1 ms avg latency, 1053.0 max latency.
329128 records sent, 63208.8 records/sec (60.28 MB/sec), 349.1 ms avg latency, 1879.0 max latency.
408542 records sent, 81708.4 records/sec (77.92 MB/sec), 433.5 ms avg latency, 2531.0 max latency.
337366 records sent, 67405.8 records/sec (64.28 MB/sec), 505.4 ms avg latency, 1556.0 max latency.
416442 records sent, 73265.7 records/sec (69.87 MB/sec), 378.6 ms avg latency, 1670.0 max latency.
360097 records sent, 72019.4 records/sec (68.68 MB/sec), 428.8 ms avg latency, 1715.0 max latency.
403744 records sent, 80748.8 records/sec (77.01 MB/sec), 388.2 ms avg latency, 1085.0 max latency.
394909 records sent, 76266.7 records/sec (72.73 MB/sec), 387.5 ms avg latency, 1689.0 max latency.
392025 records sent, 78154.9 records/sec (74.53 MB/sec), 448.3 ms avg latency, 1543.0 max latency.
386595 records sent, 77319.0 records/sec (73.74 MB/sec), 400.5 ms avg latency, 1478.0 max latency.
424468 records sent, 84893.6 records/sec (80.96 MB/sec), 360.7 ms avg latency, 1372.0 max latency.
329087 records sent, 63604.0 records/sec (60.66 MB/sec), 431.9 ms avg latency, 1677.0 max latency.
334860 records sent, 66125.6 records/sec (63.06 MB/sec), 467.7 ms avg latency, 1670.0 max latency.
215415 records sent, 41101.9 records/sec (39.20 MB/sec), 710.9 ms avg latency, 2273.0 max latency.
243600 records sent, 48237.6 records/sec (46.00 MB/sec), 688.9 ms avg latency, 1916.0 max latency.
443820 records sent, 88764.0 records/sec (84.65 MB/sec), 364.9 ms avg latency, 1812.0 max latency.
711347 records sent, 142269.4 records/sec (135.68 MB/sec), 216.4 ms avg latency, 453.0 max latency.
577303 records sent, 115460.6 records/sec (110.11 MB/sec), 263.4 ms avg latency, 1072.0 max latency.
296205 records sent, 55510.7 records/sec (52.94 MB/sec), 212.9 ms avg latency, 3721.0 max latency.
329025 records sent, 60705.7 records/sec (57.89 MB/sec), 761.2 ms avg latency, 3900.0 max latency.
287524 records sent, 57390.0 records/sec (54.73 MB/sec), 592.4 ms avg latency, 2092.0 max latency.
504626 records sent, 100423.1 records/sec (95.77 MB/sec), 305.8 ms avg latency, 1828.0 max latency.
452748 records sent, 90549.6 records/sec (86.35 MB/sec), 340.5 ms avg latency, 1327.0 max latency.
305877 records sent, 57789.0 records/sec (55.11 MB/sec), 258.8 ms avg latency, 3117.0 max latency.
423903 records sent, 84780.6 records/sec (80.85 MB/sec), 556.3 ms avg latency, 3368.0 max latency.
451797 records sent, 84988.1 records/sec (81.05 MB/sec), 315.4 ms avg latency, 1309.0 max latency.
475265 records sent, 95053.0 records/sec (90.65 MB/sec), 363.2 ms avg latency, 1313.0 max latency.
577183 records sent, 114770.9 records/sec (109.45 MB/sec), 264.1 ms avg latency, 861.0 max latency.
673152 records sent, 112661.4 records/sec (107.44 MB/sec), 230.4 ms avg latency, 1321.0 max latency.
689925 records sent, 137957.4 records/sec (131.57 MB/sec), 265.3 ms avg latency, 1400.0 max latency.
400200 records sent, 79928.1 records/sec (76.23 MB/sec), 384.9 ms avg latency, 2438.0 max latency.
369566 records sent, 73913.2 records/sec (70.49 MB/sec), 410.9 ms avg latency, 2295.0 max latency.
466609 records sent, 86908.0 records/sec (82.88 MB/sec), 318.5 ms avg latency, 1511.0 max latency.
642825 records sent, 117196.9 records/sec (111.77 MB/sec), 253.9 ms avg latency, 1139.0 max latency.
365312 records sent, 73062.4 records/sec (69.68 MB/sec), 438.2 ms avg latency, 1618.0 max latency.
546463 records sent, 108792.2 records/sec (103.75 MB/sec), 309.4 ms avg latency, 975.0 max latency.
449057 records sent, 89811.4 records/sec (85.65 MB/sec), 338.8 ms avg latency, 1284.0 max latency.
461461 records sent, 92292.2 records/sec (88.02 MB/sec), 336.2 ms avg latency, 1115.0 max latency.
425741 records sent, 84438.9 records/sec (80.53 MB/sec), 360.2 ms avg latency, 1078.0 max latency.
527314 records sent, 105462.8 records/sec (100.58 MB/sec), 288.8 ms avg latency, 1230.0 max latency.
516075 records sent, 103215.0 records/sec (98.43 MB/sec), 300.2 ms avg latency, 1021.0 max latency.
513447 records sent, 102607.3 records/sec (97.85 MB/sec), 299.7 ms avg latency, 1108.0 max latency.
392184 records sent, 78436.8 records/sec (74.80 MB/sec), 377.8 ms avg latency, 1352.0 max latency.
516621 records sent, 97715.3 records/sec (93.19 MB/sec), 261.6 ms avg latency, 1416.0 max latency.
512059 records sent, 102411.8 records/sec (97.67 MB/sec), 325.1 ms avg latency, 1509.0 max latency.
377966 records sent, 73980.4 records/sec (70.55 MB/sec), 422.2 ms avg latency, 1893.0 max latency.
503959 records sent, 100791.8 records/sec (96.12 MB/sec), 332.4 ms avg latency, 1506.0 max latency.
474116 records sent, 94407.8 records/sec (90.03 MB/sec), 321.2 ms avg latency, 1079.0 max latency.
100000000 records sent, 78434.325371 records/sec (74.80 MB/sec), 390.67 ms avg latency, 6731.00 ms max latency, 2 ms 50th, 87 ms 95th, 158 ms 99th, 275 ms 99.9th.
[root@node105 ~]#
    [root@node105 ~]# cat yinzhengjie-kafka.log　　　　　　　　　　　　　　#查看Kafka写入日志 
    100000000 records sent, 78434.325371 records/sec (74.80 MB/sec), 390.67 ms avg latency, 6731.00 ms max latency, 2 ms 50th, 87 ms 95th, 158 ms 99th, 275 ms 99.9th.

当你执行完kafka-producer-perf-test.sh 脚本时，会输出类似上面的一行，从上面一行我们可以得出一些消息
    1>.每秒平均向kafka写入了74.80 MB的数据(大概是：78433条消息，计算方式：74.8*1024*1024/1000);
    2>.每次写入的平均延迟为390.67毫秒；
    3>.最大延迟为6731.00毫秒；
    4>.87毫秒内占95%；
    5>.158毫秒内占99%；
    6>.275毫秒内占99.9%；

    二.kafka-consumer-perf-test.sh 使用案例

    1>. 测试Kafka消费的吞吐量

    [root@node105 ~]# nohup kafka-consumer-perf-test.sh --broker-list 10.1.2.102:9092  --topic yinzhengjie-kafka --messages 1000000 --fetch-size 1048576 --threads 10  > yinzhengjie-kafka-consumer.log &
[root@node105 ~]# 


相关参数说明：
--topic 
    指定topic的名称，本例为 yinzhengjie-kafka

--fetch-size 
    
    　　指定每次fetch的数据的大小，本例为1048576，也就是1M

    --messages 　　　　
    　　总共要消费的消息个数，本例为1000000，100w

    --topic
    　　指定消费的topic

    --threads
    　　指定消费的线程数为10
    　　　　

    2>.计算消费者的吞吐量

    [root@node105 ~]# cat yinzhengjie-kafka-consumer.log 
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec
2018-11-13 19:41:31:507, 2018-11-13 19:41:34:169, 953.9165, 358.3458, 1000411, 375811.7956
[root@node105 ~]# 


当你执行完 kafka-consumer-perf-test.sh 脚本时，会输出类似于上面的一行，从上面一行我们可以得出一些消息
    1>.共消费了953.9165M的数据；
    2>.消费速度为358.3458M/s；
    3>.总共消费了1000411 条消息；
    4>.每秒消费375811.7956条消息；

    三.编写脚本案例

    1>.执行脚本

    [root@node105 ~]# cat yinzhengjie-kafka.sh 
nohup kafka-consumer-perf-test.sh --broker-list 10.1.3.116:9092,10.1.3.117:9092,10.1.3.118:9092,10.1.3.119:9092,10.1.3.120:9092 --topic yinzhengjie-kafka --messages 100000000 --fetch-size 1048576 --threads 10  > /root/yinzhengjie-kafka-consumer.log &


nohup kafka-producer-perf-test.sh --num-records 100000000 --record-size 1000 --topic yinzhengjie-kafka --throughput 1000000 --producer-props bootstrap.servers=10.1.3.116:9092,10.1.3.117:9092,10.1.3.118:9092,10.1.3.119:9092,10.1.3.120:9092 > /root/yinzhengjie-kafka-producer.log &


[root@node105 ~]#

    2>.运行脚本时查看Kafka manager的监控

    3>.运行结束后查看Kafka manager的监控

    4>.查看我的Kafka性能测试结果

    生产端：
    总共需要发送的消息数：1亿条，每个记录的字节数是：1000字节，每秒钟发送的记录数是：1000000

消费端：
    模拟了10个消费者：进行消费

    指定每次fetch的数据的大小：1M
    总共要消费的消息个数：1亿条

生产端结论：
    1>.每秒平均向kafka写入了225.07 MB的数据；
    2>.每次写入的平均延迟为 129.81毫秒；
    3>.最大延迟为2422.00毫秒；

消费端结论：
    1>. 共消费了95367.4316M的数据；
    2>.消费速度为225.2047M/s；
    3>.总共消费了1亿条消息；
    4>.每秒消费236144.2369条消息；
  • 相关阅读:
    离散数学知识点总结（8）-图论
    离散数学知识点总结（7）-格
    离散数学知识点总结（6）-计数技术
    离散数学知识点总结（5）函数
    离散数学知识点总结（4）-集合
    离散数学知识点总结（3）-二元关系
    离散数学知识点总结（2）-谓词逻辑
    离散数学知识点总结（1）-命题逻辑
    镜像仓库和Harbor
    视频管理上云平台EasyNVS 2.1版本分享RTSP流和RTMP流端口发生变化是什么原因？
  • 原文地址：https://www.cnblogs.com/yinzhengjie/p/9953212.html
Copyright © 2011-2022 走看看