zoukankan      html  css  js  c++  java

  • Kafka吞吐量测试案例

     　　　　　　　　　　　　　　　　　　　　　　　　Kafka吞吐量测试案例

    　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

    版权声明：原创作品，谢绝转载！否则将追究法律责任。

    　　领英公司参考连接：https://www.slideshare.net/JiangjieQin/producer-performance-tuning-for-apache-kafka-63147600?qid=84026ff8-243f-49a7-a4d0-69976cf317b7&v=&b=&from_search=9(温馨提示，需要VPN哟～)。

    一.kafka-producer-perf-test.sh 使用案例

    1>.测试kafka写入的吞吐量

    　　我用的kafka版本是0.10.2.1，领英用的貌似是0.10.0，我使用我的版本按照它的方式敲击，发现不太好事，这就有点小尴尬了！于是它表红线的参数我就没有使用。

    [root@node105 ~]# nohup kafka-producer-perf-test.sh --num-records 100000000 --record-size 1000 --topic yinzhengjie-kafka --throughput 1000000 --producer-props bootstrap.servers=1
0.1.2.101:9092,10.1.2.102:9092,10.1.2.103:9092,10.1.2.104:9092,10.1.2.105:9092 > yinzhengjie-kafka.log &
[root@node105 ~]# 

相关参数说明：
—topic             
    指定Kafka集群的topic名称，本例为yinzhengjie-kafka

    
    --num-records       
    
    　　总共需要发送的消息数，本例为100000000
    


    --record-size      
     每个记录的字节数，本例为1000



    --throughput       
     每秒钟发送的记录数，本例为20000
    


    --producer-props    bootstrap.servers=10.1.2.101:9092,10.1.2.102:9092,10.1.2.103:9092,10.1.2.104:9092,10.1.2.105:9092        
    很显然是kafka集群的broker地址

    2>.计算生产者的吞吐量

    [root@node105 ~]# cat yinzhengjie-kafka.log 
485204 records sent, 96673.4 records/sec (92.19 MB/sec), 284.4 ms avg latency, 648.0 max latency.
591703 records sent, 118340.6 records/sec (112.86 MB/sec), 260.7 ms avg latency, 795.0 max latency.
568061 records sent, 113612.2 records/sec (108.35 MB/sec), 268.2 ms avg latency, 870.0 max latency.
576896 records sent, 115034.1 records/sec (109.71 MB/sec), 268.7 ms avg latency, 732.0 max latency.
548140 records sent, 106912.4 records/sec (101.96 MB/sec), 257.9 ms avg latency, 913.0 max latency.
568845 records sent, 112575.7 records/sec (107.36 MB/sec), 294.4 ms avg latency, 1283.0 max latency.
557990 records sent, 111598.0 records/sec (106.43 MB/sec), 279.4 ms avg latency, 828.0 max latency.
464493 records sent, 92898.6 records/sec (88.60 MB/sec), 329.3 ms avg latency, 1213.0 max latency.
488281 records sent, 97656.2 records/sec (93.13 MB/sec), 312.9 ms avg latency, 1020.0 max latency.
517560 records sent, 103512.0 records/sec (98.72 MB/sec), 294.8 ms avg latency, 1062.0 max latency.
209745 records sent, 41949.0 records/sec (40.01 MB/sec), 740.2 ms avg latency, 3722.0 max latency.
394980 records sent, 78980.2 records/sec (75.32 MB/sec), 388.7 ms avg latency, 2079.0 max latency.
445178 records sent, 89035.6 records/sec (84.91 MB/sec), 342.4 ms avg latency, 1274.0 max latency.
693946 records sent, 138789.2 records/sec (132.36 MB/sec), 223.2 ms avg latency, 469.0 max latency.
367182 records sent, 62616.3 records/sec (59.72 MB/sec), 247.3 ms avg latency, 3303.0 max latency.
631412 records sent, 126282.4 records/sec (120.43 MB/sec), 381.9 ms avg latency, 3304.0 max latency.
393343 records sent, 78668.6 records/sec (75.02 MB/sec), 386.0 ms avg latency, 1857.0 max latency.
441035 records sent, 88207.0 records/sec (84.12 MB/sec), 347.2 ms avg latency, 1018.0 max latency.
521952 records sent, 104223.6 records/sec (99.40 MB/sec), 296.4 ms avg latency, 698.0 max latency.
641069 records sent, 127779.4 records/sec (121.86 MB/sec), 238.2 ms avg latency, 560.0 max latency.
514232 records sent, 101747.5 records/sec (97.03 MB/sec), 296.9 ms avg latency, 855.0 max latency.
582718 records sent, 116543.6 records/sec (111.14 MB/sec), 269.8 ms avg latency, 633.0 max latency.
15689 records sent, 2589.8 records/sec (2.47 MB/sec), 262.5 ms avg latency, 6291.0 max latency.
300150 records sent, 34393.3 records/sec (32.80 MB/sec), 954.0 ms avg latency, 6731.0 max latency.
528932 records sent, 105786.4 records/sec (100.89 MB/sec), 595.4 ms avg latency, 5857.0 max latency.
460320 records sent, 91770.3 records/sec (87.52 MB/sec), 335.8 ms avg latency, 1688.0 max latency.
613573 records sent, 119815.1 records/sec (114.26 MB/sec), 245.9 ms avg latency, 931.0 max latency.
316608 records sent, 63321.6 records/sec (60.39 MB/sec), 502.0 ms avg latency, 2900.0 max latency.
581817 records sent, 116084.8 records/sec (110.71 MB/sec), 262.5 ms avg latency, 540.0 max latency.
636679 records sent, 125801.0 records/sec (119.97 MB/sec), 244.7 ms avg latency, 517.0 max latency.
501821 records sent, 97554.6 records/sec (93.04 MB/sec), 298.4 ms avg latency, 1026.0 max latency.
92700 records sent, 18197.9 records/sec (17.35 MB/sec), 1709.4 ms avg latency, 4843.0 max latency.
430113 records sent, 86022.6 records/sec (82.04 MB/sec), 370.9 ms avg latency, 1119.0 max latency.
318603 records sent, 63720.6 records/sec (60.77 MB/sec), 478.7 ms avg latency, 2001.0 max latency.
389034 records sent, 76732.5 records/sec (73.18 MB/sec), 379.7 ms avg latency, 1345.0 max latency.
413297 records sent, 82659.4 records/sec (78.83 MB/sec), 391.5 ms avg latency, 1016.0 max latency.
258853 records sent, 51729.2 records/sec (49.33 MB/sec), 528.6 ms avg latency, 2228.0 max latency.
437625 records sent, 87490.0 records/sec (83.44 MB/sec), 377.5 ms avg latency, 1178.0 max latency.
352050 records sent, 70227.4 records/sec (66.97 MB/sec), 424.4 ms avg latency, 1456.0 max latency.
402187 records sent, 80437.4 records/sec (76.71 MB/sec), 394.4 ms avg latency, 1452.0 max latency.
406540 records sent, 81178.1 records/sec (77.42 MB/sec), 383.9 ms avg latency, 1344.0 max latency.
352573 records sent, 66965.4 records/sec (63.86 MB/sec), 426.7 ms avg latency, 2243.0 max latency.
405849 records sent, 81169.8 records/sec (77.41 MB/sec), 406.3 ms avg latency, 1204.0 max latency.
394551 records sent, 78674.2 records/sec (75.03 MB/sec), 369.5 ms avg latency, 980.0 max latency.
371400 records sent, 74235.5 records/sec (70.80 MB/sec), 376.5 ms avg latency, 1539.0 max latency.
224325 records sent, 44141.1 records/sec (42.10 MB/sec), 730.9 ms avg latency, 2348.0 max latency.
294300 records sent, 56142.7 records/sec (53.54 MB/sec), 537.1 ms avg latency, 1441.0 max latency.
350175 records sent, 69909.2 records/sec (66.67 MB/sec), 480.5 ms avg latency, 1660.0 max latency.
350175 records sent, 66674.6 records/sec (63.59 MB/sec), 437.3 ms avg latency, 1390.0 max latency.
364683 records sent, 72936.6 records/sec (69.56 MB/sec), 440.6 ms avg latency, 1651.0 max latency.
385692 records sent, 76708.8 records/sec (73.16 MB/sec), 326.2 ms avg latency, 1466.0 max latency.
388067 records sent, 76875.4 records/sec (73.31 MB/sec), 468.7 ms avg latency, 1773.0 max latency.
312283 records sent, 62170.6 records/sec (59.29 MB/sec), 492.0 ms avg latency, 1382.0 max latency.
350175 records sent, 65983.6 records/sec (62.93 MB/sec), 432.4 ms avg latency, 1238.0 max latency.
400200 records sent, 79912.1 records/sec (76.21 MB/sec), 420.5 ms avg latency, 1244.0 max latency.
335917 records sent, 66623.8 records/sec (63.54 MB/sec), 445.0 ms avg latency, 1054.0 max latency.
364433 records sent, 70929.0 records/sec (67.64 MB/sec), 413.4 ms avg latency, 1165.0 max latency.
305884 records sent, 61176.8 records/sec (58.34 MB/sec), 536.8 ms avg latency, 1517.0 max latency.
430627 records sent, 85953.5 records/sec (81.97 MB/sec), 354.6 ms avg latency, 1919.0 max latency.
414064 records sent, 80229.4 records/sec (76.51 MB/sec), 363.1 ms avg latency, 1194.0 max latency.
304470 records sent, 60894.0 records/sec (58.07 MB/sec), 509.9 ms avg latency, 1365.0 max latency.
385418 records sent, 77083.6 records/sec (73.51 MB/sec), 417.5 ms avg latency, 1468.0 max latency.
402698 records sent, 80059.2 records/sec (76.35 MB/sec), 343.1 ms avg latency, 1147.0 max latency.
419702 records sent, 83940.4 records/sec (80.05 MB/sec), 402.7 ms avg latency, 1136.0 max latency.
272698 records sent, 54539.6 records/sec (52.01 MB/sec), 554.7 ms avg latency, 2612.0 max latency.
327574 records sent, 65514.8 records/sec (62.48 MB/sec), 452.0 ms avg latency, 1112.0 max latency.
444357 records sent, 88871.4 records/sec (84.75 MB/sec), 304.7 ms avg latency, 1523.0 max latency.
325969 records sent, 65193.8 records/sec (62.17 MB/sec), 546.3 ms avg latency, 2538.0 max latency.
391147 records sent, 78229.4 records/sec (74.61 MB/sec), 380.9 ms avg latency, 1704.0 max latency.
418355 records sent, 83671.0 records/sec (79.79 MB/sec), 380.5 ms avg latency, 1402.0 max latency.
318282 records sent, 63656.4 records/sec (60.71 MB/sec), 465.3 ms avg latency, 2996.0 max latency.
431130 records sent, 85916.7 records/sec (81.94 MB/sec), 366.3 ms avg latency, 1606.0 max latency.
398883 records sent, 79776.6 records/sec (76.08 MB/sec), 380.9 ms avg latency, 1387.0 max latency.
551592 records sent, 109791.4 records/sec (104.71 MB/sec), 270.6 ms avg latency, 1512.0 max latency.
483225 records sent, 96645.0 records/sec (92.17 MB/sec), 317.9 ms avg latency, 988.0 max latency.
317175 records sent, 58519.4 records/sec (55.81 MB/sec), 473.6 ms avg latency, 2136.0 max latency.
339750 records sent, 67665.8 records/sec (64.53 MB/sec), 491.5 ms avg latency, 2083.0 max latency.
421923 records sent, 84384.6 records/sec (80.48 MB/sec), 384.9 ms avg latency, 1260.0 max latency.
365130 records sent, 73026.0 records/sec (69.64 MB/sec), 410.0 ms avg latency, 1084.0 max latency.
346304 records sent, 69260.8 records/sec (66.05 MB/sec), 456.2 ms avg latency, 1526.0 max latency.
347720 records sent, 69544.0 records/sec (66.32 MB/sec), 420.3 ms avg latency, 1192.0 max latency.
374846 records sent, 74969.2 records/sec (71.50 MB/sec), 421.8 ms avg latency, 1478.0 max latency.
332024 records sent, 66404.8 records/sec (63.33 MB/sec), 451.3 ms avg latency, 1840.0 max latency.
413353 records sent, 82079.6 records/sec (78.28 MB/sec), 355.0 ms avg latency, 1328.0 max latency.
456013 records sent, 91202.6 records/sec (86.98 MB/sec), 362.7 ms avg latency, 1106.0 max latency.
443987 records sent, 85744.9 records/sec (81.77 MB/sec), 302.0 ms avg latency, 1187.0 max latency.
401025 records sent, 78694.1 records/sec (75.05 MB/sec), 431.8 ms avg latency, 1545.0 max latency.
339375 records sent, 67820.7 records/sec (64.68 MB/sec), 443.9 ms avg latency, 1530.0 max latency.
460950 records sent, 84655.6 records/sec (80.73 MB/sec), 304.3 ms avg latency, 1762.0 max latency.
339450 records sent, 66952.7 records/sec (63.85 MB/sec), 546.6 ms avg latency, 2097.0 max latency.
394914 records sent, 78982.8 records/sec (75.32 MB/sec), 402.1 ms avg latency, 1448.0 max latency.
291396 records sent, 57679.3 records/sec (55.01 MB/sec), 517.6 ms avg latency, 2158.0 max latency.
372347 records sent, 74469.4 records/sec (71.02 MB/sec), 397.6 ms avg latency, 1819.0 max latency.
408480 records sent, 81696.0 records/sec (77.91 MB/sec), 399.9 ms avg latency, 1226.0 max latency.
367899 records sent, 73579.8 records/sec (70.17 MB/sec), 423.3 ms avg latency, 1575.0 max latency.
270450 records sent, 54090.0 records/sec (51.58 MB/sec), 513.7 ms avg latency, 1724.0 max latency.
395764 records sent, 78276.1 records/sec (74.65 MB/sec), 387.1 ms avg latency, 1762.0 max latency.
437056 records sent, 87411.2 records/sec (83.36 MB/sec), 380.6 ms avg latency, 1763.0 max latency.
363344 records sent, 69182.0 records/sec (65.98 MB/sec), 421.1 ms avg latency, 1361.0 max latency.
450225 records sent, 87202.2 records/sec (83.16 MB/sec), 359.5 ms avg latency, 1487.0 max latency.
400200 records sent, 75623.6 records/sec (72.12 MB/sec), 265.6 ms avg latency, 2355.0 max latency.
335026 records sent, 67005.2 records/sec (63.90 MB/sec), 625.9 ms avg latency, 2568.0 max latency.
376506 records sent, 75301.2 records/sec (71.81 MB/sec), 389.1 ms avg latency, 1071.0 max latency.
281738 records sent, 56347.6 records/sec (53.74 MB/sec), 541.7 ms avg latency, 2047.0 max latency.
346980 records sent, 66255.5 records/sec (63.19 MB/sec), 461.9 ms avg latency, 1747.0 max latency.
351900 records sent, 69244.4 records/sec (66.04 MB/sec), 408.8 ms avg latency, 1318.0 max latency.
265397 records sent, 53079.4 records/sec (50.62 MB/sec), 630.5 ms avg latency, 1979.0 max latency.
424290 records sent, 84722.4 records/sec (80.80 MB/sec), 387.0 ms avg latency, 1130.0 max latency.
302624 records sent, 60500.6 records/sec (57.70 MB/sec), 504.8 ms avg latency, 1415.0 max latency.
356639 records sent, 71213.9 records/sec (67.91 MB/sec), 397.4 ms avg latency, 1366.0 max latency.
400200 records sent, 76887.6 records/sec (73.33 MB/sec), 405.1 ms avg latency, 1269.0 max latency.
435842 records sent, 87168.4 records/sec (83.13 MB/sec), 378.4 ms avg latency, 1388.0 max latency.
364558 records sent, 70801.7 records/sec (67.52 MB/sec), 411.4 ms avg latency, 1323.0 max latency.
449387 records sent, 89877.4 records/sec (85.71 MB/sec), 355.8 ms avg latency, 1039.0 max latency.
403288 records sent, 77376.8 records/sec (73.79 MB/sec), 340.2 ms avg latency, 1702.0 max latency.
417825 records sent, 80242.9 records/sec (76.53 MB/sec), 421.3 ms avg latency, 1226.0 max latency.
369900 records sent, 72686.2 records/sec (69.32 MB/sec), 417.1 ms avg latency, 1447.0 max latency.
437842 records sent, 87568.4 records/sec (83.51 MB/sec), 359.2 ms avg latency, 2027.0 max latency.
450833 records sent, 89986.6 records/sec (85.82 MB/sec), 343.0 ms avg latency, 1238.0 max latency.
468195 records sent, 93639.0 records/sec (89.30 MB/sec), 318.6 ms avg latency, 887.0 max latency.
381300 records sent, 76260.0 records/sec (72.73 MB/sec), 416.3 ms avg latency, 1537.0 max latency.
386025 records sent, 77205.0 records/sec (73.63 MB/sec), 398.5 ms avg latency, 1199.0 max latency.
355139 records sent, 71027.8 records/sec (67.74 MB/sec), 431.3 ms avg latency, 1298.0 max latency.
374749 records sent, 74770.4 records/sec (71.31 MB/sec), 353.8 ms avg latency, 1290.0 max latency.
397407 records sent, 79481.4 records/sec (75.80 MB/sec), 438.1 ms avg latency, 1648.0 max latency.
360720 records sent, 72144.0 records/sec (68.80 MB/sec), 410.4 ms avg latency, 1585.0 max latency.
364249 records sent, 72849.8 records/sec (69.47 MB/sec), 413.8 ms avg latency, 1505.0 max latency.
401363 records sent, 80272.6 records/sec (76.55 MB/sec), 388.3 ms avg latency, 1054.0 max latency.
314103 records sent, 59220.0 records/sec (56.48 MB/sec), 495.4 ms avg latency, 1338.0 max latency.
298134 records sent, 59626.8 records/sec (56.86 MB/sec), 551.7 ms avg latency, 1610.0 max latency.
295673 records sent, 59134.6 records/sec (56.40 MB/sec), 522.3 ms avg latency, 1689.0 max latency.
386488 records sent, 77297.6 records/sec (73.72 MB/sec), 400.0 ms avg latency, 1311.0 max latency.
422477 records sent, 84495.4 records/sec (80.58 MB/sec), 359.9 ms avg latency, 1357.0 max latency.
364903 records sent, 72980.6 records/sec (69.60 MB/sec), 405.4 ms avg latency, 1113.0 max latency.
333375 records sent, 63415.4 records/sec (60.48 MB/sec), 451.5 ms avg latency, 1377.0 max latency.
407003 records sent, 81400.6 records/sec (77.63 MB/sec), 390.1 ms avg latency, 1321.0 max latency.
393397 records sent, 77076.2 records/sec (73.51 MB/sec), 408.1 ms avg latency, 1285.0 max latency.
388801 records sent, 77760.2 records/sec (74.16 MB/sec), 396.4 ms avg latency, 1162.0 max latency.
383069 records sent, 76613.8 records/sec (73.06 MB/sec), 417.4 ms avg latency, 1343.0 max latency.
378705 records sent, 73735.4 records/sec (70.32 MB/sec), 391.5 ms avg latency, 2222.0 max latency.
385890 records sent, 77178.0 records/sec (73.60 MB/sec), 413.3 ms avg latency, 1166.0 max latency.
444551 records sent, 88910.2 records/sec (84.79 MB/sec), 353.1 ms avg latency, 1147.0 max latency.
347314 records sent, 69296.5 records/sec (66.09 MB/sec), 438.3 ms avg latency, 1684.0 max latency.
316545 records sent, 62508.9 records/sec (59.61 MB/sec), 446.3 ms avg latency, 1352.0 max latency.
433528 records sent, 86085.8 records/sec (82.10 MB/sec), 387.1 ms avg latency, 1173.0 max latency.
374452 records sent, 74890.4 records/sec (71.42 MB/sec), 385.6 ms avg latency, 1550.0 max latency.
338620 records sent, 67133.2 records/sec (64.02 MB/sec), 453.3 ms avg latency, 1589.0 max latency.
374775 records sent, 73993.1 records/sec (70.57 MB/sec), 412.4 ms avg latency, 1358.0 max latency.
357450 records sent, 70586.5 records/sec (67.32 MB/sec), 456.5 ms avg latency, 1546.0 max latency.
275232 records sent, 55046.4 records/sec (52.50 MB/sec), 512.0 ms avg latency, 1688.0 max latency.
442490 records sent, 88498.0 records/sec (84.40 MB/sec), 370.3 ms avg latency, 1130.0 max latency.
313110 records sent, 62622.0 records/sec (59.72 MB/sec), 498.3 ms avg latency, 1636.0 max latency.
400308 records sent, 80061.6 records/sec (76.35 MB/sec), 395.6 ms avg latency, 1373.0 max latency.
337885 records sent, 66512.8 records/sec (63.43 MB/sec), 395.0 ms avg latency, 1690.0 max latency.
446967 records sent, 89393.4 records/sec (85.25 MB/sec), 386.0 ms avg latency, 1827.0 max latency.
439940 records sent, 87988.0 records/sec (83.91 MB/sec), 355.3 ms avg latency, 1251.0 max latency.
262322 records sent, 52464.4 records/sec (50.03 MB/sec), 516.0 ms avg latency, 1680.0 max latency.
314593 records sent, 62918.6 records/sec (60.00 MB/sec), 544.6 ms avg latency, 1431.0 max latency.
312654 records sent, 61985.3 records/sec (59.11 MB/sec), 481.0 ms avg latency, 1430.0 max latency.
399757 records sent, 79951.4 records/sec (76.25 MB/sec), 386.0 ms avg latency, 1160.0 max latency.
400965 records sent, 80193.0 records/sec (76.48 MB/sec), 381.1 ms avg latency, 1262.0 max latency.
363852 records sent, 69998.5 records/sec (66.76 MB/sec), 398.8 ms avg latency, 1203.0 max latency.
536223 records sent, 107223.2 records/sec (102.26 MB/sec), 317.9 ms avg latency, 1063.0 max latency.
314202 records sent, 60962.7 records/sec (58.14 MB/sec), 425.6 ms avg latency, 2061.0 max latency.
240900 records sent, 48151.1 records/sec (45.92 MB/sec), 674.7 ms avg latency, 1933.0 max latency.
209325 records sent, 40023.9 records/sec (38.17 MB/sec), 752.9 ms avg latency, 1802.0 max latency.
256875 records sent, 51344.2 records/sec (48.97 MB/sec), 622.6 ms avg latency, 1677.0 max latency.
193350 records sent, 37986.2 records/sec (36.23 MB/sec), 762.4 ms avg latency, 1840.0 max latency.
483000 records sent, 96311.1 records/sec (91.85 MB/sec), 308.9 ms avg latency, 1841.0 max latency.
382314 records sent, 76401.7 records/sec (72.86 MB/sec), 443.2 ms avg latency, 1543.0 max latency.
435336 records sent, 80602.9 records/sec (76.87 MB/sec), 356.3 ms avg latency, 1044.0 max latency.
373638 records sent, 74727.6 records/sec (71.27 MB/sec), 430.1 ms avg latency, 1650.0 max latency.
336477 records sent, 67295.4 records/sec (64.18 MB/sec), 438.0 ms avg latency, 1648.0 max latency.
350867 records sent, 70173.4 records/sec (66.92 MB/sec), 458.6 ms avg latency, 1661.0 max latency.
499218 records sent, 99843.6 records/sec (95.22 MB/sec), 312.3 ms avg latency, 1054.0 max latency.
362143 records sent, 72428.6 records/sec (69.07 MB/sec), 433.5 ms avg latency, 1483.0 max latency.
308457 records sent, 60269.1 records/sec (57.48 MB/sec), 491.1 ms avg latency, 1619.0 max latency.
375692 records sent, 75138.4 records/sec (71.66 MB/sec), 422.2 ms avg latency, 1286.0 max latency.
387466 records sent, 76908.7 records/sec (73.35 MB/sec), 374.8 ms avg latency, 1611.0 max latency.
347292 records sent, 62428.9 records/sec (59.54 MB/sec), 464.3 ms avg latency, 1787.0 max latency.
354210 records sent, 70842.0 records/sec (67.56 MB/sec), 484.1 ms avg latency, 2107.0 max latency.
496215 records sent, 87147.0 records/sec (83.11 MB/sec), 306.1 ms avg latency, 1225.0 max latency.
400200 records sent, 75395.6 records/sec (71.90 MB/sec), 433.9 ms avg latency, 1668.0 max latency.
361743 records sent, 72348.6 records/sec (69.00 MB/sec), 453.9 ms avg latency, 1441.0 max latency.
388632 records sent, 68590.2 records/sec (65.41 MB/sec), 397.4 ms avg latency, 1394.0 max latency.
282975 records sent, 56012.5 records/sec (53.42 MB/sec), 593.2 ms avg latency, 1667.0 max latency.
340549 records sent, 68109.8 records/sec (64.95 MB/sec), 480.5 ms avg latency, 1633.0 max latency.
320276 records sent, 60750.4 records/sec (57.94 MB/sec), 451.4 ms avg latency, 1519.0 max latency.
382376 records sent, 75778.0 records/sec (72.27 MB/sec), 423.8 ms avg latency, 1378.0 max latency.
375394 records sent, 75078.8 records/sec (71.60 MB/sec), 403.5 ms avg latency, 1199.0 max latency.
442290 records sent, 88334.3 records/sec (84.24 MB/sec), 367.1 ms avg latency, 935.0 max latency.
435002 records sent, 87000.4 records/sec (82.97 MB/sec), 360.3 ms avg latency, 1660.0 max latency.
348105 records sent, 69621.0 records/sec (66.40 MB/sec), 429.2 ms avg latency, 1201.0 max latency.
378901 records sent, 75780.2 records/sec (72.27 MB/sec), 403.9 ms avg latency, 1311.0 max latency.
242099 records sent, 48419.8 records/sec (46.18 MB/sec), 587.5 ms avg latency, 2074.0 max latency.
380623 records sent, 75475.5 records/sec (71.98 MB/sec), 441.9 ms avg latency, 1329.0 max latency.
299522 records sent, 59701.4 records/sec (56.94 MB/sec), 494.5 ms avg latency, 1573.0 max latency.
374171 records sent, 74834.2 records/sec (71.37 MB/sec), 375.8 ms avg latency, 1409.0 max latency.
439817 records sent, 84547.7 records/sec (80.63 MB/sec), 371.5 ms avg latency, 1183.0 max latency.
364202 records sent, 72840.4 records/sec (69.47 MB/sec), 456.6 ms avg latency, 1347.0 max latency.
408650 records sent, 81730.0 records/sec (77.94 MB/sec), 348.7 ms avg latency, 1212.0 max latency.
386220 records sent, 77244.0 records/sec (73.67 MB/sec), 430.5 ms avg latency, 1499.0 max latency.
590650 records sent, 118130.0 records/sec (112.66 MB/sec), 257.1 ms avg latency, 1053.0 max latency.
329128 records sent, 63208.8 records/sec (60.28 MB/sec), 349.1 ms avg latency, 1879.0 max latency.
408542 records sent, 81708.4 records/sec (77.92 MB/sec), 433.5 ms avg latency, 2531.0 max latency.
337366 records sent, 67405.8 records/sec (64.28 MB/sec), 505.4 ms avg latency, 1556.0 max latency.
416442 records sent, 73265.7 records/sec (69.87 MB/sec), 378.6 ms avg latency, 1670.0 max latency.
360097 records sent, 72019.4 records/sec (68.68 MB/sec), 428.8 ms avg latency, 1715.0 max latency.
403744 records sent, 80748.8 records/sec (77.01 MB/sec), 388.2 ms avg latency, 1085.0 max latency.
394909 records sent, 76266.7 records/sec (72.73 MB/sec), 387.5 ms avg latency, 1689.0 max latency.
392025 records sent, 78154.9 records/sec (74.53 MB/sec), 448.3 ms avg latency, 1543.0 max latency.
386595 records sent, 77319.0 records/sec (73.74 MB/sec), 400.5 ms avg latency, 1478.0 max latency.
424468 records sent, 84893.6 records/sec (80.96 MB/sec), 360.7 ms avg latency, 1372.0 max latency.
329087 records sent, 63604.0 records/sec (60.66 MB/sec), 431.9 ms avg latency, 1677.0 max latency.
334860 records sent, 66125.6 records/sec (63.06 MB/sec), 467.7 ms avg latency, 1670.0 max latency.
215415 records sent, 41101.9 records/sec (39.20 MB/sec), 710.9 ms avg latency, 2273.0 max latency.
243600 records sent, 48237.6 records/sec (46.00 MB/sec), 688.9 ms avg latency, 1916.0 max latency.
443820 records sent, 88764.0 records/sec (84.65 MB/sec), 364.9 ms avg latency, 1812.0 max latency.
711347 records sent, 142269.4 records/sec (135.68 MB/sec), 216.4 ms avg latency, 453.0 max latency.
577303 records sent, 115460.6 records/sec (110.11 MB/sec), 263.4 ms avg latency, 1072.0 max latency.
296205 records sent, 55510.7 records/sec (52.94 MB/sec), 212.9 ms avg latency, 3721.0 max latency.
329025 records sent, 60705.7 records/sec (57.89 MB/sec), 761.2 ms avg latency, 3900.0 max latency.
287524 records sent, 57390.0 records/sec (54.73 MB/sec), 592.4 ms avg latency, 2092.0 max latency.
504626 records sent, 100423.1 records/sec (95.77 MB/sec), 305.8 ms avg latency, 1828.0 max latency.
452748 records sent, 90549.6 records/sec (86.35 MB/sec), 340.5 ms avg latency, 1327.0 max latency.
305877 records sent, 57789.0 records/sec (55.11 MB/sec), 258.8 ms avg latency, 3117.0 max latency.
423903 records sent, 84780.6 records/sec (80.85 MB/sec), 556.3 ms avg latency, 3368.0 max latency.
451797 records sent, 84988.1 records/sec (81.05 MB/sec), 315.4 ms avg latency, 1309.0 max latency.
475265 records sent, 95053.0 records/sec (90.65 MB/sec), 363.2 ms avg latency, 1313.0 max latency.
577183 records sent, 114770.9 records/sec (109.45 MB/sec), 264.1 ms avg latency, 861.0 max latency.
673152 records sent, 112661.4 records/sec (107.44 MB/sec), 230.4 ms avg latency, 1321.0 max latency.
689925 records sent, 137957.4 records/sec (131.57 MB/sec), 265.3 ms avg latency, 1400.0 max latency.
400200 records sent, 79928.1 records/sec (76.23 MB/sec), 384.9 ms avg latency, 2438.0 max latency.
369566 records sent, 73913.2 records/sec (70.49 MB/sec), 410.9 ms avg latency, 2295.0 max latency.
466609 records sent, 86908.0 records/sec (82.88 MB/sec), 318.5 ms avg latency, 1511.0 max latency.
642825 records sent, 117196.9 records/sec (111.77 MB/sec), 253.9 ms avg latency, 1139.0 max latency.
365312 records sent, 73062.4 records/sec (69.68 MB/sec), 438.2 ms avg latency, 1618.0 max latency.
546463 records sent, 108792.2 records/sec (103.75 MB/sec), 309.4 ms avg latency, 975.0 max latency.
449057 records sent, 89811.4 records/sec (85.65 MB/sec), 338.8 ms avg latency, 1284.0 max latency.
461461 records sent, 92292.2 records/sec (88.02 MB/sec), 336.2 ms avg latency, 1115.0 max latency.
425741 records sent, 84438.9 records/sec (80.53 MB/sec), 360.2 ms avg latency, 1078.0 max latency.
527314 records sent, 105462.8 records/sec (100.58 MB/sec), 288.8 ms avg latency, 1230.0 max latency.
516075 records sent, 103215.0 records/sec (98.43 MB/sec), 300.2 ms avg latency, 1021.0 max latency.
513447 records sent, 102607.3 records/sec (97.85 MB/sec), 299.7 ms avg latency, 1108.0 max latency.
392184 records sent, 78436.8 records/sec (74.80 MB/sec), 377.8 ms avg latency, 1352.0 max latency.
516621 records sent, 97715.3 records/sec (93.19 MB/sec), 261.6 ms avg latency, 1416.0 max latency.
512059 records sent, 102411.8 records/sec (97.67 MB/sec), 325.1 ms avg latency, 1509.0 max latency.
377966 records sent, 73980.4 records/sec (70.55 MB/sec), 422.2 ms avg latency, 1893.0 max latency.
503959 records sent, 100791.8 records/sec (96.12 MB/sec), 332.4 ms avg latency, 1506.0 max latency.
474116 records sent, 94407.8 records/sec (90.03 MB/sec), 321.2 ms avg latency, 1079.0 max latency.
100000000 records sent, 78434.325371 records/sec (74.80 MB/sec), 390.67 ms avg latency, 6731.00 ms max latency, 2 ms 50th, 87 ms 95th, 158 ms 99th, 275 ms 99.9th.
[root@node105 ~]#
    [root@node105 ~]# cat yinzhengjie-kafka.log　　　　　　　　　　　　　　#查看Kafka写入日志 
    100000000 records sent, 78434.325371 records/sec (74.80 MB/sec), 390.67 ms avg latency, 6731.00 ms max latency, 2 ms 50th, 87 ms 95th, 158 ms 99th, 275 ms 99.9th.

当你执行完kafka-producer-perf-test.sh 脚本时，会输出类似上面的一行，从上面一行我们可以得出一些消息
    1>.每秒平均向kafka写入了74.80 MB的数据(大概是：78433条消息，计算方式：74.8*1024*1024/1000);
    2>.每次写入的平均延迟为390.67毫秒；
    3>.最大延迟为6731.00毫秒；
    4>.87毫秒内占95%；
    5>.158毫秒内占99%；
    6>.275毫秒内占99.9%；

    二.kafka-consumer-perf-test.sh 使用案例

    1>. 测试Kafka消费的吞吐量

    [root@node105 ~]# nohup kafka-consumer-perf-test.sh --broker-list 10.1.2.102:9092  --topic yinzhengjie-kafka --messages 1000000 --fetch-size 1048576 --threads 10  > yinzhengjie-kafka-consumer.log &
[root@node105 ~]# 


相关参数说明：
--topic 
    指定topic的名称，本例为 yinzhengjie-kafka

--fetch-size 
    
    　　指定每次fetch的数据的大小，本例为1048576，也就是1M

    --messages 　　　　
    　　总共要消费的消息个数，本例为1000000，100w

    --topic
    　　指定消费的topic

    --threads
    　　指定消费的线程数为10
    　　　　

    2>.计算消费者的吞吐量

    [root@node105 ~]# cat yinzhengjie-kafka-consumer.log 
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec
2018-11-13 19:41:31:507, 2018-11-13 19:41:34:169, 953.9165, 358.3458, 1000411, 375811.7956
[root@node105 ~]# 


当你执行完 kafka-consumer-perf-test.sh 脚本时，会输出类似于上面的一行，从上面一行我们可以得出一些消息
    1>.共消费了953.9165M的数据；
    2>.消费速度为358.3458M/s；
    3>.总共消费了1000411 条消息；
    4>.每秒消费375811.7956条消息；

    三.编写脚本案例

    1>.执行脚本

    [root@node105 ~]# cat yinzhengjie-kafka.sh 
nohup kafka-consumer-perf-test.sh --broker-list 10.1.3.116:9092,10.1.3.117:9092,10.1.3.118:9092,10.1.3.119:9092,10.1.3.120:9092 --topic yinzhengjie-kafka --messages 100000000 --fetch-size 1048576 --threads 10  > /root/yinzhengjie-kafka-consumer.log &


nohup kafka-producer-perf-test.sh --num-records 100000000 --record-size 1000 --topic yinzhengjie-kafka --throughput 1000000 --producer-props bootstrap.servers=10.1.3.116:9092,10.1.3.117:9092,10.1.3.118:9092,10.1.3.119:9092,10.1.3.120:9092 > /root/yinzhengjie-kafka-producer.log &


[root@node105 ~]#

    2>.运行脚本时查看Kafka manager的监控

    3>.运行结束后查看Kafka manager的监控

    4>.查看我的Kafka性能测试结果

    生产端：
    总共需要发送的消息数：1亿条，每个记录的字节数是：1000字节，每秒钟发送的记录数是：1000000

消费端：
    模拟了10个消费者：进行消费

    指定每次fetch的数据的大小：1M
    总共要消费的消息个数：1亿条

生产端结论：
    1>.每秒平均向kafka写入了225.07 MB的数据；
    2>.每次写入的平均延迟为 129.81毫秒；
    3>.最大延迟为2422.00毫秒；

消费端结论：
    1>. 共消费了95367.4316M的数据；
    2>.消费速度为225.2047M/s；
    3>.总共消费了1亿条消息；
    4>.每秒消费236144.2369条消息；
  • 相关阅读:
    896. Monotonic Array单调数组
    865. Smallest Subtree with all the Deepest Nodes 有最深节点的最小子树
    489. Robot Room Cleaner扫地机器人
    JavaFX
    《Python CookBook2》 第一章 文本
    《Python CookBook2》 第一章 文本
    《Python CookBook2》 第一章 文本
    《Python CookBook2》 第一章 文本
    《Python CookBook2》 第一章 文本
    《Python CookBook2》 第一章 文本
  • 原文地址：https://www.cnblogs.com/yinzhengjie/p/9953212.html
Copyright © 2011-2022 走看看