iozone用了很久，查看源码，发现iozone其实还附带两个工具fileop和pit_server，fileop测试了POSIX常用的函数，pit_server用来测试TCP或UDP服务
今天用了一下fileop测试ceph的读写瓶颈，一下就找到问题了。首先看一下帮助
[root@node2 iozone]# ./fileop -h -------------------------------------- | Fileop | | $Revision: 1.61 $ | | | | by | | | | Don Capps | -------------------------------------- fileop [-f X ]|[-l # -u #] [-s Y] [-e] [-b] [-w] [-d <dir>] [-t] [-v] [-h] -f # Force factor. X^3 files will be created and removed. -l # Lower limit on the value of the Force factor. -u # Upper limit on the value of the Force factor. -s # Optional. Sets filesize for the create/write. May use suffix 'K' or 'M'. -e Excel importable format. -b Output best case results. -i # Increment force factor by this increment. -w Output worst case results. -d <dir> Specify starting directory. -U <dir> Mount point to remount between tests. -t Verbose output option. -v Version information. -h Help text. The structure of the file tree is: X number of Level 1 directories, with X number of level 2 directories, with X number of files in each of the level 2 directories. Example: fileop 2 dir_1 dir_2 / / sdir_1 sdir_2 sdir_1 sdir_2 / / / / file_1 file_2 file_1 file_2 file_1 file_2 file_1 file_2 Each file will be created, and then Y bytes is written to the file.
例如：1.我们测试一个文件读写最好最坏和平均的结果。(单位：次/s)
[root@node2 iozone]# ./fileop -f 1 -d ./ -e -b -w . mkdir chdir rmdir create open read write close stat access chmod readdir link unlink delete Total_files A 1 447 103991 257 370 83886 66576 22795 33288 167772 123362 45590 527 931 1004 1033 1 B 1 592 262144 855 370 83886 66576 22795 33288 167772 123362 45590 527 931 1004 1033 1 W 1 361 47663 183 370 83886 66576 22795 33288 167772 123362 45590 527 931 1004 1033 1
2.我们用verbose打印详细日志查看结果
[root@node2 iozone]# ./fileop -t -f 1 -d ./ Fileop: Working in ., File size is 1, Output is in Ops/sec. (A=Avg, B=Best, W=Worst) mkdir: Dirs = 3 Total Time = 0.006633043 seconds Avg mkdir(s)/sec = 452.28 ( 0.002211014 seconds/op) Best mkdir(s)/sec = 529.38 ( 0.001888990 seconds/op) Worst mkdir(s)/sec = 411.17 ( 0.002432108 seconds/op) chdir: Dirs = 3 Total Time = 0.000052691 seconds Avg chdir(s)/sec = 56936.25 ( 0.000017564 seconds/op) Best chdir(s)/sec = 113359.57 ( 0.000008821 seconds/op) Worst chdir(s)/sec = 28532.68 ( 0.000035048 seconds/op) rmdir: Dirs = 3 Total Time = 0.011839867 seconds Avg rmdir(s)/sec = 253.38 ( 0.003946622 seconds/op) Best rmdir(s)/sec = 909.24 ( 0.001099825 seconds/op) Worst rmdir(s)/sec = 165.32 ( 0.006048918 seconds/op) create: Files = 1 Total Time = 0.002378941 seconds Avg create(s)/sec = 420.36 ( 0.002378941 seconds/op) Best create(s)/sec = 420.36 ( 0.002378941 seconds/op) Worst create(s)/sec = 420.36 ( 0.002378941 seconds/op) write: Files = 1 Total Time = 0.000038147 seconds Avg write(s)/sec = 26214.40 ( 0.000038147 seconds/op) Best write(s)/sec = 26214.40 ( 0.000038147 seconds/op) Worst write(s)/sec = 26214.40 ( 0.000038147 seconds/op) close: Files = 1 Total Time = 0.000033855 seconds Avg close(s)/sec = 29537.35 ( 0.000033855 seconds/op) Best close(s)/sec = 29537.35 ( 0.000033855 seconds/op) Worst close(s)/sec = 29537.35 ( 0.000033855 seconds/op) stat: Files = 1 Total Time = 0.000011206 seconds Avg stat(s)/sec = 89240.51 ( 0.000011206 seconds/op) Best stat(s)/sec = 89240.51 ( 0.000011206 seconds/op) Worst stat(s)/sec = 89240.51 ( 0.000011206 seconds/op) open: Files = 1 Total Time = 0.000014067 seconds Avg open(s)/sec = 71089.90 ( 0.000014067 seconds/op) Best open(s)/sec = 71089.90 ( 0.000014067 seconds/op) Worst open(s)/sec = 71089.90 ( 0.000014067 seconds/op) read: Files = 1 Total Time = 0.000015020 seconds Avg read(s)/sec = 66576.25 ( 0.000015020 seconds/op) Best read(s)/sec = 66576.25 ( 0.000015020 seconds/op) Worst read(s)/sec = 66576.25 ( 0.000015020 seconds/op) access: Files = 1 Total Time = 0.000008821 seconds Avg access(s)/sec = 113359.57 ( 0.000008821 seconds/op) Best access(s)/sec = 113359.57 ( 0.000008821 seconds/op) Worst access(s)/sec = 113359.57 ( 0.000008821 seconds/op) chmod: Files = 1 Total Time = 0.000021935 seconds Avg chmod(s)/sec = 45590.26 ( 0.000021935 seconds/op) Best chmod(s)/sec = 45590.26 ( 0.000021935 seconds/op) Worst chmod(s)/sec = 45590.26 ( 0.000021935 seconds/op) readdir: Files = 1 Total Time = 0.001602888 seconds Avg readdir(s)/sec = 623.87 ( 0.001602888 seconds/op) Best readdir(s)/sec = 623.87 ( 0.001602888 seconds/op) Worst readdir(s)/sec = 623.87 ( 0.001602888 seconds/op) link: Files = 1 Total Time = 0.000864983 seconds Avg link(s)/sec = 1156.09 ( 0.000864983 seconds/op) Best link(s)/sec = 1156.09 ( 0.000864983 seconds/op) Worst link(s)/sec = 1156.09 ( 0.000864983 seconds/op) unlink: Files = 1 Total Time = 0.000801086 seconds Avg unlink(s)/sec = 1248.30 ( 0.000801086 seconds/op) Best unlink(s)/sec = 1248.30 ( 0.000801086 seconds/op) Worst unlink(s)/sec = 1248.30 ( 0.000801086 seconds/op) delete: Files = 1 Total Time = 0.000800133 seconds Avg delete(s)/sec = 1249.79 ( 0.000800133 seconds/op) Best delete(s)/sec = 1249.79 ( 0.000800133 seconds/op) Worst delete(s)/sec = 1249.79 ( 0.000800133 seconds/op)
从结果中，我们可以得出结论，ceph在FS挂载的情况下，其实性能瓶颈主要是在readdir:rmdir:mkdir:三个函数上面。