软件介绍
mdstest是软件的元数据操作基准测试工具，用来模拟对文件或者目录的open、stat、close操作，然后报告性能
下载软件压缩包：
yum install openmpi openmpi-devel -y
在/root/.bashrc中添加（注意也要添加mdtest的路径）
export PATH=$PATH:/usr/lib64/openmpi/bin/
source /root/.bashrc
下载
[root@lab8105 ~]# wget http://sourceforge.net/projects/mdtest/files/latest/download
解压
[root@lab8105 ~]# tar -xvf mdtest-1.9.3.tgz
修改makefile：
mdtest: mdtest.c
mpicc -Wall -D $(OS) $(LARGE_FILE) $(MDTEST_FLAGS) -g -o mdtest mdtest.c -lm
参数如下：
Usage: mdtest [-b #] [-B] [-c] [-C] [-d testdir] [-D] [-e] [-E] [-f first] [-F]
[-h] [-i iterations] [-I #] [-l last] [-L] [-n #] [-N #] [-p seconds]
[-r] [-R[#]] [-s #] [-S] [-t] [-T] [-u] [-v] [-V #] [-w #] [-y]
[-z #]
-b: branching factor of hierarchical directory structure
目录树的分支参数
-B: no barriers between phases (create/stat/remove)
不同的阶段没有隔离 (create/stat/remove)；
-c: collective creates: task 0 does all creates and deletes
共同创建: task 0 完成所有的创建和删除工作；
-C: only create files/dirs
只创建文件或目录，不作删除；
-d: the directory in which the tests will run
指出测试运行的目录（若不指定，则默认当前目录）；
-D: perform test on directories only (no files)
只对目录操作进行测试（不包括文件）；
-e: number of bytes to read from each file
从每个文件读出的文件大小
-E: only read files
只读取文件
-f: first number of tasks on which the test will run
首先运行的任务号；
-F: perform test on files only (no directories)
只创建文件，没有目录；
-h: prints help message
输出帮助信息
-i: number of iterations the test will run
测试迭代循环次数；
-I: number of items per tree node
每个树节点包含的项目
-l: last number of tasks on which the test will run
最后运行的任务号
-L: files/dirs created only at leaf level
只在目录树的“叶子”层创建文件/目录；
-n: every task will create/stat/remove # files/dirs per tree
每个任务需要再每棵树中create/stat/remove的文件/目录数；
-N: stride # between neighbor tasks for file/dir stat (local=0)
遍历时指定和相邻任务的跨度
-p: pre-iteration delay (in seconds)
每次迭代之间延时（以秒计算）
-r: only remove files/dirs
删除文件/目录
-R: randomly stat files/dirs (optional seed can be provided)
随机遍历文件/目录 ；
-s: stride between the number of tasks for each test
每次测试的任务数的跨度
-S: shared file access (file only, no directories)
共享文件访问（只针对文件操作）；
-t: time unique working directory overhead
记录特定目录的时间开销
-T: only stat files/dirs
-u: unique working directory for each task
为每个任务指定工作目录；
-v: verbosity (each instance of option increments by one)
-V: verbosity value
-w: number of bytes to write to each file
写到每个文件的字节数
-y: sync file after write completion
再写执行完后同步文件到磁盘（同步写）
-z: depth of hierarchical directory structure
目录树的深度；
NOTES:
* -N allows a "read-your-neighbor" approach by setting stride to
tasks-per-node. Do not use it with -B, as it creates race conditions.
允许"read-your-neighbor" 方法
* -d allows multiple paths for the form '-d fullpath1@fullpath2@fullpath3'
可以指定多个测试路径，'-d fullpath1@fullpath2@fullpath3'
* -B allows each task to time itself. The aggregate results reflect this
change.
允许每个任务对自己进行计时；
* -n and -I cannot be used together. -I specifies the number of files/dirs
created per tree node, whereas the -n specifies the total number of
files/dirs created over an entire tree. When using -n, integer division is
used to determine the number of files/dirs per tree node. (E.g. if -n is
10 and there are 4 tree nodes (z=1 and b=3), there will be 2 files/dirs per
tree node.)
* -R and -T can be used separately. -R merely indicates that if files/dirs
are going to be stat'ed, then they will be stat'ed randomly.
不能同时使用，因为指定了每个树节点的文件/目录数量，而-I指定的是整棵树的文件/目录数量。
Illustration of terminology:
Hierarchical directory structure (tree)
=======
| | (tree node)
=======
/ |
------ | ------
/ |
======= ======= =======
| | | | | | (leaf level)
======= ======= =======
In this example, the tree has a depth of one (z=1) and branching factor of
three (b=3). The node at the top of the tree is the root node. The level
of nodes furthest from the root is the leaf level. All trees created by
mdtest are balanced.
这个例子中，目录树深度为1（z=1）,每个节点分支为3（b=3）.最上面的节点为根节点，
关于openmpi的软件的使用
openmpi是并行的运行程序，配置的时候需要注意下
增加并行的运行节点
[root@lab8105 ~]# vim /etc/openmpi-x86_64/openmpi-default-hostfile
添加
lab8105 slots=1
lab8106 slots=1
特别注意加slots 这个是配置权重的，如果不配置，第一条有默认权重，就无法在数目小时进行均衡操作
检查是否配置成功，np为操作线程数
[root@lab8105 ~]# mpirun --allow-run-as-root -np 2 hostname
lab8105
lab8106
如果想单机执行多进程，可以用-host指定主机
[root@lab8105 ~]# mpirun -host lab8105 --allow-run-as-root -np 2 hostname
单机下多进程测试
2 task on 1 node
[root@lab8105 ~]# mpirun -host lab8105 --allow-run-as-root -np 2 mdtest -I 10 -z 5 -b 2 -d /mnt/zptest/ -t -c 2
-- started at 01/27/2015 21:06:23 --
mdtest-1.9.3 was launched with 2 total task(s) on 1 node(s)
Command line used: mdtest -I 10 -z 5 -b 2 -d /mnt/zptest/ -t -c 2
Path: /mnt/zptest
FS: 6.5 TiB Used FS: 59.2% Inodes: 0.6 Mi Used Inodes: 100.0%
2 tasks, 1260 files/directories
SUMMARY: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
Directory creation: 312.088 312.088 312.088 0.000
Directory stat : 73447.245 73447.245 73447.245 0.000
Directory removal : 255.755 255.755 255.755 0.000
File creation : 638.824 638.824 638.824 0.000
File stat : 86747.366 86747.366 86747.366 0.000
File read : 84434.232 84434.232 84434.232 0.000
File removal : 207.545 207.545 207.545 0.000
Tree creation : 39.062 39.062 39.062 0.000
Tree removal : 46.971 46.971 46.971 0.000
-- finished at 01/27/2015 21:06:43 --
多机并发测试
如下显示的2 task on 2 node
[root@lab8105 ~]# mpirun --allow-run-as-root -np 2 mdtest -I 10 -z 5 -b 2 -d /mnt/zptest/ -t -c 2
-- started at 01/27/2015 21:04:35 --
mdtest-1.9.3 was launched with 2 total task(s) on 2 node(s)
Command line used: mdtest -I 10 -z 5 -b 2 -d /mnt/zptest/ -t -c 2
Path: /mnt/zptest
FS: 6.5 TiB Used FS: 59.2% Inodes: 0.6 Mi Used Inodes: 100.0%
2 tasks, 1260 files/directories
SUMMARY: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
Directory creation: 312.355 312.355 312.355 0.000
Directory stat : 1611.000 1611.000 1611.000 0.000
Directory removal : 127.333 127.333 127.333 0.000
File creation : 309.295 309.295 309.295 0.000
File stat : 113257.534 113257.534 113257.534 0.000
File read : 203458.057 203458.057 203458.057 0.000
File removal : 98.523 98.523 98.523 0.000
Tree creation : 36.566 36.566 36.566 0.000
Tree removal : 20.191 20.191 20.191 0.000
-- finished at 01/27/2015 21:05:12 --
关于目录生成的问题：
这个是 -z 2 -b 3
[root@lab8106 mnt]# ls mdtest2/#test-dir.0/mdtest_tree.0/
mdtest_tree.1 mdtest_tree.2 mdtest_tree.3
[root@lab8106 mnt]# ls mdtest2/#test-dir.0/mdtest_tree.0/mdtest_tree.1/
mdtest_tree.4 mdtest_tree.5 mdtest_tree.6
这个是以这个目录开始的
[root@lab8106 mnt]# ls mdtest2/#test-dir.0/mdtest_tree.0/
注意
最好找一台独立机器做控制端
所有机器的hosts要有相互的hosts信息，并且发起mpirun需要与测试机器免密
注意增加参数-u 这个参数是控制每个任务都有一个自己的工作目录的
time mpirun --allow-run-as-root -np 15 --map-by node mdtest -u -n 1000 -z 1 -b 2 -d /test/2mdsnp15size0 -F -L
-F是指定只测试文件
-L指定只在子树上面创建任务