Linux IO/NFS tunning:
IO Test
=======
dd 测试读性能的命令
# time dd if=/nfsfolder/testfile of=/dev/null bs=1k
dd 测试写性能的命令
time -p dd if=/dev/zero of=test.file bs=1k count=100000
[1]Write
time -p dd if=/dev/zero of=test.file bs=1k count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 25.2433 s, 4.1 MB/s
real 25.24
user 0.03
sys 1.49
time -p dd if=/dev/zero of=test.file bs=1024k count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.477919 s, 219 MB/s
real 0.48
user 0.00
sys 0.11
[2] read file
time dd if=./test.file of=/dev/null bs=1k
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 14.1445 s, 7.2 MB/s
real 0m14.150s
user 0m0.054s
sys 0m1.929s
time dd if=./test.file of=/dev/null bs=1024k
97+1 records in
97+1 records out
102400000 bytes (102 MB) copied, 0.0465829 s, 2.2 GB/s
real 0m0.052s
user 0m0.000s
sys 0m0.037s
time rm -rf test.file
real 0m0.093s
user 0m0.000s
sys 0m0.091s
time rm -rf test.file
real 0m0.101s
user 0m0.000s
sys 0m0.098s
Tools:
======
sar -b 1 10
-b Report I/O and transfer rate statistics. The following values are displayed:
tps
Total number of transfers per second that were issued to physical devices. A
transfer is an I/O request to a physical device. Multiple logical requests can
be combined into a single I/O request to the device. A transfer is of indeu2010
terminate size.
rtps
Total number of read requests per second issued to physical devices.
wtps
Total number of write requests per second issued to physical devices.
bread/s
Total amount of data read from the devices in blocks per second. Blocks are
equivalent to sectors and therefore have a size of 512 bytes.
bwrtn/s
Total amount of data written to devices in blocks per second.
注：每1秒 显示 1次 显示 10次
sar -d -p 3 10 //3s 10 times.
Monitor NFS
1 second interval to monitor nfsstat
nfsstat -cn -Z1
Client nfs v3:
null getattr setattr lookup access readlink
0 0% 777 58% 0 0% 0 0% 111 8% 0 0%
read write create mkdir symlink mknod
0 0% 432 32% 0 0% 0 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
fsstat fsinfo pathconf commit
0 0% 0 0% 0 0% 0 0%
nfsstat -rc
Client rpc stats:
calls retrans authrefrsh
66906770 5 66907434
If retrans value is quite high, meaning that retransmissions were often necessary since the last reboot. This is a clear indication that the number of available NFS kernel threads on the server is insufficient to handle the requests from this client. The default number of threads for rpc.nfsd to start is typically eight threads.
To tell rpc.nfsd to use more kernel threads, the number of threads must be passed as an argument to it. Typically, most distributions will have a file such as /etc/sysconfig/nfs to configure this;
NFS server side:
nfsstat --all
dstat用来监控cpu,disk,network等, 类似vmstat.如下,比较直观.
[root@LMG3ADFBE2 ~] # dstat
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
1 1 98 0 0 0| 244B 66k| 0 0 | 0 0 |2139 2048
0 3 97 0 0 0| 0 0 | 195B 838B| 0 0 |2865 8355
0 1 99 0 0 0| 0 0 |1038B 628B| 0 0 | 723 537
0 2 98 0 0 0| 0 0 | 66B 358B| 0 0 |1884 1000
dstat常用的选项有：
-c 显示cpu使用情况
-d 显示磁盘使用情况
-g, 显示页面数据
-i 启用中断数据
-l 平均负载统计（1分钟，5分钟，15分钟）
-m 显示内存使用信息
-n 显示网络信息
-p enable process stats (runnable,uninterruptible, new)
-r enable I/O request stats(read, write requests)
-s 显示交换分区使用信息
-t 启用时间/日期输出
-y 系统统计(中断上下文切换)
--aio 显示异步io统计数据
--ipc ipc相关的信息
--raw: raw socket
--tcp tcp socket
--udp udp socket
--socket raw, tcp, udp
--unix unix sock
--top-cpu 显示最占用CPU的进程
--top-bio 显示最占用block IO的进程
--top-mem: 显示最耗费内存的进程
--top-io 最占用IO的进程
http://www.softpanorama.org/Net/Application_layer/NFS/nfs_performance_tuning.shtml
关于NFS ac/noac 和nocto
cto / nocto Selects whether to use close-to-open cache coherence semantics.
If neither option is specified (or if cto is specified), the
client uses close-to-open cache coherence semantics. If the nocto
option is specified, the client uses a non-standard heuristic to
determine when files on the server have changed.
Using the nocto option may improve performance for read-only
mounts, but should be used only if the data on the server changes
only occasionally. The DATA AND METADATA COHERENCE section dis-
cusses the behavior of this option in more detail.
设置这个参数如nocto后, 行为表现为.
Server A and Server B mount the same storage via NFS.
Server B nocto, ac, 当A写入一行到一个文件， 在B上看不到A写的内容。知道触发ac的timeout， 即actimo
ac/noac
ac / noac Selects whether the client may cache file attributes. If neither
option is specified (or if ac is specified), the client caches
file attributes.
To improve performance, NFS clients cache file attributes. Every
few seconds, an NFS client checks the serveru2019s version of each
fileu2019s attributes for updates. Changes that occur on the server
in those small intervals remain undetected until the client checks
the server again. The noac option prevents clients from caching
file attributes so that applications can more quickly detect file
changes on the server.
In addition to preventing the client from caching file attributes,
the noac option forces application writes to become synchronous so
that local changes to a file become visible on the server immedi-
ately. That way, other clients can quickly detect recent writes
when they check the fileu2019s attributes.
Using the noac option provides greater cache coherence among NFS
clients accessing the same files, but it extracts a significant
performance penalty. As such, judicious use of file locking is
encouraged instead. The DATA AND METADATA COHERENCE section con-
tains a detailed discussion of these trade-offs.
如果ac enable， 则文件属性及目录属性会在本地cache中（metadata cache）， 可以touch一个新文件来强制刷新这些cache。
如果server A with ac， server B with noac, 在A上向文件写入一行， 则B上可以看到最新的内容。当然如果B上设置了nocto，则看不到。
- Shall noac option on server B trigger the sync operation on server A? Yes. When the client mounted with noac option reads a file from the server, It will always fetch the most recent copy from the NFS server. If the NFS server thinks that a lock has been acquired for the same file by another client (clients acquire locks before performing write), then the NFS server asks that client to forcefully write the data, then the updated copy is provided to the client mounted with noac option.