It seems that writes/reads to regular files can't not be made non-blocking. I found the following references for support:
from The Linux Programming Interface: A Linux and UNIX System Programming Handbook:
"--- Nonblocking mode can be used with devices (e.g., terminals and pseudoterminals), pipes, FIFOs, and sockets. (Because file descriptors for pipes and sockets are not obtained using open(), we must enable this flag using the fcntl() F_SETFL operation described in Section 5.3.) O_NONBLOCK is generally ignored for regular files, because the kernel buffer cache ensures that I/O on regular files does not block, as described in Section 13.1. However, O_NONBLOCK does have an effect for regular files when mandatory file locking is employed (Section 55.4). ---"
from Advanced Programming in the UNIX Environment 2nd Ed:
"--- We also said that system calls related to disk I/O are not considered slow, even though the read or write of a disk file can block the caller temporarily. ---"
from http://www.remlab.net/op/nonblock.shtml:
"--- Regular files are always readable and they are also always writeable. This is clearly stated in the relevant POSIX specifications. I cannot stress this enough. Putting a regular file in non-blocking has ABSOLUTELY no effects other than changing one bit in the file flags. Reading from a regular file might take a long time. For instance, if it is located on a busy disk, the I/O scheduler might take so much time that the user will notice the application is frozen. Nevertheless, non-blocking mode will not work. It simply will not work. Checking a file for readability or writeability always succeeds immediately. If the system needs time to perform the I/O operation, it will put the task in non-interruptible sleep from the read or write system call. ---"
When memory is adequately available, reads/writes is performed through kernel buffering.
简单来说，对于普通的磁盘文件来说，可读可写条件总是成立的（不像socket缓冲区空了就没法读，缓冲区满了就没法写），所以设置非阻塞模式并没有意义。
问题在于，虽然系统并不认为读／写普通文件会阻塞，可是实际上由于磁盘可能比较繁忙或者比较慢等因素，导致读／写事件延迟很长，同样带来阻塞的问题，
但这种问题却根本无法避免！