问题1：Cannot run program "/bin/ls": error=11, Resource temporarily unavailable
1 15/04/22 14:46:46 INFO mapred.JobClient: Task Id : attempt_201504221017_0006_r_000077_0, Status : FAILED 2 java.lang.RuntimeException: Error while running command to get file permissions : java.io.IOException: Cannot run program "/bin/ls": error=11, Resource temporarily unavailable 3 at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047) 4 at org.apache.hadoop.util.Shell.runCommand(Shell.java:200) 5 at org.apache.hadoop.util.Shell.run(Shell.java:182) 6 at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375) 7 at org.apache.hadoop.util.Shell.execCommand(Shell.java:461) 8 at org.apache.hadoop.util.Shell.execCommand(Shell.java:444) 9 at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:712) 10 at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:448) 11 at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getOwner(RawLocalFileSystem.java:431) 12 at org.apache.hadoop.mapred.TaskLog.obtainLogDirOwner(TaskLog.java:267) 13 at org.apache.hadoop.mapred.TaskLogsTruncater.truncateLogs(TaskLogsTruncater.java:124) 14 at org.apache.hadoop.mapred.Child$4.run(Child.java:260) 15 at java.security.AccessController.doPrivileged(Native Method) 16 at javax.security.auth.Subject.doAs(Subject.java:415) 17 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) 18 at org.apache.hadoop.mapred.Child.main(Child.java:249) 19 Caused by: java.io.IOException: error=11, Resource temporarily unavailable 20 at java.lang.UNIXProcess.forkAndExec(Native Method) 21 at java.lang.UNIXProcess.<init>(UNIXProcess.java:186) 22 at java.lang.ProcessImpl.start(ProcessImpl.java:130) 23 at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028) 24 ... 15 more
该问题很可能是因为当前提交Hadoop作业的用户能够打开的文件个数限制(主要是slave结点)，可以使用ulimit -n查看当前用户能够打开的文件个数。
参考：http://mail-archives.apache.org/mod_mbox/nutch-user/201312.mbox/%3C1386546180.6104.5.camel@senf.fritz.box%3E
问题2：java.lang.OutOfMemoryError: unable to create new native thread
1 15/04/22 11:08:16 WARN hdfs.DFSClient: DataStreamer Exception: java.lang.OutOfMemoryError: unable to create new native thread 2 at java.lang.Thread.start0(Native Method) 3 at java.lang.Thread.start(Thread.java:714) 4 at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3030)
该问题并不一定是内存不够了，很可能是当前提交Hadoop作业的用户能够创建的进程个数限制(主要是slave结点)，可以使用ulimit -u查看当前用户能够创建的文件个数。
参考：http://www.nosql.se/2011/10/hadoop-tasktracker-java-lang-outofmemoryerror/
如何修改当前用户能够创建的进程个数：
在CentOS中，在vim /etc/security/limits.d/90-nproc.conf文件末尾添加如下语句：
1 * soft nproc 102400 2 * hard nproc 102400
修改完成后需要使用该用户重新登录。(如果使用Xshell这种远程软件，需要关闭该主机远程连接的标签卡，然后重新登录即可)
问题3：
是否