大数据集群常见问题总结

　　项目将近尾声，上线一切顺利，在开发过程中遇到了不少的问题，趁着空闲时间对项目中遇到的常见问题做一个总结，当作一个笔记，问题如下：

1. java.io.IOException: Could not obtain block: blk_194219614024901469_1100 file=/user/hive/warehouse/src_20180124_log/src_20180124_log

出现这种情况大多是结点断了，没有连接上。检查配置，重新启动服务即可。

 　　2.     java.lang.OutOfMemoryError: Java heap space

出现这种异常，明显是jvm内存不够得原因，要修改所有的datanode的jvm内存大小。Java -Xms1024m -Xmx4096m

一般jvm的最大内存使用应该为总内存大小的一半，我们使用的8G内存，所以设置为4096m，这一值可能依旧不是最优的值。

　　3.     IO写操作出现问题

0-1246359584298, infoPort=50075, ipcPort=50020):Got exception while serving blk_-5911099437886836280_1292 to /172.16.100.165:

java.net.SocketTimeoutException: 480000 millis timeout while waiting for channel to be ready for write. ch : java.nio.channels.SocketChannel[connected local=/

172.16.100.165:5001remote=/172.16.100.165:50930]atorg.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:185)at org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159)

atorg.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198)at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:293)

at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:387)

at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:179)

        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:94)

        at java.lang.Thread.run(Thread.java:619)

It seems there are many reasons that it can timeout, the example given in

HADOOP-3831 is a slow reading client.

连接超时，解决办法：在hadoop-site.xml中设置dfs.datanode.socket.write.timeout=0。

　　4.解决hadoop OutOfMemoryError问题：

<property>

    <name>mapred.child.java.opts</name>

   <value>-Xmx800M -server</value>

</property>

或者：hadoop jar jarfile [main class] -D mapred.child.java.opts=-Xmx800M

　　5.     Hadoop java.io.IOException: Job failed! at org.apache.hadoop.mapred.

JobClient.runJob(JobClient.java:1232) while indexing.

when i use nutch1.0,get this error:

Hadoop java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.

runJob(JobClient.java:1232) while indexing.

解决办法：可以删除conf/log4j.properties，然后可以看到详细的错误报告。如出现的是out of memory，解决办法是在给运行主类org.apache.nutch.crawl.Crawl加上参数：-Xms64m -Xmx512m。

　　6.     Namenode in safe mode

解决方法：执行命令 bin/hadoop dfsadmin -safemode leave

　　7.     java.net.NoRouteToHostException: No route to host

解决方法：sudo /etc/init.d/iptables stop

　　8．更改namenode后，在hive中运行select 依旧指向之前的namenode地址

解决办法：将metastore中的之前出现的namenode地址全部更换为现有的namenode地址

　　9.  ERROR metadata.Hive (Hive.java:getPartitions(499)) - javax.jdo.JDODataStoreException: Required table missing : ""PARTITIONS"" in Catalog "" Schema "". JPOX requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable "org.jpox.autoCreateTables"

原因：就是因为在 hive-default.xml 里把 org.jpox.fixedDatastore 设置成 true 了，应该把值设为false。

　　10.INFO hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException:Bad connect ack with firstBadLink 192.168.1.11:50010

>  INFO hdfs.DFSClient: Abandoning block blk_-8575812198227241296_1001

>  INFO hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException:

Bad connect ack with firstBadLink 192.168.1.16:50010

>  INFO hdfs.DFSClient: Abandoning block blk_-2932256218448902464_1001

>  INFO hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException:

Bad connect ack with firstBadLink 192.168.1.11:50010

> INFO hdfs.DFSClient: Abandoning block blk_-1014449966480421244_1001号的

> INFO hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException:

Bad connect ack with firstBadLink 192.168.1.16:50010

> INFO hdfs.DFSClient: Abandoning block blk_7193173823538206978_1001

>  WARN hdfs.DFSClient: DataStreamer Exception: java.io.IOException: Unable

to create new block.

at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2731)

>at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1996

at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2182)

> WARN hdfs.DFSClient: Error Recovery for block blk_7193173823538206978_1001

bad datanode[2] nodes == null

>  WARN hdfs.DFSClient: Could not get block locations. Source file "/user/umer/8GB_input"

- Aborting...

> put: Bad connect ack with firstBadLink 192.168.1.16:50010

解决方法：

1) '/etc/init.d/iptables stop' -->stopped firewall

2) SELINUX=disabled in '/etc/selinux/config' file.-->disabled selinux

　　11．某次正常运行mapreduce实例时,抛出错误

java.io.IOException: All datanodes xxx.xxx.xxx.xxx:xxx are bad. Aborting…

atorg.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2158)at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1400(DFSClient.java:1735

at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1889)

java.io.IOException: Could not get block locations. Aborting…at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2143)

at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1400(DFSClient.java:1735)

at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1889)

问题原因是linux机器打开了过多的文件导致。解决方法：用命令ulimit -n可以发现linux默认的文件打开数目为1024，修改/ect/security/limit.conf，增加hadoop soft 65535，再重新运行程序（最好所有的datanode都修改），问题解决。

　　12.  bin/hadoop jps后报如下异常：

Exception in thread "main" java.lang.NullPointerException

atsun.jvmstat.perfdata.monitor.protocol.local.LocalVmManager.activeVms(LocalVmManager.java:127)atsun.jvmstat.perfdata.monitor.protocol.local.MonitoredHostProvider.activeVms(MonitoredHostProvider.java:133)at sun.tools.jps.Jps.main(Jps.java:45)

解决方法：系统根目录/tmp文件夹被删除了。重新建立/tmp文件夹即可。

　　13. bin/hive中出现 unable to  create log directory /tmp/

解决办法：系统根目录/tmp文件夹被删除了。重新建立/tmp文件夹即可。

　　14．MySQL报错

[root@localhost mysql]# ./bin/mysqladmin -u root password '123456'

./bin/mysqladmin: connect to server at 'localhost' failed

error: 'Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)'

Check that mysqld is running and that the socket: '/tmp/mysql.sock' exists!

解决不能通过mysql .sock连接MySQL问题 这个问题主要提示是，不能通过 '/tmp/mysql .sock'连到服务器，而php标准配置正是用过'/tmp/mysql .sock'，但是一些mysql 安装方法 将 mysql .sock放在/var/lib/mysql .sock或者其他的什么地方，可以通过修改/etc/my.cnf文件来修正它，打开文件，可以看到如下的配置：

[mysqld]

socket=/var/lib/mysql.sock

改一下就好了，但也会引起其他的问题，如mysql 程序连不上了，再加一点：

[mysql]

socket=/tmp/mysql.sock

或者还可以通过修改my.ini中的配置来使用其他的mysql.sock来连，或者用这样的方法:

ln -s /var/lib/mysql/mysql.sock /tmp/mysql.sock

成功了,就是这样ln -s /var/lib/mysql/mysql.sock /tmp/mysql.sock

　　15. NameNode不能切换

less $BEH_HOME/logs/hadoop/hadoop-hadoop-zkfc-hadoop001.log

日志文件中有错误提醒：Unable to create SSH session

com.jcraft.jsch,JSchException:java.io.FileNotFoundException:~/.ssh/id_rsa (No such file or directory)

解决办法：vim $HADOOP_HOME/etc/hadoop/HDFS-site.xml

修改以下参数：

<property>

    <name>dfs.ha.fencing.ssh.private-key-files</name>

    <value>/home/hadoop/.ssh/id_rsa</value>

    <final>true</final>

    <description/>

</property>