HDFS-Could not obtain block
MapReduce Total cumulative CPU time: 33 seconds 380 msec
Ended Job = job_201308291142_4635 with errors
Error during job, obtaining debugging information...
Job Tracking URL: http://xxx /jobdetails.jsp?jobid=job_201308291142_4635
Examining task ID: task_201308291142_4635_m_000019 (and more) from job job_201308291142_4635
Examining task ID: task_201308291142_4635_m_000007 m(and more) from job job_201308291142_4635
Examining task ID: task_201308291142_4635_m_000009 (and more) from job job_201308291142_4635
Task with the most failures(5):
-----
Task ID:
task_201308291142_4635_m_000009
URL:
-----
Diagnostic Messages for this Task:
java.io.IOException: java.io.IOException: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1555036314-10.115.5.16-1375773346340:blk_-2678705702538243931_541142 file=/user/hive/warehouse/playtime/dt=20131119/access_pt.log.2013111904.log
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:330)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:246)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:215)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:200)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
- Reson
- Solution
HDFS FILE
- If HDFS block is missing
1. confirm status
Confirm missing block is exit or not.
If missing block is over 1, file is not able to read.
$ hadoop dfsadmin -report
Configured Capacity: 411114887479296 (373.91 TB)
Present Capacity: 411091477784158 (373.89 TB)
DFS Remaining: 411068945908611 (373.87 TB)
DFS Used: 22531875547 (20.98 GB)
DFS Used%: 0.01%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 20 (20 total, 0 dead)
2. detail block file
...
Status: HEALTHY
Total size: 4056908575 B (Total open files size: 3505453 B)
Total dirs: 533
Total files: 15525 (Files currently being written: 2)
Total blocks (validated): 15479 (avg. block size 262091 B) (Total open file blocks (not validated): 2)
Minimally replicated blocks: 15479 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 3.0094967
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 20
Number of racks: 1
FSCK ended at Tue Nov 19 10:17:19 KST 2013 in 351 milliseconds
The filesystem under path '/' is HEALTHY
3. remove corrupted file
.....
.........................Status: HEALTHY
Total size: 4062473881 B (Total open files size: 3505453 B)
Total dirs: 533
Total files: 15525 (Files currently being written: 2)
Total blocks (validated): 15479 (avg. block size 262450 B) (Total open file blocks (not validated): 2)
Minimally replicated blocks: 15479 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 3.0094967
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 20
Number of racks: 1
FSCK ended at Tue Nov 19 10:21:41 KST 2013 in 294 milliseconds
The filesystem under path '/' is HEALTHY
HIVE FILE
- If hive block is missing
alter table drop partition