1. 常见问题
(1)hostname设置问题。vi /etc/sysconfig/network
(2)集群/etc/hosts没有统一。
(3)yarn slave需要单独启动。./sbin/yarn-daemon.sh start resourcemanager
(4)namenode格式化需要逐个先启动journalnode。
(5)kerberos主从数据同步时获取不到ticket,一般和设置了多个hosts有关。建议hosts配置和hostname只配置一个。
(6)jsvc通过yum源安装,启动datanode会报错找不到org/apache/commons/daemon/support。改成手动编译安装后就OK了。
2. 测试
[root@master1 work]# hadoop jar wordcount-0.0.1-SNAPSHOT.jar com.javacodegeeks.examples.wordcount.WordCount hdfs://master1:9000/Input.txt output.txt 17/09/04 16:53:09 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 17/09/04 16:53:10 INFO input.FileInputFormat: Total input paths to process : 1 17/09/04 16:53:11 INFO mapreduce.JobSubmitter: number of splits:1 17/09/04 16:53:11 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1504514855214_0001 17/09/04 16:53:11 INFO impl.YarnClientImpl: Submitted application application_1504514855214_0001 17/09/04 16:53:11 INFO mapreduce.Job: The url to track the job: http://master1:8088/proxy/application_1504514855214_0001/ 17/09/04 16:53:11 INFO mapreduce.Job: Running job: job_1504514855214_0001 17/09/04 16:53:19 INFO mapreduce.Job: Job job_1504514855214_0001 running in uber mode : false 17/09/04 16:53:19 INFO mapreduce.Job: map 0% reduce 0% 17/09/04 16:53:26 INFO mapreduce.Job: map 100% reduce 0% 17/09/04 16:53:32 INFO mapreduce.Job: map 100% reduce 100% 17/09/04 16:53:32 INFO mapreduce.Job: Job job_1504514855214_0001 completed successfully 17/09/04 16:53:32 INFO mapreduce.Job: Counters: 49 File System Counters FILE: Number of bytes read=676 FILE: Number of bytes written=248505 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=416 HDFS: Number of bytes written=339 HDFS: Number of read operations=6 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=3592 Total time spent by all reduces in occupied slots (ms)=3048 Total time spent by all map tasks (ms)=3592 Total time spent by all reduce tasks (ms)=3048 Total vcore-milliseconds taken by all map tasks=3592 Total vcore-milliseconds taken by all reduce tasks=3048 Total megabyte-milliseconds taken by all map tasks=3678208 Total megabyte-milliseconds taken by all reduce tasks=3121152 Map-Reduce Framework Map input records=4 Map output records=58 Map output bytes=554 Map output materialized bytes=676 Input split bytes=94 Combine input records=0 Combine output records=0 Reduce input groups=43 Reduce shuffle bytes=676 Reduce input records=58 Reduce output records=43 Spilled Records=116 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=186 CPU time spent (ms)=1910 Physical memory (bytes) snapshot=442167296 Virtual memory (bytes) snapshot=4297965568 Total committed heap usage (bytes)=342884352 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=322 File Output Format Counters Bytes Written=339 Job was successful