前面我们学习了MapReduce编程思想和编程示例，那么本节课程同学们一起操练操练，动手完成下面的项目。
项目需求
一本英文书籍包含成千上万个单词或者短语，现在我们需要在大量的单词中，找出相同字母组成的所有anagrams(字谜)。
数据集
下面是一本英文书籍截取的一部分单词内容。猛戳此链接下载数据集
initiate
initiated
initiates
initiating
initiation
initiations
initiative
initiatives
initiator
initiators
initiatory
inject
injectant
injected
injecting
injection
injections
injector
injectors
injects
思路分析
基于以上需求，我们通过以下几步完成：
1、在 Map 阶段，对每个word（单词）按字母进行排序生成sortedWord，然后输出key/value键值对(sortedWord,word)。
2、在 Reduce 阶段，统计出每组相同字母组成的所有anagrams(字谜)。
数据处理示意流程
在下面单词中，找出相同字母组成的字谜。
cat
tar
bar
act
rat
第一步：经过 map 阶段处理
< act cat >
< art tar>
< abr bar>
< act act>
< art rat>
第二步：经过 reduce 阶段处理
< abr bar>
< act cat,act>
< art tar,rat>
程序开发
1、编写程序执行主类：AnagramMain
package com.hadoop.test; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.conf.Configured; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.util.Tool; import org.apache.hadoop.util.ToolRunner; public class AnagramMain extends Configured implements Tool{ @SuppressWarnings( "deprecation") @Override public int run(String[] args) throws Exception { Configuration conf = new Configuration(); //删除已经存在的输出目录 Path mypath = new Path(args[1]); FileSystem hdfs = mypath.getFileSystem(conf); if (hdfs.isDirectory(mypath)) { hdfs.delete(mypath, true); } Job job = new Job(conf, "testAnagram"); job.setJarByClass(AnagramMain. class); //设置主类 job.setMapperClass(AnagramMapper. class); //Mapper job.setMapOutputKeyClass(Text. class); job.setMapOutputValueClass(Text. class); job.setReducerClass(AnagramReducer. class); //Reducer job.setOutputKeyClass(Text. class); job.setOutputValueClass(Text. class); FileInputFormat.addInputPath(job, new Path(args[0])); //设置输入路径 FileOutputFormat. setOutputPath(job, new Path(args[1])); //设置输出路径 job.waitForCompletion( true); return 0; } public static void main(String[] args) throws Exception{ String[] args0 = { "hdfs://cloud004:9000/anagram/anagram.txt" , "hdfs://cloud004:9000/anagram/output"}; int ec = ToolRunner.run( new Configuration(), new AnagramMain(), args0); System. exit(ec); } }
2、编写Mapper：AnagramMapper
package com.hadoop.test; import java.io.IOException; import java.util.Arrays; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Mapper; public class AnagramMapper extends Mapper< Object, Text, Text, Text> { private Text sortedText = new Text(); private Text orginalText = new Text(); public void map(Object key, Text value, Context context) throws IOException, InterruptedException { String word = value.toString(); char[] wordChars = word.toCharArray();//单词转化为字符数组 Arrays.sort(wordChars);//对字符数组按字母排序 String sortedWord = new String(wordChars);//字符数组转化为字符串 sortedText.set(sortedWord);//设置输出key的值 orginalText.set(word);//设置输出value的值 context.write( sortedText, orginalText );//map输出 } }
3、编写Reducer：AnagramReducer
package com.hadoop.test; import java.io.IOException; import java.util.StringTokenizer; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Reducer; public class AnagramReducer extends Reducer< Text, Text, Text, Text> { private Text outputKey = new Text(); private Text outputValue = new Text(); public void reduce(Text anagramKey, Iterable< Text> anagramValues, Context context) throws IOException, InterruptedException { String output = ""; //对相同字母组成的单词，使用 ~ 符号进行拼接 for(Text anagam:anagramValues){ if(!output.equals("")){ output = output + "~" ; } output = output + anagam.toString() ; } StringTokenizer outputTokenizer = new StringTokenizer(output,"~" ); //输出anagrams（字谜）大于2的结果 if(outputTokenizer.countTokens()>=2) { output = output.replace( "~", ","); outputKey.set(anagramKey.toString());//设置key的值 outputValue.set(output);//设置value的值 context.write( outputKey, outputValue);//reduce } } }
编译和执行MapReduce作业
1、将项目编译和打包为anagram.jar，使用 SSH 客户端将 anagram.jar上传至hadoop的/home/hadoop/djt目录下。
2、使用cd /home/hadoop/djt 切换到当前目录，通过命令行执行任务。
hadoop jar anagram.jar com.hadoop.test.AnagramMain
查看运行结果
任务的最终结果输出到 HDFS ，使用如下命令查看结果。
[hadoop@cloud004 hadoop-2.2.0-x64]$ hadoop fs -cat /anagram/output/part-r-00000
部分结果集如下所示。
cehors cosher,chores,ochres,ochers
cehorst troches,hectors,torches
cehort troche,hector
cehortu toucher,couther,retouch
cehoss coshes,choses
cehrt chert,retch
cehstu chutes,tusche
cehsty chesty,scythe
ceht etch,tech
ceiijstu jesuitic,juiciest
ceiikst ickiest,ekistic
ceiilnos isocline,silicone
ceiilnoss isoclines,silicones
ceiimmnoorss commissioner,recommission
ceiimmnoorsss recommissions,commissioners
ceiimorst isometric,eroticism
ceiimost semiotic,comities
ceiinnopst inceptions,inspection
ceiinrsstu scrutinies,scrutinise
ceiinrst citrines,crinites,inciters
ceiinrt citrine,inciter
ceiinss iciness,incises
ceiintz citizen,zincite
ceiist iciest,cities
ceikln nickel,nickle
ceiklnr crinkle,clinker
ceiklnrs clinkers,crinkles
ceiklns nickels,nickles
ceiklrs slicker,lickers
ceiklrsst sticklers,strickles
ceiklrst trickles,ticklers,stickler
ceiklrt tickler,trickle
ceiklsst slickest,stickles
ceiklst keltics,stickle,tickles
ceiklt tickle,keltic
ceiknrs nickers,snicker
ceikorr rockier,corkier
ceikorst stockier,corkiest,rockiest
ceikpst skeptic,pickets
ceikrst rickets,tickers,sticker
ceil lice,ceil
ceilmop compile,polemic
ceilmopr compiler,complier
ceilmoprs compliers,compilers
ceilmops polemics,complies,compiles
ceilnoos colonise,colonies
ceilnors incloser,licensor
ceilnorss inclosers,licensors