hadoopStreamming 编程 Angels

     熟悉hadoop作业提交的人，只要明白streaming的参数就可以学会提交了，streaming提交作业比较灵活，支持多种语言，但是streaming有个缺陷就是，其封装的参数涉及到mapreduce类的就会默认其继承自org.apache.hadoop.mapred包中的类，因此继承自mapreduce包中的类不可用，但是有一个方法可以解决这个问题，就是将参数，通过-jobconf prop=value 的方式传进去。 这里的prop的名字必须是hadoop job file中那个名字。具体查看源代码。

下面一片文章很好的展示了 stream 提交 python和c语言写的作业，可作为初学参考：

作者：马士华 发表于：2008-03-05 12:51 最后更新于：2008-03-25 11:18
版权声明：可以任意转载，转载时请务必以超链接形式标明文章原始出处和作者信息。
http://www.hadoop.org.cn/hadoop/hadoop-streaming/

---

Michael G. Noll在他的Blog中提到如何在Hadoop中用Python编写MapReduce程序，韩国的gogamza在其Bolg中也提到如何用C编写MapReduce程序（我稍微修改了一下原程序,因为他的Map对单词切分使用tab键）。我合并他们两人的文章，也让国内的Hadoop用户能够使用别的语言来编写MapReduce程序。

首先您得配好您的Hadoop集群，这方面的介绍网上比较多，这儿给个链接（Hadoop学习笔记二 安装部署）。Hadoop Streaming帮 助我们用非Java的编程语言使用MapReduce，Streaming用STDIN (标准输入)和STDOUT (标准输出)来和我们编写的Map和Reduce进行数据的交换数据。任何能够使用STDIN和STDOUT都可以用来编写MapReduce程序，比如 我们用Python的sys.stdin和sys.stdout，或者是C中的stdin和stdout。

我们还是使用Hadoop的例子WordCount来 做示范如何编写MapReduce，在WordCount的例子中我们要解决计算在一批文档中每一个单词的出现频率。首先我们在Map程序中会接受到这批 文档每一行的数据，然后我们编写的Map程序把这一行按空格切开成一个数组。并对这个数组遍历按" 1"用标准的输出输出来，代表这个单词出现了一次。在Reduce中我们来统计单词的出现频率。

 

Python Code

Map: mapper.py

 
#!/usr/bin/env python
 
import sys
 
# maps words to their counts
word2count = {}
 
# input comes from STDIN (standard input)
for line in sys.stdin:
    # remove leading and trailing whitespace
    line = line.strip()
    # split the line into words while removing any empty strings
    words = filter(lambda word: word, line.split())
    # increase counters
    for word in words:
        # write the results to STDOUT (standard output);
        # what we output here will be the input for the
        # Reduce step, i.e. the input for reducer.py
        #
        # tab-delimited; the trivial word count is 1
        print '%s\t%s' % (word, 1)
 

Reduce: reducer.py

 
#!/usr/bin/env python
 
from operator import itemgetter
import sys
 
# maps words to their counts
word2count = {}
 
# input comes from STDIN
for line in sys.stdin:
    # remove leading and trailing whitespace
    line = line.strip()
 
    # parse the input we got from mapper.py
    word, count = line.split()
    # convert count (currently a string) to int
    try:
        count = int(count)
        word2count[word] = word2count.get(word, 0) + count
    except ValueError:
        # count was not a number, so silently
        # ignore/discard this line
        pass
 
# sort the words lexigraphically;
#
# this step is NOT required, we just do it so that our
# final output will look more like the official Hadoop
# word count examples
sorted_word2count = sorted(word2count.items(), key=itemgetter(0))
 
# write the results to STDOUT (standard output)
for word, count in sorted_word2count:
    print '%s\t%s'% (word, count)
 

C Code

Map: Mapper.c

 
#include 
#include 
#include 
#include 
 
#define BUF_SIZE        2048
#define DELIM   "\n"
 
int main(int argc, char *argv[]){
     char buffer[BUF_SIZE];
     while(fgets(buffer, BUF_SIZE - 1, stdin)){
            int len = strlen(buffer);
            if(buffer[len-1] == '\n')
             buffer[len-1] = 0;
 
            char *querys  = index(buffer, ' ');
            char *query = NULL;
            if(querys == NULL) continue;
            querys += 1; /*  not to include '\t' */
 
            query = strtok(buffer, " ");
            while(query){
                   printf("%s\t1\n", query);
                   query = strtok(NULL, " ");
            }
     }
     return 0;
}
h>h>h>h>

Reduce: Reducer.c

 
#include 
#include 
#include 
#include 
 
#define BUFFER_SIZE     1024
#define DELIM   "\t"
 
int main(int argc, char *argv[]){
   char strLastKey[BUFFER_SIZE];
   char strLine[BUFFER_SIZE];
   int count = 0;
 
   *strLastKey = '\0';
   *strLine = '\0';
 
   while( fgets(strLine, BUFFER_SIZE - 1, stdin) ){
          char *strCurrKey = NULL;
          char *strCurrNum = NULL;
 
          strCurrKey  = strtok(strLine, DELIM);
          strCurrNum = strtok(NULL, DELIM); /* necessary to check error but.... */
 
          if( strLastKey[0] == '\0'){
                 strcpy(strLastKey, strCurrKey);
          }
 
          if(strcmp(strCurrKey, strLastKey)){
                printf("%s\t%d\n", strLastKey, count);
                count = atoi(strCurrNum);
          }else{
                 count += atoi(strCurrNum);
          }
          strcpy(strLastKey, strCurrKey);
 
   }
   printf("%s\t%d\n", strLastKey, count); /* flush the count */
   return 0;
}
h>h>h>h>
 

首先我们调试一下源码：

chmod +x mapper.py
chmod +x reducer.py
echo "foo foo quux labs foo bar quux" | ./mapper.py | ./reducer.py
bar     1
foo     3
labs    1
quux    2
g++ Mapper.c -o Mapper
g++ Reducer.c -o Reducer
chmod +x Mapper
chmod +x Reducer
echo "foo foo quux labs foo bar quux" | ./Mapper | ./Reducer
bar     1
foo     2
labs    1
quux    1
foo     1
quux    1

你可能看到C的输出和Python的不一样,因为Python是把他放在词典里了.我们在Hadoop时,会对这进行排序,然后相同的单词会连续在标准输出中输出.

在Hadoop中运行程序

首先我们要下载我们的测试文档wget http://www.gutenberg.org/dirs/etext04/7ldvc10.txt.我们把文档存放在/tmp/doc这个目录下.拷贝测试文档到HDFS中.

bin/hadoop dfs -copyFromLocal /tmp/doc doc

运行 bin/hadoop dfs -ls doc 看看拷贝是否成功.接下来我们运行我们的MapReduce的Job.

bin/hadoop jar /home/hadoop/contrib/hadoop-0.15.1-streaming.jar  -mapper /home/hadoop/Mapper\
-reducer /home/hadoop/Reducer  -input doc/* -output c-output -jobconf mapred.reduce.tasks=1
additionalConfSpec_:null
null=@@@userJobConfProps_.get(stream.shipped.hadoopstreaming
packageJobJar: [] [/home/msh/hadoop-0.15.1/contrib/hadoop-0.15.1-streaming.jar] /tmp/streamjob60816.jar tmpDir=null
08/03/04 19:03:13 INFO mapred.FileInputFormat: Total input paths to process : 1
08/03/04 19:03:13 INFO streaming.StreamJob: getLocalDirs(): [/home/msh/data/filesystem/mapred/local]
08/03/04 19:03:13 INFO streaming.StreamJob: Running job: job_200803031752_0003
08/03/04 19:03:13 INFO streaming.StreamJob: To kill this job, run:
08/03/04 19:03:13 INFO streaming.StreamJob: /home/msh/hadoop/bin/../bin/hadoop job  -Dmapred.job.tracker=192.168.2.92:9001 -kill job_200803031752_0003
08/03/04 19:03:13 INFO streaming.StreamJob: Tracking URL: http://hadoop-master:50030/jobdetails.jsp?jobid=job_200803031752_0003
08/03/04 19:03:14 INFO streaming.StreamJob:  map 0%  reduce 0%
08/03/04 19:03:15 INFO streaming.StreamJob:  map 33%  reduce 0%
08/03/04 19:03:16 INFO streaming.StreamJob:  map 100%  reduce 0%
08/03/04 19:03:19 INFO streaming.StreamJob:  map 100%  reduce 100%
08/03/04 19:03:19 INFO streaming.StreamJob: Job complete: job_200803031752_0003
08/03/04 19:03:19 INFO streaming.StreamJob: Output: c-output
bin/hadoop jar /home/hadoop/contrib/hadoop-0.15.1-streaming.jar  -mapper /home/hadoop/mapper.py\
-reducer /home/hadoop/reducer.py  -input doc/* -output python-output -jobconf mapred.reduce.tasks=1
additionalConfSpec_:null
null=@@@userJobConfProps_.get(stream.shipped.hadoopstreaming
packageJobJar: [] [/home/hadoop/hadoop-0.15.1/contrib/hadoop-0.15.1-streaming.jar] /tmp/streamjob26099.jar tmpDir=null
08/03/04 19:05:40 INFO mapred.FileInputFormat: Total input paths to process : 1
08/03/04 19:05:41 INFO streaming.StreamJob: getLocalDirs(): [/home/msh/data/filesystem/mapred/local]
08/03/04 19:05:41 INFO streaming.StreamJob: Running job: job_200803031752_0004
08/03/04 19:05:41 INFO streaming.StreamJob: To kill this job, run:
08/03/04 19:05:41 INFO streaming.StreamJob: /home/msh/hadoop/bin/../bin/hadoop job  -Dmapred.job.tracker=192.168.2.92:9001 -kill job_200803031752_0004
08/03/04 19:05:41 INFO streaming.StreamJob: Tracking URL: http://hadoop-master:50030/jobdetails.jsp?jobid=job_200803031752_0004
08/03/04 19:05:42 INFO streaming.StreamJob:  map 0%  reduce 0%
08/03/04 19:05:48 INFO streaming.StreamJob:  map 33%  reduce 0%
08/03/04 19:05:49 INFO streaming.StreamJob:  map 100%  reduce 0%
08/03/04 19:05:52 INFO streaming.StreamJob:  map 100%  reduce 100%
08/03/04 19:05:52 INFO streaming.StreamJob: Job complete: job_200803031752_0004
08/03/04 19:05:52 INFO streaming.StreamJob: Output: python-output

当Job提交后我们还能够在web的界面http://localhost:50030/看到每一个工作的运行情况。

当Job工作完成后我们能够在c-output和python-output看到一些文件

bin/hadoop dfs -ls c-output

输入下面的命令我们能够看到我们运行完MapReduce的结果

bin/hadoop dfs -cat c-output/part-00000

用Hadoop Streaming运行MapReduce会比较用Java的代码要慢,因为有两方面的原因:

• 使用 Java API >> C Streaming >> Perl Streaming 这样的一个流程运行会阻塞IO.
• 不像Java在运行Map后输出结果有一定数量的结果集就启动Reduce的程序,用Streaming要等到所有的Map都运行完毕后才启动Reduce

如果用Python编写MapReduce的话，另一个可选的是使用Jython来转编译Pyhton为Java的原生码．另外对于C程序员更好的选择是使用Hadoop新的C++ MapReduce API Pipes来编写．不管怎样，毕竟Hadoop提供了一种不使用Java来进行分布式运算的方法．

下面是从http://www.lunchpauze.com/2007/10/writing-hadoop-mapreduce-program-in-php.html页面中摘下的用php编写的MapReduce程序,供php程序员参考：
Map: mapper.php

 
#!/usr/bin/php

 
$word2count = array();
 
// input comes from STDIN (standard input)
while (($line = fgets(STDIN)) !== false) {
   // remove leading and trailing whitespace and lowercase
   $line = strtolower(trim($line));
   // split the line into words while removing any empty string
   $words = preg_split('/\W/', $line, 0, PREG_SPLIT_NO_EMPTY);
   // increase counters
   foreach ($words as $word) {
       $word2count[$word] += 1;
   }
}
 
// write the results to STDOUT (standard output)
// what we output here will be the input for the
// Reduce step, i.e. the input for reducer.py
foreach ($word2count as $word => $count) {
   // tab-delimited
   echo $word, chr(9), $count, PHP_EOL;
}
 
?>
 

Reduce: mapper.php

 
#!/usr/bin/php

 
$word2count = array();
 
// input comes from STDIN
while (($line = fgets(STDIN)) !== false) {
    // remove leading and trailing whitespace
    $line = trim($line);
    // parse the input we got from mapper.php
    list($word, $count) = explode(chr(9), $line);
    // convert count (currently a string) to int
    $count = intval($count);
    // sum counts
    if ($count > 0) $word2count[$word] += $count;
}
 
// sort the words lexigraphically
//
// this set is NOT required, we just do it so that our
// final output will look more like the official Hadoop
// word count examples
ksort($word2count);
 
// write the results to STDOUT (standard output)
foreach ($word2count as $word => $count) {
    echo $word, chr(9), $count, PHP_EOL;
}
 
?>