Pig安装与应用

 

1.  参考说明

参考文档：

 

http://pig.apache.org/docs/r0.17.0/start.html#build

 

 

2.  安装环境说明

2.1.  环境说明

 

CentOS7.4+ Hadoop2.7.5的伪分布式环境

 

主机名	NameNode	SecondaryNameNode	DataNodes
centoshadoop.smartmap.com	192.168.1.80	192.168.1.80	192.168.1.80

 

Hadoop的安装目录为：/opt/hadoop/hadoop-2.7.5

 

3.  安装

 

 

3.1.  Pig下载

 

http://pig.apache.org/releases.html#Download

 

 

[root@server1 ~]# mkdir /opt/mongodb

[root@server1 ~]# chown -R mongodb:mongodb /opt/mongodb/

 

3.2.  Pig解压

 

将下载的pig-0.17.0.tar.gz解压到/opt/hadoop/pig-0.17.0目录下

 

4.  配置

 

4.1.  修改profile文件

vi /etc/profile

 

export PIG_HOME=/opt/hadoop/pig-0.17.0

export PATH=$PATH:$PIG_HOME/bin

 

4.2.  将JDK升级为1.8版本

 

将JDK切换成1.8的版本，并修改所有与JAVA_HOME相关的变量

 

4.3.  修改pig的配置文件

vi /opt/hadoop/pig-0.17.0/conf/pig.properties

 

exectype=mapreduce

 

4.4.  修改mapred-site.xml以启用jobhistory

 

vi /opt/hadoop/hadoop-2.7.5/etc/hadoop/mapred-site.xml

 

<property>

       <name>mapreduce.jobhistory.address</name>                

       <value>192.168.1.80:10020</value>

</property>

 

5.  启动Hadoop

 

5.1.  启动YARN与HDFS

cd /opt/hadoop/hadoop-2.7.5/sbin

 

start-all.sh

 

5.2.  启动historyserver

 

cd /opt/hadoop/hadoop-2.7.5/sbin

 

mr-jobhistory-daemon.sh start historyserver

 

6.  应用Pig工具

 

6.1.  导入文件到HDFS中

 

hadoop fs -mkdir -p /input/ncdc/micro-tab

hadoop fs -copyFromLocal sample.txt /input/ncdc/micro-tab/sample.txt

 

6.2.  启动运行Pig的交互式Shell环境

 

cd /opt/hadoop/pig-0.17.0/bin

 

pig

 

6.3.  运行任务

 

grunt> records = load '/input/ncdc/micro-tab/sample.txt' as (year:chararray, temperature:int, quality:int);

 

grunt> dump records;

 

 

 

6.4.  退出

 

 

grunt> q

 

 

6.5.  显示模式

 

cd /opt/hadoop/pig-0.17.0/bin

 

pig

 

 

grunt> records = LOAD '/input/ncdc/micro-tab/sample.txt' as (year:chararray, temperature:int, quality:int);

 

grunt> DUMP records;

 

grunt> DESCRIBE records

records: {year: chararray,temperature: int,quality: int}

grunt>

 

6.6.  过滤数据

 

grunt> filter_records = FILTER records BY temperature != 9999 AND quality IN (0, 1, 4, 5, 9);

grunt> DUMP filter_records;

 

 

 

6.7.  分组记录

 

grunt> grouped_records = GROUP filter_records BY year;

grunt> DUMP grouped_records;

grunt> DESCRIBE grouped_records;

grouped_records: {group: chararray,filter_records: {(year: chararray,temperature: int,quality: int)}}

grunt>

 

 

 

 

 

6.8.  计算最大值

 

grunt> max_temp = FOREACH grouped_records GENERATE group, MAX(filter_records.temperature);

grunt> DUMP max_temp;

 

 

 

6.9.  查看执行过程

 

grunt> ILLUSTRATE max_temp;