参考http://wiki.apache.org/nutch/RunNutchInEclipse
一、环境准备
1、下载nutch2.3源代码
- wget http://mirror.bit.edu.cn/apache/nutch/2.3/apache-nutch-2.3-src.tar.gz
- svn co https://svn.apache.org/repos/asf/nutch/branches/2.x
2、选择使用的数据库类型，以hbase为例
在conf/nutch-site.xml中增加以下属性：
- <property>
- <name>storage.data.store.class</name>
- <value>org.apache.gora.hbase.store.HBaseStore</value>
- <description>Default class for storing data</description>
- </property>
3、在ivy/ivy.xml中增加与hbase相关的依赖项，此项本已存在，但被注释掉，将注释去掉即可
- <dependency org="org.apache.gora" name="gora-hbase" rev="0.5" conf="*->default” />
4、在nutch.xml中增加以下3个属性
- <property>
- <name>http.agent.name</name>
- <value>My Nutch Spider</value>
- </property>
- <property>
- <name>http.robots.agents</name>
- <value>none</value>
- </property>
- <property>
- <name>plugin.folders</name>
- <value>/Users/liaoliuqing/0_Search/1_Nutch/1_Official/apache-nutch-2.3/build/plugins</value>
- </property>
5、执行ant eclipse
二、导入project
1、导入project
三、运行程序
1、Run as ----> Run configuration，选择project与主类
2、填写参数
/Users/liaoliuqing/Downloads/seed.txt
-Dhadoop.log.dir=logs -Dhadoop.log.file=hadoop.log
3、点击run，输出结果如下：
InjectorJob: starting at 2015-01-28 16:27:43
InjectorJob: Injecting urlDir: /Users/liaoliuqing/Downloads/seed.txt
InjectorJob: Using class org.apache.gora.hbase.store.HBaseStore as the Gora storage class.
InjectorJob: total number of urls rejected by filters: 0
InjectorJob: total number of urls injected after normalization and filtering: 1
Injector: finished at 2015-01-28 16:27:47, elapsed: 00:00:04
注意，在运行程序前，本机需要先启动hbase。
4、查看hbase中的数据
- hbase(main):003:0> scan 'webpage'
- ROW COLUMN+CELL
- com.163.www:http/ column=f:fi, timestamp=1422433667377, value=x00'x8Dx00
- com.163.www:http/ column=f:ts, timestamp=1422433667377, value=x00x00x01K/xA7:x14
- com.163.www:http/ column=mk:_injmrk_, timestamp=1422433667377, value=y
- com.163.www:http/ column=mk:dist, timestamp=1422433667377, value=0
- com.163.www:http/ column=mtdt:_csh_, timestamp=1422433667377, value=?x80x00x00
- com.163.www:http/ column=s:s, timestamp=1422433667377, value=?x80x00x00
- 1 row(s) in 0.2970 seconds