nutch

CREATE DATABASE news DEFAULT CHARACTER SET utf8mb4 DEFAULT COLLATE utf8mb4_unicode_ci;
CREATE TABLE `webpage` (
`id` varchar(767) NOT NULL,
`headers` blob,
`text` mediumtext DEFAULT NULL,
`status` int(11) DEFAULT NULL,
`markers` blob,
`parseStatus` blob,
`modifiedTime` bigint(20) DEFAULT NULL,
`score` float DEFAULT NULL,
`typ` varchar(32) CHARACTER SET latin1 DEFAULT NULL,
`baseUrl` varchar(767) DEFAULT NULL,
`content` longblob,
`title` varchar(2048) DEFAULT NULL,
`reprUrl` varchar(767) DEFAULT NULL,
`fetchInterval` int(11) DEFAULT NULL,
`prevFetchTime` bigint(20) DEFAULT NULL,
`inlinks` mediumblob,
`prevSignature` blob,
`outlinks` mediumblob,
`fetchTime` bigint(20) DEFAULT NULL,
`retriesSinceFetch` int(11) DEFAULT NULL,
`protocolStatus` blob,
`signature` blob,
`metadata` blob,
PRIMARY KEY (`id`)
) ENGINE=InnoDB
ROW_FORMAT=COMPRESSED
DEFAULT CHARSET=utf8mb4;

安装nutch2.1
A、nutch下载地址：http://apache.etoak.com/nutch/2.1/apache-nutch-2.1-src.zip,下载完成后家压缩
B、将nutch的根目录定位${APACHE_NUTCH_HOME}.
C、配置nutch对mysql的支持，修改${APACHE_NUTCH_HOME}/ivy/ivy.xml文件
将这行的注释取消<dependency org=”mysql” name=”mysql-connector-java” rev=”5.1.18″ conf=”*->default”/>
修改${APACHE_NUTCH_HOME}/conf/gora.properties文件，
注释默认存储配置
###############################
# Default SqlStore properties #
###############################
#gora.sqlstore.jdbc.driver=org.hsqldb.jdbc.JDBCDriver
#gora.sqlstore.jdbc.url=jdbc:hsqldb:hsql://localhost/nutchtest
#gora.sqlstore.jdbc.user=sa
#gora.sqlstore.jdbc.password=
新增MySql配置
###############################
# MySql properties #
###############################
gora.sqlstore.jdbc.driver=com.mysql.jdbc.Driver
gora.sqlstore.jdbc.url=jdbc:mysql://localhost:3306/news?useUnicode=true&characterEncoding=utf8&createDatabaseIfNotExist=true&autoReconnect=true&zeroDateTimeBehavior=convertToNull
gora.sqlstore.jdbc.user=root
gora.sqlstore.jdbc.password=root
D、修改${APACHE_NUTCH_HOME}/conf/nutch-site.xml 加入如下代码：
<configuration>
<property>
<name>http.agent.name</name>
<value>my nutch spider</value>
</property>

<property>
<name>http.accept.language</name>
<value>ja-jp,en-us,zh-cn,en-gb,en;q=0.7,*;q=0.3</value>
<description>Value of the “Accept-Language” request header field.
This allows selecting non-English language as default one to retrieve.
It is a useful setting for search engines build for certain national group.
</description>
</property>

<property>
<name>parser.character.encoding.default</name>
<value>utf-8</value>
<description>The character encoding to fall back to when no other information is available</description>
</property>

<property>
<name>storage.data.store.class</name>
<value>org.apache.gora.sql.store.SqlStore</value>
<description>The Gora DataStore class for storing and retrieving data.Currently the following stores are available: ….</description>
</property>
</configuration>
E、使用ant编译 ${APACHE_NUTCH_HOME}
F、设置待抓取的网站
cd ${APACHE_NUTCH_HOME}/runtime/local
mkdir -p urls
echo 'http://www.cpyjy.com/' > urls/seed.txt
G、执行爬行操作： bin/nutch crawl urls -depth 3 -topN 5
执行完在mysql中即可以查看到爬虫抓取的内容

3、安装solr，对nutch抓取的内容进行索引
（注意：参考资料中推荐使用solr4.0版本，4.0的两个版本我都试了，没有成功，所以替换为3.6.2版本）
solr下载地址：http://www.fayea.com/apache-mirror/lucene/solr/3.6.2/apache-solr-3.6.2.zip
A、解压缩下载包，
B、下载 http://nlp.solutions.asia/wp-content/uploads/2012/08/schema.xml替换${APACHE_SOLR_HOME}/example/solr/conf/schema.xml.
C、启动solr:cd ${APACHE_SOLR_HOME}/example
java -jar start.jar
D、在浏览器输入地址http://localhost:8983/solr 测试是否启动成功。
E、另起linux终端，输入如下命令，使solr对nutch抓取内容进行索引。
cd ${APACHE_NUTCH_HOME}/runtime/local/
bin/nutch solrindex http://127.0.0.1:8983/solr/ -reindex

4、测试
在浏览器输入 http://localhost:8983/solr ，看到如下界面：

在文本框中输入 content:nutch 点击查询，便可查处内容中包含nutch关键字的网页

5.乱码处理http://nlp.solutions.asia/?p=180#tab-3