Solr data import 中XML/HTTP 数据源的使用

Solr data import 中XML/HTTP 数据源的使用

参考自：http://wiki.apache.org/solr/DataImportHandler

DataImportHandler 可以通过datasource数据源索引来自于HTTP的数据。包括REST/XML和RSS/ATOM 。

在版本1.4中，推荐使用URLDataSource。

配置示例如下：

Xml代码

1. <dataSource name="b" type="HttpDataSource" baseUrl="http://host:port/" encoding="UTF-8" connectionTimeout="5000" readTimeout="10000"/>  
2. <!-- or in Solr 1.4-->  
3. <dataSource name="a" type="URLDataSource" baseUrl="http://host:port/" encoding="UTF-8" connectionTimeout="5000" readTimeout="10000"/>  

<dataSource name="b" type="HttpDataSource" baseUrl="http://host:port/" encoding="UTF-8" connectionTimeout="5000" readTimeout="10000"/> <!-- or in Solr 1.4--> <dataSource name="a" type="URLDataSource" baseUrl="http://host:port/" encoding="UTF-8" connectionTimeout="5000" readTimeout="10000"/>

 datasource的属性有：

baseUrl(可选): you should use it when the host/port changes between Dev/QA/Prod environments. Using this attribute isolates the changes to be made to the solrconfig.xml

encoding（可选）:定义响应头里面的编码方式。这个属性可以替换掉服务器的默认编码方式。

connectionTimeout(可选)：默认时间是5000ms

readTimeout（可选）：默认是10000ms

下面是一个data-config.xml示例的配置：

这是一个Slashdot RSS feed的例子。

Xml代码

1. <dataConfig>  
2.         <dataSource type="HttpDataSource" />  
3.         <document>  
4.                 <entity name="slashdot"  
5.                         pk="link"  
6.                         url="http://rss.slashdot.org/Slashdot/slashdot"  
7.                         processor="XPathEntityProcessor"  
8.                         forEach="/RDF/channel | /RDF/item"  
9.                         transformer="DateFormatTransformer">  
10.   
11.                         <field column="source"       xpath="/RDF/channel/title"   commonField="true" />  
12.                         <field column="source-link"  xpath="/RDF/channel/link"    commonField="true" />  
13.                         <field column="subject"      xpath="/RDF/channel/subject" commonField="true" />  
14.   
15.                         <field column="title"        xpath="/RDF/item/title" />  
16.                         <field column="link"         xpath="/RDF/item/link" />  
17.                         <field column="description"  xpath="/RDF/item/description" />  
18.                         <field column="creator"      xpath="/RDF/item/creator" />  
19.                         <field column="item-subject" xpath="/RDF/item/subject" />  
20.   
21.                         <field column="slash-department" xpath="/RDF/item/department" />  
22.                         <field column="slash-section"    xpath="/RDF/item/section" />  
23.                         <field column="slash-comments"   xpath="/RDF/item/comments" />  
24.                         <field column="date" xpath="/RDF/item/date" dateTimeFormat="yyyy-MM-dd'T'hh:mm:ss" />  
25.                 </entity>  
26.         </document>  
27. </dataConfig>  

<dataConfig>         <dataSource type="HttpDataSource" />         <document>                 <entity name="slashdot"                         pk="link"                         url="http://rss.slashdot.org/Slashdot/slashdot"                         processor="XPathEntityProcessor"                         forEach="/RDF/channel | /RDF/item"                         transformer="DateFormatTransformer">                          <field column="source"       xpath="/RDF/channel/title"   commonField="true" />                         <field column="source-link"  xpath="/RDF/channel/link"    commonField="true" />                         <field column="subject"      xpath="/RDF/channel/subject" commonField="true" />                          <field column="title"        xpath="/RDF/item/title" />                         <field column="link"         xpath="/RDF/item/link" />                         <field column="description"  xpath="/RDF/item/description" />                         <field column="creator"      xpath="/RDF/item/creator" />                         <field column="item-subject" xpath="/RDF/item/subject" />                          <field column="slash-department" xpath="/RDF/item/department" />                         <field column="slash-section"    xpath="/RDF/item/section" />                         <field column="slash-comments"   xpath="/RDF/item/comments" />                         <field column="date" xpath="/RDF/item/date" dateTimeFormat="yyyy-MM-dd'T'hh:mm:ss" />                 </entity>         </document> </dataConfig> 

 看到这个东西的确有点迷糊，因此下面是我们索引wikipedia的例子：

下面这个data-config.xml是用于索引http://dumps.wikimedia.org/enwiki/20100312/ 这个网页下的pages-articles.xml.bz2文件，我们需要去这个网页下载该文件，解压以后放到conf/data目录下，这个文件不压缩的大小是1.50GB。

Xml代码

1. <dataConfig>  
2.         <dataSource type="FileDataSource" encoding="UTF-8" />  
3.         <document>  
4.         <entity name="page"  
5.                 processor="XPathEntityProcessor"  
6.                 stream="true"  
7.                 forEach="/mediawiki/page/"  
8.                 url="/data/enwiki-20100312-pages-articles.xml"  
9.                 transformer="RegexTransformer,DateFormatTransformer"  
10.                 >  
11.             <field column="id"        xpath="/mediawiki/page/id" />  
12.             <field column="title"     xpath="/mediawiki/page/title" />  
13.             <field column="revision"  xpath="/mediawiki/page/revision/id" />  
14.             <field column="user"      xpath="/mediawiki/page/revision/contributor/username" />  
15.             <field column="userId"    xpath="/mediawiki/page/revision/contributor/id" />  
16.             <field column="text"      xpath="/mediawiki/page/revision/text" />  
17.             <field column="timestamp" xpath="/mediawiki/page/revision/timestamp" dateTimeFormat="yyyy-MM-dd'T'hh:mm:ss'Z'" />  
18.             <field column="$skipDoc"  regex="^#REDIRECT .*" replaceWith="true" sourceColName="text"/>  
19.        </entity>  
20.         </document>  
21. </dataConfig>  

<dataConfig>         <dataSource type="FileDataSource" encoding="UTF-8" />         <document>         <entity name="page"                 processor="XPathEntityProcessor"                 stream="true"                 forEach="/mediawiki/page/"                 url="/data/enwiki-20100312-pages-articles.xml"                 transformer="RegexTransformer,DateFormatTransformer"                 >             <field column="id"        xpath="/mediawiki/page/id" />             <field column="title"     xpath="/mediawiki/page/title" />             <field column="revision"  xpath="/mediawiki/page/revision/id" />             <field column="user"      xpath="/mediawiki/page/revision/contributor/username" />             <field column="userId"    xpath="/mediawiki/page/revision/contributor/id" />             <field column="text"      xpath="/mediawiki/page/revision/text" />             <field column="timestamp" xpath="/mediawiki/page/revision/timestamp" dateTimeFormat="yyyy-MM-dd'T'hh:mm:ss'Z'" />             <field column="$skipDoc"  regex="^#REDIRECT .*" replaceWith="true" sourceColName="text"/>        </entity>         </document> </dataConfig> 

 注意一下上面的url最好写绝对路径，我的相对路径老是出错-_-。

相关的schema.xml文件如下：

Xml代码

1. <field name="id"        type="integer" indexed="true" stored="true" required="true"/>  
2. <field name="title"     type="string"  indexed="true" stored="false"/>  
3. <field name="revision"  type="sint"    indexed="true" stored="true"/>  
4. <field name="user"      type="string"  indexed="true" stored="true"/>  
5. <field name="userId"    type="integer" indexed="true" stored="true"/>  
6. <field name="text"      type="text"    indexed="true" stored="false"/>  
7. <field name="timestamp" type="date"    indexed="true" stored="true"/>  
8. <field name="titleText" type="text"    indexed="true" stored="true"/>  
9. ...  
10. <uniqueKey>id</uniqueKey>  
11. <copyField source="title" dest="titleText"/>  

<field name="id"        type="integer" indexed="true" stored="true" required="true"/> <field name="title"     type="string"  indexed="true" stored="false"/> <field name="revision"  type="sint"    indexed="true" stored="true"/> <field name="user"      type="string"  indexed="true" stored="true"/> <field name="userId"    type="integer" indexed="true" stored="true"/> <field name="text"      type="text"    indexed="true" stored="false"/> <field name="timestamp" type="date"    indexed="true" stored="true"/> <field name="titleText" type="text"    indexed="true" stored="true"/> ... <uniqueKey>id</uniqueKey> <copyField source="title" dest="titleText"/> 

 下面会花费数小时的时间去索引数据，并且内存使用率将持续最高，请注意很多wikipedia的文章都仅仅是指向其它文章$skipDoc（1.4）的使用可以使得这些文章被忽略，当然前提是正则表达式匹配。

请注意：由于目前唯一支持delta import的EntityProcessor是SqlEntityProcessor，而XPathEntityProcessor并现在没有实现delta import。如果你想实现这些方法的话，你需要修改EntityProcessor.java