nutch 写一个indexingfilter插件

参考源：http://blog.csdn.net/amuseme_lu/article/details/6780244

1 生成一个与urlfilter-regex类似的包结构

代码路径的生成：http://www.cnblogs.com/i80386/archive/2012/09/04/2670670.html

2

public class MyIndexingFilter  implements IndexingFilter {

    public static final Log LOG = LogFactory.getLog(MyIndexingFilter.class);
    private Configuration conf;
    public void addIndexBackendOptions(Configuration conf) {
        LuceneWriter.addFieldOptions("mt", LuceneWriter.STORE.YES,
                LuceneWriter.INDEX.TOKENIZED, conf);
    }
    private NutchDocument addMyField(NutchDocument doc)  
     {  
        System.out.println("银河系");
        String value="银河系";
        doc.add("mt",value);  //这里我设置了一个固定字段，实际应该从html抽取目标字段
        return doc;  
     }  
    public NutchDocument filter(NutchDocument doc, Parse parse, Text url,
            CrawlDatum datum, Inlinks inlinks) throws IndexingException {
        addMyField(doc);
        return doc;
    }
    public Configuration getConf() {
        return this.conf;
    }
    public void setConf(Configuration arg0) {
        this.conf = arg0;
    }
}

3 生成jar包       build fat jar

4 生成plugin.xml

<plugin
   id="index-myfield"
   name="my Indexing Filter"
   version="1.0.0"
   provider-name="nutch.org">


   <runtime>
      <library name="myfield.jar">
         <export name="*"/>
      </library>
   </runtime>

   <requires>
      <import plugin="nutch-extensionpoints"/>
   </requires>

   <extension id="org.apache.nutch.indexer.myfield"
              name="Nutch My Indexing Filter"
              point="org.apache.nutch.indexer.IndexingFilter">
      <implementation id="MyIndexingFilter"
                      class="org.apache.nutch.indexer.myfield.MyIndexingFilter"/>
   </extension>

</plugin>

5 最后把打好的jar包与plugin.xml放到E:\nutch\src\plugin\index-myfield 文件夹中

6 修改conf\nutch-site.xml

<configuration>
<property>
        <name>searcher.dir</name>
        <value>E:/crawl_2</value>
</property>
    <property>  
      <name>plugin.includes</name>  
      <value>protocol-http|urlfilter-(regex|prefix|my)|parse-(html|tika)|index-(basic|anchor|myfield)|scoring-opic|urlnormalizer-(pass|regex|basic)</value>  
      <description>Regular expression naming plugin directory names to  
      include.  Any plugin not matching this expression is excluded.  
      In any case you need at least include the nutch-extensionpoints plugin. By  
      default Nutch includes crawling just HTML and plain text via HTTP,  
      and basic indexing and search plugins. In order to use HTTPS please enable   
      protocol-httpclient, but be aware of possible intermittent problems with the   
      underlying commons-httpclient library.  
      </description>  
    </property>  
</configuration>

7 启动nutch

8 在solr中检索

9 可以检索到我们需要的字段

注：如果我不是手动打jar放到 index-myfield文件夹中 ，而是直接修改nutch-site.xml 添加了 index-(basic|anchor|myfield)