Analyzer原理

【常用分词器】

• SimpleAnalyzer
• StopAnalyzer
• WhitespaceAnalyzer
• StandardAnalyze

【TokenStream】

she is a student ==〉TokenStream

 TokenStream有2个实现类。Tokenizer、TokenFilter

1)  Tokenizer

将数据进行分割形成一定的语汇(所谓语汇是指一个一个独立的词语。)。最终结果将形成TokenStream。

2) TokenFilter

按照规则对语汇进行过滤。如：StopFilter可以对停用词进行过滤。

3) 执行过程

【存储方式】

 

【应用TokenStream】

    /**
     * 使用TokenStream进行分词
     * @param str
     * @param analyzer
     */
    public static void displayTokenStream(String str, Analyzer analyzer){
        try {
            //通过Analayer获取TokenStream
            //toenStream("域名称或文件名"，输入流对象)
            TokenStream stream = analyzer.tokenStream("content", new StringReader(str));
            
            //向流中添加一个属性
            //容器，存储每次分词所对应的语汇内容
            CharTermAttribute charAttr = stream.addAttribute(CharTermAttribute.class);
            
            //通过循环语句读取语汇的内容
            while(stream.incrementToken()){
                System.out.print("[" + charAttr + "]  ");
            }
            System.out.println();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    private Version version = Version.LUCENE_35;
    
    /**
     * 测试TokenStream(英文内容)
     */
    @Test
    public void test01(){
        String str = "I'm come from Hanlin，I love Hanlin";
        System.out.println("str = " + str);
        System.out.println("====================================");
        
        //创建Analyzer对象
        Analyzer a1 = new SimpleAnalyzer(version);
        Analyzer a2 = new StopAnalyzer(version);
        Analyzer a3 = new WhitespaceAnalyzer(version);
        Analyzer a4 = new StandardAnalyzer(version);
        
        //测试TokenStream
        AnalyzerUtil.displayTokenStream(str, a1);
        AnalyzerUtil.displayTokenStream(str, a2);
        AnalyzerUtil.displayTokenStream(str, a3);
        AnalyzerUtil.displayTokenStream(str, a4);
    }

    /**
     * 测试TokenStream(中文内容)
     */
    @Test
    public void test02(){
        String str = "我来自翰林，我爱翰林";
        System.out.println("str = " + str);
        System.out.println("====================================");
        
        //创建Analyzer对象
        Analyzer a1 = new SimpleAnalyzer(version);
        Analyzer a2 = new StopAnalyzer(version);
        Analyzer a3 = new WhitespaceAnalyzer(version);
        Analyzer a4 = new StandardAnalyzer(version);
        
        //测试TokenStream
        AnalyzerUtil.displayTokenStream(str, a1);
        AnalyzerUtil.displayTokenStream(str, a2);
        AnalyzerUtil.displayTokenStream(str, a3);
        AnalyzerUtil.displayTokenStream(str, a4);
    }

TokenStream可以读取到分词内容.

【Attribute】

    /**
     * 显示语汇的基本属性
     * @param str
     * @param anlyzer
     */
    public static void displayAttributes(String str, Analyzer anlyzer){
        try {
            //获取TokenStream对象
            TokenStream stream = anlyzer.tokenStream("content", new StringReader(str));
            
            //PositionIncrementAttribute ：存储了语汇之间的位置增量
            //添加PositionIncrementAttribute属性
            PositionIncrementAttribute positionAttr = stream.addAttribute(PositionIncrementAttribute.class);
            
            //添加CharTermAttrbute
            CharTermAttribute charAttr = stream.addAttribute(CharTermAttribute.class);
            //OffsetAttribute：获取语汇的偏移数据
            OffsetAttribute offsetAttr = stream.addAttribute(OffsetAttribute.class);
            
            //语汇的分词方式类型（了解）
            TypeAttribute typeAttr = stream.addAttribute(TypeAttribute.class);
            
            //遍历每一个语汇
            while(stream.incrementToken()){
                System.out.print(positionAttr.getPositionIncrement() + "、");
                System.out.print("[" + charAttr + "  : " + offsetAttr.startOffset() + "~" + offsetAttr.endOffset()+ "(" + typeAttr.type()+ "）] " );
            }
            System.out.println();
            
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    /**
     * 测试属性的应用
     */
    @Test
    public void test03(){
        String str = "I'm come from Hanlin，I love Hanlin";
        System.out.println("str = " + str);
        System.out.println("====================================");
        
        //创建Analyzer对象
        Analyzer a1 = new SimpleAnalyzer(version);
        Analyzer a2 = new StopAnalyzer(version);
        Analyzer a3 = new WhitespaceAnalyzer(version);
        Analyzer a4 = new StandardAnalyzer(version);
        
        //测试TokenStream
        AnalyzerUtil.displayAttributes(str, a1);
        AnalyzerUtil.displayAttributes(str, a2);
        AnalyzerUtil.displayAttributes(str, a3);
        AnalyzerUtil.displayAttributes(str, a4);
    }

FlagsAttribute：标志位属性信息(了解)

PayloadAttribute：负载属性信息(了解)

说明:每一个语汇单元都存在一定的属性.通过Attribute可以获取到相关的语汇信息。