zoukankan      html  css  js  c++  java
  • Attention to encodings

    工作中发现一个有意思的bug。Jenkins上测试UTF-8 解码文件的unit test莫名其妙的fail了,之前一直跑的好好的我们也没有改动任何代码。相关代码如下:

    public Context process(Context context) throws Exception {
            Config config = context.getConfig();
            Charset charset = Charset.forName(config.getEncoding());
            StreamFactory factory = context.getFactory();
            InputStreamReader reader = new InputStreamReader(FRFUtils.resolveFile(config.getReadFile()), charset);
            BeanReaderIterable parsingReader = new BeanReaderIterable(factory.createReader(config.getStreamName(), Utils.removeBlankLines(reader)));
            context.setBMSResults(fileParser.parse(parsingReader, config, factory, context.getTimestamp()));
            return context;
        }
    
    public static InputStreamReader removeBlankLines(InputStreamReader in) {
        BufferedReader reader = new BufferedReader(in);
        StringBuilder sb = new StringBuilder();
    
        reader.lines()
            .filter(StringUtils::isNotBlank)
            .forEach(line -> {
                sb.append(line);
                sb.append('
    ');
            });
        try {
            reader.close();
        } catch (IOException e) {
            LOG.warn("removeBlankLines: unable to close the reader:{}", e.getMessage());
        }
        return new InputStreamReader(new ByteArrayInputStream(sb.toString().getBytes()));
    }

    @Test 
    public void shouldEncodeWell() throws Exception {
    String input = "data/utf8_encoded_file.txt";
    String conf = "file-utf8.xml";

    String[] args = Util.constructArgs(input, null, conf, false, false);

    //when
    List<ParsingResult> results = service.parse(args);

    assertThat(results.get()).isEqualTo("u00c1u00c9u00d3-123");
    }

    经过排查发现removeBlankLines(InputStreamReader in)方法内部对InputStreamReader 做了改动,丢失了原有的Charset,所以用的解码方法一直是server默认的方法。之前test通过因为Jenkins Linux server上默认的encoding是UTF-8,但是美国那边的运维默默的更新了服务器,default encoding变成了us-ascii. 将config中Charset加入removeBlankLines中sb.toString().getBytes和new InputStreamReader修复了bug。注意bytes流和String字符串之间的转换过程就是解码编码的过程,需要指定解码编码方式。

    new String(bytes, "UTF-8"); //decoding
    "AString".getBytes("UTF-8"); //encoding

     修正后代码如下

    public Context process(Context context) throws Exception {
            Config config = context.getConfig();
            Charset charset = Charset.forName(config.getEncoding());
            StreamFactory factory = context.getFactory();
            InputStreamReader reader = new InputStreamReader(FRFUtils.resolveFile(config.getReadFile()), charset);
            BeanReaderIterable parsingReader = new BeanReaderIterable(factory.createReader(config.getStreamName(), Utils.removeBlankLines(reader, charset)));
            context.setBMSResults(fileParser.parse(parsingReader, config, factory, context.getTimestamp()));
            return context;
        }
    
    public static InputStreamReader removeBlankLines(InputStreamReader in, Charset charset) {
        BufferedReader reader = new BufferedReader(in);
        StringBuilder sb = new StringBuilder();
    
        reader.lines()
            .filter(StringUtils::isNotBlank)
            .forEach(line -> {
                sb.append(line);
                sb.append('
    ');
            });
        try {
            reader.close();
        } catch (IOException e) {
            LOG.warn("removeBlankLines: unable to close the reader:{}", e.getMessage());
        }
        return new InputStreamReader(new ByteArrayInputStream(sb.toString().getBytes(charset)), charset);
    }
    
    @Test 
    public void shouldEncodeWell() throws Exception {
        String input = "data/utf8_encoded_file.txt";
        String conf = "file-utf8.xml";
    
        String[] args = Util.constructArgs(input, null, conf, false, false);
    
        //when
        List<ParsingResult> results = service.parse(args);
    
        assertThat(results.get()).isEqualTo("u00c1u00c9u00d3-123");
    }

    InputStream vs InputStreamReader

    The InputStream is the ancestor class of all possible streams of bytes, it is not useful by itself but all the subclasses (like the FileInputStream that you are using) are great to deal with binary data.

    On the counterpart the InputStreamReader (and its father Reader) are used specifically to deal with characters (so strings) so they handle charset encodings (utf8, iso-8859-1, and so on) gracefully.

    The simple answer is: if you need binary data you can use an InputStream (also a specific one like a DataInputStream), if you need to work with text use an InputStreamReader.

    Stream vs Reader/Writer

    Streams work at the byte level, they can read (InputStream) and write (OutputStream) bytes or list of bytes to a stream.

    Reader/Writers add the concept of character on top of a stream. Since a character can only be translated to bytes by using an Encoding, readers and writers have an encoding component (that may be set automatically since Java has a default encoding property). The characters read (Reader) or written (Writer) are automatically converted to bytes by the encoding and sent to the stream.

  • 相关阅读:
    vs2019+GLFW+GLAD出现无法解析的外部符号
    图的着色算法
    Head First C# 实验室2(冒险游戏)
    击中和击不中变换
    开运算和闭运算
    膨胀与腐蚀
    两数相加(C#数据结构和算法练习)
    C# 特性和索引(C#学习笔记06)
    C# 索引器(C#学习笔记05)
    C# yield checked,unchecked lock语句(C#学习笔记04)
  • 原文地址:https://www.cnblogs.com/codingforum/p/8552128.html
Copyright © 2011-2022 走看看