zoukankan      html  css  js  c++  java
  • Attention to encodings

    工作中发现一个有意思的bug。Jenkins上测试UTF-8 解码文件的unit test莫名其妙的fail了,之前一直跑的好好的我们也没有改动任何代码。相关代码如下:

    public Context process(Context context) throws Exception {
            Config config = context.getConfig();
            Charset charset = Charset.forName(config.getEncoding());
            StreamFactory factory = context.getFactory();
            InputStreamReader reader = new InputStreamReader(FRFUtils.resolveFile(config.getReadFile()), charset);
            BeanReaderIterable parsingReader = new BeanReaderIterable(factory.createReader(config.getStreamName(), Utils.removeBlankLines(reader)));
            context.setBMSResults(fileParser.parse(parsingReader, config, factory, context.getTimestamp()));
            return context;
        }
    
    public static InputStreamReader removeBlankLines(InputStreamReader in) {
        BufferedReader reader = new BufferedReader(in);
        StringBuilder sb = new StringBuilder();
    
        reader.lines()
            .filter(StringUtils::isNotBlank)
            .forEach(line -> {
                sb.append(line);
                sb.append('
    ');
            });
        try {
            reader.close();
        } catch (IOException e) {
            LOG.warn("removeBlankLines: unable to close the reader:{}", e.getMessage());
        }
        return new InputStreamReader(new ByteArrayInputStream(sb.toString().getBytes()));
    }

    @Test 
    public void shouldEncodeWell() throws Exception {
    String input = "data/utf8_encoded_file.txt";
    String conf = "file-utf8.xml";

    String[] args = Util.constructArgs(input, null, conf, false, false);

    //when
    List<ParsingResult> results = service.parse(args);

    assertThat(results.get()).isEqualTo("u00c1u00c9u00d3-123");
    }

    经过排查发现removeBlankLines(InputStreamReader in)方法内部对InputStreamReader 做了改动,丢失了原有的Charset,所以用的解码方法一直是server默认的方法。之前test通过因为Jenkins Linux server上默认的encoding是UTF-8,但是美国那边的运维默默的更新了服务器,default encoding变成了us-ascii. 将config中Charset加入removeBlankLines中sb.toString().getBytes和new InputStreamReader修复了bug。注意bytes流和String字符串之间的转换过程就是解码编码的过程,需要指定解码编码方式。

    new String(bytes, "UTF-8"); //decoding
    "AString".getBytes("UTF-8"); //encoding

     修正后代码如下

    public Context process(Context context) throws Exception {
            Config config = context.getConfig();
            Charset charset = Charset.forName(config.getEncoding());
            StreamFactory factory = context.getFactory();
            InputStreamReader reader = new InputStreamReader(FRFUtils.resolveFile(config.getReadFile()), charset);
            BeanReaderIterable parsingReader = new BeanReaderIterable(factory.createReader(config.getStreamName(), Utils.removeBlankLines(reader, charset)));
            context.setBMSResults(fileParser.parse(parsingReader, config, factory, context.getTimestamp()));
            return context;
        }
    
    public static InputStreamReader removeBlankLines(InputStreamReader in, Charset charset) {
        BufferedReader reader = new BufferedReader(in);
        StringBuilder sb = new StringBuilder();
    
        reader.lines()
            .filter(StringUtils::isNotBlank)
            .forEach(line -> {
                sb.append(line);
                sb.append('
    ');
            });
        try {
            reader.close();
        } catch (IOException e) {
            LOG.warn("removeBlankLines: unable to close the reader:{}", e.getMessage());
        }
        return new InputStreamReader(new ByteArrayInputStream(sb.toString().getBytes(charset)), charset);
    }
    
    @Test 
    public void shouldEncodeWell() throws Exception {
        String input = "data/utf8_encoded_file.txt";
        String conf = "file-utf8.xml";
    
        String[] args = Util.constructArgs(input, null, conf, false, false);
    
        //when
        List<ParsingResult> results = service.parse(args);
    
        assertThat(results.get()).isEqualTo("u00c1u00c9u00d3-123");
    }

    InputStream vs InputStreamReader

    The InputStream is the ancestor class of all possible streams of bytes, it is not useful by itself but all the subclasses (like the FileInputStream that you are using) are great to deal with binary data.

    On the counterpart the InputStreamReader (and its father Reader) are used specifically to deal with characters (so strings) so they handle charset encodings (utf8, iso-8859-1, and so on) gracefully.

    The simple answer is: if you need binary data you can use an InputStream (also a specific one like a DataInputStream), if you need to work with text use an InputStreamReader.

    Stream vs Reader/Writer

    Streams work at the byte level, they can read (InputStream) and write (OutputStream) bytes or list of bytes to a stream.

    Reader/Writers add the concept of character on top of a stream. Since a character can only be translated to bytes by using an Encoding, readers and writers have an encoding component (that may be set automatically since Java has a default encoding property). The characters read (Reader) or written (Writer) are automatically converted to bytes by the encoding and sent to the stream.

  • 相关阅读:
    LVM 扩容硬盘笔记
    jupyter notebook 远程访问
    samba 配置文件详解
    linux 网络挂载 windows 共享文件夹
    cmder 与 win10 wsl ( 当前目录打开wsl)
    vscode for latex
    Python 使用代理
    Python Signal(信号) 异步系统事件
    centos7 install magento
    lua笔记
  • 原文地址:https://www.cnblogs.com/codingforum/p/8552128.html
Copyright © 2011-2022 走看看