zoukankan      html  css  js  c++  java

  • 关于文件流Seek以及Read操作的一点不满

    问题

    对于读取文件某指定位置开始的一段数据的操作， 我们一般可以用如下的代码来实现：

    Read File Stream Content
    1. private static string ReadContent(string fileName, int position, int length)
    2. {
    3.     if (!File.Exists(fileName))
    4.     {
    5.         throw new FileNotFoundException("The specified file is not found : " + fileName);
    6.     }
    7.  
    8.     using(FileStream stream = new FileStream(fileName, FileMode.Open, FileAccess.Read, FileShare.Read))
    9.     using (StreamReader reader = new StreamReader(stream))
    10.     {
    11.         reader.BaseStream.Seek(position, SeekOrigin.Begin);
    12.         char[] buffer = new char[length];
    13.         reader.Read(buffer, 0, length);
    14.  
    15.         return new string(buffer, 0, length);
    16.     }
    17. }

    这样的操作在代码上看来比较直观也易于理解。 如果想在同一个文件中读取多个这样的内容段， 一般可以写成如下（指定多个位置和多个需要对应读取的长度，参数列表仅为示意）：

    Read Content With Seeking
    1. private static string[] ReadContents(string fileName, int[] positions, int[] lengths)
    2. {
    3.     if (!File.Exists(fileName))
    4.     {
    5.         throw new FileNotFoundException("The specified file is not found : " + fileName);
    6.     }
    7.  
    8.     using (FileStream stream = new FileStream(fileName, FileMode.Open, FileAccess.Read, FileShare.Read))
    9.     using (StreamReader reader = new StreamReader(stream))
    10.     {
    11.         string[] contents = new string[positions.Length];
    12.  
    13.         for (int i = 0; i < positions.Length; i++)
    14.         {
    15.             reader.BaseStream.Seek(positions[i], SeekOrigin.Begin);
    16.             char[] buffer = new char[lengths[i]];
    17.             reader.Read(buffer, 0, lengths[i]);
    18.             contents[i] = new string(buffer, 0, lengths[i]);
    19.         }
    20.  
    21.         return contents;
    22.     }
    23. }

    这看起来也没有什么问题。 但是如果我们提供一段测试程序， 就会发现出乎意料的结果：

    Test App
    1. static void Main(string[] args)
    2. {
    3.     string fileName = @"text.txt";
    4.  
    5.     using(FileStream stream = new FileStream(fileName, FileMode.Create, FileAccess.Write, FileShare.None))
    6.     using (StreamWriter writer = new StreamWriter(stream))
    7.     {
    8.         writer.Write("ABCDEFGHIJKLMNOPQ");
    9.     }
    10.  
    11.  
    12.     Console.WriteLine(ReadContent(fileName, 4, 2));
    13.     Console.WriteLine(ReadContent(fileName, 10, 2));
    14.     Console.WriteLine(ReadContent(fileName, 7, 2));
    15.     Console.WriteLine();
    16.  
    17.     string[] contents = ReadContents(fileName, new int[] { 4, 10, 7 }, new int[] { 2, 2, 2 });
    18.     foreach (var item in contents)
    19.     {
    20.         Console.WriteLine(item);
    21.     }
    22.  
    23.     Console.ReadKey();
    24. }

    输出是：

    Capture

    所以当我们在同一个流中尝试定位的时候， 类库API并没有按照我们预想的那样， 取出对应的内容。 而看起来像是， 在一个文件流对象发生第一次Seek之后， 其后的所有Seek操作都失效了！这是为什么呢？

    分析

    事实上， StreamReader为了性能的考虑， 在自己的内部内置并维护了一个byte buffer。 如果在声明StreamReader对象的时候没有指定这个buffer的尺寸， 那么它的默认大小是1k。 如果是文件流， 那么这个buffer的默认大小是4K。 所有Read操作，都直接或间接转换为了对这个buffer的操作。

    Buffer Size
    1. // Using a 1K byte buffer and a 4K FileStream buffer works out pretty well
    2. // perf-wise.  On even a 40 MB text file, any perf loss by using a 4K
    3. // buffer is negated by the win of allocating a smaller byte[], which
    4. // saves construction time.  This does break adaptive buffering,
    5. // but this is slightly faster.
    6. internal const int DefaultBufferSize = 1024;  // Byte buffer size
    7. private const int DefaultFileStreamBufferSize = 4096;
    8. private const int MinBufferSize = 128;
    Read Buffer
    1.         // This version has a perf optimization to decode data DIRECTLY into the
    2.         // user's buffer, bypassing StreamWriter's own buffer.
    3.         // This gives a > 20% perf improvement for our encodings across the board,
    4.         // but only when asking for at least the number of characters that one
    5.         // buffer's worth of bytes could produce.
    6.         // This optimization, if run, will break SwitchEncoding, so we must not do
    7.         // this on the first call to ReadBuffer.
    8.         private int ReadBuffer(char[] userBuffer, int userOffset, int desiredChars, out bool readToUserBuffer) {
    9.             charLen = 0;
    10.             charPos = 0;
    12.             if (!_checkPreamble)
    13.                 byteLen = 0;
    15.             int charsRead = 0;
    17.             // As a perf optimization, we can decode characters DIRECTLY into a
    18.             // user's char[].  We absolutely must not write more characters
    19.             // into the user's buffer than they asked for.  Calculating
    20.             // encoding.GetMaxCharCount(byteLen) each time is potentially very
    21.             // expensive - instead, cache the number of chars a full buffer's
    22.             // worth of data may produce.  Yes, this makes the perf optimization
    23.             // less aggressive, in that all reads that asked for fewer than AND
    24.             // returned fewer than _maxCharsPerBuffer chars won't get the user
    25.             // buffer optimization.  This affects reads where the end of the
    26.             // Stream comes in the middle somewhere, and when you ask for
    27.             // fewer chars than than your buffer could produce.
    28.             readToUserBuffer = desiredChars >= _maxCharsPerBuffer;
    30.             do {
    31.                 if (_checkPreamble) {
    32.                     BCLDebug.Assert(bytePos <= _preamble.Length, "possible bug in _compressPreamble.  Are two threads using this StreamReader at the same time?");
    33.                     int len = stream.Read(byteBuffer, bytePos, byteBuffer.Length - bytePos);
    34.                     BCLDebug.Assert(len >= 0, "Stream.Read returned a negative number!  This is a bug in your stream class.");
    35.  
    36.                     if (len == 0) {
    37.                         // EOF but we might have buffered bytes from previous
    38.                         // attempts to detecting preamble that needs to decoded now
    39.                         if (byteLen > 0) {
    40.                             if (readToUserBuffer) {
    41.                                 charsRead += decoder.GetChars(byteBuffer, 0, byteLen, userBuffer, userOffset + charsRead);
    42.                                 charLen = 0;  // StreamReader's buffer is empty.
    43.                             }
    44.                             else {
    45.                                 charsRead = decoder.GetChars(byteBuffer, 0, byteLen, charBuffer, charsRead);
    46.                                 charLen += charsRead;  // Number of chars in StreamReader's buffer.
    47.                             }
    48.                         }
    49.                         return charsRead;
    50.                     }
    51.  
    52.                     byteLen += len;
    53.                 }
    54.                 else {
    55.                     BCLDebug.Assert(bytePos == 0, "bytePos can be non zero only when we are trying to _checkPreamble.  Are two threads using this StreamReader at the same time?");
    56.                     byteLen = stream.Read(byteBuffer, 0, byteBuffer.Length);
    57.                     BCLDebug.Assert(byteLen >= 0, "Stream.Read returned a negative number!  This is a bug in your stream class.");
    59.                     if (byteLen == 0)  // EOF
    60.                         return charsRead;
    61.                 }
    62.  
    63.                 // _isBlocked == whether we read fewer bytes than we asked for.
    64.                 // Note we must check it here because CompressBuffer or
    65.                 // DetectEncoding will ---- with byteLen.
    66.                 _isBlocked = (byteLen < byteBuffer.Length);
    68.                 // Check for preamble before detect encoding. This is not to override the
    69.                 // user suppplied Encoding for the one we implicitly detect. The user could
    70.                 // customize the encoding which we will loose, such as ThrowOnError on UTF8
    71.                 // Note: we don't need to recompute readToUserBuffer optimization as IsPreamble
    72.                 // doesn't change the encoding or affect _maxCharsPerBuffer
    73.                 if (IsPreamble())
    74.                     continue;
    75.  
    76.                 // On the first call to ReadBuffer, if we're supposed to detect the encoding, do it.
    77.                 if (_detectEncoding && byteLen >= 2) {
    78.                     DetectEncoding();
    79.                     // DetectEncoding changes some buffer state.  Recompute this.
    80.                     readToUserBuffer = desiredChars >= _maxCharsPerBuffer;
    81.                 }
    82.  
    83.                 charPos = 0;
    84.                 if (readToUserBuffer) {
    85.                     charsRead += decoder.GetChars(byteBuffer, 0, byteLen, userBuffer, userOffset + charsRead);
    86.                     charLen = 0;  // StreamReader's buffer is empty.
    87.                 }
    88.                 else {
    89.                     charsRead = decoder.GetChars(byteBuffer, 0, byteLen, charBuffer, charsRead);
    90.                     charLen += charsRead;  // Number of chars in StreamReader's buffer.
    91.                 }
    92.             } while (charsRead == 0);
    93.  
    94.             _isBlocked &= charsRead < desiredChars;
    96.             //Console.WriteLine("ReadBuffer: charsRead: "+charsRead+"  readToUserBuffer: "+readToUserBuffer);
    97.             return charsRead;
    98.         }



    所以问题就转化为， 当第二次调用BaseStream.Seek的时候， 对应的buffer的内容并没有重新读取！所以第二次读取的时候， 对应读取的内容其实是第一次seek后， 对应的Seek位置以后4K长度的内容。这对应的缓存的起始位置已经完全不同了（或者完全不在缓存中）。

    如果想要在第二次seek前刷新缓存， 必须显式调用DiscardBufferedData（）：

    Code Snippet
    1. // DiscardBufferedData tells StreamReader to throw away its internal
    2. // buffer contents.  This is useful if the user needs to seek on the
    3. // underlying stream to a known location then wants the StreamReader
    4. // to start reading from this new point.  This method should be called
    5. // very sparingly, if ever, since it can lead to very poor performance.
    6. // However, it may be the only way of handling some scenarios where
    7. // users need to re-read the contents of a StreamReader a second time.
    8. public void DiscardBufferedData() {
    9.     byteLen = 0;
    10.     charLen = 0;
    11.     charPos = 0;
    12.     decoder = encoding.GetDecoder();
    13.     _isBlocked = false;
    14. }

    一点抱怨

    记得《Framework Design》中讲到一些.NET类库设计时的一些遗憾, 我不知道这个算不算. 我觉得自己最少算是一个熟手, 但是我遇到这个问题的时候第一感觉是很奇怪. 看到了代码的时候, 觉得代码充满tricky和smelly的味道. 类库的设计者显然恶意揣度了程序员的意图和编程能力. 设计者觉得自己在性能和可用性上找到了一个巧妙的平衡点, 但实际上不但造成了API歧义, 而且显然会导致错误的结果. 诚然, 按照统计学原理, 内容读取多发生在相近的地方; 或者说被缓存的内容有继续被读取的较大可能. 但是性能永远是建立在正确性的基础上的. 这个API令人遗憾的地方, 就是忽视了多次Seek这种需求.

    我们来揣度一下如何设计.

    如果想要做得大而全, 完全可以保持这样的一个缓存, 但是显然不能仅仅依赖于BaseStream的Seek, 而是要在StreamReader类, 或者其基类TextReader中提供Seek API来封装对BaseStream的定位操作同时也包括对缓存数据的定位操作. 这样的API是不是对程序员更友好? 我觉得是, 至少不会产生误解吧.

    如果想要做得小而精, 完全可以去掉这样的缓存机制. 取而代之, 使用程序员提供的缓存. 完全由语言的使用者来决定是否实现自己的缓存机制. 这样的语言或者类库, 同样也是健壮的, 也是可以被程序员接受的.

    总结

    最近园子里面关于C#语言自身及.NET类库的讨论深入而热烈。我私下以为， 争论是每一种语言前进的动力。 想说点什么， 突然想起了上面的这个小例子。 其实作为程序员， 可能既不关注究竟是语言支撑模式，也不关注是不是类库支撑模式。 唯希望在类库设计中，少一点上面这个例子中的灵机一动， 多一点实实在在。

    作者：Jeffrey Sun
    出处：http://sun.cnblogs.com/
    本文以“现状”提供且没有任何担保，同时也没有授予任何权利。本文版权归作者所有，欢迎转载，但未经作者同意必须保留此段声明，且在文章页面明显位置给出原文连接，否则保留追究法律责任的权利。
  • 相关阅读:
    golang/windows如何删除只读属性文件
    golang/TLS 采坑
    gsweb —— 理解HTTP协议
    gsweb —— 自己动手用golang写WEB框架
    Scala冒泡排序、快排、归并
    Hadoop自动化部署脚本
    大数据学习笔记
    vim键盘图
    什么是回调或高级函数?
    使用CSS表达式去除超链接的虚框的一些方法
  • 原文地址：https://www.cnblogs.com/sun/p/1775311.html
Copyright © 2011-2022 走看看