发个原创小工具，下载autohome 论坛帖子离线浏览

【提出问题】

autohome是个汽车门户，有时论坛里面会有一些比较好看的帖子，比如“一家四口环中国行”，主贴100多页，跟帖4000多页，看起来很爽。

但是，其论坛的JS脚本写的并不好，如果一帖图片非常多的情况下，经常有图片显示不了，很是郁闷。

于是有思路想下载帖子出来离线浏览。有人可能会说，现在有很多现成的离线浏览软件呀，不错，但是下载不了这里的图片，因为其图片URL做了个小小的手脚。

【分析问题】

1、URL规律分析

第一帖是 http://club.autohome.com.cn/bbs/thread-o-200042-19582947-1.html

第二帖子 http://club.autohome.com.cn/bbs/thread-o-200042-19582947-2.html

发现其N贴是 http://club.autohome.com.cn/bbs/thread-o-200042-19582947-N.html

2、图片分析

查看源文件，其图片的HTML为

<img id="img-0-8" name="lazypic" onload="tz.picLoaded(this)" onerror="tz.picNotFind(this)" style="700px;height:464px" src="http://x.autoimg.cn/club/lazyload.png" src9="http://club1.autoimg.cn/album/userphotos/2013/2/25/500_9bed_79b1f6c8_79b1f6c8.jpg" />

默认的src是一个等待图片，真实的src为src9属性，通过onload事件来替换src实现显示图片，超时或者出错是显示onerror事件

那么我们抓取src9就可以下载图片了

3、图片抓取尝试

比如上面的图片URL为 http://club1.autoimg.cn/album/userphotos/2013/2/25/500_9bed_79b1f6c8_79b1f6c8.jpg ，如果直接下载图片的话，服务器会拒绝，因为你在盗链。

所以最好是用HTTP 1.1的指令方式发起HTTP REQUEST，同时要传达 request.Referer 属性，可用fiddler监控

GET http://club1.autoimg.cn/album/userphotos/2013/4/3/500_43be_cde0b51b_cde0b51b.jpg HTTP/1.1
Accept: */*
Referer: http://club.autohome.com.cn/bbs/thread-o-200042-19582947-1.html
Accept-Language: zh-CN
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/6.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; KB974487)
Accept-Encoding: gzip, deflate
Connection: Keep-Alive
DNT: 1
Host: club1.autoimg.cn

为了方便图片显示，建议按照原来的路径保存图片文件，比如图片 http://club1.autoimg.cn/album/userphotos/2013/4/3/500_43be_cde0b51b_cde0b51b.jpg，则按照文件夹 club1.autoimg.cn/album/userphotos/2013/4/3/500_43be_cde0b51b_cde0b51b.jpg 来保存。

4、分页链接

为了便于浏览，下载后的分页连接要能连上

<div class="pages fs">
                <a href="forum-o-200042-1.html">返回列表</a></div>
            <div class="pages" id="x-pages1" maxindex="4927"><span class="cur">1</span><a target="_self" href="thread-o-200042-19582947-2.html">2</a><a target="_self" href="thread-o-200042-19582947-3.html">3</a><a target="_self" href="thread-o-200042-19582947-4.html">4</a><a target="_self" href="thread-o-200042-19582947-5.html">5</a><span>...</span><a target="_self" href="thread-o-200042-19582947-4927.html">4927</a><span class="gopage"><input type="text" value="1" title="输入页码，按回车快速跳转" onkeydown="if(event.keyCode==13){tz.goPage(this)}" /><span class="fs" title="共 4927 页"> / 4927 页</span></span><a target="_self" class="afpage" href="thread-o-200042-19582947-2.html" title="支持键盘 ← → 键翻页">下一页</a></div>
            <div class="jfwen">
                到第<span><input type="text" value="" class="topinp txtcenter" id="txtGoFloor1" maxlength="7"
                    title="输入楼层数，按回车快速定位" onkeydown="if(event.keyCode==13){tz.goFloor(null,'txtGoFloor1')}" /></span>楼</div>

发现本来的连接就是html文件的文件名，所以只要按原来的文件名保存就可以了。

5、无用代码过滤

将onload和onerror事件去掉，将script的始末标签替换为DIV，将无用http://开头 替换为本地./，方便本地浏览不占资源

【解决问题】

1、http

private static readonly string DefaultUserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)";
        /// <summary>  
        /// 创建GET方式的HTTP请求  ，拿来的改了下
        /// </summary>  
        /// <param name="url">请求的URL</param>  
        /// <param name="timeout">请求的超时时间</param>  
        /// <param name="userAgent">请求的客户端浏览器信息，可以为空</param>  
        /// <param name="referer">请求来源URL</param>
        /// <param name="cookies">随同HTTP请求发送的Cookie信息，如果不需要身份验证可以为空</param>  
        /// <returns></returns>  
        public static HttpWebResponse CreateGetHttpResponse(string url, int? timeout, string userAgent, string referer, CookieCollection cookies)
        {
            if (string.IsNullOrEmpty(url))
            {
                throw new ArgumentNullException("url");
            }
            HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
            request.Method = "GET";
            request.UserAgent = DefaultUserAgent;
            if (!string.IsNullOrEmpty(userAgent))
            {
                request.UserAgent = userAgent;
            }
            if (timeout.HasValue)
            {
                request.Timeout = timeout.Value;
            }
            if (referer != null)
            {
                request.Referer = referer;
            }
            if (cookies != null)
            {
                request.CookieContainer = new CookieContainer();
                request.CookieContainer.Add(cookies);
            }
            return request.GetResponse() as HttpWebResponse;
        }

2、主要动作按钮

private void btnStart_Click(object sender, EventArgs e)
        {
            btnStart.Enabled = false;
            btnStop.Enabled = true;
            timer1.Enabled = true;
            
            var dt1 = DateTime.Now;
            var dir = tbSaveDir.Text;
            var baseUrl = tbURL.Text.Replace("-1.", "-#.");
            var totalPage = (int) tbTotalPage.Value;
            var fromPage = (int) tbFromPage.Value;
            var isDefault = radioButton1.Checked;
            var html = "";
            var imgurl = "";
            var htmlFile = "";
            var imgNum = 0;

            //进度条
            progressBar1.Maximum = totalPage - fromPage + 1;
            progressBar1.Value = 1;
            progressBar2.Value = 1;

            for (int i = fromPage; i <= totalPage; i++)
            {
                //处理进度条
                progressBar1.Step = 1;
                progressBar1.PerformStep();

                //处理操作
                var url = baseUrl.Replace("#", i.ToString());
                try
                {
                    var response = HttpWebResponseUtility.CreateGetHttpResponse(url, null, null, url, null);
                    if (response != null)
                    {
                        
                        var sr = new StreamReader(response.GetResponseStream(), Encoding.GetEncoding("gb2312"),true);
                        html = sr.ReadToEnd();
                        response.Close();
                        
                        string pattern = "src9=\"http://[a-zA-Z0-9_./]+\"";
                        var gc = Regex.Matches(html, pattern);
                        //Console.WriteLine(gc);

                        //处理HTML文件
                        html = html.Replace("src=\"http://x.autoimg.cn/club/lazyload.png\" src9=\"", "src=\"");
                        html = html.Replace("http://", "");
                        html = html.Replace("onload=", "x1=");
                        html = html.Replace("onerror=", "x2=");
                        html = html.Replace("<script", "<DIV style=\"display:none\" ");
                        html = html.Replace("</script", "</DIV");
                        htmlFile = dir + Path.DirectorySeparatorChar + Path.GetFileName(url.Replace("http://", ""));
                        var sw = new StreamWriter(htmlFile, true, Encoding.GetEncoding("gb2312"));
                        sw.Write(html);
                        sw.Close();
                        sw.Dispose();
                        tbLog.AppendText(htmlFile + " ok");

                        imgNum = 0;
                        foreach (var match in gc)
                        {
                            //Console.WriteLine(match.ToString());
                            imgurl = match.ToString().Replace("\"", "").Replace("src9=", "");
                            _myQue.Enqueue(new ParamEntity(dir, imgurl, url, isDefault));

                            67                             
    
                            imgNum++;
                        }//end-foreach

                        tbLog.AppendText(", " + imgNum +" image(s)" + Environment.NewLine);
                        _totalNum += imgNum;
                    }
                }
                catch (Exception exception)
                {
                    //Console.WriteLine(exception);
                    MessageBox.Show(exception.Message);
                }
                
            }
            btnStart.Enabled = true;
            var dt2 = DateTime.Now;
            var timeUse = dt2 - dt1;
            MessageBox.Show(string.Format("页面下载已结束，耗时 {0} 分钟，请等待图片下载结束，结束后打开目录 {1} 查看下载内容。", timeUse.TotalMinutes.ToString("F2"), dir));
        }

3、下载图片

private void DownloadImage(object obj)
        {
            var pe = obj as ParamEntity;
            var tmp = pe.ImgUrl.Replace("http://", "");
            var dir = pe.SaveDir + Path.DirectorySeparatorChar + Path.GetDirectoryName(tmp);
            var filename = pe.SaveDir + Path.DirectorySeparatorChar + tmp;

            if (!Directory.Exists(dir))
            {
                Directory.CreateDirectory(dir);
            }
            try
            {
                _runNum++;
                if (pe.IsType1)
                {
                    var wc = new WebClient();
                    wc.DownloadFile(pe.ImgUrl, filename);
                }
                else
                {
                    var imgres = HttpWebResponseUtility.CreateGetHttpResponse(pe.ImgUrl, null, null, pe.PageUrl, null);
                    if (imgres != null)
                    {
                        var reader = imgres.GetResponseStream();
                        var writer = new FileStream(filename, FileMode.OpenOrCreate, FileAccess.Write);
                        var buff = new byte[512];
                        var c = 0; //实际读取的字节数
                        while ((c = reader.Read(buff, 0, buff.Length)) > 0)
                        {
                            writer.Write(buff, 0, c);
                        }
                        writer.Close();
                        writer.Dispose();
                        reader.Close();
                        reader.Dispose();
                        imgres.Close();
                    }
                }
                
            }
            catch (Exception e)
            {
                //Console.WriteLine(e.ToString());
                _logQue.Enqueue(Path.GetFileName(tmp) + " fail. " + e.Message);
            }
        }

4、timer触发器

private void timer1_Tick(object sender, EventArgs e)
        {
            //处理进度条
            progressBar2.Maximum = _totalNum;
            progressBar2.Step = 5;
            progressBar2.PerformStep();

            for (var i = 1; i <= 5; i++)
            {
                if (_myQue.Count > 0)
                {
                    //Console.WriteLine(@"RunThread ({0}) {1}", i, _runNum);
                    var pe = (ParamEntity) _myQue.Dequeue();
                    var thread = new Thread(new ParameterizedThreadStart(DownloadImage));
                    thread.Start(pe);
                }
            }
            if (_logQue.Count > 0)
            {
                tbLog.AppendText(_logQue.Dequeue().ToString() + Environment.NewLine);
            }

            Application.DoEvents();
        }

本来不想用timer的，想用一个队列，自己处理完了会再继续处理下一个，结果没写成。

【可能的技术要点】

1、http请求带referer

2、多线程，界面不阻塞（backgroundWorker，我还没改）

3、progressBar

4、Queue

【成品】

猛击下载

【心得】

不求最好，但求心安。新手的看看，大虾的指点。