我在写一个抓取网页的小程序,发现有些网页能够抓取成功呢,有些网页则要出报错“尝试自动重定向的次数太多”
如果抓取这个网页就要报错:http://spiedl.org/vsearch/servlet/VerityServlet?smode=strresults&SMODE=strsearch&possible1zone=article&pjournals=SPIEDL&KEY=SPIEDL&sort=rel&maxdisp=25&threshold=0&deliveryType=spiedl&possible1=paul&submit=Search
直接访问这个地址是没问题的
现在我把我的代码贴出来,希望高手帮忙解答下:
string strUrl= http://spiedl.org/vsearch/servlet/VerityServlet?smode=strresults&SMODE=strsearch&possible1zone=article&pjournals=SPIEDL&KEY=SPIEDL&sort=rel&maxdisp=25&threshold=0&deliveryType=spiedl&possible1=paul&submit=Search;
CookieContainer sparkc = new CookieContainer();
HttpWebRequest SparkRequest = (HttpWebRequest)WebRequest.Create(strUrl);
SparkRequest.AllowAutoRedirect = true;
SparkRequest.CookieContainer = sparkc;
HttpWebResponse SparkResponse = (HttpWebResponse)SparkRequest.GetResponse();//运行到这里就要报错!
Stream dataStream = SparkResponse.GetResponseStream();
StreamReader reader = new StreamReader(dataStream, Encoding.GetEncoding("UTF-8"));
divLeft.InnerHtml = reader.ReadToEnd();
reader.Close();
如果抓取这个网页就要报错:http://spiedl.org/vsearch/servlet/VerityServlet?smode=strresults&SMODE=strsearch&possible1zone=article&pjournals=SPIEDL&KEY=SPIEDL&sort=rel&maxdisp=25&threshold=0&deliveryType=spiedl&possible1=paul&submit=Search
直接访问这个地址是没问题的
现在我把我的代码贴出来,希望高手帮忙解答下:
string strUrl= http://spiedl.org/vsearch/servlet/VerityServlet?smode=strresults&SMODE=strsearch&possible1zone=article&pjournals=SPIEDL&KEY=SPIEDL&sort=rel&maxdisp=25&threshold=0&deliveryType=spiedl&possible1=paul&submit=Search;
CookieContainer sparkc = new CookieContainer();
HttpWebRequest SparkRequest = (HttpWebRequest)WebRequest.Create(strUrl);
SparkRequest.AllowAutoRedirect = true;
SparkRequest.CookieContainer = sparkc;
HttpWebResponse SparkResponse = (HttpWebResponse)SparkRequest.GetResponse();//运行到这里就要报错!
Stream dataStream = SparkResponse.GetResponseStream();
StreamReader reader = new StreamReader(dataStream, Encoding.GetEncoding("UTF-8"));
divLeft.InnerHtml = reader.ReadToEnd();
reader.Close();