PHP爬虫

1. 学习爬虫之前, 需要先大致了解一下guzzle   https://guzzle-cn.readthedocs.io/zh_CN/latest/quickstart.html#id2
2. dom-crawler 简单介绍   https://blog.csdn.net/wz947324/article/details/79928853
3. 使用guzzle 和 dom-crawler 实现的简单爬虫   https://www.iamle.com/archives/2202.html

4. <?php
   
   /* 参考文章 https://www.iamle.com/archives/2202.html */
   
   require_once __DIR__.'/vendor/autoload.php';
   use GuzzleHttpClient;
   use SymfonyComponentDomCrawlerCrawler;
   
   // $url = 'https://movie.douban.com/subject/25812712/?from=showing';
   // //下载网页内容
   //     $client   = new Client([
   //         'timeout' => 10,
   //         'headers' => ['User-Agent' => 'Mozilla/5.0 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)',
   //         ],
   //     ]);
   //     $response = $client->request('GET', $url)->getBody()->getContents();
   //     print_r($response); exit;
   
   print_r(json_encode(Spider(), JSON_UNESCAPED_UNICODE));
   //print_r(Spider());
    
   function Spider()
   {
       //需要爬取的页面
       $url = 'https://movie.douban.com/subject/25812712/?from=showing';
    
       //下载网页内容
       $client   = new Client([
           'timeout' => 10,
           'headers' => ['User-Agent' => 'Mozilla/5.0 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)',
           ],
       ]);
       /* 原格式输出 */
       echo '<pre>';
       $response = $client->request('GET', $url)->getBody()->getContents();
   
       //进行XPath页面数据抽取
       $data    = []; //结构化数据存本数组
       $crawler = new Crawler();
       $crawler->addHtmlContent($response);
    
       try {
           //电影名称
           //网页结构中用css选择器用id的比较容易写xpath表达式
           $data['name'] = $crawler->filterXPath('//*[@id="content"]/h1/span[1]')->text();
           //电影海报
           $data['cover'] = $crawler->filterXPath('//*[@id="mainpic"]/a/img/@src')->text();
           //导演
           $data['director'] = $crawler->filterXPath('//*[@id="info"]/span[1]/span[2]')->text();
           //多个导演处理成数组
           $data['director'] = explode('/', $data['director']);
           //过滤前后空格
           $data['director'] = array_map('trim', $data['director']);
    
           //编剧
           $data['cover'] = $crawler->filterXPath('//*[@id="info"]/span[2]/span[2]/a')->text();
           //主演
           $data['mactor'] = $crawler->filterXPath('//*[@id="info"]/span[contains(@class,"actor")]/span[contains(@class,"attrs")]')->text();
           //多个主演处理成数组
           $data['mactor'] = explode('/', $data['mactor']);
           //过滤前后空格
           $data['mactor'] = array_map('trim', $data['mactor']);
    
           //上映日期
           $data['rdate'] = $crawler->filterXPath('//*[@id="info"]')->text();
           //使用正则进行抽取
           preg_match_all("/(d{4})-(d{2})-(d{2})(.*?)/", $data['rdate'], $rdate); //2017-07-07(中国大陆) / 2017-06-14(安锡动画电影节) / 2017-06-30(美国)
           $data['rdate'] = $rdate[0];
           //简介
           //演示使用class选择器的方式
           $data['introduction'] = trim($crawler->filterXPath('//div[contains(@class,"indent")]/span')->text());
    
           //演员
           //本xpath表达式会得到多个对象结果,用each方法进行遍历
           //each是传入的参数是一个闭包,在闭包中使用外部的变量使用use方法,并使用变量指针
           $crawler->filterXPath('//ul[contains(@class,"celebrities-list from-subject")]/li')->each(function (Crawler $node, $i) use (&$data) {
               $actor['name']   = $node->filterXPath('//div[contains(@class,"info")]/span[contains(@class,"name")]/a')->text(); //名字
               $actor['role']   = $node->filterXPath('//div[contains(@class,"info")]/span[contains(@class,"role")]')->text(); //角色
               $actor['avatar'] = $node->filterXPath('//a/div[contains(@class,"avatar")]/@style')->text(); //头像
               //background-image: url(https://img3.doubanio.com/img/celebrity/medium/5253.jpg) 正则抽取头像图片
               preg_match_all("/((https|http|ftp|rtsp|mms)?://)[^s]+.(jpg|jpeg|gif|png)/", $actor['avatar'], $avatar);
               $actor['avatar'] = $avatar[0][0];
               //print_r($actor);
               $data['actor'][] = $actor;
           });
    
       } catch (Exception $e) {
    
       }
    
       return $data;
    
   }

5. 自己尝试着写了一个爬笑话的程序

   <?PHP
   require_once __DIR__.'/vendor/autoload.php';
   use GuzzleHttpClient;
   use SymfonyComponentDomCrawlerCrawler;
   
   
   /* 下载笑话网页 */
   echo '<pre>';
   xiaohua();
   
   function xiaohua($param=[])
   {
       $url = 'http://xiaodiaodaya.cn/wapindex.aspx?classid=602';    /* 最新笑话 */
   
       /* 使用guzzle请求 */
       $client = new Client();
       $response = $client->request('GET', $url)->getBody()->getContents();
       // print_r($response);
   
       /* 提取内容, 使用xpath语法练习提取内容 */
       $data = [];    /* 保存最终的数据, 并存储到数据库中 */
       $crawler = new Crawler();
       $crawler->addHtmlContent($response);
       // print_r($crawler);
   
       /* 爬取最新笑话的a标签url, 根据此url可以跳转到笑话详情页 */
       for($i=1; $i<=88; $i++){
           $second_url[] = $crawler->filterXPath('//*[@id="main"]/div/div[2]/div/a['.$i.']')->attr('href');
           /* 下面的方法也是获取url, 结果一样 */
           // $link = $crawler->filterXPath('//*[@id="main"]/div/div[2]/div/a['.$i.']')->link();
           // $uri = $link->getUri();
       }
       
       /* 再根据second_url爬取最新笑话详情 */
       foreach ($second_url as $key => $value) {
           $crawler2 = new Crawler();
           $content = $client->request('get', $value)->getBody()->getContents();
           $crawler2->addHtmlContent($content);
           $text = $crawler2->filterXPath('//*[@id="main"]/div/div[1]/text()['.$key.']')->document->textContent;
           $pattern = '/([d]、(.*))相关笑话/';
           preg_match_all($pattern, $text, $matches);
           print_r($matches[1]);
           // if($key == 2)
           //     break;
       }
   }

   在写代码的过程中也遇到了一些问题, (1) 因为涉及到了递归爬取(就是爬一个笑话列表页面, 获取到链接, 在爬链接的内容即笑话详情),  new Crawler() 必须在循环体中执行, 感觉这样会浪费内存, 100个url, 遍历100次, 实例化100个对象. 但是不这样写会报错Uncaught InvalidArgumentException: Attaching DOM nodes from multiple documents in the same crawler is forbidden.  猜测是因为调用同一个对象的addHtmlContent()方法, 是不行的.   (2) 在打印$text = $crawler2->filterXPath('//*[@id="main"]/div/div[1]/text()['.$key.']')->document->textContent;内容的时候, 这个document 属性是私有的, 我修改了源代码, 将private改成public,  因为实在找不到方法打印出结果. 最终结果截图(数据太多, 只截取部分)  (3) 感觉使用这个crawler爬虫太费劲, 而且结果我不知道怎么获取(不修改源代码的话)并且没有文档可参考, 所以不要用这个爬虫

6.