pyquery学习笔记

很早就听说了pyquery的强大。写了个简单的测试程序实验下。

思路是找个动态网页，先用PhantomJS加载，然后用PYQUERY解析。

1、随便找了个带表格的股票网页，里面有大量的股票数据，测试的目的就是抓取表格中的数据。

链接如下 

http://quote.eastmoney.com/center/BKList.html#notion_0_0?sortRule=0

2、使用PhantomJS加载。

all_url = "http://quote.eastmoney.com/center/BKList.html#notion_0_0?sortRule=0"
a_driver = webdriver.PhantomJS()
a_driver.get(all_url)

3、使用pyquery分析。之前看有的博主建议在传入pq前先用lxml的etree规整一遍，测试了发现会报错。不如不用。

Traceback (most recent call last):
File "C:/Users/Administrator/PycharmProjects/p3test/pyquery_test.py", line 25, in <module>
doc = pq(etree.fromstring(a_driver.page_source))
File "lxml.etree.pyx", line 3213, in lxml.etree.fromstring (srclxmllxml.etree.c:77616)
File "parser.pxi", line 1819, in lxml.etree._parseMemoryDocument (srclxmllxml.etree.c:116413)
File "parser.pxi", line 1700, in lxml.etree._parseDoc (srclxmllxml.etree.c:114959)
File "parser.pxi", line 1040, in lxml.etree._BaseParser._parseUnicodeDoc (srclxmllxml.etree.c:109084)
File "parser.pxi", line 573, in lxml.etree._ParserContext._handleParseResultDoc (srclxmllxml.etree.c:103323)
File "parser.pxi", line 683, in lxml.etree._handleParseResult (srclxmllxml.etree.c:104977)
File "parser.pxi", line 613, in lxml.etree._raiseParseError (srclxmllxml.etree.c:103886)
lxml.etree.XMLSyntaxError: Opening and ending tag mismatch: link line 11 and head, line 17, column 4656

详细代码如下。

all_url = "http://quote.eastmoney.com/center/BKList.html#notion_0_0?sortRule=0"

a_driver = webdriver.PhantomJS()
a_driver.get(all_url)
#doc = pq(etree.fromstring(a_driver.page_source))
doc = pq(a_driver.page_source)
print("doc=",doc("tr"))

for li in doc("tr").items():
    if len(li("td"))>=12:
        print("{0}-{1}-{2}-{3}-{4}-{5}".format(li("td").eq(0).text(),
                                                    li("td").eq(1).text(),
                                                    li("td").eq(2).text(),
                                                    li("td").eq(3).text(),
                                                    li("td").eq(4).text(),
                                                    li("td").eq(5).text(),))

输出结果如下：

1-共享经济-行情 股吧 资金流-1164.83-47.98-4.30%
2-PM2.5-行情 股吧 资金流-2804.73-81.69-3.00%
3-钛白粉-行情 股吧 资金流-1139.54-32.39-2.93%
4-美丽中国-行情 股吧 资金流-2991.04-57.94-1.98%
5-京津冀-行情 股吧 资金流-2866.82-48.62-1.73%
6-北京冬奥-行情 股吧 资金流-1168.58-18.85-1.64%
7-OLED-行情 股吧 资金流-1394.67-22.55-1.64%
8-民营医院-行情 股吧 资金流-1670.81-24.72-1.50%
9-虚拟现实-行情 股吧 资金流-951.37-12.92-1.38%
10-次新股-行情 股吧 资金流-38011.00-499.03-1.33%
11-节能环保-行情 股吧 资金流-13137.08-151.37-1.17%
12-3D玻璃-行情 股吧 资金流-1067.91-11.35-1.07%
13-病毒防治-行情 股吧 资金流-1915.40-19.41-1.02%
14-蓝宝石-行情 股吧 资金流-1981.26-19.67-1.00%
15-体育产业-行情 股吧 资金流-1978.79-19.61-1.00%
16-二胎概念-行情 股吧 资金流-1799.18-17.44-0.98%
17-食品安全-行情 股吧 资金流-3180.84-30.23-0.96%
18-海绵城市-行情 股吧 资金流-1155.37-10.92-0.95%
19-ST概念-行情 股吧 资金流-26773.39-227.49-0.86%
20-创业成份-行情 股吧 资金流-2104.93-17.20-0.82%
21-合同能源-行情 股吧 资金流-1545.17-12.25-0.80%
22-网红直播-行情 股吧 资金流-855.33-6.28-0.74%
23-AB股-行情 股吧 资金流-10146.54-74.23-0.74%
24-雄安新区-行情 股吧 资金流-1425.79-9.86-0.70%
25-车联网-行情 股吧 资金流-1014.41-7.06-0.70%
26-滨海新区-行情 股吧 资金流-11399.25-72.70-0.64%
27-国产芯片-行情 股吧 资金流-992.71-6.22-0.63%
28-王亚伟系-行情 股吧 资金流-2568.52-14.95-0.59%
29-石墨烯-行情 股吧 资金流-2490.11-13.95-0.56%
30-燃料电池-行情 股吧 资金流-2121.03-11.85-0.56%
31-金融机具-行情 股吧 资金流-1768.72-9.12-0.52%
32-人工智能-行情 股吧 资金流-970.58-4.95-0.51%
33-独家药品-行情 股吧 资金流-1684.69-8.34-0.50%
34-免疫治疗-行情 股吧 资金流-1309.56-6.32-0.49%