The method of asynchronously climbing to get the hot list of knowledge by Python
Today, the editor will share with you the relevant knowledge points about the method of Python asynchronously climbing to get the hot list. The content is detailed and the logic is clear. I believe most people still know too much about this knowledge, so share this article for your reference. I hope you can get something after reading this article. Let's take a look at it.
1. Error code: summary and detailed url cannot get import asynciofrom bs4 import BeautifulSoupimport aiohttp headers= {'user-agent':' Mozilla/5.0 (Windows NT 6.1; Win64) X64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36' 'referer':' https://www.baidu.com/s?tn=02003390_43_hao_pg&isource=infinity&iname=baidu&itype=web&ie=utf-8&wd=%E7%9F%A5%E4%B9%8E%E7%83%AD%E6%A6%9C'}async def getPages (url): async with aiohttp.ClientSession (headers=headers) as session: async with session.get (url) as resp: print (resp.status) # print status code Html=await resp.text () soup=BeautifulSoup (html 'lxml') items=soup.select (' .HotList-item') for item in items: title=item.select ('.HotList-itemTitle') [0] .text try: abstract=item.select (' .HotList-itemExcerpt') [0] .text except: abstract='No Abstract' hot=item.select ('.HotList-itemMetrics') [0] .text try: img=item. Select ('.HotList-itemImgContainer img') [' src'] except: img='No Img' print ("{}\ n {}\ n {}" .format (title Abstract,img)) if _ _ name__ = ='_ _ main__': url=' https://www.zhihu.com/billboard' loop=asyncio.get_event_loop () loop.run_until_complete (getPages (url)) loop.close ()
Second, check the JS code
It is found that detailed links, picture links, question summaries, etc., are all in JS (CSDN's developer Assistant plug-in is really easy to use)
The regular expression gets the above information:
Then there is the detailed code.
Import asyncioimport jsonimport reimport aiohttp headers= {'user-agent':' Mozilla/5.0 (Windows NT 6.1; Win64) X64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36' 'referer':' https://www.baidu.com/s?tn=02003390_43_hao_pg&isource=infinity&iname=baidu&itype=web&ie=utf-8&wd=%E7%9F%A5%E4%B9%8E%E7%83%AD%E6%A6%9C'}async def getPages (url): async with aiohttp.ClientSession (headers=headers) as session: async with session.get (url) as resp: print (resp.status) # print status code Html=await resp.text () regex=re.compile ('hotList ": (. *?) "guestFeeds":') text=regex.search (html). Group (1) # print (json.loads (text)) # json is replaced by dictionary format for item in json.loads (text): title=item ['target'] [' titleArea'] ['text'] question=item [' target'] ['excerptArea'] [' text'] hot=item ['target'] [' metricsArea'] ['text'] link=item [' Target'] ['link'] [' url'] img=item ['target'] [' imageArea'] ['url'] if not img: question='No Abstract' print ("Title: {}\ nPopular: {}\ nQuestion: {}\ nLink: {}\ nImg: {}" .format (title Hot,question,link,img)) if _ _ name__ ='_ _ main__': url=' https://www.zhihu.com/billboard' loop=asyncio.get_event_loop () loop.run_until_complete (getPages (url)) loop.close () above are all the contents of this article entitled "Python's method of asynchronously climbing to get the hottest list of knowledge" Thank you for reading! I believe you will gain a lot after reading this article. The editor will update different knowledge for you every day. If you want to learn more knowledge, please pay attention to the industry information channel.
