How to use Python to climb to benefit the Apple data of Rural Network
This article will explain in detail for you how to use Python to climb to benefit the agricultural network Apple data, the content of the article is of high quality, so the editor will share it for you as a reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.
This goal
Climb to get the information of Huinong website.
Victim address
Https://www.cnhnb.com/
Environment
Python3.6
Pycharm
Crawler code
Import tool
Import requestsimport parselimport csvimport time
Request header
Headers = {'Cookie':' deviceIdRenew=1; Hm_lvt_91cf34f62b9bedb16460ca36cf192f4c=1604579356,1604659451; deviceId=d1dd5b9-d191-406b Windows NT 971dmur391916a0e; sessionId=S_0KH64T2IHLHSO77N; lmvid=b24dcd0ad2a8f0b783f248c7ff2675a8; lmvid.sig=w1UBnTUKSDq-GfAlx6TyR_K7SjyujGIlF-1kRjTrOAI; hnUserTicket=b80e6b3a-38a3-432cMur816dMuyaeb0376228bd; hnUserId=870048250; Hm_lpvt_91cf34f62b9bedb16460ca36cf192f4c=1604659904', 'Host':' www.cnhnb.com', 'Referer':' https://www.cnhnb.com/supply/pingguo/', 'User-Agent':' Mozilla/5.0 (Windows NT 10.0 WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36',}
Parsing websites and crawling data
Def get_page_url (page_url, page_id): response_2 = requests.get (url=page_url Headers=headers) selector_2 = parsel.Selector (response_2.text) num_id = selector_2.css ('# _ layout > div > div > div > div > ul > li:nth-child (1) > a::attr (href)'). Get (). Split ('/') [2] title = selector_2.css ('. Proinfo-title::text'). Get (). Strip () # title update_time = selector_ 2.css ('. Update-time::text'). Get (). Strip ('update time:') # Update time price = selector_2.css ('.priceTxt .orange .fs30:: text'). Get (). Strip () +' yuan'# Price ads = selector_2.css ('div:nth-child (5) > span.fs14.gray6::text'). Get () # place of shipment Address get_phone_url = 'https://gateway.cnhnb.com/banana/im/operate/wechatcall' data = {' businessType':'1' 'sourceFrom':' 2, 'ticket':', "b80e6b3a-38a3-432c-816d-aeb0376228bd", 'userId':' {} '.format (num_id),} head = {' authority': 'gateway.cnhnb.com',' method': 'POST',' path':'/ banana/im/operate/wechatcall', 'scheme':' https' 'accept':' application/json, text/plain, * / *, 'accept-encoding':' gzip, deflate, br', 'accept-language':' zh-CN,zh QQ 0.9, 'cache-control':' no-cache', 'content-length':' 98, 'content-type':' application/json', 'origin':' https://www.cnhnb.com', 'pragma':' no-cache', 'referer':' https://www.cnhnb.com/gongying/{}/'.format(page_id), # 'Cookie':' deviceIdRenew=1 Hm_lvt_91cf34f62b9bedb16460ca36cf192f4c=1604579356,1604659451; deviceId=d1dd5b9-d191-406b empty', 971dmure 391916a0e; sessionId=S_0KH64T2IHLHSO77N; lmvid=b24dcd0ad2a8f0b783f248c7ff2675a8; lmvid.sig=w1UBnTUKSDq-GfAlx6TyR_K7SjyujGIlF-1kRjTrOAI; hnUserTicket=b80e6b3a-38a3-432c mur816dmuraeb0376228bd; hnUserId=870048250; Hm_lpvt_91cf34f62b9bedb16460ca36cf192f4c=1604659904', 'sec-fetch-dest':' empty', 'sec-fetch-mode':' cors', 'sec-fetch-site':' same-site', 'user-agent':' Mozilla/5.0 (Windows NT 10.0 WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36', 'xKH64WG5WL1GXPNG: '0KH64WG5WL1GXPNG,' xMeiMeiMeiL1GXPNG: '5000WG5WL1GXPNGG:' 5000WG5WL1GXPNG: 'c10e4e9a-5e19-4ba2WL5WL1KH64WL1c9e660a8d5' KH64WG5WL1c9e660a8d5' / gongying/ {} / '.format (page_id),' Xmuri client sidestep: 'Song0KH64W0GT18JX07L copyright,' xMurclientMuttink: 'b80e6b3a-38a3-432cMube 816dmuraeb0376228bdbath,' xmurclientwaytimekeeper: '1604659611092,' xmurhnlyjobless:'If you see these message, I hope you dont hack us, I hope you can join us! Please visit https://www.cnhnkj.com/job.html',} response_3 = requests.post (url=get_phone_url, json=data, headers=head) html_data = response_3.json () if html_data ['errorCode'] = = 0: dit = {' title': title, 'Update time': update_time, 'Price': price, 'Shipping address': ads Contact information: html_data ['data'] [' messageTitle']} csv_writer.writerow (dit) print (dit) else: dit = {'title': title, 'Update time': update_time, 'Price': price, 'Shipping address': ads 'contact information': 'merchant sets do not disturb'} csv_writer.writerow (dit) print (dit)
Save data
F = open ('Huinong information .csv', mode='a', encoding='utf-8-sig', newline='') csv_writer = csv.DictWriter (f, fieldnames= ['title', 'update time', 'price', 'shipping address', 'contact information']) csv_writer.writeheader () effect diagram
On how to use Python to climb to benefit the agricultural network Apple data is shared here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.
