How to use Python to crawl data
This article mainly introduces "how to crawl data with Python". In daily operation, I believe many people have doubts about how to crawl data with Python. Xiaobian consulted all kinds of materials and sorted out simple and easy operation methods. I hope to help you answer the doubts about "how to crawl data with Python"! Next, please follow the small series to learn together!
Goal: Crawl 100 pages of love stories from a given URL
Practice website
Preparation: python: version 3.7, development tools: pycharm, browser: google browser
Thinking:
Crawl web pages with requests+xpath and extract information
Regardless of whether the website has an anti-crawling strategy, the simplest way to carry headers is to use
Cycle write txt file
Encapsulate the main code as a function
A few key points:
Note the encoding format of the web page, which in general can be viewed in conjunction with response.encoding and response.headers. If there is no Content-Type in headers, encoding='utf-8'; if there is Content-Type, charset shall prevail, no charset, ISO-8859-1
Observe the structure of the web page, determine how to write procedures to achieve page turning, where the site is relatively simple, directly to the serial number +1 on the line
As for whether to use beautiful soup or xpath, use it as appropriate. Here, it is more convenient for me to use xpath directly for positioning.
Final effect:
Get 100 pages of love and save it to txt file.
major code
resultant output
Later planned improvements:
Prepare to write data to database
Building websites with flask framework
Prepare to implement a word cloud effect similar to the following.
At this point, the study of "how to crawl data with Python" is over, hoping to solve everyone's doubts. Theory and practice can better match to help you learn, go and try it! If you want to continue learning more relevant knowledge, please continue to pay attention to the website, Xiaobian will continue to strive to bring more practical articles for everyone!
