What is the introduction of the Python crawler?
In this issue, Xiaobian will bring you an introduction to Python crawlers. The article is rich in content and analyzed and described from a professional perspective. After reading this article, I hope you can gain something.
About Python crawler introduction, what is crawler? What is its architecture like? Xiao Bian summed up these points for everyone here.
1. What is a reptile?
Crawler: A program that automatically grabs information from the Internet and grabs information valuable to us from the Internet.
II. Python crawler architecture
Python crawler architecture consists of five parts: scheduler, URL manager, web downloader, web parser, application (valuable data crawled).
Scheduler: CPU equivalent to a computer, mainly responsible for scheduling URL manager, downloader, parser coordination work.
URL Manager: Including the URL address to be crawled and the URL address that has been crawled, preventing repeated URL crawling and circular URL crawling. The URL Manager is implemented in three ways: memory, database, and cache database.
Web Downloader: Download web pages by passing in a URL, convert web pages into a string, web downloader has urllib2 (Python official base module) including login, proxy, and cookies, requests(third-party package)
Web Parser: Parse a web string, extract our useful information according to our requirements, or parse it according to the parsing method of DOM tree. Web parsers have regular expressions (Intuitive, converting web pages into character strings extracts valuable information by fuzzy matching. When documents are complex, this method will be very difficult to extract data), html.parser (Python comes with), beautifulsoup (third-party plug-ins, you can use Python's own html.parser to parse, you can also use lxml to parse, compared to other kinds of powerful), lxml (third-party plug-ins, you can parse xml and HTML), html.parser and beautifulsoup and lxml are all parsed in the way of DOM tree.
Application: An application consisting of useful data extracted from a web page.
Third, the installation of the third-party library Beautiful Soup
BeautifulSoup:Python's third-party plugin for extracting data from xml and HTML, www.crummy.com/software/BeautifulSoup/
1. Install Beautiful Soup
Open cmd (command prompt), enter scripts in Python (Python version 2.7) installation directory, enter dir to see if there is pip.exe, if you can use Python's own pip command to install, enter the following command to install:
pipinstallbeautifulsoup4
2, test whether the installation is successful
Write a Python file and type:
importbs4
printbs4
Run the file, if it can be output normally, the installation is successful.
The above is what the introduction of Python crawler shared by Xiaobian is. If you happen to have similar doubts, you may wish to refer to the above analysis for understanding. If you want to know more about it, please pay attention to the industry information channel.
