Shulou(Shulou.com)06/01 Report--

This article is about how python crawled the street information of Meituan's website. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

At present, the code is still being written to simulate the login, so it is being updated slowly.

Then the last code, and then there is information about crawling districts, counties and streets.

With the last provincial and municipal information, the city-level information is taken out from the database. According to the url requested by Meituan, it is found that only the abbreviation of Pinyin can be used to form a new url to access.

According to this, crawl the data

First go to get the data from the database, take a screenshot directly, return a generator, understand the knowledge of the generator yourself, and return the city's id and its pinyin abbreviations.

The next step is to splice url, send requests, get district and county data, part of the source code.

Reassemble the url into the list, override the start_requests method, use the request.meta method to pass additional data, which can pass any data, and use response to get the passed data

The next step is to parse the required data, save the data as defined in items, and save it to the database through a pipeline. The data in the following figure is derived from the data, district and county data (3092).

Data for streets (20593)

Thank you for reading! This is the end of the article on "how python crawls the street information of Meituan website". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it out for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.