How to capture the resources of Ali cloud disk with Python
This article mainly introduces the relevant knowledge of "how to grab Alibaba cloud disk resources with Python". Xiaobian shows you the operation process through actual cases. The operation method is simple, fast and practical. I hope this article "how to grab Alibaba cloud disk resources with Python" can help you solve the problem.
web analytics
This site has two search routes: search route 1 and search route 2, which is used in this article.
Open the network under the control panel and see a get request for seach.html.
There are several parameters, four key parameters:
Page: Number of pages,
Keyword: Search keyword
category: file category, all, video, image, doc, audio, zip, others, all by default in script
search_model: The route to search
Also in the control panel, see this page jump to Alibaba cloud disk to get the real link is above the title. Use bs4 to parse the div(class=resource-item border-dashed-yee) tag on the page to get the address of the jump network disk, and parse the p tag under div to get the resource date.
Capture and parse
First install the required bs4 third-party library for parsing the page.
pip3 install bs4
Below is the script code for crawling and parsing web pages, sorted in descending order by date.
import requestsfrom bs4 import BeautifulSoupimport stringword = input ('Please enter the name of the resource to search for:') headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36'}result_list = []for i in range(1, 11): print ('Searching page {}'. format(i)) params = { 'page': i, 'keyword': word, 'search_folder_or_file': 0, 'is_search_folder_content': 0, 'is_search_path_title': 0, 'category': 'all', 'file_extension': 'all', 'search_model': 0 } response_html = requests.get('https://www.alipanso.com/search.html', headers = headers,params=params) response_data = response_html.content.decode() soup = BeautifulSoup(response_data, "html.parser"); divs = soup.find_all('div', class_='resource-item border-dashed-eee') if len(divs)
