How does Python crawl the music material of the website
This article mainly explains "how Python crawls the music material of the website". Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn "Python how to crawl website music material"!
Preface basic development environment
Python 3.6
Pycharm
The use of related modules import osimport concurrent.futuresimport requestsimport parsel
Install Python and add it to the environment variable, and pip installs the relevant modules you need.
First, determine the demand
If you want to verify that this link is the real download address for audio, you can copy the link and paste it into a new window.
Https://downsc.chinaz.net/Files/DownLoad/sound1/202102/s830.mp3
The old idea is that some of the parameters in the copy link are searched in the developer's tools, and it is clear that S830 is the ID of audio.
Search S830 to find the source and find the download address in the web page. After obtaining the audio download address, you need to splice the url yourself.
Web data is not complex, relatively speaking, it is relatively simple.
1. Request the current web page data to obtain the audio address and audio title.
2. Just save the download
Third, code implementation
Get audio ID and audio title
Def main (html_url): html_data = get_response (html_url). Text selector = parsel.Selector (html_data) lis = selector.css ('# AudioList. Container. Audio-item') for li in lis: name = li.css ('.name::text'). Get (). Strip () src = li.css (' audio::attr (src)'). Get () audio_url = 'https:' + src save (name) Audio_url) print (name, audio_url)
Save data
Def save (name, audio_url): header = {'Upgrade-Insecure-Requests':' 1century, 'User-Agent':' Mozilla/5.0 (Windows NT 10.0) WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36'} audio_content = requests.get (url=audio_url, headers=header). Content path = 'audio\\' if not os.path.exists (path): os.mkdir (path) with open (path + name + '.mp3', mode='wb') as f: f.write (audio_content)
Here you want to give a new headers parameter, otherwise you won't be able to download it. The code runs all the time, but there is no response.
Multi-thread crawling
If _ _ name__ ='_ main__': executor = concurrent.futures.ThreadPoolExecutor (max_workers=5) for page in range (1,31): url = f 'https://sc.chinaz.com/yinxiao/index_{page}.html' # main (url) executor.submit (main, url)
At this point, I believe you have a deeper understanding of "Python how to crawl website music material". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!
