How to use python to crawl weather data

Editor to share with you how to use python to climb weather data, I believe most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's learn about it!

Preface

Practice a crawler every day and climb a relatively simple website

Import requestsfrom bs4 import BeautifulSoupimport pandas as pdimport timedef get_data (url): resp = requests.get (url) html = resp.content.decode ('gbk') soup = BeautifulSoup (html,' html.parser') tr_list = soup.find_all ('tr') dates, conditions, tempmin, tempmax = [] [] for data in tr_list [1:]: sub_data = data.text.split () dates.append (sub_data [0]) conditions.append (''.join (sub_ data [1: 3])) tempmax.append (sub_data [3]) tempmin.append (sub_data [5]) _ data = pd.DataFrame () _ data ["date"] = Dates _ data ['Weather condition'] = conditions _ data ['maximum temperature'] = tempmax _ data ['minimum temperature'] = tempmin return _ data#data_1_month = get_data ('this is the URL for any January data') only need to be modified here Specific examples are as follows: print ('crawling January data') data_1_month = get_data ('http://www.tianqihoubao.com/lishi/huichang/month/201101.html')print(' grabbing February data') data_2_month = get_data ('http://www.tianqihoubao.com/lishi/huichang/month/201102.html')print(' crawling March data') data_3_month = get_ Data ('http://www.tianqihoubao.com/lishi/huichang/month/201103.html')print(' is crawling April data') data_4_month = get_data ('http://www.tianqihoubao.com/lishi/huichang/month/201104.html')print(' is grabbing May data') data_5_month = get_data ('http://www.tianqihoubao.com/lishi/huichang/month/201105.html')print(' is positive In crawling June data') data_6_month = get_data ('http://www.tianqihoubao.com/lishi/huichang/month/201106.html')print(' is crawling July data') data_7_month = get_data ('http://www.tianqihoubao.com/lishi/huichang/month/201107.html')print(' is crawling August data') data_8_month = get_data ('http://www.tianqihoubao .com / lishi/huichang/month/201108.html') print ('fetching September data') data_9_month = get_data ('http://www.tianqihoubao.com/lishi/huichang/month/201109.html')print(' is grabbing October data') data_10_month = get_data ('http://www.tianqihoubao.com/lishi/huichang/month/201110.html')print(' is crawling November data') Data_11_month = get_data ('http://www.tianqihoubao.com/lishi/huichang/month/201111.html')print(' is crawling December data') data_12_month = get_data ('http://www.tianqihoubao.com/lishi/huichang/month/201112.html')print(' data crawled successfully! Data for that year are being consolidated Please wait a moment.') time.sleep (3) data = pd.concat ([data_1_month, data_2_month, data_3_month, data_4_month, data_5_month, data_6_month, data_7_month, data_8_month, data_9_month, data_10_month, data_11_month Data_12_month]. Reset_index (drop=True) print (data) print ('data integration is complete!') # data crawling results will be saved under the same path of this code. Above are all the contents of the article "how to use python to crawl weather data". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!