How to climb the data of Amap Metro Line and Station by Python
This article is about how Python crawls Gaode map subway line and station data. Xiaobian thinks it is quite practical, so share it with everyone for reference. Let's follow Xiaobian and have a look.
preface
The text and pictures of this article come from the network, only for learning and communication, and do not have any commercial use. The copyright belongs to the original author. If you have any questions, please contact us in time for processing.
# coding=utf-8import requestsimport timeimport jsonimport astimport osimport utilsfrom lxml import etreePAGE_URL = 'http://map.amap.com/subway/index.html?& 1100'DATA_URL = 'http://map.amap.com/service/subway? srhdata='HEADER = { "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"}def fetchAllCity(url, header): r = requests.get(url, header) html = r.content element = etree.HTML(html) options = element.xpath("//a[contains(@class, 'city')]") cities = [] for option in options: city = { 'id': option.get('id'), 'name': option.get('cityname'), 'text': option.text } cities.append(city) return citiesdef parseCityData(citys): lw = open('./ lwkt.txt', 'w') lw.write('wkt' + '\n') pw = open('./ pwkt.txt', 'w') pw.write('wkt' + '\n') for city in citys: parseCityPointFromApi(city, lw, pw)def parseCityPointFromApi(city, lw, pw): url = DATA_URL + "{}_drw_{}.json".format(city['id'], city['name']) print(url) json_str = requests.get(url).text res = json.loads(json_str) res = res['l'] for r in res: st = r['st'] coords = [] for s in st: _coords = s.get('sl', '').split(',') coords.append(_coords) pwkt = '"POINT ({} {})"'.format(_coords[0], _coords[1]) pw.write(pwkt + '\n') new_coords = ','.join(['%s %s' % (coord[0], coord[1]) for coord in coords]) lwkt = '"LINESTRING(%s)"' % new_coords lw.write(lwkt + '\n')def main(): cities = fetchAllCity(PAGE_URL, HEADER) print(cities) parseCityData(cities)if __name__ == '__main__': main() Thank you for reading! About "Python how to crawl Gaode map subway line and site data" This article is shared here, I hope the above content can be of some help to everyone, so that everyone can learn more knowledge, if you think the article is good, you can share it to let more people see it!
