How to use Python crawler Scrapy to crawl Chinese seismic data

How to use Python crawler Scrapy to crawl Chinese seismic data, for this problem, this article details the corresponding analysis and solution, hoping to help more friends who want to solve this problem find a simpler and easier way.

preface

I always feel that there are a lot of earthquake news recently, so I want to climb a wave of earthquake-related data to see.

development tools

Python version: 3.6.4

Related modules:

scrapy module;

jieba module;

pyecharts module;

folium module;

wordcloud module;

fake_useragent module;

And some Python modules.

environment construction

Install Python and add it to the environment variables, pip install the relevant modules you need.

data crawling

First, create a new project:

Then open the China Seismological Network:

http://news.ceic.ac.cn/index.html

It is found that seismic data within a specified time and latitude range can be queried by using the query function that comes with the website:

picture

OK, start writing code.

First define the data we want to crawl in the items.py file:

class EarthquakeItem(scrapy.Item):

Create a main.py function under spiders folder to write our crawler main program, where we crawl all the earthquake data recorded on the website since January 1, 2000 (in fact, after running the code, we found that there are only earthquake data T_T from 2012 in the website):

'''main'''

OK, that's it. Just run the following command under the first earthquake folder:

scrapy crawl earthquake -o eqdata.json -t json

The crawled data is saved in the eqdata.json file.

About how to use Python crawler Scrapy to crawl Chinese seismic data questions to share here, I hope the above content can be of some help to everyone, if you still have a lot of doubts, you can pay attention to the industry information channel to learn more related knowledge.