Use scrapy to easily climb Douban 9-point list books and store them in mysql database

Note: Most of the contents refer to http://www.cnblogs.com/voidsky/p/5490798.html, but the original text is not stored in the database.

First create a project douban9fen

kuku@ubuntu:~/pachong$ scrapy startproject douban9fenNew Scrapy project 'douban9fen', using template directory '/usr/local/lib/python2.7/dist-packages/scrapy/templates/project', created in: /home/kuku/pachong/douban9fenYou can start your first spider with: cd douban9fen scrapy genspider example example.comkuku@ubuntu:~/pachong$ cd douban9fen/

First, we need to determine the information we want to capture, including three fields: (1) title,(2) rating, and (3) author

Then, let's analyze the next, using Firefox browser, go to www.douban.com/doulist/1264675/

Press F12 to debug the above page

Follow steps 1, 2, and 3 to view the div to which each object belongs, and close the debug window.

Furthermore, right-click in the page to view the page source code, and in the page source code to view the place where the div tag in Search 3 is class bd doulist-subject

According to the principle of first big then small, we first use bd doulist-subject to find each book, and then extract the information in the loop.

Extract book frame:

'//div[@class="bd doulist-subject"]'

Extraction topic:

'div[@class="title"]/a/text()'

Extraction score:

'div[@class="rating"]/span[@class="rating_nums"]/text()'

Extract author: (Use regular convenient point here)

'(.*?) show databases;+--------------------+| Database |+--------------------+| information_schema || csvt04 || douban9fen || doubandianying || mysql || performance_schema || web08 |+--------------------+7 rows in set (0.00 sec)mysql> use douban9fen;Reading table information for completion of table and column namesYou can turn off this feature to get a quicker startup with -ADatabase changedmysql> show tables;+----------------------+| Tables_in_douban9fen |+----------------------+| douban9fen |+----------------------+1 row in set (0.00 sec)mysql> select * from douban9fen;

Shows that it can be successfully written to the database.