Centos7 production environment IP proxy pool (python)
Recently, I was studying crawlers and needed to deploy the IP proxy pool in front of me, so I found proxy pool in open source China. It can automatically crawl the IP of several free IP proxy websites in China, and verify the availability of IP in real time, and the database is SSDB.
IP proxy Pool website:
Http://www.data5u.com/
Http://www.data5u.com/free/
Http://www.data5u.com/free/gngn/index.shtml
Http://www.data5u.com/free/gnpt/index.shtml
Http://www.66ip.cn/
Http://www.ip181.com/
Http://www.xicidaili.com/nn
Http://www.xicidaili.com/nt
Http://www.goubanjia.com/free/gngn/index.shtml
Http://www.xdaili.cn/ipagent/freeip/getFreeIps?page=1&rows=10
2 yum-y install git
3 yum-y install wget
4 yum install curl-devel expat-devel gettext-devel openssl-devel zlib-devel
5 yum install gcc perl-ExtUtils-MakeMaker epel-release gcc-c++
8 cd / usr/src/
16 wget https://www.kernel.org/pub/software/scm/git/git-2.9.5.tar.gz # install the cloning tool git
17 tar-xzf git-2.9.5.tar.gz
18 cd git-2.9.5
19 make prefix=/usr/local/git all
20 make prefix=/usr/local/git install
21 echo "export PATH=$PATH:/usr/local/git/bin" > > / etc/bashrc
22 source / etc/bashrc
23 cd..
24 git clone https://github.com/jhao104/proxy_pool.git # Clone proxy_pool
25 cd proxy_pool/
26 python-V # View python version 2.7.5
27 yum-y install python34 # install python 3.4
28 wget-no-check-certificate https://bootstrap.pypa.io/get-pip.py
29 python3 get-pip.py # install pip
30 pip install-r requirements.txt # proxy_pool installation dependency package
32 cd / usr/local/
33 git clone https://github.com/ideawu/ssdb.git # Clone SSDB
34 cd ssdb
35 yum-y install autoconf
37 cd deps/snappy-1.1.0/ # compile Snappy
38. / configure
39 make
40 cd / usr/local/ssdb
41 make # install SSDB
42 make install
43 ln-sf / usr/local/ssdb/ssdb-server / usr/local/bin/ssdb-server
44 ln-sf / usr/local/ssdb/tools/ssdb-cli / usr/local/bin/ssdb-cli
45 ln-sf / usr/local/ssdb/tools/ssdb-dump / usr/local/bin/ssdb-dump
46 ln-sf / usr/local/ssdb/tools/ssdb-repair / usr/local/bin/ssdb-repair
47 ln-sf / usr/local/ssdb/tools/ssdb.sh / etc/rc.d/init.d/ssdb
48 chkconfig-add ssdb
49 chkconfig ssdb on
50 systemctl stop firewalld.service # turn off the firewall
51 systemctl disable firewalld.service
52 firewall-cmd-state
53 pip install-upgrade pyssdb
Start proxy_pool sequence
1 cd / usr/local/ssdb
2. / ssdb-server-d. / ssdb.conf-s start
Note: when the ssdb service starts, it will generate a ssdb.pid file in the var directory. When ssdb shuts down unexpectedly, this file will not be deleted, so when the ssdb service is restarted, there will be an error, so you can restart manually by running the following two lines of command.
. / ssdb-server ssdb.conf-s stop
. / ssdb-server-d. / ssdb.conf-s restart
3 cd / usr/src/proxy_pool/Run
4 python3.4 main.py # start with python3.4
Client access
Http://ip:5010/
Http://ip:5010/get (randomly call an IP and port)
Http://ip:5010/get_all (access all available IP and port)
For specific use, please refer to https://github.com/jhao104/proxy_pool
This article provides details of the deployment under cenots7. I would like to thank contributor and j_hao104 for their selfless dedication!
