The principle and example that crawler ip pool can change ip

This article mainly explains "the principle and example of how the reptile ip pool can change ip". Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Next let the editor to take you to learn "crawler ip pool can change the principle of ip and examples to illustrate" it!

Principle

1. The enclosure uses an ordered collection of Redis to represent the deduplication and status of dialing.

At the same time, it is also the central module and basic module, connecting other modules.

2. The acquisition module obtains agents from the agent website on a regular basis.

Transfer the acquired agent to the enclosure and save it to the database.

3. The detection module obtains all agents from the enclosure on a regular basis.

The agent is detected and different identities are set for the agent according to different detection results.

4. The interface module connects the service interface through WebAPI.

Interface connects to the database and returns available agents in the form of Web.

Example

From multiprocessing import Processimport timefrom api import appfrom getter import Getterfrom tester import TesterTESTER_CYCLE = 20GETTER_CYCLE = 20TESTER_ENABLED = TrueGETTER_ENABLED = TrueAPI_ENABLED = Trueclass Scheduler (): "scheduling module" def schedule_tester (self " Cycle=TESTER_CYCLE): "" scheduled test agent: param cycle:: return: "" tester = Tester () while True: print ("the tester is running") tester.run () time.sleep (cycle) def schedule_getter (self) Cycle=GETTER_CYCLE): "" timing acquisition agent: param cycle:: return: "" getter = Getter () while True: print ("start crawling agent") getter.run () time.sleep (cycle) def schedule_api (self): "" Open api: return: "" app.run () def run (self): print ("Agent pool is running") if TESTER_ENABLED: test_process = Process (target=self.schedule_tester) test_process.start () if GETTER_ENABLED: getter_process = Process (target=self.schedule_getter) Getter_process.start () if API_ENABLED: api_process = Process (target=self.schedule_api) api_process.start () if _ _ name__ = ='_ _ main__': a = Scheduler () a.run () so far I believe that everyone has a deeper understanding of the "reptile ip pool can change the principle of ip and examples", might as well come to the actual operation of it! Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!