How to avoid being anti-crawled when crawling
This article mainly introduces "how to prevent anti-crawling when crawling". In daily operation, I believe that many people have doubts about how to prevent anti-crawling when crawling. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful for everyone to answer the doubt of "how to prevent being anti-crawling" when crawling! Next, please follow the editor to study!
1. Counterfeit User-Agent. Set User-Agent to User-Agent in the browser to fake browser access.
Headers = {'User-Agent':'Mozilla/5.0 (X11; Linux x86 / 64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'} resp = requests.get (url,headers = headers)
2. Using proxies, multiple proxy IP can be used in the ip proxy pool to prevent the same IP from being sealed due to excessive requests.
Proxies = {'http':' http://10.10.10.10:8765','https':'https://10.10.10.10:8765'}resp = requests.get (url,proxies = proxies) at this point, the study on "how to prevent crawlers from being anti-crawled" is over. I hope I can solve everyone's doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!
