Scrapy randomly changes the User-Agent method
1 Scrapy set User-Agent1.1 to set the USER_AGENT parameter in the settings.py file can achieve 1.1.1 using the fixed User-Agent method
Add the following code to the settings.py file (USER_AGENT is fixed), and you can change the blue font content below by yourself with what kind of agent you use.
USER_AGENT = 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)
1.1.2 using the random User-Agent method
In the settings.py file, you can create an agent list and then randomly take a value from it and assign it to the USER_AGENT variable.
The code is as follows:
Import random
UserAgentList = [
"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)"
"Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
]
USER_AGENT = random.choice (UserAgentList) 1.1.3 use a third-party user-agent library (fake-useragent) to implement the agent setting method
Add the following code to the settings.py file:
From fake_useragent import UserAgent
Ua = UserAgent ()
USER_AGENT = ua.random1.2 1.2.1 by writing the middleware method by writing the user-agent method in the setting file
Write a UserAgentList list in the setting.py file
UserAgentList = [
'Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1500.55 Safari/537.36'
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10, 8) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1309.0 Safari/537.17'
]
Enable DOWNLOADER_MIDDLEWARES in the setting.py file and comment out the default
DOWNLOADER_MIDDLEWARES = {
# 'amazon.middlewares.MyCustomDownloaderMiddleware': 543
'amazon.middlewares.MyUserAgentMiddleware': 543
}
Add the following code to the middlewares.py file:
Rom scrapy.downloadermiddlewares.useragent import UserAgentMiddleware
From amazon.settings import UserAgentList
Import randomclass MyUserAgentMiddleware (UserAgentMiddleware):
Def _ _ init__ (self, user_agent=''):
Self.user_agent = user_agent
Def process_request (self, request, spider):
Agent = random.choice (list (UserAgentList))
Request.headers ['User-Agent'] = agent1.2.2 sets the User-Agent method using the fake-useragent library
Install the fake-useragent library (useragent's library)
Pip install fake-useragent
Import fake-useragent libraries in a middlewares.py file
From fake_useragent import UserAgent
Write a middleware middleware in middlewares.py file
From fake_useragent import UserAgent
From scrapy.downloadermiddlewares.useragent import UserAgentMiddlewareclass MyUserAgentMiddleware (UserAgentMiddleware):
Def _ _ init__ (self, user_agent=''):
Self.user_agent = user_agent
Def process_request (self, request, spider):
Ua = UserAgent ()
Request.headers ['User-Agent'] = ua.random
Enable DOWNLOADER_MIDDLEWARES in the setting.py file and comment out the default
DOWNLOADER_MIDDLEWARES = {
# 'amazon.middlewares.MyCustomDownloaderMiddleware': 543
'amazon.middlewares.MyUserAgentMiddleware': 543
}
