How to use HTTP Agent with user name and password Authentication by Selenium + Firefox

Editor to share with you Selenium + Firefox how to use the HTTP agent with user name and password authentication, I believe most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to know it!

Students who are familiar with Firefox know that Firefox cannot set a user name and password when configuring HTTP agents. Most of the paid HTTP agents need to authenticate their username and password (some also support IP whitelist, but only if your IP needs to be fixed). This makes it very inconvenient to automate with Selenium + Firefox, because each time you start a new browser instance, an authorization verification window pops up and is asked to enter a user name and password (as shown in the following figure), interrupting the automation process.

In addition, Firefox does not provide command-line arguments to set the username and password (PS:phantomjs has arguments like-- proxy-auth). Is there really no solution?

Through research, Luo Zhipeng's technicians have finally found an effective and stable solution:

First introduce an important role, its home page is https://addons.mozilla.org/en-US/firefox/addon/close-proxy-authentication/. Close-proxy-authentication implements the automatic completion of proxy username and password authentication (Proxy Authentication), which provides an extensions.closeproxyauth.authtoken parameter to set the agent's username and password, whose value is the base64-encoded username and password pair (as shown in the following figure). Close-proxy-authentication uses this value to construct a "Proxy-Authorization: Basic dGVzdDp0ZXN0" hair to the proxy server for authentication, which is how it works.

We want to use this plug-in to automatically complete HTTP agent authentication in Selenium + Firefox. The process is as follows:

(1) add the close-proxy-authentication plug-in dynamically through the Firefox configuration option (no plug-ins are loaded by default)

(2) set the IP and port parameters of the HTTP proxy through the configuration option

(3) set the extensions.closeproxyauth.authtoken value to base64encode ("user name: password")

(4) the close-proxy-authentication plug-in will automatically complete the authorization verification process of the agent when you visit the website, and the authentication window will not pop up again.

Here is the complete test code:

View plain copy to clipboard print?

# coding: utf-8

# selenium_firefox_proxy_auto_auth.py

Import sys

Import time

From base64 import b64encode

From selenium import webdriver

# path to the close-proxy-authentication plug-in

# https://addons.mozilla.org/en-US/firefox/addon/close-proxy-authentication/

PROXY_HELPER_DIR = 'close_proxy_authentication-1.1.xpi'

Def test ():

# HTTP (S) type proxy parameters

Proxy_host = '221.229.204.91'

Proxy_port = 8888

Proxy_username ='*'

Proxy_password ='*'

Fp = webdriver.FirefoxProfile ()

# add Agent Authentication plug-in

Fp.add_extension (PROXY_HELPER_DIR)

# set proxy parameters

Fp.set_preference ('network.proxy.type', 1)

Fp.set_preference ('network.proxy.http', proxy_host)

Fp.set_preference ('network.proxy.http_port', proxy_port)

# set authtoken (that is, the username and password for agent authentication) to the close-proxy-authentication plug-in

Credentials ='{}: {} '.format (proxy_username, proxy_password)

Credentials = b64encode (credentials)

Fp.set_preference ('extensions.closeproxyauth.authtoken', credentials)

Firefox = webdriver.Firefox (firefox_profile=fp)

# access http://httpbin.org/ip to echo the current IP

Firefox.get ('http://httpbin.org/ip')

Time.sleep (1000)

If _ _ name__ = ='_ _ main__':

Test ()

Test environment:

View plain copy to clipboard print?

Firefox V53.0

Geckodriver v0.18.0

Selenium V3.8.0

Close-proxy-authentication V1.1

The above environment involves packing and downloading files. Address: http://pan.webscraping.cn:8000/index.php/s/PMDjc77gbCFJzpO

Special attention should be paid to:

(1) the latest version of close-proxy-authentication is currently v1.1, and it is not compatible with the latest version of Firefox. The technical test of Luo Zhipeng found that versions below Firefox v56.0 are compatible with close-proxy-authentication v1.1.

(2) different versions of geckodriver (webdriver program of Firefox) support different versions of Firefox. For more information on which versions are supported, please see the releases page of geckodriver.

The test results are shown in the following figure. Instead of popping up the authentication window, visiting httpbin.org/ip directly echoes the IP of the HTTP agent:

The above is all the contents of the article "Selenium + Firefox how to use HTTP Agent with user name and password Authentication". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!