How the python crawler urljoin () generates links

This article is about how the python crawler urljoin () generates links. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

With the urlunparse () and urlunsplit () methods, we can merge links, but only if there is an object of a specific length, and each part of the link should be clearly separated. In addition, there is the urljoin () method.

1. Urljoin () can generate links. Provide base_url (basic link) as the first parameter and the new link as the second parameter.

2. This method will analyze the scheme, netloc and path of base_url, supplement the missing parts of the new link, and finally return the result.

Example

From urllib.parse import urljoin print (urljoin ('http://www.baidu.com',' FAQ.html')) print (urljoin ('http://www.baidu.com',' https://cuiqingcai.com/FAQ.html'))print(urljoin('http://www.baidu.com/about.html', 'https://cuiqingcai.com/FAQ.html'))print(urljoin('http://www.baidu.com/about.html',) 'https://cuiqingcai.com/FAQ.html?question=2'))print(urljoin('http://www.baidu.com?wd=abc',' https://cuiqingcai.com/index.php'))print(urljoin('http://www.baidu.com','? category=2#comment') print (urljoin ('www.baidu.com','? category=2#comment')) print (urljoin ('www.baidu.com#comment','? category=2'))

Output

Http://www.baidu.com/FAQ.htmlhttps://cuiqingcai.com/FAQ.htmlhttps://cuiqingcai.com/FAQ.htmlhttps://cuiqingcai.com/FAQ.html?question=2https://cuiqingcai.com/index.phphttp://www.baidu.com?category=2#commentwww.baidu.com?category=2#commentwww.baidu.com?category=2, thank you for your reading! This is the end of the article on "python crawler urljoin () how to generate links". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it out for more people to see!