How to use the urlparse method of python crawler

This article mainly introduces how to use python reptile urlparse method, the article is very detailed, has a certain reference value, interested friends must read it!

1. URL identification and segmentation can be realized, and standard URL can be split by using urlparse() method.

from urllib.parse import urlparseresult = urlparse('http://www.baidu.com/index.html;user? id=5#comment')print(type(result), result)

Here we use the urlparse() method to parse a URL. First, output the type of the parsing result, and then output the result.

The results are as follows:

ParseResult(scheme='http', netloc='www.baidu.com', path='/index.html', params='user', query='id=5', fragment='comment')

There is also API usage, which has three parameters. urlstring: is required, i.e. the URL to be resolved.

scheme Default protocol (such as http or https)

from urllib.parse import urlparse result = urlparse('www.baidu.com/index.html;user? id=5#comment', scheme='https')print(result)

The results are as follows:

ParseResult(scheme ='https ', netloc ='', path ='www.baidu.com/index.html', params ='user', query ='id = 5', fragment ='comment ') The above is "python crawler urlparse method how to use" All the contents of this article, thank you for reading! Hope to share the content to help everyone, more relevant knowledge, welcome to pay attention to the industry information channel!