How to understand the web server crawler protocol robot in linux
In this issue, the editor will bring you about how to understand the web server crawler protocol robot in linux. The article is rich in content and analyzes and describes it from a professional point of view. I hope you can get something after reading this article.
Summary of common configuration contents of robots.txt: (prompt to put it directly under the web root directory, you can use Baidu webmaster tool robot tool to test)
User-agent: Baiduspider
Allow: /
Disallow: / * .jpg$
Disallow: / * .jpeg$
Disallow: / * .gif$
Disallow: / * .png$
Disallow: / * .bmp$
User-agent: Googlebot
Allow: /
Disallow: / * .jpg$
Disallow: / * .jpeg$
Disallow: / * .gif$
Disallow: / * .png$
Disallow: / * .bmp$
User-agent: 360Spider
Allow: /
Disallow: / * .jpg$
Disallow: / * .jpeg$
Disallow: / * .gif$
Disallow: / * .png$
Disallow: / * .bmp$
User-agent: msnbot
Allow: /
Disallow: / * .jpg$
Disallow: / * .jpeg$
Disallow: / * .gif$
Disallow: / * .png$
Disallow: / * .bmp$
User-agent: Sosospider
Allow: /
Disallow: / * .jpg$
Disallow: / * .jpeg$
Disallow: / * .gif$
Disallow: / * .png$
Disallow: / * .bmp$
User-agent: YoudaoBot
Allow: /
Disallow: / * .jpg$
Disallow: / * .jpeg$
Disallow: / * .gif$
Disallow: / * .png$
Disallow: / * .bmp$
Root@ubuntu:/var/www# cat robots.txt
User-agent: Baiduspider
Allow: /
Disallow: / * .jpg$
Disallow: / * .jpeg$
Disallow: / * .gif$
Disallow: / * .png$
Disallow: / * .bmp$
User-agent: Googlebot
Allow: /
Disallow: / * .jpg$
Disallow: / * .jpeg$
Disallow: / * .gif$
Disallow: / * .png$
Disallow: / * .bmp$
User-agent: 360Spider
Allow: /
Disallow: / * .jpg$
Disallow: / * .jpeg$
Disallow: / * .gif$
Disallow: / * .png$
Disallow: / * .bmp$
User-agent: msnbot
Allow: /
Disallow: / * .jpg$
Disallow: / * .jpeg$
Disallow: / * .gif$
Disallow: / * .png$
Disallow: / * .bmp$
User-agent: Sosospider
Allow: /
Disallow: / * .jpg$
Disallow: / * .jpeg$
Disallow: / * .gif$
Disallow: / * .png$
Disallow: / * .bmp$
User-agent: YoudaoBot
Allow: /
Disallow: / * .jpg$
Disallow: / * .jpeg$
Disallow: / * .gif$
Disallow: / * .png$
Disallow: / * .bmp$
User-agent: bingbot
Allow: /
Disallow: / * .jpg$
Disallow: / * .jpeg$
Disallow: / * .gif$
Disallow: / * .png$
Disallow: / * .bmp$
User-agent: Sogou web spider/4.0
Allow: /
Disallow: / * .jpg$
Disallow: / * .jpeg$
Disallow: / * .gif$
Disallow: / * .png$
Disallow: / * .bmp$
User-agent: *
Disallow: /
The above is the editor for you to share how to understand the web server crawler protocol robot in linux. If you happen to have similar doubts, please refer to the above analysis. If you want to know more about it, you are welcome to follow the industry information channel.
