Shulou(Shulou.com)06/03 Report--

This article introduces the relevant knowledge of "how to solve the problem of asynchronous loading of crawler data". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

1. Built-in browser kernel: built-in browser refers to starting the browser core in the crawler to get the page after js rendering, just like collecting static pages.

There are three kinds of tools commonly used: Selenium, HtmlUnit, and PhantomJs

These tools can help us solve the problem of asynchronous data loading, but they are all inefficient and unstable.

2. Reverse analysis.

The data of the js rendering page is obtained from the backend in the way of Ajax, and we just need to find the appropriate Ajax request connection in order to get the required data. One of the advantages of reverse analysis is that the data obtained are all in json format, which is convenient for analysis; second, the probability of interface change is smaller than that of the page.

Again, Ajax has two drawbacks. In Ajax, you need patience and skill because you need to find what you want in a large number of requests; the reverse parsing method is helpless for pages rendered by JavaScript.

These are two solutions for asynchronous data loading, which must be selected on a case-by-case basis. Built-in browsers are more reliable for some pages rendered using JavaScript fragments. In other cases, reverse parsing is better because its performance and stability are more reliable than the built-in browser kernel.

Crawlers encounter data asynchronous loading, try these two methods. Nowadays, many projects are separated from the front end and back end, which will make the data asynchronous loading problem more prominent.

This is the end of the content of "how to solve the problem of asynchronous loading of crawler data". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.