Get the App
SLTechnology News&Howtos  ›  Internet Technology  › 

The reptile of java

Shulou Source: shulou.com Published: 2022-06-03 06:43:51 11月23日 Update

A recent study of crawlers crawling links to websites:

1. Need to get all hyperlinks

two。 Exclude the crawled links and remove the heavy ones

3. Research on the breadth and depth of reptiles (ps: it has not been thoroughly studied so far)

The following is the implementation code:

Import java.io.BufferedReader;import java.io.InputStream;import java.io.InputStreamReader;import java.net.URL;import java.net.URLConnection;import java.util.ArrayList;import java.util.HashMap;import java.util.HashSet;import java.util.List;import java.util.Map;import java.util.Set;import java.util.regex.Matcher;import java.util.regex.Pattern;import javax.swing.plaf.synth.SynthSpinnerUI;public class study {private static List waitforUrl=new ArrayList () / / store the crawled url and wait for the crawled private static Set goforUrl=new HashSet (); / / store the crawled url private static Map allUrldepth=new HashMap (); / / judge the crawling depth of all url by private static int Maxdepth=2; public static void main (String [] args) {String urlstr= ".; study.gourl (urlstr, 1) } public static void gourl (String urlstr,int depath) {if (! (goforUrl.contains (urlstr) | | depath > Maxdepth) {goforUrl.add (urlstr); try {URL url=new URL (urlstr); URLConnection urlConn=url.openConnection (); / / establish url link InputStream is=urlConn.getInputStream () / / get the page content through the link to InputStreamReader isr=new InputStreamReader (is, "utf-8"); / / convert the byte stream into a byte stream BufferedReader br=new BufferedReader (isr); / / read the byte stream StringBuffer sb=new StringBuffer (); / / instantiate the StringBuffer to store the read data String line=null While ((line=br.readLine ())! = null) {sb.append (line); / / System.out.println (line); Pattern p = Pattern.compile ("

Tags: Links threads problems methods storage research crawlers objects tags depth greed between code content first half simultaneously characters bytes instances breadth Apple Docker Huawei Linux macOS MariaDB Microsoft MySQL NVidia OPPO Reno Shulou Tech Info Microsoft Shulou Information Shulou Technology Apple