How to use Nutch

This article mainly explains "how to use Nutch". The content in the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn how to use Nutch.

Nutch has now reached version 2.2.2, and version 1.x has been updated to 1.8. here, for example, the API of some command-line tools in 1.8has changed, so it is not easy to get started.

# transfer to run Nutch#

Download and install Nutch

Under ${NUTCH_HOME}, mkdir urls

Cd urls

Touch seed.txt

Edit seed.txt, write: http://nutch.apache.org

Edit ${NUTCH_HOME} / conf/regex.urlfilter.txt

Replace

'# accept anything else+.

With

+ ^ http://([a-z0-9]*\.)*nutch.apache.org/

Crawl web page: bin/nutch crawl urls-dir crawl-depth 3-topN 5 Note: the 1.8 version of this command has changed

# install Solr#

Download and install Solr, the latest version of 4.8when I used it.

Cd ${SOLR_HOME} / example

Java-jar start.jar

Verify installation: http://localhost:8983/solr/

# Nutch and Solr integration # Note here: there are only two things that the document says:

Replace ${SOLR_HOME} / example/solr/collection1/conf/schema.xml with ${NUTCH_HOME} / conf/schema-solr4.xml and rename schema-solr4.xml to schema.xml

Add after line 351 in schema.xml (in fact, just add it in the types tag):

The integration is done here. Now all you have to do is restart solr and index the data crawled by Nutch to solr with the following command.

Under ${NUTCH_HOME}, run:

Bin/nutch solrindex http://127.0.0.1:8983/solr/ crawl/crawldb-linkdb crawl/linkdb crawl/segments/*

Then visit: http://localhost:8983/solr/

Thank you for your reading, the above is the content of "how to use Nutch", after the study of this article, I believe you have a deeper understanding of how to use Nutch, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!