Get the App
SLTechnology News&Howtos  ›  Internet Technology  › 

Modify hadoop script to change the loading order of jar in CLASSPATH

Shulou Source: shulou.com Published: 2022-06-03 06:22:47 11月23日 Update

Let's talk about the environment first. there are two clusters, one is new and the other is old. We are going to debug the new one and then turn off the old one.

New: Cloudera Express 5.6.0cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc

Old: Cloudera Express 5.0.5 cdccccmcccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc

A problem is found in the process of building a new cluster. Execute the following command to create an index on the lzo file. Job cannot be submitted to the specified queue in the new cluster. The same command is normal in the old cluster:

Hadoop jar hadoop-lzo.jar com.hadoop.compression.lzo.DistributedLzoIndexer-D mapreduce.job.queuename=copy / data/lzodata/

The configuration difference between the two clusters is that the configuration of the old cluster lzo is manual, while the new cluster is configured with the parcel package of GPLEXTRAS.

The new cluster configured with GPLEXTRAS defaults to loading lzo's jar package into the hadoop classpath path, resulting in DistributedLzoIndexer not being submitted to the correct queue.

Because in the new version of hadoop-lzo.jar

Com.hadoop.compression.lzo.DistributedLzoIndexer modified the way to get Configuration

Configuration conf = new Configuration (); Job job = new Job (conf)

In the old version,

Job job = new Job (getConf ())

The source code is modified and then re-uploaded to the server that submitted the job, which only works when the job is submitted.

Modify the location of the script. Please make a backup before modifying it:

/ opt/cloudera/parcels/CDH-5.6.0-1.cdh6.6.0.p0.45/lib/hadoop/bin

Modify export CLASSPATH=/opt/cloudera/userlib/hadoop-lzo.jar:$CLASSPATH in script

The / opt/cloudera/userlib/hadoop-lzo.jar above is the path you want to load

Tags: Cluster configuration script two command path queue location role backup manual file method time server source code version environment index result Apple Docker Huawei Linux macOS MariaDB Microsoft MySQL NVidia OPPO Reno Shulou Information Microsoft MySQL NVidia Huawei