Hadoop operation and maintenance record series (26)

Deploy jupyterlab + pyspark2 + hive within a minute, as long as spark2 is working properly on yarn.

Recently, Party A has been asking Party B to install jupyterlab, and then use spark2, so there is nothing we can do about it. I don't want to install jupyter in the production cluster myself. There is still potential danger in Terminal, which is said to be not in line with the regulations of Party A's security team, but the business team cried and said no, so they took a fancy to the code hints of jupyter and asked for everything. What can I do? In fact, you can't write 1000 lines of python a year. Why do you need a code hint? Hey.

In fact, this kind of Party An is quite good, and later we will talk about an even worse Party A.

Cluster environment

Hive 1.1.0 included with CDH 5.14.0

Spark 2.1.3 Apache version

Jupyterlab in Anaconda3

According to the official statement of spark2, spark2.x only supports hive version 1.2.x or above, but hive version 1.1.x can also be supported by verification in use. But 2.3is absolutely impossible to use 1.1.x version, and the CDH system can not be upgraded to 6, so Party A can only compromise with 2.1.3 version, another: CDH5 comes with version 1.6.

Anaconda can just download the official installation package, which I installed directly under / opt/anaconda3.

Open spark/conf/spark-env.sh and enter as follows

Export PYSPARK_PYTHON=/opt/anaconda3/bin/pythonexport PYSPARK_DRIVER_PYTHON=/opt/anaconda3/bin/jupyter-labexport PYSPARK_DRIVER_PYTHON_OPTS='-ip=172.16.191.30-port=8890'

Use the conda or spark command to install sparkmagic and ipykernel 4.9.0

Pip install sparkmagicpip/conda install ipykernel==4.9.0

Because ipykernel 5.x is not compatible with sparkmagic, an exception of future will be reported. 4.9.0 is required.

Https://github.com/jupyter-incubator/sparkmagic/issues/492

And then, there is no such thing.

Run pyspark2-master yarn to launch the yarn-based jupyterlab directly.

Start jupyterlab directly.

-A record of evil deeds of Party A-

A few days ago, I went to talk to a company about how bigtop packages and compiles hadoop ecology. the company does outsourcing projects for a state-owned enterprise. how boring and abnormal Party An is.

The Hadoop ecological series should be fully Chinese and packaged into a distribution version, including log4j logs, and the content must also be in Chinese. if Party A goes further, it will have to rewrite the java language in easy language.