Running Spark on Yarn

Download Spark

For Hadoop 2.7, download Spark 2.4.6 from https://archive.apache.org/dist/spark/spark-2.4.6/

Configure Spark

export SPARK_HOME=/root/spark-2.4.6-bin-hadoop2.7
export PATH=$PATH:$SPARK_HOME/bin
export HADOOP_CONF_DIR=/root/hadoop-2.10.2/etc/hadoop
export LD_LIBRARY_PATH=/opt/hadoop/lib/native:$LD_LIBRARY_PATH

Start an Example Spark Application

spark-submit --class org.apache.spark.examples.SparkPi \
    --master yarn \
    --deploy-mode cluster \
    --queue default \
    /root/spark-2.4.6-bin-hadoop2.7/examples/jars/spark-examples*.jar \
    10

Reference

• https://spark.apache.org/docs/latest/running-on-yarn.html

Leave a message

Name

Email



Send