Skip to content

hadoop.md

Latest commit

 

History

History
112 lines (73 loc) · 2.26 KB

hadoop.md

File metadata and controls

112 lines (73 loc) · 2.26 KB

Hadoop

NOT PORTED YET

Healthchecks

Authenticate with Kerberos first (if you're not using Kerberos, you really should be otherwise your cluster has zero security).

kinit

MapR Cluster

This sub-section is for MapR managed Hadoop clusters only.

Centrify does default_ccache_name=KCM:... so you will need to export KRB5CCNAME but this messes up some hadoop commands so only prefix it to lines like 'maprlogin kerberos' and 'sqlline'

Authenticate mapr using the kerberos ticket from kinit above.

maprlogin kerberos

running services:

maprcli node list -columns hostname,svc
maprcli node cldbmaster

configured services:

maprcli node list -columns hostname,csvc
maprcli alarm list

HDFS

hadoop fs -ls /

Yarn

yarn node -list | tee /dev/stderr | grep -c RUNNING

MapReduce

Run a calculate Pi MapReduce job across the cluster nodes:

hadoop jar /opt/mapr/hadoop/hadoop-*/share/hadoop/mapreduce/hadoop-mapreduce-excamples-*-mapr-*.jar pi 10 100

Spark

Run a calculate Pi Spark job across the cluster nodes:

/opt/mapr/spark/spark-*/bin/spark-submit --master yarn --class org.apache.spark.examples.SparkPi /opt/mapr/spark/spark-*/examples/jars/spark-examples_*-mapr-*.jar 10 100

Drill

Avoid errors for user accounts writing to query log:

cd /opt/mapr/drill/drill-*/logs/ && touch sqlline{,_queries}.log
chmod -v 0666 /opt/mapr/drill/drill-*/logs/sqlline{,_queries}.log
avoid flexjson classpath error:
ln -sv /opt/mapr/lib/flexjson-*.jar /opt/mapr/drill/drill-*/jars/

Should configure Drill with mapr/<cluster_name>@$REALM instead so that ZooKeeper can connect to any of them:

Any spaces in the connection string will results in an IllegalArgumentException

sqlline -u "jdbc:drill:drillbit=$(hostname -f);auth=kerberos;principal=mapr/$(hostname -f)@$REALM" <<< "SELECT * FROM sys.drillbits;"

Oozie

/opt/mapr/oozie/oozie-*/bin/oozie admin -oozie http://$(hostname -f):11000/oozie -status

MapR-DB:

mapr dbshell <<< "list"

Ported from private Knowledge Base pages 2010+