星期日, 7月 05, 2015

安裝 Spark in Ubuntu 12.04 小記

最近由於研究需要
要開始研究 Apache Spark
先從大家常安裝的 Ubuntu 12.04 開始
之後來研究 CentOS 與 openSUSE
:-)

安裝 Spark in Ubuntu 12.04
OS: Ubuntu 12.04 LTS
java:  openjdk 7
scala: 2.9.1
spark: 1.4.0 with pre-build for Hadoop 2.6 and later

因為用的是 EasyCloud服務, 所以已經有先 sudo 為 root


先更新 apt-get
#apt-get   update

看看有沒有 jps  指令
#jps
如果沒有就安裝 openjdk 套件
#apt-get  install  openjdk-7-jdk

再次 測試 jps
#jps

查詢 scala 套件
#apt-cache   search   ^scala

安裝 scala
#apt-get   install  scala

查詢 scala 版本
# apt-cache   showpkg   scala

進入  scala shell 界面
# scala
Welcome to Scala version 2.9.1 (OpenJDK 64-Bit Server VM, Java 1.6.0_24).
Type in expressions to have them evaluated.
Type :help for more information.

scala>

按 Ctrl + d 離開

下載 Spark

選取 Spark 版本
選取 套件類型
點選 Download Spark後面的連結  spark-1.4.0-bin-hadoop2.6.tgz
螢幕快照 2015-07-04 下午12.33.36.png
選擇最接近的下載點

螢幕快照 2015-07-04 下午12.33.36.png

由於我是在 VM 裡面所以使用 wget 方式
# wget    http://www.gtlib.gatech.edu/pub/apache/spark/spark-1.4.0/spark-1.4.0-bin-hadoop2.6.tgz

# ls
spark-1.4.0-bin-hadoop2.6.tgz

解壓縮 spark
# tar   zxvf   spark-1.4.0-bin-hadoop2.6.tgz

觀察相關資訊
# ls
spark-1.4.0-bin-hadoop2.6  spark-1.4.0-bin-hadoop2.6.tgz

# ls spark-1.4.0-bin-hadoop2.6/bin/
beeline             pyspark2.cmd      spark-class       sparkR.cmd        spark-submit
beeline.cmd         pyspark.cmd       spark-class2.cmd  spark-shell       spark-submit2.cmd
load-spark-env.cmd  run-example       spark-class.cmd   spark-shell2.cmd  spark-submit.cmd
load-spark-env.sh   run-example2.cmd  sparkR            spark-shell.cmd
pyspark             run-example.cmd   sparkR2.cmd       spark-sql


# spark-1.4.0-bin-hadoop2.6/bin/spark-shell
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/07/05 01:54:41 INFO SecurityManager: Changing view acls to: root
15/07/05 01:54:41 INFO SecurityManager: Changing modify acls to: root
15/07/05 01:54:41 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
15/07/05 01:54:41 INFO HttpServer: Starting HTTP Server
15/07/05 01:54:42 INFO Utils: Successfully started service 'HTTP class server' on port 49845.
Welcome to
     ____              __
    / __/__  ___ _____/ /__
   _\ \/ _ \/ _ `/ __/  '_/
  /___/ .__/\_,_/_/ /_/\_\   version 1.4.0
     /_/

Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.6.0_24)
Type in expressions to have them evaluated.
Type :help for more information.
15/07/05 01:54:48 WARN Utils: Your hostname, nchc-vm resolves to a loopback address: 127.0.1.1; using 10.2.0.8 instead (on interface eth0)
15/07/05 01:55:18 INFO SparkILoop: Created sql context (with Hive support)..
SQL context available as sqlContext.

scala>


先記起來

~ enjoy it

沒有留言: