最近由於研究需要
要開始研究 Apache Spark
先從大家常安裝的 Ubuntu 12.04 開始
之後來研究 CentOS 與 openSUSE
:-)
安裝 Spark in Ubuntu 12.04
OS: Ubuntu 12.04 LTS
java: openjdk 7
scala: 2.9.1
spark: 1.4.0 with pre-build for Hadoop 2.6 and later
因為用的是 EasyCloud服務, 所以已經有先 sudo 為 root
先更新 apt-get
#apt-get update
看看有沒有 jps 指令
#jps
如果沒有就安裝 openjdk 套件
#apt-get install openjdk-7-jdk
再次 測試 jps
#jps
查詢 scala 套件
#apt-cache search ^scala
安裝 scala
#apt-get install scala
查詢 scala 版本
# apt-cache showpkg scala
進入 scala shell 界面
# scala
Welcome to Scala version 2.9.1 (OpenJDK 64-Bit Server VM, Java 1.6.0_24).
Type in expressions to have them evaluated.
Type :help for more information.
scala>
按 Ctrl + d 離開
下載 Spark
選取 Spark 版本
選取 套件類型
點選 Download Spark後面的連結 spark-1.4.0-bin-hadoop2.6.tgz
選擇最接近的下載點
由於我是在 VM 裡面所以使用 wget 方式
# wget http://www.gtlib.gatech.edu/pub/apache/spark/spark-1.4.0/spark-1.4.0-bin-hadoop2.6.tgz
# ls
spark-1.4.0-bin-hadoop2.6.tgz
解壓縮 spark
# tar zxvf spark-1.4.0-bin-hadoop2.6.tgz
觀察相關資訊
# ls
spark-1.4.0-bin-hadoop2.6 spark-1.4.0-bin-hadoop2.6.tgz
# ls spark-1.4.0-bin-hadoop2.6/bin/
beeline pyspark2.cmd spark-class sparkR.cmd spark-submit
beeline.cmd pyspark.cmd spark-class2.cmd spark-shell spark-submit2.cmd
load-spark-env.cmd run-example spark-class.cmd spark-shell2.cmd spark-submit.cmd
load-spark-env.sh run-example2.cmd sparkR spark-shell.cmd
pyspark run-example.cmd sparkR2.cmd spark-sql
# spark-1.4.0-bin-hadoop2.6/bin/spark-shell
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/07/05 01:54:41 INFO SecurityManager: Changing view acls to: root
15/07/05 01:54:41 INFO SecurityManager: Changing modify acls to: root
15/07/05 01:54:41 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
15/07/05 01:54:41 INFO HttpServer: Starting HTTP Server
15/07/05 01:54:42 INFO Utils: Successfully started service 'HTTP class server' on port 49845.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.4.0
/_/
Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.6.0_24)
Type in expressions to have them evaluated.
Type :help for more information.
15/07/05 01:54:48 WARN Utils: Your hostname, nchc-vm resolves to a loopback address: 127.0.1.1; using 10.2.0.8 instead (on interface eth0)
15/07/05 01:55:18 INFO SparkILoop: Created sql context (with Hive support)..
SQL context available as sqlContext.
scala>
先記起來
~ enjoy it
沒有留言:
張貼留言