Spark 本機安裝 with Ansible 小記

Spark 本機安裝 with Ansible

現在的實驗環境會透過 Ansible 來管理

所以整理了一下 spark local 套件安裝的 ansible playbook

• 使用 group 方式控管
• 安裝 spark 相關套件, 透過變數詢問下載不同版本, 預設是 1.4.0
• 使用 when: 方式針對不同的 OS 來安裝, 目前有 openSUSE Leap, CentOS, 還有 ubuntu
• 目前已測試 openSUSE 42.1 / CentOS 7.2 / Ubuntu 14.04

檔案 spark_local.yml

- name: Install spark with local

 hosts: sparkSingle

#  透過 vars_prompt 動態詢問要裝的版本, 預設是 1.4.0

 vars_prompt:

   - name: "spark_version"

     prompt: "Please enter spark version, example 1.4.0"

     private: no

     default: "1.4.0"

 tasks:

   - name: Install java, wget  with openSUSE Leap

# Use sudo before ansible 1.9

#      sudo: True

     become: True

# 使用 pkg 加上 {{ item }} 配合後面的 with_item 來安裝多個套件

     zypper: name={{ item }}

     with_items:

       - java-1.7.0-openjdk-devel

       - wget

     when: ansible_distribution == "openSUSE Leap"

   - name: Install java, wget with CentOS

     become: True

     yum: name={{ item }}

     with_items:

       - java-1.7.0-openjdk-devel

       - wget

     when: ansible_distribution == "CentOS"

   - name: Install java, wget and scala with Ubuntu

     become: True

     apt: name={{ item }} update_cache=yes

     with_items:

       - openjdk-7-jdk

       - wget

       - scala

     when: ansible_distribution == "Ubuntu"

# ----------------------------------------------------------

   - name: Get scala

     shell: wget http://www.scala-lang.org/files/archive/scala-2.10.1.tgz

     when: ansible_distribution == "CentOS" or ansible_distribution == "openSUSE Leap"

   - name: Exact source code of scala

     shell: tar xvf scala-2.10.1.tgz

     when: ansible_distribution == "CentOS" or ansible_distribution == "openSUSE Leap"

   - name: Move scala to /usr/lib

     become: True

     shell: mv scala-2.10.1 /usr/lib

     when: ansible_distribution == "CentOS" or ansible_distribution == "openSUSE Leap"

   - name: create link for scala

     become: True

     shell: ln -s /usr/lib/scala-2.10.1/  /usr/lib/scala

     when: ansible_distribution == "CentOS" or ansible_distribution == "openSUSE Leap"

   - name: touch .bashrc

     shell: touch ~/.bashrc

     when: ansible_distribution == "CentOS" or ansible_distribution == "openSUSE Leap"

   - name: export $PATH to .bashrc

     lineinfile: dest=~/.bashrc line="export PATH=$PATH:/usr/lib/scala/bin"

     when: ansible_distribution == "CentOS" or ansible_distribution == "openSUSE Leap"

# ----------------------------------------------------------

# 由於沒有在 general option 使用 sudo, 這邊在使用 shell module 抓下來就是使用者自己的身份額非 root

# 這邊用 spark 1.4.0 版本

   - name: Download spark with pre-build hadoop

       shell: wget    http://archive.apache.org/dist/spark/spark-{{ spark_version }}/spark-{{ spark_version }}-bin-hadoop2.6.tgz

# 解開 source code

   - name: Exact pre-build source

     shell: tar zxvf spark-*.tgz

   - name: Rename spark folder

     shell: mv ~/spark*/ ~/spark

# 測試 jps 指令

   - name: Testing jps command

     command: jps

     register: jps

   - debug: var=jps.stdout_lines

執行方式

將要安裝 spark 的主機加入 sparkSingle 群組

然後執行 spark_local.yml 即可

先記下來

~ enjoy it