使用 gsutil 工具建立 Google Cloud Storage in openSUSE Leap 15 小記
OS: openSUSE Leap 15
首先在 GCP 平台上面已經建立專案然後啟動計費
接下來是安裝 Google Cloud SDK, 參考
使用一般使用者開啟終端機
檢查 python 版本
> python --version
Python 2.7.14
將目錄切換到使用者家目錄
>cd
- 這邊會切換到家目錄的原因是, 之前練習安裝 Cloud SDK 的時候, 位置在 ~/下載 目錄
- 遇到安裝時出現 “UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4” 這樣的錯誤訊息, 查了一下網路上的文章是編碼的問題, 將SDK 放在沒有中文的目錄路徑下就沒有問題了
下載 Cloud SDK
進行 SHA256 check SUM, 跟官方比對, 確認檔案無誤
> sha256sum google-cloud-sdk-240.0.0-linux-x86_64.tar.gz
5b7ad4e2632c77d5643162af6e1a586d22c5da82fe4e2ec18c411c831c6f5a44 google-cloud-sdk-240.0.0-linux-x86_64.tar.gz
進行解壓縮
> tar zxvf google-cloud-sdk-240.0.0-linux-x86_64.tar.gz
安裝 google cloud SDK
> ./google-cloud-sdk/install.sh
Welcome to the Google Cloud SDK!
To help improve the quality of this product, we collect anonymized usage data
and anonymized stacktraces when crashes are encountered; additional information
is available at <https://cloud.google.com/sdk/usage-statistics>. You may choose
to opt out of this collection now (by choosing 'N' at the below prompt), or at
any time in the future by running the following command:
gcloud config set disable_usage_reporting true
Do you want to help improve the Google Cloud SDK (Y/n)?
Your current Cloud SDK version is: 240.0.0
The latest available version is: 243.0.0
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Components │
├──────────────────┬──────────────────────────────────────────────────────┬──────────────────────────┬───────────┤
│ Status │ Name │ ID │ Size │
├──────────────────┼──────────────────────────────────────────────────────┼──────────────────────────┼───────────┤
│ Update Available │ BigQuery Command Line Tool │ bq │ < 1 MiB │
│ Update Available │ Cloud SDK Core Libraries │ core │ 10.2 MiB │
│ Update Available │ Cloud Storage Command Line Tool │ gsutil │ 3.8 MiB │
│ Not Installed │ App Engine Go Extensions │ app-engine-go │ 56.6 MiB │
│ Not Installed │ Cloud Bigtable Command Line Tool │ cbt │ 6.4 MiB │
│ Not Installed │ Cloud Bigtable Emulator │ bigtable │ 5.6 MiB │
│ Not Installed │ Cloud Datalab Command Line Tool │ datalab │ < 1 MiB │
│ Not Installed │ Cloud Datastore Emulator │ cloud-datastore-emulator │ 18.4 MiB │
│ Not Installed │ Cloud Datastore Emulator (Legacy) │ gcd-emulator │ 38.1 MiB │
│ Not Installed │ Cloud Firestore Emulator │ cloud-firestore-emulator │ 40.5 MiB │
│ Not Installed │ Cloud Pub/Sub Emulator │ pubsub-emulator │ 33.4 MiB │
│ Not Installed │ Cloud SQL Proxy │ cloud_sql_proxy │ 3.8 MiB │
│ Not Installed │ Emulator Reverse Proxy │ emulator-reverse-proxy │ 14.5 MiB │
│ Not Installed │ Google Cloud Build Local Builder │ cloud-build-local │ 6.0 MiB │
│ Not Installed │ Google Container Registry's Docker credential helper │ docker-credential-gcr │ 1.8 MiB │
│ Not Installed │ gcloud Alpha Commands │ alpha │ < 1 MiB │
│ Not Installed │ gcloud Beta Commands │ beta │ < 1 MiB │
│ Not Installed │ gcloud app Java Extensions │ app-engine-java │ 104.8 MiB │
│ Not Installed │ gcloud app PHP Extensions │ app-engine-php │ │
│ Not Installed │ gcloud app Python Extensions │ app-engine-python │ 6.0 MiB │
│ Not Installed │ gcloud app Python Extensions (Extra Libraries) │ app-engine-python-extras │ 28.5 MiB │
│ Not Installed │ kubectl │ kubectl │ < 1 MiB │
└──────────────────┴──────────────────────────────────────────────────────┴──────────────────────────┴───────────┘
To install or remove components at your current SDK version [240.0.0], run:
$ gcloud components install COMPONENT_ID
$ gcloud components remove COMPONENT_ID
To update your SDK installation to the latest version [243.0.0], run:
$ gcloud components update
Modify profile to update your $PATH and enable shell command
completion?
Do you want to continue (Y/n)? Y
The Google Cloud SDK installer will now prompt you to update an rc
file to bring the Google Cloud CLIs into your environment.
Enter a path to an rc file to update, or leave blank to use
[/home/sakana/.bashrc]:
Backing up [/home/sakana/.bashrc] to [/home/sakana/.bashrc.backup].
[/home/sakana/.bashrc] has been updated.
==> Start a new shell for the changes to take effect.
For more information on how to get started, please visit:
https://cloud.google.com/sdk/docs/quickstarts
因為Google Cloud SDK 有針對 ~/.bashrc 進行改動, 所以要開一個新的終端機才算完整
開啟新的終端機
進行比對 ( 主要是加入命令路徑以及命令補齊功能 )
> diff ~/.bashrc ~/.bashrc.backup
29,34d28
<
< # The next line updates PATH for the Google Cloud SDK.
< if [ -f '/home/sakana/google-cloud-sdk/path.bash.inc' ]; then . '/home/sakana/google-cloud-sdk/path.bash.inc'; fi
<
< # The next line enables shell command completion for gcloud.
< if [ -f '/home/sakana/google-cloud-sdk/completion.bash.inc' ]; then . '/home/sakana/google-cloud-sdk/completion.bash.inc'; fi
執行 gcloud init 初始化 SDK
> ./google-cloud-sdk/bin/gcloud init
Welcome! This command will take you through the configuration of gcloud.
Your current configuration has been set to: [default]
You can skip diagnostics next time by using the following flag:
gcloud init --skip-diagnostics
Network diagnostic detects and fixes local network connection issues.
Checking network connection...done.
Reachability Check passed.
Network diagnostic passed (1/1 checks passed).
You must log in to continue. Would you like to log in (Y/n)? Y
會要求登入或是選擇 google 帳號, 詢問是否允許 google Cloud SDK 存取
Updates are available for some Cloud SDK components. To install them,
please run:
$ gcloud components update
You are logged in as: [YOUR_ACCOUNT@gmail.com].
Pick cloud project to use:
[1] steadfast-task-xxxyyy
[2] Create a new project
Please enter numeric choice or text value (must exactly match list
item): 輸入目前專案或是建立一個專案
Your current project has been set to: [steadfast-task-xxxyyy].
Do you want to configure a default Compute Region and Zone? (Y/n)? Y
Which Google Compute Engine zone would you like to use as project
default?
[26] asia-east1-b
[27] asia-east1-a
[28] asia-east1-c
Please enter numeric choice or text value (must exactly match list
item): 27 這邊我輸入 27 是在台灣的機房
Your project default Compute Engine zone has been set to [asia-east1-a].
You can change it by running [gcloud config set compute/zone NAME].
Your project default Compute Engine region has been set to [asia-east1].
You can change it by running [gcloud config set compute/region NAME].
Created a default .boto configuration file at [/home/sakana/.boto]. See this file and
[https://cloud.google.com/storage/docs/gsutil/commands/config] for more
information about configuring Google Cloud Storage.
Your Google Cloud SDK is configured and ready to use!
Run `gcloud help config` to learn how to change individual settings
This gcloud configuration is called [default]. You can create additional configurations if you work with multiple accounts and/or projects.
Run `gcloud topic configurations` to learn more.
Some things to try next:
* Run `gcloud --help` to see the Cloud Platform services you can interact with. And run `gcloud help COMMAND` to get help on any gcloud command.
* Run `gcloud topic --help` to learn about advanced features of the SDK like arg files and output formatting
到這邊 Cloud SDK 設定完成
觀察一下剛剛提到的 ~/.boto
> egrep -v '^#|^$' ~/.boto
[Credentials]
[Boto]
https_validate_certificates = True
[GoogleCompute]
[GSUtil]
content_language = en
default_api_version = 2
[OAuth2]
接下來開始使用 gsutil 指令建立 google cloud storage
> gsutil mb -l asia-east1 gs://test20190427
Creating gs://test20190427/...
- mb 是建立 bucket
- 預設是 Region 等級, 所以 -l 後面加上 asia-east1 就好, 不需要指定 zone (asia-east1-a) , 不然會有錯誤
到 GCP console 觀察
接下來嘗試把圖片丟到剛剛建立的 Bucket
> gsutil cp kitten.png gs://test20190427
Copying file://kitten.png [Content-Type=image/png]...
- [1 files][164.3 KiB/164.3 KiB]
Operation completed over 1 objects/164.3 KiB.
可以到 GCP console 觀察
測試下載 Bucket 中的檔案到本機
> gsutil cp gs://test20190427/kitten.png kitten2.png
Copying gs://test20190427/kitten.png...
/ [1 files][164.3 KiB/164.3 KiB]
Operation completed over 1 objects/164.3 KiB.
嘗試將檔案複製到 Bucket 中的資料夾
> gsutil cp gs://test20190427/kitten.png gs://test20190427/just-a-folder/kitten3.png
Copying gs://test20190427/kitten.png [Content-Type=image/png]...
/ [1 files][164.3 KiB/164.3 KiB]
Operation completed over 1 objects/164.3 KiB.
列出 Bucket 的內容
> gsutil ls gs://test20190427/
gs://test20190427/kitten.png
gs://test20190427/just-a-folder/
嘗試設定物件的權限
設定公開存取
> gsutil acl ch -u AllUsers:R gs://test20190427/kitten.png
Updated ACL on gs://test20190427/kitten.png
刪除權限
> gsutil acl ch -d AllUsers gs://test20190427/kitten.png
Updated ACL on gs://test20190427/kitten.png
刪除物件
> gsutil rm gs://test20190427/kitten.png
Removing gs://test20190427/kitten.png...
/ [1 objects]
Operation completed over 1 objects.
刪除 Bucket
> gsutil rm -r gs://test20190427
Removing gs://test20190427/just-a-folder/kitten3.png#1556341698915616...
/ [1 objects]
Operation completed over 1 objects.
Removing gs://test20190427/...
額外練習 gsutil rync 指令
如果覺得一個一個指令下太麻煩, 也可以嘗試 gsutil rsync 指令
建立一個實驗資料夾 images, 裡面有圖片兩張
> gsutil rsync images/ gs://test20190427/
WARNING: gsutil rsync uses hashes when modification time is not available at
both the source and destination. Your crcmod installation isn't using the
module's C extension, so checksumming will run very slowly. If this is your
first rsync since updating gsutil, this rsync can take significantly longer than
usual. For help installing the extension, please see "gsutil help crcmod".
Building synchronization state...
Starting synchronization...
Copying file://images/kitten2.png [Content-Type=image/png]...
Copying file://images/opensuse.jpg [Content-Type=image/jpeg]...
- [2 files][182.0 KiB/182.0 KiB]
Operation completed over 2 objects/182.0 KiB.
上面的作法會把 images 目錄下的兩張圖片直接複製到 test20190427 bucket 底下
可是這樣以後管理並不方便, 所以嘗試另外一種方式
在目的地加上目錄名稱 images
> gsutil rsync images/ gs://test20190427/images
Building synchronization state...
Starting synchronization...
Copying file://images/kitten2.png [Content-Type=image/png]...
Copying file://images/opensuse.jpg [Content-Type=image/jpeg]...
- [2 files][182.0 KiB/182.0 KiB]
Operation completed over 2 objects/182.0 KiB.
接下來將 本機 images 目錄內的其中一張圖片刪除, 再次執行一樣的指令
> gsutil rsync images/ gs://test20190427/images
Building synchronization state...
Starting synchronization...
這邊會發現沒有影響
慎重其實我們來實驗兩個 options -n 以及 -d
> gsutil rsync -dn images/ gs://test20190427/images
Building synchronization state...
Starting synchronization...
Would remove gs://test20190427/images/kitten2.png
- -d delete, 如果來源端不存在, 目的地端也會刪除
- -n dry run, 不實際執行, 先看執行的輸出
另外也有一個重要的 option -r recursive
- 如果沒有加上 -r , 只會針對當下目錄的物件, 不會包含子目錄還有子目錄的物件
> gsutil rsync -dr images/ gs://test20190427/images
Building synchronization state...
Starting synchronization...
Copying file://images/test2/kitten2.png [Content-Type=image/png]...
- [1 files][164.3 KiB/164.3 KiB]
Operation completed over 1 objects/164.3 KiB.
如果要加速同步, 可以使用 -m 選項
- -m muti-threaded
這樣又前進 GCP 一小步
~ enjoy it
Reference:
