使用 curl 測試 CDN 小記

使用 curl 測試 CDN 小記

OS: openSUSE Leap 15.1

Curl: 7.60.0

因為公司也有 CDN 的服務, 所以就來 study 一下如何測試 CDN

找了網路上的文章來進行實驗

> curl  -I  img1.momoshop.com.tw

HTTP/1.1 200 OK

Content-Type: text/html

Content-Length: 206

Connection: keep-alive

Date: Thu, 13 Feb 2020 06:37:14 GMT

Last-Modified: Wed, 04 Sep 2019 08:33:23 GMT

ETag: "5d6f76d3-ce"

Expires: Fri, 12 Feb 2021 06:37:14 GMT

Cache-Control: max-age=31536000

X-Image-Server: a1-imgweb62

X-Cache-Server: a1-rproxy64

X-Cache-Status: HIT

Cache-Control: public

Accept-Ranges: bytes

X-Cache: Hit from cloudfront

Via: 1.1 ff0896bca963fdb839934a38daad05c9.cloudfront.net (CloudFront)

X-Amz-Cf-Pop: TPE50-C1

X-Amz-Cf-Id: s_DGUUBq7VKqDZprbRdA75kJeG5So0PLUwMRyWjqwA4TxjSefOmfEg==

Age: 1923399

• -I, --head (HTTP  FTP FILE) Fetch the  headers only! 顯示 Headers
• Cache-Control:
• public: 如果回應標記為「public」，即使具備關聯的 HTTP 認證，甚至回應狀態碼無法正常快取，回應也可以供使用者快取。在大多數情況下，「public」並不是必要項目，因為明確的快取資訊 (例如「max-age」) 已表示 回應可供快取。
• 參考 https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/http-caching?hl=zh-tw
• max-age=31536000
• 最多快取 31536000 秒, 也就是 365 天
• X-Cache: 是否在 CDN 有快取
• HIT: 在 CDN 上有快取
• MISS: 在 CDN 上面沒有快取, 會跟原站抓
• X-Amz-Cf-Pop: 這個欄位是快取的站, 這邊可以觀察是 Taipei 的 Pop 點
• 但是不同的供應商, 會有不同的欄位, Cloudflare 可能就是 CF-RAY: 56fc8a38f94545d0-TPE
• ETag:
• ETag HTTP 標題傳遞驗證權杖, 透過驗證權杖進行高效率的資源更新檢查：如果資源未變更，則不會傳輸任何資料
• 參考 https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/http-caching?hl=zh-tw

• Expires: 過期時間
• 瀏覽器收到這個 Response 之後就會把這個資源給快取起來，當下一次使用者再度造訪這個頁面或是要求這個圖片的資源的時候，瀏覽器會檢視「現在的時間」是否有超過這個 Expires。如果沒有超過的話，那瀏覽器「不會發送任何 Request」，而是直接從電腦裡面已經存好的 Cache 拿資料。
• 參考 https://blog.techbridge.cc/2017/06/17/cache-introduction/
• 從 Chrome 觀察

• RFC2616 規範 max-age 會蓋掉 Expires, 所以實際上用到的是 Cache-Control: max-age

剛剛是針對某個網站

哪如果是針對 CDN 可能快取的物件呢? 

> curl -I  https://img1.momoshop.com.tw/goodsimg/0006/750/353/6750353_L.jpg?t=1581517754

HTTP/2 200 

content-type: image/jpeg

content-length: 22654

date: Fri, 06 Mar 2020 13:40:47 GMT

last-modified: Wed, 28 Aug 2019 15:00:03 GMT

etag: "5d6696f3-587e"

expires: Sat, 06 Mar 2021 13:40:47 GMT

cache-control: max-age=31536000

x-image-server: b1-imgweb61

access-control-allow-origin: *

access-control-max-age: 1000

access-control-allow-methods: POST, GET, OPTIONS, DELETE, PUT

access-control-allow-headers: x-requested-with, Content-Type, origin, authorization, accept, client-security-token

x-cache-server: a1-rproxy65

x-cache-status: HIT

cache-control: public

accept-ranges: bytes

x-cache: Miss from cloudfront

via: 1.1 5c4e99b05f9fd6102a3e039a6bd98968.cloudfront.net (CloudFront)

x-amz-cf-pop: TPE50-C1

x-amz-cf-id: JrqvYZNlLjV5zHS323SyuQ8pqzJgciSDOW4sEl1kl1Wj6ky_D4jWjg==

上面有提到過的就不重提

• X-cache: Miss from cloudfront ( Cloudfront 上面沒有快取 ), 所以 CDN 會跟原站抓
• 這邊也可以觀察到沒有 age 的 header, 也就是說沒有物件在 CDN 上已經快取的時間
• 也可以去觀察 x-cache-server 以及 x-image-server 資訊
• last-modified:
• 會跟 If-Modified-Since 搭配使用, 如果超過時間但是沒有變動, 還是會從快取出

很短的時間重新抓一次

> curl -I https://img1.momoshop.com.tw/goodsimg/0006/750/353/6750353_L.jpg?t=1581517754

HTTP/2 200 

content-type: image/jpeg

content-length: 22654

date: Fri, 06 Mar 2020 13:40:47 GMT

last-modified: Wed, 28 Aug 2019 15:00:03 GMT

etag: "5d6696f3-587e"

expires: Sat, 06 Mar 2021 13:40:47 GMT

cache-control: max-age=31536000

x-image-server: b1-imgweb61

access-control-allow-origin: *

access-control-max-age: 1000

access-control-allow-methods: POST, GET, OPTIONS, DELETE, PUT

access-control-allow-headers: x-requested-with, Content-Type, origin, authorization, accept, client-security-token

x-cache-server: a1-rproxy65

x-cache-status: HIT

cache-control: public

accept-ranges: bytes

x-cache: Hit from cloudfront

via: 1.1 41d05f95698dc73a0a8a2f90b9eb739a.cloudfront.net (CloudFront)

x-amz-cf-pop: TPE50-C1

x-amz-cf-id: YL_Ii4eNQTB0VfTIlpSo77A8cHB4aOk7Mv_K4dhix_OqBhbW6tkqHw==

age: 28

• 這邊可以觀察到 CloudFront 已經有資料, 然後也有 age 的資料

接下來進行另外一個測試 就是 支援 TLS 的版本

現在 TLS 版本的支援, 很多瀏覽器都會要求到一定的 TLS 版本, 目前 大概都要支援到 TLS 1.2 以上的版本

> curl  -I  -v  --tlsv1.0  https://img1.momoshop.com.tw

* Rebuilt URL to: https://img1.momoshop.com.tw/

*   Trying 13.35.153.7...

* TCP_NODELAY set

* Connected to img1.momoshop.com.tw (13.35.153.7) port 443 (#0)

* ALPN, offering h2

* ALPN, offering http/1.1

* TLSv1.0 (OUT), TLS handshake, Client hello (1):

* TLSv1.0 (IN), TLS alert, Server hello (2):

* error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version

* stopped the pause stream!

* Closing connection 0

curl: (35) error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version

• 這邊使用 -v 來顯示詳細資訊
• --tlsv1.0 指定使用 TLS 1.0 協定
• 這邊可以觀察到 使用 TLS 1.0 是被拒絕的

接下來嘗試 TLS 1.1 and TLS 1.2

> curl  -I  -v --tlsv1.1  https://img1.momoshop.com.tw

* Rebuilt URL to: https://img1.momoshop.com.tw/

*   Trying 13.35.153.65...

* TCP_NODELAY set

* Connected to img1.momoshop.com.tw (13.35.153.65) port 443 (#0)

* ALPN, offering h2

* ALPN, offering http/1.1

* TLSv1.1 (OUT), TLS handshake, Client hello (1):

* TLSv1.1 (IN), TLS handshake, Server hello (2):

* TLSv1.1 (IN), TLS handshake, Certificate (11):

* TLSv1.1 (IN), TLS handshake, Server key exchange (12):

* TLSv1.1 (IN), TLS handshake, Server finished (14):

* TLSv1.1 (OUT), TLS handshake, Client key exchange (16):

* TLSv1.1 (OUT), TLS change cipher, Client hello (1):

* TLSv1.1 (OUT), TLS handshake, Finished (20):

* TLSv1.1 (IN), TLS handshake, Finished (20):

* SSL connection using TLSv1.1 / ECDHE-RSA-AES128-SHA

* ALPN, server accepted to use h2

* Server certificate:

*  subject: CN=*.momoshop.com.tw

*  start date: Jun  3 00:00:00 2019 GMT

*  expire date: Jul  3 12:00:00 2020 GMT

*  subjectAltName: host "img1.momoshop.com.tw" matched cert's "*.momoshop.com.tw"

*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon

*  SSL certificate verify ok.

* Using HTTP2, server supports multi-use

* Connection state changed (HTTP/2 confirmed)

* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0

* Using Stream ID: 1 (easy handle 0x564c5fcb9040)

> HEAD / HTTP/2

> Host: img1.momoshop.com.tw

> User-Agent: curl/7.60.0

> Accept: */*

> 

* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!

< HTTP/2 200 

HTTP/2 200 

> curl -I -v --tlsv1.2 https://img1.momoshop.com.tw

* Rebuilt URL to: https://img1.momoshop.com.tw/

*   Trying 13.35.153.7...

* TCP_NODELAY set

* Connected to img1.momoshop.com.tw (13.35.153.7) port 443 (#0)

* ALPN, offering h2

* ALPN, offering http/1.1

* TLSv1.2 (OUT), TLS handshake, Client hello (1):

* TLSv1.2 (IN), TLS handshake, Server hello (2):

* TLSv1.2 (IN), TLS handshake, Certificate (11):

* TLSv1.2 (IN), TLS handshake, Server key exchange (12):

* TLSv1.2 (IN), TLS handshake, Server finished (14):

* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):

* TLSv1.2 (OUT), TLS change cipher, Client hello (1):

* TLSv1.2 (OUT), TLS handshake, Finished (20):

* TLSv1.2 (IN), TLS handshake, Finished (20):

* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256

* ALPN, server accepted to use h2

* Server certificate:

*  subject: CN=*.momoshop.com.tw

*  start date: Jun  3 00:00:00 2019 GMT

*  expire date: Jul  3 12:00:00 2020 GMT

*  subjectAltName: host "img1.momoshop.com.tw" matched cert's "*.momoshop.com.tw"

*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon

*  SSL certificate verify ok.

* Using HTTP2, server supports multi-use

* Connection state changed (HTTP/2 confirmed)

* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0

* Using Stream ID: 1 (easy handle 0x56535129a040)

> HEAD / HTTP/2

> Host: img1.momoshop.com.tw

> User-Agent: curl/7.60.0

> Accept: */*

> 

* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!

< HTTP/2 200 

HTTP/2 200 

• 其實如果沒有特別指定, 會用比較高的版本去連接
• 這邊可以觀察 TLS 1.1 and TLS 1.2 都支援

同場加映:

然後也可以用另外一個指令來檢查 TLS, 就是 sslscan 指令

> sslscan  https://img1.momoshop.com.tw

Version: 1.11.10

OpenSSL 1.0.2p-fips  14 Aug 2018

OpenSSL version does not support SSLv2

SSLv2 ciphers will not be detected

OpenSSL version does not support SSLv3

SSLv3 ciphers will not be detected

Testing SSL server img1.momoshop.com.tw on port 443 using SNI name img1.momoshop.com.tw

  TLS Fallback SCSV:

Server does not support TLS Fallback SCSV

  TLS renegotiation:

Session renegotiation not supported

  TLS Compression:

Compression disabled

  Heartbleed:

TLS 1.2 not vulnerable to heartbleed

TLS 1.1 not vulnerable to heartbleed

TLS 1.0 not vulnerable to heartbleed

  Supported Server Cipher(s):

Preferred TLSv1.2  128 bits ECDHE-RSA-AES128-GCM-SHA256   Curve P-256 DHE 256

Accepted  TLSv1.2 128 bits  ECDHE-RSA-AES128-SHA256       Curve P-256 DHE 256

Accepted  TLSv1.2 128 bits  ECDHE-RSA-AES128-SHA          Curve P-256 DHE 256

Accepted  TLSv1.2 256 bits  ECDHE-RSA-AES256-GCM-SHA384   Curve P-256 DHE 256

Accepted  TLSv1.2 256 bits  ECDHE-RSA-AES256-SHA384       Curve P-256 DHE 256

Accepted  TLSv1.2 256 bits  ECDHE-RSA-AES256-SHA          Curve P-256 DHE 256

Accepted  TLSv1.2 128 bits  AES128-GCM-SHA256            

Accepted  TLSv1.2 256 bits  AES256-GCM-SHA384            

Accepted  TLSv1.2 128 bits  AES128-SHA256                

Accepted  TLSv1.2 256 bits  AES256-SHA                   

Accepted  TLSv1.2 128 bits  AES128-SHA                   

Preferred TLSv1.1  128 bits ECDHE-RSA-AES128-SHA          Curve P-256 DHE 256

Accepted  TLSv1.1 256 bits  ECDHE-RSA-AES256-SHA          Curve P-256 DHE 256

Accepted  TLSv1.1 256 bits  AES256-SHA                   

Accepted  TLSv1.1 128 bits  AES128-SHA                   

  SSL Certificate:

Signature Algorithm: sha256WithRSAEncryption

RSA Key Strength:    2048

Subject:  *.momoshop.com.tw

Altnames: DNS:*.momoshop.com.tw

Issuer:   Amazon

Not valid before: Jun  3 00:00:00 2019 GMT

Not valid after:  Jul 3 12:00:00 2020 GMT

然後也常常看到網路上的人使用下列的選項

> curl  -ksvo  /dev/null  https://img1.momoshop.com.tw

• -k, --insecure
• -s, --silent, Silent or quiet mode. Don't show progress meter or error messages.
• -v, --verbose
•  -o, --output <file>

整理一下今天學習到的知識重點

也許可以利用 curl -I 的方式來看 Header 來觀察是否有 CDN, 也許 Header 不一樣

> curl -I  https://media.etmall.com.tw

HTTP/2 403 

server: HiNetCDN/1908

date: Sat, 07 Mar 2020 15:00:16 GMT

content-type: text/html

content-length: 1147

vary: Accept-Encoding

x-cache: MISS, MISS, MISS

x-request-id: 08a83a10d0e14edecab443f4bde4c753

> curl  -I  http://cf.shopee.tw

HTTP/1.1 404 Not Found

Server: NWS_Oversea_AP

Connection: keep-alive

Date: Sat, 07 Mar 2020 15:00:36 GMT

Last-Modified: Sat, 07 Mar 2020 15:00:00 GMT

Content-Type: text/plain; charset=utf-8

Content-Length: 19

X-Content-Type-Options: nosniff

X-Daa-Tunnel: hop_count=2

X-NWS-LOG-UUID: 5423788696303268195 b38e6f5a9afc4fde3c81e984a88d9c0f

Access-Control-Allow-Origin: *

X-CDN: tencent

X-Cache-Lookup: Hit From Upstream

可以藉由觀察 Cache-Control 來觀察是否快取

• public
• 可以由任何快取給存取
• private
• 快取只給一個使用者使用，且不能被共用的快取伺服器給儲存過。隱私視窗(無痕模式)的快取就可能是這樣子。
• no-cache
• 快取伺服器在把已儲存的複製版本傳給請求者之前，先會送一個請求給網頁伺服器做驗證
• no-store
• 快取不該存取任何的使用者請求或者伺服器的回覆。每個請求都是送到原始的伺服器去取得資源。
• 參考 firefox 上面的文件 https://developer.mozilla.org/zh-TW/docs/Web/HTTP/Caching

期限的部分一般則是可以看 Cache-Control 的 max-age 來決定

會使用 etag 或是 last-modified 來進行快取內容驗證

再度前進一小步 :)

~ enjoy it

Reference:

• https://help.onapp.com/hc/en-us/articles/222042948-How-do-I-use-CURL-to-check-a-CDN-resource-
• https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/http-caching?hl=zh-tw
• https://blog.techbridge.cc/2017/06/17/cache-introduction/
• https://developer.mozilla.org/zh-TW/docs/Web/HTTP/Caching