Reference

• AWS EC2 Instances: Choose Your EC2 Family Wisely - Infostretch

可以來看看的 AWS lab website

Reference

• Well-Architected Tool :: AWS Well-Architected Labs

使用 s3cmd 來操作 aws s3 動作

Tips

• How to find zero byte files in Amazon S3 – iTecNote
  • s3cmd ls -r s3://bucketname | awk '{if ($3 == 0) print}'

Reference

• s3tools/s3cmd: Official s3cmd repo – Command line tool for managing Amazon S3 and CloudFront services
• Running S3cmd commands – DreamHost Knowledge Base

最近公司改為使用 team 來作為溝通的工具，所以也就試著將需要的通知改為介接到 team 囉

Reference

• AWS Health Notifications to MS Teams

了解一下 boto3 中 client & resource 的差異

Reference

• python - Difference in boto3 between resource, client, and session? - Stack Overflow

使用 amazon management 的 WMAA 來跑 Apache Airflow 是個要開始使用 airflow 的好方式

how to access WMAA airflow

• Accessing an Amazon MWAA environment - Amazon Managed Workflows for Apache Airflow

example

• airflow/airflow/providers/amazon/aws/example_dags at main · apache/airflow

airflow token

• Creating an Apache Airflow web login token - Amazon Managed Workflows for Apache Airflow

Reference

• Using Terraform to build a serverless Airflow via Amazon Managed Workflows and automatic DAG sync using GitHub Actions · ds/dx - a data science & ml engineering blog
• claudiobizzotto/aws-mwaa-terraform: AWS MWAA Quick Start With Terraform. 有依照這個 github 來跑起一個 airflow。要設定 kms 為 custom key 時，還需要在試試看

使用 KMS key 可以參考的:

• Customer managed keys for Data Encryption - Amazon Managed Workflows for Apache Airflow
• Amazon MWAA execution role - Amazon Managed Workflows for Apache Airflow 依據這篇來修改 execution role 權限
  • 另外是要自行 grand 權限
    • aws_kms_grant doesn’t accept service principals in grantee_principal fiend · Issue #13994 · hashicorp/terraform-provider-aws
    • aws kms create-grant --region us-east-2 --key-id key-id-xxxxxxxxxx-33333 --grantee-principal "airflow.us-east-2.amazonaws.com" --retiring-principal "airflow.us-east-2.amazonaws.com" --operations Decrypt Encrypt GenerateDataKey ReEncryptFrom ReEncryptTo RetireGrant DescribeKey --name wmaa_s3_key_grant
• idealo/terraform-aws-mwaa: Terraform module to setup Managed Workflows with Apache Airflow. (Airflow as managed service by AWS) 可以改用這個 module 來建立 wmaa 試試
• Setting up MWAA to use a KMS key - DEV Community

有時間要來試試看

• Tutorial: Configuring the aws-mwaa-local-runner in a Continuous Delivery (CD) pipeline - Amazon Managed Workflows for Apache Airflow 可以整合 CI/CD 的方式

Compare

• AWS Managed Workflows for Apache Airflow vs. Glue | by Minseok Song | Towards Data Science

一直以來，說到要做大量的資料分析 or 說到 Data Warehousing 都會提到 AWS redshift
找時間來看看何時使用 redshift & 適用的情形如何

Reference

• Amazon Redshift vs Redshift Spectrum vs Amazon Aurora

使用 serverless framework 搭配 AWS resource group，來方便的進行 cloud watch 的觀察

Reference

• An easy way to manage Serverless project resources by using AWS Resource Groups | by Yi Ai | The Cloud Builders Guild | Medium

使用這篇來記錄一下，自己有遇到過的 cognito issue

Issue List

How do I reset a Cognito user’s password that has expired?

• 問題：新建的 user 超過 7 天沒有 login 進去改 password
  • error message : Temporary password has expired and must be reset by an administrator.
• 執行的 cmd 範例
  • aws cognito-idp admin-create-user --region us-east-1 --user-pool-id us-east-1_youruserpoolid --username theusername --message-action RESEND
• resend 後，預設的密碼會改變，要看信件中的 temporary password
• How do I reset a Cognito user’s password that has expired? - Stack Overflow

使用 AWS SES 寄出通知信時，尤其是 gmail 時，會顯示為 via amazonses.com，然後此封 email 會被歸類為 spam
需要設定 DKIM 驗證，來避免此狀況

Reference

• How to prevent from gmail showing via amazonses.com? - MailBluster
• How to Set Up AWS SES and Avoid Spam Folders | by Rob Wilson | AWS in Plain English
• What is a DNS DKIM record? | Cloudflare
• 設定 SPF, DKIM, DMARC Email 驗證解決「Gmail 未經驗證網域」警告 - Soul & Shell Blog
• What is a DNS SPF record? | Cloudflare

同場加映

• Amazon – SES 設定 Email，讓使用者寄信/伺服器收信/轉寄給管理者 | jsnWork