aws-airflow-wmaa

使用 amazon management 的 WMAA 來跑 Apache Airflow 是個要開始使用 airflow 的好方式

how to access WMAA airflow

• Accessing an Amazon MWAA environment - Amazon Managed Workflows for Apache Airflow

example

• airflow/airflow/providers/amazon/aws/example_dags at main · apache/airflow

airflow token

• Creating an Apache Airflow web login token - Amazon Managed Workflows for Apache Airflow

Reference

• Using Terraform to build a serverless Airflow via Amazon Managed Workflows and automatic DAG sync using GitHub Actions · ds/dx - a data science & ml engineering blog
• claudiobizzotto/aws-mwaa-terraform: AWS MWAA Quick Start With Terraform. 有依照這個 github 來跑起一個 airflow。要設定 kms 為 custom key 時，還需要在試試看

使用 KMS key 可以參考的:

• Customer managed keys for Data Encryption - Amazon Managed Workflows for Apache Airflow
• Amazon MWAA execution role - Amazon Managed Workflows for Apache Airflow 依據這篇來修改 execution role 權限
  • 另外是要自行 grand 權限
    • aws_kms_grant doesn’t accept service principals in grantee_principal fiend · Issue #13994 · hashicorp/terraform-provider-aws
    • aws kms create-grant --region us-east-2 --key-id key-id-xxxxxxxxxx-33333 --grantee-principal "airflow.us-east-2.amazonaws.com" --retiring-principal "airflow.us-east-2.amazonaws.com" --operations Decrypt Encrypt GenerateDataKey ReEncryptFrom ReEncryptTo RetireGrant DescribeKey --name wmaa_s3_key_grant
• idealo/terraform-aws-mwaa: Terraform module to setup Managed Workflows with Apache Airflow. (Airflow as managed service by AWS) 可以改用這個 module 來建立 wmaa 試試
• Setting up MWAA to use a KMS key - DEV Community

有時間要來試試看

• Tutorial: Configuring the aws-mwaa-local-runner in a Continuous Delivery (CD) pipeline - Amazon Managed Workflows for Apache Airflow 可以整合 CI/CD 的方式

Compare

• AWS Managed Workflows for Apache Airflow vs. Glue | by Minseok Song | Towards Data Science