Reference

• Local Development Setup for MWAA with PyCharm | by Satyajit Roy | SelectFrom 有依照這方式，建立起 local wmaa docker
• aws/aws-mwaa-local-runner: This repository provides a command line interface (CLI) utility that replicates an Amazon Managed Workflows for Apache Airflow (MWAA) environment locally.
• How to Setup a Local MWAA Development Environment | by Tejas Marvadi | CodeX | Medium

使用 docker 來跑 apache airflow 來玩玩 tutorial 的程式

cmd for list dag and task

`

initialize the database tables

airflow db init

print the list of active DAGs

airflow dags list

prints the list of tasks in the “tutorial” DAG

airflow tasks list tutorial

prints the hierarchy of tasks in the “tutorial” DAG

airflow tasks list tutorial –tree
`

local 跑測試的方式

• 跑一個特定的 task
  • airflow tasks test tutorial templated 2015-06-01 # testing templated
• 跑一個特定的 DAG
  • airflow dags test [dag_id] [logical_date]

Reference

• Running Airflow in Docker — Airflow Documentation
• Tutorial — Airflow Documentation

列一下，目前看到的 airflow 資源

Tutorial

• Quick Start — Airflow Documentation
• Concepts — Airflow Documentation

ecosystem

• Ecosystem | Apache Airflow 可以看看有哪些 resource 囉

airflow command line

• config command
  • list config: airflow config list
  • get single config: airflow config get-value core executor
  • Configuration Reference — Airflow Documentation

decorator

• Introduction to Airflow Decorators - Airflow Guides
• DAGs — Airflow Documentation

Reference

• Apache Airflow：工作流程管理控制台. 上一篇「淺談 Cloud Dataflow & Apache Beam… | by Terrence Toh | Hahow Tech
• Airflow, Prefect, and Dagster: An Inside Look | by Pedram Navid | Towards Data Science

Compare

• Airflow, Prefect, and Dagster: An Inside Look | by Pedram Navid | Towards Data Science

使用 amazon management 的 WMAA 來跑 Apache Airflow 是個要開始使用 airflow 的好方式

how to access WMAA airflow

• Accessing an Amazon MWAA environment - Amazon Managed Workflows for Apache Airflow

example

• airflow/airflow/providers/amazon/aws/example_dags at main · apache/airflow

airflow token

• Creating an Apache Airflow web login token - Amazon Managed Workflows for Apache Airflow

Reference

• Using Terraform to build a serverless Airflow via Amazon Managed Workflows and automatic DAG sync using GitHub Actions · ds/dx - a data science & ml engineering blog
• claudiobizzotto/aws-mwaa-terraform: AWS MWAA Quick Start With Terraform. 有依照這個 github 來跑起一個 airflow。要設定 kms 為 custom key 時，還需要在試試看

使用 KMS key 可以參考的:

• Customer managed keys for Data Encryption - Amazon Managed Workflows for Apache Airflow
• Amazon MWAA execution role - Amazon Managed Workflows for Apache Airflow 依據這篇來修改 execution role 權限
  • 另外是要自行 grand 權限
    • aws_kms_grant doesn’t accept service principals in grantee_principal fiend · Issue #13994 · hashicorp/terraform-provider-aws
    • aws kms create-grant --region us-east-2 --key-id key-id-xxxxxxxxxx-33333 --grantee-principal "airflow.us-east-2.amazonaws.com" --retiring-principal "airflow.us-east-2.amazonaws.com" --operations Decrypt Encrypt GenerateDataKey ReEncryptFrom ReEncryptTo RetireGrant DescribeKey --name wmaa_s3_key_grant
• idealo/terraform-aws-mwaa: Terraform module to setup Managed Workflows with Apache Airflow. (Airflow as managed service by AWS) 可以改用這個 module 來建立 wmaa 試試
• Setting up MWAA to use a KMS key - DEV Community

有時間要來試試看

• Tutorial: Configuring the aws-mwaa-local-runner in a Continuous Delivery (CD) pipeline - Amazon Managed Workflows for Apache Airflow 可以整合 CI/CD 的方式

Compare

• AWS Managed Workflows for Apache Airflow vs. Glue | by Minseok Song | Towards Data Science