aws_glue_study_note
基本概念
- Data Store
遇到的問題
- 設定 connection 後，無法連結到 RDS
- 出現的 error code 為 “At least one security group must open all ingress ports.To limit traffic, the source security group in your inbound rule can be restricted to the same security group”
- google 後，Error的情形為Error: Inbound Rule in Security Group Required，照著說明的方式進行，test connection 就通過囉
- 原因：security group 沒有設定好，沒有 allow security-group 本身所有的 tcp 都可以連
- Setting Up a VPC to Connect to JDBC Data Stores
轉換 json 的資料
- Transform and Import a JSON file into Amazon Redshift with AWS Glue
- Simplify Querying Nested JSON with the AWS Glue Relationalize Transform
連動的方式
- How can I use AWS Glue workflows to automatically start a job when a crawler run completes?
- How can I use a Lambda function to automatically start an AWS Glue job when a crawler run completes?
ETL job
目前 glue 的 job, 可以用 純 python or PySpark 來寫 script code, 我目前是用 PySpark 來嘗試
- Program AWS Glue ETL Scripts in Python
- AWS Glue ETL Code Samples 有關 ETL job 的範例