s3

近期的 project 使用 AWS CloudFront 為前端，來拉儲存在 S3 中的資料，以提供一個 web server 的方式
上傳 server.ini 後，使用 browser 瀏覽時會變成是下載檔案，查詢後，發現是 MIME 的原因

Reference

• Working with object metadata - Amazon Simple Storage Service
  • System-defined metadata，default 的 content-type 為 binary/octet-stream
• How can I change the MIME/Content-Type associated with my objects stored in Wasabi? – Wasabi Knowledge Base
• How to set correct MIME type on videos in Amazon S3

使用 s3cmd 來操作 aws s3 動作

Tips

• How to find zero byte files in Amazon S3 – iTecNote
  • s3cmd ls -r s3://bucketname | awk '{if ($3 == 0) print}'

Reference

• s3tools/s3cmd: Official s3cmd repo – Command line tool for managing Amazon S3 and CloudFront services
• Running S3cmd commands – DreamHost Knowledge Base

要用 python 讀取儲存在 S3 中的 gzip 檔案資料
使用的過程中，boto3 有用 s3_client 也有用 s3 resource 取得 S3 object 的方式，需要找時間再研究這兩者的差異

code sample

def load_gzip(client, bucket, key): response = client.get_object(Bucket=bucket, Key=key) content = response['Body'].read() with gzip.GzipFile(fileobj=io.BytesIO(content), mode='rb') as fh: file_content = fh.read() return file_content.decode("utf-8")

讀多個 S3 檔案的方式

• python - How to read multiple files in a directory, all of which are csv.gzip with Airflow S3 Hook or boto3? - Stack Overflow

Reference

• amazon web services - Reading contents of a gzip file from a AWS S3 in Python - Stack Overflow
• To use gzip file between python application and S3 directly for Python3
• How to store and retrieve gzip-compressed objects in AWS S3
• Reading a Specific File from an S3 bucket Using Python – SQLServerCentral

在找看看如何將 parquet file 寫到 S3 時，意外找到 aws-data-wrangler package，覺得還滿好用的

Reference

• awslabs/aws-data-wrangler: Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
• aws-data-wrangler/tutorials at main · awslabs/aws-data-wrangler 建議這份 tutorial 要整份看過一次
• amazon s3 - How to write parquet file from pandas dataframe in S3 in python - Stack Overflow

由 S3 trigger 到 step function 需要經過一個 event 的步驟，目前測試出來的方式為

1. S3 設定送給 Event Bridge
2. 設定 Event Bridge rule
3. Event Bridge 送給 step function

Reference

• Tutorial: Log Amazon S3 object-level operations using EventBridge - Amazon EventBridge
• Trigger an ECS job when an S3 upload completes | by Maxime Bury | Medium
• Starting a State Machine Execution in Response to Amazon S3 Events - AWS Step Functions 範例中是用 AWS CloudTrail 為中介，實作上是改用 Event Bridge 來做
• New – Use Amazon S3 Event Notifications with Amazon EventBridge | Noise 參考這篇，微調後進行的設定
• AWS EventBridge vs S3 Notification - DEV Community
• terraform EventBridge

進階參考

• Build a serverless event-driven workflow with AWS Glue and Amazon EventBridge - AWS Feed
• Introducing the Amazon EventBridge service integration for AWS Step Functions | AWS Compute Blog

如何透過 api-gateway 將檔案上傳到 S3

Reference

• Hit the 6MB Lambda payload limit? Here’s what you can do. | theburningmonk.com - 參考 Option 1: use API Gateway service proxy
• S3 Uploads — Proxies vs Presigned URLs vs Presigned POSTs | by Zac Charles | Medium
• serverless-operations/serverless-apigateway-service-proxy: This Serverless Framewrok plugin supports the AWS service proxy integration feature of API Gateway. - This Serverless Framewrok plugin supports the AWS service proxy integration feature of API Gateway.
• Tutorial: Create a REST API as an Amazon S3 proxy in API Gateway - Amazon API Gateway 官方文件, 手動的作法參考

其他 search 到的一些記錄

• Use S3 bucket with API Gateway Proxy integration to return binary files | by Hemnanirohan | Geek Culture | Medium

單純的將 s3 object 分享出來，但是 open s3 bucket access

Reference

• How can I grant public read access to some objects in my Amazon S3 bucket?
  • 要記得先修改 bucket level 的 permission

列一下用 S3 signed URL 相關的事項

Reference

• How S3 Signed URLs work
• AWS S3 signed URLs handbook e-book