使用 Amazon S3 批量操作处理多达数十亿个对象.pdf

1、 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.S T G 3 6 0Using Amazon S3 Batch Operations to operate on up to billions of objectsFred ZhangSenior Product ManagerMayank GuptaSoftware Development Manager 2025,Amaz

2、on Web Services,Inc.or its affiliates.All rights reserved.AgendaWhat is S3 Batch OperationsWhat do you need to create an S3 Batch Operations jobA use case:copy 20 billion objectsUnder the hoodSummary 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.What is S3 Batch OperationsAllows

3、 to operate on tens to billions of objects at scaleScalableCopy,Replicate,Tagging,and moreOperationsServicing tens of thousands of jobs and billions of objects dailyVolumeSupporting other S3 features such as Compute ChecksumsFoundational 2025,Amazon Web Services,Inc.or its affiliates.All rights rese

4、rved.What do you need to create an S3 Batch Operations job Object list/Manifest Operation/Action Permissions Report(optional)2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.Use case:Copy 20 Billion objectsLet S3 generate manifest for youUse a single job to process 20 billion objec

5、tsLet S3 create IAM role with permission policies in S3 Management ConsolePreviouslyNowGenerate an inventory report for a manifestSplit the objects into multiple jobsCreate IAM role manually in IAM center 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.UP TO 10X FASTERUnder the ho

6、od 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.Partitions 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.How does S3 Batch Operations workPre-processingExecutionPost-processing 2025,Amazon Web Services,Inc.or its affi