高性能工作负载的文件存储.pdf

1、 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.S T G 4 1 6File storage for high-performance workloadsSebastien BerubeSr.Specialist Solution ArchitectFile Storage ServicesTom McDonaldPrincipal Workload Solution Ar

2、chitectAWS Storage Services 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.AgendaScale-out vs scale-upFSx Intelligent-TieringBest practicesNetworking,EC2,data loadersTesting methodologyCustomer use cases 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.Lustre Ar

3、chitectureActivePassiveActiveActiveActiveActiveActiveActiveOSSOSTsMDS1/MDT1MDS2/MDT2ActiveMGS/MGTFile ServerFile StorageMonolithic(Amazon FSx for OpenZFS)Scale-out(Amazon FSx for Lustre)MonolithicScale-outComponent communication over IPC(10s 100s ns)Components communication over network(100s of ms)S

4、cale by increasing instance sizeScale by adding componentsLimited to 1 active instanceAll instances are activeLower max throughput/scalabilityHighest throughput(TB/s)/scalabilityLowest latency(0.2-0.4 ms)Higher latency(0.5-0.8 ms)Faster individual metadata operationSlower individual metadata operati

5、onsFixed metadata performanceScale-out metadata 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.BottlenecksAchieving a balanced system means nothing is over/under provisionedEverything you consume in AWS is part of a service catalogHost processing/memory/networking must be in bala

6、nce with the filesystem provisioned throughput/IOPS/latencySharp edges include:-Networking-Latency-Data Layout-Data LoadersMonitoring is crucial!2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.Elastic Fabric Adapter(EFA)vs non-EFA FilesystemsEFA filesystems are newer and faster ri