调整、扩展、成功：优化 Amazon EKS 上的 Spark 性能.pdf

1、 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.C N S 4 2 0Shuffle,Scale,Succeed:Optimizing Spark Performance on Amazon EKSVara BonthuHe/himPrincipal Open Source Specialist SAAWSManabu McCloskeyHe/himSenior Open S

2、ource EngineerAWS 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.AgendaWhy Spark on EKS?Configuring Spark on EKS for Scale and EfficiencyDynamic Scaling with Spark+KarpenterAccelerating compute with Apache Gluten+VeloxRemote Shuffle Service with Apache CelebornBest practices&next

3、 stepsQ&A 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.Configuring Spark on EKS for Scale and Efficiency 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.Spark on EKS for ScaleEKS Control Plane

4、API ServerData LakeS3 BucketAmazon ECRNode AutoscalerKubernetes SchedulerUsersExecutor 1Executor 2Spark Shuffle Storage950GB-NVMe SSDRAID 0(/mnt/k8s-disks/0)Driver Podr8g.xlarge-4vCPU/64GBOn Demandr8g.4xlarge-4vCPU/64GBSpotApache Spark 3.5.2 Docker Image Driver pod on On-Demand instances Executor on

5、 Spot or Reserved instances Karpenter for Dynamic autoscaling Graviton instances for better performance NVMe SSD for shuffle storage Amazon ECR for Spark images-Spark Operator Apache YuniKorn for custom batch schedulerSpark config 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.20

6、25,Amazon Web Services,Inc.or its affiliates.All rights reserved.The Scaling Challenge Dynamic Data and Cost Efficiency 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.Spark Dynamic AllocationEKS Control PlaneAPI ServerData LakeS3 BucketAmazon ECRNode AutoscalerKubernetes Schedule