将 Apache Kafka 转换为可扩展的消息队列.pdf

1、 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.O P N 4 1 3Transforming Apache Kafka into a Scalable Message QueueSubham RakshitSr.Streaming Solution ArchitectAmazon Web ServicesMasudur Rahaman SayemSr.Streaming S

2、olution ArchitectAmazon Web Services 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.AgendaApache Kafka IntroLimitations of Kafka in Certain ScenariosQueues for KafkaDeep Dive into internal implementationSecurity,Observability&Scaling ConsiderationsDemoLimitations and RoadmapTakea

3、ways 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.Apache KafkaP1P2P3Topic:customer01234567Consumer AProducerConsumer Breadwrite(offset=4)(offset=6)Replication 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.Apache KafkaP1P2P3Consumer GroupConsumer 1Consumer 2

4、Topic:customer 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.Limitations of Kafka in Certain Scenarios 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.Scale consumers to process peak loadstReco

5、rdsConsumer Lag 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.Scale consumers to process peak loadstRecordsConsumer Lag Ingress traffic is higher at certain times Need more consumers to handle peak loadP1P2P3Consumer 1Consumer 2Consumer 3Consumer GroupParallelism limited by part

6、ition countNeed to“over-partition”for peak 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.Head-of-line blocking*PartitionOKOKBlocked-Slow/Bad Record Kafka ensures strict ordering per partition Slow or faulty message blocks all subsequent messages from being processed To mitigate