LinkedIn 上流式和批处理融合的秘诀.pdf

1、Secret Sauce of Stream&Batch ConvergenceSanil Jain|Staff Engineer at LinkedIn2018LinkedInApache Samza 2019Managed Stream Processing Platform 2024Stream Batch Convergence2021Realtime ML infraApache Beam Feature AccessFeature GenerationApache FlinkApache FlinkNEUAbout Me01StrategyStream Batch Unificat

2、ion02ArchitectureUnified Platform03Ads&Tracking PlatformUse CasesConvergence PlatformMission:Provide a unified platform for expressing computation logic for stream/batch processing independent of where data is stored(HDFS or Kafka)Why solve this problem?Development cost:Learn different API/Engines/D

3、ata SemanticsMaintenance cost:No code sharing/Operational overhead Infra Cost:Maintain different ecosystems&enginesConvergence Platform:What it takes?Storage Convergence:Provide an abstraction of Unified tables as Data access API for accessing different data sourcesCompute Convergence:Provide Unifie

4、d Compute API to author data processing logic for batch/streaming computation.Our Strategy Stream&Batch Unification01Bet on Flink as a Unified engineStreamingBatchInfiniteOut-of-orderDynamicFiniteOragnized Static DatasetProactive plan for optimizing execution time&resourcesStream Processing Semantic

5、s Watermark,Retraction,Checkpointing,Local StateExecution Stages,Adaptive Execution,Speculative Execution,Data SkippingQuery Optimization,Scheduling,Shuffle,State Backend,ConnectorsCompute Convergence:API:Apache Beam vs Flink native Table/SQLStreaming Benchmarks ResultsBeam on Flink runner is much s

6、lower almost 10 x than Flink native Java APIsBatch Benchmark Results Beam on Flink Runner performs 2-3x slower than the corresponding Flink SQL implementation and uses almost 2x more resourcesPlease see slides at the end for details on benchmarking setup and details.Storage Convergence:Unified Table