流式处理一切——高效数据流处理的模式.pdf

1、Streaming all the thingsPatterns of Effective Data Stream ProcessingAdi PolakConfluent Data is everywhere!adipolakbatchData streamingApplicationsAdi PolakOReilly AuthorPeople Manager,Software EngineerStreaming,big data field,including ML at scaleAdiPolakChallenges1.Throughput and Scalability2.Latenc

2、y and Real-Time Processing3.Exactly-Once Semantics4.Join operations5.Error Handling and Recovery6.Complex Event Processing7.Guarding the Gates Data IntegrityadipolakChallenges1.Throughput and Scalability2.Latency and Real-Time Processing3.Exactly-Once Semantics4.Join operations5.Error Handling and R

3、ecovery6.Complex Event Processing7.Guarding the Gates Quality,Security and ComplianceadipolakChallenges1.Throughput and Scalability2.Latency and Real-Time Processing3.Exactly-Once Semantics4.Join operations5.Error Handling and Recovery6.Complex Event Processing7.Guarding the Gates-Quality,Security a

4、nd ComplianceadipolakOut Lamba architectureadipolakIn Kappa ArchitectureadipolakadipolakEventsStateTime Event Storage Ingestion ProcessingadipolakEvent timeStorage timeImmutable=deterministicIngestProcessingMutable!=deterministicTime and CalculationsadipolakIs event late?Compare event timestamp to t

5、he most current watermark:If(timestamp the watermark)event is on time.timeline5watermark2Late event 6on timeadipolakadipolakWindowsTime drivenallowed lateness for late arriving eventsType of windowadipolakTumbling WindowadipolakSliding Window adipolakSession WindowadipolakEnd to End System adipolakE

6、nd to End System adipolaksinkEnd to End System adipolaksink?adipolakadipolakadipolakadipolakLets put all of these different pieces together:Once all of the operators complete their pre-commit,they issue a commit.If at least one pre-commit fails,all others are aborted,and we roll back to the previous