Apache Flink 中的流处理和批处理融合.pdf

1、HDFSStorageComputing EngineControl PlaneSupportingToolingsDataModelingDataMetricsLoggingAlertingTestingReleasingData freshnessScalabilityThroughputCostStabilityOperabilityData ApplicationsHDFSStorageComputing EngineControl PlaneSupportingToolingsDataModelingDataYARNMetricsLoggingAlertingTestingRelea

2、singData ApplicationsHDFSStorageComputing EngineControl PlaneSupportingToolingsDataModelingDataYARNMetricsLoggingAlertingTestingReleasingData ApplicationsHDFSStorageComputing EngineControl PlaneSupportingToolingsDataModelingDataYARNMetricsLoggingAlertingTestingReleasingData ApplicationsHDFSStorageCo

3、mputing EngineControl PlaneSupportingToolingsDataModelingDataYARNMetricsLoggingAlertingTestingReleasingData ApplicationsHDFSStorageComputing EngineControl PlaneSupportToolingsDataModelingDataYARNMetricsLoggingAlertingTestingReleasingData ApplicationsOnline ApplicationKafkaStreaming Feature Generatio

4、nHDFSETLFeature StoreBatch Feature GenerationThe features are generated in nearline.Batch backfill jobs are needed occasionally to apply the same computing logic on the historical data.E.g.new features onboarding,error correction.HDFSYARNOnline ApplicationStreaming And Batch Feature GenerationThe fe

5、atures are generated in nearline.Batch backfill jobs are needed occasionally to apply the same computing logic on the historical data.E.g.new features onboarding,error correction.Unified StorageUser Activity EventsFeaturesUser Activity EventsFeaturesOne FormatK-V,Queue,Range ScanMigration CostLearni

6、ng CostMaintenance CostDevelopment CostExecution CostData Infra Cost ModelStream and Batch UnificationA“new”design paradigmNo need to distinguish between streaming and batchMetricsLoggingAlertingStorageComputing EngineControl PlainSupportingToolingsDataModelingDataTestingReleasingData ApplicationsA“