Netflix 支持多样化的机器学习系统.pdf

1、QConSF/November 18,2024David Berg,Romain CledatSupporting ML Systemsdiverse1Computer Vision and Media Understanding Intelligent InfrastructurePayments and Growth2AdsContent Demand ModelingRecommendations and personalizationContent Knowledge GraphDiverse ML Use Cases3Identity resolution:Massively par

2、allel computationMassive dataBuilding model explainers:Diverse environmentsEvent drivenMedia processing:Spiky inference of long running computationsSpecialized resourcesContent decision:Complex orchestration of ETLs,training and scoringMetaflow platform at Netflix 3,788 unique Metaflow flows 332 uni

3、que users 4,302,524 total flow executions with complete lineage 12 PB user artifact storage Servicing multiple organizations Data Science and Engineering Platform and other engineering Machine Learning Platform Data Engineering and Insights4Platform principles56Minimize cognitive loadReduce anxietyR

4、educe attentional loadReduce memory loadUser centric ML platform design7Foundational components that build on top of one another prevent anxiety and allow for extensibilityThe“house of cards”effect8Components that are naturally composable with similar levels of abstraction and with similar interface

5、 aesthetics reduce attention and memoryThe“puzzle”effect9Complexity that is handled for you,not pushed onto you,reduces anxiety,attention and memoryThe“waterbed”effectdef request_with_exp_retry(url,attempts,policy,failure_policy,does_raise):def request(url):Introduction to Metaflow10Computedef compu

6、te(input):.return outputInputOutput11Basic MetaflowstartabjoinendProcess12Execution and data movementf56ab3 :4d89c28:3Flow/0/a/0-x:d89c28Flow/0/b/0-x:f56ab3Flow/0/join/0-x:f56ab3Flow/0/end/0-x:f56ab3a4abb6:1Flow/0/start/0-x:a4abb6NodeORbob:sampled_modelalice:unsampled_modelstartabjoinend#Access Alic