使用 Delta Lake 和微服务的半结构化通信数据.pdf

1、2024 Databricks Inc.All rights reservedAlex Wilcoxson andAlex Wilcoxson and Greg OttGreg Ott20242024-0606-10101DELTA LAKE AND DELTA LAKE AND MICROSERVICESMICROSERVICES2024 Databricks Inc.All rights reservedEvolving our data store to handle increasing Evolving our data store to handle increasing comp

2、lexity and volume of communication datacomplexity and volume of communication data2SEMISEMI-STRUCTURED STRUCTURED COMMUNICATION DATA COMMUNICATION DATA WITH DELTA LAKE AND WITH DELTA LAKE AND MICROSERVICESMICROSERVICES2024 Databricks Inc.All rights reserved3VOLUME AND COMPLEXITYVOLUME AND COMPLEXITY

3、Short message data is the new normalShort message data is the new normal430%430%Increase in Short Increase in Short Message Data YoYMessage Data YoY2024 Databricks Inc.All rights reserved4VOLUME AND COMPLEXITYVOLUME AND COMPLEXITYChallenges scaling for individual message volumeChallenges scaling for

4、 individual message volumeWebJobs and Services2024 Databricks Inc.All rights reserved5NEW DATA STORENEW DATA STOREScaling for volume and complexityScaling for volume and complexity2024 Databricks Inc.All rights reservedIve worked at Relativity for 11 years.NET,SQL Server,and front-end full stackDist

5、ributed systems,Kubernetes,SparkData Engineering and Machine Learning6ALEX WILCOXSONALEX WILCOXSONStaff Software EngineerStaff Software Engineer2024 Databricks Inc.All rights reservedIve worked as a developer at Relativity for 7 yearsOriginally started with DevOps,then onto C#and JS and finally into

6、 Scala and RustWorked with AI/Analytics/Big data for the past 5 years7GREG OTTGREG OTTStaff Software EngineerStaff Software Engineer2024 Databricks Inc.All rights reserved Introduction to RelativityOne and short message data Architecture overview:Dive into the tech stack Deep dive into system and le