大规模高效AI服务：LLM和向量搜索的近内存处理加速.pdf

1、Gaurav AgarwalHoward BorchewJayjeet ChakrabortyEfficient AI Serving at ScaleEfficient AI Serving at ScaleGaurav Agarwal Distinguished Engineer,MarvellHoward BorchewDistinguished Engineer,MarvellJayjeet Chakraborty Intern,Marvell/Researcher,UCSCFuture Technologies Initiative:Data Centric ComputingInc

2、reasingly Diverse WorkloadsComputational complexityMemory requirements,access patternsLatency sensitivityData-Centric WorkloadsLLMs,Vector DBs,KV-Caches,DLRMComplex PipelinesAgentic AIRecommendation SystemsRise Of Heterogenous Disaggregated SystemsScale-Up InterconnectCPUCPUGPUMemory w/ComputePCIe/C

3、XLNetworkXPUGPUMemory ApplianceStorageServerDisaggregated computing is essential for efficient,scalable,cost-effective solutionsDeep-Memory HierarchiesHigh cost of data movementFar memory access latenciesPointers in far memoryCaching and Prefetching LimitationsLow arithmetic intensityIrregular acces

4、s patternsSparsityPoor spatial and temporal localityCompute-centric approachesChallenges w/Heterogeneous ArchitecturesCompute Centric Approaches Are Inefficient For Data Centric WorkloadsL1$L2$LLCL1$L2$LLCPCIe/CXL HBML2$L2$CPUXPUXPUCPUDRAMDRAMQPIDataMoveHBMDataMoveDataMoveDataMoveGPU CPU coupled arc

5、hitecturesHW Based Coherent MemoryCXL based architecturesOpen standards-based interconnectProcessing in memory(PIM)Processing near memory(PNM)Paradigm Shift:Bring Compute Close To DataSource:AMD InstinctSource:NVIDIA GraceTightly IntegratedTightly CoupledSource:CXL ConsortiumProcessing Near Data Is

6、A Compelling OptionAccelerator LogicCPU CoreCPU CoreCoherence&Memory LogicPCIe/CXL.io LogicInternal IO DevicesAcceleratorCXL Link dynamically multiplexes IO,Cache,and Memory Protocols in Flit format on PCIe PHY LayerCXL.ioCXL.cacheCXL.memCXL.ioCXL.cacheCXL.memHost MemoryOptional Device MemoryHostLoo