3414 - LMCache：降低企业级 LLM 性能成本.pdf

1、Orlando,FLOctober 69IBM TechXchange 2025Junchen JiangLMCache Lab,UChicago,Tensormesh Inc.Martin HickeyIBM Research,STSM,Open Technology3414Lower LLM Performance Costs in the Enterprise 29 yr tech career in enterprise and open source software Core maintainer in LMCache Emeritus Helm core maintainer a

2、nd TOC member Contributor to LMCache,Helm,Kubernetes,Open Telemetry Open source developer at IBM GitHub:hickeymaMartin Hickey Associate Professor,UChicago CS 15+years of systems research experience NSF CAREER,CMU SCS Best Dissertation,2x Google Faculty Awards in ML Systems Pioneered KV caching techn

3、ologies KV cache encoding,blending,sharing,translation Co-creator of LMCache CEO,Tensormesh,Inc.Email:junchenjtensormesh.aiJunchen JiangThe Trends:LLM Inference will be HUGEOnly 10 companies are dedicated to training new LLMs.Itll be unthinkable not to have intelligence integrated into every product

4、 and service.Itll just be an expected,obvious thing.Sam Altman,OpenAIBut1,000,000sof apps and orgs run LLM inferenceAccording to Gartner,over 80%of AI hardware will be dedicated to inferenceby 2028Armand Ruiz,IBMLong-Context inference:the biggest OpportunitiesIn the next year,youre going to see very

5、 large context windows,.When they are delivered at scale,its going to impact the worldat a scale no one understands yet.Eric Schmidt,Former Google CEONewsBusiness docsChat historyBooksLLMVideosCode reposAudioMeeting notesLong-Context inference:the biggest ChallengeExisting approaches fail at scale.-

6、Caching is criticalBottleneck:Prefilloutput textLLMinput queryUserloooooooong contexttimeKV cachePrefill(long)DecodeTime to first tokenWhy KV cache?Traditionally,KV cache speeds up inference with a limited scope It exists only in the lifecycle of a single query in a single engines GPU memory However