姜慧强-长文本 LLMs 推理优化：动态稀疏性算法的应用实践.pdf

1、长文本长文本 LLMs LLMs 推理优化：动推理优化：动态稀疏性算法的应用实践态稀疏性算法的应用实践演讲人：姜慧强目 录01LLMs Inference优化方法02长文本推理挑战03观察与解决方案04总结和未来展望01LLMs Inference优化方法LLMs Inference优化方法Resummarized based on https:/arxiv.org/abs/2312.15234.LLMs Inference优化方法Q-SparseBitNetYOCO02长文本推理挑战Long-context will unlock intelligent applicationsRepo-l

2、evel Debugging/QASelf-play ReasoningLong-video Understanding/GenerationLong-context LLMs Inference Bottleneck in Pre-fillingO*1.5=21.78 mins=60+A100*20s TTFTLong Prefilling Latency,30 minutes to process 1M tokens on an A100 for an 8B LLM.How to Optimal the Attention FLOPsLong-context LLMs Inference

3、Bottleneck in KV CacheKV Cache Storage Issue,Storing 512K tokens requires 62GB of GPU memory in fp16.03观察与解决方案Observation 1:Attention is Dynamically Sparse Figure.The dynamic sparsity in attention.(a)How much attention scores can top-k(k=4096)columns cover in a 128k context.(b)Less attention scores

4、are retrieved when reusing the top-k indices from another examples,indicating its dynamic nature.Visualizations are based on LLaMa-3-8B with a single A100.in decodingin prefillingFigure.(c)The dynamic sparsity of each layer and head in Llama-3-8B model in the KV retrieval test of 100k tokens.The blu

5、e curve shows that dynamically selecting top-1000 critical tokens achieves an 89%average recovery ratio,indicating high attention sparsity.In contrast,the orange curve shows that statically using the initial top-1000 critical tokens drops the ratio to 71%.(c)Dynamically sparsity in decoding.(a)Atten

6、tion is sparse.(b)Sparsity of attention is dynamic.Sparse Attention via Vector RetrievalAttention from retrieval perspective=1expOut-of-Distribution degrades the effectiveness of ANNSRetrieve key by an inner-product distance(a)Scanned Vectors(b)Mahalanobis DistanceRetrievalAttentionRetrievalAttentio