07-goodput-shi-ni-suo-xu-de-yi-qie-ji-yu-slo-de-llm-tui-li-fu-wu-ji-zhun-ce-shi-zheng-yu-chen-.pptx

1、闪电演讲Goodput 是你所需的一切基于 SLO 的 LLM 推理服务基准测试Lightning Talk:Goodput is All You Need:SLO-based LLM Inference Benchmarking,软件工程师,2025/11/15,郑宇宸,Yu-Chen Cheng,Software Engineer,郑宇宸 Yu-Chen ChengSoftware Engineer,Focus on building AI/LLM infrastructure based on KubernetesPreviously Heywhale WizardQuantIncomi

2、ng SDE Tencent Cloud CSIG TI-ONE,CONTENT,目录,01,Background,02,LLM Inference Benchmarking,03,Takeaways,BackgroundLLM Inference and Benchmarking,LLM Inference,(Source:https:/huggingface.co/blog/tngtech/llm-performance-prefill-decode-concurrent-requests),Prefill:Computation-intensive,Decode:Memory-inten

3、sive,LLM Inference Stack on Kubernetes,LLM Inference Stack on Kubernetes,Inference,Parameter TuningTensor ParallelismData ParallelismExpert Parallelism,FrameworksvLLMSGLangTersorRT LLM,OptimizationSpeculative DecodingFlash Attention,LLM Inference Stack on Kubernetes,KV Cache Management,KV Cache Offl

4、oading,KV Cache Sharing,LLM Inference Stack on Kubernetes,Orchestration,Routing and Load BalancingPrefix-Aware RoutingFairness RoutingSLO-Aware Routing,PD Disaggregation,Autoscaling,LLM Inference Metrics,TTFTTime to First Token.Initial response time when the first output token is generated.TTFT=time

5、_at_first_token-start_time,TPOTTime Per Output Token.The average time between two subsequent generated tokens.TPOT=(e2e_latency-TTFT)/(num_output_tokens-1),(Source:https:/,Service Level Objective(SLO),(Source:https:/,Chat ApplicationsP99 TTFT 200msP99 ITL 50ms,Retrieval-Augmented Generation(RAG)P99

6、TTFT 300msP99 ITL 100msP99 Request Latency 3s,Offline BatchMax Throughput 100 RPS,Code CompletionP99 Request Latency 2s,Pareto Frontier+SLO Attainment,Example:LLM Performance Explorer by BentoML,(Source:https:/https:/,SLO Violation,What is Goodput?,Goodput measures the number of completed requests t