大规模基因人工智能：它能带来什么成本是多少以及如何减轻痛苦.pdf

1、1GenAI at Scale:What it Enables,Costs,and How to Reduce PainMark KurtzFormer CTO Neural Magic,Acquired by Red HatMember of Technical Staff,Office of the CTO,Red Hat2What well discuss today?The state of LLMs and real-world deployment trends?Challenges&key decisions in production?How vLLM unlocks effi

2、cient LLM serving?Model compression with LLM Compressor?Fine-Tuning with InstructLab?Tuning Your Deployments?Accelerate with Red Hat AI3The state of LLMs and real-world deployment trends4Why LLMs?Understand and generate natural,human-like text with unprecedented accuracyTrained to predict the next t

3、oken,then tuned for human preferencesNearly every company is exploring how to integrate LLMsCurrent StateOnly 3%-10%of prototypes make it to production$106B projected to be spent on inference this yearProjected StateOver 30%of prototypes expected to reach production in the next 2 yearsInference spen

4、d projected to hit$255 Billion by 2030State of the Industry5Code and content generationAugments human workflows(e.g.writing,coding)Efficiency gains across engineering and creative teams of 30%or more*varies depending on who you askSummarizationDistills key insights from long documentsUsed for meetin

5、g notes,reviews,articles,and moreQuestion Answering SystemsAnswers queries using internal or external sourcesCommonly with RAG:retrieval augmented generationPowers chatbots,support tools,and intelligent searchProduction Use Cases61.Rapid Prototypinga.Focus on usefulness not performance2.Accuracy Eva

6、luationsa.Define eval criteria:academic e.g.lm eval harness,public e.g.arena hard&human eval,internal evaluationsb.Explore architectures and sizesc.Optionally fine-tune on internal data3.Inference Performance Testinga.Ensure model is fast enough and cost-effective4.Limited Deployment&A/B Testinga.Re