构建LLM系统评估微观指标的框架.pdf

1、LLM Micro MetricsSeptember 20231Nov 2024Denys LinkovHead of MLSo youre thinking about changing a system prompt?2LLM Micro MetricsNov 2024You run all of your evaluations,they pass.3LLM Micro MetricsNov 2024You release,things are good until4LLM Micro MetricsNov 20241.Prompt in language2.Conversation i

2、n language3.Response in English!?5LLM Micro MetricsNov 20246LLM Micro MetricsNov 2024Building LLM platforms is challenging7LLM Micro MetricsNov 2024Voiceflow is an AI Agent Platform8LLM Micro MetricsNov 2024250k usersSome context on Voiceflows scale40+languages60+enterprisesWhat makes a good LLM res

3、ponse?9LLM Micro MetricsNov 2024Philosophical question right?10LLM Micro MetricsNov 20241.LLMs generate plausible responses2.People often dont agree on what is good11LLM Micro MetricsNov 2024Different Options for Scoring12LLM Micro MetricsNov 2024re.search(“Review0-9$,txt)dist.cos(“I like potatoes”,

4、“Potatoes”)gpt4(“I like potatoes”,“Potatoes”)rouge(“I like potatoes”,“Potatoes”)Lesson one The flaws of one metric13LLM Micro MetricsNov 2024Challenges with Semantic Similarity14LLM Micro MetricsNov 2024Flaws in Cosine Similarity15LLM Micro MetricsNov 2024Phrasetext-embedding-3-largebge-m3stella_en_

5、1.5BI am a potatoI am a humanI am hungry“I like to eat potatoes”Flaws in Cosine Similarity16LLM Micro MetricsNov 2024Phrasetext-embedding-3-largebge-m3stella_en_1.5BI am a potato0.7120.7120.8600.8600.9560.956I am a human0.3080.6490.947I am hungry0.430.7370.952“I like to eat potatoes”How about LLM as

6、 a judge?17LLM Micro MetricsNov 2024Prefer longer responses18Verbosity Bias in Preference Labeling by Large Language ModelsLLM Micro MetricsNov 2024Humans as judges?19LLM Micro MetricsNov 2024Standardized Exams20LLM Micro MetricsNov 2024“If you just graded them based on length without ever reading t