评估人工智能代理：来自亚马逊代理系统的真实案例.pdf

1、 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.A M Z 4 0 2Yunfei BaiKashif ImranAllie ColinEvaluating AI agents:real-world lessons from Amazons a

2、gent systemsHe/himPrincipal Solutions ArchitectAmazon Web ServicesHe/himSenior Manager,Cloud/Applied AI ArchitectureAmazon Web ServicesShe/herSenior Manager,Head of Product and ScienceAmazon 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.AI agent evaluation at Amazon Example 1:ev

3、aluating agent tool-use Example 2:evaluating agent reasoning Example 3:evaluating multi-agent system Key takeaways Agenda 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.AI agent evaluationPlanning/multi-step reasoningFunction call and tool-useMemory managementEvaluation strategyT

4、ask completionOperations,costs and RAIApp-specific agent 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.Common challenges in AI agent evaluation Real-world performanceBlack box frustrationComplexity overwhelmPerformance monitoringFramework lock-inEvaluation data quality 2025,Amaz

5、on Web Services,Inc.or its affiliates.All rights reserved.AI agent evaluation at AmazonOnline traceDEFINE INPUTS RESULTS SHARINGDashboardS3 bucketLLM metricsAI agent evaluatorEVALUATIONOffline traceAUDITING/MONITORINGBehavior analysis and performance monitoringAgent final response evaluationLLM eval

6、uationAI AGENT EVALUATION LIBRARYIntent DetectionMemoryPlanningMulti-turnRAGAgent component evaluationTool-callReasoning 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.Example 1:evaluating agent tool-useTool-call accuracyResponse correctnessFunction relevanceM