从流到句子：使用 Apache Flink 实现实时生成式人工智能.pdf

1、Real-Time Generative AI with Apache FlinkFrom Stream to SentenceDaren WongAmazon Web Services01Use CaseReal-time Football Commentary AI Case Study02ChallengesHow to solve it with Flink03DemoHow it works04Future Big IdeasFuture of Live Sport Commentary with AI and FlinkUse Case:Live Sports Commentary

2、Input:Live video/image stream Output:Sports CommentaryNave approachFeed a livestream to LLM and ask it to generate the commentariesCommentary is passed to a TTS(text-to-speech)client to generate audio01LatencyLLMs have high latency02ScalabilityThrottling and Context Window size limit 03Hallucination

3、LLM has no access to real time dataChallenges04Enrichment/SinkEnriching data generated by LLM into a Sink05Fault ToleranceLLM server/client failure recovery06Late EventsHandling out-of-order and late eventsA better approach with FlinkPrompt Engineering with role contextExample input event in json:pl

4、ayer_locations:”Alice:98,55,”Bob:90,60,”Carol:95,50,”David:90,50,ball_location:98,55We know that the ball 98,55 is with Alice 98,55 and she is close to the goal 100,50.Return output in json:“Comment”:“Alice with the ball is very close to the goal now!Is she going to score or will Carol defend succes

5、sfully?”AsyncFunctionRole promptingYou are a football commentatorContextTeams,players,field dimensions,etcInstructionOver the next few prompts,I will give you event updates.Based on the given new event and by keeping track of past events,make an exciting comment.Few shots learningHow it worksCNN(YOL

6、O)modeltimestamp:100,player_locations:”Alice:98,55,”Bob:90,60,”Carol:95,50,”David:90,50,ball_location:98,55Prompt LLM for the next commentPersist event and comment in state/context memoryKafkaTTS(Text-to-speech)Stitch the video and speech togetherContent Delivery Network(CDN)Challenges01LatencyLLMs