朱江云-从 LLM 到 Omni：vLLM-Omni 的全模态推理架构演进.pdf

《朱江云-从 LLM 到 Omni：vLLM-Omni 的全模态推理架构演进.pdf》由会员分享，可在线阅读，更多相关《朱江云-从 LLM 到 Omni：vLLM-Omni 的全模态推理架构演进.pdf（37页珍藏版）》请在三个皮匠报告上搜索。

1、从 LLM 到 Omni：vLLM-Omni 的全模态推理架构演进朱江云vLLM-Omni CommiterAbout usRoger WangvLLM&vLLM-Omni CommitterHan GaovLLM-OmniCommitterHongsheng LiuvLLM-OmniCommitterJiangyun ZhuvLLM&vLLM-Omni Committerdocs.vllm.ai/projects/vllm-omni/en/latestWednesday 19:30 PDTOur GoalBuild the fastest andeasiest-to-use open-sou

2、rceOmni-Modality model inference&serving engineOmni-Modality modelsOmni-modality:Text,image,video,and audio data processingNon-autoregressive Architectures:extend the AR support of vLLM to Diffusion Transformers(DiT)and other parallel generation modelsHeterogeneous outputs:from traditional text gene

3、ration to multimodal outputsfrom vllm_omni import Omni#Example prompts.inputs=prompt:prompt,multi_modal_data:video:video_frames,audio:audio_data,#Create an omni with HF model name.omni=Omni(model=“Qwen/Qwen3-Omni-30B-A3B-Instruct)#Generate texts and audio from the multi-modality inputs.outputs=omni.

4、generate(inputs)vLLM-mni API(1):Omni classA Python interface for offline batched inference for Qwen3-Omni/Qwen-Imagehttps:/docs.vllm.ai/projects/vllm-omni/en/latest/examples/from vllm_omni import Omni#Example prompts.inputs=A cup of coffee on the table“#Create an omni with HF model name.omni=Omni(mo

5、del=Qwen/Qwen-Image-2512)#Generate image from multi-modality inputs.outputs=omni.generate(inputs)vLLM-Omni API(2):OpenAI-compatible serverA FastAPI-based server for online serving for Qwen3-Omni$vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct-omni-port 8091$curl-sS-X POST http:/localhost:8091/v1/chat/co

6、mpletions -H Content-Type:application/json -d model:Qwen/Qwen3-Omni-30B-A3B-Instruct,messages:Why is this video funny?sampling_params_list:$sampling_params_list,ServerClientvLLM-Omni API(2):OpenAI-compatible server$vllm serve Qwen/Qwen-Image-2512-omni-port 8091$curl-X POST http:/localhost:8091/v1/im