从 EPD 到 SGLang-Omni：图像密集场景推理加速实践与下一代全模态推理架构演进-刘斯宇.pdf

1、1.Transmission:ZMQ IPC OverheadBottleneck:Serialization and memory copying latency when moving massive vision tensors across process boundaries(Tokenizer Scheduler).Overview:Critical Bottlenecks in the Multimodal Pipeline2.Synchronization:The ViT Scaling ParadoxBottleneck:Communication-to-computatio

2、n ratio is too high for Vision Encoders.Evidence:TP 8(523ms)TP 4(465ms)on Qwen2.5-VL-72B(H20).3.Preprocessing:The Hashing FrictionBottleneck:Synchronous,CPU-bound image hashing blocks the request pipeline before scheduling begins.Tokenizer(Image Processing)Serialization&ZMQ(Copying to Socket)step1st

3、ep2step3DeserializationSchedulerstep4Data TransferThe Scaling Challenge:Highres image=Big Tensors that choke standard IPC pipes.The Technical Cost:ZMQ relies on CPU-bound serialization.Multiple socket-based copies create a massive costCumulative Latency:In high-concurrency scenarios,transmission del

4、ays accumulate across the queue.send_pyobjThe 4-Step Shared Memory HandlerStep 1.Write:Tokenizer writes vision tensors directly to Shared Memory,bypassing Python Pickle serialization.Step 2.Dispatch:ZMQ sends only a Metadata Handle(the Pointer).Step 3.Map:Scheduler attaches to the SHM segment and re

5、constructs the tensor via torch.frombuffer(Zero-Copy).Step 4.Unlink:The memory segment is unlinked immediately after mapping to ensure resource safety and prevent leaks.See more#16402Concurrency=1Concurrency=5IMAGE_COUNT=10IMAGE_RES=720pQwen2.5-VL-7B-InstructWhy Tensor Parallelism(TP)Fails for Visio

6、n EncodersThe Evidence:Performance RegressionBenchmark:Qwen2.5-VL-72B on H20 GPUs.The Paradox:Increasing TP size leads to higher latency.TP 4(Baseline):465.8 msTP 8(Regression):523.8 ms(+12.4%Latency)Observation:Vertical scaling(TP)has reached a negative-return threshold.2.Root Cause:Communication o