为 Transformer 模型构建视觉推理系统的挑战.pdf

1、Challenges in Architecting Vision Inference Systems for Transformer ModelsCheng WangCTO&SVP,Software&ArchitectureFlex LogixBlind person and the elephant Receptive field is a big limitation2 2023 Flex Logix Technologies,Inc.CNNs are limited by receptive field3 2023 Flex Logix Technologies,Inc.Smaller

2、 receptive fieldSmaller features&objectsLarger receptive fieldLarger features&objectsTransformers use context from whole image4 2023 Flex Logix Technologies,Inc.CNN:Elephant?Transformers:Elephant!Receptive FieldReceptive FieldMAYBEVERY HARD Uses CNN backbone for feature extraction&transformer“head”T

3、ransformer Encoder extracts features from all patches for context Decoder makes predictions based on all extracted features Transformer Encoder/Decoder operations are very different from CNNDETR 2020 The de-facto vision transformer model5 2023 Flex Logix Technologies,Inc.InferX provides flexibility

4、essential for transformers:Each TPU can stream data with:TPU,L1 weight mem,L2 Data mem&DDR TPU natively supports mixed precision Flexible activations in EFLX eFPGA More data bandwidth vs Network-on-Chip based AI Also more flexible data manipulationInferX dynamic TPU Flexible,balanced&memory-efficien

5、t6 2023 Flex Logix Technologies,Inc.First stage is positional encode:PE values are stored eFPGA ROMs EFLX lookup PE“on the fly”to add to the K/Q matrix into the attention headDiving into vision transformers 7 2023 Flex Logix Technologies,Inc.CNN output Second stage multiplies input with 3 matrices f

6、or each head(Q/K/V):Each matrix maps to TPU weightsDiving into vision transformers(2)8 2023 Flex Logix Technologies,Inc.Attention Head#0Attention Head#1 Main part of multi-head attention layer is a challenge on traditional edge accelerators:The(Q,K,V)for each matrix is activation