leveraging-wasm-for-portable-ai-inference-across-gpus-cpus-os-cloud-native-environments-zhi-wasmnanogpu-cpu-xu-ni-re-chang-hou-mao-chan-zhe-zha-ailia-miley-fu-hung-ying-tai-second-state.pdf

1、Wasm for Portable AI Inference Across GPUs,CPUs,OS&Cloud-Native EnvironmentsHung-Ying Tai,WasmEdgeMiley Fu,WasmEdgeGitHub/Twitter:mileyfu/mileyfuhydai/hydai_twhttps:/ the server side?MicroservicesSaaS/UDFStreaming dataEmbeddedAI inferenceWasm landscape September 6,2023Why Wasm for AI Inference?Probl

2、em:The need for consistent,efficient AI inference across diverse environments.Key Points:Cross-platform consistency(run anywhere without rebuilding image targeting different underlying architecture or cuda version etc).High performance and small footprint.Security and sandboxing benefits.Wasm in Clo

3、ud-Native EnvironmentsIntegration with Kubernetes for scalable AI deployments.Use with Docker for containerized AI inference.Benefits for microservices and serverless architectures.WasmEdge:A WebAssembly Runtime Optimized for AILightweight,high-performance WebAssembly runtime.Designed for AI,IoT,and

4、 edge computing.Compatibility with TensorFlow Lite,ONNX,and other AI frameworks.LlamaEdge,AI tools built on top of WasmEdge runtimea versatile AI deployment platform that enables running large language models(LLMs)across various environments,including edge devices,cloud infrastructure,and multiple p

5、latforms.across various hardware and OS.CPU and GPU compatibility.Execution on different operating systems(Windows,Linux,macOS).Flexibility in deployment:from PCs to cloud environments.Key Features and BenefitsLightweight:The core runtime and API server are less than 30MB,making it suitable for reso

6、urce-constrained devices.Fast:Takes full advantage of device hardware and software acceleration for optimal performance.Cross-platform:Enables LLM deployment across CPUs,GPUs,TPUs,NPUs etc.Open-source:Built on open standards,fostering community contributions and customization.Privacy-focused:Allows