“Bridging the Gap: Streamlining the Process of Deploying AI onto Processors,” a Presentation from SqueezeBits
0 likes24 views
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/may-2025-embedded-vision-summit-replay/ Taesu Kim, Chief Technology Officer at SqueezeBits, presents the “Bridging the Gap: Streamlining the Process of Deploying AI onto Processors” tutorial at the May 2025 Embedded Vision Summit. Large language models (LLMs) often demand hand-coded conversion scripts for deployment on each distinct processor-specific software stack—a process that’s time-consuming and prone to error. In this presentation, Kim introduces a model-agnostic approach designed to streamline LLM deployment, especially for NVIDIA GPUs. Kim demonstrates how his company’s automated approach cuts through the complexity of constantly evolving software stacks, enabling faster, more reliable LLM adoption. You’ll gain practical insights into a future-proof strategy that boosts coverage for new and upcoming LLM architectures, all while reducing manual coding effort.
“Bridging the Gap: Streamlining the Process of Deploying AI onto Processors,” a Presentation from SqueezeBits
- 1. Bridging the Gap: Streamlining the Process of Deploying AI onto Processors Taesu Kim CTO SqueezeBits Inc.
- 2. The Challenge of AI Deployment • Supporting diverse models • Computer vision • Larger models (LLMs, diffusion …) • Multiple hardware targets (GPUs, Mobile, ..) • Manual conversion scripts needed • Innovation is getting slowed down © 2025 SqueezeBits Inc. 2 New Models & HWs
- 3. Model-Agnostic Conversion Process • PyTorch 2.0 with several tools to support model- agnostic deployment • TorchDynamo: Python-level just-in-time compiler • TorchInductor: Fast codegen with loop level IR • AOTAutograd: Ahead-of-time graph tracer / deep learning compiler integration • Robust and fast, but sometimes harder to use © 2025 SqueezeBits Inc. 3 User Model Script High Level Computation Graph (Torch IR) Low Level Computation Graph (ATen / Prim IR) Compiled Graph
- 4. Our solution: OwLite • Native integration with PyTorch • Supports all PyTorch operators • Multiple precisions, formats, and quantization algorithms • E.g., INT8, FP8 (E4M3, E3M4) • Layer-wise fine-grained quantization • Applicable through simple UI © 2025 SqueezeBits Inc. 4
- 5. Our solution: OwLite © 2025 SqueezeBits Inc. 5 • Quantization-aware-training support • Compressed models can be trained again for accuracy recovery! • Users can reuse their own data loader and training scripts. • Fine-tuned models can be deployed to target devices with same configuration.
- 6. Our solution: OwLite © 2025 SqueezeBits Inc. 6 torch.nn.Conv2d torch.nn.BatchNorm2d torch.nn.SiLU torch.nn.Conv2d + TensorRT Fused Kernel 1 TensorRT Fused Kernel 2 Supports Diverse Hardware
- 7. OwLite in Vision Applications © 2025 SqueezeBits Inc. 7 • Available tasks (examples): • Image classification, object detection, image segmentation, text classification, re-identification, face landmark, pose estimation, and many more • Supports up to 1B parameter models • Models with too many nodes to visualize are currently not supported. • Bring your own model! • Even supports transformer-based ones! (Tested on a NVIDIA A6000, TensorRT)
- 8. Consider Deployment from Model Training Stage © 2025 SqueezeBits Inc. 8 • Models must be trained considering their performance upon deployment. • Larger models with low precision can outperform smaller models. • Rapid prototyping and validation are crucial. Model Evaluation Model Training Model Deployment Data Ingestion Service Monitoring
- 9. Ditto: Model-Agnostic Converter for LLMs © 2025 SqueezeBits Inc. 9 • Model-agnostic converter for LLMs • Currently supports TensorRT-LLM for NVIDIA GPUs • No need for hand-coded conversion script! • Converts models in Transformers library to TensorRT-LLM engines • Diverse graph optimizations to support LLM-specific features Huggingface LLM Model TensorRT-LLM Engine Predefined TRT Network Handwritten Checkpoint Conversion Ditto
- 10. Fits on Chips: Revolutionizing LLMs Deployment © 2025 SqueezeBits Inc. 10 • “Click, Benchmark, Deploy.” • Diverse serving frameworks & hardware • vLLM (NVIDIA GPUs, Intel Gaudi) • TensorRT-LLM (NVIDIA GPUs with Ditto) • More to come (sglang for GPUs, etc.) • Tool for non-expert users • Helps optimize LLM serving – reduce your LLM serving cost!
- 11. Conclusions • Reduce development time with model-agnostic deployment pipelines. • Optimize performance by embedding deployment considerations into the training stage. • Cut serving costs dramatically by exploring a wide range of configuration options. • Leverage existing tools to streamline and accelerate your deployment workflow. © 2025 SqueezeBits Inc. 11
- 12. Try It Now! © 2025 SqueezeBits Inc. 12 • Our deployment pipelines are being served as both open-source software and SaaS toolkits. • Start deploying your own models today with OwLite and Fits on Chips • OwLite has free-tier offers for developers (come visit us at our booth #817!) • Fits on Chips is being served as free. Try it now!
- 13. OwLite (Quantization and Deployment) https://owlite.ai Fits on Chips (LLM Deployment) https://fitsonchips.ai Torch-TRTLLM (Ditto, Open Source) https://github.com/SqueezeBits/Torch-TRTLLM SqueezeBits Tech Blog https://blog.squeezebits.com Come visit us at booth #817 for demo! © 2025 SqueezeBits Inc. 13 Resources