“Image Tokenization for Distributed Neural Cascades,” a Presentation from Google and VeriSilicon
0 likes•61 views
Derek Chow, Software Engineer at Google, and Shang-Hung Lin, Vice President of NPU Technology at VeriSilicon, co-present the “Image Tokenization for Distributed Neural Cascades” tutorial at the May 2025 Embedded Vision Summit. Multimodal LLMs promise to bring exciting new abilities to devices! As we see foundational models become more capable, we see compute requirements grow as well. It is not uncommon to see LLMs grow to tens of billions of parameters, at a rate faster than what embedded processors can provide. In this talk, Chow and Lin introduce the concept of a “neural cascade,” a scheme that allows for division of computation across devices. They present a recipe for constructing a neural cascade from a pre-existing LLM and they show how this system harmonizes edge and cloud devices to enable new experiences.
“Image Tokenization for Distributed Neural Cascades,” a Presentation from Google and VeriSilicon
- 1. Image Tokenization for Distributed Neural Cascades Derek Chow Software Engineer, Google Shang-Hung Lin Vice President of NPU Technology VeriSilicon
- 2. What is Tokenization? Tokenization is the process of converting a sensor modality into a neural encoding. © 2025 VeriSilicon and Google 2
- 3. Examples of Tokenizers © 2025 VeriSilicon and Google 3
- 4. Tokenizer is a Feature Extractor © 2025 VeriSilicon and Google 4 ResNet101 Classification Detection Segmentation • Serves as a feature extractor for a neural network • Enables features like classification, generation, RAG
- 5. Multimodal AI © 2025 VeriSilicon and Google 5
- 6. © 2025 VeriSilicon and Google 6 SigLIP / Gemma
- 7. Tokenization Creates a Form a Data Compression © 2025 VeriSilicon and Google 7 • Tokenizer and detokenizer act as a Codec • Saves power during transmission • Saves capacity at rest
- 8. Compute Memory Bandwidth High High High Medium Medium Medium Medium Low Low Low Low Low Low Low Low © 2025 VeriSilicon and Google 8 Diverse Hardware Ecosystem
- 9. © 2025 VeriSilicon and Google 9 World’s Leading Smart Home Products
- 10. Can we combine the strengths of multiple devices for GenAI experiences? We think yes. © 2025 VeriSilicon and Google 10
- 11. Anatomy of a Neural Cascade © 2025 VeriSilicon and Google 11 Yes Yes No Tokenizer Tokenizer Tokenizer Image Tokens Image Tokens Gating Model
- 12. Building a Large Gating Model • We can build a gating model using a VLM • Provide a prompt to describe what you want to detect. i.e.: “Is there an animal present?” • Feed tokenized image into VLM • Check probability of emitting “Yes” or “No” © 2025 VeriSilicon and Google 12 “Is there an animal present?” Text Embedder VLM Image Tokenizer P(“Yes”), P(“No”) VLM Based Gating Model
- 13. Distilling a Smaller Gating Model © 2025 VeriSilicon and Google 13 “Is there an animal present?” Text Embedder VLM Image Tokenizer P(“Yes”), P(“No”) Student Gating Model Teacher Gating Model Gradient Updates
- 14. Composing Models © 2025 VeriSilicon and Google 15 Image Tokenizer Distilled Animal Detector VLM “Describe what the animal is doing” “The squirrel is eating your avocado!” Image Tokens Image Tokens Embedded Device
- 15. Cascades Beyond Two Devices © 2025 VeriSilicon and Google 16 Image Tokens Audio Tokens Health Tokens RAG Queries
- 16. Squeezing Neural Cascade Frontend into Small Devices • Knowledge distillation 17 • Quantization • Sparsity, weight sharing • Hybrid architecture © 2025 VeriSilicon and Google
- 17. Image Token Compression • Reducing image token numbers by text prompt © 2025 VeriSilicon and Google 18 QueCC (ICLR 2025, arxiv:2411.03312) 16x Compression Ratio 36x 144x 576x
- 18. 19 Project Open Se Cura – Edge and Cloud Collaborative Computing Extremely low power consumption • Always on • Ambient computing Realizing large models everywhere • Responsiveness • Privacy (local & cloud) • Computational resources Cloud computing © 2025 VeriSilicon and Google
- 19. Kelvin: A RISC-V ML Accelerator for Edge Kelvin is a RISC-V based ML Accelerator • Open-source design as part of Open Se Cura • Provides familiar framework for programming ML kernels to experts with SIMD/GPU experience • Support for RISC-V Vector and Matrix extensions is in development, targeting 256+ MACs/cycle • Security extensions via CHERI are on our roadmap © 2025 VeriSilicon and Google 20 S C A L A R ML SIMD SIMD T C M
- 20. VeriSilicon AI-Computing IP Product Lineup © 2025 VeriSilicon and Google 21 Inferencing Training Inferencing VIP9X00 (NPU IP) CC9X00TC-MP (GPGPU+NPU IP) Embedded Devices Data Center Server Chips Edge Serer Chips VIP9X00CC (NPU+GPGPU IP) VIP Nano/PICO Sub TOPS Inferencing Incremental Training
- 21. 22 High Efficiency Inference NPU for VLMs & LLMs Qwen2 1.5B VIP9000 4 TOPS 16 GB/s LLaMA2 7B VIP9000 40 TOPS 128 GB/s LLaMA3 70B VIP9400 160 TOPS 512 GB/s Embedded Devices AI-PC, Mobile Edge Server © 2025 VeriSilicon and Google
- 22. Summary and Challenges Summary • Tokenizers provide a framework building multi-modal LLMs • Distillation based training can create a gating mechanism to separate tokenizers from the LLM • Once separated, compute can be distributed between embedded devices and the cloud Challenges • Technical • Memory and compute scaling for tokenizers and LLMs • Infrastructure for training distributed models • Ecosystem • Changing model landscape • Diverse hardware landscape • Fostering community 23 © 2025 VeriSilicon and Google
- 23. Gemma https://ai.google.dev/gemma Project Open Se Cura https://www.opensecura.googlesourc e.com VeriSilicon NPU IP https://www.verisilicon.com/en/IPPor tfolio/VivanteNPUIP 2025 Embedded Vision Summit Visit us at booth 508! 24 Resources MAIN ENTRANCE © 2025 VeriSilicon and Google