WebAssembly is Key to Better LLM Performance

0 likes167 views

Samy Fodil, the founder and CEO of Taubyte, emphasizes the importance of WebAssembly (WASM) for enhancing the performance of large language models (LLMs) in coding and cloud applications. Taubyte provides a platform that allows local code to easily scale to global production while addressing challenges related to LLM inference, orchestration, caching, and resource management. By using WASM, developers can achieve lightweight, fast, and secure LLM inference and deployment solutions with simplified implementation and efficient resource handling.

Technology

WebAssembly is Key to Better LLM Performance

1. Samy Fodil Founder & CEO of Taubyte WebAssembly is Key to Better LLM Performance

2. Samy Fodil Founder & CEO of Taubyte

3. Taubyte An Open Source Cloud Platform on Autopilot, where coding in local environment equals scaling to global production. 🚀 Local Coding Global Production

4. Taubyte Locally

5. Taubyte In Prod

6. https://tau.how

7. LLMs Generative AI

8. INPUT Large Language Models OUTPUT

9. Proprietary Open Source ✅Train ✅Host Large Language Models

10. LLM Training

11. LLM Inference 📦Huge PyTorch Images 🔗Complex Dependencies 🔒Lock GPU ressources 🗿Primitive Orchestration Problems

12. 🪽Lightweight 🪣Sand-boxed ⚡Easy to orchestrate LLM Inference WASM to the rescue

13. 🪽 Lightweight <2MB ~4GB

14. 🪽 Lightweight ⚡Fast to provision 🌱Cheap to cache Closer to Data (Data Gravity) & User (Edge Computing)

15. 🪣Sand-boxed 🛡️Secure 🕹️Interfaces Which means No, Restricted, Virtual or Mocked: Networking, Filesystem and more

16. LLM Inference You can use transformers directly from python. This will result in PyTorch and other dependencies. 📦Huge PyTorch Images 🔗Complex Dependencies 🔒Lock GPU ressources

17. LLM Inference You can for example use onnx, llama.cpp, candle (which wrapps llama.cpp) or TensorRT-LLM and have a lower foot print. But it comes with a few challenges: 🔒Lock GPU ressources 🧠Way harder to implement

18. Scaling LLM Inference (lvl1) 🎹Orchestration 🌱Caching

19. Inference Engines for LLM 🤹Load balancing? 🚦Routing?

20. With WASM If we made LLM and AI, in general, available through a set of common host calls we can combine benfits of Inference Engines 🎹Orchestration 🌱Caching WebAssembly 🪽Lightweight 🪣Sand-boxed

21. 🤹Load balancing? 🚦Routing?

22. github.com/taubyte/tau Will provide: 🤹Load-balancing 🚦Routing 🎹Orchestration It’ll also provide abstractions so what’s built locally will work in production with no changes.

23. APP HTTP Inference Engine HTTP HTTP WASM MODULE HOST CALL HOST CALL HOST CALL HTTP HTTP Inference Host Module The idea 1️⃣ 2️⃣

24. Implementation of 1️⃣ tau implement a protocols called `gateway` that will determine what host will be best suited to serve the request based on: WebAssembly caching and dependency modules availability (including host modules) Host Module Resource availability Host Resource availability Other constraints defined by developer like data gravity

25. Node Running Gateway Proto Gateway Protocol Serving Node (Substrate) Serving Node (Substrate) Serving Node (Substrate) Satellite (i.e. LLM Inference) Satellite Satellite MUXED TUNNEL MUXED TUNNEL M UXED TUN N EL ORBIT PROTO ORBIT PROTO ORBIT PROTO

26. Host Module Resource Availability This is still to be implemented feature that will ask Host Module to provide basic metrics for the gateway: Caching score. Example: is the particular model loaded. Resources Availability Score. Example: Is there enough GPU mem to spin-up the model Queue Score. Example: If Host Module uses queues, how filled is the queue.

27. github.com/taubyte/vm-orbit Extending WebAssembly Runtime in a secure way. orbit external process PROXY WASM CALL PROXY ACCESS TO MODULE MEMORY

28. Example

29. github.com/ollama-cloud trunings ollama into a satellite

30. Compile llama.cpp Build plugin

31. Install dreamland Start local Cloud Attach plug-in

32. Login to Local Cloud

33. Create a project

34. Create a function

35. Call Generate Stream Tokens

36. No SDK! Check my previous project github.com/samyfodil/taubyte-llama-satellite Which actually has a nice SDK!

37. Trigger the function

38. - Copy ollama plugin to /tb/plugins - Add it to config In production