Weave AI Controllers (Weave GitOps Office Hours)
0 likes20 views
The document outlines the deployment of large language models (LLMs) on Kubernetes using the Weave AI platform. It introduces the LM Controller, which simplifies the deployment and management of LLMs, making them accessible to development teams through self-service options. Key features include integration with existing DevOps pipelines, a curated LLM catalog, and pre-built engines for various CPU architectures.
Weave AI Controllers (Weave GitOps Office Hours)
- 1. Confidential do not distribute 1 Generative AI Automation for private Enterprise LLMs Part 1: LM-Controller
- 2. Confidential do not distribute 2 ● AI Models and Applications are the new class of Kubernetes workloads ● We start tackling this from LLMs ● Enterprise already invested in CPU-based Kubernetes clusters Enterprise AI workloads
- 3. Confidential do not distribute 3 ● AI Application Developers shouldn’t worry about the complexity of model deployment. ● Platform Teams: LLMs become platform components ○ Security and Governance: signing and verification ○ RBAC and Tenancy ○ Standardization across organizations ○ Available for the Dev teams via self-service portals Why Weave AI?
- 4. Confidential do not distribute 4 ● Day 0 - Out-of-the-box experiences ○ weave-ai install ○ weave-ai run zephyr-7b-beta ● Day 1 - Integrate them to your DevOps / GitOps pipelines ○ weave-ai install --export ● Day 2 - Build and maintain model catalog for the Dev teams ○ flux commands ○ Fine-tuning models / RAG data pipelines Why Weave AI?
- 5. Confidential do not distribute 5
- 6. Confidential do not distribute 6 ● The first controller released as part of the Weave AI Controllers ● LM Controller is a Flux controller that helps deploy Large Language Models on Kubernetes. ● It supports LLMs in the Flux OCI format. ● It uses Flux Source Controller as the in-cluster model cache. What is LM Controller?
- 7. Confidential do not distribute 7 LLMs are snowflakes
- 8. Confidential do not distribute 8 Hugging Face Compatible Models GitHub / GitLab CI Your App LLM Serving Your Data CPU or GPU on Cloud or on-Prem fine-tuning store packaged pulled deploy context manage LLM as Flux OCI
- 9. Confidential do not distribute 9 Why use LM Controller? LLM Serving LLMs injects all required information to the deployment units LM Controller
- 10. Confidential do not distribute 10 ● A curated list of LLM catalog ○ In Flux’s OCI format ● Flux’s Source Controller as in-Cluster model Cache ○ No PVC required ● A controller that takes care of this and that LLM parameters for you ● A set of pre-built OpenAI API Compatible engines ○ No-AVX, AVX, AVX2, AVX512 and more to come ● An easy-to-use CLI What Weave AI provides so far
- 11. Confidential do not distribute 11 It’s Demo Time