14 November 2024 - Conf 42 - Prompt Engineering - Codeless Generative AI Pipelines_ Prompt Automation
0 likes98 views
The document presents a discussion on the challenges of unstructured data and introduces vector embeddings as a solution, highlighting the role of Milvus, an open-source vector database designed for AI projects. It outlines features of Milvus and various data processing techniques, including retrieval-augmented generation to enhance data analysis. Additionally, it includes references to multiple programming resources and tools related to data handling and AI integrations.
14 November 2024 - Conf 42 - Prompt Engineering - Codeless Generative AI Pipelines_ Prompt Automation
- 1. 1 | © Copyright 2024 Zilliz 1 Codeless Generative AI Pipelines: Prompt Automation Tim Spann @ Zilliz
- 2. 2 | © Copyright 2024 Zilliz 2 2 | © Copyright 10/22/23 Zilliz 2 | © Copyright 2024 Zilliz Tim Spann Principal Developer Advocate, Milvus [email protected] https://www.linkedin.com/in/timothyspann/ https://x.com/PaaSDev
- 3. 3 | © Copyright 2024 Zilliz 3 These Slides X
- 4. 4 | © Copyright 2024 Zilliz 4 The Challenge of Unstructured Data ● Problem: Unstructured data comes in lots of forms, no easy way to interact with it all ● Solution: Vector embeddings ● How: Neural networks e.g. embedding models Vector Databases
- 5. 5 | © Copyright 2024 Zilliz 5 Unstructured Data is Everywhere Unstructured data is any data that does not conform to a predefined data model. Currently, 90% of unstructured data is never analyzed. Images Videos and more! Text
- 6. Vn, 1 … … … 1 2 3 4 5 Transform into Vectors Unstructured Data Images User Generated Content Video Documents Audio Vector Embeddings Perform Approximate Nearest Neighbor Similarity Search Perform Query Get Results Store in Vector Database How Similarity Search Works
- 7. 7 | © Copyright Zilliz 7 About Milvus Milvus is an open-source vector database for GenAI projects. pip install on your laptop, plug into popular AI dev tools, and push to production with a single line of code. 29K GitHub Stars 25M Downloads 250 Contributors 2,600 Forks Easy Setup Pip-install to start coding in a notebook within seconds Integration Plug into OpenAI, Langchain, LlmaIndex, and many more Reusable Code Write once, and deploy with one line of code into the production environment Feature-rich Dense & sparse embeddings, filtering, reranking and beyond
- 8. 8 | © Copyright Zilliz 8 pip install pymilvus Milvus Lite
- 9. 9 Retrieval-Augmented Generation (RAG) 2024 A technique that combines the strength of retrieval-based and generative models: ● Improve accuracy and relevance ● Eliminate hallucination ● Provide domain-specific knowledge
- 10. 1 0 RAG : an economic perspective 2024 A business model that bridges public data and private data ● Data sovereignty ● You can't and shouldn't give your private data to others
- 11. 11 | © Copyright Zilliz 11
- 12. 12 | © Copyright Zilliz 12 DataFlow Pipelines Can Help External Context Ingest Ingesting, routing, clean, enrich, transforming, parsing, chunking and vectorizing structured, unstructured, semistructured, binary data and documents Prompt engineering Crafting and structuring queries to optimize LLM responses Context Retrieval Enhancing LLM with external context such as Retrieval Augmented Generation (RAG) Roundtrip Interface Act as a Discord, REST, Kafka, SQL, Slack bot to roundtrip discussions
- 13. 13 | © Copyright Zilliz 13 https://medium.com/cloudera-inc/getting-ready-for-apache-nifi-2-0-5a5e6a67f450 NiFi 2.0.0 Features ● Python Integration ● Parameters ● JDK 21+ ● JSON Flow Serialization ● Rules Engine for Development Assistance ● Run Process Group as Stateless ● flow.json.gz https://cwiki.apache.org/confluence/display/NIFI/NiFi+2.0+Release+Goals
- 14. 14 | © Copyright Zilliz 14 Extract Company Names ● Python 3.10+ ● Hugging Face, NLP, SpaCY, PyTorch https://github.com/tspannhw/FLaNK-python-ExtractCompanyName-processor
- 15. 15 | © Copyright Zilliz 15 CaptionImage ● Python 3.10+ ● Hugging Face ● Salesforce/blip-image-captioning-large ● Generate Captions for Images ● Adds captions to FlowFile Attributes ● Does not require download or copies of your images https://github.com/tspannhw/FLaNK-python-processors
- 16. 16 | © Copyright Zilliz 16 RESNetImageClassification ● Python 3.10+ ● Hugging Face ● Transformers ● Pytorch ● Datasets ● microsoft/resnet-50 ● Adds classification label to FlowFile Attributes ● Does not require download or copies of your images https://github.com/tspannhw/FLaNK-python-processors
- 17. 17 | © Copyright Zilliz 17 NSFWImageDetection ● Python 3.10+ ● Hugging Face ● Transformers ● Falconsai/nsfw_image_detection ● Adds normal and nsfw to FlowFile Attributes ● Gives score on safety of image ● Does not require download or copies of your images https://github.com/tspannhw/FLaNK-python-processors
- 18. 18 | © Copyright Zilliz 18 FacialEmotionsImageDetection ● Python 3.10+ ● Hugging Face ● Transformers ● facial_emotions_image_detection ● Image Classification ● Adds labels/scores to FlowFile Attributes ● Does not require download or copies of your images https://github.com/tspannhw/FLaNK-python-processors
- 19. 19 | © Copyright Zilliz 19 Letʼs do a metamorphosis on your data. Donʼt fear changing data. You don’t need to be a brilliant writer to stream data. Franz Kafka was a German-speaking Bohemian novelist and short-story writer, widely regarded as one of the major figures of 20th-century literature. His work fuses elements of realism and the fantastic. Wikipedia YES, FRANZ, IT’S KAFKA
- 20. 20 | © Copyright Zilliz 20 20 Streams Replication Manager (SRM) • Event Replication engine for Kafka • Supports active-active, multi-cluster, cross DC replication scenarios • Leverage Kafka Connect for scalability and HA • Replicate data and configurations (ACL, partitioning, new topics, etc) • Offset translation for simplified failover • Integrate replication monitoring with SMM
- 21. 21 | © Copyright Zilliz 21 Open Source Edition •Apache NiFi in Docker •Try new features quickly •Develop applications locally ● Docker NiFi ○ docker run --name nifi -p 8443:8443 -d -e SINGLE_USER_CREDENTIALS_USERNAME=admin -e SINGLE_USER_CREDENTIALS_PASSWORD=ctsBtRBKHRAx69EqUghv vgEvjnaLjFEB apache/nifi:latest ● Licensed under the ASF License ● Unsupported ● NiFi 1.28 and NiFi 2.0.0 https://hub.docker.com/r/apache/nifi
- 22. 22 | © Copyright Zilliz 22 https://medium.com/cloudera-inc/streaming-street-cams-to-yolo-v8-with-python-and-nifi-to-minio-s3-3277e73723ce Street Cameras
- 23. 23 | © Copyright Zilliz 23 | © Copyright Zilliz 23 RESOURCES
- 24. 24 | © Copyright Zilliz 24 Vector Database Resources Give Milvus a Star! Chat with me on Discord! https://github.com/milvus-io/milvus
- 27. https://medium.com/@tspann/unstructured-data-processing-with-a-raspberry-pi-ai-kit-c959dd7fff47 Raspberry Pi AI Kit Hailo Edge AI
- 28. 28 | © Copyright 2024 Zilliz 28 28 This week in Milvus, Towhee, Attu, GPT Cache, Gen AI, LLM, Apache NiFi, Apache Flink, Apache Kafka, ML, AI, Apache Spark, Apache Iceberg, Python, Java, Vector DB and Open Source friends. https://bit.ly/32dAJft https://github.com/milvus-io/milvus AIM Weekly by Tim Spann