【RAG落地利器】向量数据库Weaviate部署与使用教程,收藏这一篇就够了!!

Weaviate 部署

1. 简介

Weaviate 是一种开源的向量搜索引擎数据库,允许以类属性的方式存储 JSON 文档,并将机器学习向量附加到这些文档上,以在向量空间中表示它们。Weaviate 支持语义搜索、问答提取、分类等功能,并且可以通过 GraphQL-API 轻松访问数据。

官网地址:https://weaviate.io/

2. 安装 Weaviate

从 Docker Hub 下载 Weaviate 的最新镜像:

docker pull semitechnologies/weaviate:latest   

如果拉取镜像速度较慢,可以尝试替换镜像源。

2.3 运行 Weaviate 容器

使用以下命令运行 Weaviate 容器:

docker run -d --name weaviate \
    --restart=always \
    -p 8080:8080 \
    -p 50051:50051 \
    -e "AUTHENTICATION_APIKEY_ENABLED=true" \
    -e "AUTHENTICATION_APIKEY_ALLOWED_KEYS=test-secret-key,test2-secret-key" \
    -e "AUTHENTICATION_APIKEY_USERS=test@2024.com,test2@2024.com" \
    -e "AUTHORIZATION_ADMINLIST_ENABLED=true" \
    -e "AUTHORIZATION_ADMINLIST_USERS=test@2024.com" \
    -e "AUTHORIZATION_ADMINLIST_READONLY_USERS=test2@2024.com" \
    -e WEAVIATE_HOSTNAME=0.0.0.0 \
    semitechnologies/weaviate:latest

参数说明

  • -d: 让容器在后台运行。

  • --name weaviate: 给容器命名为weaviate

  • --restart=always: 配置容器在宿主机重启后自动启动。

  • -p 8080:8080: 将容器内的 8080 端口映射到宿主机的 8080 端口。

  • -p 50051:50051: 将容器内的 50051 端口映射到宿主机的 50051 端口。

  • -e "AUTHENTICATION_APIKEY_ENABLED=true": 启用 API 密钥认证功能。

  • -e "AUTHENTICATION_APIKEY_ALLOWED_KEYS=test-secret-key,test2-secret-key": 指定允许使用的 API 密钥列表。

  • -e "AUTHENTICATION_APIKEY_USERS=test@2024.com,test2@2024.com": 关联密钥与用户邮箱。

  • -e "AUTHORIZATION_ADMINLIST_ENABLED=true": 开启管理员列表授权。

  • -e "AUTHORIZATION_ADMINLIST_USERS=test@2024.com": 指定管理员列表中的用户。

  • -e "AUTHORIZATION_ADMINLIST_READONLY_USERS=test2@2024.com": 指定只读权限的用户列表。

  • -e WEAVIATE_HOSTNAME=0.0.0.0: 设置 Weaviate 的主机名,监听所有可用网络接口。

  • semitechnologies/weaviate:latest: 指定要从 Docker Hub 下载并运行的 Weaviate 镜像的最新版本。

3. 测试连接

3.1 安装 Weaviate 客户端

使用 pip 安装 Weaviate 客户端:

pip install -U weaviate-client   
3.2 编写测试脚本

创建一个test.py文件,内容如下:

import weaviate
from weaviate.auth import AuthApiKey

# 连接到本地部署的 Weaviate
client = weaviate.connect_to_local(
    auth_credentials=AuthApiKey("test-secret-key")
)

# 或者自定义连接
client = weaviate.connect_to_custom(
    skip_init_checks=False,
    http_host="127.0.0.1",
    http_port=8080,
    http_secure=False,
    grpc_host="127.0.0.1",
    grpc_port=50051,
    grpc_secure=False,
    # 对应 AUTHENTICATION_APIKEY_ALLOWED_KEYS 中的密钥
    # 注意:此处只需要密钥即可,不需要用户名称
    auth_credentials=AuthApiKey("test-secret-key")
)

# 检查连接是否成功
print(client.is_ready())

# 关闭连接
print(client.close())
3.3 运行测试脚本

在终端中运行测试脚本:

python test.py   

如果输出True,则表示连接成功。

可以通过浏览器访问地址:

http://localhost:8080/v1/docs

使用python操作Weaviate向量数据库

以下是使用 Python 操作 Weaviate 向量数据库的完整示例,涵盖连接数据库、检查集合是否存在、创建集合、插入数据、查询数据以及删除集合等操作。

1. 安装 Weaviate Python 客户端

首先,确保你已经安装了 Weaviate 的 Python 客户端:

pip install weaviate-client   

2. 连接 Weaviate 数据库

import weaviate
from weaviate.auth import AuthApiKey

# 连接到本地 Weaviate 实例
client = weaviate.connect_to_local(
    auth_credentials=AuthApiKey("test-secret-key")
)

# 或者自定义连接
client = weaviate.connect_to_custom(
    http_host="127.0.0.1",
    http_port=8080,
    http_secure=False,
    grpc_host="127.0.0.1",
    grpc_port=50051,
    grpc_secure=False,
    auth_credentials=AuthApiKey("test-secret-key")
)

# 检查连接是否成功
print(client.is_ready())

3. 检查集合是否存在

def check_collection_exists(client: weaviate.WeaviateClient, collection_name: str) -> bool:
    """
    检查集合是否存在
    :param client: Weaviate 客户端
    :param collection_name: 集合名称
    :return: True 或 False
    """
    try:
        collections = client.collections.list_all()
        return collection_name in collections
    except Exception as e:
        print(f"检查集合异常: {e}")
        return False

4. 创建集合

def create_collection(client: weaviate.WeaviateClient, collection_name: str):
    """
    创建集合
    :param client: Weaviate 客户端
    :param collection_name: 集合名称
    """
    collection_obj = {
        "class": collection_name,
        "description": "A collection for product information",
        "vectorizer": "none",  # 假设你会上传自己的向量
        "vectorIndexType": "hnsw",
        "vectorIndexConfig": {
            "distance": "cosine",
            "efConstruction": 200,
            "maxConnections": 64
        },
        "properties": [
            {
                "name": "text",
                "description": "The text content",
                "dataType": ["text"],
                "tokenization": "word",
                "indexFilterable": True,
                "indexSearchable": True
            }
        ]
    }
    try:
        client.collections.create_from_dict(collection_obj)
        print(f"创建集合 '{collection_name}' 成功.")
    except weaviate.exceptions.UnexpectedStatusCodeException as e:
        print(f"创建集合异常: {e}")

5. 插入数据

def save_documents(client: weaviate.WeaviateClient, collection_name: str, documents: list):
    """
    向集合中插入数据
    :param client: Weaviate 客户端
    :param collection_name: 集合名称
    :param documents: 文档列表
    """
    collection = client.collections.get(collection_name)
    for doc in documents:
        content = doc  # 假设文档是简单的字符串
        vector = [0.1, 0.2, 0.3]  # 假设这是你的向量
        properties = {
            "text": content
        }
        try:
            uuid = collection.data.insert(properties=properties, vector=vector)
            print(f"文档添加内容: {content[:30]}..., uuid: {uuid}")
        except Exception as e:
            print(f"添加文档异常: {e}")

6. 查询数据

def query_vector_collection(client: weaviate.WeaviateClient, collection_name: str, query: str, k: int) -> list:
    """
    从集合中查询数据
    :param client: Weaviate 客户端
    :param collection_name: 集合名称
    :param query: 查询字符串
    :param k: 返回的结果数量
    :return: 查询结果列表
    """
    vector = [0.1, 0.2, 0.3]  # 假设这是你的查询向量
    collection = client.collections.get(collection_name)
    response = collection.query.near_vector(
        near_vector=vector,
        limit=k
    )
    documents = [res.properties['text'] for res in response.objects]
    return documents

7. 删除集合

def delete_collection(client: weaviate.WeaviateClient, collection_name: str):
    """
    删除集合
    :param client: Weaviate 客户端
    :param collection_name: 集合名称
    """
    try:
        client.collections.delete(collection_name)
        print(f"删除集合 '{collection_name}' 成功.")
    except Exception as e:
        print(f"删除集合异常: {e}")

8. 示例使用

if __name__ == "__main__":
    # 连接 Weaviate
    client = weaviate.connect_to_local(auth_credentials=AuthApiKey("test-secret-key"))

    collection_name = "MyCollection"

    # 检查集合是否存在
    if not check_collection_exists(client, collection_name):
        # 创建集合
        create_collection(client, collection_name)

    # 插入数据
    documents = ["This is a test document.", "Another document for testing."]
    save_documents(client, collection_name, documents)

    # 查询数据
    query_results = query_vector_collection(client, collection_name, "test", 2)
    print("查询结果:", query_results)

    # 删除集合
    delete_collection(client, collection_name)

    # 关闭连接
    client.close()

如何实现语义检索

在TrusRAG项目中,对上面教程进行了封装,具体链接如下:

https://github.com/gomate-community/TrustRAG/blob/pipeline/trustrag/modules/engine/weaviate_cli.py

WeaviateEngine实现如下:

from typing import List, Dict, Any, Optional, Union
import numpy as np
import weaviate
from weaviate import WeaviateClient
from weaviate.collections import Collection
import weaviate.classes.config as wc
from weaviate.classes.config import Property, DataType
from trustrag.modules.retrieval.embedding import EmbeddingGenerator
from  weaviate.classes.query import MetadataQuery

class WeaviateEngine:
    def __init__(
            self,
            collection_name: str,
            embedding_generator: EmbeddingGenerator,
            client_params: Dict[str, Any] = {
                "http_host": "localhost",
                "http_port": 8080,
                "http_secure": False,
                "grpc_host": "localhost",
                "grpc_port": 50051,
                "grpc_secure": False,
            },
    ):
        """
        Initialize the Weaviate vector store.

        :param collection_name: Name of the Weaviate collection
        :param embedding_generator: An instance of EmbeddingGenerator to generate embeddings
        :param weaviate_client_params: Dictionary of parameters to pass to Weaviate client
        """
        self.collection_name = collection_name
        self.embedding_generator = embedding_generator

        # Initialize Weaviate client with provided parameters
        self.client = weaviate.connect_to_custom(
            skip_init_checks=False,
            **client_params
        )

        # Create collection if it doesn't exist
        if not self._collection_exists():
            self._create_collection()

    def _collection_exists(self) -> bool:
        """Check if collection exists in Weaviate."""
        try:
            collections = self.client.collections.list_all()
            collection_names = [c.lower() for c in collections]
            return self.collection_name in collection_names
        except Exception as e:
            print(f"Error checking collection existence: {e}")
            return False

    def _create_collection(self):
        """Create a new collection in Weaviate."""
        try:
            self.client.collections.create(
                name=self.collection_name,
                # Define properties of metadata
                properties=[
                    wc.Property(
                        name="text",
                        data_type=wc.DataType.TEXT
                    ),
                    wc.Property(
                        name="title",
                        data_type=wc.DataType.TEXT,
                        skip_vectorization=True
                    ),
                ]
            )
        except Exception as e:
            print(f"Error creating collection: {e}")
            raise

    def upload_vectors(
            self,
            vectors: Union[np.ndarray, List[List[float]]],
            payload: List[Dict[str, Any]],
            batch_size: int = 100
    ):
        """
        Upload vectors and payload to the Weaviate collection.

        :param vectors: A numpy array or list of vectors to upload
        :param payload: A list of dictionaries containing the payload for each vector
        :param batch_size: Number of vectors to upload in a single batch
        """
        if not isinstance(vectors, np.ndarray):
            vectors = np.array(vectors)
        if len(vectors) != len(payload):
            raise ValueError("Vectors and payload must have the same length.")

        collection = self.client.collections.get(self.collection_name)

        # Process in batches
        for i in range(0, len(vectors), batch_size):
            batch_vectors = vectors[i:i + batch_size]
            batch_payload = payload[i:i + batch_size]

            try:
                with collection.batch.dynamic() as batch:
                    for idx, (properties, vector) in enumerate(zip(batch_payload, batch_vectors)):
                        # Separate text content and other metadata
                        text_content = properties.get('description',
                                                      '')  # Assuming 'description' is the main text field
                        metadata = {k: v for k, v in properties.items() if k != 'description'}

                        # Prepare the properties dictionary
                        properties_dict = {
                            "text": text_content,
                            "title": metadata.get('title', f'Object {idx}')  # Using title from metadata or default
                        }

                        # Add the object with properties and vector
                        batch.add_object(
                            properties=properties_dict,
                            vector=vector
                        )
            except Exception as e:
                print(f"Error uploading batch: {e}")
                raise

    def search(
            self,
            text: str,
            query_filter: Optional[Dict[str, Any]] = None,
            limit: int = 5
    ) -> List[Dict[str, Any]]:
        """
        Search for the closest vectors in the collection based on the input text.

        :param text: The text query to search for
        :param query_filter: Optional filter to apply to the search
        :param limit: Number of closest results to return
        :return: List of payloads from the closest vectors
        """
        # Generate embedding for the query text
        vector = self.embedding_generator.generate_embedding(text)
        print(vector.shape)
        collection = self.client.collections.get(self.collection_name)

        # Prepare query arguments
        query_args = {
            "near_vector": vector,
            "limit": limit,
            "return_metadata": MetadataQuery(distance=True)
        }

        # Add filter if provided
        if query_filter:
            query_args["filter"] = query_filter

        results = collection.query.near_vector(**query_args)

            # Convert results to the same format as QdrantEngine
        payloads = []
        for obj in results.objects:
            payload = obj.properties.get('metadata', {})
            payload['text'] = obj.properties.get('text', '')
            payload['_distance'] = obj.metadata.distance
            payloads.append(payload)

        return payloads


    def build_filter(self, conditions: List[Dict[str, Any]]) -> Dict[str, Any]:
        """
        Build a Weaviate filter from a list of conditions.

        :param conditions: A list of conditions, where each condition is a dictionary with:
                         - key: The field name to filter on
                         - match: The value to match
        :return: A Weaviate filter object
        """
        filter_dict = {
            "operator": "And",
            "operands": []
        }

        for condition in conditions:
            key = condition.get("key")
            match_value = condition.get("match")
            if key and match_value is not None:
                filter_dict["operands"].append({
                    "path": [f"metadata.{key}"],
                    "operator": "Equal",
                    "valueString": str(match_value)
                })

        return filter_dict if filter_dict["operands"] else None  

测试代码如下:

from trustrag.modules.retrieval.embedding import SentenceTransformerEmbedding
from trustrag.modules.engine.weaviate_cli import WeaviateEngine
if __name__ == '__main__':
    # 初始化 MilvusEngine
    local_embedding_generator = SentenceTransformerEmbedding(model_name_or_path=r"H:\pretrained_models\mteb\all-MiniLM-L6-v2", device="cuda")
    weaviate_engine = WeaviateEngine(
        collection_name="startups",
        embedding_generator=local_embedding_generator,
        client_params={
            "http_host": "localhost",
            "http_port": 8080,
            "http_secure": False,
            "grpc_host": "localhost",
            "grpc_port": 50051,
            "grpc_secure": False,
        }
    )

    documents = [
        {"name": "SaferCodes", "images": "https://safer.codes/img/brand/logo-icon.png",
         "alt": "SaferCodes Logo QR codes generator system forms for COVID-19",
         "description": "QR codes systems for COVID-19.\nSimple tools for bars, restaurants, offices, and other small proximity businesses.",
         "link": "https://safer.codes", "city": "Chicago"},
        {"name": "Human Practice",
         "images": "https://d1qb2nb5cznatu.cloudfront.net/startups/i/373036-94d1e190f12f2c919c3566ecaecbda68-thumb_jpg.jpg?buster=1396498835",
         "alt": "Human Practice -  health care information technology",
         "description": "Point-of-care word of mouth\nPreferral is a mobile platform that channels physicians\u2019 interest in networking with their peers to build referrals within a hospital system.\nHospitals are in a race to employ physicians, even though they lose billions each year ($40B in 2014) on employment. Why ...",
         "link": "http://humanpractice.com", "city": "Chicago"},
        {"name": "StyleSeek",
         "images": "https://d1qb2nb5cznatu.cloudfront.net/startups/i/3747-bb0338d641617b54f5234a1d3bfc6fd0-thumb_jpg.jpg?buster=1329158692",
         "alt": "StyleSeek -  e-commerce fashion mass customization online shopping",
         "description": "Personalized e-commerce for lifestyle products\nStyleSeek is a personalized e-commerce site for lifestyle products.\nIt works across the style spectrum by enabling users (both men and women) to create and refine their unique StyleDNA.\nStyleSeek also promotes new products via its email newsletter, 100% personalized ...",
         "link": "http://styleseek.com", "city": "Chicago"},
        {"name": "Scout",
         "images": "https://d1qb2nb5cznatu.cloudfront.net/startups/i/190790-dbe27fe8cda0614d644431f853b64e8f-thumb_jpg.jpg?buster=1389652078",
         "alt": "Scout -  security consumer electronics internet of things",
         "description": "Hassle-free Home Security\nScout is a self-installed, wireless home security system. We've created a more open, affordable and modern system than what is available on the market today. With month-to-month contracts and portable devices, Scout is a renter-friendly solution for the other ...",
         "link": "http://www.scoutalarm.com", "city": "Chicago"},
        {"name": "Invitation codes", "images": "https://invitation.codes/img/inv-brand-fb3.png",
         "alt": "Invitation App - Share referral codes community ",
         "description": "The referral community\nInvitation App is a social network where people post their referral codes and collect rewards on autopilot.",
         "link": "https://invitation.codes", "city": "Chicago"},
        {"name": "Hyde Park Angels",
         "images": "https://d1qb2nb5cznatu.cloudfront.net/startups/i/61114-35cd9d9689b70b4dc1d0b3c5f11c26e7-thumb_jpg.jpg?buster=1427395222",
         "alt": "Hyde Park Angels - ",
         "description": "Hyde Park Angels is the largest and most active angel group in the Midwest. With a membership of over 100 successful entrepreneurs, executives, and venture capitalists, the organization prides itself on providing critical strategic expertise to entrepreneurs and ...",
         "link": "http://hydeparkangels.com", "city": "Chicago"},
        {"name": "GiveForward",
         "images": "https://d1qb2nb5cznatu.cloudfront.net/startups/i/1374-e472ccec267bef9432a459784455c133-thumb_jpg.jpg?buster=1397666635",
         "alt": "GiveForward -  health care startups crowdfunding",
         "description": "Crowdfunding for medical and life events\nGiveForward lets anyone to create a free fundraising page for a friend or loved one's uncovered medical bills, memorial fund, adoptions or any other life events in five minutes or less. Millions of families have used GiveForward to raise more than $165M to let ...",
         "link": "http://giveforward.com", "city": "Chicago"},
        {"name": "MentorMob",
         "images": "https://d1qb2nb5cznatu.cloudfront.net/startups/i/19374-3b63fcf38efde624dd79c5cbd96161db-thumb_jpg.jpg?buster=1315734490",
         "alt": "MentorMob -  digital media education ventures for good crowdsourcing",
         "description": "Google of Learning, indexed by experts\nProblem: Google doesn't index for learning. Nearly 1 billion Google searches are done for \"how to\" learn various topics every month, from photography to entrepreneurship, forcing learners to waste their time sifting through the millions of results.\nMentorMob is ...",
         "link": "http://www.mentormob.com", "city": "Chicago"},
        {"name": "The Boeing Company",
         "images": "https://d1qb2nb5cznatu.cloudfront.net/startups/i/49394-df6be7a1eca80e8e73cc6699fee4f772-thumb_jpg.jpg?buster=1406172049",
         "alt": "The Boeing Company -  manufacturing transportation", "description": "",
         "link": "http://www.boeing.com", "city": "Berlin"},
        {"name": "NowBoarding \u2708\ufe0f",
         "images": "https://static.above.flights/img/lowcost/envelope_blue.png",
         "alt": "Lowcost Email cheap flights alerts",
         "description": "Invite-only mailing list.\n\nWe search the best weekend and long-haul flight deals\nso you can book before everyone else.",
         "link": "https://nowboarding.club/", "city": "Berlin"},
        {"name": "Rocketmiles",
         "images": "https://d1qb2nb5cznatu.cloudfront.net/startups/i/158571-e53ddffe9fb3ed5e57080db7134117d0-thumb_jpg.jpg?buster=1361371304",
         "alt": "Rocketmiles -  e-commerce online travel loyalty programs hotels",
         "description": "Fueling more vacations\nWe enable our customers to travel more, travel better and travel further. 20M+ consumers stock away miles & points to satisfy their wanderlust.\nFlying around or using credit cards are the only good ways to fill the stockpile today. We've built the third way. Customers ...",
         "link": "http://www.Rocketmiles.com", "city": "Berlin"}

    ]
    vectors = weaviate_engine.embedding_generator.generate_embeddings([doc["description"] for doc in documents])
    print(vectors.shape)
    payload = [doc for doc in documents]

    # Upload vectors and payload
    weaviate_engine.upload_vectors(vectors=vectors, payload=payload)
    


    # 构建过滤器并搜索
    conditions = [
        {"key": "city", "match": "Berlin"},
    ]
    custom_filter = weaviate_engine.build_filter(conditions)

    # 搜索柏林的度假相关创业公司
    results = weaviate_engine.search(
        text="vacations",
        # query_filter=custom_filter,
        limit=5
    )
    print(results)

输出如下:

{'text': "Fueling more vacations\nWe enable our customers to travel more, travel better and travel further. 20M+ consumers stock away miles & points to satisfy their wanderlust.\nFlying around or using credit cards are the only good ways to fill the stockpile today. We've built the third way. Customers ...", '_distance': 0.5216099619865417}   

最后

为什么要学AI大模型

当下,⼈⼯智能市场迎来了爆发期,并逐渐进⼊以⼈⼯通⽤智能(AGI)为主导的新时代。企业纷纷官宣“ AI+ ”战略,为新兴技术⼈才创造丰富的就业机会,⼈才缺⼝将达 400 万!

DeepSeek问世以来,生成式AI和大模型技术爆发式增长,让很多岗位重新成了炙手可热的新星,岗位薪资远超很多后端岗位,在程序员中稳居前列。

在这里插入图片描述

与此同时AI与各行各业深度融合,飞速发展,成为炙手可热的新风口,企业非常需要了解AI、懂AI、会用AI的员工,纷纷开出高薪招聘AI大模型相关岗位。
在这里插入图片描述
最近很多程序员朋友都已经学习或者准备学习 AI 大模型,后台也经常会有小伙伴咨询学习路线和学习资料,我特别拜托北京清华大学学士和美国加州理工学院博士学位的鲁为民老师给大家这里给大家准备了一份涵盖了AI大模型入门学习思维导图、精品AI大模型学习书籍手册、视频教程、实战学习等录播视频 全系列的学习资料,这些学习资料不仅深入浅出,而且非常实用,让大家系统而高效地掌握AI大模型的各个知识点。

这份完整版的大模型 AI 学习资料已经上传CSDN,朋友们如果需要可以微信扫描下方CSDN官方认证二维码免费领取【保证100%免费

AI大模型系统学习路线

在面对AI大模型开发领域的复杂与深入,精准学习显得尤为重要。一份系统的技术路线图,不仅能够帮助开发者清晰地了解从入门到精通所需掌握的知识点,还能提供一条高效、有序的学习路径。

img

但知道是一回事,做又是另一回事,初学者最常遇到的问题主要是理论知识缺乏、资源和工具的限制、模型理解和调试的复杂性,在这基础上,找到高质量的学习资源,不浪费时间、不走弯路,又是重中之重。

AI大模型入门到实战的视频教程+项目包

看视频学习是一种高效、直观、灵活且富有吸引力的学习方式,可以更直观地展示过程,能有效提升学习兴趣和理解力,是现在获取知识的重要途径

在这里插入图片描述
光学理论是没用的,要学会跟着一起敲,要动手实操,才能将自己的所学运用到实际当中去,这时候可以搞点实战案例来学习。
在这里插入图片描述

海量AI大模型必读的经典书籍(PDF)

阅读AI大模型经典书籍可以帮助读者提高技术水平,开拓视野,掌握核心技术,提高解决问题的能力,同时也可以借鉴他人的经验。对于想要深入学习AI大模型开发的读者来说,阅读经典书籍是非常有必要的。
在这里插入图片描述

600+AI大模型报告(实时更新)

这套包含640份报告的合集,涵盖了AI大模型的理论研究、技术实现、行业应用等多个方面。无论您是科研人员、工程师,还是对AI大模型感兴趣的爱好者,这套报告合集都将为您提供宝贵的信息和启示。
在这里插入图片描述

AI大模型面试真题+答案解析

我们学习AI大模型必然是想找到高薪的工作,下面这些面试题都是总结当前最新、最热、最高频的面试题,并且每道题都有详细的答案,面试前刷完这套面试题资料,小小offer,不在话下
在这里插入图片描述

在这里插入图片描述

这份完整版的大模型 AI 学习资料已经上传CSDN,朋友们如果需要可以微信扫描下方CSDN官方认证二维码免费领取【保证100%免费

### 如何在本地部署支持 RAG向量数据库 #### 1. 部署前准备 为了成功部署支持 RAG向量数据库,需先完成必要的准备工作。这包括安装依赖环境、选择合适的硬件配置以及明确业务需求。通常情况下,RAG 系统会利用开源的向量数据库来实现高效的数据检索功能[^2]。 #### 2. 数据嵌入处理 数据嵌入是整个流程中的重要环节之一。通过预训练或定制化的嵌入模型(如 BERT 或 Sentence Transformers),可将非结构化数据转化为高维向量表示形式。这些向量随后会被存储至选定的向量数据库中以便后续操作[^3]。 #### 3. 向量数据库的选择安装 目前市面上存在多种优秀的开源向量数据库解决方案可供选择,例如 Milvus、Qdrant 和 Weaviate 等。每种方案都有其独特的优势,在实际应用过程中可以根据具体场景做出最佳决策: - **Milvus**: 提供强大的矢量相似度计算能力,并且兼容性强; - **Qdrant**: 易于集成并具备良好的性能表现; - **Weaviate**: 支持语义搜索同时还集成了知识图谱特性; 对于本地部署而言,推荐优先考虑米尔沃斯 (Milvus),因为它拥有活跃社区支持并且文档详尽易于上手。以下是基于 Docker Compose 文件快速启动 Milvus 实例的方法示例: ```yaml version: '3' services: milvus: image: milvusdb/milvus:v2.0.0 container_name: milvus_cpu_docker_compose environment: - MILVUS_CPU_ONLY=true ports: - "19530:19530" - "8080:8080" volumes: - ./data:/var/lib/milvus ``` 运行命令 `docker-compose up` 即可在本机环境中启动一个完整的 Milvus 服务实例。 #### 4. 构建索引优化查询效率 一旦完成了基础架构搭建工作之后,则需要进一步关注如何提高系统的整体效能。其中一个重要方面就是合理设计并向量集合创建相应的索引来加速近邻查找过程。常见的索引类型有 IVF_FLAT, HNSW 等,开发者应依据实际情况选取最适合当前任务需求的那种方式。 #### 5. 测试验证阶段 最后一步是对已建立起来的整体框架进行全面测试以确保各项指标均达到预期目标水平之上。可以通过编写自动化脚本来模拟真实世界里的各种可能情况来进行压力测试分析系统稳定性及响应速度等方面的表现状况。 ---
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值