语义相似度：从STSB到Paraphrase任务

Sentence Transformers专注于句子和文本嵌入，支持超过100种语言。利用深度学习技术，特别是Transformer架构的优势，将文本转换为高维向量空间中的点，使得相似的文本在几何意义上更接近。 语义搜索：构建高效的语义搜索系统，找到最相关的查询结果。信息检索与重排：在大规模文档集合中查找相关文档并重新排序。聚类分析：将文本自动分组，发现隐藏的主题或模式。摘要挖掘：识别和提取文本的主要观点。平行句对挖掘：在多语言数据中找出对应的翻译句子。💥pip安装：pip install -U sentence-transformers💥conda安装：conda install -c conda-forge sentence-transformers快速使用：from sentence_transformers import SentenceTransformer model = SentenceTransformer("all-MiniLM-L6-v2")# 加载all-MiniLM-L6-v2，这是一个在超过 10 亿个训练对的大型数据集上微调的 MiniLM 模型 sentences = [ "The weather is lovely today.", "It's so sunny outside!", "He drove to the stadium.",] embeddings = model.encode(sentences)print(embeddings.shape) # 计算所有句子对之间的相似度similarities = model.similarity(embeddings, embeddings)print(similarities)输出： Cross Encoder计算给定文本对的相似度得分。通常比Sentence Transformer模型慢，因为它需要对每一对而不是每个文本进行计算交叉编码器（Cross Encoder）经常被用来对Sentence Transformer模型的top-k个结果进行重新排序。💯Cross Encoder (又名 reranker) 模型的用法与 Sentence Transformers 类似：from sentence_transformers.cross_encoder import CrossEncoder# 我们选择要加载的CrossEncoder模型model = CrossEncoder("cross-encoder/stsb-distilroberta-base") # 定义查询句子和语料库query = "A man is eating pasta."corpus = [ "A man is eating food.", "A man is eating a piece of bread.", "The girl is carrying a baby.", "A man is riding a horse.", "A woman is playing violin.", "Two men pushed carts through the woods.", "A man is riding a white horse on an enclosed ground.", "A monkey is playing drums.", "A cheetah is running behind its prey.",] # 对句子进行排名ranks = model.rank(query, corpus) print("Query: ", query)for rank in ranks: print(f"{rank['score']:.2f}\t{corpus[rank['corpus_id']]}") import numpy as np# 使用 NumPy 进行排序sentence_combinations = [[query, sentence] for sentence in corpus]scores = model.predict(sentence_combinations) ranked_indices = np.argsort(scores)[::-1]print("Scores:", scores)print("Indices:", ranked_indices)输出：Sentence Transformer模型用法 计算给定文本或图像的固定大小的向量表示（嵌入）。嵌入计算往往是高效的，嵌入相似度计算非常快。适用于广泛的任务，例如语义文本相似度、语义搜索、聚类、分类、释义挖掘等。通常用作两步检索过程的第一步，其中使用Cross-Encoder模型对双编码器的前 k 个结果进行重新排序。语义文本相似度💫对于语义文本相似度 (STS)，我们希望为所有相关文本生成嵌入并计算它们之间的相似度。相似度得分最高的文本对在语义上最相似from sentence_transformers import SentenceTransformer model = SentenceTransformer("all-MiniLM-L6-v2") sentences1 = [ "The new movie is awesome", "The cat sits outside", "A man is playing guitar",] sentences2 = [ "The dog plays in the garden", "The new movie is so great", "A woman watches TV",] embeddings1 = model.encode(sentences1)embeddings2 = model.encode(sentences2) similarities = model.similarity(embeddings1, embeddings2) for idx_i, sentence1 in enumerate(sentences1): print(sentence1) for idx_j, sentence2 in enumerate(sentences2): print(f" - {sentence2: <30}: {similarities[idx_i][idx_j]:.4f}")返回一个 3x3 矩阵，其中包含embeddings1和embeddings2之间所有可能对的相应余弦相似度分数 💫输出：相似度计算 可以通过多种方式改变此值：1. 通过使用所需的相似度函数初始化 SentenceTransformer 实例：from sentence_transformers import SentenceTransformer, SimilarityFunction model = SentenceTransformer("all-MiniLM-L6-v2", similarity_fn_name=SimilarityFunction.DOT_PRODUCT)2. 通过直接在 SentenceTransformer 实例上设置值：from sentence_transformers import SentenceTransformer, SimilarityFunction model = SentenceTransformer("all-MiniLM-L6-v2")model.similarity_fn_name = SimilarityFunction.DOT_PRODUCTSentence Transformers 实现了两种方法来计算嵌入之间的相似度from sentence_transformers import SentenceTransformer, SimilarityFunctionmodel = SentenceTransformer("all-MiniLM-L6-v2") sentences = [ "The weather is lovely today.", "It's so sunny outside!", "He drove to the stadium.",]embeddings = model.encode(sentences) similarities = model.similarity(embeddings, embeddings)print(similarities) model.similarity_fn_name = SimilarityFunction.MANHATTANprint(model.similarity_fn_name) similarities = model.similarity(embeddings, embeddings)print(similarities) 输出：语义搜索 🧊语义搜索旨在通过理解搜索查询的语义含义和要搜索的语料库来提高搜索准确率。与只能根据词汇匹配查找文档的关键字搜索引擎不同，语义搜索在给定同义词、缩写和拼写错误的情况下也能表现良好。语义搜索背后的理念是将语料库中的所有条目（无论是句子、段落还是文档）嵌入到向量空间中。在搜索时，查询被嵌入到相同的向量空间中，并从语料库中找到最接近的嵌入。这些条目应该与查询具有较高的语义相似度。对称与非对称语义搜索🧊我们设置的一个关键区别是对称与非对称语义搜索：对于对称语义搜索：搜索类似问题：您的查询可能是“如何在线学习 Python？”想查找“如何在网上学习 Python？”这样的条目对于非对称语义搜索：我们通常会有一个简短的查询（例如问题或一些关键字）并且想要找到一个较长的段落来回答该查询。例如，查询“什么是 Python”，并且您想要找到段落“Python 是一种解释型、高级和通用的编程语言。Python 的设计理念……”。对于非对称任务，翻转查询和语料库中的条目通常没有意义。