SlideShare a Scribd company logo
3
Most read
4
Most read
5
Most read
1
Aniruddha Chakrabarti
Cloud Consulting & Architecture Lead
Accenture - Cloud First, Data & AI
Agenda
1 Vector Databases, About Pinecone
2 Pinecone Organization, Project and Client libraries
3 Index and Collection
4 Data – Insert, Update, Delete
5 Data - Fetch & Search
6 Other
What is a Vector Database
• Applications that involve large language models, generative AI, and semantic search rely on vector
embeddings, a type of data that represents semantic information.
• Embeddings are generated by AI models (such as Large Language Models) and have a large number
of attributes or features, making their representation challenging to manage. In the context of AI and
machine learning, these features represent different dimensions of the data that are essential for
understanding patterns, relationships, and underlying structures
• Traditional scalar-based databases can’t keep up with the complexity and scale of such data, making
it difficult to extract insights and perform real-time analysis.
• Vector databases like Pinecone offer optimized storage and querying capabilities for embeddings.
• First, use the embedding model to create vector embeddings for
the content we want to index.
• The vector embedding is inserted into the vector database, with some
reference to the original content the embedding was created from.
• When the application issues a query, we use the same embedding
model to create embeddings for the query, and use those embeddings
to query the database for similar vector embeddings. And as
mentioned before, those similar embeddings are associated with the
original content that was used to create them.
About Pinecone
• Pinecone is a managed, cloud-native, vector database/ DBaaS.
• Few other examples of vector databases include Qdrant, Milvus, Chroma, Weaviate etc.
Other databases like Postgres, Redis and Sqlite provides extensions to handle Vector data.
• Pinecone currently is one of the most popular Vector Database.
• Suitable for storing vector embeddings (also called text embeddings) that provide long
term memory to Large Language Models. large language models, generative AI, and
semantic search rely on vector embeddings
• Vector embedding or text embeddings are a type of data that represents semantic
information. This information allows AI applications to gain understanding and maintain a
long-term memory that they can draw upon when executing complex tasks
• It helps in storing, querying, filtering and searching vector data.
• Operations are low latency and scale to billions of vectors
• Pinecone is a NoSQL datastore and so it’s eventually consistent
• Provides a mongo like API
Pinecone object hierarchy
• Organization – An organization in Pinecone
is a higher-level construct that contains
multiple projects. So, it is a set of projects
that use the same billing.
• Project – Each organization contains one or
more projects that share the same
organization owners and billing settings.
• Index – An index is similar to a table in
relational databases. It contains records that
store vectors.
• Record – A record consists of an id, vector
data (array of float/ numbers) and metadata
in the form of key value pairs.
• Collection - Used to backup an Index (along
with associated data)
Organization
Project N
Project 2
Project 1
Index N
Index 1
Record 1 (Id, Vector, Metadata)
Record N (Id, Vector, Metadata)
Collection 1 Collection N
Billing Roles - Organization
Owners and Users
Pinecone Client in Python
• Install Pinecone client
pip install pinecone-client
• Import Pinecone library and initialize the client by passing Pinecone api key and name of the Pinecone
environment.
import pinecone
import os
pinecone_api_key = os.environ.get('PINECONE_API_KEY')
env = os.environ.get('PINECONE_ENVIRONMENT')
pinecone.init(api_key=pinecone_api_key,
environment=env)
pinecone.version()
VersionResponse(server='2.0.11', client='2.2.4')
Organization in Pinecone
• A Pinecone organization is a set of projects that use the same billing.
• When an account is created in Pinecone a project gets created. Additional projects could
be created from Settings > Projects.
Project in Pinecone
• Each Pinecone project contains a number of indexes and users. Only a user who belongs to the
project can access the indexes in that project. Each project also has at least one project owner.
• Create a project in the Pinecone web console by specifying a name, Cloud providers (GCP, AWS,
Azure), Deployment location, Pod limit (maximum number of pods that can be used in total)
• whoami method could be used to retrieve the project id
pinecone.whoami()
WhoAmIResponse(username='bd6e4bc', user_label='default', projectname='f638a37’)
• Project name could be also retrieved using Config property. Apart from Project, environment, log
level, api key etc could be retrieved
pinecone.Config.PROJECT_NAME
pinecone.Config.ENVIRONMENT
pinecone.Config.API_KEY
Index
• An index is the highest-level organizational
unit of vector data in Pinecone (like a table in
relational database)
• Pinecone indexes store records – each
record can contain vector data, which is
array of floating-point numbers / decimals.
• It accepts and stores vectors, serves queries
over the vectors it contains, and does other
vector operations over its contents.
• A Pinecone Project can contain many
Indexes. Each Index contains multiple
records.
Organization
Project N
Project 2
Project 1
Index N
Index 1
Record 1 (Vector)
Record N (Vector)
Record
• A Pinecone Project can
contain many Indexes. Each
Index contains multiple
records.
• A record contains an
• ID,
• Values which is a vector
or array of float/
numbers
• Additional Metadata
(optional).
• As Pinecone is a NoSQL
vector database, defining
the schema is not reqd.
Organization
Project N
Project 2
Project 1
Index N
Index 1
1 [ 1.0,2.0, …. 10.0] {“key1”: “val1”, “key2”: “val2”}
2 [ 1.0,2.0, …. 10.0] {“key1”: “val1”, “key2”: “val2”}
N [ 1.0,2.0, …. 10.0] {“key1”: “val1”, “key2”: “val2”}
ID Values Metadata
Index – creating
• Create an index by passing index name and size/ dimension of the vector to be stored in index.
pinecone.create_index(name="first-index", dimension=5)
• Additional parameters could be passed like distance metric type and no of shards.
pinecone.create_index(name="second-index", dimension=5, metric="cosine", shards=1)
Index – creating
• Distance metric could be of three type –
o Euclidean - This is used to calculate the distance between two data points in a plane. It is one of the most
commonly used distance metric.
o Cosine - This is often used to find similarities between different documents. This is the default value. The
advantage is that the scores are normalized to [-1,1] range.
o Dotproduct - This is used to multiply two vectors. You can use it to tell us how similar the two vectors are. The
more positive the answer is, the closer the two vectors are in terms of their directions.
• Distance metric could be of three type – No of pods and pod type also could be specified –
pinecone.create_index(name="third-index", dimension=10, metric="cosine", shards=3,
pods=5, pod_type="p2")
Index – listing index, getting details and deleting
• List Pinecone indexes
pinecone.list_indexes()
['first-index’]
• Get details of an index
pinecone.describe_index("second-index")
IndexDescription(name='second-index', metric='cosine', replicas=1, dimension=5.0,
shards=1, pods=1, pod_type='starter', status={'ready': True, 'state': 'Ready'},
metadata_config=None, source_collection=‘’)
• Delete an index
pinecone.delete_index("first-index")
Index – scaling and configuring
• An index could be scaled up or down -
pinecone.scale_index(name='second-index', replicas=3)
• An index could be updated or reconfigured to a different pod type and replica count.
pinecone.configure_index(name='second-index', pod_type=“s1", replicas = 5)
• There are three types of pods available –
o s1 – storage optimized pod. provide large storage capacity and lower overall costs with slightly higher query
latencies than p1 pods. They are ideal for very large indexes with moderate or relaxed latency requirements.
o p1 - performance-optimized pods provide very low query latencies, but hold fewer vectors per pod than s1
pods. They are ideal for applications with low latency requirements (<100ms).
o p2 - p2 pod type provides greater query throughput with lower latency. For vectors with fewer than 128
dimension and queries where topK is less than 50, p2 pods support up to 200 QPS per replica and return
queries in less than 10ms. This means that query throughput and latency are better than s1 and p1.
o Starter – used in free plan.
• Get statistics about the index -
index.describe_index_stats()
{'dimension': 5, 'index_fullness': 9e-05, 'namespaces': {'': {'vector_count': 9}},
'total_vector_count': 9}
Insert data
• To insert, update, delete or perform any operation, Get a reference to the index created before -
index = pinecone.Index("second-index")
• Insert a single record by using the upsert method. The record contain Id, vector embeddings and
optional metadata -
index.upsert([("hello world", [1.0, 2.234, 3.34, 5.6, 7.8])])
• Insert multiple records using the same operations passing all the records as array -
index.upsert(
[
("Bangalore", [1.0, 2.234, 3.34, 5.6, 7.8]),
("Kolkata", [2.0, 1.234, 3.34, 5.6, 7.8]),
("Chennai", [3.0, 5.234, 3.34, 5.6, 7.8]),
("Mumbai", [4.0, 6.234, 3.34, 5.6, 7.8])
])
• Insert records (multiple) with metadata
index.upsert(
[
("Delhi", [1.0, 2.234, 3.34, 5.6, 7.8], {"type": "city", "sub-type": "metro"}),
("Pune", [2.0, 1.234, 3.34, 5.6, 7.8], {"type": "city", "sub-type": "non-metro"})
])
Update data – partial and full update
• Pinecone supports Full update and partial update of records.
• Full Update allows updating both vector values and metadata, while partial update allowing either
vector values or metadata
• To insert, update, delete or perform any operation, Get a reference to the index created before -
index = pinecone.Index("second-index")
• To partially update the record, use the update method
Following code update the vector values of the record -
index.update(id="hello world", values=[2.0, 3.1, 6.4, 9.6, 11.8])
Following code update the metadata the record -
index.update(id="hello-world", metadata={"city1":"Blore", "city2":"Kolkata"})
• To fully update the record use the same upsert method used earlier to insert the data -
index.upsert([("hello-world", [10.0, 20.1, 31.4, 55.6, 75.8], {"city1":"Blore",
"city2":"Kolkata", "city3":"Delhi"})])
Backup data using Collection
• Collections are used to backup an Index. A collection is a static copy of your index that only
consumes storage
• Create a collection by specifying name of the collection and the name of the index -
pinecone.create_collection(name="bakcup_collection", source=“first-index")
pinecone.list_collections()
['bakcup-collection’]
pinecone.describe_collection("bakcup-collection")
Collection – listing, get details and deleting
• All existing collections could be easily listed -
pinecone.list_collections()
['bakcup-collection’]
• All existing collections could be easily listed -
pinecone.describe_collection("bakcup-collection")
• Delete collection -
pinecone.delete_collection("bakcup-collection")
Dense vector vs Sparse data
• Pinecone supports both Dense vector and Sparse vector.
• Till now, we have only played with Dense vectors.
Index 1
1 [ 1.0,2.0, …. 10.0] indices [1,2], values [10,0, 20.5] {“key1”: “val1”, “key2”: “val2”}
2 [ 1.0,2.0, …. 10.0] indices [1,2], values [10,0, 20.5] {“key1”: “val1”, “key2”: “val2”}
N [ 1.0,2.0, …. 10.0] indices [1,2], values [10,0, 20.5] {“key1”: “val1”, “key2”: “val2”}
ID Dense Vector Sparse Vector Metadata
both Dense vector and Sparse vector could be part of a record
Inserting sparse data
• Sparse vector values can be upserted alongside dense vector values -
index.upsert(vectors=[{'id': 'id1', 'values': [0.1, 0.2, 0.3, 0.4, 0.5],
'sparse_values':{ 'indices':[1,2], 'values': [10.0, 20.5] } }])
• Note that you cannot upsert a record with sparse vector values without dense vector values
index.upsert(vectors=[{'id': 'id1', 'sparse_values':{ 'indices':[1,2], 'values':
[10.0, 20.5] } }])
ValueError: Vector dictionary is missing required fields: ['values']
Fetch data, and update data
• Fetch data by passing ids of the record
index.fetch(ids=['Bangalore', 'Kolkata’])
{'namespace': '',
'vectors': {'Bangalore': {'id': 'Bangalore', 'metadata': {}, 'values': [1.0, 2.234,
3.34, 5.6, 7.8]},
'Kolkata': {'id': 'Kolkata', 'metadata': {}, 'values': [2.0, 1.234,
3.34, 5.6, 7.8]}}}
• Update a record – both values (vector) and metadata could be updated
index.update(id='Bangalore', set_metadata={"type":"city", "sub-type":"non-metro"},
values=[1.0, 2.0, 3.0, 4.0, 5.0])
index.fetch(ids=['Bangalore’])
{'namespace': '',
'vectors': {'Bangalore': {'id': 'Bangalore', 'metadata': {'sub-type': 'non-metro',
'type': 'city'}, 'values': [1.0, 2.0, 3.0, 4.0, 5.0]}}}
Query data
• Query data by vector match
index.query(vector=[1.0, 2.0, 3.0, 5, 7.0], top_k=3, include_values=True)
{'matches': [{'id': 'Bangalore', 'score': 0.999296784, 'values': [1.0, 2.0, 3.0,
5.0, 7.0]},
{'id': 'Delhi', 'score': 0.997676671, 'values': [1.0, 2.234, 3.34,
5.6, 7.8]},
{'id': 'Den Haag', 'score': 0.997676671, 'values': [1.0, 2.234, 3.34,
5.6, 7.8]}], 'namespace': ‘’}
• Apart from id and values (vector data), metadata of each matched record also could be retrieved
index.query(vector=[1.0, 2.0, 3.0, 5, 7.0], top_k=3, include_values=True,
include_metadata=True)
Namespace
• Pinecone allows you to partition the records in an index into namespaces. Queries and other
operations are then limited to one namespace, so different requests can search different subsets of
your index.
• A new namespace could be created by inserting a record to an index by specifying a new namespace
-
index.upsert(
vectors = [
("Howrah", [1.0, 2.234, 3.34, 5.6, 7.8], {"type": "city", "sub-type":
"metro"}),
("Siliguri", [2.0, 1.234, 3.34, 5.6, 7.8], {"type": "city", "sub-type":
"non-metro"})
], namespace='my-first-namespace')
• By default each index contains a single default index.

More Related Content

What's hot (20)

PPTX
Fraud and Risk in Big Data
Umma Khatuna Jannat
 
PPTX
Delta Lake with Azure Databricks
Dustin Vannoy
 
PDF
Data and AI reference architecture
Willy Marroquin (WillyDevNET)
 
PDF
[XConf Brasil 2020] Data mesh
ThoughtWorks Brasil
 
PDF
BigQuery for Beginners
Better&Stronger
 
PPTX
Cloud provenance
Minnu C Tomy
 
PPTX
Delay Tolerant Network (DTN)
Haya Saani
 
PPTX
Architecting a datalake
Laurent Leturgez
 
PDF
Technical Deep Dive: Using Apache Kafka to Optimize Real-Time Analytics in Fi...
confluent
 
PDF
Introduction on Data Science
Edureka!
 
PDF
Observability for modern applications
MoovingON
 
PDF
Resource reservation protocol
Atakan ATAK
 
PPTX
bigquery.pptx
Harissh16
 
PDF
Meetup OpenTelemetry Intro
DimitrisFinas1
 
PPTX
Orion Context Broker NGSI-v2 Overview for Developers That Already Know NGSI-v...
Fermin Galan
 
PDF
Knowledge Graphs and Graph Data Science: More Context, Better Predictions (Ne...
Neo4j
 
PDF
Big Data Trends
Collabor8now Ltd
 
PPT
Unit-3_BDA.ppt
PoojaShah174393
 
PDF
Hyperspace for Delta Lake
Databricks
 
Fraud and Risk in Big Data
Umma Khatuna Jannat
 
Delta Lake with Azure Databricks
Dustin Vannoy
 
Data and AI reference architecture
Willy Marroquin (WillyDevNET)
 
[XConf Brasil 2020] Data mesh
ThoughtWorks Brasil
 
BigQuery for Beginners
Better&Stronger
 
Cloud provenance
Minnu C Tomy
 
Delay Tolerant Network (DTN)
Haya Saani
 
Architecting a datalake
Laurent Leturgez
 
Technical Deep Dive: Using Apache Kafka to Optimize Real-Time Analytics in Fi...
confluent
 
Introduction on Data Science
Edureka!
 
Observability for modern applications
MoovingON
 
Resource reservation protocol
Atakan ATAK
 
bigquery.pptx
Harissh16
 
Meetup OpenTelemetry Intro
DimitrisFinas1
 
Orion Context Broker NGSI-v2 Overview for Developers That Already Know NGSI-v...
Fermin Galan
 
Knowledge Graphs and Graph Data Science: More Context, Better Predictions (Ne...
Neo4j
 
Big Data Trends
Collabor8now Ltd
 
Unit-3_BDA.ppt
PoojaShah174393
 
Hyperspace for Delta Lake
Databricks
 

Similar to Pinecone Vector Database.pdf (20)

PDF
High Performance JSON Search and Relational Faceted Browsing with Lucene
lucenerevolution
 
PPTX
Unit 2- Data Warehouse Logical Design.pptx
Rakesh Bachchan
 
PDF
Red Hat Summit Connect 2023 - Redis Enterprise, the engine of Generative AI
Luigi Fugaro
 
PDF
Elasticsearch Introduction at BigData meetup
Eric Rodriguez (Hiring in Lex)
 
PPTX
PgVector + : Enable Richer Interaction with vector database.pptx
aranjan11
 
PDF
data science with python_UNIT 2_full notes.pdf
mukeshgarg02
 
PPTX
Session 2
HarithaAshok3
 
PDF
What's new in pandas and the SciPy stack for financial users
Wes McKinney
 
PDF
Virtual training intro to InfluxDB - June 2021
InfluxData
 
PPTX
05 k-means clustering
Subhas Kumar Ghosh
 
PDF
2021 04-20 apache arrow and its impact on the database industry.pptx
Andrew Lamb
 
PDF
A Maturing Role of Workflows in the Presence of Heterogenous Computing Archit...
Ilkay Altintas, Ph.D.
 
PPTX
Day 1 - Technical Bootcamp azure synapse analytics
Armand272
 
PPTX
Examiness hints and tips from the trenches
Ismail Mayat
 
DOCX
Resume (1)
Rajat Goswami
 
PPTX
this includes basics about python modules and packages introduction
ssuseree48e0
 
PPTX
Python short notes on modules and applications
ssuseree48e0
 
PDF
Indexator_oct2022.pdf
Daniel JACOB
 
KEY
Introduction to the Semantic Web
Nuxeo
 
DOC
Abinitio Experienced resume-Anilkumar
anilkumar kagitha
 
High Performance JSON Search and Relational Faceted Browsing with Lucene
lucenerevolution
 
Unit 2- Data Warehouse Logical Design.pptx
Rakesh Bachchan
 
Red Hat Summit Connect 2023 - Redis Enterprise, the engine of Generative AI
Luigi Fugaro
 
Elasticsearch Introduction at BigData meetup
Eric Rodriguez (Hiring in Lex)
 
PgVector + : Enable Richer Interaction with vector database.pptx
aranjan11
 
data science with python_UNIT 2_full notes.pdf
mukeshgarg02
 
Session 2
HarithaAshok3
 
What's new in pandas and the SciPy stack for financial users
Wes McKinney
 
Virtual training intro to InfluxDB - June 2021
InfluxData
 
05 k-means clustering
Subhas Kumar Ghosh
 
2021 04-20 apache arrow and its impact on the database industry.pptx
Andrew Lamb
 
A Maturing Role of Workflows in the Presence of Heterogenous Computing Archit...
Ilkay Altintas, Ph.D.
 
Day 1 - Technical Bootcamp azure synapse analytics
Armand272
 
Examiness hints and tips from the trenches
Ismail Mayat
 
Resume (1)
Rajat Goswami
 
this includes basics about python modules and packages introduction
ssuseree48e0
 
Python short notes on modules and applications
ssuseree48e0
 
Indexator_oct2022.pdf
Daniel JACOB
 
Introduction to the Semantic Web
Nuxeo
 
Abinitio Experienced resume-Anilkumar
anilkumar kagitha
 
Ad

More from Aniruddha Chakrabarti (20)

PDF
Mphasis-Annual-Report-2018.pdf
Aniruddha Chakrabarti
 
PDF
Thomas Cook and Accenture expand relationship with 10 year technology consult...
Aniruddha Chakrabarti
 
PDF
NLP using JavaScript Natural Library
Aniruddha Chakrabarti
 
PPTX
Dart programming language
Aniruddha Chakrabarti
 
PPTX
Third era of computing
Aniruddha Chakrabarti
 
PPTX
Golang - Overview of Go (golang) Language
Aniruddha Chakrabarti
 
PPTX
Amazon alexa - building custom skills
Aniruddha Chakrabarti
 
PDF
Using Node-RED for building IoT workflows
Aniruddha Chakrabarti
 
PDF
Mphasis Digital - Use Go (gloang) for system programming, distributed systems...
Aniruddha Chakrabarti
 
PDF
Using Swift for all Apple platforms (iOS, watchOS, tvOS and OS X)
Aniruddha Chakrabarti
 
PDF
Future of .NET - .NET on Non Windows Platforms
Aniruddha Chakrabarti
 
PPTX
CoAP - Web Protocol for IoT
Aniruddha Chakrabarti
 
PPTX
Groovy Programming Language
Aniruddha Chakrabarti
 
PDF
Mphasis Digital POV - Emerging Open Standard Protocol stack for IoT
Aniruddha Chakrabarti
 
PPTX
Level DB - Quick Cheat Sheet
Aniruddha Chakrabarti
 
PPTX
Overview of CoffeeScript
Aniruddha Chakrabarti
 
PPTX
memcached Distributed Cache
Aniruddha Chakrabarti
 
PPTX
Redis and it's data types
Aniruddha Chakrabarti
 
PPTX
pebble - Building apps on pebble
Aniruddha Chakrabarti
 
Mphasis-Annual-Report-2018.pdf
Aniruddha Chakrabarti
 
Thomas Cook and Accenture expand relationship with 10 year technology consult...
Aniruddha Chakrabarti
 
NLP using JavaScript Natural Library
Aniruddha Chakrabarti
 
Dart programming language
Aniruddha Chakrabarti
 
Third era of computing
Aniruddha Chakrabarti
 
Golang - Overview of Go (golang) Language
Aniruddha Chakrabarti
 
Amazon alexa - building custom skills
Aniruddha Chakrabarti
 
Using Node-RED for building IoT workflows
Aniruddha Chakrabarti
 
Mphasis Digital - Use Go (gloang) for system programming, distributed systems...
Aniruddha Chakrabarti
 
Using Swift for all Apple platforms (iOS, watchOS, tvOS and OS X)
Aniruddha Chakrabarti
 
Future of .NET - .NET on Non Windows Platforms
Aniruddha Chakrabarti
 
CoAP - Web Protocol for IoT
Aniruddha Chakrabarti
 
Groovy Programming Language
Aniruddha Chakrabarti
 
Mphasis Digital POV - Emerging Open Standard Protocol stack for IoT
Aniruddha Chakrabarti
 
Level DB - Quick Cheat Sheet
Aniruddha Chakrabarti
 
Overview of CoffeeScript
Aniruddha Chakrabarti
 
memcached Distributed Cache
Aniruddha Chakrabarti
 
Redis and it's data types
Aniruddha Chakrabarti
 
pebble - Building apps on pebble
Aniruddha Chakrabarti
 
Ad

Recently uploaded (20)

PDF
Thread In Android-Mastering Concurrency for Responsive Apps.pdf
Nabin Dhakal
 
PDF
Mobile CMMS Solutions Empowering the Frontline Workforce
CryotosCMMSSoftware
 
PDF
Unlock Efficiency with Insurance Policy Administration Systems
Insurance Tech Services
 
PDF
Efficient, Automated Claims Processing Software for Insurers
Insurance Tech Services
 
PPTX
Engineering the Java Web Application (MVC)
abhishekoza1981
 
PDF
HiHelloHR – Simplify HR Operations for Modern Workplaces
HiHelloHR
 
PPT
MergeSortfbsjbjsfk sdfik k
RafishaikIT02044
 
PPTX
Writing Better Code - Helping Developers make Decisions.pptx
Lorraine Steyn
 
PDF
vMix Pro 28.0.0.42 Download vMix Registration key Bundle
kulindacore
 
PPTX
MiniTool Power Data Recovery Full Crack Latest 2025
muhammadgurbazkhan
 
PPTX
Feb 2021 Cohesity first pitch presentation.pptx
enginsayin1
 
PDF
Streamline Contractor Lifecycle- TECH EHS Solution
TECH EHS Solution
 
PPTX
MailsDaddy Outlook OST to PST converter.pptx
abhishekdutt366
 
PDF
iTop VPN With Crack Lifetime Activation Key-CODE
utfefguu
 
PDF
Why Businesses Are Switching to Open Source Alternatives to Crystal Reports.pdf
Varsha Nayak
 
PDF
Executive Business Intelligence Dashboards
vandeslie24
 
PDF
Understanding the Need for Systemic Change in Open Source Through Intersectio...
Imma Valls Bernaus
 
PPTX
How Apagen Empowered an EPC Company with Engineering ERP Software
SatishKumar2651
 
PDF
Odoo CRM vs Zoho CRM: Honest Comparison 2025
Odiware Technologies Private Limited
 
PPTX
Equipment Management Software BIS Safety UK.pptx
BIS Safety Software
 
Thread In Android-Mastering Concurrency for Responsive Apps.pdf
Nabin Dhakal
 
Mobile CMMS Solutions Empowering the Frontline Workforce
CryotosCMMSSoftware
 
Unlock Efficiency with Insurance Policy Administration Systems
Insurance Tech Services
 
Efficient, Automated Claims Processing Software for Insurers
Insurance Tech Services
 
Engineering the Java Web Application (MVC)
abhishekoza1981
 
HiHelloHR – Simplify HR Operations for Modern Workplaces
HiHelloHR
 
MergeSortfbsjbjsfk sdfik k
RafishaikIT02044
 
Writing Better Code - Helping Developers make Decisions.pptx
Lorraine Steyn
 
vMix Pro 28.0.0.42 Download vMix Registration key Bundle
kulindacore
 
MiniTool Power Data Recovery Full Crack Latest 2025
muhammadgurbazkhan
 
Feb 2021 Cohesity first pitch presentation.pptx
enginsayin1
 
Streamline Contractor Lifecycle- TECH EHS Solution
TECH EHS Solution
 
MailsDaddy Outlook OST to PST converter.pptx
abhishekdutt366
 
iTop VPN With Crack Lifetime Activation Key-CODE
utfefguu
 
Why Businesses Are Switching to Open Source Alternatives to Crystal Reports.pdf
Varsha Nayak
 
Executive Business Intelligence Dashboards
vandeslie24
 
Understanding the Need for Systemic Change in Open Source Through Intersectio...
Imma Valls Bernaus
 
How Apagen Empowered an EPC Company with Engineering ERP Software
SatishKumar2651
 
Odoo CRM vs Zoho CRM: Honest Comparison 2025
Odiware Technologies Private Limited
 
Equipment Management Software BIS Safety UK.pptx
BIS Safety Software
 

Pinecone Vector Database.pdf

  • 1. 1 Aniruddha Chakrabarti Cloud Consulting & Architecture Lead Accenture - Cloud First, Data & AI
  • 2. Agenda 1 Vector Databases, About Pinecone 2 Pinecone Organization, Project and Client libraries 3 Index and Collection 4 Data – Insert, Update, Delete 5 Data - Fetch & Search 6 Other
  • 3. What is a Vector Database • Applications that involve large language models, generative AI, and semantic search rely on vector embeddings, a type of data that represents semantic information. • Embeddings are generated by AI models (such as Large Language Models) and have a large number of attributes or features, making their representation challenging to manage. In the context of AI and machine learning, these features represent different dimensions of the data that are essential for understanding patterns, relationships, and underlying structures • Traditional scalar-based databases can’t keep up with the complexity and scale of such data, making it difficult to extract insights and perform real-time analysis. • Vector databases like Pinecone offer optimized storage and querying capabilities for embeddings. • First, use the embedding model to create vector embeddings for the content we want to index. • The vector embedding is inserted into the vector database, with some reference to the original content the embedding was created from. • When the application issues a query, we use the same embedding model to create embeddings for the query, and use those embeddings to query the database for similar vector embeddings. And as mentioned before, those similar embeddings are associated with the original content that was used to create them.
  • 4. About Pinecone • Pinecone is a managed, cloud-native, vector database/ DBaaS. • Few other examples of vector databases include Qdrant, Milvus, Chroma, Weaviate etc. Other databases like Postgres, Redis and Sqlite provides extensions to handle Vector data. • Pinecone currently is one of the most popular Vector Database. • Suitable for storing vector embeddings (also called text embeddings) that provide long term memory to Large Language Models. large language models, generative AI, and semantic search rely on vector embeddings • Vector embedding or text embeddings are a type of data that represents semantic information. This information allows AI applications to gain understanding and maintain a long-term memory that they can draw upon when executing complex tasks • It helps in storing, querying, filtering and searching vector data. • Operations are low latency and scale to billions of vectors • Pinecone is a NoSQL datastore and so it’s eventually consistent • Provides a mongo like API
  • 5. Pinecone object hierarchy • Organization – An organization in Pinecone is a higher-level construct that contains multiple projects. So, it is a set of projects that use the same billing. • Project – Each organization contains one or more projects that share the same organization owners and billing settings. • Index – An index is similar to a table in relational databases. It contains records that store vectors. • Record – A record consists of an id, vector data (array of float/ numbers) and metadata in the form of key value pairs. • Collection - Used to backup an Index (along with associated data) Organization Project N Project 2 Project 1 Index N Index 1 Record 1 (Id, Vector, Metadata) Record N (Id, Vector, Metadata) Collection 1 Collection N Billing Roles - Organization Owners and Users
  • 6. Pinecone Client in Python • Install Pinecone client pip install pinecone-client • Import Pinecone library and initialize the client by passing Pinecone api key and name of the Pinecone environment. import pinecone import os pinecone_api_key = os.environ.get('PINECONE_API_KEY') env = os.environ.get('PINECONE_ENVIRONMENT') pinecone.init(api_key=pinecone_api_key, environment=env) pinecone.version() VersionResponse(server='2.0.11', client='2.2.4')
  • 7. Organization in Pinecone • A Pinecone organization is a set of projects that use the same billing. • When an account is created in Pinecone a project gets created. Additional projects could be created from Settings > Projects.
  • 8. Project in Pinecone • Each Pinecone project contains a number of indexes and users. Only a user who belongs to the project can access the indexes in that project. Each project also has at least one project owner. • Create a project in the Pinecone web console by specifying a name, Cloud providers (GCP, AWS, Azure), Deployment location, Pod limit (maximum number of pods that can be used in total) • whoami method could be used to retrieve the project id pinecone.whoami() WhoAmIResponse(username='bd6e4bc', user_label='default', projectname='f638a37’) • Project name could be also retrieved using Config property. Apart from Project, environment, log level, api key etc could be retrieved pinecone.Config.PROJECT_NAME pinecone.Config.ENVIRONMENT pinecone.Config.API_KEY
  • 9. Index • An index is the highest-level organizational unit of vector data in Pinecone (like a table in relational database) • Pinecone indexes store records – each record can contain vector data, which is array of floating-point numbers / decimals. • It accepts and stores vectors, serves queries over the vectors it contains, and does other vector operations over its contents. • A Pinecone Project can contain many Indexes. Each Index contains multiple records. Organization Project N Project 2 Project 1 Index N Index 1 Record 1 (Vector) Record N (Vector)
  • 10. Record • A Pinecone Project can contain many Indexes. Each Index contains multiple records. • A record contains an • ID, • Values which is a vector or array of float/ numbers • Additional Metadata (optional). • As Pinecone is a NoSQL vector database, defining the schema is not reqd. Organization Project N Project 2 Project 1 Index N Index 1 1 [ 1.0,2.0, …. 10.0] {“key1”: “val1”, “key2”: “val2”} 2 [ 1.0,2.0, …. 10.0] {“key1”: “val1”, “key2”: “val2”} N [ 1.0,2.0, …. 10.0] {“key1”: “val1”, “key2”: “val2”} ID Values Metadata
  • 11. Index – creating • Create an index by passing index name and size/ dimension of the vector to be stored in index. pinecone.create_index(name="first-index", dimension=5) • Additional parameters could be passed like distance metric type and no of shards. pinecone.create_index(name="second-index", dimension=5, metric="cosine", shards=1)
  • 12. Index – creating • Distance metric could be of three type – o Euclidean - This is used to calculate the distance between two data points in a plane. It is one of the most commonly used distance metric. o Cosine - This is often used to find similarities between different documents. This is the default value. The advantage is that the scores are normalized to [-1,1] range. o Dotproduct - This is used to multiply two vectors. You can use it to tell us how similar the two vectors are. The more positive the answer is, the closer the two vectors are in terms of their directions. • Distance metric could be of three type – No of pods and pod type also could be specified – pinecone.create_index(name="third-index", dimension=10, metric="cosine", shards=3, pods=5, pod_type="p2")
  • 13. Index – listing index, getting details and deleting • List Pinecone indexes pinecone.list_indexes() ['first-index’] • Get details of an index pinecone.describe_index("second-index") IndexDescription(name='second-index', metric='cosine', replicas=1, dimension=5.0, shards=1, pods=1, pod_type='starter', status={'ready': True, 'state': 'Ready'}, metadata_config=None, source_collection=‘’) • Delete an index pinecone.delete_index("first-index")
  • 14. Index – scaling and configuring • An index could be scaled up or down - pinecone.scale_index(name='second-index', replicas=3) • An index could be updated or reconfigured to a different pod type and replica count. pinecone.configure_index(name='second-index', pod_type=“s1", replicas = 5) • There are three types of pods available – o s1 – storage optimized pod. provide large storage capacity and lower overall costs with slightly higher query latencies than p1 pods. They are ideal for very large indexes with moderate or relaxed latency requirements. o p1 - performance-optimized pods provide very low query latencies, but hold fewer vectors per pod than s1 pods. They are ideal for applications with low latency requirements (<100ms). o p2 - p2 pod type provides greater query throughput with lower latency. For vectors with fewer than 128 dimension and queries where topK is less than 50, p2 pods support up to 200 QPS per replica and return queries in less than 10ms. This means that query throughput and latency are better than s1 and p1. o Starter – used in free plan. • Get statistics about the index - index.describe_index_stats() {'dimension': 5, 'index_fullness': 9e-05, 'namespaces': {'': {'vector_count': 9}}, 'total_vector_count': 9}
  • 15. Insert data • To insert, update, delete or perform any operation, Get a reference to the index created before - index = pinecone.Index("second-index") • Insert a single record by using the upsert method. The record contain Id, vector embeddings and optional metadata - index.upsert([("hello world", [1.0, 2.234, 3.34, 5.6, 7.8])]) • Insert multiple records using the same operations passing all the records as array - index.upsert( [ ("Bangalore", [1.0, 2.234, 3.34, 5.6, 7.8]), ("Kolkata", [2.0, 1.234, 3.34, 5.6, 7.8]), ("Chennai", [3.0, 5.234, 3.34, 5.6, 7.8]), ("Mumbai", [4.0, 6.234, 3.34, 5.6, 7.8]) ]) • Insert records (multiple) with metadata index.upsert( [ ("Delhi", [1.0, 2.234, 3.34, 5.6, 7.8], {"type": "city", "sub-type": "metro"}), ("Pune", [2.0, 1.234, 3.34, 5.6, 7.8], {"type": "city", "sub-type": "non-metro"}) ])
  • 16. Update data – partial and full update • Pinecone supports Full update and partial update of records. • Full Update allows updating both vector values and metadata, while partial update allowing either vector values or metadata • To insert, update, delete or perform any operation, Get a reference to the index created before - index = pinecone.Index("second-index") • To partially update the record, use the update method Following code update the vector values of the record - index.update(id="hello world", values=[2.0, 3.1, 6.4, 9.6, 11.8]) Following code update the metadata the record - index.update(id="hello-world", metadata={"city1":"Blore", "city2":"Kolkata"}) • To fully update the record use the same upsert method used earlier to insert the data - index.upsert([("hello-world", [10.0, 20.1, 31.4, 55.6, 75.8], {"city1":"Blore", "city2":"Kolkata", "city3":"Delhi"})])
  • 17. Backup data using Collection • Collections are used to backup an Index. A collection is a static copy of your index that only consumes storage • Create a collection by specifying name of the collection and the name of the index - pinecone.create_collection(name="bakcup_collection", source=“first-index") pinecone.list_collections() ['bakcup-collection’] pinecone.describe_collection("bakcup-collection")
  • 18. Collection – listing, get details and deleting • All existing collections could be easily listed - pinecone.list_collections() ['bakcup-collection’] • All existing collections could be easily listed - pinecone.describe_collection("bakcup-collection") • Delete collection - pinecone.delete_collection("bakcup-collection")
  • 19. Dense vector vs Sparse data • Pinecone supports both Dense vector and Sparse vector. • Till now, we have only played with Dense vectors. Index 1 1 [ 1.0,2.0, …. 10.0] indices [1,2], values [10,0, 20.5] {“key1”: “val1”, “key2”: “val2”} 2 [ 1.0,2.0, …. 10.0] indices [1,2], values [10,0, 20.5] {“key1”: “val1”, “key2”: “val2”} N [ 1.0,2.0, …. 10.0] indices [1,2], values [10,0, 20.5] {“key1”: “val1”, “key2”: “val2”} ID Dense Vector Sparse Vector Metadata both Dense vector and Sparse vector could be part of a record
  • 20. Inserting sparse data • Sparse vector values can be upserted alongside dense vector values - index.upsert(vectors=[{'id': 'id1', 'values': [0.1, 0.2, 0.3, 0.4, 0.5], 'sparse_values':{ 'indices':[1,2], 'values': [10.0, 20.5] } }]) • Note that you cannot upsert a record with sparse vector values without dense vector values index.upsert(vectors=[{'id': 'id1', 'sparse_values':{ 'indices':[1,2], 'values': [10.0, 20.5] } }]) ValueError: Vector dictionary is missing required fields: ['values']
  • 21. Fetch data, and update data • Fetch data by passing ids of the record index.fetch(ids=['Bangalore', 'Kolkata’]) {'namespace': '', 'vectors': {'Bangalore': {'id': 'Bangalore', 'metadata': {}, 'values': [1.0, 2.234, 3.34, 5.6, 7.8]}, 'Kolkata': {'id': 'Kolkata', 'metadata': {}, 'values': [2.0, 1.234, 3.34, 5.6, 7.8]}}} • Update a record – both values (vector) and metadata could be updated index.update(id='Bangalore', set_metadata={"type":"city", "sub-type":"non-metro"}, values=[1.0, 2.0, 3.0, 4.0, 5.0]) index.fetch(ids=['Bangalore’]) {'namespace': '', 'vectors': {'Bangalore': {'id': 'Bangalore', 'metadata': {'sub-type': 'non-metro', 'type': 'city'}, 'values': [1.0, 2.0, 3.0, 4.0, 5.0]}}}
  • 22. Query data • Query data by vector match index.query(vector=[1.0, 2.0, 3.0, 5, 7.0], top_k=3, include_values=True) {'matches': [{'id': 'Bangalore', 'score': 0.999296784, 'values': [1.0, 2.0, 3.0, 5.0, 7.0]}, {'id': 'Delhi', 'score': 0.997676671, 'values': [1.0, 2.234, 3.34, 5.6, 7.8]}, {'id': 'Den Haag', 'score': 0.997676671, 'values': [1.0, 2.234, 3.34, 5.6, 7.8]}], 'namespace': ‘’} • Apart from id and values (vector data), metadata of each matched record also could be retrieved index.query(vector=[1.0, 2.0, 3.0, 5, 7.0], top_k=3, include_values=True, include_metadata=True)
  • 23. Namespace • Pinecone allows you to partition the records in an index into namespaces. Queries and other operations are then limited to one namespace, so different requests can search different subsets of your index. • A new namespace could be created by inserting a record to an index by specifying a new namespace - index.upsert( vectors = [ ("Howrah", [1.0, 2.234, 3.34, 5.6, 7.8], {"type": "city", "sub-type": "metro"}), ("Siliguri", [2.0, 1.234, 3.34, 5.6, 7.8], {"type": "city", "sub-type": "non-metro"}) ], namespace='my-first-namespace') • By default each index contains a single default index.