Comparing three data ingestion approaches where Apache Kafka integrates with a distributed graph database in real time | Benyue (Emma) Liu, TigerGraph

Comparing three data ingestion
approaches where Apache Kafka
integrates with a distributed
graph database in real time
Benyue (Emma) Liu
Product Manager, TigerGraph
April, 2021

© 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Today's Speakers
Benyue (Emma) Liu
Senior Product Manager, TigerGraph
● BS in Engineering from Harvey Mudd College, MS in
Engineering Systems from MIT
● Prior work experience at Oracle and MarkLogic
● Focus - Cloud, Containers, Enterprise Infra, Monitoring,
Management, Connectors, Developer Tools,
Applications
2

2
TigerGraph Data Ingestion System
Architecture: Internal Kakfa Component
TigerGraph Cloud and Kafka Data
Pipeline Use Cases
Built-in Kafka Loader in TigerGraph
Today’s Outline
4
3
3
1 Graph Analytics Overview

Graph Analytics
Overview
4

5
By 2025, graph technologies will be used
in 80% of data and analytics innovations,
up from 10% in 2021, facilitating rapid
decision making across the enterprise.1
“To Graph or Not to Graph? That Is Not
the Question — You Will Graph.”2
Mark Beyer, Distinguished VP Analyst
1Gartner, Top Trends in Data and Analytics for 2021, 16 February 2021
2Gartner, Graph Steps Onto the Main Stage of Data and Analytics: A Gartner Trend Insight Report, 14, December 2020

© 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM |
Why Graph Analytics?
Source: Gartner - Top 10 data and analytics Trends for 2019
Graph deployments are going deeper, wider and operational:
Need to make it accessible to non-technical users
6
● Definition - Graph analytics is a set of analytic techniques that
allows for the exploration of relationships between entities of
interest such as organizations, people and transactions.
● Forecasted growth - 100% annually through 2022
● What’s driving the growth
○ Need to ask complex questions across complex data, which is
not always practical or even possible at scale using SQL
queries. (RDBMS requires time-consuming & expensive table
joins!)
● What’s needed for broad adoption of graph data stores
○ Graph data stores can efficiently model, explore and query data
with complex interrelationships across data silos, but the need for
specialized skills has limited their adoption to date.
Customer
Supplier
Location 2
Product
Payment
PURCHASED
Location 1
Order

© 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM
Who is TigerGraph?
Corporate Overview Video
We provide advanced analytics on connected data
○ The only scalable graph database for the enterprise
○ HTAP graph database, foundational for AI and ML solutions
○ SQL-like query language (GSQL) accelerates time to solution
○ Cloud Neutral: Google GCP, Microsoft Azure,
Our customers include:
○ The largest companies in financial, healthcare, telecoms, media,
utilities and innovative startups in cybersecurity, ecommerce and
finserv
Founded in 2012, HQ in Redwood City, California
7

© 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM
How Our Customers Use TigerGraph?
8
Find most influential
users/customers
Find similar users/customers Who are the patients that are going through a particular type of
journey that results in an adverse health outcome?
Is the
Uncover hidden connections Is the new credit card applicant or transaction connected to
known fraudsters?
Recommend next best action Can I run a real-time credit score algorithm and recommend an
offer based on the customer’s credit profile & need?
Which users are driving higher usage or adoption of my product or
service?
Detect connected users
(communities)
What is average spend over time across a community of connected
users (fin. services, airlines, healthcare, retail..)?

TigerGraph
System Architecture
Overview
(Kafka as a Key Component)
9

The TigerGraph Difference
Feature Design Difference Benefit
Real-Time Deep-Link Querying ● Native Graph design
● C++ engine, for high performance
● Storage Architecture
● Uncovers hard-to-find patterns
● Operational, real-time
● HTAP: Transactions+Analytics
Handling Massive Scale ● Distributed DB architecture
● Massively parallel processing
● Compressed storage reduces
footprint and messaging
● Integrates all your data
● Automatic partitioning
● Elastic scaling of resource usage
In-Database Analytics ● GSQL: High-level yet Turing-
complete language
● User-extensible graph algorithm
library, runs in-DB
● ACID (OLTP) and Accumulators
(OLAP)
● Avoids transferring data
● Richer graph context
● In-DB machine learning
5 to 10+ hops deep
10

TigerGraph Architecture - Kafka as a Key Component
11

Data Ingestion Steps Inside of TigerGraph
12
Step 3
Each GPE consumes the
partial data updates,
processes it and puts it on
disk.
Loading Jobs and POST use
UPSERT semantics:
● If vertex/edge doesn't
yet exist, create it.
● If vertex/edge already
exists, update it.
Step 1
Data integration through the
following ways to ingest in
user source data.
● Bulk load of data files or
a Kafka stream in CSV or
JSON format
● HTTP POSTs via REST
services (JSON)
● GSQL Insert commands
Step 2
Dispatcher takes in the data
ingestion requests in the form of
updates to the database.
1. Query IDS to get internal
IDs
2. Convert data to internal
format
3. Send data to one or more
corresponding GPEs

Data Ingestion (Internal)
13
Incremental
Data
Nginx Restpp
GPE GPE GPE
Disk Disk Disk
CSV/JSON Insert/Update/Delete
Vertices and Edges
Listen to
corresponding
topic for new
messages
Acknowledge
Response
Incoming
Outgoing
Synchronize
data to disk
GSE(IDS)
ID Translation
Kafka Kafka Kafka
Server 1 Server 2 Server 3
Kafka Cluster
In-memory
copy of data

Kafka and TigerGraph
-Native Kafka Loader
14
External
Kafka
Cluster

Kafka and TigerGraph Data Pipeline
Static
Data
Sources
Streaming
Data
Sources
Kafka
Loader
15

Kafka Loader - Speed to Value from Real-time
Streaming Data
• Reduce Data Availability Gap and Accelerate Time to Value
• Native Integration with Real-time Streaming Data and Batch
Data
• Enables Real-time Graph Feature Updates with Streaming
Data in Machine Learning Use Cases
• Decrease Learning Curve With Familiar Syntax
• GSQL Support with Consistent Data Loading Syntax
• Maintain Separation of Control for Data Loading
• Designed with Built-in MultiGraph Support
16

Kafka Loader : Three Steps
Consistent with GSQL Data Loading Steps
Step 1: Define the Data Source
Step 2: Create a Loading Job
Step 3: Run the Loading Job
17

Kafka Loader High Level Architecture
● Connect to External Kafka Cluster
● User Commands Through GSQL Server
● Configuration Settings:
○ Config 1: Kakfa Cluster Configuration
○ Config 2: Topic/Partition/Offset Info
18

DEMO
19

Kafka and tgcloud.io
Data Loading Use Cases
20

What is TigerGraph Cloud (tgcloud.io)?
● TigerGraph Cloud is a distributed graph Database-as-a-Service
● Cloud Infrastructure Included
● Out-of-box graph starter kits - industry use case library
● Complete operational support by TigerGraph
○ Upgrades
○ Patches
○ Maintenance
○ Status Monitoring
● Flexible and hourly billing rates by credit card and through prepaid bulk
“cloud credits”
● Multiple versions and multiple clusters supported
● Easy provisioning for distributed clusters
● Built in Encryption and Security
21

TigerGraph Cloud Architecture
TigerGraph Cloud Portal
GraphStudio Admin Portal
GSQL
Web Shell
TigerGraph
Database
+
Graph Starter Kit
GraphStudio Admin Portal
GSQL
Web Shell
TigerGraph
Database
+
Graph Starter Kit
…...
…...
22

Azure Blob + Kafka + tgcloud.io
23
Kafka
Loader
Kafka Connect
Azure Blob Storage
Source Connector
Azure Blob Storage
https://docs.confluent.io/kafka-connect-azure-blob-storage-source/current/index.html

Google Cloud Storage + Kafka + tgcloud.io
24
Kafka
Loader
Kafka Connect
Google Cloud
Storage Source
Connector
https://docs.confluent.io/kafka-connect-gcs-source/current/overview.html

Amazon Kinesis + Kafka + tgcloud.io
25
Kafka
Loader
Kafka Connect
Amazon Kinesis
Source Connector
https://docs.confluent.io/kafka-connect-kinesis/current/index.html

DEMO
26

Summary
27

Summary
● Graph Analytics Overview
● TigerGraph Data Ingestion System Architecture: Kafka as
an Internal Component
● Built-in Kafka Loader in TigerGraph
● TigerGraph Cloud (tgcloud.io) and Kafka Data Pipeline Use
Cases
28

Get Started for Free
● Try TigerGraph Cloud ( tgcloud.io )
● Download TigerGraph’s Developer Edition
● Take a Test Drive - Online Demo
● Get TigerGraph Certified
● Join the Community
@TigerGraphDB /tigergraph /TigerGraphDB /company/TigerGraph
29

Comparing three data ingestion approaches where Apache Kafka integrates with a distributed graph database in real time | Benyue (Emma) Liu, TigerGraph

More Related Content

What's hot (20)

Similar to Comparing three data ingestion approaches where Apache Kafka integrates with a distributed graph database in real time | Benyue (Emma) Liu, TigerGraph (20)

More from HostedbyConfluent (20)

Recently uploaded (20)

Comparing three data ingestion approaches where Apache Kafka integrates with a distributed graph database in real time | Benyue (Emma) Liu, TigerGraph