Agile Data Engineering - Intro to Data Vault Modeling (2016)

KENT GRAZIANO
AGILE DATA ENGINEERING:
INTRODUCTION TO DATA VAULT DATA
MODELING
@KentGraziano kentgraziano.com

Agenda
2
Bio
What do we mean by Agile?
What is a Data Vault?
Where does it fit in an DW/BI architecture
How to design a Data Vault model
Being “agile” with Data Vault
What’s new in DV 2.0

My Bio
3
› Senior Technical Evangelist, Snowflake Computing
› Oracle ACE Director (BI/DW)
› Certified Data Vault Master and DV 2.0 Practitioner
› Data Modeling, Data Architecture and Data Warehouse
Specialist
• 30+ years in IT
• 25+ years of Oracle-related work
• 20+ years of data warehousing experience
› Member – DAMA Houston
› Former-Member: Boulder BI Brain Trust
(http://www.boulderbibraintrust.org/)
› Author & Co-Author of a bunch of books
• The Business of Data Vault Modeling
• The Data Model Resource Book (1st Edition)
› Blogger: The Data Warrior
› Past-President of Oracle Development Tools User Group
and Rocky Mountain Oracle User Group

Manifesto for Agile Software Development
4
“We are uncovering better ways of developing software by
doing it and helping others do it.
Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right, we value
the items on the left more.”
http://agilemanifesto.org/

Applying the Agile Manifesto to DW
(C) Kent Graziano 5
User Stories instead of requirements documents
Time-boxed iterations
› Iteration has a standard length
› Choose one or more user stories to fit in that
iteration
Rework is part of the game
› There are no “missed requirements”... only those
that haven’t been delivered or discovered yet.

Data Vault Definition
TDAN.com Article 6
The Data Vault is a detail oriented, historical tracking and uniquely linked
set of normalized tables that support one or more functional areas of
business.
It is a hybrid approach encompassing the best of breed between 3rd normal
form (3NF) and star schema. The design is flexible, scalable, consistent
and adaptable to the needs of the enterprise.
Architected specifically to meet the needs of today’s
enterprise data warehouses
DAN LINSTEDT: Defining the Data Vault

What is Data Vault Trying to Solve?
(C) Kent Graziano 7
What are our other Enterprise Data Warehouse
options?
› Third-Normal Form (3NF): Complex primary keys (PK’s)
with cascading snapshot dates
› Star Schema (Dimensional): Difficult to reengineer fact
tables for granularity changes
Difficult to get it right the first time
Not adaptable to rapid business change
NOT AGILE!

Data Vault Time Line
© LearnDataVault.com 8
20001960 1970 1980 1990
E.F. Codd invented
relational modeling
Chris Date and
Hugh Darwen
Maintained and
Refined Modeling
1976 Dr Peter Chen
Created E-R
Diagramming
Early 70’s Bill Inmon
Began Discussing
Data Warehousing
Mid 60’s Dimension &
Fact Modeling
presented by General
Mills and Dartmouth
University
Mid 70’s AC Nielsen
Popularized
Dimension & Fact
Terms
Mid – Late 80’s Dr Kimball
Popularizes Star Schema
Mid 80’s Bill Inmon
Popularizes Data
Warehousing
Late 80’s – Barry Devlin and
Dr Kimball Release
“Business Data Warehouse”
1990 – Dan Linstedt
Begins R&D on Data
Vault Modeling
2000 – Dan
Linstedt releases
first 5 articles on
Data Vault
Modeling

Data Vault Evolution
(C) Kent Graziano 9
The work on the Data Vault approach began in the early 1990s, and completed around 1999.
Throughout 1999, 2000, and 2001, the Data Vault design was tested, refined, and deployed
into specific customer sites.
In 2002, the industry thought leaders were asked to review the architecture.
This is when I attend my first DV seminar in Denver and met Dan!
In 2003, Dan began teaching the modeling techniques to the mass public.
In 2014, Dan introduced DV 2.0!

Where does a Data Vault Fit?
STAGING EDW
DATA VAULT
DATA MARTS
(STAR SCHEMAS)
DATA MARTS
(STAR SCHEMAS)
DATA MARTS
(STAR SCHEMAS)

Where does Data Vault fit?
©Oracle Corp 11
Data Vault
goes here

Data Vault: 3 Simple Structures
EDW
DATA VAULT
HUB
LINK
SATELITE
01
02
03

Data Vault Core Architecture
HUBS
Unique List of
Business Keys
LINKS
Unique List of
Relationships
across keys
SATELITES
Descriptive Data
› Satellites have one and only one parent table
› Satellites cannot be “Parents” to other tables
› Hubs cannot be child tables

Common Attributes
Required –
all structures
› Primary key – PK
› Load date time
stamp – DTS
› Record source –
REC_SRC
Required –
Satellites only
› Load end date
time stamp –
LEDTS
› Optional in DV
2.0
Optional –
Hubs & Links
only
› Last seen dates
– LSDTs
› MD5KEY
(REQUIRED IN
DV 2.0)
Optional –
Satellites only
› Load sequence ID –
LDSEQ_ID
› Update user –
UPDT_USER
› Update DTS –
UPDT_DTS
› MD5DIFF

1. Hub = Business Keys
(C) Kent Graziano 15
Hubs = Unique Lists of Business Keys
Business Keys are used to TRACK and IDENTIFY key information
New: DV 2.0 uses MD5 Hash of the BK for the PK

2: Links = Associations
Links = Transactions and Associations
They are used to hook together multiple sets of information
In DV 2.0 the BK attributes may migrate to the Links for faster query

Modeling Links - 1:1 or 1:M?
Today Tomorrow With a Link in The Data Vault
Relationship is
a 1:1 so why
model a Link?
The business rule
can change to a
1:M.
You discover new
data later.
No need to
change the EDW
structure.
Existing data is
fine.
New data is
added.

3. Satellites = Descriptors
Satellites provide context for the Hubs and the Links
Tracks changes over time - Like SCD 2
In DV 2.0 use HASH_DIFF to detect changes

Data Vault Model Flexibility (Agility)
Goes beyond
standard 3NF
Based on natural
business keys
Hyper normalized
› Hubs and Links only hold keys and meta data
› Satellites split by rate of change and/or source
Enables Agile data modeling
› Easy to add to model without having to change existing
structures and load routines
• Relationships (links) can be dropped and created on-demand.
› No more reloading history because of a missed requirement
Not system surrogate keys
Allows for integrating data across functions
and source systems more easily
› All data relationships are key driven

Data Vault Extensibility
(C) LearnDataVault.com 20
Adding new
components to the
EDW has NEAR
ZERO impact to:
› Existing Loading
Processes
› Existing Data Model
› Existing Reporting &
BI Functions
› Existing Source
Systems
› Existing Star
Schemas and Data
Marts

Data Vault Productivity
› Standardized modeling rules
• Highly repeatable and learnable modeling
technique
• Can standardize load routines
o Delta Driven process
o Re-startable, consistent loading patterns.
• Can standardize extract routines
o Rapid build of new or revised Data Marts
• Can be automated
• Can use a BI-meta layer to virtualize the
reporting structures
o Example: OBIEE Business Model and
Mapping tool
o Example: BOBJ Universe Business Layer
• Can put views on the DV structures as well
o Simulate ODS/3NF or Star Schemas

Data Vault Adaptability
› The Data Vault holds granular
historical relationships.
• Holds all history for all time, allowing any
source system feeds to be reconstructed
on-demand
o Easy generation of Audit Trails for data
lineage and compliance.
o Data Mining can discover new
relationships between elements
o Patterns of change emerge from the
historical pictures and linkages.
› The Data Vault can be accessed by
power-users

Other Benefits of a Data Vault
› Modeling it as a DV forces integration
of the Business Keys upfront
• Good for organizational alignment
› An integrated data set with raw data
extends it’s value beyond BI:
• Source for data quality projects
• Source for master data
• Source for data mining
• Source for Data as a Service (DaaS) in
an SOA (Service Oriented Architecture).

Other Benefits of a Data Vault
› Upfront Hub integration simplifies the
data integration routines required to
load data marts.
• Helps divide the work a bit.
› It is much easier to implement security
on these granular pieces.
› Granular, re-startable processes enable
pin-point failure correction.
› It is designed and optimized for real-
time loading in its core architecture
(without any tweaks or mods).

How to be Agile using DV
Model iteratively
› Use Data Vault data
modeling technique
› Create basic components,
then add over time
Virtualize the Access Layer
› Don’t waste time building
facts and dimensions up
front
ETL and testing takes too
long
› “Project” objects using
pattern-based DV model with
database views (or BI meta
layer)
Users see real reports with
real data
› Can always build out for
performance in another
iteration

26
WHAT IS
THE
WORLD'S
SMALLEST
DATA VAULT?

Worlds Smallest Data Vault
Hub Customer
Hub_Cust_Seq_ID
Hub_Cust_Num
Hub_Cust_Load_DTS
Hub_Cust_Rec_Src
Hub_Cust_Seq_ID
Sat_Cust_Load_DTS
Sat_Cust_Load_End_DTS
Sat_Cust_Name
Sat_Cust_Rec_Src
Satellite
Customer Name
› The Data Vault doesn’t have to be “BIG”.
› A Data Vault can be built incrementally.
› Reverse engineering one component of the
existing models is not uncommon.
› Building one part of the Data Vault, then
changing the marts to feed from that vault is a
best practice.
› The smallest Enterprise Data Warehouse
consists of two tables:
• One Hub,
• One Satellite

Notably…
28
› In 2008 Bill Inmon stated that the
“Data Vault is the optimal approach for
modeling the EDW in the DW2.0
framework.” (DW2.0)
› The number of Data Vault users in the
US surpassed 500 in 2010 and grows
rapidly (http://danlinstedt.com/about/dv-
customers/)

Organizations using Data Vault
29
› WebMD Health Services
› Anthem Blue-Cross Blue Shield
› MD Anderson Cancer Center
› Denver Public Schools
› Independent Purchasing Cooperative
(IPC, Miami)
• Owner of Subway
› Kaplan
› US Defense Department
› Colorado Springs Utilities
› State Court of Wyoming
› Federal Express
› US Dept. Of Agriculture

What’s New in DV2.0?
Modeling
Structure
Includes…
› NoSQL, and Non-
Relational DB
systems, Hybrid
Systems
› Minor Structure
Changes to support
NoSQL
01 02 03 04
New ETL
Implementation
Standards
› For true real-time
support
› For NoSQL support
New
Architecture
Standards
› To include support
for NoSQL data
management
systems
New
Methodology
Components
› Including CMMI, Six
Sigma, and TQM
› Including Project
Planning, Tracking,
and Oversight
› Agile Delivery
Mechanisms
› Standards, and
templates for
Projects

What’s New in DV2.0?
This model is
fully compliant
with Hadoop,
needs NO
changes to work
properly
Note:
Business Keys
replicated to the Link
structure for “join”
capabilities on the
way out to Data
Marts.

Summary
32
Data Vault provides a data modeling technique that allows:
Model Agility Productivity So? Agile Data
Warehousing?
01 02 03
› Enabling rapid
changes and
additions
› Enabling low
complexity
systems with high
value output at a
rapid pace
› Easy projections
of dimensional
models

33
› Available on Amazon:
http://www.amazon.com/Better-
Data-Modeling-Introduction-
Engineering-ebook /dp/
B018BREV1C/
Shameless Plug:

34
› Available on Amazon.com
› Soft Cover or Kindle Format
› Now also available in PDF at
LearnDataVault.com
› Hint: Kent is the Technical Editor
Super Charge
Your Data Warehouse

35
› Available on Amazon:
http://www.amazon.com/Buildin
g-Scalable-Data-Warehouse-
Vault/dp/0128025107/
New DV 2.0 Book

Data Vault References
37
www.youtube.com/LearnDataVault www.facebook.com/learndatavault
www.learndatavault.com
www.danlinstedt.com

Contact Information
39
KENT GRAZIANO
Snowflake Computing
www.snowflake.net
kent.graziano@snowflake.net
@KentGraziano
http://kentgraziano.com

Agile Data Engineering - Intro to Data Vault Modeling (2016)

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Agile Data Engineering - Intro to Data Vault Modeling (2016) (20)

More from Kent Graziano (8)

Recently uploaded (20)

Agile Data Engineering - Intro to Data Vault Modeling (2016)

Editor's Notes