SlideShare a Scribd company logo
Thank you for joining us, the webinar will start at:
10:00 Pacific / 12:00 Central / 13:00 East / 18:00 UK Time
Manage Databases like Codebases
Before we start
• You will be on mute for the duration of the event
• We are now talking so please type a message
in the Questions box in the Control Panel if you can’t
hear us (please check your speakers and
GoToWebinar audio settings first)
• There will be a Q+A session at the end but please
feel free to type your questions in the Questions
box in the Control Panel in advance
• A recording of the full webinar will be put up online
Presenters
Tim O'Brien
• Analyst, CTO
• Gleanster Research
Yaniv Yehuda @yanivyehuda
• Co-Founder & CTO at DBmaestro
• Presenter at world-wide conferences: ODTUG, ilOUG
Kyle Hailey @kylehhailey
• Technical Evangelist at Delphix
• Oracle ACE, member of the OakTable Network
About Delphix
• Founded in 2008, launched in 2010
• CEO Jedidiah Yueh (founder of Avamar: >$1B revenue))
• Based in Silicon Valley, Global Operations
• 10% of Fortune 500
About DBmaestro
• Founded in 2008, launched in 2010
• Founded by Yariv Tabac and Yaniv Yehuda
• Headquartered in Israel, Global Operations
Manage Databases like Codebases
Kyle Hailey & Yaniv Yehuda
The Business Need
Copyright@2010, Gartner RAS Core Research
80%
of unplanned
downtime is
due to Change
50%
More than
of this, half is
due to human
errors
40% of changes FAIL Copyright@2008, Juniper Networks, Inc.
Code, Databases, and
Agility in 2014
May20,2014
Tim O’Brien
Analyst, CTO
Gleanster Research
Industry
Trends
Code in 2014
 Distributed& AsynchronousNew Normal
 Self-serviceCriticaltoProductivity
 IncreasingVariety (fromJavascript,Java, .NET,
Ruby,Python+everything)
Everything is Automated
Industry
Trends
Databases in 2014
 More thanjustRelational
 IncreasingVariety (Oracle,Mongo)
 Management? Notas Matureas Code
Oddity
 Big Datamarginallymore manageablethan
traditionalRDBMS
 “We’restillwaitingforthedatabase…”
Code is King
Modern Enterprise
 Fully distributed development
 Continuous Deployment Pipelines
 Everything Automated! - DevOps, PaaS,
IaaS
“Every Company is a Software
Company”
How did we get here?
Scale
Development
Open Source @ Scale
 Google Chrome, Mozilla Firefox
 Android, Apache httpd
 Linux*, FreeBSD
Characteristics
 Rapid Iterations, Frequent Releases
 Numerous Experiments (or Branches)
 Thousands of developers
 OSS @ Scale established best practices
now in use.
Scale
Development
Organizations @ Scale
 Facebook,Wal-Mart, eBay, Salesforce,
Intuit
 Goldman Sachs, Apple, Amazon,Yahoo!
 Google, SSA, Ford,Toyota
Characteristics
 Rapid Iterations, Frequent Releases
 Numerous Experiments (or Branches)
 Thousands of developers, ManyTeams
@Scale
Everything is
Continuous
Continuous integration and
deployment
Modern set ofTools Integrated with:
 Source Control – Perforce, Clearcase, Git
 Infrastructure-as-a-Service – Openstack
 Platform-as-a-Service – AWS, Heroku
 Databases? - Still an Integration Nightmare
Case in Point
 Example Fortune 500 Relaunch
 Re-architecture: More distributed, Move to
a PaaS
 Scope: Everything, 2Year Migration Project
 Size: 1500 Developers, Several Continents,
$100M+
 Results?
 Code: Distributed Projects, Continuous
Deployment
 Infrastructure: Self-service, On-demand,
Agile
 Database: Delays, Manual Builds, Cost
Overruns
RootCause
Analysis
 Code is Agile
 Developers move FAST.
 Branches are fungible, disposable, personal.
 Deployments managed with DevOps tools
(immediately)
 Traditional Databases are Not
 Costly to Setup
 Often require Dedicated Hardware
 Little automation. No Change Management
“The rest of the department is
Virtual but our Databases are
stuck in 1998.”
Worst
Practices
(we’reall used to)
 Lack of Database Change Management:
 Shared Developer Databases Results in Contention
 Costly, Slow Environment Provisioning
 Testing and Staging Not Synchronized with
Production
 Drag on Productivity and Efficiency
 Opportunity lost
 “We’ve accepted that the databases causes
delays”
What We’ve Seen
1. QA: Expensive, Slow
2. Development: Bottlenecks & bugs
3. Provisioning: Delays
1. QA: Expensive, Slow : Long Build times
Build Time
96% of QA time was building environment
$.04/$1.00 actual testing vs. setup
Build
Build Time
QA
Test
QA
Test
Build
1. QA: Expensive, Slow: bugs found late
Build QA Env QA Build QA Env QA
Sprint 1 Sprint 2 Sprint 3
Bug CodeX
0
10
20
30
40
50
60
70
1 2 3 4 5 6 7
Delay in Fixing the bug
Cost
To
Correct
Software Engineering Economics – Barry Boehm (1981)
Build QA Env QA
2. Development: Bottlenecks & bugs: Bottlenecks
Frustration Waiting
2. Development: Bottlenecks & bugs: subsets
cause bugs
2. Development: Bottlenecks & bugs: subsets
cause bugs
3. Provisioning Delays : weeks to months
Management
DBA
System Admin
Storage Admin
Developers
Submit
Request
Disk
Capacity?
Approve
Request $$
(2 Weeks)
Approve
Request $$
(1 Week)
Request
Additional
Storage?
Provision
Capacity
File System
Configured?
Configure LUNS &
Build File System
Coordinate
Replication w/
Infrastructure
Re-
Parameterize &
Configure DB
Mount
Recovery DB
to Specific PIT
Begin
Work
Approve
Request $$
(2 Weeks)
(3 Days)
(3 Days)
(2 Days)
(3 Days)
(3 Days)
…….1-2 Weeks of Approvals, Delays, and Provisioning……
24
Developer Asks for DB Get Access
Manager approves
DBA Request
system
Setup DB
System
Admin
Request
storage
Setup
machine
Storage
Admin
Allocate
storage
(take snapshot)
3. Provisioning Delays : culture of no
What We’ve Seen
1. QA: Higher costs more code re-work
2. Development: Bottlenecks & bugs
3. Provisioning: Delays
Three Physical Copies Three Virtual Copies
Install Delphix on Commodity Intel Hardware
Intel hardware
Allocate Any Storage to Delphix
Allocate Storage
Any type
One time backup of source database
Database
Production
Instance
File systemFile system
DxFS (Delphix) Compress Data
Database
Production
Instance
Data is
compressed
typically 1/3
size
File system
Incremental forever change collection
Database
Production
Instance
File system
Changes
• Collected incrementally forever
• Old data purged
Time Window
File system
Typical Architecture
Production
Instance
Development
Instance
QA
Instance
UAT
Instance
Database
File systemFile system
Database
File systemFile system
Database
File systemFile system
Database
File systemFile system
With Delphix
Database
File system
Production
Instance
Database
Development
Instance
Database
QA
Instance
Database
UAT
Instance
Parallel environments
Instance
Instance
Instance
Instance
Source
Delphix and DBmaestro
What We’ve Seen With Delphix
1. QA: Low cost, fast bug detection
2. Development: Parallelized, less bugs
3. Provisioning: Fast, Culture of Yes
1. QA Efficient : Lower cost
B
u
i
l
d
T
i
m
e
QA Test
1% of QA time was building environment
$.99/$1.00 actual testing vs. setup
Build Time
QA Test
Build
1. QA Fast : bugs found fast and fixed
Sprint 1 Sprint 2 Sprint 3
Bug CodeX
QA QA
Build QA
Env
Q
A
Build QA
Env
Q
A
Sprint 1 Sprint 2 Sprint 3
Bug
Cod
e
X
2. Development : Parallelize
gif by Steve Karam
2. Development: Eliminate bugs
3. Provisioning: Fast, Efficient, Self Service, Culture
of Yes!
What We’ve Seen With Delphix
1. QA: Low cost, fast bug detection
2. Development: Parallelized, less bugs
3. Provisioning: Fast, Culture of Yes
But …
How do we manage database changes as code
changes and automate deployment of changes?
Dealing with Risk
 Smaller and more focused changes are easier to manage (Agile…)
 Automation of repeating tasks lowers risk of (human) error
 Development and Operations should work in synergy (DevOps)
45
• More rapid changes
• Reacting quickly to market needs
• Getting ahead of competition
• Fewer changes backed out
• Better collaboration
• Fewer defects
• Ultimately better service
• Happy customers
• Profitability
Why Continuous Delivery?
46
• Team and process
• Version everything
• Automate your tests
• Fix it, properly (no out of process changes!)
• Automate your deployments
Database Continuous Delivery?
47
A quick poll
48
• The database holds your essential information
• Changes can impact the entire system
• Need to be synchronized with other changes
• Often overlooked
Database is a Key Component
49
• There is more to a database than SQL scripts
– Schema structure
– Code
– Content and meta-content
– Internal dependencies
– …
• Ensure that changes are not made without
authorization
• Ensure no out-of-process changes
Reaching Inside the Database
50
• Old adage but true
• The database is often neglected and therefore can
become the weakest link
• Essential from a compliance point of view
• Should be the strongest link
The Weakest Link In a Chain
51
Challenges…
Un-coordinated Process
Silos in development
Deployment delays…
Out-of-Process Changes
Errors in production
Lack of confidence in automation
Code overrides
Different methodologies
Lack of history
Missing changes
wrong versions
52
Two isolated Processes
Version Control Process Development Process
Check-Out
Script
Modify Script
Get updated
Script from DB
Check-In
Script
Compile
Script
in DB
Debug Script
in DB
?
?
?
?
A
A’
53
X
1.11.1.11.11.21.31.41.51.61.7
Build Once Deploy Many
Int QA Stage Prod
Database Deploy Script
Environment
Re-Base (due to defects)
Dev
Dev
Dev
Model
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
1.11.11.41.7
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
Out of Process
Change
X
X
X
X
X
? 1.1.1
X
54
Dealing with challenges…
Coordinated ProcessTraceability
Start in the
Beginning
No Out-of-Process Changes
Impact Analysis
Automation
Task Based Development
Well Defined Processes
What is DBmaestro TeamWork
• Database Enforced Change Management
+ Database version control
+ Plugs into the ALM (change request, tickets & work items)
+ Database merge & change impact analysis
• DevOps Solution for databases
+ Baseline aware deployment automation, rollback &
recovery
+ Plugs into release management & Continuous
Delivery
How?
• Database version control
– Enforced Check Out/In
– Labels
– Rollback/Undo
– Audit trail reports
• Database impact analysis
– Utilizes version control repository information
– 3-way analysis
• Database deployment automation
– API
– Baselines
– Conflict resolution
– Customized business logic
57
Version Control - One Enforced Process
58
1.11.21.31.41.51.61.7
Build & Deploy On Demand
*
Int QA Stage Prod
Database Deploy Script
Environment* Execute the same script
being executed at the
Stage environment
Re-Base (due to defects)
Dev
Dev
Dev
Model
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
1.1 1.4
1.4 1.7
1.1.1 1.7
1.1 1.1 1.11.41.7
File Based
Version Control
Out of Process
Change
1.1.11.7 1.1.11.7
Safety Net For Deployment Automation
Source vs.
Target
Action
= No Action
≠
?
Source vs.
Baseline
Target vs.
Baseline
Action
= = No Action
≠ = Deploy Changes
= ≠ Protect Target
≠ ≠ Merge Changes
You do not have all
of the information
With Baselines and 3 way
analysis the unknown is
now known
Simple Compare & Sync Baseline aware Deployment
An index exists in Target (Production) but not in source (QA).
What should we do? Drop the index or not?
Benefits - Development
• Database change repository
• Follow SCM best practices (Check-Out/Check-In)
• All changes are documented
• Manage who can do what, where, when & why
Benefits - Operations
• Integrated deployment engine
• Business level audit
• Roles & responsibilities enforcement
Benefits - Management
• Complete visibility into changes in progress
• Management reports, Audit reports
• No silos
+
Time Window
Instance
Instance
Instance
Virtual DatabaseDev1
Dev2
Trunk
Virtual Database
Virtual Database
Developer 1 modify
Developer 2
modify
DB
VC
Dev2
Dev1Merge to dev1
Trunk
DB
VC
Fork
Fork
Fork
Fork
Safe and Efficient Work Process
• Clone relevant number of virtual copies of the Trunk
– Dev Team 1-n, Developer a-z, Hot fix environment, Etc…
– Work in parallel, save time
• Rely on enforced change process
– Know exactly who did what, when and why
• Deployment automation
– Save time, mitigate risk
– Continuous integration, Continuous Delivery
• Deal with conflicts & merge them into the Trunk
• Test before deploy to production
– Pre-prod clone
Q&A
Kyle Hailey @kylehhailey
Delphix: delphix.com
Yaniv Yehuda @yanivyehuda
DBmaestro: dbmaestro.com
Tim O'Brien
Gleanster: gleanster.com
Thanks!
Kyle Hailey @kylehhailey
Delphix: delphix.com
Yaniv Yehuda @yanivyehuda
DBmaestro: dbmaestro.com
Tim O'Brien
Gleanster: gleanster.com

More Related Content

What's hot (20)

PPTX
The Future of Change Management and DevOps for Dummies
DBmaestro - Database DevOps
 
PPTX
Iltam database version control
uridbmaestro
 
PPTX
Database CI/CD Pipeline
muhammadhashir57
 
PPTX
Version Control meets Database Control
DBmaestro - Database DevOps
 
PDF
The future of DevOps: fully left-shifted deployments with version control and...
Red Gate Software
 
PDF
Continuous Integration for Oracle Database Development
Vladimir Bakhov
 
PPTX
Redgate database DevOps demo webinar (with Git & Jenkins)
Red Gate Software
 
PPTX
DevOps Drive-In: Automate Database Deployments in Your Continuous Delivery Pi...
Serena Software
 
PPT
Database continuous integration, unit test and functional test
Harry Zheng
 
PPTX
Continuous delivery for databases
DevOpsGroup
 
PPTX
Automate Database Deployment - SQL In The City Workshop
Red Gate Software
 
PDF
Using Redgate, AKS and Azure to bring DevOps to your Database
Red Gate Software
 
PDF
Microsoft SQL Server Continuous Integration
Mark Ginnebaugh
 
PPTX
Successfully migrating existing databases to Azure
Red Gate Software
 
PPSX
Continuous Integration - Oracle Database Objects
Prabhu Ramasamy
 
PPTX
What Is Slowing My Application Releases?
Datical
 
PDF
How the Big Data of APM can Supercharge DevOps
CA Technologies
 
PDF
Embracing DevOps through database migrations with Flyway
Red Gate Software
 
PPTX
New DevOps for the DBA
Kellyn Pot'Vin-Gorman
 
PDF
Common Challenges in DevOps Change Management
Matt Ray
 
The Future of Change Management and DevOps for Dummies
DBmaestro - Database DevOps
 
Iltam database version control
uridbmaestro
 
Database CI/CD Pipeline
muhammadhashir57
 
Version Control meets Database Control
DBmaestro - Database DevOps
 
The future of DevOps: fully left-shifted deployments with version control and...
Red Gate Software
 
Continuous Integration for Oracle Database Development
Vladimir Bakhov
 
Redgate database DevOps demo webinar (with Git & Jenkins)
Red Gate Software
 
DevOps Drive-In: Automate Database Deployments in Your Continuous Delivery Pi...
Serena Software
 
Database continuous integration, unit test and functional test
Harry Zheng
 
Continuous delivery for databases
DevOpsGroup
 
Automate Database Deployment - SQL In The City Workshop
Red Gate Software
 
Using Redgate, AKS and Azure to bring DevOps to your Database
Red Gate Software
 
Microsoft SQL Server Continuous Integration
Mark Ginnebaugh
 
Successfully migrating existing databases to Azure
Red Gate Software
 
Continuous Integration - Oracle Database Objects
Prabhu Ramasamy
 
What Is Slowing My Application Releases?
Datical
 
How the Big Data of APM can Supercharge DevOps
CA Technologies
 
Embracing DevOps through database migrations with Flyway
Red Gate Software
 
New DevOps for the DBA
Kellyn Pot'Vin-Gorman
 
Common Challenges in DevOps Change Management
Matt Ray
 

Viewers also liked (20)

PPTX
SCM Patterns for Agile Architectures
Brad Appleton
 
PDF
Go2Group_secrets of high-performing software teams_EAD event_san jose_Doug Bass
Go2Group, Inc.
 
PPTX
Trustworthy Transparency and Lean Traceability
Brad Appleton
 
PDF
How do you deliver your applications to the cloud?
Michael Elder
 
PPT
P4 Branching Overview
Go2Group, Inc.
 
PPT
Tui Travel - Overcoming the Challenges of Agile Methods
DBmaestro - Database DevOps
 
PPTX
Jenkins Plugin
DelphixCorp
 
PDF
Software Configuration Management Problemas e Soluções
elliando dias
 
PDF
Faking Hell
Giovanni Asproni
 
PPTX
DBTA Data Summit : Eliminating the data constraint in Application Development
Kyle Hailey
 
PDF
Is agile adoption losing steam?
Go2Group, Inc.
 
PPTX
Kscope 2013 delphix
Kyle Hailey
 
PDF
Delphix modernization whitepaper
Franco_Dagosto
 
PPTX
WANTED: Seeking Single Agile Knowledge Development Tool-set
Brad Appleton
 
PPTX
Continuous delivery made possible
mimmozzo_
 
PDF
Test case management and requirements traceability
Go2Group, Inc.
 
PPTX
Agile Configuration Management Environments
Brad Appleton
 
PPTX
Preventing the Next Deployment Issue with Continuous Performance Testing and ...
Correlsense
 
PPTX
Delphix Workflow for SQL Server
rcaccia
 
PDF
MuleSoft Connect 2015 - Go2Group presentation
Go2Group, Inc.
 
SCM Patterns for Agile Architectures
Brad Appleton
 
Go2Group_secrets of high-performing software teams_EAD event_san jose_Doug Bass
Go2Group, Inc.
 
Trustworthy Transparency and Lean Traceability
Brad Appleton
 
How do you deliver your applications to the cloud?
Michael Elder
 
P4 Branching Overview
Go2Group, Inc.
 
Tui Travel - Overcoming the Challenges of Agile Methods
DBmaestro - Database DevOps
 
Jenkins Plugin
DelphixCorp
 
Software Configuration Management Problemas e Soluções
elliando dias
 
Faking Hell
Giovanni Asproni
 
DBTA Data Summit : Eliminating the data constraint in Application Development
Kyle Hailey
 
Is agile adoption losing steam?
Go2Group, Inc.
 
Kscope 2013 delphix
Kyle Hailey
 
Delphix modernization whitepaper
Franco_Dagosto
 
WANTED: Seeking Single Agile Knowledge Development Tool-set
Brad Appleton
 
Continuous delivery made possible
mimmozzo_
 
Test case management and requirements traceability
Go2Group, Inc.
 
Agile Configuration Management Environments
Brad Appleton
 
Preventing the Next Deployment Issue with Continuous Performance Testing and ...
Correlsense
 
Delphix Workflow for SQL Server
rcaccia
 
MuleSoft Connect 2015 - Go2Group presentation
Go2Group, Inc.
 
Ad

Similar to Delphix and DBmaestro (20)

PPTX
manage databases like codebases
DBmaestro - Database DevOps
 
PPTX
DevOps for Database webinar
DBmaestro - Database DevOps
 
PDF
Workshop: Delivering chnages for applications and databases
Eduardo Piairo
 
PDF
DataOps, DevOps and the Developer: Treating Database Code Just Like App Code
DevOps.com
 
PDF
Delivering changes for databases and applications
Eduardo Piairo
 
PDF
Relational Database CI/CD
Jasmin Fluri
 
PDF
Delivering Changes for Applications and Databases
Miguel Alho
 
PPTX
Geek Sync I In Database Automation We Trust
IDERA Software
 
PDF
Geek Sync | Is Your Database Environment Ready for DevOps?
IDERA Software
 
PDF
A Continuous Delivery Safety Net for Databases
IBM UrbanCode Products
 
PPTX
Challenges and best practices of database continuous delivery
DBmaestro - Database DevOps
 
PPTX
The Challenges & Pitfalls of Database Continuous Delivery
Perforce
 
PPTX
Webinar - Devops platform for the evolving enterprise
DBmaestro - Database DevOps
 
PPTX
Webinar: "DBMaestro: Database Enforced Change Management (DECM) tool"
Emerasoft, solutions to collaborate
 
PPTX
Webinar: "In database automation we trust"
Emerasoft, solutions to collaborate
 
PPTX
Continuous Database Delivery - 7/12/2018
David P. Moore
 
PPTX
How Financial Institutions Must Enforce DevOps Organizational Policy
Tamir Belzer
 
PPTX
Bringing DevOps to the Database
Michaela Murray
 
PDF
Delivering changes for applications and databases
Eduardo Piairo
 
PPTX
BGOUG "Agile Data: revolutionizing database cloning'
Kyle Hailey
 
manage databases like codebases
DBmaestro - Database DevOps
 
DevOps for Database webinar
DBmaestro - Database DevOps
 
Workshop: Delivering chnages for applications and databases
Eduardo Piairo
 
DataOps, DevOps and the Developer: Treating Database Code Just Like App Code
DevOps.com
 
Delivering changes for databases and applications
Eduardo Piairo
 
Relational Database CI/CD
Jasmin Fluri
 
Delivering Changes for Applications and Databases
Miguel Alho
 
Geek Sync I In Database Automation We Trust
IDERA Software
 
Geek Sync | Is Your Database Environment Ready for DevOps?
IDERA Software
 
A Continuous Delivery Safety Net for Databases
IBM UrbanCode Products
 
Challenges and best practices of database continuous delivery
DBmaestro - Database DevOps
 
The Challenges & Pitfalls of Database Continuous Delivery
Perforce
 
Webinar - Devops platform for the evolving enterprise
DBmaestro - Database DevOps
 
Webinar: "DBMaestro: Database Enforced Change Management (DECM) tool"
Emerasoft, solutions to collaborate
 
Webinar: "In database automation we trust"
Emerasoft, solutions to collaborate
 
Continuous Database Delivery - 7/12/2018
David P. Moore
 
How Financial Institutions Must Enforce DevOps Organizational Policy
Tamir Belzer
 
Bringing DevOps to the Database
Michaela Murray
 
Delivering changes for applications and databases
Eduardo Piairo
 
BGOUG "Agile Data: revolutionizing database cloning'
Kyle Hailey
 
Ad

More from Kyle Hailey (20)

PPTX
Hooks in postgresql by Guillaume Lelarge
Kyle Hailey
 
PPTX
Performance insights twitch
Kyle Hailey
 
PPT
History of database monitoring
Kyle Hailey
 
PPT
Ash masters : advanced ash analytics on Oracle
Kyle Hailey
 
PPT
Successfully convince people with data visualization
Kyle Hailey
 
PPTX
Virtual Data : Eliminating the data constraint in Application Development
Kyle Hailey
 
PPTX
Accelerate Develoment with VIrtual Data
Kyle Hailey
 
PPTX
Delphix and Pure Storage partner
Kyle Hailey
 
PPTX
Mark Farnam : Minimizing the Concurrency Footprint of Transactions
Kyle Hailey
 
PDF
Dan Norris: Exadata security
Kyle Hailey
 
PDF
Martin Klier : Volkswagen for Oracle Guys
Kyle Hailey
 
PPTX
What is DevOps
Kyle Hailey
 
PPTX
Data as a Service
Kyle Hailey
 
PPTX
Data Virtualization: Revolutionizing data cloning
Kyle Hailey
 
PPTX
Denver devops : enabling DevOps with data virtualization
Kyle Hailey
 
PPTX
Oracle Open World 2014: Lies, Damned Lies, and I/O Statistics [ CON3671]
Kyle Hailey
 
PPT
Jonathan Lewis explains Delphix
Kyle Hailey
 
PDF
Oaktable World 2014 Toon Koppelaars: database constraints polite excuse
Kyle Hailey
 
PDF
Profiling the logwriter and database writer
Kyle Hailey
 
PDF
Oaktable World 2014 Kevin Closson: SLOB – For More Than I/O!
Kyle Hailey
 
Hooks in postgresql by Guillaume Lelarge
Kyle Hailey
 
Performance insights twitch
Kyle Hailey
 
History of database monitoring
Kyle Hailey
 
Ash masters : advanced ash analytics on Oracle
Kyle Hailey
 
Successfully convince people with data visualization
Kyle Hailey
 
Virtual Data : Eliminating the data constraint in Application Development
Kyle Hailey
 
Accelerate Develoment with VIrtual Data
Kyle Hailey
 
Delphix and Pure Storage partner
Kyle Hailey
 
Mark Farnam : Minimizing the Concurrency Footprint of Transactions
Kyle Hailey
 
Dan Norris: Exadata security
Kyle Hailey
 
Martin Klier : Volkswagen for Oracle Guys
Kyle Hailey
 
What is DevOps
Kyle Hailey
 
Data as a Service
Kyle Hailey
 
Data Virtualization: Revolutionizing data cloning
Kyle Hailey
 
Denver devops : enabling DevOps with data virtualization
Kyle Hailey
 
Oracle Open World 2014: Lies, Damned Lies, and I/O Statistics [ CON3671]
Kyle Hailey
 
Jonathan Lewis explains Delphix
Kyle Hailey
 
Oaktable World 2014 Toon Koppelaars: database constraints polite excuse
Kyle Hailey
 
Profiling the logwriter and database writer
Kyle Hailey
 
Oaktable World 2014 Kevin Closson: SLOB – For More Than I/O!
Kyle Hailey
 

Recently uploaded (20)

PDF
[Solution] Why Choose the VeryPDF DRM Protector Custom-Built Solution for You...
Lingwen1998
 
PPTX
Tally software_Introduction_Presentation
AditiBansal54083
 
PPTX
Homogeneity of Variance Test Options IBM SPSS Statistics Version 31.pptx
Version 1 Analytics
 
PPTX
Change Common Properties in IBM SPSS Statistics Version 31.pptx
Version 1 Analytics
 
PPTX
AEM User Group: India Chapter Kickoff Meeting
jennaf3
 
PDF
Odoo CRM vs Zoho CRM: Honest Comparison 2025
Odiware Technologies Private Limited
 
PPTX
ChiSquare Procedure in IBM SPSS Statistics Version 31.pptx
Version 1 Analytics
 
PDF
iTop VPN With Crack Lifetime Activation Key-CODE
utfefguu
 
PDF
4K Video Downloader Plus Pro Crack for MacOS New Download 2025
bashirkhan333g
 
PDF
Empower Your Tech Vision- Why Businesses Prefer to Hire Remote Developers fro...
logixshapers59
 
PPTX
Agentic Automation: Build & Deploy Your First UiPath Agent
klpathrudu
 
PDF
Unlock Efficiency with Insurance Policy Administration Systems
Insurance Tech Services
 
PDF
Alexander Marshalov - How to use AI Assistants with your Monitoring system Q2...
VictoriaMetrics
 
PPTX
Agentic Automation Journey Session 1/5: Context Grounding and Autopilot for E...
klpathrudu
 
PPTX
Finding Your License Details in IBM SPSS Statistics Version 31.pptx
Version 1 Analytics
 
PDF
Why Businesses Are Switching to Open Source Alternatives to Crystal Reports.pdf
Varsha Nayak
 
PDF
Digger Solo: Semantic search and maps for your local files
seanpedersen96
 
PDF
vMix Pro 28.0.0.42 Download vMix Registration key Bundle
kulindacore
 
PDF
Wondershare PDFelement Pro Crack for MacOS New Version Latest 2025
bashirkhan333g
 
PDF
Generic or Specific? Making sensible software design decisions
Bert Jan Schrijver
 
[Solution] Why Choose the VeryPDF DRM Protector Custom-Built Solution for You...
Lingwen1998
 
Tally software_Introduction_Presentation
AditiBansal54083
 
Homogeneity of Variance Test Options IBM SPSS Statistics Version 31.pptx
Version 1 Analytics
 
Change Common Properties in IBM SPSS Statistics Version 31.pptx
Version 1 Analytics
 
AEM User Group: India Chapter Kickoff Meeting
jennaf3
 
Odoo CRM vs Zoho CRM: Honest Comparison 2025
Odiware Technologies Private Limited
 
ChiSquare Procedure in IBM SPSS Statistics Version 31.pptx
Version 1 Analytics
 
iTop VPN With Crack Lifetime Activation Key-CODE
utfefguu
 
4K Video Downloader Plus Pro Crack for MacOS New Download 2025
bashirkhan333g
 
Empower Your Tech Vision- Why Businesses Prefer to Hire Remote Developers fro...
logixshapers59
 
Agentic Automation: Build & Deploy Your First UiPath Agent
klpathrudu
 
Unlock Efficiency with Insurance Policy Administration Systems
Insurance Tech Services
 
Alexander Marshalov - How to use AI Assistants with your Monitoring system Q2...
VictoriaMetrics
 
Agentic Automation Journey Session 1/5: Context Grounding and Autopilot for E...
klpathrudu
 
Finding Your License Details in IBM SPSS Statistics Version 31.pptx
Version 1 Analytics
 
Why Businesses Are Switching to Open Source Alternatives to Crystal Reports.pdf
Varsha Nayak
 
Digger Solo: Semantic search and maps for your local files
seanpedersen96
 
vMix Pro 28.0.0.42 Download vMix Registration key Bundle
kulindacore
 
Wondershare PDFelement Pro Crack for MacOS New Version Latest 2025
bashirkhan333g
 
Generic or Specific? Making sensible software design decisions
Bert Jan Schrijver
 

Delphix and DBmaestro

  • 1. Thank you for joining us, the webinar will start at: 10:00 Pacific / 12:00 Central / 13:00 East / 18:00 UK Time Manage Databases like Codebases
  • 2. Before we start • You will be on mute for the duration of the event • We are now talking so please type a message in the Questions box in the Control Panel if you can’t hear us (please check your speakers and GoToWebinar audio settings first) • There will be a Q+A session at the end but please feel free to type your questions in the Questions box in the Control Panel in advance • A recording of the full webinar will be put up online
  • 3. Presenters Tim O'Brien • Analyst, CTO • Gleanster Research Yaniv Yehuda @yanivyehuda • Co-Founder & CTO at DBmaestro • Presenter at world-wide conferences: ODTUG, ilOUG Kyle Hailey @kylehhailey • Technical Evangelist at Delphix • Oracle ACE, member of the OakTable Network
  • 4. About Delphix • Founded in 2008, launched in 2010 • CEO Jedidiah Yueh (founder of Avamar: >$1B revenue)) • Based in Silicon Valley, Global Operations • 10% of Fortune 500
  • 5. About DBmaestro • Founded in 2008, launched in 2010 • Founded by Yariv Tabac and Yaniv Yehuda • Headquartered in Israel, Global Operations
  • 6. Manage Databases like Codebases Kyle Hailey & Yaniv Yehuda
  • 7. The Business Need Copyright@2010, Gartner RAS Core Research 80% of unplanned downtime is due to Change 50% More than of this, half is due to human errors 40% of changes FAIL Copyright@2008, Juniper Networks, Inc.
  • 8. Code, Databases, and Agility in 2014 May20,2014 Tim O’Brien Analyst, CTO Gleanster Research
  • 9. Industry Trends Code in 2014  Distributed& AsynchronousNew Normal  Self-serviceCriticaltoProductivity  IncreasingVariety (fromJavascript,Java, .NET, Ruby,Python+everything) Everything is Automated
  • 10. Industry Trends Databases in 2014  More thanjustRelational  IncreasingVariety (Oracle,Mongo)  Management? Notas Matureas Code Oddity  Big Datamarginallymore manageablethan traditionalRDBMS  “We’restillwaitingforthedatabase…”
  • 11. Code is King Modern Enterprise  Fully distributed development  Continuous Deployment Pipelines  Everything Automated! - DevOps, PaaS, IaaS “Every Company is a Software Company” How did we get here?
  • 12. Scale Development Open Source @ Scale  Google Chrome, Mozilla Firefox  Android, Apache httpd  Linux*, FreeBSD Characteristics  Rapid Iterations, Frequent Releases  Numerous Experiments (or Branches)  Thousands of developers  OSS @ Scale established best practices now in use.
  • 13. Scale Development Organizations @ Scale  Facebook,Wal-Mart, eBay, Salesforce, Intuit  Goldman Sachs, Apple, Amazon,Yahoo!  Google, SSA, Ford,Toyota Characteristics  Rapid Iterations, Frequent Releases  Numerous Experiments (or Branches)  Thousands of developers, ManyTeams
  • 14. @Scale Everything is Continuous Continuous integration and deployment Modern set ofTools Integrated with:  Source Control – Perforce, Clearcase, Git  Infrastructure-as-a-Service – Openstack  Platform-as-a-Service – AWS, Heroku  Databases? - Still an Integration Nightmare
  • 15. Case in Point  Example Fortune 500 Relaunch  Re-architecture: More distributed, Move to a PaaS  Scope: Everything, 2Year Migration Project  Size: 1500 Developers, Several Continents, $100M+  Results?  Code: Distributed Projects, Continuous Deployment  Infrastructure: Self-service, On-demand, Agile  Database: Delays, Manual Builds, Cost Overruns
  • 16. RootCause Analysis  Code is Agile  Developers move FAST.  Branches are fungible, disposable, personal.  Deployments managed with DevOps tools (immediately)  Traditional Databases are Not  Costly to Setup  Often require Dedicated Hardware  Little automation. No Change Management “The rest of the department is Virtual but our Databases are stuck in 1998.”
  • 17. Worst Practices (we’reall used to)  Lack of Database Change Management:  Shared Developer Databases Results in Contention  Costly, Slow Environment Provisioning  Testing and Staging Not Synchronized with Production  Drag on Productivity and Efficiency  Opportunity lost  “We’ve accepted that the databases causes delays”
  • 18. What We’ve Seen 1. QA: Expensive, Slow 2. Development: Bottlenecks & bugs 3. Provisioning: Delays
  • 19. 1. QA: Expensive, Slow : Long Build times Build Time 96% of QA time was building environment $.04/$1.00 actual testing vs. setup Build Build Time QA Test QA Test Build
  • 20. 1. QA: Expensive, Slow: bugs found late Build QA Env QA Build QA Env QA Sprint 1 Sprint 2 Sprint 3 Bug CodeX 0 10 20 30 40 50 60 70 1 2 3 4 5 6 7 Delay in Fixing the bug Cost To Correct Software Engineering Economics – Barry Boehm (1981) Build QA Env QA
  • 21. 2. Development: Bottlenecks & bugs: Bottlenecks Frustration Waiting
  • 22. 2. Development: Bottlenecks & bugs: subsets cause bugs
  • 23. 2. Development: Bottlenecks & bugs: subsets cause bugs
  • 24. 3. Provisioning Delays : weeks to months Management DBA System Admin Storage Admin Developers Submit Request Disk Capacity? Approve Request $$ (2 Weeks) Approve Request $$ (1 Week) Request Additional Storage? Provision Capacity File System Configured? Configure LUNS & Build File System Coordinate Replication w/ Infrastructure Re- Parameterize & Configure DB Mount Recovery DB to Specific PIT Begin Work Approve Request $$ (2 Weeks) (3 Days) (3 Days) (2 Days) (3 Days) (3 Days) …….1-2 Weeks of Approvals, Delays, and Provisioning…… 24 Developer Asks for DB Get Access Manager approves DBA Request system Setup DB System Admin Request storage Setup machine Storage Admin Allocate storage (take snapshot)
  • 25. 3. Provisioning Delays : culture of no
  • 26. What We’ve Seen 1. QA: Higher costs more code re-work 2. Development: Bottlenecks & bugs 3. Provisioning: Delays
  • 27. Three Physical Copies Three Virtual Copies
  • 28. Install Delphix on Commodity Intel Hardware Intel hardware
  • 29. Allocate Any Storage to Delphix Allocate Storage Any type
  • 30. One time backup of source database Database Production Instance File systemFile system
  • 31. DxFS (Delphix) Compress Data Database Production Instance Data is compressed typically 1/3 size File system
  • 32. Incremental forever change collection Database Production Instance File system Changes • Collected incrementally forever • Old data purged Time Window File system
  • 33. Typical Architecture Production Instance Development Instance QA Instance UAT Instance Database File systemFile system Database File systemFile system Database File systemFile system Database File systemFile system
  • 37. What We’ve Seen With Delphix 1. QA: Low cost, fast bug detection 2. Development: Parallelized, less bugs 3. Provisioning: Fast, Culture of Yes
  • 38. 1. QA Efficient : Lower cost B u i l d T i m e QA Test 1% of QA time was building environment $.99/$1.00 actual testing vs. setup Build Time QA Test Build
  • 39. 1. QA Fast : bugs found fast and fixed Sprint 1 Sprint 2 Sprint 3 Bug CodeX QA QA Build QA Env Q A Build QA Env Q A Sprint 1 Sprint 2 Sprint 3 Bug Cod e X
  • 40. 2. Development : Parallelize gif by Steve Karam
  • 42. 3. Provisioning: Fast, Efficient, Self Service, Culture of Yes!
  • 43. What We’ve Seen With Delphix 1. QA: Low cost, fast bug detection 2. Development: Parallelized, less bugs 3. Provisioning: Fast, Culture of Yes But … How do we manage database changes as code changes and automate deployment of changes?
  • 44. Dealing with Risk  Smaller and more focused changes are easier to manage (Agile…)  Automation of repeating tasks lowers risk of (human) error  Development and Operations should work in synergy (DevOps)
  • 45. 45 • More rapid changes • Reacting quickly to market needs • Getting ahead of competition • Fewer changes backed out • Better collaboration • Fewer defects • Ultimately better service • Happy customers • Profitability Why Continuous Delivery?
  • 46. 46 • Team and process • Version everything • Automate your tests • Fix it, properly (no out of process changes!) • Automate your deployments Database Continuous Delivery?
  • 48. 48 • The database holds your essential information • Changes can impact the entire system • Need to be synchronized with other changes • Often overlooked Database is a Key Component
  • 49. 49 • There is more to a database than SQL scripts – Schema structure – Code – Content and meta-content – Internal dependencies – … • Ensure that changes are not made without authorization • Ensure no out-of-process changes Reaching Inside the Database
  • 50. 50 • Old adage but true • The database is often neglected and therefore can become the weakest link • Essential from a compliance point of view • Should be the strongest link The Weakest Link In a Chain
  • 51. 51 Challenges… Un-coordinated Process Silos in development Deployment delays… Out-of-Process Changes Errors in production Lack of confidence in automation Code overrides Different methodologies Lack of history Missing changes wrong versions
  • 52. 52 Two isolated Processes Version Control Process Development Process Check-Out Script Modify Script Get updated Script from DB Check-In Script Compile Script in DB Debug Script in DB ? ? ? ? A A’
  • 53. 53 X 1.11.1.11.11.21.31.41.51.61.7 Build Once Deploy Many Int QA Stage Prod Database Deploy Script Environment Re-Base (due to defects) Dev Dev Dev Model 1.1 1.2 1.2 1.3 1.3 1.4 1.4 1.5 1.5 1.6 1.6 1.7 1.11.11.41.7 1.1 1.2 1.2 1.3 1.3 1.4 1.4 1.5 1.5 1.6 1.6 1.7 1.1 1.2 1.2 1.3 1.3 1.4 1.4 1.5 1.5 1.6 1.6 1.7 Out of Process Change X X X X X ? 1.1.1 X
  • 54. 54 Dealing with challenges… Coordinated ProcessTraceability Start in the Beginning No Out-of-Process Changes Impact Analysis Automation Task Based Development Well Defined Processes
  • 55. What is DBmaestro TeamWork • Database Enforced Change Management + Database version control + Plugs into the ALM (change request, tickets & work items) + Database merge & change impact analysis • DevOps Solution for databases + Baseline aware deployment automation, rollback & recovery + Plugs into release management & Continuous Delivery
  • 56. How? • Database version control – Enforced Check Out/In – Labels – Rollback/Undo – Audit trail reports • Database impact analysis – Utilizes version control repository information – 3-way analysis • Database deployment automation – API – Baselines – Conflict resolution – Customized business logic
  • 57. 57 Version Control - One Enforced Process
  • 58. 58 1.11.21.31.41.51.61.7 Build & Deploy On Demand * Int QA Stage Prod Database Deploy Script Environment* Execute the same script being executed at the Stage environment Re-Base (due to defects) Dev Dev Dev Model 1.1 1.2 1.2 1.3 1.3 1.4 1.4 1.5 1.5 1.6 1.6 1.7 1.1 1.4 1.4 1.7 1.1.1 1.7 1.1 1.1 1.11.41.7 File Based Version Control Out of Process Change 1.1.11.7 1.1.11.7
  • 59. Safety Net For Deployment Automation Source vs. Target Action = No Action ≠ ? Source vs. Baseline Target vs. Baseline Action = = No Action ≠ = Deploy Changes = ≠ Protect Target ≠ ≠ Merge Changes You do not have all of the information With Baselines and 3 way analysis the unknown is now known Simple Compare & Sync Baseline aware Deployment An index exists in Target (Production) but not in source (QA). What should we do? Drop the index or not?
  • 60. Benefits - Development • Database change repository • Follow SCM best practices (Check-Out/Check-In) • All changes are documented • Manage who can do what, where, when & why
  • 61. Benefits - Operations • Integrated deployment engine • Business level audit • Roles & responsibilities enforcement
  • 62. Benefits - Management • Complete visibility into changes in progress • Management reports, Audit reports • No silos
  • 63. +
  • 64. Time Window Instance Instance Instance Virtual DatabaseDev1 Dev2 Trunk Virtual Database Virtual Database Developer 1 modify Developer 2 modify DB VC
  • 66. Safe and Efficient Work Process • Clone relevant number of virtual copies of the Trunk – Dev Team 1-n, Developer a-z, Hot fix environment, Etc… – Work in parallel, save time • Rely on enforced change process – Know exactly who did what, when and why • Deployment automation – Save time, mitigate risk – Continuous integration, Continuous Delivery • Deal with conflicts & merge them into the Trunk • Test before deploy to production – Pre-prod clone
  • 67. Q&A Kyle Hailey @kylehhailey Delphix: delphix.com Yaniv Yehuda @yanivyehuda DBmaestro: dbmaestro.com Tim O'Brien Gleanster: gleanster.com
  • 68. Thanks! Kyle Hailey @kylehhailey Delphix: delphix.com Yaniv Yehuda @yanivyehuda DBmaestro: dbmaestro.com Tim O'Brien Gleanster: gleanster.com

Editor's Notes

  • #5: Growing almost 300% a year Founded in 2008, launched in 2010 Jedidiah Yueh, President and CEO Founded Avamar in 1999, sold to EMC in 2006, VP Product Mgmt at EMC Avamar: >$1B revenue, 150 Employees: HQ in Menlo Park, SF, Boston, DC, London, NY and Atlanta Growing 250% annually – 130+ customers including 100 Fortune1000 Customers
  • #8: 80% of outages impacting mission-critical services caused by people and process issues thru 2015, with the majority of those outages (50%+) caused by change/configuration/release integration and hand-off issues (Gartner RAS Core Research Note G00208328 Ronni J. Colville, George Spafford [October 27, 2010] – Strategic Planning Assumption(s) “Top Seven Considerations for Configuration Management for Virtual and Cloud )
  • #19: I’m not sure if you have run into these situations at your organization or if you can imagine some of these situations But here are some of the issues we at Delphix are seeing in the industry with the companies we are talking to. Let’s look at the 5 points in more detail
  • #20: We talked to Presbyterian Healthcare And they told us that they spend 96% of their QA cycle time building the QA environment And only 4% actually running the QA suite This happens for every QA suite meaning For every dollar spent on QA there was only 4 cents of actual QA value And that 96% cost is infrastructure time and overhead
  • #21: Because of the time required to set up QA environments The actual QA tests suites lag behind the end of a sprint or code freeze Meaning that the amount of time that goes by after the introduction of a bug in code and before the bug is found increases And the more time that goes by after the introduction of a bug into the code The more dependent code is written on top of the bug Increasing the amount of code rework required after the bug is finally found In his seminal book that some of you may be familiar with, “Software Engineering Economics”, author Barry Boehm Introduce the computer world to the idea that the longer one delays fixing a bug in the software development life cycle The more expensive it is to to fix that bug and these cost rise exponentially as delays increase
  • #22: Not sure if you’ve run into this but I have personally experience the following When I was talking to one development group at Ebay, they Shared a single copy of the production database between the developers on that team. What this sharing of a single copy of production meant, is that whenever a Developer wanted to modified that database, they had to submit their changes to code Review and that code review took 1 to 2 weeks. I don’t know about you, but that kind of delay would stifle my motivation And I have direct experience with the kind of disgruntlement it can cause. When I was last a DBA, all schema changes went through me. It took me about half a day to process schema changes. That delay was too much so it was unilaterally decided by They developers to go to an EAV schema. Or entity attribute value schema Which mean that developers could add new fields without consulting me and without stepping on each others feat. It also mean that SQL code as unreadable and performance was atrocious. Besides creating developer frustration, sharing a database also makes refreshing the data difficult as it takes a while to refresh the full copy And it takes even longer to coordinate a time when everyone stops using the copy to make the refresh All this means is that the copy rarely gets refreshed and the data gets old and unreliable
  • #23: To circumvent the problems of sharing a single copy of production Many shops we talk to create subsets. One company we talked to , RBS spends 50% of time copying databases have to subset because not enough storage subsetting process constantly needs fixing modification Now What happens when developers use subsets -- ****** -----
  • #25: The biggest and most pervasive problem we see is slow build times. In order to set up an database copy for a development environments Requires submitting a request to management who has to review it Then if the request is granted, it is passed to the DBA who has to coordinate with the Sysadmin who has to coordinate with the storage admin. In such a situation it makes sense that copying a large database would take a long time But even when we talk to someone who uses netapp storage snapshots like Electronics Art and Ariba, they said even using storage snapshot sit took days to weeks get a database clone copy due to the coordination between DBA, sys admin and storage admin At many of the customers we talk to provisioning a database clone copy takes weeks or months. One large global bank quotes us as taking typically 6 months to provision a database clone copy environment. Requirements: self service for app teams Requirements: end-to-end automation Metrics: # people, process, time for delivery So far we have talked about the weight of infrastructure on app delivery. Of course, to control and manage that infrastructure, firms layer on a large set of bureaucratic processes, change control, approvals, procurement, governance, etc etc. So the operational and organizational hurdles then create an even bigger drag on IT and app development. Here’s an example from one banking customer. Once the app developer puts in a request for a new development environment, there’s at least a week long wait for management approvals. Then project DBA work with the sysadmin and storage groups for capacity. If more capacity needs to be allocated, it’s 3 more days. If more needs to be purchased, weeks or months. If a copy of production data is needed, the process needs to wait on a production DBA, who might be busy with production issues. Recovering the database to a specific point in time and configuration can also take days. It is very common for two weeks to pass between a developer request and a ready environment. The process can be repeated for multiple environments, for data refreshes, and for integration across multiple systems. With Delphix, turns stop signs into green lights. Provisioning, refresh, rollback, and data integration happen nearly instantly and do not trigger approvals from production systems or require additional storage. That is why KLA is able to deliver 5 times the output from its SAP teams… Without Delphix, it’s impossible for organizations to implement the level of agile processes they desire. The management of data, and the bureaucracy of data management, slows things down too much.
  • #26: Due to the constraints of building clone copy database environments one ends up in the “culture of no” Where developers stop asking for a copy of a production database because the answer is “no” If the developers need to debug an anomaly seen on production or if they need to write a custom module which requires a copy of production they know not to even ask and just give up.
  • #27: I’m not sure if you have run into these situations at your organization or if you can imagine some of these situations But here are some of the issues we at Delphix are seeing in the industry with the companies we are talking to. Let’s look at the 5 points in more detail
  • #28: In the physical database world, 3 clones take up 3x the storage. In the virtual world 3 clones take up 1/3 the storage thanks to block sharing and compression
  • #29: installs an any Intel commodity hardware as a VM supports Oracle 9.2-12c, standard edition, enterprise edition, single instance and RAC on AIX, Sparc, HPUX, LINUX support SQL Server
  • #30: EMC, Netapp, Fujitsu, Or newer flash storage like Violin, Pure Storage, Fusion IO etc Recent benchmark with Delphix and Pure Storage showed that a Flash and Delphix solution is an order of magnitude cheaper than spinning disk and more performant
  • #31: Delphix does a one time only copy of the source database onto Delphix
  • #37: Quote from a customer “Delphix GUI is what Oracle Enterprise Manager would look like if Apple had designed it” Delphix inter face is user friendly, polished and easy to use
  • #39: Presbyterian when from 10 hour builds to 10 minute builds Total Investment in Test Environment: $2M/year 10 QA engineers DBA, storage team dedicated to support testing App, Oracle server, storage, backups Restore load competes with backup jobs Requirements: fast data refresh, rollback Data delivery takes 480 out of 500 minute test cycle (4% value) $.04/$1.00 actual testing vs. setup
  • #41: For example Stubhub went from 5 copies of production in development to 120 Giving each developer their own copy
  • #42: Stubhub estimated a 20% reduction in bugs that made it to production