Retail architecture target
- 1. Platform Architecture for OmniChannel Retail O’Reilly Software Architecture Conference London, October 16, 2017
- 2. Background VP, Architecture and Chief Architect – Target Former Chief Architect & Head of Digital Engineering - Best Buy
- 3. Plan for Today • Context of the omnichannel retail environment • Legacy architecture overview • Platform architecture • Platform examples Ask questions any time.
- 5. Target is Omnichannel • 1,806 stores in the United States • 38 distribution centers in the U.S • 323,000 team members worldwide • Global locations in India • Target.com is the fourth most-visited retail website in the U.S. with more than 26 million unique visitors each month on average
- 6. Omnichannel Retail Commerce • Physical • Online • Mobile • Partner • Voice • Text • Social Customer Interaction • Store Associate • Call Center • Mobile App • Online Chat • Email • Text • Social • Augmented Reality Customer Fulfillment • Shopping bags • Ship to home • Ship to store • In-Store pickup • Car park pickup • Same day delivery • Partner direct ship • Digital goods
- 7. Omnichannel Retail Evolving Rapidly Every step of the retail value chain is being disrupted • Import and Logistics (Direct to Consumer, Crowdsourced Delivery) • Selection & Curation (Online Search and Recommendations) • Trips to Stores (Same Day Delivery, Personal Shoppers) • Online Mega-Retailers (Everything Store) • Online Micro-Retailers (Single Category Specialists) • Checkout process (Self-checkout, Automated checkout)
- 9. 70% of capabilities tied to mainframe 3000+ Applications Stores Digital Supply Chain Marketing Merchandising Product Development Corporate 7000+ RDBMS Operationally siloed Legacy Footprint
- 10. External Cloud Infrastructure Owned Datacenter Leased Datacenter External Cloud Store . . . Store 1800 Stores Ware house Ware house . . . 32 Warehouses Owned Datacenter Digital SiloStore Silo Supply Chain Silo Corporate Silo Each box contains infrastructure onsite Office . . . Office ~40 Offices Worldwide
- 11. Official Data Sources Direct DB Connections Store SiloDigital Silo Marketing Silo Supply Chain Silo Corporate Silo UDS HR Finance Batch ETLs UDS Legacy Architecture UDS = Unofficial Data Sources POS CRM WMS Logist ics Pricing UDSUDSUDSUDSUDSUDSUDSUDS In- Store .COM Mobile X
- 12. Mainframe Dependency Graph Each node is a table, each name is an application that writes to that table
- 14. What is a Platform? A set of technologies that are the fundamental building blocks of custom applications 1. Primitive components 2. A defined surface 3. Extension points A Platform Has:
- 15. What is a Retail Platform? A set of primitive APIs and Services that represent the data, processes and business logic required to complete customer transactions
- 16. Retail Platform Primitives - Examples • Item API • Price API • Inventory API • Location API • Tax API • Customer API • Worker API • Checkout API • Cart API • Restrictions API • Returns API • Address Verification API • Item Movement API • Shift Management API Data Process and Logic Primitive componentsA Platform Has:
- 17. Retail Defined Surface The complete set of retail platform primitives that define the core components of a retailer A defined surfaceA Platform Has:
- 18. Retail Platform Extension Points All retail platform primitives can be extended by users of the platform Extension pointsA Platform Has:
- 19. Worker Price API Retail Platform Architecture Custo mer Locati on Tax OrderPrice ItemPromo Vendor Data that cannot be derived from other data or is generated during common business processes of the company, divided into the logical domain entities of the business Promo API Tax API Custo mer API Location API Order API Item API Worker API Vendor API Core Data Primitive component
- 20. Item-Location-Price API Retail Platform Architecture Pre-joined and cached core data, used to pre-calculate commonly used data patterns and protect core data services from excessive load Customer-Orders API Item-Offer API Item-Location-Price Cache Customer-Orders Cache Item-Offer Cache Core Data Aggregations Primitive component
- 21. Checkout API Retail Platform Architecture The generic components of a business process, presented as an API, that can be used by all channels that execute the business process Cart API Item Setup API Offer Setup API Payment API Fulfillment API Returns API Fundamental Business Process Primitive component
- 22. Restrictions API Retail Platform Architecture The proprietary logic of the business, presented as an API, that is used by all channels that require that business logic Cart Price API Worker Pay API Address Verification API Fundamental Business Logic Primitive component
- 23. Business Logic Retail Platform Architecture Business Process Data Aggregation Worker Price API Custo mer Locati on Tax OrderPrice ItemPromo Vendor Promo API Tax API Custo mer API Location API Order API Item API Worker API Vendor API Platform Logic Data Platform Process Data Platform Cache Fundamental Platform Components Primitive components
- 24. Business Logic Retail Platform Architecture Price API Promo API Tax API Custo mer API Location API Order API Item API Worker API Vendor API Restrictions API Cart Price API Worker Pay API Addr Verify API Checkout API Cart API Item Setup API Offer Setup API Payment API Fulfillment API Returns API Business Process Data Aggregations Item-Location-Price API Customer-Orders API Item-Offer API Core Data Retail Platform Surface A defined surface
- 25. What is a Tenant? • Tenant drives an interaction with an actor • Tenants are coarse grained around a channel (digital, store, supply chain, corporate) • Isolation from other tenants • Can only call services within the tenant, or provided by the platform A user of the platform that builds applications using and extending the platform primitives
- 26. Business Logic Extension Retail Platform Architecture Business Process Extension Service Aggregator Forward Cache Tenant Process Data Tenant Logic Data View Controller User Interface Tenant Specific APIs Tenant Data View Controller User Interface View Controller User Interface Single Tenant Components Extension points
- 27. Retail Platform Architecture Tenant Tenant and Platform Tenant Tenant Business Logic APIs Business Process APIs Data Aggregation APIs Core Data APIs – READ ONLY Worker Custo mer Locati on Tax OrderPrice ItemPromo Vendor Tenants Platform
- 28. Eventual Consistency • The Core Data layer contains the true real-time operational data – source of truth • Core Data is exposed to the Platform as READ ONLY • All other layers are caches • Tenants operate almost exclusively off caches • Tenant applications must be designed for eventual consistency
- 29. Events • All data changes are events • Tenants do not write to Core Data • Tenants emit events • Core Data listeners process events
- 30. Retail Platform Architecture Data Change Event Item-Price API Tenants Platform Merchandising Tenant Item-Price Cache Price Management API Item-Price Cache Price API Price Price Event Price Event Listener Price Update Real Price Event Item-Price Event Listener Item-Price Event Tenant Item-Price Event Listener Price Management UI
- 31. Retail Platform Architecture Tenant Verification Tenants Platform Price API Price Price Event Price Event Listener Price Update Merchandising Tenant Item-Price Cache Price Management APITenant Real Price Event Listener Price Management UI Real Price Event
- 32. Retail Platform Architecture Tenant Complete Platform Tenant Tenant Business Logic APIs Business Process APIs Data Aggregation APIs Core Data APIs – READ ONLY Worker Custo mer Locati on Tax OrderPrice ItemPromo Vendor Tenants Platform DataChangeEvents TenantEvents
- 34. Retail Platform Architecture Data Analytics Tenants Platform Merchandising Tenant Item-Price Cache Price Management Price API Price Price Event Price Event Listener Price Update Price Event Listener Price Data Set Analytics Platform Real Price Event
- 35. Retail Platform Architecture Finance Tenant Stream Processing Real Time Price Change Listener Tenants Platform Merchandising Tenant Item-Price Cache Price Management Price Event Price Change Event Listener Price Data Set Analytics Platform Price Streaming Query Price Change Event X
- 37. Retail Platform Architecture Process Extension Business Process APIs Tenants Platform Digital Tenant Item-Price Cache Target.com Digital Checkout API Extension Inventory Cache Checkout API Store Tenant Item-Price Cache Point of Sale Store Checkout API Extension Inventory Cache 1. List of items 2. Price and promos 3. Fulfillment options 4. Payment type
- 38. Retail Platform Architecture Process Extension Business Process APIs Tenants Platform Checkout API Store Tenant Item-Price Cache Point of Sale Store Checkout API Extension Inventory Cache 1. List of items 2. Price and promos 3. Fulfillment options 4. Payment type 1. N/A 2. Store only promo 3. N/A 4. Chip and Pin
- 39. Retail Platform Architecture Process Extension Business Process APIs Tenants Platform Checkout API Digital Tenant Item-Price Cache Target.com Digital Checkout API Extension Inventory Cache 1. List of items 2. Price and promos 3. Fulfillment options 4. Payment type 1. Retrieve prior cart 2. Digital coupon 3. Address or Pickup? 4. Paypal, Bitcoin
- 41. Generic Platform Architecture Tenant Tenant Services Tenant View Controllers Tenants Platform DataChangeEvents TenantEvents No Tenant to Tenant Calls
- 42. Retail Platform Architecture All Transactions are Events Tenants Platform Merchandising Tenant Item-Price Cache Price Management Price API Price Price Event Price Event Listener Price Update OK
- 43. Retail Platform Architecture Events are Open to All Tenants Platform Merchandising Tenant Item-Price Cache Price Management Price API Price Price Event Price Event Listener Price Update Price Event Price Event Listener Price Data Set Analytics Platform OK
- 44. Retail Platform Architecture Eventually Consistent Tenants Platform Merchandising Tenant Item-Price Cache Price Management Price API Price Price Event Price Event Listener Price Update Price Event Tenant Price Event Listener 1 1
- 45. Platform Operating Principles • No Tenant to Tenant calls • All transactions are Events • Events are (almost) completely open to all • All caching layers are eventually consistent
- 46. Scalability Principles • Protect Core Data services • Platform Aggregations and Tenant caches • Throttle events to Core Data layer • Asynchronous writes • Serve majority of traffic from tenant layer • Distribute data to edge • All caching layers are eventually consistent
- 47. Handling Failure
- 48. Platform Micro-Failures • Individual platform service failure • Tenant to Platform network failure Tenant decides how to handle failure
- 49. Retail Platform Micro-failure - Read Tenants Platform Merchandising Tenant Item-Price Cache Price Management Price API Price 503 Price Aggregator Use last cached value Indicate in UI price is old
- 50. Platform Macro-Failures • Platform down • Network failures • DDOS • Exceeded capacity • Excessive latency Tenant decides how to handle failure
- 51. Retail Platform Macro-failure – Platform Network Business Process APIs Platform Digital Tenant Item-Price Cache Target.com Digital Checkout API Extension Inventory Cache Checkout API 1. Checkout down 2. Browse & Search OK 3. Store Locator OK Digital Browse & Search 503 XStore Locator OK OK Cloud Data Center Location Cache
- 53. Architecture Governance Considerations • Every system has a context: Platform or Tenant • Every system has a defined scope: Data, Process, Logic, Aggregation, UI • Tenants are decoupled from other tenants & the platform • Context decides amount of enterprise governance
- 54. A DSL for Architecture • Establish an architecture and engineering language
- 55. Retail Platform Architecture Tenant Tenant Governance Tenant Tenant Tenants Isolated impact Open to new technologies Build and deploy quickly Tenants make many technology decisions
- 56. Retail Platform Architecture Platform Governance Business Logic APIs Business Process APIs Data Aggregation APIs Core Data APIs – READ ONLY Worker Custo mer Locati on Tax OrderPrice ItemPromo Vendor Platform DataChangeEvents TenantEvents Wide impact Strong technology standards Durable API contracts Enterprise decision process
- 57. Architecture Considerations • Adopt new technologies in Tenants • Innovate and experiment in Tenants • Learn and evaluate for graduation to Platform Technology Renewal
- 58. Use Conway’s Law Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure. ~ M. Conway http://www.melconway.com/Home/Conways_Law.html This is exactly what we want!
- 59. O’Reilly Software Architecture Conference • https://www.safaribooksonline.com/library/view/oreilly-software- architecture/9781491985274/ • But it’s not out there yet, still waiting approval from PR.
- 60. Questions
Editor's Notes
- #3: Lead Enterprise Architecture at both of these retailers. Omnichannel retail language began around 2011.
- #6: Target has 1,806 stores in the United States 38 distribution centers in the United States 323,000 team members worldwide 70 + billion in revenue online business at target.com global locations in India To date, Target.com is the fourth most-visited retail Website in the U.S. with more than 26 million unique visitors each month on average.
- #7: As an omnichannel retailer you have to do all these things well, and make them look seamless and integrated. Must excel at everything
- #12: Official Datasources are incomplete, because of the potential for writes not to make it back to the official datastore, teams actually have to source from the other UDS as well to get a complete picture.
- #17: Not a complete list. You might question why we have Worker API and Shift Mangement API. To complete a customer transaction in a store, it’s necessary to have a worker present.
- #20: The only location of the operational data of the company, all writes must pass through these systems before it is reality in the company.
- #25: We put it all together and we get a defined surface. Extension points will come later.
- #29: Not a CAP Theorem discussion. If a Tenant can’t write to fundamental data, than what do they do? Events next slide.
- #30: Not a CAP Theorem discussion.
- #31: Price change event starts in Merchandising tenant. Seems complicated but each component is small and isolated. All event from tenants are read by fundamental data listeners and decide whether to persist that event. Not until that event is persisted should the Platform reflect the change. Once a Fundamental data system produces event, the Platform quickly cascades that to all interested systems.
- #32: Merchandising tenant wants to know when Price is written. I may want to notify the UI that the Price Event was taken. In that case I would a add a Tenant Price Event Listener to see when my generated event caused the fundamental data system to change.
- #34: So we’ve described the operational system. Now let’s add in analytics and business intelligence
- #35: Seems complicated but each component is small and isolated. All event from tenants are read by fundamental data listeners and decide whether to persist that event. Not until that event is persisted should the Platform reflect the change. Once a Fundamental data system produces event, the Platform quickly cascades that to all interested systems.
- #36: Add a streaming event in another tenant.
- #37: How does a Tenant make a Process or Logic extension
- #38: A Checkout API consists of four things
- #39: A Checkout API consists of four things
- #40: Digital tenant may have separate coupons, fulfillment types or payment types
- #42: No Tenant to Tenant calls No East West
- #43: No writes to fundamental data, all transactions are events. Seems complicated but each component is small and isolated. All event from tenants are read by fundamental data listeners and decide whether to persist that event. Not until that event is persisted should the Platform reflect the change. Once a Fundamental data system produces event, the Platform quickly cascades that to all interested systems.
- #44: Events are open to everyone, even external to the platform
- #45: Eventually consistent
- #50: Fundamental Data read fails, use last cached value Tenant decides to allow expired cache reads
- #52: What if the entire platform is down due to network failure? Failure levels, Checkout is down but digital search and store locator is up. Omnichannel we can still help our guests find stores and inventory, but no checkout Business decision not to take orders while Platform API is down, we could cache locally and update the platform later
- #56: Tenant’s have a lot of freedom to innovate and choose new technology But if they make bad choices in technology or development, it’s all on them There are a few enterprise standards, but tenants can choose much of their stack. EX: RDBMS, OS, expensive items Creates a cycle of technology renewal and innovation.
- #57: Platform is governed for stability Reuse similar architectures and patterns across applications Evolves more slowly, expect APIs to last at least 1 year