How Big Data and Hadoop Integrated into BMC ControlM at CARFAX
Download as PPTX, PDF•2 likes•150,446 views
Carfax utilizes a unified workload automation platform to manage over 129,000 processes daily across five data centers, leveraging big data and Hadoop for efficient data processing. With the ability to handle diverse data formats from 34,000 sources, Carfax emphasizes the importance of data integrity and automation in its operations. The integration of Hadoop into existing systems aims to streamline processes and reduce reliance on legacy systems while fostering agile practices within the company.
How Big Data and Hadoop Integrated into BMC ControlM at CARFAX
- 1. © Copyright 4/7/2015 BMC Software, Inc1 Robert Stinnett (@robertstinnett) CARFAX Automation Analyst October 14, 2014 We Came, We Saw, We Processed
- 2. © Copyright 4/7/2015 BMC Software, Inc2 How #BigData and #Hadoop integrated into @BMCControlM at CARFAX CARFAX helps millions of people buy, sell and service their used cars better.
- 3. © Copyright 4/7/2015 BMC Software, Inc3 Agenda 1. Workload Automation at CARFAX 2. Big Data and Hadoop Initiative 3. Reduce, Reuse, Recycle 4. Batches, Unite!
- 4. © Copyright 4/7/2015 BMC Software, Inc4 Support traditional batch, data transfer and SLA/SLE management across various datacenters. Everything under one roof. Integrate with other software packages to create an Enterprise wide workload management system. Capacity on demand. React to the hyper- growing business. Workload Automation at CARFAX 1 Manage Integrate Scale
- 5. © Copyright 4/7/2015 BMC Software, Inc5 • 129,000+ processes a day • 350 batch nodes • 5 Different Datacenters • 1 Unified Workload Management Platform
- 6. © Copyright 4/7/2015 BMC Software, Inc6 Data can be loaded and live less than 30 minutes after receipt 1 record to process today, 1 million records to process tomorrow Data is what makes CARFAX who we are. We can’t afford to “hope we got it right”. It’s All About The Data #Fast #Dynamic #Reliable
- 7. © Copyright 4/7/2015 BMC Software, Inc7 Where Does it Come From? 34,000 Data Sources 13 billion records in our VHDB Data comes in many formats, even pictures and PDFs! CARFAX receives data in “any format, any time, any method”. We process more data in an hour than many businesses do in an entire month.
- 8. © Copyright 4/7/2015 BMC Software, Inc8 Our business is all about data. We’ve been doing Big Data long before it was “cool”. It had to integrate with our currently business processes. It is a vital part of our data services and will provide and consume data from many other applications. We see Hadoop and other Big Data initiatives replacing many of our legacy data processing systems. Big Data & Hadoop 2 #BigData #Integration #Future
- 9. © Copyright 4/7/2015 BMC Software, Inc9 Hadoop is still evolving and maturing. Thousands of pilot projects out there, but very few production installations. We learned we were one of the pioneers when it came to integration.
- 10. © Copyright 4/7/2015 BMC Software, Inc10 Nobody wanted to reinvent the wheel. We didn’t need another scheduling system. Hadoop team wanted to hit he ground running. Reuse what they were already familiar with. Integrate with our existing DevOps practices across CARFAX. No silos! Reuse, Reduce, Recycle 3 #DoMoreWithLess #TimetoMarket #DevOps
- 11. © Copyright 4/7/2015 BMC Software, Inc11 Agile Development + Agile Operations Increased usage of data center automation and configuration management tools #DevOps
- 12. © Copyright 4/7/2015 BMC Software, Inc12 We’ve managed to free up our personnel to do awesome things, and let the automation handle the routine stuff. Integration, more agile operations, compliance and remediation. These are what we see on the horizon. This wasn’t a one person, or one team project. It was a whole company initiative. It has been an amazing journey, yet we’ve only just begun. Batches, Unite! 4 #Today #Tomorrow #AmazingJourney
- 13. © Copyright 4/7/2015 BMC Software, Inc13 Overall Integration Strategy Basic Batch File Transfer Under Control-M Control Database Processing, Web Services Bladelogic, Java Informatica Start “Run this script”SUMMARY Today “Manage our workloads” Future “Make it all just happen” 20102008 20132004 2012 Beyond Hadoop, SAS, ServiceNOW, “Write Our Own”
- 14. © Copyright 4/7/2015 BMC Software, Inc14 #KeyTakeaways 1. Hadoop isn’t a silo, it’s part of other IT processes. 2. Significantly reduce your learning curve by using what you already have. 3. Integrations reduce management headaches. 4. Evolution from batch to workload
- 15. © Copyright 4/7/2015 BMC Software, Inc15 @robertstinnett robertstinnett@ Carfax.com