Data Pipelines and Tools to Integrate with Power BI and Spotfire.pdf
- 1. Data Pipelines and Tools to Integrate with Power BI and Spotfire Steve Dolha & Ryan Mross
- 2. ⬤ Introduction ⬤ Considerations in Software Selection ⬤ The Real Issue is Data Access ⬤ How Can a Data Pipeline Help? ⬤ Where Have We Seen Success? ⬤ Building a Data Pipeline ⬤ Where To Go From Here ⬤ Q&A Agenda
- 3. Steve Dolha ⬤ Chief Data Officer ⬤ 40+ Years Oil & Gas Data Analysis Experience Introductions Ryan Mross ⬤ Senior Data Analytics Specialist ⬤ 12+ Years Oil & Gas Data Analysis Experience
- 4. ⬤ Power BI vs Spotfire vs Tableau ⬤ Virtualized vs Physical Data Layers, and their related software ⬤ Define your Key Criteria ⬤ Cost (initial and ongoing) ⬤ Performance/features ⬤ Extensibility ⬤ Ease of Use (learning curve) ⬤ Ease of Support ⬤ Existing Skillsets Considerations on Software Choices
- 5. Example: Business Intelligence Software Matrix Mapping Extensibility Data Wrangling Statistical Analysis Data Volumes Mobile Access Spotfire Multiple Layers, shape files, custom map layers, bubble charts Full python, and R integration. Mods Data canvas R, matlab, SAS, python Only limited by hardware Static Emails, App (with license), Web dev possible Tableau Bubble and area map layers, WMS layers Dashboard extensions offered Tableau prep Need separate python and R servers. “Chatty" when querying. A couple million rows ideal Tableau app for published content PowerBI Dynamic shading, multiple map sources. Better with paid add-ons External tools available Power-query R and python not supported on server Performance drop-off over 1 million rows Online and offline reports
- 6. ⬤ Regardless of what software/technology you chose, the most important thing is the DATA ⬤ Pick a path and commit to making it meet your business needs ⬤ We are vendor neutral ⬤ Focus more on the ideas and strategies you need to be successful Let's Shift the Narrative
- 7. ⬤ Finding vetted data is a challenge ⬤ Location, quantity, and types of data are complex. ⬤ Everyone has business rules, but they are often not well defined (aggregation, hierarchies, joins, unit conversions) ⬤ Publishing reports is time consuming A Common Issue – Trusted Data is Hard to Find
- 8. ⬤ Cubes enable access to vetted, business focused data, often through excel. ⬤ However, BI tools prefer flat data and there is often inefficiency flattening cube data ⬤ The structures are rigid and require effort to change. They are not flexible or agile. ⬤ It is also very challenging to combine cube data with other sources Where to Start – Cubes and Tabular Models
- 9. ⬤ A data pipeline is a standardized process for sourcing, processing/calculating, and delivering data to end users ⬤ It is not necessarily a data warehouse. It is the processing steps, not the physical location of the final data sets. ⬤ They are meant to be agile, flexible, and dynamic based on ever changing business requirements The Next Step – A Data Pipeline
- 10. ⬤ In your current analytics process, you already perform calculations and transform data. ⬤ A Data Pipeline moves this process further back in the chain ⬤ Lets do this work once, vet it, and share it among all end users! You’re Already Halfway There
- 11. ⬤ TIBCO Data Virtualization @ Hammerhead ⬤ Merged in proprietary data from internal databases with GDC public Data ⬤ Wrote business rules into the views for reusable calculations ⬤ Created dynamic well lists based on set criteria for always up-to-date data sets ⬤ Published standardized and vetted views for end user self service Where Have We Seen Success?
- 12. ⬤ DataBricks & TIBCO Data Virtualization @ Arc ⬤ Used DataBricks to create a standard Data Access Layer (DAL) for all reports to source data from ⬤ DataBricks unifies many disparate data sources into holistic data sets. Used TDV to flatten MDX cubes for Spotfire consumption ⬤ Accessing this data from both Power BI and Spotfire, for unified reports, regardless of BI Tool. Where Have We Seen Success?
- 13. ⬤ Efficient calculation over large data volumes ⬤ Search and understand the data (source, quality, calculations) ⬤ Standardize business rules ⬤ Meld disparate data sources ⬤ The end goal is always Self Service Data Pipeline Business Capabilities
- 14. ⬤ Assess your tool and analytical requirements based on what you're trying to do. Pick a tool ⬤ Build your Data Pipeline from the ground up based on business reporting and analysis requirements ⬤ Model your data (standardize and consolidate) ⬤ Build the pipeline. Pick a technology and implement your solution ⬤ Vet the data ⬤ Publish for end users Building a Data Pipeline
- 15. ⬤ Find your capabilities and requirements. Build a plan. ⬤ Ask for help! There are tons of resources available to help you along this journey ⬤ Be Data and Tool Agnostic. Pick the tool that fits best for you now, with room to grow in the future. ⬤ Be prepared to iterate, you won't get it right the first time. Where Do We Go From Here?
- 16. Questions
- 17. Thank you! Contact us: General / Sales Inquiries Phone: 403.475.2494 Email: [email protected] Head Office Address Suite 520 800 – 5th Avenue SW Calgary, AB T2P 3T6