Dsc 2021 presentation_radovan_bacovic

Editor's Notes

• #2: Abstract: Providing a walkthrough over the process of creating, improving, and scaling the Data product using a modern DevOps stack. Exposing details of the use case how we embrace the open-source philosophy to help us provide faster time to market. Will discuss how to use the advantage of the internal product to make us more agile in the daily job of creating great data products. Story in the beginning: connect DSC with GitLab.com, connect the audience with the intro. Tell the story about: We are using our product we are using to build our product to build our data product.
• #3: Long story short, what we are doing. But, what is wrong with this (dramatize)?
• #4: This is ultimately confusing, unclear and unacceptable to say - …. and it is true! How?
• #5: Let me tell you the story Why I am here, whom I represent Want to give you the context how to build the data product/services Why to iterate fast Why to use the best you have around you (great DevOps tool) Why to give to the community Why to be transparent Why to focus on results
• #6: About the company - couple of words to share GitLab is an open core company which develops software for the software development lifecycle
• #7: “Everyone Can Contribute” – GitLab encourages its wider community to contribute back to the product through code and feedback.
• #8: Provide the context about values
• #9: GitLab is The DevOps Platform delivered as a single application to help you iterate faster and innovate together.
• #10: If you are curious
• #11: If you are lazy
• #12: 4 crucial numbers to expose - numbers are confirmed
• #13: We are all-remote company.
• #14: 1,300+ team members in more than 65 countries and regions around the world as of July 31, 2021.
• #15: GitLab has an estimated 30 million users (both Paid and Free) from startups to global enterprises.
• #16: GitLab releases a new version of the product on the 22nd of every month. No delays, no excuses, no exceptions.
• #17: GitLab’s handbook has over 2,000 web pages of text.
• #18: Move the audience to the use case part
• #19: Data team mission: Deliver Results That Matter With Trusted and Scalable Data Solutions
• #20: Platform state Team forming life cycle Data volume is not so high at the moment Descriptive Analysis: Understanding what happened Diagnostic Analysis: Understanding what made it happen Predictive Analysis: Understanding what is likely to happen Prescriptive Analysis: Understanding what action to perform to make it happen
• #21: Platform state Team forming life cycle Data volume is not so high at the moment Descriptive Analysis: Understanding what happened Diagnostic Analysis: Understanding what made it happen Predictive Analysis: Understanding what is likely to happen Prescriptive Analysis: Understanding what action to perform to make it happen
• #22: Data team maturity Gap between Data team maturity VS GitLab.com maturity Data volume is growing - how to scale the entire ecosystem in a efficient way? Orchestrate, harmonize and synchronize multiple teams and stakeholders. How to increase visibility in the company “I don't even know you or your team exists”? Dramatize here: what to do and how to act - how to overcome, how to be “best of the breed”
• #23: Data team maturity Gap between Data team maturity VS GitLab.com maturity Data volume is growing - how to scale the entire ecosystem in a efficient way? Orchestrate, harmonize and synchronize multiple teams and stakeholders. How to increase visibility in the company “I don't even know you or your team exists”? Dramatize here: what to do and how to act - how to overcome, how to be “best of the breed”
• #24: Data team maturity Gap between Data team maturity VS GitLab.com maturity Data volume is growing - how to scale the entire ecosystem in a efficient way? Orchestrate, harmonize and synchronize multiple teams and stakeholders. How to increase visibility in the company “I don't even know you or your team exists”? Dramatize here: what to do and how to act - how to overcome, how to be “best of the breed”
• #25: Data team maturity Gap between Data team maturity VS GitLab.com maturity Data volume is growing - how to scale the entire ecosystem in a efficient way? Orchestrate, harmonize and synchronize multiple teams and stakeholders. How to increase visibility in the company “I don't even know you or your team exists”? Dramatize here: what to do and how to act - how to overcome, how to be “best of the breed”
• #26: Data team maturity Gap between Data team maturity VS GitLab.com maturity Data volume is growing - how to scale the entire ecosystem in a efficient way? Orchestrate, harmonize and synchronize multiple teams and stakeholders. How to increase visibility in the company “I don't even know you or your team exists”? Dramatize here: what to do and how to act - how to overcome, how to be “best of the breed”
• #27: Data team maturity Gap between Data team maturity VS GitLab.com maturity Data volume is growing - how to scale the entire ecosystem in a efficient way? Orchestrate, harmonize and synchronize multiple teams and stakeholders. How to increase visibility in the company “I don't even know you or your team exists”? Dramatize here: what to do and how to act - how to overcome, how to be “best of the breed”
• #28: Data team maturity Gap between Data team maturity VS GitLab.com maturity Data volume is growing - how to scale the entire ecosystem in a efficient way? Orchestrate, harmonize and synchronize multiple teams and stakeholders. How to increase visibility in the company “I don't even know you or your team exists”? Dramatize here: what to do and how to act - how to overcome, how to be “best of the breed”
• #29: Open source: create to and for community get the feedback and contribution from wider community
• #30: dogfooding Explain what the dogfooding is
• #31: Dogfooding Gitlab git flow - push frequently Issue -> MR Embrace and cultivate Devops culture Iterate fast, move fast, release frequently
• #32: Dogfooding Gitlab git flow - push frequently Issue -> MR Embrace and cultivate Devops culture Iterate fast, move fast, release frequently
• #33: Dogfooding Gitlab git flow - push frequently Issue -> MR Embrace and cultivate Devops culture Iterate fast, move fast, release frequently
• #34: Dogfooding Gitlab git flow - push frequently Issue -> MR Embrace and cultivate Devops culture Iterate fast, move fast, release frequently
• #35: People Project management Hiring process as its best Stable counterpart Agile and resilient organization - ability to move quick and transform as needed
• #36: Tech stack: Snowflake Python Stitch, Fivetran Cloud (AWS, GCP) Sisense Mention 3 rising stars as it is not too much in use here (Serbia): DBT Meltano Singer.io (taps and targets)
• #37: Tech stack: Snowflake Airflow Python Stitch, Fivetran Cloud (AWS, GCP) Sisense Mention 3 rising stars as it is not too much in use here (Serbia): DBT Meltano Singer.io (taps and targets)
• #38: Tech stack: DBT Snowflake Python Stitch, Fivetran Cloud (AWS, GCP) Sisense Meltano Singer.io (taps and targets)
• #39: Tech stack: DBT Snowflake Python Stitch, Fivetran Cloud (AWS, GCP) Sisense Meltano Singer.io (taps and targets)
• #40: Tech stack: DBT Snowflake Python Stitch, Fivetran Cloud (AWS, GCP) Sisense Meltano Singer.io (taps and targets)
• #41: Testing and checkpoint pipelines NPMI data checks Testing pipelines Code quality checks pipeline DBT: job run test run Python pipeline code quality and linters (mypy, black) test coverage (pytest) Snowflake pipeline Security and compliance pipeline
• #42: Fastly created, short lived and isolated test environment Snowflake Zero Copy Clone Destroyed each week Created on demand - per issue
• #43: Various type of well defined tools/frameworks Embrace open source tools (singer, meltano…) and frameworksEDW - Enterprise data warehouse
• #44: Family of Data roles, well defined, well described: Data Analyst Data Scientist Analytics Engineer Data Engineer Manager, Data Director, Data
• #45: Family of Data roles, well defined, well described: Data Analyst Data Scientist Analytics Engineer Data Engineer Manager, Data Director, Data
• #46: Family of Data roles, well defined, well described: Data Analyst Data Scientist Analytics Engineer Data Engineer Manager, Data Director, Data
• #47: Family of Data roles, well defined, well described: Data Analyst Data Scientist Analytics Engineer Data Engineer Manager, Data Director, Data
• #48: Everything starts with issue Dogfooding Include more and more stages from www-gitlab-com into our development/testing/release process Trusted Data Model What is your single source of truth? Reliable metrics Keep it simple
• #49: Everything starts with issue Dogfooding Include more and more stages from www-gitlab-com into our development/testing/release process Trusted Data Model What is your single source of truth? Reliable metrics Keep it simple
• #50: ELTD - D stands for document (Handbook first company as SSOT)
• #51: Deliver Results That Matter With Trusted and Scalable Data Solutions
• #52: Deliver Results That Matter With Trusted and Scalable Data Solutions
• #53: Building blocks of EDW
• #54: Move the audience to the conclusion
• #55: Let me conclude and expose the key takeaways for today's presentation. Before moving to the last point, want to know do you have any questions?
• #56: Dramatize: We are not done yet - we consider our work in the persistent state of iteration (look for the value: iteration) Support from the core product gave us wind in the back to embrace, learn and apply DevOps culture in the team and tons of flexibility in order to experiment, change and adopt various solutions in uncertain environments. Go for the “boring” solution (someone should call it “best praxis”) Be strict and disciplined to keep everything in the source control Be even more strict when it comes to test your code - and automate things you can Be cruel and insensitive when it comes to your codebase quality checks - and automate them Make your documentation first class citizen and a single source of truth (SSOT). Iterate and move fast - and try not to get lost the bigger picture.
• #57: Dramatize: We are not done yet - we consider our work in the persistent state of iteration (look for the value: iteration) Support from the core product gave us wind in the back to embrace, learn and apply DevOps culture in the team and tons of flexibility in order to experiment, change and adopt various solutions in uncertain environments. Go for the “boring” solution (someone should call it “best praxis”) Be strict and disciplined to keep everything in the source control Be even more strict when it comes to test your code - and automate things you can Be cruel and insensitive when it comes to your codebase quality checks - and automate them Make your documentation first class citizen and a single source of truth (SSOT). Iterate and move fast - and try not to get lost the bigger picture.
• #58: Dramatize: We are not done yet - we consider our work in the persistent state of iteration (look for the value: iteration) Support from the core product gave us wind in the back to embrace, learn and apply DevOps culture in the team and tons of flexibility in order to experiment, change and adopt various solutions in uncertain environments. Go for the “boring” solution (someone should call it “best praxis”) Be strict and disciplined to keep everything in the source control Be even more strict when it comes to test your code - and automate things you can Be cruel and insensitive when it comes to your codebase quality checks - and automate them Make your documentation first class citizen and a single source of truth (SSOT). Iterate and move fast - and try not to get lost the bigger picture.
• #59: Dramatize: We are not done yet - we consider our work in the persistent state of iteration (look for the value: iteration) Support from the core product gave us wind in the back to embrace, learn and apply DevOps culture in the team and tons of flexibility in order to experiment, change and adopt various solutions in uncertain environments. Go for the “boring” solution (someone should call it “best praxis”) Be strict and disciplined to keep everything in the source control Be even more strict when it comes to test your code - and automate things you can Be cruel and insensitive when it comes to your codebase quality checks - and automate them Make your documentation first class citizen and a single source of truth (SSOT). Iterate and move fast - and try not to get lost the bigger picture.
• #60: Dramatize: We are not done yet - we consider our work in the persistent state of iteration (look for the value: iteration) Support from the core product gave us wind in the back to embrace, learn and apply DevOps culture in the team and tons of flexibility in order to experiment, change and adopt various solutions in uncertain environments. Go for the “boring” solution (someone should call it “best praxis”) Be strict and disciplined to keep everything in the source control Be even more strict when it comes to test your code - and automate things you can Be cruel and insensitive when it comes to your codebase quality checks - and automate them Make your documentation first class citizen and a single source of truth (SSOT). Iterate and move fast - and try not to get lost the bigger picture.
• #61: Dramatize: We are not done yet - we consider our work in the persistent state of iteration (look for the value: iteration) Support from the core product gave us wind in the back to embrace, learn and apply DevOps culture in the team and tons of flexibility in order to experiment, change and adopt various solutions in uncertain environments. Go for the “boring” solution (someone should call it “best praxis”) Be strict and disciplined to keep everything in the source control Be even more strict when it comes to test your code - and automate things you can Be cruel and insensitive when it comes to your codebase quality checks - and automate them Make your documentation first class citizen and a single source of truth (SSOT). Iterate and move fast - and try not to get lost the bigger picture.
• #62: Dramatize: We are not done yet - we consider our work in the persistent state of iteration (look for the value: iteration) Support from the core product gave us wind in the back to embrace, learn and apply DevOps culture in the team and tons of flexibility in order to experiment, change and adopt various solutions in uncertain environments. Go for the “boring” solution (someone should call it “best praxis”) Be strict and disciplined to keep everything in the source control Be even more strict when it comes to test your code - and automate things you can Be cruel and insensitive when it comes to your codebase quality checks - and automate them Make your documentation first class citizen and a single source of truth (SSOT). Iterate and move fast - and try not to get lost the bigger picture.
• #63: Dramatize: We are not done yet - we consider our work in the persistent state of iteration (look for the value: iteration) Support from the core product gave us wind in the back to embrace, learn and apply DevOps culture in the team and tons of flexibility in order to experiment, change and adopt various solutions in uncertain environments. Go for the “boring” solution (someone should call it “best praxis”) Be strict and disciplined to keep everything in the source control Be even more strict when it comes to test your code - and automate things you can Be cruel and insensitive when it comes to your codebase quality checks - and automate them Make your documentation first class citizen and a single source of truth (SSOT). Iterate and move fast - and try not to get lost the bigger picture.
• #64: Dramatize: We are not done yet - we consider our work in the persistent state of iteration (look for the value: iteration) Support from the core product gave us wind in the back to embrace, learn and apply DevOps culture in the team and tons of flexibility in order to experiment, change and adopt various solutions in uncertain environments. Go for the “boring” solution (someone should call it “best praxis”) Be strict and disciplined to keep everything in the source control Be even more strict when it comes to test your code - and automate things you can Be cruel and insensitive when it comes to your codebase quality checks - and automate them Make your documentation first class citizen and a single source of truth (SSOT). Iterate and move fast - and try not to get lost the bigger picture.
• #65: Long story short, what we are doing. But, what is wrong with this (dramatize)?
• #66: Give me one change to repeat our mission statement
• #67: For any additional questions or info needed, looking forward to hearing from you. Do not hesitate to contact me with any questions.
• #68: And of course…. thank you. See you around. Adios amigos!