#ATAGTR2019 Presentation "Top 10 quality engineering best practices to achieve quality-at-speed" By Vinod Sundararaju Antony, Senthilkumar Thirumalaisamy, Santhosh Kumar Vasudevan & Ameet Pradhan
0 likes•73 views
This document outlines 10 best practices for quality engineering to achieve quality at speed, including establishing an "automate first" mindset, impact-based testing, eliminating dependencies through virtualization and containerization, enabling continuous integration/delivery, improving security with DevSecOps, continuous performance testing, leveraging cloud capabilities to scale automation, shifting to intelligent DevOps, behavior driven development, and leveraging artificial intelligence. It provides examples and strategies for implementing each best practice.
#ATAGTR2019 Presentation "Top 10 quality engineering best practices to achieve quality-at-speed" By Vinod Sundararaju Antony, Senthilkumar Thirumalaisamy, Santhosh Kumar Vasudevan & Ameet Pradhan
- 1. #ATAGTR2019 TOP 10 QUALITY ENGINEERING BEST PRACTICES TO ACHIEVE QUALITY-AT-SPEED By : Vinod Sundararaju Antony, Senthilkumar Thirumalaisamy, Santhosh Kumar Vasudevan & Ameet Pradhan 14th 15th Dec 2019
- 2. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 2 TOP 10 QUALITY ENGINEERING BEST PRACTICES TO ACHIEVE QUALITY-AT-SPEED The Digital era will see a proliferation of intelligent products, transaction channels and influencers available at the fingertips of a digital customer. • Need to deliver improved customer experience, at speed and scale with higher efficiency • Quality, Customer Experience & Trust (Data privacy and Security) are therefore becoming more important than ever • It has become paramount for enterprises to assure quality across the application lifecycle with better test processes, tools, people and governance. This paper provides : • A guideline to transform traditional testing approaches into best-in-class QA for the Digital ICE (intelligent, continuous and early) age. • Describes business case studies with best practices that illustrate testing strategies for functional, automation teams, DevOps engineers, production support teams and non-functional teams to deliver quality at speed.
- 3. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 3 Establish an “Automate First” Mind Set Strategy • Prioritize Automation – Risk based approach • Invert Automation Pyramid: focus on unit level automation • Early Automation : write components during Dev • Use test design pattern and principles : Page Object, Screenplay, SOLID,GRASP • Static code analysis • Build reusable components • Test design automation Framework • Build a Robust Automation Framework • Data driven • Page Object Model • Reporting • API Libraries • Integrates with Test management ,Version control and CI tools • Supports Web, Mobile, API, SAP, Mainframe application, etc. Collaboration • Automation - A team effort • Developer to follow coding standards in UI • Support from Dev for Automated build and deployments - Enables faster QA Infrastructure • Scalable: Identify un- interrupted automation execution platform on VMs - Dockers / Kubernetes • AWS – Helps configure test run machines based on the requirements , full admin access Reporting • Intelligent ROI capture : Automated measure or capture of ROI • Dashboard - Create online real time QA reports and dashboard 01 02 03 04 05 1
- 4. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 4 Automated Impact Based Testing Traditional Approach Run entire tests for even “one- line change” of code Change Based Testing • Analyze build dependencies: Baseline code Vs new build code • Identifying set of files/code impacted and their respective tests to be executed Inside-Out Approach • Begin with details about the situation and identify risks associated with them • Study a product and repeatedly check “What can go wrong here?” Outside-In Approach • Identify a predefined list of risks and determine whether they apply to the situation • Organize RBT based on Risk watch list, risk / task matrix and component risk matrix. a) Change based testing (CBT) b) Heuristic Risk-Based Testing Module-based Approach Manual Identification of tests based on impacted modules Next-Gen CBT • Predictive test selection - model predicts probability of each test failing for a newly proposed code change • ML Model leverages Gradient-boosted decision-tree algorithm • Jira to catalogue and prioritize risks • Jubula to mitigate them • Mylyn to mix everything (Risk Based Interface)Tools • VectorCAST • Parasoft DTP • Coverity Test Advisor etc.Tools 2
- 5. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 5 Eliminate dependencies through Virtualization and Containerization 01 02 05 04 03 When to Adapt? • Components are not available or are restricted • Critical component testing impacts business Benefits : Enables TDD , “Shift Left” QA: replicate/fix defects faster, continuous testing , enables parallel testing: test the pre-prod/UAT in parallel Containerization • AWS - Amazon ECS, Amazon EKS (Amazon ECS for Kubernetes) and AWS Fargate • Docker Swarm Return on Investment • Dev/QA Staff Productivity savings, • Reduced costs through shifting defects left, • Reduced Infrastructure costs and • Time to market savings (days saved) • Capital and Operational expenditure reduction Ways of creating/generating SV • Recording Traffic : Record Process Model • Ground up from WSDL, REST • Using Request/Response (RR) pairs for Web Services Microservices Testing • Require heavy orchestration for service virtualization • “Mountebank” approach : Open source, lightweight tool , provides cross-platform multi-protocol test doubles 3 Virtualize anything, test anything • Mainframe , Platforms (Oracle, Siebel, SAP) • SOA protocols (SOAP, HTTP, REST, XML) • Banking message format (SWIFT) • Teleconference & SMS gateway providers (WebEx, Skype), Card processing services (PayPal)
- 6. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 6 Enable Continuous Everything DevOps Continuous Integration (CI) Continuous Delivery (CD) Continuous Deployment (CI/CD) Best Practices : • Gated check-in build system • Setting up a change control board • Automatically identify issues in every commit and pull requests • Speed up the test automation phase Manage Source Code Repository for ease of Release Management Workflow • Centralized Workflow • Feature Branch Workflow • GitFlow Workflow Implement SCM in Test for better CI Collaboration through ChatOps Continuous Monitoring in DevOps Continuous Inspection Infrastructure as Code (IaC) – Runtime Execution Infrastructure Monitoring • Visualize events and get alerts in real time • Monitor compute resources, storage, network and measure inventory usage, and health of an infrastructure's resources Application Performance Monitoring (APM) • Helps find performance bottlenecks • Provides usage and performance of application dependency systems like web services, databases, caching, etc. • Code Quality Check using Static Code Analysis • Management of code quality of new, changed and added code before anything else – Water leak paradigm • Infrastructure via source control, thus providing a detailed audit trail for changes • Enables collaboration around infrastructure configuration and provisioning, most notably between dev and ops • Connects people, process and automation in a transparent flow • Helps collaborate and control DevOps pipeline through BOTs 4
- 7. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 7 Continuous Everything: Improve Security through DevSecOps DevSecOps automatically detect alerts on vulnerable components, outdated libraries, and license compliance issues in code as defined by OWASP guidelines. It enables Some of the best practices to be improve security in DevOps model is represented below: Detection and solution for vulnerable open source components Generate comprehensive open source inventory reports per project or build Enforcing open source license compliance Identification outdated open source libraries with recommendations Embrace a DevSecOps model Embedded Governance and Cybersecurity functions: • Identity and access management (IAM), privilege management • Firewalling / unified threat management, code review • configuration management, and vulnerability management Control, monitor, and audit access with privileged access management Deploy a secured jump server • Multi-factor authentication • Adaptive access authorization • Use session monitoring to provide oversight 4
- 8. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 8 Continuous Everything : Continuous Performance Test Continuous performance testing involves: Developing and deploying low volume load tests to capture performance metrics of critical application user flows, API end points and database queries of the systems without applying stress. Key focus is to capture performance metrics such as latency, throughput, bandwidth, Time to First Byte (TTFB), application http response codes (404, 500), CPU, Memory, Disk I/O and others • Benefits continuous Performance Testing in CI/CD • Early detection of performance issues or degradation trends. • Quick feedback to developers on check-in – resolution of performance issues or bottlenecks in the code early in SDLC • Help identify latency issues, over the time performance degradations, location specific latency issues • Very helpful for cloud hosted applications and services to measure long term stability and site performance on regular basis. Pre requisites • List of application’s critical pages, end to end user flow scenarios, API end points and database queries • Measure and analyze on regular basis as automated performance test • Establish baselines performance • Execute performance tests on daily basis in production environment along with real users’ traffic to capture performance metrics. 4
- 9. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 9 Scale Automation : Leverage the Power of Cloud Identify un-interrupted execution platform for running automation test scripts in multiple Virtual Machines (on-demand cloud based infrastructure) 5 Planned In-Progress Completed GIT Repository Continuous Integration Automation Reports Defects Test Planning & Feasibility Requirement Refinement Failure Analysis Execution Environments Test Data (Application URLs / Sites) Parallel execution across selected markets Automation Framework JVM Libraries / Reusable methods Automation Tests / Reusable Tests APP Select combination of sites/markets & QA checks Docker is lightweight container helps in running tests distributed with NO minimal costs compared to traditional distributed execution platforms AWS : Helps to configure test run machines based on the requirements. Full administrator access with any software, browsers, tools and access with different operating systems
- 10. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 10 Shift from DevOps to Intelligent DevOps Intelligent DevOps to make data-driven decisions that help them improve software delivery Automation Analysis • Employ algorithms and intelligence to extract meaning from data, execute judgment-based activities using diagnostic analytics, and make recommendations based on patterns leveraging advanced automation systems Automation that learns and evolves • Adaptive continuous delivery pipeline. E.g. - Correlate code quality and defect detection and slippage patterns to dynamically decide what tests to be run • DevOps process optimization based on insight across the lifecycle. E.g. - Determine the level of failure risk by correlating production log data with past code change data • Usage of parallel pipeline to prevent disruption , Release Orchestration InteliDevOps - Intelligent Development Operations • InteliDevOps is an intelligent build & release management system designed to fail fast and fail cheap by prioritizing tests & injecting them as dynamic stages 1 2 3 Generation of automated test cases ( from manual) using NLP Self-healing automation Automated generation of test scenarios based on analytics on production logs Automated verification of system requirements based on CX analytics 6
- 11. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 11 Behavior Driven Development WHY Less code, less maintenance, more acceptance testing, readability, standardization WHEN Early automation, business driven model WHAT Business value, collaboration, traceability, coverage, metrics Testing APIs, UI and non-UI components of any application by Domain-Specific Language (DSL) Collaboration Strategy Imperative vs Declarative Gherkin Tag Feature Mapping Design Pattern Structured using “Context- Action-Outcome” pattern Aligned Features • For identification of different release, version, type, etc. • Traceability - Map with requirements • Helps team write great Executable Specifications Natural Language Processing (NLP) testing platform to enhance QE automation and improve quality Page Object Pattern aims to move logic out of UI elements BDD Frameworks 7
- 12. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 12 Test SMART : Leverage the power of Artificial Intelligence (AI) Create Automation using AI • AI-driven Web & Mobile App Test : User interaction /behaviour is recorded by AI to auto generates test scripts • Smart API Test Generator • AI based Visual Test Validation across OS, browsers and screen resolution Maintenance through Self Healing AI engine auto updated ‘dynamic locator’ to self heal automation scripts Automated Failure Analysis • Traditional failure analysis by manual investigation and categorization of automation error logs, screenshots, etc.. and provide meaningful interpretation for test reports • AI-driven Application Failure Log Analysis - Automating and building predictive model that continuously learn from error logs such as system logs, application logs, database log tables, etc.. (pattern based) to identify failure root cause Robotic Process Automation Automation across large scale standard systems like SAP, Microsoft Dynamics 365, Salesforce etc. Leveraging types of AI Automation • AI-driven automation test creation • AI which monitors HTTP(S) traffics to generate API scenarios • Visual Validation Automation Test • Self-healing systems • AI for automation failure analysis 8
- 13. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 13 Extend Application Testing in Production - Shift-Right QA • Canary Release - push of programming code changes to a small group of end users • A/B Testing - compares two versions of a product hosted in production • Blue-green deployment • Split testing • Automated Roll back strategy • Visual Testing across OS, browsers and screen resolution. Tools like Applitools helps in pixel-to-pixel testing. • New Relic Synthetic User Testing – a suite of automated, scriptable tools to monitor your websites, critical business transactions, and API endpoints. • Infrastructure Integration Testing (IIT) • Disaster Recovery (DR) Testing Testing Web Services with live production traffic • Live production traffic helps to test new version of a web service and to determine if unpredictable changes in API usage cause unexpected behavior, such as slower response times or deviations in the CPU consumption. Helps reveal • Potential weak points and causes of failure • How efficiently servers can be restarted • How fast the app can switch to backup systems and • How resilient the software is? • How long can entire application run on an average, in practice, before anything slows down, goes down, crashes, or freezes? Adapt deployment strategies Web Services Testing Test Methods APM in Real time Shift Right QA Shift Right QA : Brings in the business perspective in testing for improved quality through end to end business component testing and defect prediction/trend analytics, prevent the performance bottlenecks and increase system performance. Some of the best practices are: 9
- 14. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 14 Shift from traditional Testing to Quality Engineering metrics ‘Single View of Quality’ @Speed and Scale : Establish a ‘Metrics Driven’ QE management framework aimed to measure, manage and optimize ‘Business Outcome Aligned Metrics’ and plan for progressive continuous improvements 10 BUSINESS OUTCOMES DELIVERY OUTCOMES KPIs Faster Time To Market Cycle Time Reduction QA Vs ADM Cycle Time Automation Coverage % Regression Automation Coverage Automation effectiveness % Defects identified through automation Reduce Total Cost of Ownership COQ Reduction QA vs. AD spend QA spend on Run the Business - High Maturity/ Stable applications Improved User experience Advocacy / Brand Value End user satisfaction score % Application Availability