Functional and Non-functional Test automation
5 likes1,752 views
The document discusses the goals and methodologies of functional and non-functional test automation, emphasizing the importance of selecting the right test cases and tools for successful automation. It outlines different types of test automation frameworks and the significance of designing tests that are resilient to changes. Additionally, the document covers automation's return on investment (ROI) and the various types of non-functional testing, including performance, security, and integration testing.
Functional and Non-functional Test automation
- 1. Functional & Non-functional Test Automation Dr Ganesh Neelakanta Iyer QA Architect GANESHNIYER [email protected] http://ganeshniyer.com
- 2. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.2 About Me GANESHNIYER
- 3. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.3 Quick Poll
- 4. Do you have some test automation in place for your testing?
- 5. Do you have required functional test automation in place?
- 6. Do you have test automation for functional and non-functional (performance, load, scalability, security, …) pieces of testing?
- 7. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.7 Agenda Goal of successful automation Types of test automation frameworks Functional Test automation Non-Functional test automation – Major types
- 8. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.8 Goals of Successful Automation
- 9. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.9 The Goal: Successful Automation Suggestions: • Automate the ‘Right’ Test Cases • Test Early / Test Often • Define Quality Test Data • Design Automated Tests that can resist change • Selecting the right Automation Tool Graphic from qatestlab.com
- 10. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.10 Automate the ‘Right’ Test Cases • Repetitively Run Tests (aka Smoke or Acceptance tests) • Data intensive tests • Frequently Used Functionality • High Risk, Business-critical functionality • Tests that are difficult to run manually • Tests that are run on differing hardware/software platforms and configurations • Tests that take a lot of effort or time Graphic from qatestlab.com
- 11. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.11 Test Early / Test Often Start as early in the process as possible – Could start with Automated Unit testing – Gradually build your Automation Suite The more you test, the more bugs you find. The earlier bugs are found the cheaper they are to fix!
- 12. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.12 Define Quality Test Data With good data… – Writing and Maintaining Automated tests is easier. – With Data driven tests you can run more iterations and find more bugs! – Can extend existing Automated tests as new features are developed – without having to edit the actual automated test itself. – Stay proactive in updating/defining data.
- 13. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.13 Automation Approach Keep Automated tests small and focused – Easy to debug – Easy to manage – Can be reused and shared These can then be grouped into a larger Automated test suite organized by functional area. With the right framework, users can create large/complex Automated tests that are also easy to maintain! Graphic from checkpointech.com
- 14. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.14 Design Automated Tests Resistant to Change If possible, avoid using screen coordinates. Componentize your test scripts Ensure that your development team uses unique names for each control and object!
- 15. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.15 How long do Automated Tests Last? At some point, the application will change and the script will break. • If you are using GUI Capture/Playback a major revision to the UI may break your tests. If the script breaks before the automation effort was repaid, it is better off as a manual test.
- 16. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.16 Selecting the ‘‘Right” Automation Tool Can be used by business users No programming skills required Simple Automate 60-70% of core business processes in a few days Rapid return on investment Fast When underlying applications change, automation continues to work Simple maintenance of your business process portfolio Resilient True end-to-end business process validation covering all major technologies and platforms Complete
- 17. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.17 Return On Investment (ROI)
- 18. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.18 Classic ROI Calculation ROI = BENEFIT/COST Automation Cost = Price Of HW + Price of SW + Development Cost + Maintenance Cost + Execution Cost Manual Testing Cost = Development Cost + Maintenance Cost + Execution Cost ROI = (Manual Testing Cost - Automation Cost)/Automation Cost Looks right, isn’t it?
- 19. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.19 Problems with Classic ROI Calculation You can’t compare Automated Testing and Manual Testing. There are not the same You can’t compare cost of multiple execution of automated tests vs. manual tests. You would never dream of executing that many test cases manually.
- 20. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.20 Automation Real ROI ROI value IS NOT the value of Automation vs. Cost of executing these tests manually Automation ROI value IS the benefit of this type of testing, and it can be: Reducing Time to Market Increased Test Efficiency (Productivity) Increased Test Effectiveness
- 21. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.21 Types of test automation frameworks
- 22. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.22 Types of Test Automation Frameworks Modular Data-Driven Keyword–Driven Model-Based
- 23. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.23 Modular framework The Modular framework is the natural progression from Record-and- Playback The modular framework seeks to minimize this repetition of code by grouping similar actions into “modules” (e.g.: login) Test Data is in a script
- 24. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.24 Data-Driven/Keyword Driven These frameworks are similar in that the data is separated from the test script The script is just a "driver" or delivery mechanism for the data. The difference: • In keyword-driven testing, the navigation data and test data are contained in the data source • In data-driven testing, only test data is contained in the data source.
- 25. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.25 Model-Based testing Model-based testing is software testing in which test cases are derived in whole or in part from a model that describes some (usually functional) aspects of the system. Model-based testing for complex software systems is still an evolving field.
- 26. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.26 Functional test automation
- 27. Typically what we speak about is all about functional automation
- 28. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.28 Functional Testing Web Telerik Test Studio Sahi Selenium Sikuli Web Mobile Desktop Backend Mobile Mobile Test Studio Telerik DeviceCloud Appium Desktop QTP
- 29. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.29 Testing Mobile Apps in Cloud Telerik DeviceCloud >300 29 27 Devices Manufacturers OS versions
- 30. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.30 Compatibility Testing; Web, Device, OS, DB,…. Automation: Telerik Test Studio Telerik DeviceCloud Sahi Selenium Compatibility Testing Accessibility testing Globalization and localization testing Compatibility under different situations
- 31. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.31 Non-Functional test automation – Major types
- 32. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.32 Major Non-functional test dimensions in a Nut-shell Non- functional Security API Connectivity PerformanceResilience Integration
- 33. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.33 Security Security Traversal vulnerability User access/ roles Identity federation management Multi-tenant penetration User Access / Roles Web UI Security Multi- Tenant DB Vulnerabilities Automation: Veracode Websecurify ZAP tool Scripts
- 34. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.34 Security at various stages Planning Architecture and Design Implementation Level Whitebox testing Blackbox testing Reviews Code Doc Log Regional Security Compliance Ruleset System Config Network Sniffing File Integrity Checking PO/ PM QA Dev Architect / Lead Dev QA QA Dev QA QADev Architect / Lead Dev Architect / Lead QA
- 35. Different components of each aspect- what needs to be tested/assured Security protection methods Authentication Access Control Cryptography Firewall, Anti-virus etc Secured Infrastructure Web Security URL Manipulation SQL Injection, Other injection attacks Session fixation, XSS API Security Testing Cloud Security Multi-tenant penetration (for ISVs) Data protection in Cloud Cloud Identity Management Cloud Infrastructure security Database Security Reliability and Integrity DB Disclosure Security in data mining Data security Vulnerabilities and attacks Network, Memory- related, Metadata injection Traversal vulnerability DoS/DDoS attacks Encryption in networks/browsers Programmed Injection Privacy, Legal issues and Ethics Web privacy Law and Compliance Email security Ethical issues Emerging Technologies and Security Aspects Mobile App Mobile Web IoT sensors Information Security Security Issues within the code
- 36. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.36 How we plan?– Four types of activities High-level test approach and test execution frequency QA Reviews • Doc Review • Code Review Static Analysis (Whitebox Testing) •Analyze source for security vulnerabilities typically at the programming and/or testing software life cycle (SLC) phases Service Level Dynamic Testing • Test the run time services exposed by applications for various security testing Business Level Dynamic Testing •Application Runtime analysis, E2E business use cases To happen regularly on a scheduled manner For short releases Once in the release For long releases, 2-3 times during the release cycle Sanity Suite to run on Continuous Integration Automate the cases and run once in a sprint Automate and run once in a sprint Use tools for scanning 1-3 times based on release cycle
- 38. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.38 API Invocation API Invocation Connectivity and invocation API load API security Multi-tenancy Automation: SoapUI Node.js PySys In-house Frameworks
- 39. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.39 High Availability (HA) and Performance Testing Automation: Telerik Test Studio NeoLoad JMeter HA & Performance Time to deploy Multi-tenancy Latency Connectivity and reliability with 3rd parties Reliability and availability HA/ Performance Testing Availability 24 x 7 – Endurance Reliability Latency Stress
- 40. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.40 24 x 7 Testing VMs / Build Integration Machines ANT ScriptTests Reports using Kendo UI controls Decides parallelism HTTP POST reqBatch Script Decides frequency Logs SUT Uses any test environments and leverage existing tests Batch Script Decides the frequency ANT Script We configure #Threads, #Tests to run in parallel Reports Shown using Keno UI controls / widgets
- 41. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.41 Resilience and Failover Ability of a server, network, storage system, or an entire data center, to continue operating even when there has been an equipment failure, power outage or other disruption Software error Hardware error Human error Natural Disaster Picture Credit: http://imperfectspirituality.com/wp-content/uploads/2012/08/Moving- mountains-with-resilience.jpg
- 42. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.42 Resilience and Failover Resilience Making one instance down Making multiple instances down Disaster Recovery Application Interface (UI / API) IaaS Logs ELB HA Instance 2Stubs for 3rd Parties HA Instance 1
- 43. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.43 Integration Integration P2P Integration System Integration End-to-End Integration SSO Customer Scenarios Picture Credit: http://goo.gl/wgNKi3
- 44. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.44 A Typical Integration Scenario in Progress OpenEdge DB OData OData REST Service Telerik Mobile QA Dashboard App DMZ REST Wrapper REST Service
- 45. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.45 Typical End-to-End Scenario Registration Land on product dashboard Click on subscription details Choose a plan Checkout process Generate invoice Product usage Log-out LDAP Community Export Compliance, Marketing Marketing, Billing, Support Billing, Invoicing LDAP Billing Metering Billing
- 46. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.46 Integration Test Challenges and Approaches Multiple systems behave differently No automated deletion of test data in some systems Unpredictable delays in updating various systems Different types of environments for testing
- 47. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.47 Integration Test Automation With Cloud Telerik Test Studio as the UI framework, C# as the Runtime framework Output Repository Test Manager Test Logger Run Time Test Framework API Calls UI Framework Stubs for 3rd Parties Test Repository Test suite Libraries (APIs) Product 1 Product 2 Product 3
- 48. There are much more…
- 51. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.51 Major Test Dimensions in a Nut Shell Security Performance Resilience & Failover Compatibility API Connectivity Integration Traversal vulnerability Time to deploy Making one instance down Accessibility Connectivity and invocation P2P Integration User access/ roles Multi-tenancy Making multiple instances down Globalization and localization API load System Integration Identity federation management Latency Disaster Recovery Compatibility under different situations API security End-to-End Integration Multi-tenant penetration Connectivity and reliability with 3rd parties Multi-tenancy SSO Reliability and availability Customer Scenarios
- 52. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.52 Automation Frameworks Telerik Test Studio, Telerik DeviceCloud Sahi / Selenium scripts Telerik Test Studio for Load / Performance Testing Neoload, JMeter SOAP UI / Node.js / PySys Telerik Test Studio (with C#) TestNG + Java + Sahi combined approach Integration Functional (Web/Mobile UI) Performance API Invocation Veracode, Websecurify, ZAP ToolSecurity