SlideShare a Scribd company logo

Антон Семенченко | (EPAM Systems, DPI.Solutions )Сравнительный анализ инструментов Desktop-ной автоматизации

Download as PPTX, PDF
RIF-Technology
RIF-Technology

The document is a comprehensive analysis of automation tools for desktop applications, categorizing and comparing 15 different tools based on various criteria such as stakeholders' orientation, technical aspects, and mixed criteria. It includes methodologies for selecting appropriate tools and criteria tailored to specific projects, complemented by summarized scores and trends in automation. The document also discusses implications for future trends and recommendations for choosing suitable automation solutions.

Technology
1 of 99
Downloaded 12 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
Anton Semenchenko Сравнительный анализ инструментов Автоматизации Desktop AUT
Agenda, part 1 (general) 1. Problem 2. Solutions 2016
Agenda, part 2 (tools and criteria's) 1. Tools to be compared (15) 2. How why we selected this list of tools? 3. Comparison criteria types (3) 4. Stakeholders oriented comparison criteria (7) 5. Mixed comparison criteria (7) 6. Tech stuff oriented comparison criteria (8) 7. How why we selected these lists of criteria's? 8. How to select proper criteria's for your project
Agenda, part 3 (comparison analyses) 1. Mixed comparison criteria 2. Tech stuff oriented comparison criteria 3. Stakeholders oriented comparison criteria 4. Define our “standard” context 5. Summarized scores 6. How to calculate scores 7. How to use scores / presentation 8. 4 summarized tables
Agenda, part 4 (tools, “how to” and examples) 1. How to define proper tool based on selected criteria's 2. How to link information from presentation to QA Automation metrics 3. How to link information from presentation to Project Health Check 4. How to link information from presentation to QA Automation ROI 5. Tools tiny overview 6. Tools overview structure 7. Example of tool usage structure
Agenda, part 5 (trends, science and “what’s next”) 1. Define a Trend! Is it possible ..? 2. Trend – an option 3. Why so? 4. What’s next
Problem • There is an implicit leader for Web automation
Problem • It’s not that simple if to talk about desktop apps
Tools to be compared • TestComplete Desktop • Unified Functional Testing (UFT) • Ranorex • Telerik Test Studio • Zeenyx AscentialTest • MS VS Coded UI • CUIT • AUTOIT • Sikuli • Jubula • Robot Framework • Winium • WinAppDriver • QTWebDriver • PyWinAuto
How why we selected this list of tools?
Comparison criteria types 1. Stakeholders oriented 2. Tech stuff oriented 3. Mixed
Stakeholders oriented comparison criteria 1. Approximate complexity of auto-test development 2. Approximate complexity of auto-test support 3. Approximate “entrance” level 4. Required technical skills level 5. Tests readability 6. How fast tests run 7. Ability to re-use "Business-Logic" layer in other technical context
Mixed comparison criteria 1. Supported platforms 2. Supported technologies 3. Licensing 4. Maturity 5. Record-Play system support 6. Standard actions pack
Tech stuff oriented comparison criteria 1. Programming languages support 2. Have tools for mapping 3. Self-Made architecture support 4. Data-Driven testing support 5. Test-Driven development support 6. Key-word driven 7. Behavior Driven Development support 8. Continues integration system support
How why we selected these lists of criteria's?
How to select proper criteria's for your project
Mixed comparison criteria
Supported platforms – “the best” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10. Jubula 11. Robot Framework 12. Winium 13. WinAppDriver 14. QTWebDriver 15. PyWinAuto
Supported platforms Tool Platforms Mark TestComplete Desktop Windows Unified Functional Testing Windows Ranorex Windows Telerik Test Studio Windows Zeenyx AscentialTest Windows MS VS Coded UI ; CUIT Windows AUTOIT Windows Sikuli Windows, Unix-like Good Jubula Windows, Unix-like Good Robot Framework Windows, Unix-like Good Winium / WinAppDriver ; QTWebDriver Windows / Windows; Cross-Platform / ; Good PyWinAuto Windows
Supported technologies – “the best” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10.Jubula 11. Robot Framework 12. Winium 13. WinAppDriver 14. QTWebDriver 15. PyWinAuto
Supported technologies Tool Technologies Mark TestComplete Desktop C/C++, WinForms, WPF, Java, Qt Unified Functional Testing WinForms, WPF, Java, SAP Ranorex WinForms, WPF, Java, Qt, SAP Telerik Test Studio WPF Bad Zeenyx AscentialTest Win Forms, WPF, Java Bad MS VS Coded UI ; CUIT Win Forms (partial), WPF Bad AUTOIT OS level Good Sikuli Image recognition based Good Jubula WinForms, WPF, Java Bad Robot Framework Uses AutoIT (and co inside) Good Winium / WinAppDriver ; QTWebDriver WinForms, WPF / Any ; QT Bad PyWinAuto Win32 API, WinForms (partial, Win32 API bases) Bad
Licensing – “the worst” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10. Jubula 11. Robot Framework 12. Winium 13. WinAppDriver 14. QTWebDriver 15. PyWinAuto
Licensing Tool License Mark TestComplete Desktop Paid Bad Unified Functional Testing Paid Bad Ranorex Paid Bad Telerik Test Studio Paid Bad Zeenyx AscentialTest Paid Bad MS VS Coded UI ; CUIT Paid Bad AUTOIT Free Sikuli Open source Good Jubula Open source Good Robot Framework Open source Good Winium / WinAppDriver ; QTWebDriver Open source Good PyWinAuto Open source Good
Maturity – “the worst” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10.Jubula 11.Robot Framework 12.Winium 13.WinAppDriver 14.QTWebDriver 15.PyWinAuto
Maturity Tool Maturity TestComplete Desktop Good Unified Functional Testing Good Ranorex Good Telerik Test Studio Good Zeenyx AscentialTest MS VS Coded UI ; CUIT Good AUTOIT Sikuli Jubula Robot Framework Winium / WinAppDriver ; QTWebDriver Bad PyWinAuto
Record-Play support – do we really need it?
Record-Play support Tool Record-Play Mark TestComplete Desktop Yes Good Unified Functional Testing Yes Good Ranorex Yes Good Telerik Test Studio Yes Good Zeenyx AscentialTest No MS VS Coded UI ; CUIT No AUTOIT No Sikuli No Jubula No Robot Framework No Winium / WinAppDriver ; QTWebDriver No PyWinAuto No
Standard actions pack – do we really need it?
Standard actions pack Tool STD actions Mark TestComplete Desktop No Unified Functional Testing No Ranorex No Telerik Test Studio No Zeenyx AscentialTest Yes Good MS VS Coded UI ; CUIT No AUTOIT No Sikuli Yes Good Jubula Yes Good Robot Framework No Winium / WinAppDriver ; QTWebDriver No PyWinAuto Yes / No (via SWAPY)
Tech stuff oriented comparison criteria
Programming languages – “the best” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10. Jubula 11. Robot Framework 12. Winium 13. WinAppDriver 14. QTWebDriver 15. PyWinAuto
Programming languages support Tool Language Mark TestComplete Desktop Python, C#Script, JScript, C++Script, VBScript, DelphiScript Good Unified Functional Testing VBScript Bad Ranorex C#, VB.Net Telerik Test Studio C#, VB.Net Zeenyx AscentialTest Own DSL Bad MS VS Coded UI ; CUIT C#, VB.Net AUTOIT Own Basic-like language Bad Sikuli Jython, Java Jubula - Robot Framework Own DSL, Java, Python Winium / WinAppDriver ; QTWebDriver Java, JavaScript, PHP, Python, Ruby, C# Good PyWinAuto CPython
Tools for mapping – “the best” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10. Jubula 11. Robot Framework 12. Winium 13. WinAppDriver 14. QTWebDriver 15. PyWinAuto
Tools for mapping Tool Tools for mapping Mark TestComplete Desktop Yes Good Unified Functional Testing Yes Good Ranorex Yes Good Telerik Test Studio Yes Good Zeenyx AscentialTest Yes / No Good MS VS Coded UI ; CUIT No AUTOIT No Sikuli Yes / No Jubula Yes Good Robot Framework No Winium / WinAppDriver ; QTWebDriver No PyWinAuto No
Custom architecture – “the worst” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10. Jubula 11. Robot Framework 12. Winium 13. WinAppDriver 14. QTWebDriver 15. PyWinAuto
Custom architecture Tool Custom architecture Mark TestComplete Desktop Yes / No Unified Functional Testing Yes / No Ranorex Yes / No Telerik Test Studio Yes / No Zeenyx AscentialTest No Bad MS VS Coded UI ; CUIT Yes Good AUTOIT No Bad Sikuli Yes Good Jubula No / Yes Robot Framework Yes Good Winium / WinAppDriver ; QTWebDriver Yes Good PyWinAuto Yes Good
DDT support – “the worst” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10. Jubula 11. Robot Framework 12. Winium 13. WinAppDriver 14. QTWebDriver 15. PyWinAuto
DDT support Tool DDT support Mark TestComplete Desktop Yes Good Unified Functional Testing Yes Good Ranorex Yes Good Telerik Test Studio Yes Good Zeenyx AscentialTest Yes Good MS VS Coded UI ; CUIT Yes Good AUTOIT No Bad Sikuli Yes / No Good Jubula Yes Good Robot Framework Yes Good Winium / WinAppDriver ; QTWebDriver Yes Good PyWinAuto Yes Good
TDD support – “the worst” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10.Jubula 11. Robot Framework 12. Winium 13. WinAppDriver 14. QTWebDriver 15. PyWinAuto
TDD support Tool TDD Mark TestComplete Desktop Yes / No Good Unified Functional Testing Yes / No Good Ranorex Yes / No Good Telerik Test Studio Yes / No Good Zeenyx AscentialTest No Bad MS VS Coded UI ; CUIT Yes Good AUTOIT No Bad Sikuli Yes / No Good Jubula Yes Good Robot Framework Yes Good Winium / WinAppDriver ; QTWebDriver Yes Good PyWinAuto Yes Good
Key-word driven – “the best” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10. Jubula 11. Robot Framework 12. Winium 13. WinAppDriver 14. QTWebDriver 15. PyWinAuto
Key-word driven support Tool Key-word Mark TestComplete Desktop No Bad Unified Functional Testing No Bad Ranorex Yes / No Telerik Test Studio Yes / No Zeenyx AscentialTest Yes Good MS VS Coded UI ; CUIT Yes / No AUTOIT No Bad Sikuli Yes / No Jubula No Bad Robot Framework Yes Good Winium / WinAppDriver ; QTWebDriver Yes / No PyWinAuto Yes / No
BDD support – “the worst” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10. Jubula 11. Robot Framework 12. Winium 13. WinAppDriver 14. QTWebDriver 15. PyWinAuto
BDD support Tool BDD Mark TestComplete Desktop No Bad Unified Functional Testing No Bad Ranorex Yes Good Telerik Test Studio Yes Good Zeenyx AscentialTest No Bad MS VS Coded UI ; CUIT Yes Good AUTOIT No Bad Sikuli Yes Good Jubula No Bad Robot Framework Yes / No Winium / WinAppDriver ; QTWebDriver Yes Good PyWinAuto Yes Good
CI support – “the worst” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10. Jubula 11. Robot Framework 12. Winium 13. WinAppDriver 14. QTWebDriver 15. PyWinAuto
CI support Tool CI Mark TestComplete Desktop Automated Build Studio Unified Functional Testing Jenkins plugin Ranorex Jenkins Telerik Test Studio Bamboo Zeenyx AscentialTest Test Execution Management MS VS Coded UI ; CUIT Any Good AUTOIT - / Any Sikuli - / Any Java-compatible Jubula No Bad Robot Framework Jenkins plugin Winium / WinAppDriver ; QTWebDriver Any Good PyWinAuto Any Good
Stakeholders oriented comparison criteria
Define our “standard” context
Approximate complexity of auto-test development Tool Development Mark TestComplete Desktop ~3h Unified Functional Testing ~3h Ranorex ~2h Good Telerik Test Studio ~2h Good Zeenyx AscentialTest ~2h Good MS VS Coded UI ; CUIT ~3h ; 2h ; Good AUTOIT ~1h Good Sikuli ~2h Good Jubula ~2h Good Robot Framework ~4h Winium / WinAppDriver ; QTWebDriver ~3h / 6h -> 2h / Bad -> Good PyWinAuto ~1h Good
Approximate complexity of auto-test support (per year) Tool Support Mark TestComplete Desktop ~3h Bad Unified Functional Testing ~3h Bad Ranorex ~2h Good Telerik Test Studio ~2h Good Zeenyx AscentialTest ~3h Bad MS VS Coded UI ; CUIT ~2h ; 1h Good AUTOIT ~4h Bad Sikuli ~5h Bad Jubula ~2h Good Robot Framework ~1h Good Winium / WinAppDriver ; QTWebDriver ~2h / 10h -> 1h Good / Bad -> Good PyWinAuto ~2h Good
Approximate “entrance” level – “the best” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10.Jubula 11.Robot Framework 12.Winium 13.WinAppDriver 14.QTWebDriver 15.PyWinAuto
Approximate “entrance” level Tool Level TestComplete Desktop High Unified Functional Testing High Ranorex Telerik Test Studio Zeenyx AscentialTest MS VS Coded UI ; CUIT High AUTOIT Low Sikuli Low Jubula Robot Framework High Winium / WinAppDriver ; QTWebDriver High -> PyWinAuto
Required “technical skills” level – “the best” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10. Jubula 11. Robot Framework 12. Winium 13. WinAppDriver 14. QTWebDriver 15. PyWinAuto
Required “technical skills” level Tool Level TestComplete Desktop Unified Functional Testing Ranorex Telerik Test Studio Zeenyx AscentialTest Low MS VS Coded UI ; CUIT High ; AUTOIT Low Sikuli Low Jubula Low Robot Framework High Winium / WinAppDriver ; QTWebDriver High -> PyWinAuto Low
Test readability – “the worst” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10. Jubula 11. Robot Framework 12. Winium 13. WinAppDriver 14. QTWebDriver 15. PyWinAuto
Test readability Tool Level TestComplete Desktop Unified Functional Testing Ranorex Telerik Test Studio Zeenyx AscentialTest High MS VS Coded UI ; CUIT AUTOIT Low Sikuli High Jubula High Robot Framework - > High Winium / WinAppDriver ; QTWebDriver - > High PyWinAuto High
How fast tests run – “the best” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10. Jubula 11. Robot Framework 12. Winium 13. WinAppDriver 14. QTWebDriver 15. PyWinAuto
How fast tests run Tool Level TestComplete Desktop Bad Unified Functional Testing Bad Ranorex Telerik Test Studio Zeenyx AscentialTest MS VS Coded UI ; CUIT Good AUTOIT Good Sikuli Bad Jubula Bad Robot Framework Good Winium / WinAppDriver ; QTWebDriver Good PyWinAuto Good
Ability to re-use "Business-Logic" layer Tool “BDD” Mark TestComplete Desktop No Bad Unified Functional Testing No Bad Ranorex Yes Good Telerik Test Studio Yes Good Zeenyx AscentialTest No Bad MS VS Coded UI ; CUIT Yes Good AUTOIT No Bad Sikuli Yes Good Jubula No Bad Robot Framework Yes Good Winium / WinAppDriver ; QTWebDriver Yes Good PyWinAuto Yes Good
Summarized scores
How to calculate scores
How to use scores 
Stakeholders oriented score Tool Score TestComplete Desktop -2 Unified Functional Testing -2 Ranorex +3 Telerik Test Studio +3 Zeenyx AscentialTest +1 MS VS Coded UI ; CUIT +1 AUTOIT +1 Sikuli +3 Jubula +2 Robot Framework +2 Winium / WinAppDriver ; QTWebDriver +2 PyWinAuto +6
Mixed score Tool Score TestComplete Desktop +1 Unified Functional Testing +1 Ranorex +1 Telerik Test Studio 0 Zeenyx AscentialTest -1 MS VS Coded UI ; CUIT -1 AUTOIT +1 Sikuli +4 Jubula +1 Robot Framework +2 Winium / WinAppDriver ; QTWebDriver -2 PyWinAuto -1
Tech stuff oriented score Tool Score TestComplete Desktop +2 Unified Functional Testing 0 Ranorex +4 Telerik Test Studio +4 Zeenyx AscentialTest -1 MS VS Coded UI ; CUIT +4 AUTOIT -6 Sikuli +4 Jubula +1 Robot Framework +4 Winium / WinAppDriver ; QTWebDriver +6 PyWinAuto +5
Summarized score Tool Stub TestComplete Desktop +1 Unified Functional Testing -1 Ranorex +8 Telerik Test Studio +7 Zeenyx AscentialTest -1 MS VS Coded UI ; CUIT +4 AUTOIT -4 Sikuli +11 Jubula +4 Robot Framework +8 Winium / WinAppDriver ; QTWebDriver +6 PyWinAuto +10
How to define proper tool based on selected criteria's
How to 1. link information from presentation to QA Automation metrics 2. link information from presentation to Project Health Check 3. link information from presentation to QA Automation ROI
Tools tiny overview
Tools overview structure 1. Pros 2. Cons 3. What kind of project / product / problem / situation certain tools could be used for!
Example of tool usage structure 1. Plus several examples of each tool usage — Example structure: • Values: — Value the individual — Act as a team — Strive for excellence — Focus on customer — Act with integrity • Prisms: — Technology — Delivery — Leadership
Project A
Project A
Project A
Test Complete Desktop 1. Pros • Low entrance level • High level of test scripts’ flexibility • Huge knowledge base (at about MSDN level) • Wide choice of script languages which look like common languages 2. Cons • Very expensive license • Very specific own script languages
Unified functional testing 1. Pros • Low “entrance” level • High level of test scripts’ flexibility • Good tech support 2. Cons • Strict integration with other HP solutions • Very specific own DSL
Ranorex 1. Pros • Low “entrance” level • Script tests are written on common languages (C#, VB,Net) • Good tech support 2. Cons • Paid license
Telerik Test Studio (Desktop) 1. Pros • Low “entrance” level • Great parameterization of Keyword tests • DDT support using common formats (CSV, XLS, DB) • Converting tests to common languages (C#, VB.NET) 2. Cons • Only WPF-applications
Zeenyx 1. Pros • Supporting complex logic • Great organization of DDT • Using standard .Net libraries support 2. Cons • Need time to learn how to use • Specific own DSL
MS VS Coded UI 1. Pros • “Native” for Windows • Supports a huge set of UI technologies • Generated UI Map • Ready to go infrastructure • Good documentation and support 2. Cons • License cost • Relatively “low level” API
MS VS Coded UI + CUIT 1. Pros • The same as for MS VS Coded UI • Elegant “High level” API 2. Cons • The same as for MS VS Coded UI
AutoIT 1. Pros • Easy • Universal • Free 2. Cons • There is no ready-to-use verification instruments • Test = exe file • There is no ready-to-use reports
Sikuli 1. Pros • IDE is easy to learn and use • Standard actions pack • Supports an ability to write tests using common languages (Java, Python) • Supports an ability to work on different platforms and with any applications • Free 2. Cons • Low test’s reliability • Slow tests work • No ability to work with texts • Complicated to support tests
Jubula 1. Pros • IDE is easy to use • Supports an ability to work on requirement base • Integrated DB for storing test data and results • Free 2. Cons • No flexibility which is ingrain to script tests • No CI support
Robot Framework 1. Pros • Their own not complex and easy-to-read Keyword-based language • Plugins for different IDE’s • Work with different Oss • Different programming languages support • Tools for creating user-own libraries • Free 2. Cons • High entrance level
Winium 1. Pros • Familiar syntax and API • Supports all the languages that are supported by Selenium WebDriver • Free 2. Cons • “Immature” testing tool • Incomplete way of locating elements • A lack of documentation
WinAppDriver 1. Pros • Familiar syntax and API • “Native” for Windows • Free 2. Cons • “Immature” testing tool • Complicated (in special case usage) • A lack of documentation
QTWebDriver 1. Pros • Familiar syntax and API • QT Applications oriented / “Native” (unique tool) • Free 2. Cons • “Immature” testing tool • Complicated (in special case usage) • A lack of documentation
PyWinAuto 1. Pros • Extremely simple to use • Easy to support • Free 2. Cons • Do not support all popular UI technologies • CPython only
Define a Trend! Is it possible ..?
Trend 1. There is a potential leader for Desktop Automation
Why so? “Scientific” technical prove of Trend • Hegel’s dialectics • Bifurcation mathematical apparat (Bifurcation Theory) • Sedov’s law of hierarchical compensation • Pannov-Snuks Vertical • Big History
Why so? Non-technical scientific prove of Trend • Peter Drucker “Management. Challenges for the 21st Century” Note: It’s a topic of the whole big conversation, and I’m sure we’re going to get back to it, but not today…
How to 1. use this presentation on different project phases 2. use this presentation based on main project roles
What’s next (just a possible way) • Shu 1. Use Presentation 1. Please, follow recommendations a) “How to select proper criteria's for your project” b) “How to define proper tool based on selected criteria's” c) “How to link information from presentation to QA Automation metrics” d) “How to link information from presentation to Project Health Check” e) “How to link information from presentation to QA Automation ROI” f) “How to use this presentation on different project phases” g) “how to use this presentation based on main project roles”
What’s next • Ha 1. Update a set of criteria's 2. Update a set of tools 3. Update Presentation 4. Read “Scientific” prove of Trend
What’s next • Ri 1. Re-Read “Scientific” prove of Trend 2. Update a set of criteria's 3. Update a set of tools 4. Update Presentation 5. Predict the “Trend” 6. Manage the “Trend”
Next iteration  • Move from static (Presentation) to dynamic (Application) • For example, “https://telescope.epam.com”
CONTACT ME Anton_Semenchenko@epam.com semenchenko_anton_v https://www.linkedin.com/in/anton- semenchenko-612a926b https://www.facebook.com/semenche nko.anton.v https://twitter.com/comaqa Thanks for your attention Anton Semenchenko DPI.Solutions EPAM Systems www.comaqa.by www.corehard.by

More Related Content

What's hot (20)

PPTX
QA/Test Engineering Perspectives
Roopesh Kohad
 
PPTX
Shift left as first transformation step into Quality Assurance
Zbyszek Mockun
 
PDF
Build & Release Engineering
Pranesh Vittal
 
PPTX
Agile Testing by Example
Mikalai Alimenkou
 
PPTX
Team Foundation Server Process Templates For Effective Project Management
Aaron Bjork
 
PPTX
Teamwork and agile methodologies
Stefano Paluello
 
PPTX
Manual testing1
Raghu Sirka
 
ODP
xUnit and TDD: Why and How in Enterprise Software, August 2012
Justin Gordon
 
PPTX
Extreme programming (xp)
Mohamed Abdelrahman
 
PPT
Tech talks #1- Unit testing and TDD
DUONG Trong Tan
 
PPTX
agile vs. traditional methodologies
SWE department, Bogazici university
 
PPTX
Introduction to the Agile Methods
softwareacademy
 
PPT
Agile Engineering Practices
Vernon Stinebaker
 
PPT
Testing in Agile Projects
sriks7
 
DOC
Saurav_kumar
Saurav Kumar
 
PDF
Agile Testing
Naresh Jain
 
PPTX
Software development lifecycle_release_management
netdbncku
 
DOC
Saurav_Kumar
Saurav Kumar
 
DOCX
Sandeep Kamath Resume
Sandeep Kamath
 
ODP
Extreme Programming
Knoldus Inc.
 
QA/Test Engineering Perspectives
Roopesh Kohad
 
Shift left as first transformation step into Quality Assurance
Zbyszek Mockun
 
Build & Release Engineering
Pranesh Vittal
 
Agile Testing by Example
Mikalai Alimenkou
 
Team Foundation Server Process Templates For Effective Project Management
Aaron Bjork
 
Teamwork and agile methodologies
Stefano Paluello
 
Manual testing1
Raghu Sirka
 
xUnit and TDD: Why and How in Enterprise Software, August 2012
Justin Gordon
 
Extreme programming (xp)
Mohamed Abdelrahman
 
Tech talks #1- Unit testing and TDD
DUONG Trong Tan
 
agile vs. traditional methodologies
SWE department, Bogazici university
 
Introduction to the Agile Methods
softwareacademy
 
Agile Engineering Practices
Vernon Stinebaker
 
Testing in Agile Projects
sriks7
 
Saurav_kumar
Saurav Kumar
 
Agile Testing
Naresh Jain
 
Software development lifecycle_release_management
netdbncku
 
Saurav_Kumar
Saurav Kumar
 
Sandeep Kamath Resume
Sandeep Kamath
 
Extreme Programming
Knoldus Inc.
 

Viewers also liked (20)

PPTX
Вячеслав Черников (Binwell) | Xamarin на практике
RIF-Technology
 
PPTX
Сергей Смирнов (Altair Engineering Inc.) | Организация работы распределенной ...
RIF-Technology
 
PPTX
Ксения Стернина | (Mail.Ru Group)Gamer Experience Research на различных этапа...
RIF-Technology
 
PPTX
Юрий Буянов | (Одноклассники)Нюансы разработки мобильного мессенджера
RIF-Technology
 
PPT
Алексей Раменский (Тэглайн) | Обзор рынка российской заказной веб-разработки ...
RIF-Technology
 
PPTX
I, For One, Welcome Our New Robot Overlords
Steve Malsam
 
PDF
Олег Бунин (Онтико) | Менеджмент и бизнес-процессы в разработке highload-прое...
RIF-Technology
 
PDF
Continuous Deployment pipeline demonstration spiced with Robot Framework and ...
Antti Pohjonen
 
PPTX
Automation patterns on practice
automated-testing.info
 
PDF
Robot framework
Prayoch Rujira
 
PPTX
Taming robotframework
泰 増田
 
PPTX
Test automation within a scrum process
Kushan Shalindra Amarasiri - Technical QE Specialist
 
PPT
Groovyscriptingformanualandautomationtestingusingrobotframework 141221014703-...
Bhaskara Reddy Sannapureddy
 
PPTX
Sphinx + robot framework = documentation as result of functional testing
plewicki
 
PPTX
Robot Framework
Onur Baskirt
 
PPTX
Test Cases are dead, long live Checklists!
SQALab
 
PDF
Functional Tests Automation with Robot Framework
laurent bristiel
 
PDF
「うねり」を産んだ言葉と 自分なりの越境のやり方
Yoh Nakamura
 
PPTX
ATAGTR2017 Expanding test horizons with Robot Framework
Agile Testing Alliance
 
PDF
Robot Framework Introduction & Sauce Labs Integration
Sauce Labs
 
Вячеслав Черников (Binwell) | Xamarin на практике
RIF-Technology
 
Сергей Смирнов (Altair Engineering Inc.) | Организация работы распределенной ...
RIF-Technology
 
Ксения Стернина | (Mail.Ru Group)Gamer Experience Research на различных этапа...
RIF-Technology
 
Юрий Буянов | (Одноклассники)Нюансы разработки мобильного мессенджера
RIF-Technology
 
Алексей Раменский (Тэглайн) | Обзор рынка российской заказной веб-разработки ...
RIF-Technology
 
I, For One, Welcome Our New Robot Overlords
Steve Malsam
 
Олег Бунин (Онтико) | Менеджмент и бизнес-процессы в разработке highload-прое...
RIF-Technology
 
Continuous Deployment pipeline demonstration spiced with Robot Framework and ...
Antti Pohjonen
 
Automation patterns on practice
automated-testing.info
 
Robot framework
Prayoch Rujira
 
Taming robotframework
泰 増田
 
Test automation within a scrum process
Kushan Shalindra Amarasiri - Technical QE Specialist
 
Groovyscriptingformanualandautomationtestingusingrobotframework 141221014703-...
Bhaskara Reddy Sannapureddy
 
Sphinx + robot framework = documentation as result of functional testing
plewicki
 
Robot Framework
Onur Baskirt
 
Test Cases are dead, long live Checklists!
SQALab
 
Functional Tests Automation with Robot Framework
laurent bristiel
 
「うねり」を産んだ言葉と 自分なりの越境のやり方
Yoh Nakamura
 
ATAGTR2017 Expanding test horizons with Robot Framework
Agile Testing Alliance
 
Robot Framework Introduction & Sauce Labs Integration
Sauce Labs
 
Ad

Similar to Антон Семенченко | (EPAM Systems, DPI.Solutions )Сравнительный анализ инструментов Desktop-ной автоматизации (20)

PDF
International journal of computer science and innovation vol 2015-n1-paper2
sophiabelthome
 
PDF
Automation Testing of Web based Application with Selenium and HP UFT (QTP)
IRJET Journal
 
PDF
Analysis of Emerging Automation Tools in Industry
IJAEMSJORNAL
 
DOCX
summary
ANSHU GOYAL
 
PPTX
Automation testing
Mona M. Abd El-Rahman
 
PDF
Automated Testing: An Edge Over Manual Software Testing
ijtsrd
 
PDF
Ijcatr04051014
Editor IJCATR
 
PPTX
[DevDay 2017] Automation Testing - Speaker: Nghia Khuong - Project Manager at...
DevDay Da Nang
 
PDF
Choosing right-automation-tool
BabuDevanandam
 
PDF
Top Automated Testing Tools for 2024_ A Comprehensive Guide.pdf
kalichargn70th171
 
PDF
Discover the power of QA automation testing
Softweb Solutions
 
PDF
Test Automation Tool comparison – HP UFT/QTP vs. Selenium
Aspire Systems
 
PDF
Top Automated UI Testing Tools 2023.pdf
pcloudy2
 
PDF
A Study Of Automated Software Testing Automation Tools And Frameworks
Tony Lisko
 
PDF
Overview and Analysis of Automated Testing Tools: Ranorex, Test Complete, Se...
IRJET Journal
 
PDF
Top 10 Automation Testing Tools in 2020
Alaina Carter
 
PDF
YuryMakedonov_GUI_TestAutomation_QAI_Canada_2007_14h
Yury M
 
PPT
Software Testing ISTQB study material part2.ppt
pavan7211
 
PDF
automation framework
ANSHU GOYAL
 
PDF
Top Software Testing Tools for 2024 - All You Need to Know About.pdf
flufftailshop
 
International journal of computer science and innovation vol 2015-n1-paper2
sophiabelthome
 
Automation Testing of Web based Application with Selenium and HP UFT (QTP)
IRJET Journal
 
Analysis of Emerging Automation Tools in Industry
IJAEMSJORNAL
 
summary
ANSHU GOYAL
 
Automation testing
Mona M. Abd El-Rahman
 
Automated Testing: An Edge Over Manual Software Testing
ijtsrd
 
Ijcatr04051014
Editor IJCATR
 
[DevDay 2017] Automation Testing - Speaker: Nghia Khuong - Project Manager at...
DevDay Da Nang
 
Choosing right-automation-tool
BabuDevanandam
 
Top Automated Testing Tools for 2024_ A Comprehensive Guide.pdf
kalichargn70th171
 
Discover the power of QA automation testing
Softweb Solutions
 
Test Automation Tool comparison – HP UFT/QTP vs. Selenium
Aspire Systems
 
Top Automated UI Testing Tools 2023.pdf
pcloudy2
 
A Study Of Automated Software Testing Automation Tools And Frameworks
Tony Lisko
 
Overview and Analysis of Automated Testing Tools: Ranorex, Test Complete, Se...
IRJET Journal
 
Top 10 Automation Testing Tools in 2020
Alaina Carter
 
YuryMakedonov_GUI_TestAutomation_QAI_Canada_2007_14h
Yury M
 
Software Testing ISTQB study material part2.ppt
pavan7211
 
automation framework
ANSHU GOYAL
 
Top Software Testing Tools for 2024 - All You Need to Know About.pdf
flufftailshop
 
Ad

Recently uploaded (20)

PPTX
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
PDF
Using FME to Develop Self-Service CAD Applications for a Major UK Police Force
Safe Software
 
PDF
CIFDAQ Token Spotlight for 9th July 2025
CIFDAQ
 
PDF
Blockchain Transactions Explained For Everyone
CIFDAQ
 
PDF
From Code to Challenge: Crafting Skill-Based Games That Engage and Reward
aiyshauae
 
PDF
Smart Trailers 2025 Update with History and Overview
Paul Menig
 
PPTX
COMPARISON OF RASTER ANALYSIS TOOLS OF QGIS AND ARCGIS
Sharanya Sarkar
 
PDF
Fl Studio 24.2.2 Build 4597 Crack for Windows Free Download 2025
faizk77g
 
PDF
Exolore The Essential AI Tools in 2025.pdf
Srinivasan M
 
PDF
Log-Based Anomaly Detection: Enhancing System Reliability with Machine Learning
Mohammed BEKKOUCHE
 
PPTX
"Autonomy of LLM Agents: Current State and Future Prospects", Oles` Petriv
Fwdays
 
PDF
Bitcoin for Millennials podcast with Bram, Power Laws of Bitcoin
Stephen Perrenod
 
PDF
Transcript: New from BookNet Canada for 2025: BNC BiblioShare - Tech Forum 2025
BookNet Canada
 
PDF
DevBcn - Building 10x Organizations Using Modern Productivity Metrics
Justin Reock
 
PDF
July Patch Tuesday
Ivanti
 
PDF
Empower Inclusion Through Accessible Java Applications
Ana-Maria Mihalceanu
 
PDF
LLMs.txt: Easily Control How AI Crawls Your Site
Keploy
 
PDF
HubSpot Main Hub: A Unified Growth Platform
Jaswinder Singh
 
PPTX
UiPath Academic Alliance Educator Panels: Session 2 - Business Analyst Content
DianaGray10
 
PDF
"AI Transformation: Directions and Challenges", Pavlo Shaternik
Fwdays
 
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
Using FME to Develop Self-Service CAD Applications for a Major UK Police Force
Safe Software
 
CIFDAQ Token Spotlight for 9th July 2025
CIFDAQ
 
Blockchain Transactions Explained For Everyone
CIFDAQ
 
From Code to Challenge: Crafting Skill-Based Games That Engage and Reward
aiyshauae
 
Smart Trailers 2025 Update with History and Overview
Paul Menig
 
COMPARISON OF RASTER ANALYSIS TOOLS OF QGIS AND ARCGIS
Sharanya Sarkar
 
Fl Studio 24.2.2 Build 4597 Crack for Windows Free Download 2025
faizk77g
 
Exolore The Essential AI Tools in 2025.pdf
Srinivasan M
 
Log-Based Anomaly Detection: Enhancing System Reliability with Machine Learning
Mohammed BEKKOUCHE
 
"Autonomy of LLM Agents: Current State and Future Prospects", Oles` Petriv
Fwdays
 
Bitcoin for Millennials podcast with Bram, Power Laws of Bitcoin
Stephen Perrenod
 
Transcript: New from BookNet Canada for 2025: BNC BiblioShare - Tech Forum 2025
BookNet Canada
 
DevBcn - Building 10x Organizations Using Modern Productivity Metrics
Justin Reock
 
July Patch Tuesday
Ivanti
 
Empower Inclusion Through Accessible Java Applications
Ana-Maria Mihalceanu
 
LLMs.txt: Easily Control How AI Crawls Your Site
Keploy
 
HubSpot Main Hub: A Unified Growth Platform
Jaswinder Singh
 
UiPath Academic Alliance Educator Panels: Session 2 - Business Analyst Content
DianaGray10
 
"AI Transformation: Directions and Challenges", Pavlo Shaternik
Fwdays
 

Антон Семенченко | (EPAM Systems, DPI.Solutions )Сравнительный анализ инструментов Desktop-ной автоматизации

  • 2. Agenda, part 1 (general) 1. Problem 2. Solutions 2016
  • 3. Agenda, part 2 (tools and criteria's) 1. Tools to be compared (15) 2. How why we selected this list of tools? 3. Comparison criteria types (3) 4. Stakeholders oriented comparison criteria (7) 5. Mixed comparison criteria (7) 6. Tech stuff oriented comparison criteria (8) 7. How why we selected these lists of criteria's? 8. How to select proper criteria's for your project
  • 4. Agenda, part 3 (comparison analyses) 1. Mixed comparison criteria 2. Tech stuff oriented comparison criteria 3. Stakeholders oriented comparison criteria 4. Define our “standard” context 5. Summarized scores 6. How to calculate scores 7. How to use scores / presentation 8. 4 summarized tables
  • 5. Agenda, part 4 (tools, “how to” and examples) 1. How to define proper tool based on selected criteria's 2. How to link information from presentation to QA Automation metrics 3. How to link information from presentation to Project Health Check 4. How to link information from presentation to QA Automation ROI 5. Tools tiny overview 6. Tools overview structure 7. Example of tool usage structure
  • 6. Agenda, part 5 (trends, science and “what’s next”) 1. Define a Trend! Is it possible ..? 2. Trend – an option 3. Why so? 4. What’s next
  • 7. Problem • There is an implicit leader for Web automation
  • 8. Problem • It’s not that simple if to talk about desktop apps
  • 9. Tools to be compared • TestComplete Desktop • Unified Functional Testing (UFT) • Ranorex • Telerik Test Studio • Zeenyx AscentialTest • MS VS Coded UI • CUIT • AUTOIT • Sikuli • Jubula • Robot Framework • Winium • WinAppDriver • QTWebDriver • PyWinAuto
  • 10. How why we selected this list of tools?
  • 11. Comparison criteria types 1. Stakeholders oriented 2. Tech stuff oriented 3. Mixed
  • 12. Stakeholders oriented comparison criteria 1. Approximate complexity of auto-test development 2. Approximate complexity of auto-test support 3. Approximate “entrance” level 4. Required technical skills level 5. Tests readability 6. How fast tests run 7. Ability to re-use "Business-Logic" layer in other technical context
  • 13. Mixed comparison criteria 1. Supported platforms 2. Supported technologies 3. Licensing 4. Maturity 5. Record-Play system support 6. Standard actions pack
  • 14. Tech stuff oriented comparison criteria 1. Programming languages support 2. Have tools for mapping 3. Self-Made architecture support 4. Data-Driven testing support 5. Test-Driven development support 6. Key-word driven 7. Behavior Driven Development support 8. Continues integration system support
  • 15. How why we selected these lists of criteria's?
  • 16. How to select proper criteria's for your project
  • 18. Supported platforms – “the best” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10. Jubula 11. Robot Framework 12. Winium 13. WinAppDriver 14. QTWebDriver 15. PyWinAuto
  • 19. Supported platforms Tool Platforms Mark TestComplete Desktop Windows Unified Functional Testing Windows Ranorex Windows Telerik Test Studio Windows Zeenyx AscentialTest Windows MS VS Coded UI ; CUIT Windows AUTOIT Windows Sikuli Windows, Unix-like Good Jubula Windows, Unix-like Good Robot Framework Windows, Unix-like Good Winium / WinAppDriver ; QTWebDriver Windows / Windows; Cross-Platform / ; Good PyWinAuto Windows
  • 20. Supported technologies – “the best” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10.Jubula 11. Robot Framework 12. Winium 13. WinAppDriver 14. QTWebDriver 15. PyWinAuto
  • 21. Supported technologies Tool Technologies Mark TestComplete Desktop C/C++, WinForms, WPF, Java, Qt Unified Functional Testing WinForms, WPF, Java, SAP Ranorex WinForms, WPF, Java, Qt, SAP Telerik Test Studio WPF Bad Zeenyx AscentialTest Win Forms, WPF, Java Bad MS VS Coded UI ; CUIT Win Forms (partial), WPF Bad AUTOIT OS level Good Sikuli Image recognition based Good Jubula WinForms, WPF, Java Bad Robot Framework Uses AutoIT (and co inside) Good Winium / WinAppDriver ; QTWebDriver WinForms, WPF / Any ; QT Bad PyWinAuto Win32 API, WinForms (partial, Win32 API bases) Bad
  • 22. Licensing – “the worst” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10. Jubula 11. Robot Framework 12. Winium 13. WinAppDriver 14. QTWebDriver 15. PyWinAuto
  • 23. Licensing Tool License Mark TestComplete Desktop Paid Bad Unified Functional Testing Paid Bad Ranorex Paid Bad Telerik Test Studio Paid Bad Zeenyx AscentialTest Paid Bad MS VS Coded UI ; CUIT Paid Bad AUTOIT Free Sikuli Open source Good Jubula Open source Good Robot Framework Open source Good Winium / WinAppDriver ; QTWebDriver Open source Good PyWinAuto Open source Good
  • 24. Maturity – “the worst” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10.Jubula 11.Robot Framework 12.Winium 13.WinAppDriver 14.QTWebDriver 15.PyWinAuto
  • 25. Maturity Tool Maturity TestComplete Desktop Good Unified Functional Testing Good Ranorex Good Telerik Test Studio Good Zeenyx AscentialTest MS VS Coded UI ; CUIT Good AUTOIT Sikuli Jubula Robot Framework Winium / WinAppDriver ; QTWebDriver Bad PyWinAuto
  • 26. Record-Play support – do we really need it?
  • 27. Record-Play support Tool Record-Play Mark TestComplete Desktop Yes Good Unified Functional Testing Yes Good Ranorex Yes Good Telerik Test Studio Yes Good Zeenyx AscentialTest No MS VS Coded UI ; CUIT No AUTOIT No Sikuli No Jubula No Robot Framework No Winium / WinAppDriver ; QTWebDriver No PyWinAuto No
  • 28. Standard actions pack – do we really need it?
  • 29. Standard actions pack Tool STD actions Mark TestComplete Desktop No Unified Functional Testing No Ranorex No Telerik Test Studio No Zeenyx AscentialTest Yes Good MS VS Coded UI ; CUIT No AUTOIT No Sikuli Yes Good Jubula Yes Good Robot Framework No Winium / WinAppDriver ; QTWebDriver No PyWinAuto Yes / No (via SWAPY)
  • 30. Tech stuff oriented comparison criteria
  • 31. Programming languages – “the best” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10. Jubula 11. Robot Framework 12. Winium 13. WinAppDriver 14. QTWebDriver 15. PyWinAuto
  • 32. Programming languages support Tool Language Mark TestComplete Desktop Python, C#Script, JScript, C++Script, VBScript, DelphiScript Good Unified Functional Testing VBScript Bad Ranorex C#, VB.Net Telerik Test Studio C#, VB.Net Zeenyx AscentialTest Own DSL Bad MS VS Coded UI ; CUIT C#, VB.Net AUTOIT Own Basic-like language Bad Sikuli Jython, Java Jubula - Robot Framework Own DSL, Java, Python Winium / WinAppDriver ; QTWebDriver Java, JavaScript, PHP, Python, Ruby, C# Good PyWinAuto CPython
  • 33. Tools for mapping – “the best” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10. Jubula 11. Robot Framework 12. Winium 13. WinAppDriver 14. QTWebDriver 15. PyWinAuto
  • 34. Tools for mapping Tool Tools for mapping Mark TestComplete Desktop Yes Good Unified Functional Testing Yes Good Ranorex Yes Good Telerik Test Studio Yes Good Zeenyx AscentialTest Yes / No Good MS VS Coded UI ; CUIT No AUTOIT No Sikuli Yes / No Jubula Yes Good Robot Framework No Winium / WinAppDriver ; QTWebDriver No PyWinAuto No
  • 35. Custom architecture – “the worst” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10. Jubula 11. Robot Framework 12. Winium 13. WinAppDriver 14. QTWebDriver 15. PyWinAuto
  • 36. Custom architecture Tool Custom architecture Mark TestComplete Desktop Yes / No Unified Functional Testing Yes / No Ranorex Yes / No Telerik Test Studio Yes / No Zeenyx AscentialTest No Bad MS VS Coded UI ; CUIT Yes Good AUTOIT No Bad Sikuli Yes Good Jubula No / Yes Robot Framework Yes Good Winium / WinAppDriver ; QTWebDriver Yes Good PyWinAuto Yes Good
  • 37. DDT support – “the worst” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10. Jubula 11. Robot Framework 12. Winium 13. WinAppDriver 14. QTWebDriver 15. PyWinAuto
  • 38. DDT support Tool DDT support Mark TestComplete Desktop Yes Good Unified Functional Testing Yes Good Ranorex Yes Good Telerik Test Studio Yes Good Zeenyx AscentialTest Yes Good MS VS Coded UI ; CUIT Yes Good AUTOIT No Bad Sikuli Yes / No Good Jubula Yes Good Robot Framework Yes Good Winium / WinAppDriver ; QTWebDriver Yes Good PyWinAuto Yes Good
  • 39. TDD support – “the worst” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10.Jubula 11. Robot Framework 12. Winium 13. WinAppDriver 14. QTWebDriver 15. PyWinAuto
  • 40. TDD support Tool TDD Mark TestComplete Desktop Yes / No Good Unified Functional Testing Yes / No Good Ranorex Yes / No Good Telerik Test Studio Yes / No Good Zeenyx AscentialTest No Bad MS VS Coded UI ; CUIT Yes Good AUTOIT No Bad Sikuli Yes / No Good Jubula Yes Good Robot Framework Yes Good Winium / WinAppDriver ; QTWebDriver Yes Good PyWinAuto Yes Good
  • 41. Key-word driven – “the best” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10. Jubula 11. Robot Framework 12. Winium 13. WinAppDriver 14. QTWebDriver 15. PyWinAuto
  • 42. Key-word driven support Tool Key-word Mark TestComplete Desktop No Bad Unified Functional Testing No Bad Ranorex Yes / No Telerik Test Studio Yes / No Zeenyx AscentialTest Yes Good MS VS Coded UI ; CUIT Yes / No AUTOIT No Bad Sikuli Yes / No Jubula No Bad Robot Framework Yes Good Winium / WinAppDriver ; QTWebDriver Yes / No PyWinAuto Yes / No
  • 43. BDD support – “the worst” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10. Jubula 11. Robot Framework 12. Winium 13. WinAppDriver 14. QTWebDriver 15. PyWinAuto
  • 44. BDD support Tool BDD Mark TestComplete Desktop No Bad Unified Functional Testing No Bad Ranorex Yes Good Telerik Test Studio Yes Good Zeenyx AscentialTest No Bad MS VS Coded UI ; CUIT Yes Good AUTOIT No Bad Sikuli Yes Good Jubula No Bad Robot Framework Yes / No Winium / WinAppDriver ; QTWebDriver Yes Good PyWinAuto Yes Good
  • 45. CI support – “the worst” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10. Jubula 11. Robot Framework 12. Winium 13. WinAppDriver 14. QTWebDriver 15. PyWinAuto
  • 46. CI support Tool CI Mark TestComplete Desktop Automated Build Studio Unified Functional Testing Jenkins plugin Ranorex Jenkins Telerik Test Studio Bamboo Zeenyx AscentialTest Test Execution Management MS VS Coded UI ; CUIT Any Good AUTOIT - / Any Sikuli - / Any Java-compatible Jubula No Bad Robot Framework Jenkins plugin Winium / WinAppDriver ; QTWebDriver Any Good PyWinAuto Any Good
  • 49. Approximate complexity of auto-test development Tool Development Mark TestComplete Desktop ~3h Unified Functional Testing ~3h Ranorex ~2h Good Telerik Test Studio ~2h Good Zeenyx AscentialTest ~2h Good MS VS Coded UI ; CUIT ~3h ; 2h ; Good AUTOIT ~1h Good Sikuli ~2h Good Jubula ~2h Good Robot Framework ~4h Winium / WinAppDriver ; QTWebDriver ~3h / 6h -> 2h / Bad -> Good PyWinAuto ~1h Good
  • 50. Approximate complexity of auto-test support (per year) Tool Support Mark TestComplete Desktop ~3h Bad Unified Functional Testing ~3h Bad Ranorex ~2h Good Telerik Test Studio ~2h Good Zeenyx AscentialTest ~3h Bad MS VS Coded UI ; CUIT ~2h ; 1h Good AUTOIT ~4h Bad Sikuli ~5h Bad Jubula ~2h Good Robot Framework ~1h Good Winium / WinAppDriver ; QTWebDriver ~2h / 10h -> 1h Good / Bad -> Good PyWinAuto ~2h Good
  • 51. Approximate “entrance” level – “the best” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10.Jubula 11.Robot Framework 12.Winium 13.WinAppDriver 14.QTWebDriver 15.PyWinAuto
  • 52. Approximate “entrance” level Tool Level TestComplete Desktop High Unified Functional Testing High Ranorex Telerik Test Studio Zeenyx AscentialTest MS VS Coded UI ; CUIT High AUTOIT Low Sikuli Low Jubula Robot Framework High Winium / WinAppDriver ; QTWebDriver High -> PyWinAuto
  • 53. Required “technical skills” level – “the best” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10. Jubula 11. Robot Framework 12. Winium 13. WinAppDriver 14. QTWebDriver 15. PyWinAuto
  • 54. Required “technical skills” level Tool Level TestComplete Desktop Unified Functional Testing Ranorex Telerik Test Studio Zeenyx AscentialTest Low MS VS Coded UI ; CUIT High ; AUTOIT Low Sikuli Low Jubula Low Robot Framework High Winium / WinAppDriver ; QTWebDriver High -> PyWinAuto Low
  • 55. Test readability – “the worst” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10. Jubula 11. Robot Framework 12. Winium 13. WinAppDriver 14. QTWebDriver 15. PyWinAuto
  • 56. Test readability Tool Level TestComplete Desktop Unified Functional Testing Ranorex Telerik Test Studio Zeenyx AscentialTest High MS VS Coded UI ; CUIT AUTOIT Low Sikuli High Jubula High Robot Framework - > High Winium / WinAppDriver ; QTWebDriver - > High PyWinAuto High
  • 57. How fast tests run – “the best” tool? 1. TestComplete Desktop 2. Unified Functional Testing (UFT) 3. Ranorex 4. Telerik Test Studio 5. Zeenyx AscentialTest 6. MS VS Coded UI 7. CUIT 8. AUTOIT 9. Sikuli 10. Jubula 11. Robot Framework 12. Winium 13. WinAppDriver 14. QTWebDriver 15. PyWinAuto
  • 58. How fast tests run Tool Level TestComplete Desktop Bad Unified Functional Testing Bad Ranorex Telerik Test Studio Zeenyx AscentialTest MS VS Coded UI ; CUIT Good AUTOIT Good Sikuli Bad Jubula Bad Robot Framework Good Winium / WinAppDriver ; QTWebDriver Good PyWinAuto Good
  • 59. Ability to re-use "Business-Logic" layer Tool “BDD” Mark TestComplete Desktop No Bad Unified Functional Testing No Bad Ranorex Yes Good Telerik Test Studio Yes Good Zeenyx AscentialTest No Bad MS VS Coded UI ; CUIT Yes Good AUTOIT No Bad Sikuli Yes Good Jubula No Bad Robot Framework Yes Good Winium / WinAppDriver ; QTWebDriver Yes Good PyWinAuto Yes Good
  • 62. How to use scores 
  • 63. Stakeholders oriented score Tool Score TestComplete Desktop -2 Unified Functional Testing -2 Ranorex +3 Telerik Test Studio +3 Zeenyx AscentialTest +1 MS VS Coded UI ; CUIT +1 AUTOIT +1 Sikuli +3 Jubula +2 Robot Framework +2 Winium / WinAppDriver ; QTWebDriver +2 PyWinAuto +6
  • 64. Mixed score Tool Score TestComplete Desktop +1 Unified Functional Testing +1 Ranorex +1 Telerik Test Studio 0 Zeenyx AscentialTest -1 MS VS Coded UI ; CUIT -1 AUTOIT +1 Sikuli +4 Jubula +1 Robot Framework +2 Winium / WinAppDriver ; QTWebDriver -2 PyWinAuto -1
  • 65. Tech stuff oriented score Tool Score TestComplete Desktop +2 Unified Functional Testing 0 Ranorex +4 Telerik Test Studio +4 Zeenyx AscentialTest -1 MS VS Coded UI ; CUIT +4 AUTOIT -6 Sikuli +4 Jubula +1 Robot Framework +4 Winium / WinAppDriver ; QTWebDriver +6 PyWinAuto +5
  • 66. Summarized score Tool Stub TestComplete Desktop +1 Unified Functional Testing -1 Ranorex +8 Telerik Test Studio +7 Zeenyx AscentialTest -1 MS VS Coded UI ; CUIT +4 AUTOIT -4 Sikuli +11 Jubula +4 Robot Framework +8 Winium / WinAppDriver ; QTWebDriver +6 PyWinAuto +10
  • 67. How to define proper tool based on selected criteria's
  • 68. How to 1. link information from presentation to QA Automation metrics 2. link information from presentation to Project Health Check 3. link information from presentation to QA Automation ROI
  • 70. Tools overview structure 1. Pros 2. Cons 3. What kind of project / product / problem / situation certain tools could be used for!
  • 71. Example of tool usage structure 1. Plus several examples of each tool usage — Example structure: • Values: — Value the individual — Act as a team — Strive for excellence — Focus on customer — Act with integrity • Prisms: — Technology — Delivery — Leadership
  • 75. Test Complete Desktop 1. Pros • Low entrance level • High level of test scripts’ flexibility • Huge knowledge base (at about MSDN level) • Wide choice of script languages which look like common languages 2. Cons • Very expensive license • Very specific own script languages
  • 76. Unified functional testing 1. Pros • Low “entrance” level • High level of test scripts’ flexibility • Good tech support 2. Cons • Strict integration with other HP solutions • Very specific own DSL
  • 77. Ranorex 1. Pros • Low “entrance” level • Script tests are written on common languages (C#, VB,Net) • Good tech support 2. Cons • Paid license
  • 78. Telerik Test Studio (Desktop) 1. Pros • Low “entrance” level • Great parameterization of Keyword tests • DDT support using common formats (CSV, XLS, DB) • Converting tests to common languages (C#, VB.NET) 2. Cons • Only WPF-applications
  • 79. Zeenyx 1. Pros • Supporting complex logic • Great organization of DDT • Using standard .Net libraries support 2. Cons • Need time to learn how to use • Specific own DSL
  • 80. MS VS Coded UI 1. Pros • “Native” for Windows • Supports a huge set of UI technologies • Generated UI Map • Ready to go infrastructure • Good documentation and support 2. Cons • License cost • Relatively “low level” API
  • 81. MS VS Coded UI + CUIT 1. Pros • The same as for MS VS Coded UI • Elegant “High level” API 2. Cons • The same as for MS VS Coded UI
  • 82. AutoIT 1. Pros • Easy • Universal • Free 2. Cons • There is no ready-to-use verification instruments • Test = exe file • There is no ready-to-use reports
  • 83. Sikuli 1. Pros • IDE is easy to learn and use • Standard actions pack • Supports an ability to write tests using common languages (Java, Python) • Supports an ability to work on different platforms and with any applications • Free 2. Cons • Low test’s reliability • Slow tests work • No ability to work with texts • Complicated to support tests
  • 84. Jubula 1. Pros • IDE is easy to use • Supports an ability to work on requirement base • Integrated DB for storing test data and results • Free 2. Cons • No flexibility which is ingrain to script tests • No CI support
  • 85. Robot Framework 1. Pros • Their own not complex and easy-to-read Keyword-based language • Plugins for different IDE’s • Work with different Oss • Different programming languages support • Tools for creating user-own libraries • Free 2. Cons • High entrance level
  • 86. Winium 1. Pros • Familiar syntax and API • Supports all the languages that are supported by Selenium WebDriver • Free 2. Cons • “Immature” testing tool • Incomplete way of locating elements • A lack of documentation
  • 87. WinAppDriver 1. Pros • Familiar syntax and API • “Native” for Windows • Free 2. Cons • “Immature” testing tool • Complicated (in special case usage) • A lack of documentation
  • 88. QTWebDriver 1. Pros • Familiar syntax and API • QT Applications oriented / “Native” (unique tool) • Free 2. Cons • “Immature” testing tool • Complicated (in special case usage) • A lack of documentation
  • 89. PyWinAuto 1. Pros • Extremely simple to use • Easy to support • Free 2. Cons • Do not support all popular UI technologies • CPython only
  • 90. Define a Trend! Is it possible ..?
  • 91. Trend 1. There is a potential leader for Desktop Automation
  • 92. Why so? “Scientific” technical prove of Trend • Hegel’s dialectics • Bifurcation mathematical apparat (Bifurcation Theory) • Sedov’s law of hierarchical compensation • Pannov-Snuks Vertical • Big History
  • 93. Why so? Non-technical scientific prove of Trend • Peter Drucker “Management. Challenges for the 21st Century” Note: It’s a topic of the whole big conversation, and I’m sure we’re going to get back to it, but not today…
  • 94. How to 1. use this presentation on different project phases 2. use this presentation based on main project roles
  • 95. What’s next (just a possible way) • Shu 1. Use Presentation 1. Please, follow recommendations a) “How to select proper criteria's for your project” b) “How to define proper tool based on selected criteria's” c) “How to link information from presentation to QA Automation metrics” d) “How to link information from presentation to Project Health Check” e) “How to link information from presentation to QA Automation ROI” f) “How to use this presentation on different project phases” g) “how to use this presentation based on main project roles”
  • 96. What’s next • Ha 1. Update a set of criteria's 2. Update a set of tools 3. Update Presentation 4. Read “Scientific” prove of Trend
  • 97. What’s next • Ri 1. Re-Read “Scientific” prove of Trend 2. Update a set of criteria's 3. Update a set of tools 4. Update Presentation 5. Predict the “Trend” 6. Manage the “Trend”
  • 98. Next iteration  • Move from static (Presentation) to dynamic (Application) • For example, “https://telescope.epam.com”