[2016/2017] RESEARCH in software engineering
4 likes•1,541 views
Ivano Malavolta discusses various aspects of software engineering research, emphasizing the importance of empirical methods, architectural modeling, and collaboration. The document outlines course structures, homework tasks, and specific research themes, as well as the distinction between software engineering and computer science. Additionally, it presents findings from a study on hybrid mobile apps, highlighting user perceptions and performance comparisons between hybrid and native applications.
![[2016/2017] RESEARCH in software engineering](https://image.slidesharecdn.com/ivano01seresearch-161220160404/85/2016-2017-RESEARCH-in-software-engineering-2-320.jpg)
![Empirical software engineering
Scientific use of quantitative and qualitative data to
– understand and
– improve
software products and software development processes
Data is central to address any research question
Issues related to validity addressed continuously
[Victor Basili]](https://image.slidesharecdn.com/ivano01seresearch-161220160404/85/2016-2017-RESEARCH-in-software-engineering-42-320.jpg)
[2016/2017] RESEARCH in software engineering
- 1. Ivano Malavolta RESEARCH in software engineering VRIJE UNIVERSITEIT AMSTERDAM
- 3. Hello Empirical software engineering + Software Architecture + MDE + applied to Complex systems Autonomous robots Mobile-enabled systems If you think good architecture is expensive, try bad architecture. ... Brian Foote and Joseph Yoder
- 4. Course overview (from my perspective) Module name #hours Instructor Introduction to the course 2 Inverardi SE principles, Eclipse IDE, MDE 8 Iovino Research in software engineering 2 Malavolta Collaborative software development 2 Malavolta Software architecture 6 Malavolta Modern development paradigms 2 Malavolta Code generation, applications of MDE, LAB, etc. 10 Iovino Principles of software testing 4 Bertolino Homework 1 Christmas break Homework 2
- 5. Homework 1 Tasks • create an AADL specification describing the architecture of a chosen system • write an Acceleo program that takes as input the AADL model and produces an HTML5 report or other artifact • write a report describing the performed activities Goals • to understand what are the advantages of SE principles like abstraction and separation of concerns • to concretely understand what architectural modeling means • to be exposed to concerns and issues related to dependability • to understand how to use MDE techniques like model transformations and code generators Tentative deadline 20/12/2016
- 6. Homework 2 Tasks • pick a set of articles related to a chosen research theme • carefully read them and explore the state of the art about the research theme • write a report about your findings • make a presentation to the classroom Goals • to study a specific area of software engineering that may be of interest to you • to design a scientific paper • to train yourself in reading and writing scientific papers Tentative deadline 20/01/2017 Research themes will be available soon
- 7. Roadmap Software engineering research Empirical strategies Writing good research papers
- 9. The application of engineering to software We deal with software systems that are: – large and complex – built by teams – exist in many versions – last many years – undergo changes Programming skills are not enough “Physicist example” Software engineering Programmer: complete program Software engineer: software component
- 10. Software engineering VS computer science Computer Science – Computability, algorithms and complexity, programming languages, data structures, databases, artificial intelligence, etc. Software Engineering – The APPLICATION of computer science, mathematics, project management to build high quality software
- 11. My vision about research Research Theory Industrial projectsProgramming Experimentation Ivano Malavolta. Research Statement. November 2013. http://goo.gl/99N5AS
- 12. Research objective Key objectives • Quality àutility + functional correctness • Cost à development + use • Timeliness à good-enough result, when it’s needed Address problems that affect practical software Real world practical PROBLEM Real world practical SOLUTION ?
- 14. Example
- 15. Research strategy Real world practical PROBLEM Real world practical SOLUTION Research setting IDEALIZED PROBLEM Research setting SOLUTION to IDEALIZED PROBLEM Research product (technique, method, model, system, …)
- 17. Validation of the results Real world practical PROBLEM Real world practical SOLUTION Research setting IDEALIZED PROBLEM Research setting SOLUTION to IDEALIZED PROBLEM Research product (technique, method, model, system, …)
- 18. Validation of the results Real world practical PROBLEM Real world practical SOLUTION Research setting IDEALIZED PROBLEM Research setting SOLUTION to IDEALIZED PROBLEM Research product (technique, method, model, system, …) Validation task 1 Does the product solve the idealized problem?
- 19. Validation of the results Real world practical PROBLEM Real world practical SOLUTION Research setting IDEALIZED PROBLEM Research setting SOLUTION to IDEALIZED PROBLEM Research product (technique, method, model, system, …) Validation task 1 Does the product solve the idealized problem? Validation task 2 Does the product help to solve the practical problem?
- 20. Validation of the results: example
- 22. Types of research questions FEASIBILITY CHARACTERIZATION METHOD/MEANS GENERALIZATION DISCRIMINATION Does X exist, and what is it? Is it possible to do X at all? What are the characteristics of X? What exactly do we mean by X? What are the varieties of X, and how are they related? How can we do X? What is a better way to do X? How can we automate doing X? Is X always true of Y? Given X, what will Y be? How do I decide whether X or Y?
- 23. LAB
- 25. Research results Real world practical PROBLEM Real world practical SOLUTION Research setting IDEALIZED PROBLEM Research product (technique, method, model, system, …)
- 26. Types of research results QUALITATIVE & DESCRIPTIVE MODELS TECHNIQUES SYSTEM EMPIRICAL MODELS ANALYTIC MODELS Report interesting observations Generalize from (real-life) examples Structure a problem area; ask good questions Invent new ways to do some tasks, including implementation techniques Develop ways to select from alternatives Embody result in a system, using the system both for insight and as carrier of results Develop empirical predictive models from observed data Develop structural models that permit formal analysis
- 27. LAB
- 29. Research validation Real world practical PROBLEM Real world practical SOLUTION Research setting IDEALIZED PROBLEM Research setting SOLUTION to IDEALIZED PROBLEM Research product (technique, method, model, system, …) Validation task 1 Does the product solve the idealized problem? Validation task 2 Does the result help to solve the practical problem?
- 30. Types of research validation PERSUASION I thought hard about this, and I believe… IMPLEMENTATION Here is a prototype of a system that… EVALUATION Given these criteria, the object rates as… ANALYSIS Given the facts, here are consequences… Formal model Rigorous derivation and proof Empirical model Data on use in controlled situation EXPERIENCE Report on use in practice Qualitative model Narrative Decision criteria Comparison of systems in actual use Empirical model Data, usually statistical, on practice
- 31. LAB
- 32. Building blocks for research Feasibility Characterization Method/means Generalization Selection Qualitative model Technique System Empirical model Analytic model Persuasion Implementation Evaluation Analysis Experience Question Result Validation
- 33. Is this a good plan? Feasibility Characterization Method/means Generalization Selection Qualitative model Technique System Empirical model Analytic model Persuasion Implementation Evaluation Analysis Experience Question Result Validation
- 34. A common good plan Feasibility Characterization Can X be done better? Generalization Selection Qualitative model Technique Build Y Empirical model Analytic model Persuasion Implementation Measure Y, compare to X Analysis Experience Question Result Validation
- 35. Is this a good plan? Feasibility Characterization Method/means Generalization Selection Qualitative model Technique System Empirical model Analytic model Persuasion Implementation Evaluation Analysis Experience Question Result Validation
- 36. A common, but bad, plan Feasibility Characterization Method/means Generalization Selection Qualitative model Technique System Empirical model Analytic model Persuasion Implementation Evaluation Analysis Experience Question Result Validation
- 37. Two other good plans Can X be done at all? Characterization Is X always true of Y? Selection Qualitative model Technique Build a Y that does X Empirical model Formally model Y, prove X “Look it works!” Implementation Check proof Experience Question Result Validation Method/means Evaluation
- 38. LAB Choose a research paper and try to map it into the building blocks of SE research Feasibility Characterization Method/means Generalization Selection Qualitative model Technique System Empirical model Analytic model Persuasion Implementation Evaluation Analysis Experience Question Result Validation
- 39. “NO-NO”s for software engineering research • Assume that a result demonstrated fro a 10K-line system will scale to a 500K-line system • Expect everyone to do things “my way” • Believe functional correctness is sufficient • Assume the existence of a complete, consistent specification • Just build things without extracting enduring lessons • Devise a solution in ignorance of how the world really works
- 40. How do you trust a research then? 1. What are the problems from the real world? – Are they “real” and widespread? 2. Are the solutions general? What are their limits? Real world practical PROBLEM Real world practical SOLUTION ? EMPIRICAL SOFTWARE ENGINEERING
- 41. Empirical strategies* *We will have an advanced course on this topic
- 42. Empirical software engineering Scientific use of quantitative and qualitative data to – understand and – improve software products and software development processes Data is central to address any research question Issues related to validity addressed continuously [Victor Basili]
- 43. Intuition • You apply the scientific method – ask yourself a question – observe a phenomenon – formulate an hypothesis – build a prediction – setup an experiment – perform analysis on your results
- 44. Example End Users’ Perception of Hybrid Mobile Apps in the Google Play Store Ivano Malavolta, Stefano Ruberto Tommaso Soru, Valerio Terragni Ivano Malavolta Gran Sasso Science Institute (Italy) [email protected] ABSTRACT Recently, companies like IBM and Adobe and a growing community of developers advocate hybrid mobile apps development as a possible solution to mobile platforms fragmentation. Hybrid mobile apps are consistent across platforms and built on web standards. In this study, we present an empirical investigation into mobile hybrid apps. Our goal is to identify and analyse the traits and distinctions of publicly available hybrid mobile apps from end users’ perspective. The study has been conducted by mining 11,917 free apps and 3,041,315 reviews from the Google Play Store, and analyzing them from the end users’ perception perspective. The results of this study build an objective and reproducible snapshot about how hybrid mobile development is performing “in the wild” in real projects, thus establishing a base for future methods and techniques for developing hybrid mobile apps. FINDINGS • hybrid development frameworks are perceived as better suited for data-intensive mobile apps, whereas they perform poorly when dealing with low-level, platform- specific features • end users value hybrid and native apps similarly • in some categories, end users perceive native apps better than hybrid apps with respect to performance and the presence of bugs RESEARCH QUESTIONS What is the difference between hybrid and native mobile apps as perceived by end users? – RQ1: What is the difference in the perceived value between hybrid and native mobile apps? – RQ2: What is the difference in the perceived performance between hybrid and native mobile apps? – RQ3: What is the difference in the perceived bugginess between hybrid and native mobile apps? – RQ4: What is the difference in the initial download overhead between hybrid and native mobile apps?
- 45. Dimensions of empirical studies “In the lab” versus “in the wild” studies Qualitative versus quantitative studies Primary versus secondary studies
- 46. “In the lab” versus “in the wild” studies Common “in the lab” methods – Controlled experiments – Literature reviews – Simulations Common “in the wild” methods – Quasi-experiments – Case studies – Survey research – Ethnographies – Action research
- 47. Examples
- 48. Qualitative versus quantitative studies Qualitative research studying objects in their natural setting and letting the findings emerge from the observations – inductive process – the subject is the person Quantitative research quantifying a relationship with the aim to identify a cause- effect relationship – fixed implied factors – focus on collected quantitative data à promotes comparison and statistical analyses They are complementary
- 49. Primary versus secondary studies Primary studies empirical studies in which we directly make measurements or observations about the objects of interest, whether by surveys, experiments, case studies, etc. Secondary studies empirical studies that do not generate any data from direct measurements, but: – analyze a set of primary studies – usually seek to aggregate the results from these in order to provide stronger forms of evidence about a phenomenon
- 50. Examples
- 51. …and what about this?
- 52. Types of empirical studies • Survey • Case study • Experiment
- 53. Survey Def: a system for collecting information from or about people to describe, compare or explain their knowledge, attitudes and behavior Often an investigation performed in retrospect Interviews and questionnaires are the primary means of gathering qualitative or quantitative data These are done through taking a sample which is representative from the population to be studied
- 54. Example: our survey on arch. languages 1. ALs Identification – Definition of a preliminary set of ALs – Systematic search 2. Planning the Survey 3. Designing the survey 4. Analyzing the Data – vertical analysis (and coding) + horizontal analysis
- 55. Case study Def: an empirical enquiry to investigate one instance (or a small number of instances) of a contemporary software engineering phenomenon within its real-life context, especially when the boundary between phenomenon and context cannot be clearly specified Observational study Data collected to track a specific attribute or establishing relationships between different attributes Multivariate statistical analysis is often applied
- 56. Example
- 57. Experiment Def: an empirical enquiry that manipulates one factor or variable of the studied setting 1. Identify and understand the variables that play a role in software development, and the connections between variables 2. Learn cause-effect relationships between the development process and the obtained products 3. Establish laws and best practices about software construction
- 60. Examples of empirical studies in SE • Nejati, Javad, and Aruna Balasubramanian. "An in-depth study of mobile browser performance." Proceedings of the 25th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 2016. • Hassan, Safwat, Weiyi Shang, and Ahmed E. Hassan. "An empirical study of emergency updates for top android mobile apps." Empirical Software Engineering (2016): 1-42. • Tufano, Michele, et al. "An empirical investigation into the nature of test smells." Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering. ACM, 2016. • Linares-Vásquez, Mario, et al. "Mining energy-greedy api usage patterns in android apps: an empirical study." Proceedings of the 11th Working Conference on Mining Software Repositories. ACM, 2014. • Procaccianti, Giuseppe, Héctor Fernández, and Patricia Lago. "Empirical evaluation of two best practices for energy-efficient software development." Journal of Systems and Software 117 (2016): 185-198.
- 61. How to have an impact in reality? This is called technology transfer
- 62. Writing good software engineering papers
- 63. Let’s reconsider our SE research process… Research questions Research results Research validation
- 64. Research questions The program committee looks for – a clear statement of the specific problem you solved – the question about software development you answered – an explanation of how the answer will help solve an important software engineering problem You'll devote most of your paper to describing your result, but you should begin by explaining what question you are answering and why the answer matters Research questions
- 65. Research results Explain precisely – what you have contributed to the catalogue of software engineering knowledge – how this is useful beyond your own project
- 66. Why should the reader believe your result? Show evidence that your result • is valid • helps to solve the problem
- 67. A couple of words on the abstract of a paper • People judge papers by their abstracts – usually decide whether to read the whole paper based on abstract • It's important for the abstract to tell the whole story – background and context – goals – method – results – conclusion • Often it is the only part freely accessible
- 68. Example of an abstract structure: 1. Two or three sentences about the current state of the art, identifying a particular problem 2. One or two sentences about what this paper contributes to improving the situation 3. One or two sentences about the specific result of the paper and the main idea behind it 4. A sentence about how the result is demonstrated or defended
- 69. Coming back to the initial example… State of the art Overall contribution Specific results Validation ✗ ✓✓✗ ✗
- 70. Second try… State of the art Overall contribution Specific results Validation
- 71. What this lecture means to you? Software engineering != computer science You now know how to carry on research in SE Don’t focus on the “size” of the problem, but on – the relevance (the practical, but also the theoretical!) – the accuracy in the investigation (problem and evaluation research) When conducting empirical research, don’t make claims you cannot eventually measure
- 72. Suggested readings about scientific writing • Shaw, M. (2003), Writing Good Software Engineering Research Papers, in Lori A. Clarke; Laurie Dillon & Walter F. Tichy, ed., 'ICSE' , IEEE Computer Society, , pp. 726-737 . • Shaw, M. (2002), 'What makes good research in software engineering?', STTT 4 (1) , 1-7 .
- 74. Contact Ivano Malavolta | Assistant professor Vrije Universiteit Amsterdam iivanoo [email protected] www.ivanomalavolta.com
- 75. Acknowledgement Some parts of this lecture are inspired by: • Ivica Crnkovic’s lecture on software engineering research at Mälardalen University (Sweden) • Matthias Galster ‘s tutorial titled “Introduction to Empirical Research Methodologies” at ECSA 2014