[05-B] Experiment design (advanced)
1 like•250 views
The document discusses advanced experimental design techniques focusing on various types, including factorial and Latin square designs, to model effects and interactions between factors. It emphasizes the importance of randomization, balancing designs, and instrumentation in conducting experiments. Additionally, it highlights the complexities involved with multiple factors and treatments while providing examples for clarity.
[05-B] Experiment design (advanced)
- 1. ‹#› Het begint met een idee Experiment design (advanced) Ivano Malavolta
- 2. Vrije Universiteit Amsterdam Advanced design types Instrumentation 2 Roadmap Ivano Malavolta / S2 group / Experiment design
- 3. Vrije Universiteit Amsterdam 3 Line of reasoning Ivano Malavolta / S2 group / Experiment design
- 4. Vrije Universiteit Amsterdam We can have the following cases: ● 1 factor and 2 treatments (1F-2T) ● 1 factor and >2 treatments (1F-MT) ● 2 factors and 2 treatments (2F-2T) ● >2 factors, each one with >=2 treatments (MF-MT) 4 Advanced design types Ivano Malavolta / S2 group / Experiment design
- 5. Vrije Universiteit Amsterdam 5 2 factors ● Things become more complex ● Interaction must be modeled: ○ 𝜏i : effect of treatment i (level of factor A) ○ 𝛽j : effect of treatment j (level of factor B) ○ (𝜏𝛽)ij : effect of the interaction between 𝜏i and 𝛽j ● Two factors may interact with each other Ivano Malavolta / S2 group / Experiment design
- 6. Vrije Universiteit Amsterdam 6 Definition of effect Ivano Malavolta / S2 group / Experiment design ● Null hypothesis: no difference in means ● Typical model of an outcome (dependent variable): ○ Yij = 𝜇 + Ti + error Outcome for subject j under treatment i Average of all values Effect of treatment i (offset)
- 7. Vrije Universiteit Amsterdam 7 Definition of effect Ivano Malavolta / S2 group / Experiment design 𝜇 = avg(Y) Ti = avg(Yi) - avg(Y) 𝜇 𝜇 + Ti 𝜑(Y) Y
- 8. Vrije Universiteit Amsterdam 8 Interaction Interaction: non-additive effects between factors Additive model Yij = 𝜇 + 𝜏i + 𝛽j + error Non-Additive model Yij = 𝜇 + 𝜏i + 𝛽j + (𝜏𝛽)ij + error https://goo.gl/XUq8tp
- 9. Vrije Universiteit Amsterdam 9 2 factors, 2 treatments (2F-2T) Ivano Malavolta / S2 group / Experiment design ● Example: ○ Objects: 8 applications ○ Factor 1: Connection Protocol ■ Treatment 1: HTTP ■ Treatment 2: HTTPS ○ Factor 2: Sorting Algorithm ■ Treatment 1: BubbleSort ■ Treatment 2: QuickSort
- 10. Vrije Universiteit Amsterdam 10 2F-2T: factorial design Ivano Malavolta / S2 group / Experiment design ● Consider all possible combinations of treatments ● Each treatment is randomly assigned to experimental objects ● Balanced design: each combination is assigned to an equal number of objects Factor 1: Connection Protocol Treatment 1: HTTP Treatment 2: HTTPS Factor 2: Sorting Treatment 1: BubbleSort Application 4,6 Application 1,7 Treatment 2: QuickSort Application 2,3 Application 5,8
- 11. Vrije Universiteit Amsterdam 11 2F-2T: factorial design Ivano Malavolta / S2 group / Experiment design ● 𝜇i: mean of the dependent variable for treatment i ● 𝜇i=avg(P) ● 𝜏i : effect of treatment i (HTTP/HTTPS) of factor A (Conn. Protocol) ● 𝛽j : effect of treatment j (Bubble/Quick) of factor B (Sorting) ● (𝜏𝛽)ij : effect of the interaction between 𝜏i and 𝛽j
- 12. Vrije Universiteit Amsterdam 12 2F-2T: factorial design Ivano Malavolta / S2 group / Experiment design Null hypothesis: H0A: 𝜏1 = 𝜏2 = 0 Null hypothesis: H0B: 𝛽1 = 𝛽2 = 0 Null hypothesis: H0AB: (𝜏𝛽)ij = 0 ∀ i,j
- 13. Vrije Universiteit Amsterdam 13 2F-2T: factorial design Ivano Malavolta / S2 group / Experiment design Alternative hypothesis: H1A: ∃ i | 𝜏i ≠ 0 Alternative hypothesis: H1AB: ∃ (i,j) | (𝜏𝛽)ij ≠ 0 Alternative hypothesis: H1B: ∃ j | 𝛽j ≠ 0
- 14. Vrije Universiteit Amsterdam 14 2F-2T: 2-stage nested design Ivano Malavolta / S2 group / Experiment design ● Example: ○ Objects: 8 applications ○ Factor 1: Interface ■ Treatment 1: Web-based ■ Treatment 2: Client-based ○ Factor 2: Programming Language ■ Treatment 1,1: PHP ■ Treatment 1,2: ASP ■ Treatment 2,1: Java ■ Treatment 2,2: C++
- 15. Vrije Universiteit Amsterdam 15 2F-2T: 2-stage nested design Ivano Malavolta / S2 group / Experiment design ● One of the two factors has different treatments with respect to the other factor ● Balanced design, randomized application Factor 1: Interface Treatment 1: Web-based Treatment 2: Client-based Factor 2: Programming Language Factor 2: Programming Language Treatment 1,1: PHP Treatment 1,2: ASP Treatment 2,1: Java Treatment 1,2: C++ Application 1,3 Application 6,2 Application 7,8 Application 5,4
- 16. Vrije Universiteit Amsterdam 16 More than 2 factors Ivano Malavolta / S2 group / Experiment design ● Number of experimental groups explodes ● Total number of trials is at least n*n ● More trials = more subjects (larger sample size)
- 17. Vrije Universiteit Amsterdam 17 Factorial designs FULL-COVERAGE ○ every possible combination of all the alternatives of all the factors N = #trials k = #factors ni = #levels for i-th factor
- 18. Vrije Universiteit Amsterdam 18 Factorial designs Discover the effects of each factor and its interactions with the other factors BEWARE: combinatorial curse N = #trials k = #factors ni = #levels for i-th factor SOLUTIONS?
- 19. Vrije Universiteit Amsterdam 19 Latin square designs Ivano Malavolta / S2 group / Experiment design ● Latin Square: an n x n array with n different symbols ● Divide factors in main factor and co-factors (or blocking factor) ○ Levels of the main factors are the "letters” (in the cells) ○ Levels of the co-factors are rows and columns ● All levels of the main factor occurs per each blocking factor
- 20. Vrije Universiteit Amsterdam 20 Latin square designs Ivano Malavolta / S2 group / Experiment design ● 3 factors ○ Code Size: Small, Medium, Large (Main Factor) ○ Programming Language: Java, C++, C (Blocking Factor) ○ Operating System: Windows, Linux, OS X (Blocking Factor) ● Total number of groups: ○ Full 33 factorial design: 27 ○ Latin Square: 9
- 21. Vrije Universiteit Amsterdam 21 Latin square designs Ivano Malavolta / S2 group / Experiment design Factor 1: Programming Language Treatment 1: Java Treatment 2: C++ Treatment 3: C Factor 2: Operating System Treatment 1: Windows Small Medium Large Treatment 2: Linux Medium Large Small Treatment 2: OS X Large Small Medium
- 22. Vrije Universiteit Amsterdam 22 Another example ● Main factor: object-orientation of language (A, B, C, D) ● Co-factor 1: size of the project (Very small to Very large) ● Co-factor 2: team experience (T1, T2, T3, T4)
- 23. Vrije Universiteit Amsterdam 23 Pitfalls of Latin squares Ivano Malavolta / S2 group / Experiment design ● Incomplete or partial design ○ Key: balancing + randomization ● Assumption: factors do not interact ● Randomization is limited by design
- 24. Vrije Universiteit Amsterdam 24 Latin squares applicabile to 1-factor studies ● 1 factor - 1 block Uninteresting factor Factor of interest
- 25. Vrije Universiteit Amsterdam 25 Latin squares applicabile to 1-factor studies ● 1 factor à2 blocks It is equivalent to superimposing two different Latin squares:
- 26. Vrije Universiteit Amsterdam 26 Instrumentation Ivano Malavolta / S2 group / Experiment design
- 27. Vrije Universiteit Amsterdam 27 Instrumentation ● Instrumentation must not affect our control of the experiment Goal: Provide a way to conduct and monitor the experiment ● Types of instrumentation: ○ Objects (e.g. servers, apps) ○ Guidelines (checklists, documentation) ○ Measurement tools (power meters, profiling software, etc.) Ivano Malavolta / S2 group / Experiment design
- 28. Vrije Universiteit Amsterdam ● You know how to design an experiment ● Experiment design is an essential choice when doing an experiment ● Constraints on statistical methods ● If possible, use a simple design ● Maximize the usage of the available objects ▪ automation will be your friend here 28 What this lecture means to you? Ivano Malavolta / S2 group / Experiment design
- 29. Vrije Universiteit Amsterdam 29 Ivano Malavolta / S2 group / Experiment design Readings Chapter 8 Chapter 5 IMPORTANT - Checklist for making a good experiment design (section 5.9)