Receiver Operating Characteristic (ROC) curve analysis. 19.12
- 1. An assessment of the self-reported version of the Swedish Strengths and Difficulties Questionnaire among children and adolescents 12-16 years old Kenisha S. Russell Jonsson Irina Vartanova
- 2. SDQ studies • Study type 1: Examination of the psychometric properties (alpha coefficients) Internal consistency Retest reliability ?? • Study type 2: Factor stucture (factor analysis & SEM) Controversy of the five versus three structure versus bifactor • Study type 3: Validity (ROC Analysis, mean comparison) specificity & sensitivity Convergent validity ??
- 3. Data • Community Sample Survey of children and young peoples mental health (Grodan) conducted in 2009, collected data from students in grade 6 and 9 (roughly between age 11- 17). In total there were 172,000 respondents. • Service Contact Sample During 1 mars – 30 september 2014 data a collected from 2 648 children and young people from 27 municipalities in Sweden who visited a healthcare center.
- 4. Psychometric properties (1) Internal consistency reliability (Cronbachs Alpha) of the total difficulty scores and subscores SDQ scale Community Service Contact Widenfelt et al. (2003) Goodman (2001) Koskelainen et al. (2000) Total difficulties 0.63 0.56 0.70 0.80 0.71 Emotional symptoms 0.69 0.67 0.63 0.66 0.69 Conduct problems 0.55 0.55 0.47 0.60 0.57 Hyperactivity- inattention 0.66 0.71 0.66 0.67 0.66 Peer Problems 0.54 0.59 0.39 0.41 0.63 Prosocial 0.68 0.67 0.60 0.66 0.69
- 5. Factor structure (1) Community sampleSpecified 5 factor analyses. Oblimin rotation (community sample) Prosocial Emotion Hyper Conduct Peer Somatic -0.004 0.582 0.019 0.105 -0.040 Worries 0.044 0.733 -0.014 -0.041 0.040 Unhappy 0.044 0.730 -0.081 0.130 0.154 Clingy -0.059 0.548 0.116 -0.181 0.094 Afraid 0.050 0.493 0.063 -0.100 0.103 Tantrum -0.070 0.408 0.107 0.335 -0.031 Robeys -0.406 0.070 0.032 0.187 -0.205 Fights -0.068 0.011 0.057 0.788 -0.006 Lies -0.022 0.024 0.167 0.489 0.180 Steals -0.062 0.034 0.038 0.544 0.079 Restles -0.006 -0.003 0.757 0.024 -0.044 Fidgety 0.034 -0.047 0.877 0.008 0.037 Distrac -0.137 0.359 0.362 0.076 -0.047 Reflect -0.456 0.137 0.094 0.150 -0.157 Rattends -0.451 0.315 0.212 0.026 -0.110 Loner 0.027 0.198 -0.003 -0.026 0.530 Friend -0.228 0.103 -0.072 0.079 0.638 Popular -0.383 0.117 0.046 -0.152 0.475 Bullied 0.112 0.203 0.032 0.280 0.558 Oldbest 0.250 -0.030 0.069 0.029 0.470 Consid 0.553 0.129 -0.001 -0.244 -0.121 Shares 0.461 0.046 0.031 0.088 -0.165 Caring 0.615 0.258 -0.002 -0.067 -0.217 Kind 0.498 -0.001 0.048 -0.258 -0.002 Helpout 0.779 -0.036 -0.0002 0.003 0.029
- 6. Factor structure (2) Service contact sampleSpecified 5 factor analyses. Oblimin rotation (community sample) Prosocial Emotion Hyper Conduct Peer Somatic 0.104 0.506 0.058 0.099 0.010 Worries 0.026 0.759 -0.069 -0.108 -0.002 Unhappy 0.108 0.680 -0.017 0.060 0.142 Clingy -0.110 0.553 0.039 -0.131 0.077 Afraid 0.023 0.459 -0.100 0.008 0.034 Tantrum -0.036 0.366 0.094 0.504 0.015 Robeys -0.254 -0.005 0.111 0.413 -0.091 Fights -0.038 -0.026 0.017 0.739 0.024 Lies 0.052 -0.181 0.111 0.558 0.301 Steals -0.191 0.046 0.089 0.366 0.030 Restles -0.013 -0.031 0.914 -0.032 0.045 Fidgety 0.006 -0.012 0.885 0.0002 -0.028 Distrac -0.046 0.396 0.384 0.203 -0.064 Reflect -0.124 -0.015 0.147 0.494 -0.078 Rattends -0.137 0.337 0.294 0.231 -0.036 Loner -0.191 0.247 -0.108 -0.151 0.435 Friend -0.089 0.061 -0.034 -0.089 0.751 Popular -0.198 0.073 0.044 0.095 0.587 Bullied 0.140 -0.022 0.098 0.129 0.661 Oldbest 0.148 0.080 -0.046 0.124 0.426 Consid 0.491 0.052 0.094 -0.479 0.020 Shares 0.561 0.034 -0.013 0.046 -0.144 Caring 0.727 0.104 0.007 -0.023 -0.134 Kind 0.435 -0.025 0.057 -0.275 -0.017 Helpout 0.765 -0.038 -0.090 -0.001 0.066
- 7. Validity (1) Descriptive stats
- 8. Validity (2) Descriptive stats
- 9. Validity (3) ROC Analysis Receiver operating curves In a ROC curve the true positive rate (Sensitivity) is plotted as a function of the false positive rate (100-Specificity) for different cut- off points of a parameter. The area under the ROC curve (AUC) a measure of how well a parameter can distinguish between two diagnostic groups (community/service contact) a method for reducing the entire ROC curve to a single quantitative index of diagnostic accuracy
- 10. Validity (4) Caseness True positive: cases with condition classified as positive False positive: cases without condition classified as positiveFalse negative: cases with condition classified as negative True negative: cases without condition classified as negative
- 11. Validity (5) Sensitivity–Specificity Report Emotional Problems: Detailed report of sensitivity and specificity Cutpoint Sensitivity Specificity Correctly Classified LR+ LR- ( >= 0 ) 100.00% 0.00% 0.53% 1.0000 ( >= 1 ) 96.75% 16.52% 16.94% 1.1590 0.1965 ( >= 2 ) 91.64% 35.90% 36.19% 1.4295 0.2330 ( >= 3 ) 81.65% 54.18% 54.33% 1.7821 0.3387 ( >= 4 ) 71.79% 68.67% 68.68% 2.2909 0.4109 ( >= 5 ) 55.93% 79.78% 79.65% 2.7661 0.5524 ( >= 6 ) 41.95% 87.91% 87.67% 3.4693 0.6604 ( >= 7 ) 27.97% 93.21% 92.86% 4.1158 0.7729 ( >= 8 ) 17.10% 96.42% 96.00% 4.7774 0.8597 ( >= 9 ) 7.37% 98.42% 97.94% 4.6694 0.9412 ( >= 10 ) 3.00% 99.38% 98.87% 4.8338 0.9761 ( > 10 ) 0.00% 100.00% 99.47% 1.0000 Cutpoint:indicate the rating used to classify subjects with/without a condition Probability of correctly classifying those with a condition Probability of correctly classifying those without a condition The ratio of the probability of a negative test among truly positive subjects to the probability of a negative test among truly negative subjects The ratio of the probability of a positive test among truly positive subjects to the probability of a positive test among truly negative subjects
- 12. Validity (6) AUC OBS AUC SD LLCI ULCI Total difficulties 151803 0.7076 0.0089 0.69017 0.72508 Emotional 151803 0.7541 0.0083 0.73778 0.77049 Conduct 151803 0.5761 0.0097 0.55715 0.59510 Hyperactivity-inattention 151803 0.5768 0.0104 0.55634 0.59728 Peer 151803 0.6104 0.0102 0.59051 0.63038 Prosocial 151803 0.5595 0.0098 0.54025 0.57879
- 13. Validity (7) ROC-Emotion 0.000.250.500.751.00 Sensitivity 0.00 0.25 0.50 0.75 1.00 1 - Specificity Area under ROC curve = 0.7541 service contact versus community sample Emotional Problems
- 14. Validity (8) ROC-Conduct 0.000.250.500.751.00 Sensitivity 0.00 0.25 0.50 0.75 1.00 1 - Specificity Area under ROC curve = 0.5761 service contact versus community sample Conduct Diffiulties
- 15. Validity (9) ROC -Hyper 0.000.250.500.751.00 Sensitivity 0.00 0.25 0.50 0.75 1.00 1 - Specificity Area under ROC curve = 0.5768 service contact versus community sample Hyperactivity-inattention
- 16. Validity (10) ROC- Peer 0.000.250.500.751.00 Sensitivity 0.00 0.25 0.50 0.75 1.00 1 - Specificity Area under ROC curve = 0.6104 service contact versus community sample Peer Problems
- 17. Validity (11) ROC-Prosocial 0.000.250.500.751.00 Sensitivity 0.00 0.25 0.50 0.75 1.00 1 - Specificity Area under ROC curve = 0.5595 service contact versus community sample Prosocial Behaviour
- 18. Validity (12) ROC-Total difficulties 0.000.250.500.751.00 Sensitivity 0.00 0.25 0.50 0.75 1.00 1 - Specificity Area under ROC curve = 0.7076 service contact versus community sample Total Difficulties
- 19. Dilemma • AUC low for some of the subscores -> community sample is too high or the service contact sample is too low or vice versa. How to solve this??? • Compare results with other countries (specifically nordic sample) • Further analyses, restricting/more emphasis on service sample reason for the visit number of visit who contacted the service center (parent/ child/teacher/other adult) reason for contact
- 20. FACTOR STRUCTURES- THE SWEDISH CONTRIBUTION
- 21. Explorative (EFA) vs Confirmative (CFA) Factor Analysis • In EFA, the factor structure is inferred from the obtained correlation matrix. • In CFA,the obtained correlation matrix is compared with a specified theoretical model. • The result of comparison is goodness of fit of the specified model. Thus, we can compare different factor structures for better understanding of the analyzed questionnaire.
- 22. EFA vs CFA Correlation matrix Factor structure Correlation matrix Theoretical model compared Model fit
- 23. Bifactor models – the latest suggestion of model fit improvement Kobor et al., 2013 Casi et al., 2015
- 24. Alternative models fit Model chisq df RMSEA CFI TLI Original 5-factor model 198,004 265 0.068 0.896 0.882 5-factor model with acquiescence style 125,592 259 0.051 0.942 0.932 Alternative 3-factor model 269,164 272 0.080 0.853 0.838 3-factor model with acquiescence style 232,670 268 0.073 0.880 0.865 Bifactor model (Kobor et al., 2013) 96,664 240 0.044 0.961 0.952 Bifactor model (Casi et al., 2015) 164,093 252 0.063 0.914 0.898 Different model fit measures Best fit model Model currently testing
- 25. Reference Caci, H., Morin, A. J., & Tran, A. (2015). Investigation of a bifactor model of the Strengths and Difficulties Questionnaire. European child & adolescent psychiatry, 24, pp 1291-1301. Choi,B.C.K. 1998. Slopes of a receiver operating characteritic curve and the likelihood ratio for a diagnostic test.American Journal of Epidemiology 148:1127-1132. Di Riso, D., Salcuni, S., Chessa, D., Raudino, A., Lis, A., & Altoè, G. (2010). The Strengths and Difficulties Questionnaire (SDQ). Early evidence of its reliability and validity in a community sample of Italian children. Personality and Individual Differences, 49(6), 570-575. Essau, C. A., Olaya, B., Anastassiou‐Hadjicharalambous, X., Pauli, G., Gilvarry, C., Bray, D., ... & Ollendick, T. H. (2012). Psychometric properties of the Strength and Difficulties Questionnaire from five European countries. International journal of methods in psychiatric research, 21(3), 232- 245. Goodman, R. (1997). The Strengths and Difficulties Questionnaire: a research note. Journal of child psychology and psychiatry, 38(5), 581-586. Goodman, R., Meltzer, H., & Bailey, V. (1998). The Strengths and Difficulties Questionnaire: A pilot study on the validity of the self-report version. European child & adolescent psychiatry, 7(3), 125-130. Goodman, A., Lamping, D. L., & Ploubidis, G. B. (2010). When to use broader internalising and externalising subscales instead of the hypothesised five subscales on the Strengths and Difficulties Questionnaire (SDQ): data from British parents, teachers and children. Journal of abnormal child psychology,38(8), 1179-1191. Hanley,J.A and B.J. McNeil.1982.The meaning and the use of the area under a receiver operating characteristic (ROC) curve. Radiology 143:9- 36. Kóbor, A., Takács, Á., & Urbán, R. (2013). The bifactor model of the Strengths and Difficulties Questionnaire. European Journal of Psychological Assessment, 29, pp. 299-307.