Receiver Operating Characteristic (ROC) curve analysis. 19.12

1. An assessment of the self-reported version of the Swedish Strengths
and Difficulties Questionnaire among children and adolescents 12-16
years old
Kenisha S. Russell Jonsson
Irina Vartanova

2. SDQ studies
•Study type1: Examination of the psychometricproperties(alpha
coefficients)
Internal consistency
Retestreliability??
•Study type2: Factor stucture (factoranalysis& SEM)
Controversyof the fiveversusthreestructureversusbifactor
•Study type3:Validity (ROCAnalysis, meancomparison)
specificity& sensitivity
Convergentvalidity??

3. Data
•Community Sample
Survey ofchildrenand young peoplesmental health(Grodan) conductedin
2009, collecteddata from students in grade6 and 9 (roughlybetweenage 11-
17). In total therewere172,000 respondents.
•Service Contact Sample
During1 mars –30 september 2014 data a collectedfrom 2 648 childrenand
young peoplefrom 27 municipalitiesin Sweden whovisiteda healthcare
center.

4. Psychometric properties(1)
Internal consistency reliability (CronbachsAlpha) of the total difficulty scores and
subscores
SDQ scale Community Service
Contact
Widenfelt
et al. (2003)
Goodman
(2001)
Koskelainen
et al. (2000)
Total difficulties 0.63 0.56 0.70 0.80 0.71
Emotional
symptoms
0.69 0.67 0.63 0.66 0.69
Conduct problems 0.55 0.55 0.47 0.60 0.57
Hyperactivity-
inattention
0.66 0.71 0.66 0.67 0.66
Peer Problems 0.54 0.59 0.39 0.41 0.63
Prosocial 0.68 0.67 0.60 0.66 0.69

5. Factor structure(1) Community sample
Specified 5 factor analyses. Obliminrotation (community sample)
Prosocial Emotion Hyper Conduct Peer
Somatic -0.004 0.582 0.019 0.105 -0.040
Worries 0.044 0.733 -0.014 -0.041 0.040
Unhappy 0.044 0.730 -0.081 0.130 0.154
Clingy -0.059 0.548 0.116 -0.181 0.094
Afraid 0.050 0.493 0.063 -0.100 0.103
Tantrum -0.070 0.408 0.107 0.335 -0.031
Robeys -0.406 0.070 0.032 0.187 -0.205
Fights -0.068 0.011 0.057 0.788 -0.006
Lies -0.022 0.024 0.167 0.489 0.180
Steals -0.062 0.034 0.038 0.544 0.079
Restles -0.006 -0.003 0.757 0.024 -0.044
Fidgety 0.034 -0.047 0.877 0.008 0.037
Distrac -0.137 0.359 0.362 0.076 -0.047
Reflect -0.456 0.137 0.094 0.150 -0.157
Rattends -0.451 0.315 0.212 0.026 -0.110
Loner 0.027 0.198 -0.003 -0.026 0.530
Friend -0.228 0.103 -0.072 0.079 0.638
Popular -0.383 0.117 0.046 -0.152 0.475
Bullied 0.112 0.203 0.032 0.280 0.558
Oldbest 0.250 -0.030 0.069 0.029 0.470
Consid 0.553 0.129 -0.001 -0.244 -0.121
Shares 0.461 0.046 0.031 0.088 -0.165
Caring 0.615 0.258 -0.002 -0.067 -0.217
Kind 0.498 -0.001 0.048 -0.258 -0.002
Helpout 0.779 -0.036 -0.0002 0.003 0.029

6. Factor structure(2) Service contactsample
Specified 5 factor analyses. Obliminrotation (community sample)
Prosocial Emotion Hyper Conduct Peer
Somatic 0.104 0.506 0.058 0.099 0.010
Worries 0.026 0.759 -0.069 -0.108 -0.002
Unhappy 0.108 0.680 -0.017 0.060 0.142
Clingy -0.110 0.553 0.039 -0.131 0.077
Afraid 0.023 0.459 -0.100 0.008 0.034
Tantrum -0.036 0.366 0.094 0.504 0.015
Robeys -0.254 -0.005 0.111 0.413 -0.091
Fights -0.038 -0.026 0.017 0.739 0.024
Lies 0.052 -0.181 0.111 0.558 0.301
Steals -0.191 0.046 0.089 0.366 0.030
Restles -0.013 -0.031 0.914 -0.032 0.045
Fidgety 0.006 -0.012 0.885 0.0002 -0.028
Distrac -0.046 0.396 0.384 0.203 -0.064
Reflect -0.124 -0.015 0.147 0.494 -0.078
Rattends -0.137 0.337 0.294 0.231 -0.036
Loner -0.191 0.247 -0.108 -0.151 0.435
Friend -0.089 0.061 -0.034 -0.089 0.751
Popular -0.198 0.073 0.044 0.095 0.587
Bullied 0.140 -0.022 0.098 0.129 0.661
Oldbest 0.148 0.080 -0.046 0.124 0.426
Consid 0.491 0.052 0.094 -0.479 0.020
Shares 0.561 0.034 -0.013 0.046 -0.144
Caring 0.727 0.104 0.007 -0.023 -0.134
Kind 0.435 -0.025 0.057 -0.275 -0.017
Helpout 0.765 -0.038 -0.090 -0.001 0.066

7. Validity(1) Descriptivestats

8. Validity(2) Descriptivestats

9. Validity (3) ROC Analysis
Receiver operating curves
In a ROC curve the true positive rate (Sensitivity) is plotted as a
function of the false positive rate (100-Specificity) for different cut-
off points of a parameter.
The area under the ROC curve (AUC)
a measure of how well a parameter can distinguish between two
diagnostic groups (community/service contact)
a method for reducing the entire ROC curve to a single
quantitativeindex of diagnosticaccuracy

10. Validity (4) Caseness
True positive: cases
withcondition
classifiedas positive
Falsepositive: cases
withoutcondition
classifiedas positive
Falsenegative:
caseswithcondition
classifiedas negative
True negative: cases
withoutcondition
classifiedas negative

11. Validity(5) Sensitivity–Specificity ReportEmotional Problems: Detailed report of sensitivity and specificity
CutpointSensitivitySpecificity
Correctly
ClassifiedLR+ LR-
( >= 0 )100.00%0.00% 0.53% 1.0000
( >= 1 )96.75% 16.52% 16.94% 1.15900.1965
( >= 2 )91.64% 35.90% 36.19% 1.42950.2330
( >= 3 )81.65% 54.18% 54.33% 1.78210.3387
( >= 4 )71.79% 68.67% 68.68% 2.29090.4109
( >= 5 )55.93% 79.78% 79.65% 2.76610.5524
( >= 6 )41.95% 87.91% 87.67% 3.46930.6604
( >= 7 )27.97% 93.21% 92.86% 4.11580.7729
( >= 8 )17.10% 96.42% 96.00% 4.77740.8597
( >= 9 )7.37% 98.42% 97.94% 4.66940.9412
( >= 10 )3.00% 99.38% 98.87% 4.83380.9761
( > 10 )0.00% 100.00%99.47% 1.0000
Cutpoint:indicatethe
rating usedto classify
subjectswith/withouta
condition
Probability of correctly
classifyingthosewitha
condition
Probability of correctly
classifyingthosewithouta
condition
Theratioofthe
probabilityofa
negativetestamong
trulypositivesubjects
totheprobabilityofa
negativetestamong
trulynegativesubjects
Theratioofthe
probabilityofapositive
testamongtrulypositive
subjectsto the
probabilityofapositive
testamongtrulynegative
subjects

12. Validity (6) AUC
OBS AUC SD LLCI ULCI
Total difficulties 1518030.70760.00890.690170.72508
Emotional 1518030.75410.00830.737780.77049
Conduct 1518030.57610.00970.557150.59510
Hyperactivity-inattention 1518030.57680.01040.556340.59728
Peer 1518030.61040.01020.590510.63038
Prosocial 1518030.55950.00980.540250.57879

13. Validity (7) ROC-Emotion0.00
0.25
0.50
0.75
1.00
Sensitivity
0.00 0.25 0.50 0.75 1.00
1 - Specificity
Area under ROC curve = 0.7541
service contact versus community sample
Emotional Problems

14. Validity (8) ROC-Conduct0.00
0.25
0.50
0.75
1.00
Sensitivity
0.00 0.25 0.50 0.75 1.00
1 - Specificity
Area under ROC curve = 0.5761
service contact versus community sample
Conduct Diffiulties

15. Validity (9) ROC -Hyper0.00
0.25
0.50
0.75
1.00
Sensitivity
0.00 0.25 0.50 0.75 1.00
1 - Specificity
Area under ROC curve = 0.5768
service contact versus community sample
Hyperactivity-inattention

16. Validity (10) ROC-Peer0.00
0.25
0.50
0.75
1.00
Sensitivity
0.00 0.25 0.50 0.75 1.00
1 - Specificity
Area under ROC curve = 0.6104
service contact versus community sample
Peer Problems

17. Validity (11) ROC-Prosocial0.00
0.25
0.50
0.75
1.00
Sensitivity
0.00 0.25 0.50 0.75 1.00
1 - Specificity
Area under ROC curve = 0.5595
service contact versus community sample
Prosocial Behaviour

18. Validity (12) ROC-Total difficulties0.00
0.25
0.50
0.75
1.00
Sensitivity
0.00 0.25 0.50 0.75 1.00
1 - Specificity
Area under ROC curve = 0.7076
service contact versus community sample
Total Difficulties

19. Dilemma
•AUC lowfor someofthe subscores-> communitysampleis toohigh
or the service contactsampleis toolowor vice versa.
Howto solvethis???
•Compareresultswithothercountries(specificallynordicsample)
•Furtheranalyses, restricting/moreemphasison service sample
reasonfor the visit
numberofvisit
whocontactedthe service center (parent/ child/teacher/otheradult)
reasonfor contact

20. FACTOR STRUCTURES-THE SWEDISH CONTRIBUTION

21. Explorative (EFA) vs Confirmative (CFA)
Factor Analysis
•In EFA, the factor structure is inferred from the obtained correlation
matrix.
•In CFA,theobtained correlation matrixis compared with a specified
theoretical model.
•The result of comparison is goodness of fit of the specifiedmodel.
Thus, we can compare different factor structures for better
understanding of the analyzed questionnaire.

22. EFA vs CFA
Correlation matrix
Factor structure
Correlation matrix
Theoretical model
compared
Model fit

23. Bifactormodels –the latest suggestion of
model fit improvement
Koboret al., 2013 Casiet al., 2015

24. Alternative models fit
Model
chisq dfRMSEA CFITLI
Original 5-factor model 198,0042650.0680.8960.882
5-factor model with acquiescence style125,5922590.0510.9420.932
Alternative 3-factor model 269,1642720.0800.8530.838
3-factor model with acquiescence style232,6702680.0730.8800.865
Bifactormodel (Koboret al., 2013) 96,6642400.0440.9610.952
Bifactormodel (Casiet al., 2015) 164,0932520.0630.9140.898
Different modelfit
measures
Best fit model
Modelcurrently
testing

25. Reference
Caci, H., Morin, A. J., & Tran, A. (2015). Investigationofa bifactormodelofthe Strengthsand DifficultiesQuestionnaire.Europeanchild&
adolescentpsychiatry,24,pp1291-1301.
Choi,B.C.K. 1998. Slopesof a receiver operating characteriticcurveand the likelihoodratiofor a diagnostictest.AmericanJournal of
Epidemiology148:1127-1132.
Di Riso, D., Salcuni, S., Chessa, D., Raudino, A., Lis, A., & Altoè, G. (2010). The Strengths and Difficulties Questionnaire (SDQ). Early evidence of
its reliability and validity in a community sample of Italian children. Personality and Individual Differences, 49(6), 570-575.
Essau, C. A., Olaya, B., Anastassiou‐Hadjicharalambous, X., Pauli, G., Gilvarry, C., Bray, D., ... & Ollendick, T. H. (2012). Psychometricpropertiesof
the Strengthand DifficultiesQuestionnairefrom fiveEuropeancountries.International journal ofmethodsin psychiatricresearch,21(3), 232-
245.
Goodman, R. (1997). The Strengths and Difficulties Questionnaire: a research note. Journal of child psychology and psychiatry, 38(5), 581-586.
Goodman, R., Meltzer, H., & Bailey, V. (1998). The Strengths and Difficulties Questionnaire: A pilot study on the validity ofthe self-report
version. European child & adolescent psychiatry, 7(3), 125-130.
Goodman, A., Lamping, D. L., & Ploubidis, G. B. (2010). When to use broader internalisingand externalisingsubscales instead of the
hypothesisedfive subscales on the Strengths and Difficulties Questionnaire (SDQ): data from British parents, teachers and children.Journal of
abnormal child psychology,38(8), 1179-1191.
Hanley,J.Aand B.J. McNeil.1982.The meaningand the useof the area under a receiver operating characteristic(ROC) curve. Radiology143:9-
36.
Kóbor, A., Takács, Á., & Urbán, R. (2013). The bifactormodelofthe Strengthsand DifficultiesQuestionnaire.EuropeanJournal ofPsychological
Assessment, 29, pp. 299-307.