Reliability in behavioural research with practical example

Reliability in behavioural research with practical example for teaching purpose

1. Dr. SanchitaGarai,Mr. AmitavaPanja& Dr.SanjitMaiti
Reliability in Behavioural Research

2. Context of ‘reliability’ and ‘validity’
•Quantification of human behaviour: Measuring instruments to observe human
behaviour
•Two important aspects of measuring instruments:
a.Validity
b.Reliability
Source: Mohajan(2017)

3. What is Reliability?
Synonyms for reliability: Dependability, Stability, Consistency, Accuracy
Three approaches for defining reliability:
a.Stability approach:
“Ifwemeasurethesamesetofobjectsagainandagainwiththesameorcomparable
measuringinstrument,willwegetthesameorsimilarresults?”
b. Accuracy approach:
“ Are the measures obtained from a measuring instrument the “true” measures of the property
measured?”
c. Solving theoretical or practical problems:
“How much error of measurement there is in a measuring instrument?”
Definition:Reliability is the accuracy or precision of a measuring instrument.

4. Defining ‘Reliability’
•Reliabilityistheaccuracyorprecisionofameasuringinstrument.(Kerlinger,1964)
•Reliabilityistheagreementbetweentwoeffortstomeasurethesametraitthrough
maximallysimilarmethod.(Campbell&Fiske,1967)
•Reliabilityisamajorconcernwhenapsychologicaltestisusedtomeasuresome
attributeorbehaviour.(RosenthalandRosnow,1991)
•Reliabilityisconsistencyofmeasurement(Bollen,1989),orstabilityofmeasurement
overavarietyofconditionsinwhichbasicallythesameresultsshouldbeobtained
(Nunnally,1978).

5. Estimates of Reliability
Methods of estimating reliability can be roughly categorized into two groups:
a.Two separate test administrations
b.One test administration
•Typical methods to estimate test reliability in behavioural research are:
a.Test-retest reliability
b.Alternative forms
c.Split-halves
d.Inter-rater reliability
e.Internal consistency
•Therearethreemainconcernsinreliabilitytesting:equivalence,stabilityover
time,andinternalconsistency.

6. Estimates of Reliability
Source: Drost(2011)

7. •Refers to the temporal stabilityof a test from one measurement session to another.
•Administer the test to a group of respondents and then administer the same test to the same
respondents at a later date.The correlation between scores on the identical tests given at different
times operationally defines its test-retest reliability.
•Despiteitsappeal,thetest-retestreliabilitytechniquehasseverallimitations(Rosenthal&Rosnow,
1991).
a.Memoryeffect:Whentheintervalbetweenthefirstandsecondtestistooshort,respondents
mightrememberwhatwasonthefirsttestandtheiranswersonthesecondtestcouldbeaffectedby
memory.
b.Maturationeffect:Happenswhentheintervalbetweenthetwotestsistoolong,difference.
Maturationreferstochangesinthesubjectfactorsorrespondents(otherthanthoseassociated
withtheindependentvariable)thatoccurovertimeandcauseachangefromtheinitial
measurementstothelatermeasurements(tandt+1).
Test-Retest Reliability

8. Test retest example

9. •Thismethodinvolvesconstructingtwosimilarformsofatest/scale(i.e.bothformshavethe
samecontent)andadministeringbothformstothesamegroupofexamineeswithinavery
shorttimeperiod.
•Thecorrelationbetweenobservedscoresonthealternatetest/scaleforms,isnestimateof
thereliabilityofeitheroneofthealternateforms.
•Thiscorrelationcoefficientisknownascoefficientofequivalence.
•Ifthecorrelationbetweenthealternativeformsislow,itcouldindicatethatconsiderable
measurementerrorispresent,becausetwodifferentscaleswereused.
•Severalofthelimitsofthetest-retestmethodarealsotrueofthealternativeformstechnique.
Alternative forms of reliability

10. Alternative forms

12. Feature Test-Retest Reliability Alternate Forms Reliability
Test Version The same test is administered twice.
Two differentversions (called
alternateforms)ofthetestare
created.Ideally,theseformsmeasure
thesameskillorknowledge.
Administration
Thetestisadministeredtwicetothe
samegroupofpeople.Thetimegap
betweenthetwoadministrationscan
varydependingonthetypeoftest.
Eachformofthetestisadministeredto
thesamegroupofpeople,ideallyatthe
sametime.
Time Gap
Measured between the two
administrationsofthesametest.
Notapplicable.Bothformsare
administeredideallyatthesame
time.
Measures Consistency Of
Scoresonthesametestovertime.This
canbeaffectedbymemory,practice
effects,orchangesintheparticipants'
knowledgeorskills.
Scoresontwodifferentbutequivalent
versionsofthetest.Thishelpsassessif
thetestitselfisconsistentacross
differentversions.

13. •Underthismethod,test/scaledevelopersdividethescale/testintotwohalves,sothatthe
firsthalfformsthefirstpartoftheentiretest/scaleandthesecondhalfformsthe
remainingpartofthetest/scale.
•Bothhalvesarenormallyofequallengthsandtheyaredesignedinsuchawaythateach
isanalternateformoftheother.
•Estimationofreliabilityisbasedoncorrelatingtheresultsofthetwohalvesofthe
sametest/scale.
•Incontrasttothetest-retestandalternativeformmethods,thesplit-halfapproachis
usuallymeasuredinthesametimeperiod.Thecorrelationbetweenthetwohalvestestsmust
becorrectedtoobtainthereliabilitycoefficientforthewholetest(Nunnally,1978;Bollen,1989).
Split-half approach

14. Split-half approach

16. Advantages and disadvantages of
Split-half approach
Advantage:
•First, the effect of memory discussed previously does not operate with this approach.
•Also,apracticaladvantageisthatthesplit-halvesareusuallycheaperandmoreeasilyobtained
thanovertimedata(Bollen,1989).
Disadvantage:
•Testsmustbeparallelmeasures-thatis,thecorrelationbetweenthetwohalveswillvaryslightly
dependingonhowtheitemsaredivided.
•Nunnally(1978)suggestsusingthesplit-halfmethodwhenmeasuringvariabilityofbehavioursover
shortperiodsoftimewhenalternativeformsarenotavailable.
•Forexample,theevenitemscanfirstbegivenasatestand,subsequently,onthesecondoccasion,theodd
itemsasthealternativeform.Thecorrectedcorrelationcoefficientbetweentheevenandodditemtestscores
willindicatetherelativestabilityofthebehaviouroverthatperiodoftime.

17. Example:Twojudgesrating10personsonaparticulartest(i.e.,judgesratingpeople‘s
competencyintheirwritingskills).
•Thecorrelationbetweentheratingsmadebythetwojudgeswilltellusthereliabilityofeither
judgeinthespecificsituation.
Interrater reliability
Whenratersorjudgesareusedtomeasurebehaviour,thereliabilityoftheirjudgments
orcombinedinternalconsistencyofjudgmentsisassessed(Rosenthal&Rosnow,1991).

18. •Concernsthereliabilityofthetestcomponents.
•Internalconsistencymeasuresconsistencywithintheinstrumentandquestionshowwellaset
ofitemsmeasuresaparticularbehaviourorcharacteristicwithinthetest.
•Foratesttobeinternallyconsistent,estimatesofreliabilityarebasedontheaverage
intercorrelationsamongallthesingleitemswithinatest.
•Themostpopularmethodoftestingforinternalconsistencyinthebehaviouralsciencesis
coefficientalpha.
•CoefficientalphawaspopularizedbyCronbach(1951),whorecogniseditsgeneralusefulness.As
aresult,itisoftenreferredtoasCronbach’salpha.
•Coefficientsofinternalconsistencyincreasesasthenumberofitemsgoesup,toacertainpoint.
Internal consistency of reliability

19. •Thisisthemostwidelyusedmethodofestimatingreliabilityusingasingletest
administration.
•Cronbach’sAlpha(??????)iscalculatedbasedonthefollowingformula:
•Where,
kisthenumberofitemsonthetest/scale
??????
2
??????isthevarianceofitemi,
??????
2
??????isthetotaltestvariance

20. AccordingtoKerlinger(1964),principlebehindtheimprovementofreliabilityinaslightly
differentformof‘MaxMinCon’principle-
“Maximizethevarianceoftheindividualdifferencesandminimizetheerrorvariance”
•Writingtheitemsofpsychologicalandeducationalmeasuringinstrumentsunambiguously.
•Ifaninstrumentisnotreliableenough,addmoreitemsofequalkindandquality.
•Clearandstandardinstructionstendtoreduceerrorsofmeasurement.
Improvement of reliability