Qualitative and quantitative research Methods

It looks at methods in Qualitative and quantitative research

1. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
VALIDITY AND RELIABILITY
© LOUIS COHEN, LAWRENCE MANION AND
KEITH MORRISON

2. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
STRUCTURE OF THE CHAPTER
•Defining validity
•Validity in quantitative, qualitative and mixed
methods research
•Types of validity
•Triangulation
•Ensuring validity
•Reliability
•Reliability in quantitative and qualitative research
•Validity and reliability in interviews, experiments,
questionnaires, observations, tests, life histories
and case studies

3. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
WHAT IS VALIDITY?
A demonstration that a particular instrument
measures what it intends, purports or claims to
measure.
A demonstration that an account accurately
represents those features that it is intended to
describe, represent, explain or theorise.
The extent to which interpretations of data are
warranted by the theories and evidence used.

4. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
BASES OF VALIDITY IN
QUANTITATIVE RESEARCH
BASES OF VALIDITY IN
QUALITATIVE RESEARCH
Controllability Natural
Isolation, control, manipulation of
variables
Thick description
Replicability Uniqueness
Predictability Emergence, unpredictability
Generalizability Uniqueness
Context-freedom Context-boundedness
Fragmentation and atomization Holism
Randomization of samples Purposive sample/no sampling
Neutrality Value-ladenness of observations
Objectivity Confirmability
Observability Observable and non-observable
meanings/ intentions
Inference Description, inference, explanation
‘Etic’ research ‘Emic’ research
Observations Meanings

5. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
BASES OF RELIABILITY IN
QUANTITATIVE RESEARCH
BASES OF RELIABILITY IN
QUALITATIVE RESEARCH
Reliability Dependability
Demonstrability Trustworthiness
Stability and replicability Stability and replicability
Parallel forms Parallel forms
Context-freedom Context-specificity
Objectivity Authenticity
Coverage of domain Comprehensiveness of situation
Verification of data and analysisHonesty and candour
Answering research questionsDepth of response
Meaningfulness to the researchMeaningfulness to respondents
Parsimony Richness

6. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
BASES OF RELIABILITY IN
QUANTITATIVE RESEARCH
BASES OF RELIABILITY IN
QUALITATIVE RESEARCH
Objectivity Confirmability
Internal consistency Credibility
Generalizability Transferability
Inter-rater reliability Inter-rater reliability
Triangulation Triangulation
Accuracy and precision Accuracy and comprehensiveness
Neutrality Multiple interests represented
Consistency Consistency
Alternative forms (equivalence)
Split-half
Inter-item correlation

7. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
VALIDITY IN QUANTITATIVE AND
QUALITATIVE RESEARCH
Validity in quantitative research often concerns:
objectivity, generalizability, replicability,
predictability, controllability, nomothetic
statements.
Validity in qualitative research often concerns:
honesty, richness, authenticity, depth, scope,
subjectivity, strength of feeling, catching
uniqueness, idiographic statements.

8. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
TYPES OF VALIDITY
•Catalytic
•Concurrent
•Consequential
•Construct
•Content
•Criterion-related
•Convergent and discriminant
•Cross-cultural
•Cultural validity
•Descriptive
•Ecological
•Evaluative
•External
•Face
•Internal
•Interpretive
•Jury
•Predictive
•Systemic
•Theoretical

9. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
VALIDITY IN QUANTITATIVE RESEARCH
•Concurrent
•Construct
•Content
•Criterion-related
•Convergent and discriminant
•Cross-cultural
•Evaluative
•External
•Face
•Internal
•Jury
•Predictive
•Theoretical

10. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
VALIDITY IN QUALITATIVE RESEARCH
Credibility
•The truth value (replacing the quantitative concepts of
internal validity)
Transferability
•Generalizability (replacing the quantitative concept of
external validity)
Dependability
•Consistency (replacing the quantitative concept of reliability)
Confirmability
•Neutrality (replacing the quantitative concept of objectivity)

11. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
MAXWELL’S VALIDITY IN QUALITATIVE RESEARCH
(MAXWELL)
Descriptive validity
•the factual accuracy of the account, that it is not made up,
selective, or distorted; objectively factual) and credible
Interpretive validity
•the ability of the research to catch the meaning, terms,
interpretations, intentions that situations and events, i.e. data,
have for the participants/ subjects themselves, in their terms
Theoretical validity
•the theoretical constructions that the researcher brings to the
research theoretical validity is the extent to which the research
explains phenomena; construct validity

12. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
MAXWELL’S VALIDITY IN QUALITATIVE RESEARCH
(MAXWELL)
Generalizability
•the view that the theory generated may be useful in
understanding other similar situations; generalizing within
specific groups or communities, situations or
circumstances and, beyond, to outsidercommunities,
situations or circumstance
Evaluative validity
•the application of an evaluative, judgmental stance
towards that which is being researched, rather than a
descriptive, explanatory or interpretive framework.

13. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
VALIDITY IN MIXED METHODS RESEARCH
•Representation
•Legitimation
•Sample integration
•Inside–outside
•Weakness minimization
•Sequential
•Conversion
•Paradigmatic mixing
•Commensurability
•Multiple validities
•Political
•Integration (of methods)
•Integration

14. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
INTERNAL VALIDITY
Demonstration that the explanation of a particular event,
issue or set of data which a piece of research provides can
actually be sustained by the data and the research.
The findings must describe accurately the phenomena being
researched.
Truth value and credibility of interpretations and conclusions.

15. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
DIRECTION
OF CAUSALITY
MATURATION
TESTING
THREATS TO
VALIDITY AND
RELIABILITY
TYPE I AND
TYPE II
ERRORS
INSTRUMENT-
ATION
OPERATIONAL-
IZATION
REACTIVITY
HISTORY
EXPERIMENTAL
MORTALITY
CONTAMIN-
ATION

16. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
INTERNAL VALIDITY IN QUALITATIVE RESEARCH
•Confidence in the data
•Authenticity of the data (the ability of the research to
report a situation through the eyes of the participants)
•Cogency of the data
•Soundness of the research design
•Plausibility of the data and interpretation
•Credibility of the data
•Auditability of the data
•Dependability of the data
•Confirmabilityof the data
•Clarity on the kinds of claim made from the research
(e.g. definitional, descriptive, explanatory, theory
generative)

17. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
EXTERNAL VALIDITY
The degree to which the results can be generalized to
the wider population, cases, settings, times or
situations, i.e. the transferability of the findings.

18. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
ESTABLISHING VALIDITY IN QUALITATIVE RESEARCH
•Prolonged engagement in the field
•Persistent observation
•Triangulation
•Leaving an audit trail
•Respondent validation
•Weighting the evidence (giving priority)
•Checking for representativeness
•Checking for researcher effects
•Making contrast/comparisons
•Theoretical sampling
•Checking the meaning of outliers
•Using extreme cases

19. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
ESTABLISHING VALIDITY IN QUALITATIVE RESEARCH
•Ruling out spurious relations
•Replicating a finding
•Referential adequacy
•Following up surprises
•Structural relationships
•Peer debriefing
•Rich and thick description
•Looking for possible sources of invalidity
•Assessing rival explanations
•Negative case analysis
•Confirmatory data analysis
•Effect sizes

20. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
THREATS TO VALIDITY IN QUANTITATIVE RESEARCH
•History
•Maturation
•Statistical regression
•Testing
•Instrumentation
•Selection bias
•Experimental mortality
•Instrument reactivity
•Selection–maturation interaction
•Type I and Type II errors

21. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
VALIDITY PROBLEMS IN CROSS-CULTURAL RESEARCH
•Failure to operationalize elements of cultures.
•Whose construction of ‘culture’ to adopt:
‘emic’/‘etic’.
•False attribution of causality to cultural factors rather
than non-cultural factors.
•Directions of causality.
•Ecological fallacy.

22. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
VALIDITY PROBLEMS IN CROSS-CULTURAL RESEARCH
•Sampling and instrumentation.
•Convergent and discriminant validity.
•Response bias and preparation of participants.
•Language problems.
•Problems of equivalence (conceptual, psychological,
meaning, instrument, understanding, significance,
relevance, measurement, linguistic).

23. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
THREATS TO EXTERNAL VALIDITY
IN QUANTITATIVE RESEARCH
•Failure to describe independent variables explicitly.
•Lack of representativeness of available and target populations.
•Hawthorne effect.
•Inadequate operationalizing of dependent variables.
•Sensitization/reactivity to experimental/research conditions.
•Interaction effects of extraneous factors and experimental/
research treatments.
•Invalidity or unreliability of instruments.
•Ecological validity.
•Multiple treatment validity.

24. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
THE HAWTHORNE EFFECT
•Date
–Between 1927 and 1932 researchers carried out experiments at the
Western Electric Company’s Hawthorne plant.
•Purposes
–To examine the effects of changes to working conditions on output of
workers
•Sample
–Six women, chosen as average workers
•Method
–Women worked in a test room. Output measured under different
conditions (e.g. no change → change to method of payment →
introduce two rest periods → introduce six rest periods → changes in
lighting conditions, early clocking-off, five-day working week → return
to initial conditions)
•Duration
–15 weeks

25. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
THE HAWTHORNE EFFECT
Results:
•Output rose steadily during test period and after the test
period.
Conclusion:
•Output did not seem to depend on test conditions. Increased
output seemed to be due to the fact that the people had been
involved in the experiment itself, i.e. the act of research had
affected the results. The results were a research of the
research itself.
Implications:
•The act of being involved in research itself affects the results.

26. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
THREATS TO EXTERNAL VALIDITY
IN QUALITATIVE RESEARCH
•Selection effects
•Setting effects
•History effects
•Construct effects

27. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
ENSURING VALIDITY AT THE DESIGN STAGE
•Choose an appropriate timescale
•Ensure adequate resources for the research
•Select appropriate methodology
•Select appropriate instruments
•Use an appropriate sample
•Ensure reliability
•Select appropriate foci
•Avoid having biased researcher(s)

28. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
ENSURING VALIDITY AT THE DATA-COLLECTION
STAGE
•Reduce the Hawthorne effect
•Minimize reactivity
•Avoid dropout rates amongst respondents
•Take steps to avoid non-return of questionnaires
•Avoid too long or too short an interval between pre-tests and
post-tests
•Ensure inter-rater reliability
•Match control and experimental groups
•Ensure standardized procedures for gathering data
•Build on the motivations of respondents
•Tailor instruments to situational factors
•Address researcher characteristics

29. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
ENSURING VALIDITY AT THE DATA-ANALYSIS STAGE
•Use respondent validation
•Avoid subjective interpretation of data
•Reduce the halo effect
•Use appropriate statistical treatments
•Recognize extraneous factors which may affect data
•Avoid poor coding of qualitative data
•Avoid making inferences/generalizations beyond the data
•Avoid equating correlations and causes
•Avoid selective use of data
•Avoid unfair aggregation of data
•Avoid degrading the data
•Avoid Type I and/or Type II errors

30. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
ENSURING VALIDITY AT THE REPORTING STAGE
•Avoid using data selectively and unrepresentatively
•Indicate the context and parameters of the research
•Present the data without misrepresenting the message
•Make claims which are sustainable by the data
•Avoid inaccurate or wrong reporting of data
•Ensure that the research questions are answered
•Release research results neither too soon nor too late

31. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
RELIABILITY
•Reliability is an umbrella term for dependability,
consistency and replicability over time, over
instruments and over groups of respondents.
•Can we believe the results?
•Can we trust the results?
•Reliability is concerned with precision and accuracy.
•Reliability is concerned with consistency.

32. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
RELIABILITY IN QUANTITATIVE
AND QUALITATIVE RESEARCH
Reliability in quantitative research
•consistency (stability), accuracy, predictability,
equivalence, replicability, concurrence, descriptive
and causal potential.
Reliability in qualitative research
•accuracy, fairness, dependability,
comprehensiveness, respondent validation,
‘checkability’, empathy, uniqueness, explanatory
and descriptive potential, confirmability.

33. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
RELIABILITY IN QUANTITATIVE RESEARCH
Reliability as stability
•Consistency over time and samples
Reliability as equivalence
•Equivalent forms of same instrument;
•Inter-rater reliability;
Reliability as internal consistency
•Split half reliability (e.g. for test items)

34. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
TRIANGULATION
•Paradigms
•Methodologies
•Instruments
•Researchers
•Time
•Location
•Theories
•Samples
•Participants
•Data

35. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
SPLIT-HALF RELIABILITY
(Spearman-Brown)
Reliability =
r = the actual correlation between the two halves of
the instrument;
Reliability = = = 0.919r
r
1
2 85.01
)85.0(2
 185
70.1

36. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
RELIABILITY IN QUALITATIVE RESEARCH
•Credibility
•Neutrality
•Confirmability
•Dependability
•Consistency
•Applicability
•Trustworthiness
•Transferability

37. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
RELIABILITY AND REPLICATION
IN QUALITATIVE RESEARCH
Repeat:
•The status position of the researcher
•The choice of informants/respondents
•The social situations and conditions
•The analytic constructs used
•The methods of data collection and analysis
Address:
•Stability of observations
•Parallel forms
•Inter-rater reliability
•Respondent validation

38. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
IMPROVING RELIABILITY
Minimiseexternal sources of variation
Standardise conditions under which measurement occurs
Improve researcher consistency
Broaden the sample of measurement questions by:
•adding similar questions to the instrument;
•increasing the number of researchers (triangulation);
•increasing the number of occasions in an observational study.
Exclude extreme responses (outliers)

39. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
RELIABILITY AND VALIDITY AT ALL STAGES
•Design and methodology
•Sampling
•Instrumentation
•Timing
•Data collection
•Data analysis
•Data interpretation
•Reporting