Clinical lab-method validation.ppt

How to proceed with method validation in clinical chemistry

1. Method Validation
To understand College of American Pathologists
(CAP) requirements of method validation in a clinical
laboratory
Dr. SibtainAhmed

2. What is validation?
•“Validationistheprocessoftestingameasurement
proceduretoassessitsperformanceandtodetermine
whetherthatperformanceisacceptable.”[CLSI]

3. Method Validation
Why must we validate?
When should we validate?
What should we validate?
What is method validation?

4. What is Method Validation?
–Method validation and/or verification is the process
by which a method is determined to be fit for
purpose and intended use.
–Although method validation/verification are often
used interchangeably, validation is usually
performed on in-house and/or modified methods,
while verification is taking a marketed/unmodified
assay and verifying its performance.

5. Purposes of Method Validation Studies
To quantifiably characterize system performance
To assess potential for error
To identify method-to-method differences
To meet regulatory guidelines

6. When to Do Method Validation Studies
When considering purchasing a new system
When placing a new system into service
At regular intervals to assess on-going system performance
When troubleshooting questionable system performance

7. Tests to Validate
•Waived tests are approved by the FDA for home use and by definition are
simple to perform (i.e., pregnancy test) –do not require validation.
•Non-waived tests included moderately and highly complex tests –requires
validation
•Degree of validation/verification depends on status of the test method (FDA-
approved/Non-FDA-approved/Modified/Unmodified)
•Unmodified FDA-approved: using as intended by the manufacturer and
licensed for use by the FDA
•Modified or Non-FDA approved: using test kit for indications other than as
intended by the manufacturer; not licensed for use by the FDA.

8. The 6 Elements of Method
Validation:
•If FDA-approved/Unmodified
–only Reportable Range,
Precision, Accuracy and
Reference Intervals need to be
verified.
•If Non-FDA
approved/Modified –all 6
elements must be performed,
including sensitivity and
specificity.

9. samplesareanalyzedaccordingtoanynumberofschemesdepending
upontheanalyteavailability:
Twiceadayinduplicatefor10days.
Onceadayfor20days.
Twiceadayinquadruplicatefor5days.
Selectsufficientmaterialattwolevels:
Patientsamples
QCmaterial
Referencematerials
Standards
Suitablematerialsthathaveaknownvalue.
Precision
Multiple measurements are close to each others
i.e. Reproducibility
Gives information related to random error
Introduction
What is
needed
How we
perform the
testing

10. Experiment Design:
Run one sample 20 time once a day for within run precision and for between run
precision run one sample 2-4 times per day.
Data Input Requirements:
20 reading of single specimen,
allowable total error
allowable random error
Statistics:
MEAN
SD
CV
2SD RANGE
SD GOAL
Interpretation :The precision test pass if computed SD below the SD goal.

11. Accuracy
How close to the true value Comparison of methods
Closeness of the agreement between the result
of a measurement and a true value of the
measurand. [CLSI]
Gives information related to systematic error
Introduction
40 different specimens
Cover reportable range of method
What is
needed
Duplicate measurements of each specimen
on each method
Minimum of five days, prefer over 20
(since replicate testing is same)
How we
perform the
testing

12. Accuracy studies
Test results from the new
method against an existing
method which is known to
be accurate
Two or more samples in which the concentration of the
selected analyte concentrations spans but does not exceed
the analytical measuring interval of the assay.
Comparison of Method/ Instrument
Recovery of expected values
from assigned values

13. what is needed for accuracy studies ?
Acontrolsampleorcertifiedreferencematerial
Proficiencytestingmaterial
Materialsprovidedbythemethodmanufacturerforverificationof
truenessorqualitycontrol.

14. Method comparison
•WHEN IT IS
PERFORMED?
To validate a new method being brought into
the lab, by demonstrating that it is
statistically identical to the method currently
in use.
HOW IT IS PERFORMED?
Assay about 20-40 patient samples on
both instruments. The specimens should
be somewhat evenly-distributed across
the analytesreportable range.

15. Purpose: To validate a new method being brought into the lab, by demonstrating that it is
statistically identical to the method currently in use.
Experiment Design Requirements:
Run 20-40 specimens that cover reportable range properly on both instruments.
Statistics:
Correlation coefficient(R)
SLOPE
INTERCEPT
Interpretation:
CORELATION COFICIENT SHOULD BE >0.9
95% confidence interval for the Deming Slope includes 1.000
95% confidence interval for the Deming Intercept includes 0.000
Alternate method comparison

16. Purpose:
The Two Instrument Comparison (2IC) procedure will satisfy the CLIA requirement for a lab to compare
instruments performing the same test on a semi-annual basis
Experiment Design:
Select 5-10 Specimens that cover reportable range and run on both instruments one time.
Data Requirements:
5-10 reading of specimens
Total allowable error
Reportable range
Statistics:
Average error index
Slope
Intercept
standard error estimate
Interpretation:
Test passes, when error index of each pair (x-y) is between 1.0 to -1.0
Test fails ,when error index of any pair greater than 1.0 or less than -1.0.
Two instrument comparison

17. Purpose: Multiple Instrument Comparison (MIC) compares 3or more instruments to determine whether
they are within Total Allowable Error (TEa) of the target.
Experiment Design Requirements:
Select 3-19 instrument and run 3-5 specimens on each instrument that cover reportable range of test.
Select one instrument as a target instrument.
Data Requirements:
3-5 reading of each instrument
Total allowable error
Reportable range
Statistics:
OBSERVE ERROR
ERROR INDEX
ALLOWABLE ERROR
Interpretation:
The experiment passes if every non target instrument is within error index limit of -1 to 1 .
Multi instrument comparison

18. Reportable Range / Linearity
Lowest and highest test results that are reliable
Especially important with two point calibrations
Analytical Measurement Range (AMR) and derived Clinical Reportable Range (CRR)
Introduction
What is
needed
How we
perform the
testing
Material manufactured specially for linearity checking.
CAP linearity survey or validation materials
Quality control or calibration material.
Single Patient samples.
Pool patient sample.
Spiked patient samples
Run 3-11 (5 is sufficient) specimens in triplicate with known concentration
spread across the reportable range; ideally the lowest and highest
specimen should challenge the limit of range.

19. Verifying reportable range
•AMR must be verified before method introduced, and at least
every 6 months (and after recalibration or major maintenance)
while in use
•CRR is a clinical decision by lab director, and does not require
experiments or re-validation; however, dilution or
concentration protocols must be specified in methods

20. How we evaluate linearity ?
Reportable/Analyticalmeasuringrange(linearity)isverifiediftwo
conditionsaremet:
Assignedvaluesofthelowestandhighestspecimenshouldbe
withinproximitylimitsofthereportablerangelimit.
Thesetwospecimensareacceptablyaccuratewithrespecttototal
allowableerror.

21. Experiment Design:
Select 3-5 specimens that cover reportable range/AMR and run 3-5 times each specimen
Data In put Requirements:
Low and high reportable range of analyte
Proximity limits
3-5 reading of each specimens
Target value of each specimen(mean)
Allowable error
Statistics
Mean
Slope
Intercept
Error
Percent
Recovery
Interpretation:
The midpoints of target ranges for the lowest and highest specimens respectively are with in
the proximity limit s of the reportable range limits
These two specimens also pass accuracy.
The method is linear if it is possible to draw straight line that passes within a user defined
allowable error of each specimen point.

22. Reference Intervals
Normal range in healthy population
Used for diagnosis/clinical interpretation of results
Introduction
•Test and compare to manufacturer: If all 20 specimens fall within the specified range, the
reference ranges have been verified and you can adopt manufacturer suggestions and
incorporate into your SOP. If not, need to establish reference ranges for your population
(240 specimens: 120 each of male and female).
•Another lab may have established reference ranges that can be used for verification
purposes; must be documented.
What is
needed
How we
perform the
testing
Enter the readings in EP evualtor along with, the
"proposed" reference interval that is to be verified.

23. PURPOSE: Reference interval verification with respect to population
Experiment Design Requirements:
Select 20 specimens of healthy individuals and run one time per specimen
Data Requirements:
20 reading of health individual specimens
Proposed Reference interval range
Statistics:
MEAN
SD
MEDIAN
RANGE
Frequency
Interpretation:
If more than 10% of results are out side the proposed reference interval which mean verification fail,
otherwise passes.

24. Sensitivity
Lowest reliable value; lower limit of
detection, especially of interest in drug testing
and tumor markers
Different terminologies used by different
manufacturers
Introduction
Blanksolution:(oftenusethezerocalibrator)
Lowlevelofpositivesample(slightlyabove
themanufacture'sstatedlowestdetectionlimit)
What is
needed
Run 20 blanks or low level samples;
How we
perform the
testing

25. Purpose: To verify the vendor claim sensitivity
Experiment Design Requirements:
Replicate response measurements are made for two specimens
blank or zero concentration sample minimum 10 replicate
And 2
nd
sample of known low concentration minimum 3 replicate
Data Requirements:
10 reading of zero concentration sample
3 reading of low concentration sample
manufacturer’s claim sensitivity
Statistics:
Mean
SD
2 SD 95% confidence interval for Limit of Blank
Interpretation:
Manufacture sensitivity claim is verified when the computed LOB is less than manufacturer ‘s
claim.

26. Specificity
Determination of how well a method measures the
analyte of interest accompanied by potential
interfering materials
Introduction
Standard solutions, participant specimens
or pools
Interferer solutions (standard solutions, if
possible; otherwise, pools or specimens)
added at high concentrations
What is
needed
Duplicate measurements.
The lab can use the manufacturer’s provided list
How we
perform the
testing

27. Specificity: How We Evaluate the Data
Tabulate results for pairs of samples
(dilution and interferent)
Calculate means for each (dilution
and interferent)
Calculate the differences
Calculate the average interference of
all specimens tested at a given
concentration of interference

28. Carry Over
Carryover” in laboratory testing, is defined as “the
contamination of a specimen by the previous one”.
Select specimens for the analytes to be tested.
Select one patient/control specimen with a very low level of
the analyte and one with a very high level.
Select specimens with enough volume so that they can each be
run approximately 10 to 11 times.
WHAT IS NEEDED ?

29. when to perform carry over studies?
Study must be performed, as applicable, as a part of initial
evaluation of an instrument.
Study is repeated after major maintenance or repair of the
pipetting assembly of instrument.
Evaluation for carryover is not required for automatic
pipettes that use disposable tips.
The instrument must be dedicated to the analysis of one
analyte during this experiment and must do this experiment
for one analyte at a time.
Routine performance of tests must be suspended.

30. How to perform carry over?
Run in following order :
3 Low specimens
2 High specimens
1Low specimen
2 High specimens
4 Low specimens
2 High specimens
1 Low Specimen
2 High specimens
1 Low Specimen
2 High specimens
1 Low Specimen

31. Experiment design: Twospecimen are prepared ,one with very high concentration and one with very
low concentration .these specimens are aliquot out in to a total of 21 samples :11 with low and 10 with
high concentration.
Run sample in following sequence on analyzer.
LLLHHLHHLLLLHHLHHLHHL
Data Requirements:
21 reading of low and high specimen as above sequence.
Statistics
SD
Mean
Error Limit
Carry over
carry over: The mean of H-L results minus the mean of L-L results
Error limit: Three times the SD of L-L results
Interpretation :Carry over test passes if carry over is less than error limit.

32. Validation for qualitative tests:
Precision
Accuracy

33. Precision for qualitative assay
Runs of specimens with analyte concentrations near the cutoff
point .
Three specimens, one at cutoff, one just below cutoff, and one
just above cutoff (±20% recommended) .
Replicate measurements of each of three specimens (20 each,
minimum).

34. How to evaluate?
The results obtained from the negative and positive controls
will be used to calculate the CV and compared to the
manufacturer’s claims for reproducibility.
The laboratory CV should be less than or equal to the
manufacturer’s stated CV. In the event that an assay does not
perform as expected

35. What is needed for accuracy studies for
qualitative testing?
Samples with known values, such as :
proficiency testing samples or commercial standards, may be
used as the reference method.
use of EQA samples or other commercially prepared
reference material with known values.

36. accuracy/method comparison for qualitative
testing
What is needed ?
•A minimum of 10 samples for each expected result.
For example, if a test method gives results of
“Positive/Negative”, the accuracy study must include 10
known positives and 10 known negatives.
•Two levels of quality control must be run each day that testing
is performed, not including controls internal to the kit
cartridge/testing device.

37. How it is performed?
•The performance of qualitative tests is most commonly
described in terms of sensitivity and specificity.
•By using a contingency table:
Method being Validated
Diagnostic Sensitivity and Specificity
(Results from Comparison Study) Total
Positive Negative
Positive # true positive (TP)# false positive (FP)TP+FP
Negative # false negative (FN)# true negative (TN)FN+TN
Total TP+FN FP+TN N

38. How it is evaluated?
Compare the results calculated above with the manufacturer’s
stated claims for Sensitivity, Specificity and Agreement found
in the test kit package insert.
Results must be equal to, or greater than, the manufacturer’s
claims for the method to be considered accurate

39. PURPOSE: To compare two qualitative method are correlated to each other
Experiment Design Requirements:
Run 20-40 specimen from both method which consist off positive ,negative and borderline results
equally.
Data Requirements:
20-40 reading of specimens
Statistics:
Total agreement
Positive agreement
Negative agreement
Cohen kappa agreement
Interpretation:
Cohen kappa agreement should be >75% passes.
Total agreement should be >90%
qualitative/semi qualitatine method comparision

40. Thank you !