Subjectivity Subjectivity Subjectivity Subjectivity

Subjectivity Subjectivity Subjectivity

1. Quality Risk Management, ICH Q9(R1) Training Slides  Subjectivity in Quality Risk Management PART II:  What Subjectivity in Quality Risk Management is - Background and Theory International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use

2. Legal Notice This presentation is protected by copyright and may, with the exception of the ICH logo, be used, reproduced, incorporated into other works, adapted, modified, translated or distributed under a public license provided that ICH's copyright in the presentation is acknowledged at all times. In case of any adaption, modification or translation of the presentation, reasonable steps must be taken to clearly label, demarcate or otherwise identify that changes were made to or based on the original presentation. Any impression that the adaption, modification or translation of the original presentation is endorsed or sponsored by the ICH must be avoided. The presentation is provided "as is" without warranty of any kind. In no event shall the ICH or the authors of the original presentation be liable for any claim, damages or other liability arising from the use of the presentation. The above-mentioned permissions do not apply to content supplied by third parties. Therefore, for documents where the copyright vests in a third party, permission for reproduction must be obtained from this copyright holder. ​

3. Agenda This training material on Subjectivity in Quality Risk Management (QRM) is divided into two different presentations: Part I: how to Identify and Control Subjectivity   Key points from ICH Q9(R1) regarding Subjectivity in QRM Why Subjectivity in QRM can be a problem The Benefits of Controlling Subjectivity Practical Tips & Examples Controlling and minimizing subjectivity across the four elements of QRM Risk Assessment Risk Control Risk Communication Risk Review Case Study Key take-home points Part II: what Subjectivity in Quality Risk Management is – Background and Theory Key points from ICH Q9(R1) regarding Subjectivity in QRM The role of Uncertainty, Heuristics and Bias in relation to Subjectivity Risk Perception Factors How Subjectivity can influence the activities that make up the QRM process Subjectivity in R isk Scoring Scales/Models Key take-home points

4. Subjectivity in QRM – Key Points from ICH Q9(R1) 4 Question: ​ The original (2005) version of ICH Q9 did not specifically address subjectivity in QRM. Why was the topic of subjectivity identified for elaboration in ICH Q9(R1)? Answer:​ Highly subjective risk scoring methods and differences in how risks are assessed and how hazards, risks, and harms are perceived by different stakeholders can lead to varying levels of effectiveness in the management of risks. (Ref. ICH Q9(R1) Concept Paper, November 2019) Subjectivity can directly impact the effectiveness of risk management activities and the decisions made. Therefore, it is important that subjectivity is managed and minimized.  (See ICH Q9(R1), Introduction)

5. Key Points from ICH Q9(R1), cont’d 5 Question:   What are the responsibilities of decision-makers in QRM with regard to subjectivity ? Answer:  Decision-makers should assure that subjectivity in quality risk management activities is managed and minimized, to facilitate scientifically robust risk-based decision-making. (See ICH Q9(R1), Section 4.1, Responsibilities)

6. 6 Question: ​ What stages of QRM are impacted by subjectivity and how is it introduced into QRM? Answer:​ Subjectivity can impact every stage of a quality risk management process, especially the identification of hazards and the estimation of probability of occurrence and severity of harm. It can also impact the estimation of risk reduction and the effectiveness of decisions made from quality risk management activities. (S ee ICH Q9 (R1), Section 5.3, Managing and Minimizing Subjectivity) Subjectivity can be introduced in quality risk management through differences in how risks are assessed and in how hazards, harms and risks are perceived by different stakeholders, (e.g., bias). Subjectivity can also be introduced when risk questions are inadequately defined, and when tools have poorly designed risk scoring scales. (S ee ICH Q9 (R1), Section 5.3, Managing and Minimizing Subjectivity) Key Points from ICH Q9(R1), cont’d

7. 7 Question: ​ Can subjectivity be eliminated from QRM? What needs to be done? Answer:​ While subjectivity cannot be completely eliminated from quality risk management activities, it may be controlled by addressing bias and assumptions, the proper use of quality risk management tools and maximizing the use of relevant data and sources of knowledge. (See ICH Q10, Section 1.6.1) All participants involved with quality risk management activities should acknowledge, anticipate, and address the potential for subjectivity. (See ICH Q9(R1), Section 5.3. Managing and Minimizing Subjectivity) Key Points from ICH Q9(R1), cont’d

8. Why Subjectivity in QRM can be a problem Hazards may not be identified appropriately, and as a result, the risks associate with certain hazards may go unmanaged. When risks presented by hazards are being assessed, the probabilities of occurrence and the severities of harm that are arrived at may be underestimated or overestimated.  In addition, subjectivity in assumptions about the value or effectiveness of certain risk controls can result in an over estimation of the extent of risk reduction that they deliver – this can lead to a ‘false sense of security’ in those controls. 8 Subjectivity in QRM can lead to biased risk assessment outcomes, and these can give way to sub-optimal risk controls, poor risk-based decisions, and ineffective QRM outcomes generally.  !

9. There are many definitions for Subjectivity , and it is described in various ways. Some indicate that it is based on, or influenced by, personal feelings, taste, or opinions. Others describe it as relating to the influence of personal feelings or opinions when considering and representing facts. It is often contrasted with Objectivity , which is described as a view of truth or reality that is free of any individual's biases, interpretations, feelings, and imaginings . This presentation discusses a number of factors that can give rise to Subjectivity in QRM outputs – namely : Uncertainty, Heuristics, Bias, and Risk Perception. Note: These factors are inter-related and are not completely separate from each other. What is Subjectivity? 9

10. What leads to Subjectivity in QRM? Uncertainty, Heuristics, Bias SUBJECTIVITY This relates to one’s personal interpretations, impressions, feelings, perceptions, preferences, experiences, exposure to information, etc. There can be subjectivity in professional judgments, e.g., when data are being interpreted, as well as in value judgments, e.g., when deciding which hazards matter the most in a risk assessment, etc. I t is important to be aware of the factors that introduce subjectivity into QRM activities, and to minimize their adverse effects ​ UNCERTAINTY I n a QRM context, this is considered to be a lack of  knowledge about hazards, harms and, consequently, their associated risks. The ICH Q9(R1) Guideline states that t ypical sources of uncertainty include: gaps in knowledge gaps in pharmaceutical science and process understanding gaps in the sources of harm (e.g., failure modes of a process, sources of variability) and the probability of detection of problems HEURISTICS These are regarded as mental shortcuts, simplifications, or cognitive ‘rules of thumb’ that allow people to make decisions and solve problems quickly. But they can also be a source of bias and lead to errors in judgment.  . BIAS and COGNITIVE BIAS  Bias is considered to be any systematic error. Cognitive biases are considered to be systematic errors associated with thought processes. They are sometimes described as systematic and universally occurring tendencies, inclinations or dispositions that skew or distort information processes in ways that make their outcome inaccurate, suboptimal, or simply wrong. 10 Risk Perception Factors

11. Some degree of uncertainty is generally unavoidable when performing Risk Assessments, given the accepted probabilistic nature of risk ICH Q9(R1) defines risk as the combination of the probability of occurrence of harm and the severity of that harm. Unless the source of the hazard or harm is eliminated, uncertainty cannot be avoided when one tries to assess and manage the resulting risks. ICH Q9(R1) states that uncertainty is due to the combination of: incomplete knowledge about a process its expected or unexpected variability Typical sources of uncertainty include: gaps in knowledge gaps in pharmaceutical science and process understanding gaps in the sources of harm (e.g., failure modes of a process, sources of variability) the probability of detection of problems The role of Uncertainty… 11

12. Heuristics are considered to be cognitive processes that come into play when people make judgments under uncertainty – i.e. when information is lacking They also occur when there are information processing limitations in place Heuristics are regarded as mental shortcuts , or 'rules of thumb' that reduce complex problems down to simple, faster altternatives. They are commonly employed, often unknowingly, when arriving at estimates or making decisions when there is uncertainty. Heuristics can lead to quick decisions which can, in some situations, be beneficial In other situations, they can introduce bias and lead to inaccurate assumptions, decisions and judgements.  The role of Heuristics… 12 Note : In the literature, the terms ‘Bias’ and ‘Heuristics’ are sometimes used interchangeably – e.g., some publications refer to heuristics as a kind of bias , whilst others refer to heuristics as a source of bias . In this training material, heuristics are considered to give rise to biases.

13. Much of today‘s understanding of heuristics and their resulting biases can be traced back to the influential work of Kahneman and Tversky in the early 1970s Kahneman and Tversky demonstrated that human judgements often depart substantially from normative standards based on probability theory or simple logic. They referred to heuristics as mental shortcuts that reduce the complex tasks of assessing probabilities and predicting values to simpler judgemental operations. The Heuristic of Representativeness is one such example – this is where the way in which A resembles or is representative of B influences the judgements people make. Heuristics, cont’d… 13 Example: With regard to tossing a coin 6 times, people usually assess the sequence HTHTTH as being more likely than HHHTTT or HHHHTH . This is because the sequence HTHTTH appears more representative of randomness than the other two sequences. The ‘Gambler’s Fallacy’ is another example of this heuristic – the more bets lost, the more the gambler feels a win is now due, even though each new turn is independent of the last . Ref. Kahneman, D., Tversky, A., ‘Subjective Probability: A Judgement of Representativeness, Cognitive Psychology, 3:430-354, 1972

14. 14 Is a knowledge of Heuristics relevant to Quality Risk Management? Yes, as knowing about heuristics is important when trying to minimise subjectivity in QRM outputs. Risk Assessment is, by definition, probabilistic. As explained by Kahneman and Tversky (1972), numerous investigations have found that people “do not follow the principles of probability theory in judging the likelihood of uncertain events”. They explained how this was “hardly surprising, because many of the laws of chance are neither intuitively apparent, nor easy to apply”. Apparently, “people replace the laws of chance by heuristics, which sometimes yield reasonable estimates and quite often do not.” Heuristics, cont’d… Ref. Kahneman, D., Tversky, A., ‘Subjective Probability: A Judgement of Representativeness, Cognitive Psychology, 3:430-354, 1972

15. The Encyclopaedia of Behavioural Neuroscience, 2nd edition (2021), indicates that Heuristics can come into play when dealing with data-limitations – such limitations may relate to information processing limitations, a lack of expertise, etc. This is important from a Risk Assessment perspective, because so many risk assessment tools and methodologies require probability of occurrence ratings to be assigned to hazards, failure modes, harms, etc., and the data needed to arrive at accurate estimates are often limited. Fortunately, t here has been extensive research performed into the ways that heuristics can influence probability of occurrence estimates. Examples include*: The Heuristic of Anchoring and Adjustment The Heuristic of Availability The Heuristic of Representativeness Ref . Encyclopaedia   of Behavioural Neuroscience, 2nd edition, Ed. S. D. Sala, Elsevier, September 2021 * For simple strategies that can be used in the pharmaceutical environment to counteract the effects of these heuristics, see O’Donnell, K., Strategies for Addressing the Problems of Subjectivity & Uncertainty in Quality Risk Management Exercises: Part I: The Role of Human Heuristics, J. Val. Technol. 2010, Summer, 76-84 Heuristics, cont’d 15

16. There is much written in the literature about bias: A bias is considered to be a systematic error. Some sources suggest that bias is an inclination or prejudice for or against one thing or person. Biases can be innate or learned. People may develop biases for or against an individual, a group, or a belief (Steinbock & Bonnie, 1978). There are many different types of biases. The role of Bias… 16 Cognitive Bias is the main kind of bias addressed in this training material. This is discussed on the next few slides. Ref . Steinbock & Bonnie, ‘Speciesism and the Idea of Equality’. Philosophy. 53 (204): 247–256, 1978

17. Korteling et al (2021) refer to cognitive biases as “systematic, universally occurring tendencies, inclinations, or dispositions that skew or distort information processes in ways that make their outcome inaccurate, suboptimal, or simply wrong.” Hu et al (2021) describe cognitive biases as “systematic patterns of deviation from norm or rationality in judgment, which affect us in many areas of life, such as social situations, memory recall, what we believe, and our behaviors .” Examples of some common cognitive biases are shown on the next few slides. Cognitive Bias 17 Cognitive Biases are considered to be ‘systematic cognitive dispositions or inclinations in human thinking and reasoning that often do not comply with the tenets of logic, probability reasoning, and plausibility.’ (Ref. Encyclopaedia of Behavioural Neuroscience, 2nd edition, 2021) Similar descriptions are provided in other publications. Ref . Encyclopaedia   of Behavioural Neuroscience, 2nd edition, Ed. S. D. Sala, Elsevier, September 2021 Ref . Korteling   et al., ‘Human versus Artificial Intelligence’, Artif . Intell , 25 March 2021 Ref . Hu et al, ‘Cognitive Biases in Understanding Shale Gas Exploration’, Oct 2021

18. Examples of Cognitive Biases 18 BIAS: What this means or may result in: ANCHORING BIAS People sometimes over-rely on the first piece of information they hear. This can give rise to b iased decisions toward previously acquired information. (Note: This bias is sometimes referred to as Anchoring and Adjustment , and it is sometimes called a heuristic.) AVAILABILITY BIAS This relates to the tendency to judge the frequency, importance, or likelihood of an event by the ease with which relevant instances of it come to mind. People sometimes overestimate the importance of information that is readily available to them. PRESENT BIAS The tendency to opt for a smaller present reward today than to wait for a larger future reward, in a trade-off situation. RECENCY BIAS The tendency to weigh the most recent information more heavily than older data.  KNOWLEDGE/ STATUS QUO BIAS The tendency to systematically prefer the current state of affairs or what is well-known over the unknown. GROUP-THINK BIAS Th e practice of thinking or making decisions as a group – this can result in unchallenged, poor-quality decision-making. (This is also described as the psychological phenomenon that occurs within a group of people in which the desire for harmony or conformity results in irrational or dysfunctional decision-making outcomes.) IN-GROUP BIAS The tendency to favor one’s own group above that of others. This can result in a tendency for people to give preferential treatment to others they perceive to be members of their own group.

19. Examples of Cognitive Biases, cont’d 19 BIAS: What this means or may result in: CONFIRMATION BIAS The tendency to select, interpret, focus on and remember information in a way that confirms one’s preconceptions, views or expectations. CONJUNCTION FALLACY The tendency to consider a combination of conditions more likely than only one of those conditions. This can result in an assumption that specific conditions are more probable than a more general version of those same conditions. SUNK-COST FALLACY The tendency to consistently continue a chosen course or an investment with negative outcomes rather than alter it. Sometimes, people justify increased investment in a decision, based on the cumulative prior investment. BANDWAGON EFFECT The tendency to adopt beliefs and behaviors when they have already been adopted by others. LOUDEST VOICE BIAS The tendency for loudest voices in a meeting to dominate discussions, driving away other participants who have as much (or more) to contribute, but don't want to participate in a shouting match. FUNDAMENTAL ATTRIBUTION ERROR The tendency for people to over-emphasize personality-based explanations for behaviours observed in others while under-emphasizing the role and power of situational influences on the same behaviour. INFORMATION BIAS The tendency to seek information when it does not impact the outcome. OUTCOME BIAS The tendency to evaluate a decision based on its outcome rather than on the factors that led to the decision. This is essentially about judging the quality of a decision by how well things turned out, without considering how thoughtfully those decisions were made.

20. 1 Anchoring Bias What would be the severity score for this failure mode? 20 People can be over-reliant on the first piece of information they hear, especially if it comes from a leader. No anchoring in place Manager Operators anchor to their manager 9 3 Operator 2 Operator 1 5 4 Operator I Operator 2

21. Availability Bias I have never seen this contamination event in my experience. This is a low risk. I’ve seen this contamination event at 2 other companies I audited. This is not a low risk. 21 Things that come readily to mind are more representative than is actually the case. People overestimate the importance of information that is readily available to them. Expert II Expert I

22. Recency Bias !? I have worked through 3 FMEAs recently. I am sure that the risk ratings in this new one are going to be similar on average. 22 The tendency to weigh the most recent information more heavily than older data. Operator I Operator II

23. Knowledge/Status Quo Bias !? I always focus on QC when a process investigation has to be carried out. I feel knowledgeable in a Lab and I am sure we will find valuable information there 23 To systematically prefer what is well-known over the unknown. We should consider a 6M approach with a multidisciplinary Team instead QA Staff Member I QA Staff Member II

24. Confirmation Bias !? I know this equipment very well. It is not worth spending additional time there for our cross-contamination issue. 24 The tendency to search for, interpret, focus on and remember information in a way that confirms one’s preconceptions. There have been recent changes to the procedure for using this item of equipment that are not being considered! Engineer I Engineer II

25. 25 In addition to Uncertainty, Heuristics and Bias , it is also useful to consider the influences of perception when dealing with hazards and risks, and how perception can lead to subjectivity. The following slides present some points in this regard.

26. ICH Q9(R1) states the following in relation to perception: It is commonly understood that risk is defined as the combination of the probability of occurrence of harm and the severity of that harm. However, achieving a shared understanding of the application of risk management among diverse stakeholders is difficult because each stakeholder might perceive different potential harms, place a different probability on each harm occurring and attribute different severities to each harm. Risk perception is also discussed in the ISO 31000 Risk Management Standard: It explains how stakeholders form judgements about risks based on their own perceptions of those risks. In this context it refers to differences in values, needs, assumptions and concerns. Risk Perception 26 Factors relating to perception can make it difficult to reach agreement on the acceptability of a risk, or on the suitability of a course of action proposed to address the risk. In addition, h ow hazards and risks are perceived is complicated by the influence of psychological and cognitive factors. Factors relating to perception can make it difficult to reach agreement on the acceptability of a risk, or on the suitability of a course of action proposed to address the risk. In addition, h ow hazards and risks are perceived is complicated by the influence of psychological and cognitive factors.

27. R isk Perception has long been an important area of research for psychologists and others across various disciplines Litai’s work in the late 1970s uncovered a lot about how people perceive hazards and their related risks. He produced a listing of quantifiable ‘risk factors’ that related to risk perception. These risk factors were shown on a dichotomous scale - a two-point scale that presents options that are opposite each other. This list indicated how differently hazards and risks are perceived when they meet certain criteria, such as whether a hazard is man-made or natural, whether it is old or new, whether its effects are ordinary or catastrophic, whether a hazard is voluntarily assumed or not. Litai’s research found that: The public’s acceptance of risks from hazards that could be considered voluntarily assumed was up to 100 times higher than when the hazards were considered involuntarily a ssumed. Risk Perception, cont’d 27 Example of a voluntarily assumed hazard: Smoking cigarettes Example of an involuntarily assumed hazard: H igh voltage electricity lines near one’s home Ref. Litai, D. ‘A risk comparison methodology for the assessment of acceptable risk’, PhD Thesis, Massachusetts Institute of Technology, Cambridge, Mass., 1980

28. ‘ Dreadfulness’ and other factors: Research by Fischhoff et al indicated that risk factors can be grouped into three main categories The degree of ‘dreadfulness’ associated with the issue The degree to which the risk was understood The number of people exposed to the risk in question These categories can then be used to define what is called a ‘Risk Space’: When a hazard comes within this space, a person’s perception of the risk tends to be significantly affected than when a hazard is outside this space. Risk Perception, cont’d 28 Consider the potential risks presented by glass particulates in glass vials. The above research work is useful to know about when assessing and managing such risks. Ref . Fischhoff, B., Slovic , P., Lichtenstein, S., Fault trees: sensitivity of estimated failure probabilities to problem representation, J. Exp. Psychol. – Human Perception Perf. 4:330-44, 1978

29. The glass particles example… The potential risks presented by glass particles in glass vials (i.e. injectable medicines) might fall into Fischhoff et al‘s Dreadfulness category. And in accordance with Litai’s Risk Factors , these risks may also be characterised as involuntarily assumed risks , with perceived catastrophic consequences. While the known occurrence rates of glass particles in such products are usually low, these kinds of risks can be subject to problems of mis-perception among stakeholders. Risk Perception, cont’d 29 So how might such problems of risk perception be counteracted? There are various things that can be done… see next slide

30. When documenting such risks and when communicating them, one can address the following: The known incidence rates of glass particles in filled vials… And the data that support those rates. How glass particulates are controlled at a practical level during a) empty vial manufacture, b) vial handling & depyrogenation , c) vial filling, stoppering, capping, etc. Any design features in the drug product manufacturing process that can reduce the incidence of glass particles in the filled and sealed vials... e.g., the empty vials are inverted prior to depyrogenation . Any design features in the finished drug product that can reduce the consequences of glass particles, if they are present in the product (e.g., a filtered needed is supplied with the filled vials and this may capture any glass particles that are present). The known effectiveness of such controls and design features. Any important assumptions and sources of uncertainty in the risk assessment.  Risk Perception, cont’d 30 Doing these things can help ensure that people are better informed about the risks in question and about how those risks are managed. This can reduce the problems presented by risk perception factors.

31. Subjectivity in Hazard Identification People with similar experiences may perceive differences when identifying hazards in a risk assessment. All hazards should be identified, including those that may already be controlled. Surely the most prevalent hazard is that contamination is transferred outside the Weigh & Dispense Room and to personnel PPE. I have seen it before.  We do not need to address this, we are trained to be careful and that would never happen Even Subject Matter Experts (SMEs) may perceive differences when performing hazard identification. The SME in the red chair remembers a recent occurrence of another event (recency bias). The SME in the blue chair does not want to assess potential hazards if there are mitigation processes already in place (status quo bias). 31 Note: This slide is based on materials and figures submitted to the ICH Q9(R1) EWG by the Pharmaceutical Research and Manufacturers of America (PhRMA).

32. I don’t agree. I have never seen that happen. That would be is extremely unlikely to occur. Contamination can frequently be transferred outside the Weigh & Dispense Room and to personnel PPE. I have seen it happen before. SMEs may perceive differences when scoring probability of occurrence. The SME in the red chair remembers a recent occurrence of another event (recency bias). The SME in the blue chair never experienced that event and does not believe it can happen (conservatism bias). Subjectivity in Estimating Probabilities of Occurrence 32 Note: This slide is based on materials and figures submitted to the ICH Q9(R1) EWG by the Pharmaceutical Research and Manufacturers of America (PhRMA).

33. Probabilities of Occurrence, cont’d Many factors can impact upon the probability estimates that are arrived at… The process being risk assessed may be new, and there may be little (or no) data on how often elements of it may fail, or how often similar elements have gone wrong in the past. Even with established processes, there may be little useful data available on the failure rates of things. One can use complaint data and deviation information, but these may only represent part of the picture. One can elicit experts to estimate probabilities of occurrence, but research shows that human heuristics and other factors can lead experts to make biased decisions about probabilities (Goldberg 1959, Faust, 1985). 33 Ref. Goldberg, L. R. “The effectiveness of clinicians’ judgements: the diagnosis of organic brain damage from the Bender-Gestalt test”, Journal of Consulting Psychologists, 23:23-33, 1959 Ref. Faust, D. Declarations versus investigations: The case for the special reasoning abilities and capabilities of the expert witness in psychology/psychiatry, Journal of Psychiatry & Law, 13(1–2), pp 33–59, 1985

34. I do not think so. There are no detection methods in place.  Contamination in the Weigh & Dispense Room can be easily detected when it occurs and cannot be missed. It has been detected a few times.   The SME in the red chair asssumes that all contamination will be obvious because it has been found before and was clearly visible (a vailabilty bias ) The SME in the blue chair believes contamination can be easily missed if not checked regularly. Subjectivity in Estimating Detectability 34 Note: This slide is based on materials and figures submitted to the ICH Q9(R1) EWG by the Pharmaceutical Research and Manufacturers of America (PhRMA).

35. Understanding Bias in Machine Learning (ML) Models Machine Learning: A method of data analysis that automates analytical model building. It is a branch of artificial intelligence based on the idea that systems can learn from data, identify patterns, and make decisions with minimal human intervention. Growing volumes and varieties of available data, in conjunction with advances in computational processing, enable more applications of machine learning (ML). With regard to pharmaceuticals, it is important to recognize that the application of digitalization and emerging technologies in the manufacture and control of drug (medicinal) products can lead to risk reduction, when such technologies are fit for their intended use. However, they can also introduce other risks that may need to be controlled. Current applications of Machine Learning include, but are not limited to: Process Monitoring/Control: ML algorithms, in combination with spectroscopical methods, are deployed to monitor process parameters – leading to increased process knowledge. Predictive Maintenance: Historical data (training data) are used to train ML algorithms to predict the optimum time for maintenance or replacement of equipment, to reduce production downtime/bottlenecks, etc. Technology Transfer: ML algorithms are deployed to predict process performance. 35 It is important to consider the potential for bias to inadvertently be incorporated into ML algorithms and applications.

36. Understanding Bias in Machine Learning (ML) Models cont’d Bias in relation to Machine Learning: This can be considered to be a systematic error arising from an erroneous assumption in the algorithm’s modeling.  The assumption in this case is that the training data on which the ML algorithm relies is not reflective of reality. An observed high degree of accuracy in the predictions of the Artificial Intelligence (AI) model does NOT necessarily mean the model is stable and learning accurately. The model may be generating biased outputs that go undetected. How is such bias introduced? Insufficient data – there is simply not enough data inputted into the ML model to ensure non-biased outputs, or the data collected is not representative of the population of interest. Inconsistent data collecting – Collecting data in a way that does not ensure a representative sample. What are the consequences of such bias? Lost investments and lower revenue. Lower customer satisfaction. Most importantly, POOR DECISIONS through decison error resulting from model bias. 36

37. Sources of Bias in Machine Learning models Here are some real-life examples of how bias can impact AI models and have negative consequences: College acceptance decisions – Ref. Lilah Burke, “The death and life of an admissions algorithm”, Inside Higher Ed, December 14, 2020 Criminal sentencing and parole decisions – Ref. Karen Hao, “AI is sending people to jail—and getting it wrong”, MIT Technology Review, January 21, 2019 Hiring decisions - Ref. Miranda Bogen , “All the ways hiring algorithms can introduce bias”, Harvard Business Review, May 6, 2019

38. Ensure that training data is representative of the population Educate/train personnel in the organization about the potential for model bias Include domain experts and impacted parties in the model development process  Ensure that team members are well-versed on the underlying data, features, etc., of the AI model – this is very important. Develop a strategy for dealing with missing data and unbalanced data: Missing data – identify appropriate algorithms to deal with missing data; impute missing data via imputation (note – it is important to utilize an imputation method that is fit for purpose as each method poses their own pros/cons) Unbalanced data – use a sampling strategy to over/under sample certain classes (i.e., the over/under-represented subset of the data) to prevent algorithm bias Identify and utilize fit-for-purpose indicators to measure the model’s performance Monitor the model over time to proactively identify model degradation and update it with new training data, as needed Tips for Minimizing AI Model Bias

39. Many scales used in Risk Assessment methodologies are constructed using what are called ‘Ordinal’ numbers These scales, e.g., 1 to 5, or 1 to 10, just indicate a relative order of things – they do not represent actual units of measurement The magnitude of the individual values is not meaningful in a numerical sense ( Conrow , 2003) Such scales are inherently subjective, and one should exercise caution when using them Consider the Probability of Occurrence scale on the right: A Probability of Occurrence of 4 is of course higher than a Probability of Occurrence of 2, but it is not necessarily twice as high Subjectivity in Risk Scoring Scales/Models 39     Probability of Occurrence 5 Highest 4 3 2 1 Lowest Movie ratings based on Stars are an everyday example of an Ordinal Scale Movie A has a 4-star rating, Movie B has a 2-star rating Is Movie A twice as good as Movie B??? Ref. Conrow , E. H., Effective Risk Management: Some Keys to Success, 2nd Edition, 2003, General Publication Series, American Institute of Aeronautics and Astronautics

40. See the rating scale shown on the right as an example: Such scales can be quite subjective A time range that goes from 1 week to a thousand years can be difficult to comprehend and apply, and data will likely not be available that expresses events in that way A person using such a scale may have a different understanding of what the word descriptors in the scale (e.g., occasional, rare) mean, versus what the scale indicates they mean The bottom four levels in this scale are logarithmic , but the top two are not, and this can make the scale difficult to actually use in practice Subjectivity in Risk Scoring Scales/Models 40     Likelihood of failure ​ 6 Frequent ​ Once in a week 5 5 Moderate ​ Once in 1 month ​ 4 Occasional ​ Once in 1 year ​ 3 ​ Infrequent ​ Once in 10 years   ​ ​ 2 ​ Rare ​ Once in 100 years   ​ ​ 1 ​ Very Rare ​ Once in 1000 years ​ Sometimes, Probability of Occurrence scales contain quantitative likelihood rates that are intended to explain what the different levels on the scale mean.

41. Risk is usually expressed as the combination of Probability of Occurrence and Severity of Harm Numerical or word-based scales are often used when Probability and Severity estimates are being arrived at, and such scales can be a source of subjectivity There can be several reasons for this: The various levels making up the scales may not be well defined , and it may be unclear as to what they actually mean The various levels may not be well differentiated from each other, and it may be unclear as to when to assign one rating over another The words used to describe the various levels may mean different things to different people Consider these scales, for example. What do you think of them? Subjectivity in Risk Scoring Scales/Models 41     Severity of Harm 5 Patient Death 4 Patient Injury 3 Moderate Patient Impact 2 Minor Patient Impact 1 Negligible Patient Impact Probability of Occurrence 5 Very Likely 4 Frequent 3 Occasional 2 Unlikely 1 Rare

42. As discussed in earlier slides, severity and occurrence scales are fundamental for estimating the amount of risk. When combining severity and occurrence scales in risk charts, areas with High/Medium/Low risk can be identified. Acceptability of risk and what is considered High /Medium/Low needs to be agreed by the organization to fit the specific risk question . Some models use numerical calculations to evaluate a risk and place it in the risk chart; other models use other methods. Each method has its bias – as illustrated by the next slides. When prioritizing risks for risk control, such effects should be considered. The following slides show various methods for creating risk acceptability charts . In these examples , r isk acceptability thresholds are shown as green (low risk), yellow (medium risk) and red (high risk ). Subjectivity in Risk Scoring Scales/Models Likelihood of Occurrence Remote Unlikely Occasional Likely Frequent Severity Critical Major Moderate Minor Negligible Note : The risk acceptability charts shown on this and subsequent slides are fictitious examples illustrating how different rating scales and methods can affect where a risk is placed on the chart, and thus, how this may affect risk acceptability. The charts used show risk areas (green, yellow and red) of different sizes. In other approaches, risk scales and charts can be used based on what best fits the risk question. There is no ideal number of levels for a risk rating scale. Most scales have at least three levels, in order to avoid “Yes or No” scenarios. Additional levels may enable more complex risk ratings. However, some risk questions do not require sophisticated rating scales. The number of levels can be adjusted to be commensurate with the level of risk.

43. Subjectivity in Risk Scoring: C ombining S and O ratings Risks can be expressed in different ways. A very straight-forward way is to directly combine Severity (S) and Occurrence (O) terminology ratings: Likelihood of Occurrence Remote Unlikely Occasional Likely Frequent Severity Critical Critical, Remote Critical, Unlikely Critical, Occasional Critical, Likely Critical, Frequent Major Major, Remote Major, Unlikely Major, Occasional Major, Likely Major, Frequent Moderate Moderate, Remote Moderate, Unlikely Moderate, Occasional Moderate, Likely Moderate, Frequent Minor Minor, Remote Minor, Unlikely Minor, Occasional Minor, Likely Minor, Frequent Negligible Negligible, Remote Negligible, Unlikely Negligible, Occasional Negligible, Likely Negligible, Frequent 43 Advantage : This approach enables unique risk ratings to be determined. Each cell covers a single combination of severity and occurrence that will be assessed for its acceptability. Disadvantage : Determining risk acceptability based on such qualitative terms requires more time in preparation than using numerical, quantitative scales. Risk A Risk B This approach takes different severity and occurrence ratings and combines them in a risk chart (or matrix). Each cell in the matrix has a unique ID based on the ratings, and it has a unique acceptability level (Red/Yellow/Green) assigned to it.

44. Another important consideration with Ordinal Scales…. It is common practice in some Risk Assessment methodologies to multiply the ratings assigned to Probability (Likelihood) of Occurrence and Severity of Harm, to arrive at a risk level or estimate: Probability of Occurrence = 3 Severity of Harm = 7 Risk = 3 x 7 = 21 The numerical risk that is arrived at through the use of ordinal numbers is subjective. (Note: There is subjectivity in all estimates of risk, and it is not possible to eliminate subjectivity, but efforts should be made to control and minimize it.) Subjectivity in Risk Scoring Scales/Models 44   Likelihood of Occurrence 1 Remote 3 Unlikely 5 Occasional 7 Likely 9 Frequent Severity 9 Critical 9 27 45 63 81 7 Major 7 21 35 49 63 5 Moderate 5 15 25 35 45 3 Minor 3 9 15 21 27 1 Negligible 1 3 5 7 9 Consider the risk circled below… This risk was based on an Unlikely occurrence rating (3) and a Major severity rating (7). The Risk (3 x 7 = 21) was considered moderate in level. It is useful to consider subjectivity – how much subjectivity was there in the Unlikely rating? What data supported that rating? What if the Likelihood had been 5 (Occasional) instead of 3 (Unlikely)? This small change in one rating (Likelihood, from 3 to 5) results in a large change in the estimated risk (21 to 35). The risk would then have been unacceptable . Failing to recognize such subjectivity is problematic, as it can result in one assigning too much weight (or importance) to the actual risk numbers that are arrived at. For example , low risk numbers may lead to risk acceptance decisions without considering the risks more fully. And in some cases, risk controls may actually be required.

45. Subjectivity in Risk Scoring Risk is often expressed by assigning numerical ratings, as shown in the matrix below: In this example, each rating scale is assigned numbers from 1-9. Risk is then ‘calculated’  by multiplying the severity and occurrence numbers. This helps simplify risk acceptability decisions. On the downside, very different harms may appear equally significant. For example, see the two risks circled in this risk chart/matrix: Likelihood of Occurrence 1 Remote 3 Unlikely 5 Occasional 7 Likely 9 Frequent Severity 9 Critical 9 27 45 63 81 7 Major 7 21 35 49 63 5 Moderate 5 15 25 35 45 3 Minor 3 9 15 21 27 1 Negligible 1 3 5 7 9 45 Tip: When Risk Assessment methodologies/tools like the one above are used, it can be useful to express the risks more completely, so that they are better differentiated: e.g., Risk A = 27 (Critical Sev ., Unlikely Occ.) and Risk B = 27 (Minor Sev ., Frequent Occ.) This helps make the risk expressions more objective, and thus less subjective. Note: It is also important to carefully critique risk scoring scales such as this one, and to consider what risk levels are to be assigned Red, Yellow and Green labels. Having the same risk score may result in treating both risks equally - in terms of the need for risk control - even when the potential impact on individual patients is significantly higher for Risk A. Just considering the risk score is not very objective - it is somewhat subjective. In the chart on this slide, risk above a threshold of 27 is considered high/red, because that fits with the numbers – however, compared to the chart on the slide 47, this gives a different acceptability. Risk A : This has a critical severity (e.g., severe harm or death). While its occurrence is rated as unlikely, it is not remote. This hazard should be a concern. Risk B : This risk is very different. It is considered to result in minor (but not negligible) patient impact that will likely occur frequently.

46. In ICH Q9(R1), risk is defined as “The combination of the probability of occurrence of harm and the severity of that harm (ISO/IEC Guide 51:2014)”. There are different ways to combine such probabilities of occurrence and severities: The multiplication approach shown on slides 48 & 49 is one approach. ‘ Tallying ’ is another approach: this is where each risk is represented by a 2-digit number, with the Severity rating as the first digit. Tallying is sometimes expressed as ‘ Sev . | Occ.’  Mathematically, using the scales below, tallying involves weighting the severity ratings by a factor of 10, as in: Risk = (10 x Sev .) + Occ. Tallying can provide a different risk scoring perspective, augmenting data analysis efforts. Both approaches are shown below. Using both approaches together can be useful. Subjectivity in Risk Scoring: Tallying S|O Risks can be expressed in different ways... one such way is called Tallying 46 S x O OCC 1 OCC 3 OCC 5 SEV 5 5 15 25 SEV 3 3 9 15 SEV 1 1 3 5 S | O OCC 1 OCC 3 OCC 5 SEV 5 51 53 55 SEV 3 31 33 35 SEV 1 11 13 15 Note : Risk scores differ significantly between the two charts. In the S x O approach, some risk scores can be found in more than one cell, e.g., risk scores of 5 and 15. This implies identical risk levels for different combinations (via multiplication) of S and O ratings. In the Tallying approach (S | O) shown above on the right, all cells have unique risk scores, and the more severe risks to patients can be better prioritized for risk control.

47. In some Risk Assessment tools, such as FMEA and FMECA, failure modes are prioritised for risk control considerations based on Risk Priority Numbers (RPNs) The RPN is sometimes considered the product of Severity (S) x Probability (P) x Detectability (D) If scales of 1-5 are used for S, P & D, the range of RPNs will be 1-125. If scales of 1-10 are used, the range will be 1-1000 RPNs are useful – they are easy to understand and apply It is common practice to define an RPN threshold (or cut-off number) Above the threshold, risk controls will be required At or b elow the threshold, risk controls are often not required, as those risks are often considered acceptable Subjectivity in Risk Scoring Scales/Models 47 It is useful to reflect on such approaches: As RPNs are usually based on ordinal scales, there will be subjectivity in their values Defining an RPN threshold is one way to prioritise failure modes for risk control activities, but what should the threshold be? What determines it? How subjective is it? Rigorously applying such thresholds in light of these subjectivity considerations can be problematic. If an RPN range is 1-125 and the threshold is set at 40, it may not be correct to take the position that a failure mode with an RPN of 36 (4x3x3) requires no risk control?

48. In relation to RPN thresholds, consider the following scenario… A manufacturing site is inspected and is issued an observation related to not having a defined cleaning revalidation program in place. The site commits to performing a risk assessment to determine its cleaning revalidation requirements. An FMEA is used – this has rating scales for Severity, Probability and Detection of 1 through 5. The RPN range is 1 to 125. The site FMEA procedure indicates that risk controls will be required for failure modes with RPNs > 60 This is approximately half-way up the RPN range (1-125). The RPN threshold (or cut-off) is set at > 60. In the FMEA, none of the failure modes that were risk assessed exceed the RPN threshold. Subjectivity in Risk Scoring Scales/Models 48 In this scenario, it is worth considering that only 11 possible RPN combinations exceed the RPN threshold of 60. And while above the threshold of 60 (i.e. 61 through 125) represents 52% of the RPN range, the 11 RPN combinations > 60 represent only 9% of all possible combinations (i.e., 11 out of 125 possible combinations). So RPN thresholds should be chosen with care!

49. Subjectivity in Managing Failure Modes 49 49 Risk Matrix, (S x O) Risk Priority Number, (S x O x D) Consider a Risk Assessment where six different Failure Modes all have the same RPN – 81. Presenting the Failure Modes visually , using the matrices shown below, is a useful way to differentiate between Failure Modes, and, thus, to prioritize their risk control activities. Likelihood of Occurrence 1 Remote 3 Unlikely 5 Occasional 7 Likely 9 Frequent Severity 9 Critical 9 27 45 63 81 7 Major 7 21 35 49 63 5 Moderate 5 15 25 35 45 3 Minor 3 9 15 21 27 1 Negligible 1 3 5 7 9 The Risk Matrix on the left shows the risks associated with 6 different Failure Modes. Each of these Failure Modes has an RPN of 81 when detectability is taken into account – on the right. Presenting the RPNs in this visual way allows one to see that the RPNs of 81 are not all the same – they differ markedly in Severity, Occurrence and Detectability. This can allow one to prioritize any required risk control activities appropriately. When prioritizing risk controls on the right, consider Failure Modes with the highest Risk Index first.

50. Take-home Points/Conclusions Subjectivity can be observed across all parts of the QRM process, from Risk Assessment through to Risk Review There are many strategies that can be adopted to control and minimize subjectivity. A number of these are outlined in Part I of this presentation. This training material, in Parts I and II, together with the Case Study referred to in Part I , illustrates the sources of subjectivity and things to consider when working to control subjectivity. These relate to the following: Taking steps that reduce the effects of heuristics and bias Having fit-for-purpose risk scoring methods  Making sure that QRM tools are used appropriately and as intended Maximising the use of relevant data and sources of knowledge 50

51. 51 Click here for Part I of this presentation

52. Acknowledgement to the ICH Q9(R1) Expert Working Group (EWG) For any questions, please contact the ICH Secretariat: [email protected] EWG Acknowledgment and Contact