QRM in the GMP Environment: Ten Years On—Are Medicines Any Safer Now? A Regulators Perspective | IVT

Peer Reviewed: Risk


The past ten years since the finalisation of ICH Q9 have seen significant efforts by industry and regulators to increase use of risk-based approaches in pharmaceutical applications.  The GMPs, in particular, have undergone very significant changes to reflect the concepts of QRM, both in the US and EU.  Despite these efforts, the question remains as to whether medicines are any safer now than they were when the ICH Q9 guideline was originally introduced.  This paper discusses GMP and product quality considerations relative to QRM. 

From my own perspective, it seems that many of the issues we had before 2005 are still with us, and historical evidence indicates that certain types of problems are increasing.  Some of these relate to serious quality defects, product recalls, and product shortages.  Quantitative historical data are provided and specific case studies are presented.  While many factors may have contributed to increases, published data are significant and of serious concern.  Why has this occurred?     

Four key problems seem to be prevalent across the industry with respect to its approaches to risk assessment and QRM work in general are discussed.

  • Lack of good science
  • Too little rigour 
  • Poor management of knowledge
  • Overuse of formal risk assessments.

Five suggestions for improvements in how to improve our approaches to QRM are proposed:

  • Develop QRM Certification Programmes utilising related research
  • Work to Better Understand Informal vs. Formal Risk Assessment
  • Utilise Knowledge Management (KM) in QRM
  • More Focus on Science
  • Apply Increased Rigour

While there is much food for thought for the pharmaceutical manufacturing sector presented here, it is important to also recognise that problem issues cited in this paper also apply to the GMP inspectorates and their wider National Competent Authorities that regulate that sector.  Within competent authorities, work including associated training is already underway to address a number of these challenges.  The pharmaceutical manufacturing sector should also consider what advanced training and learning it requires to better achieve QRM benefits.

It is within a spirit of openness and reflection that this paper seeks to present these ideas for discussion and debate.



Over the past ten years since the finalisation of ICH Q9 (1), there have been significant efforts by industry and regulators to increase use of risk-based approaches in pharmaceutical applications.  This has included initiatives by regulators to update the GMPs and other official guidance documents to incorporate increased provisions relating to risk and its management (2-11), and the publication of an array of industry-led publication, guides, and technical reports from PDA, ISPE, ASTM, and others.  These offer practical guidance for how to implement the concepts of Quality Risk Management (QRM) in the GMP environment (12–18).  There have also been many papers and studies published in the literature in this area (19–26).   

The GMPs, in particular, have undergone very significant changes to reflect the concepts of QRM.  In the US, FDA has undertaken many initiatives to incorporate risk-based concepts and requirements into its regulatory guidances and to encourage innovation and modernisation in GMP.  This work started well before 2005 when ICH Q9 was finalised.  Its 21st Century initiative (2) led the way, resulting in many useful and innovative guidances for industry such as the Process Analytical Technology (PAT) guidance of 2004 (3).  This placed a heavy emphasis on risk analysis and its role in supporting PAT applications.  The modernisation work led by FDA continues, and their influential 2011 guidance for industry on process validation included a heavy emphasis on QRM (4).    

The EU has similarly revamped its regulatory requirements to incorporate the principles of QRM.  Changes range from the new risk-based Pharmaceutical Quality System requirements (2008 and 2013 revisions to Chapter 1 of the EU GMP Guide) to the new supply chain provisions introduced into Chapter 5 (Production) in March 2015 (5).  The latter now requires the level of supervision of suppliers of starting materials to be proportionate to the risks posedby the individual materials.  Chapter 8, Complaints and Recalls, underwent a very significant revision in March 2013; it was entirely rewritten to incorporate the concepts of risk assessment and QRM.  For example, Chapter 8 now formally addresses the need for risk-based decision making and risk-reducing actions in response to quality defect issues (5).  In the area of qualification and validation, the October 2015 revision of Annex 15 to the EU Guide for the first time officially linked process parameter criticality with risk assessment work.  It stated that the basis by which process parameters and quality attributes are identified as critical or non-critical should be clearly documented, taking into account the results of any risk assessment activities (5).

There is no doubt that ICH Q9 has had a significant impact on the pharmaceutical sector.  Together with ICH Q8, 10 and 11, major advancements in the modernisation of the GMPs to reflect risk-based concepts in almost all areas of GMP have resulted.  However, the question remains as to whether medicines are any safer now than they were when the ICH Q9 guideline was introduced ten years ago.  While there are many different areas one could study when considering this question such as pharmacovigilance data, Marketing Authorisation withdrawals and drug shortages, the focus of this paper is on GMP and specifically on product quality considerations.



The aforementioned regulatory and industry initiatives each had product quality and patient safety as their overall focus, and they promoted the increased use of risk assessment and QRM in the GMP environment.  But, it is important that we ask ourselves some hard questions in relation to what has been achieved for patients, such as:

  • Has all this risk assessment and QRM work delivered the expected benefits? 
  • Has ICH Q9 been implemented as intended?

And the ultimate question: 

  • Are medicines safer now than they were ten years ago before ICH Q9?

From my own perspective, it seems that many of the issues we had before 2005 are still with us (27, 28), and evidence indicates that certain types of problems are increasing.  For a useful discussion on this area, see the review presented by Waldron in 2015 (28).  Some of these relate to:

  • Serious quality defects and product recalls continuing to occur globally;
  • MA non-compliance issues sometimes resulting in the cessation of batch release
  • Product shortages as a result of serious GMP failures.

Looking at quality defect data alone, it is interesting to see the increasing trend seen at the HPRA over the last ten years, as shown in Figure 1 and Table 1 below.  

 Table 1

Figure 1:  Serious (Major and Critical) Quality Defects Reported to HPRA, Ireland, 2004-2014 (Source: HPRA 2015)

Figure 1 


Table 1: Quality Defects & Recall Statistics, HPRA, Ireland, 2005-2014 (Source: HPRA 2015)

Quality Defect data from the EMA on Centralised Products also indicates an increase in reported suspected Quality Defects between 1999 and June 2014 as shown in Figure 2.


Figure 2

Figure 2:  Reported Suspected Quality Defects on Centralised Products, 1999 – June 2014 (Source: EMA, 2015)

There are probably many factors at play behind the increases in quality defect reports seen in the last ten years.  These include higher levels of reporting of quality defects by manufacturers, and possibly more products in the marketplace, among others.  It is important that such data be carefully studied and evaluated to draw scientific conclusions from the data set.  Formal research projects are currently underway in Ireland and at the EMA to do just that, via HPRA, Regulatory Science Ireland and EMA initiatives.  However, in my opinion, these factors alone probably do not fully explain the increasing trends that have been seen, and factors related to globalisation of the supply chain resulting in increased complexity and thus in more opportunity for processes to fail may also be important.  Either way, the data above are significant and give serious rise to concern.   

Specific Examples

Some examples of serious quality defect issues investigated by HPRA in recent years include:  

  • Product mix-up issues.  In March 2015, lipid and aqueous units intended for a 600g neonatal baby were incorrectly labelled.  In August 2015, 60mg tablets were packed in 30mg packs of a duloxetine medicine.
  • Faulty/Defective batches.  In 2012-2014 there were reports that indicated that certain adrenaline auto-injector pens, used during a life-threatening anaphylactic reaction, might deliver no dose or a reduced dose.
  • Packaging and labelling issues.  In 2013, the wrong colour coding was used on vials of Methotrexate.  In 2014, important pregnancy prevention information was missing from the package leaflet of a medicine that was teratogenic in nature. 
  • Sterility assurance and product contamination issues.  In 2015 and earlier, various sterility assurance concerns, such as from recurring media fill failures at manufacturing sites, were identified leading to batch recalls. In addition, there has been various product contamination incidents reported, such as glass particles in vials, mould contamination in tablets, and others.

When one considers the fact that most of these defective batches were manufactured using qualified equipment, trained staff, and validated manufacturing processes, one has to question what is happening.  

  • Have the manufacturing and control processes currently in place been properly risk assessed, and have they been designed using QRM principles?  
  • Were qualification and validation activities actually based on risk, as required by Annex 15 of the EU GMPs?  If so, why are defective batches of medicines still being produced, with the same kind of defects over and over – product contamination, mix-ups, labelling errors, etc.? 
  • What areas need attention in the QRM activities at manufacturing sites? 



GMP Inspectors have noted an increasing use of risk assessment and QRM activities at sites since ICH Q9 was finalised in 2005.  From my own experience, risk assessment and control have been the areas that have received greatest attention at sites, with risk review, risk communication lagging somewhat behind.  While some good practices have been seen in these areas, significant concerns remain, especially in relation to the lack of good science, robustness, and rigour that are associated with many of risk assessment and QRM-related activities.  The following is a series of brief real-life case study examples showing some of the inspectional deficiencies that the HPRA has cited in recent years in these areas.  The first two case studies related to finished product manufacturers, and the second two relate to API-manufacturing sites. 

Case Study 1:  QRM and Process Robustness of a Tablet Manufacturing Process

While described as a QRM exercise, this example is probably better described as a risk assessment exercise.  It was performed by a finished product facility, it was proactive in nature, and its stated aim was to identify risks affecting process robustness and also to assess the adequacy of the existing CPPs and the control strategy

This case exemplified a proactive approach to risk assessment in order to generate increased process understanding and improved levels of process control.  The company used a risk assessment tool that they developed in-house, demonstrating initiative and indicating a general approach to GMP that was continuous improvement focused - the site worked to improve its approach to risk assessment appropriate to the exercise at hand – a very positive initiative.  

However, when examined during inspection, it was evident that the risk assessment exercise was lacking in a number of important areas, and several deficiencies were cited including the following:

  • There was no procedure in place for how risk assessment was to be performed, and it was unclear how the different classes of risks (low, medium, high) were to be managed.
  • Tablet packaging operations were not within the scope of the exercise despite the fact that two recalls had recently occurred for packaging-related quality defects. 
  • The risk assessment failed to address the root causes of the risk events that had been identified, and there was no space on the risk assessment form to document any such root causes.
  • The adequacy of existing critical process parameters (CPPs) from a risk perspective had not been assessed. 
  • There had been no assessment of whether new CPPs were required to control the process. 
  • There was no assessment of the qualification or validation status of the various controls that were identified as important for risk mitigation.

This case study demonstrates a clear example of an inadequate approach to Risk Assessment on several fronts - procedural weaknesses, a failure to address the aims of the exercise, and a lack of robustness as evidenced by the failure to identify any of the root causes of the failure event.

Case Study 2:  Risk Assessment to Support a Change Control Proposal

The aim of this risk assessment was to assign revised calibration frequencies to instruments and other equipment at a finished product site using risk-based criteria.  The exercise identified two “high risks,” each relating to out-of-calibration instruments.  The risk assessment report stated that these risks were mitigated by three types of detection controls as follows:

  • Daily verification checks
  • In-process controls
  • Finished product testing.

However, on closer examination, it was found that many types of instruments and items of equipment included within the scope of this assessment, such as pressure probes and temperature transmitters, were not subjected to any kind of daily verification checks.  In relation to in-process controls and finished product testing, it was not clear how those activities could detect out-of-calibration equipment.  Furthermore, the risk assessment approach had a significant potential to assign the same twelve-month default calibration frequency to both GMP critical and GMP non-critical instruments after they were in use for over 24 months.  This was perhaps an unintended feature of the risk assessment approach that was used, but it did not withstand regulatory scrutiny.  Deficiencies were cited with reflected each of the above points. 

Case Study 3:  Risk Assessment to Support a Deviation Investigation

The aim of the risk assessment in this case was to assess the risks presented by a screen breakage incident in an API process and to decide on batch release.  

The context for this risk assessment was that, at the end of an API production campaign, a metal mesh screen used in one of the final steps was found to have become broken at some stage during the campaign.  A large section of the mesh material was torn from the rim of the screen.  Some of the batches had already been released and shipped.  The remaining batches were still at the API site and were quarantined.  An FMEA was performed to help decide whether to release the remaining batches.  A number of significant problems were identified with the risk assessment.  For example:

  • A medium severity rating was assigned to the identified hazard of screen wire material ending up in the medicinal product.  It was not documented why a piece of stainless steel up to 850µm in length in a tablet represented a hazard of medium severity and not one with a high severity, and no clinical assessment was documented of the potential impact on patients. 
  • While a low likelihood rating was assigned for metal fragments having entered a batch, there was evidence that this may not have been correct - a 15 cm piece of mesh wire had broken off while washing the screen during the investigation.
  • Finally, the risk assessment concluded that the issue presented a low risk of batch contamination and this conclusion was used to justify the release of the quarantined batches to a drug product-manufacturing site.  No reprocessing or reworking was carried out on the API batches.

When considering this case and the decision made based on the risk assessment, it is interesting to consider whether the same approach to this deviation incident would have occurred ten years ago before ICH Q9 was in place.  With such a serious equipment failure, would the same decision have been made to release the remaining batches without any reprocessing or reworking operation?  While we don’t know the answer to that, we do know that some months after release, a complaint was received from the drug product manufacturing site (the customer) stating that metal had been detected in tablets processed using those API lots. 

This case study presents clear evidence of the failure of a risk assessment process in relation to good decision making and in preventing the release of defective product; it ultimately failed to adequately protect patients.

Case Study 4:  Risk Assessment to Support a PAT Change Control

The aim of this risk assessment was to assess the risks of introducing a PAT instrument into an API process.  Change control documentation had been initiated to replace an off-line particle size in process test used to determine the end of re-crystallisation before discharging the batch to a filter dryer with an in-line PAT measurement.  The in-line instrument would be used to monitor the physical characteristics of the slurry (chord distribution length) to determine when the re-crystallisation process was complete.  This would deliver time and cost savings through the elimination of the off-line in-process particle size test.  It would also lead to eliminating a later bulk density test normally performed on the dried API. 

The risk assessment was reviewed during inspection, and while certain positive elements were identified, a number of important deficiencies were also noted.  For example:

  • The failure mode root cause analysis part of the assessment lacked rigour and was essentially meaningless; for several failure modes, the potential root causes were essentially the same as the failure modes, as in the following example:  

Potential Failure Mode: Fouling of Instrument

Potential Cause: Build-up of product on glass probe

This is problematic for two reasons: Poor root-cause analysis affects how probabilities of occurrence are estimated or assigned.  It is the probability of occurrence of each potential cause of a failure mode that needs to be assigned in risk assessment exercises, not the probability of occurrence of the failure mode itself (see reference 29).  If the potential failure mode causes are not identified, the probability of each will not be considered, and the risk score or rating for the failure mode in question will be assigned based on a probability of occurrence that has nothing to do with its potential causes.  Secondly, when the potential causes of failure modes are not identified in risk assessments, it is unlikely that adequate preventative controls to reduce their probabilities of occurrence will be identified.  This can then lead to inadequate risk control and mitigation strategies being identified via risk assessment work.  

  • Several low severity ratings were not scientifically justified.  For example, for one of the risks assessed in this risk assessment, the potential failure mode was:  “Confusion as to which reactor is being read by the instrument.”  The potential effect of this failure mode was documented as: “Incorrect Results,” and theCurrent Control was listed as “Programme tested.” 

A Severity Rating of 1 was assigned to the effect of this failure mode. This indicated that the effect would not cause a risk to health and did not impact quality or regulatory compliance.  This led to a Risk Priority Number (RPN) of 1 (lowest possible RPN) being assigned to the failure mode, leading to no risk control or mitigation actions being identified.  This was despite the fact that there were no preventative controls of any kind documented in the risk assessment for this type of failure mode, which has to be regarded as one with potentially very serious implications – completely wrong test results being generated for the batch undergoing re-crystallisation.   

It was clear that not only was this particular failure mode evaluated very poorly in the risk assessment, the lack of good science applied here (and in other parts of the risk assessment also) rendered the outputs of the assessment to be highly questionable.  It also led to a false sense of security in the control strategy that was in place for the PAT instrument.    

  • For a number of failure modes, low detectability scores indicating a high likelihood of detection had been assigned which were not supported by adequate detection-related controls.  One example was for a failure mode related to metal contamination.  In this case, a detectability rating of 1 was assigned, meaning that such contamination in a batch would certainly be detected, yet there was no test for metal or other ongoing detection controls documented in the assessment to support this rating.   

This low detectability score led to a low RPN being assigned to the failure mode and again, this resulted in no risk control or mitigation actions being identified.   



Looking back over ten years of inspecting risk assessments and QRM work at manufacturing sites as well as reviewing numerous risk assessment reports submitted by companies in response to quality defect issues, a number of recurring problems come to mind.  These have contributed to the industry not realising the full benefits of its QRM-related activities.  They have adversely affected many related GMP activities, such as risk-based approaches to CAPA, change management, process validation, process understanding, product quality reviews, and others.  This has probably also partly contributed to the desired state, as envisaged by ICH Q10, not being achieved to any meaningful degree.  This is one reason that ICH Q12, Lifecycle Management, has been initiated.  

Four key problems seem to be prevalent across the industry with respect to its approaches to risk assessment and QRM work in general.  These are as follows:

 1. Lack of Good Science

Many of the risk assessments are lacking in good science.  This leads to numerous other problems, not least of which is their ability to withstand any level of scrutiny by the regulator, but more importantly, their role in risk mitigation for patients.

The following are common deficiencies that lack good science:

  • Probability of occurrence estimates that are not based on any kind of historical data, preventative controls, or on modelling data
  • Assumptions regarding risk severity and detection that are totally unsound, and which are not supported by controls which influence failure mode severity or detectability
  • Making important decisions based on RPN values that fail to recognise that those values are derived only from ordinal scale numbers, that they are not mathematically meaningful.  They are often associated with high levels of subjectivity, uncertainty and guesswork.  See paper by Waldron in this Journal in relation to risk analysis and risk rating scales (30)
  • Over-relying on RPN thresholds or cut-offs that have no scientific basis but which greatly influence risk assessment outputs.  It is common that when RPNs fall under the cited threshold, no consideration is given to the need for any additional risk control or mitigation
  • Documenting controls for risk mitigation that are not relevant to the failure mode being assessed, such as were detection controls are used as the basis for low probability of occurrence of failure modes.

2. Too Little Rigour 

Many risk assessments suffer from a lack of rigour.  This can have negative consequences manifest across all aspects of risk assessments.  These include the basic risk question through to hazard identification, assessment and control, risk review, and risk communication.  A simple example, relating to how the risk question or problem statement at the outset of a risk assessment was documented follows:  

In the aforementioned risk assessment exercise relating to the PAT instrument in the API re-crystallisation process, the following risk question was documented at the top of the risk assessment form.    

‘What could affect patient safety along the product lifecycle?’ 

While good in many respects – it is focussed on patient safety and product lifecycle - it is also very high level and is not in any way specific to the task at hand.  This activity intended to risk assess the proposed introduction of a PAT instrument based on chord distribution length measurement in a re-crystallisation step in an API manufacturing process.  This kind of risk question could probably be applied to every single risk assessment in a GMP facility and is of questionable value.  For example:

  • It fails to recognise that the step of the manufacturing process to which it relates is quite far removed from the patient – a slurry recrystallization step in an API process
  • It does not in any way lead one’s mind to thinking about what specifically might go wrong with the proposed change control – introducing the PAT probe into the process and using it to justify removing an off-line particle size test as well as a bulk density test on the finished API
  • It does not address a key concern that would seem to be quite important – the accuracy and validity of the chord distribution monitoring results that the PAT instrument would generate. 

When the risk assessment was reviewed on inspection, it was immediately evident that inadequate consideration had been given to the risks relating to inaccurate or invalid chord distribution monitoring results.  The only reference to this in the risk assessment was in relation to reading results from the wrong reactor.  

There were no other failure modes documented that specifically addressed the potential risk of generating inaccurate or invalid chord distribution monitoring results, and it was unclear what controls were in place which would assure the accuracy and validity of the PAT data and results.  How the risk question had been posed seems to have greatly affected the failure mode identification, risk rating and risk control elements of the exercise.   

Had increased rigour been applied when the risk question was being written, perhaps these weaknesses could have been prevented, and perhaps a more specific risk question might have been documented, such as 

‘What may affect the accuracy and validity of the chord distribution results generated by the PAT instrument?’ 


‘Is it safe to replace the off-line particle size test with a PAT monitoring system?’   

In my experience, a lack of rigour in risk assessment work is often a result of the following factors:

  • Using risk questions that are too high level and not tailored to the objective of the exercise;
  • Focusing on a very large number of failure modes and only superficially assessing them with subjective  RPN-type approaches; 

This can lead to very questionable risk assessment outputs and conclusions. I would recommend focusing on a much smaller number of failure modes and assessing them much more rigorously.  After all, if the pharmaceutical quality system is operating effectively, normal GMP controls (such as in documentation, training, equipment qualification, process validation, cleaning, etc.) should address the risks presented by most basic failure modes.

  • Not differentiating between failure modes and their causes – as explained in the earlier example, this can have several negative consequences;
  • Failing to recognise that potential failure modes can have more than one root cause – this has implications for how probability estimates are applied;
  • Failing to the link to outputs of the risk assessment back into the quality system; for example, using risk assessment to improve a control strategy but failing to update validation and qualification protocols.  This in turn leads to a lack of risk-based process validation and equipment qualification and to a greater potential to produce defective medicines. 

When it comes to control strategy considerations, an important question that is often overlooked during risk assessments is:  ‘What validation or qualification work, if any, is required for this control?’  Often there is a clear disconnect between risk assessment reports and process/cleaning/method validation activities, and many of the controls that get documented on risk assessment forms as being important in controlling or mitigating risks don’t make their way into validation and qualification protocols.  Thus, the effectiveness of those controls in risk reduction is not assured. 

3. Poor Management of Knowledge

In the years since the finalisation of ICH Q9, and in particular with the publishing of ICH Q10 in 2008, the role that knowledge management plays in delivering high quality medicines to patients has been recognised.  Indeed, ICH Q10 identified QRM and knowledge management as Key Enablers for an effective Pharmaceutical Quality System.  However, from my experience, effective knowledge management does not support much of the QRM work during inspections.  

Typical weaknesses in this area would include:

  • Existing knowledge about the purpose and nature of different kinds of GMP controls is sometimes overlooked in risk assessments.  For example, in Case Study 1 above, the daily verification of calibration checks on some instruments and other items of equipment above was an example of a GMP control that was in place for some instruments and equipment, and key site staff would have known this, but this control was regarded in the risk assessment exercise as a control that reduced the risk of all instruments going out of calibration.
  • Overlooking key empirical data in decision making (heuristic of representativeness), and not translating that data into usable knowledge.  An example of this can be seen in Case Study 2, where a low probability score is assigned to contamination in the metal screen breakage incident despite data that the screen was so damaged that contamination was a real possibility
  • Ignoring key sources of knowledge pertinent to the risk assessment.  For example, knowledge held by equipment or ingredient suppliers is not utilized in QRM.

4. Overuse of Formal Risk Assessments

Regulatory inspectors are now seeing formal risk assessments for a very large number of activities and problem issues at manufacturing sites.  These risk assessments are predominantly based on FMEA-type approaches, and are probably considered by most practitioners to represent formal risk assessments.  While this is positive in some respects – it reflects industry’s ongoing attempts to build in risk-based decision making across the board - it is also not without its problems, particularly when formal risk assessment tools such as FMEA are inherently weak and flawed in certain important areas.  

For example, while FMEA is a useful risk prioritisation tool, most approaches to FMEA that I have seen are highly subjective in their RPN outputs.  There are few if any design elements built into FMEA that limit the amount of subjectivity that is associated with the RPNs that are generated.  FMEA is also somewhat weak in relation to hazard identification and root cause analysis.  FMEA tools generally do not contain design features that dictate how brainstorming for failure mode identification should be performed, and they do not offer any kind of a structured approach to root cause analysis for the failure modes that are identified.  The root cause analyses that we often see on inspection via FMEA are quite superficial as in the PAT case study above, and lead to ineffective risk mitigation.  FMEA worksheets also generally contain no columns that relate to the qualification or validation requirements of the GMP controls that are identified during FMEA exercises as being important in risk control or mitigation.

A consequence is that, when formal but flawed risk assessment tools are used, their outputs are more often than not regarded as true and factual and are not scientifically challenged.  In this regard, the level of subjectivity and guesswork that is associated with those outputs - the failure modes identified, the RPNs generated, or the decisions for additional risk control / mitigation that are made - are often overlooked.  This is the problem with the use of such formal tools – they seem to result in a false sense of security in their outputs.  

To me, many of the problem issues that arise during manufacture (e.g. serious deviations, product defect reports, failed cleaning validation activities, etc.) do not always require formal risk assessments to help decide how to react to them, or how to prevent them from recurring.  Sometimes what is needed is better and more structured approaches to root cause analyses, more evidence-based investigations to determine the real extent of the problem rather than relying on subjective probability estimates in formal risk assessment exercises, and these being coupled with more effective CAPAs and improved process designs. 

Risk-based decision making does not always require the use of formal risk assessment tools to be effective.  It is the same for decision making within GMP Inspectorates.  If a serious GMP failure is reported to them that implicates a particular manufacturing site (such as a product mix-up incident, a serious compliance issue, or a reported lack of sterility assurance with a product), one does not need to apply a formal risk assessment tool to decide whether to go and perform a for-cause GMP inspection at that site.  Considering the facts and taking account the general risk issues for patients is often a much more effective approach.  We should not forget to use common sense!



Despite the multiple flaws in QRM practices within the industry described above, hope is not lost.  The remainder of this paper offers suggestions on how to improve our approaches to QRM.  Five ideas are proposed.

1. Develop QRM Certification Programmes utilising related research

The GMPs are becoming increasingly reliant upon QRM to support decision-making in almost all areas (see reference 5).  However, competencies in QRM have not kept pace.  A decade has passed since ICH Q9 was finalised, and we find ourselves in a position where little has improved, especially in relation to risk-based decision making.  In fact, one could argue that decision-making may have suffered through the use of highly subjective and biased approaches to risk assessment and QRM. 

As evidenced by the case studies and other examples presented in this paper, flawed risk assessment processes and tools are contributing to a lack of effective process control, increased risks of generating defective and harmful medicines, sub-standard approaches to validation and qualification, and GMP decisions relating to deviations and change control that are highly suspect.  All of this indicates that better competencies in QRM are needed if these problems are to be overcome.  People are urgently needed within the GMP environment who can learn about and apply the outcomes of research from other fields and disciplines.  There is a wealth of peer-reviewed research available on areas that are highly relevant to QRM such as expert opinion elucidation for probability estimations (31-34), System Reliability Analysis (32-33), risk perception and human heuristics (37-47), Probabilistic Risk Assessment and Monte Carlo simulations (48-59), and general risk-based decision making (31).   Such learnings could help those in the GMP environment address the following questions with effective solutions:

  • How can we quantitatively measure how much risk reduction our risk assessment work is really delivering?  Much of GMP is about quantitative measurements – measuring the variability associated with analytical test methods via intermediate precision experiments during method validation is a simple example – but such approaches seem to be totally lacking when it comes to QRM-related activities.    
  • How can we deal with the factors that adversely affect brainstorming and decision making during risk assessment exercises (e.g. subjectivity, risk perception, human heuristics)?
  • How can we obtain reliable probability estimates for failure modes and hazards?

Developing QRM Certification programmes that are tailored for the GMP environment has the potential to lead to better trained QRM facilitators, the correct use of QRM tools, more effective QRM work, better manufacturing and control processes, and ultimately better quality medicines for patients.

2. Work to Better Understand Informal vs. Formal Risk Assessment

There is some debate as to what the terms ‘informal’ and ‘formal’ risk assessment actually mean.  While this does not seem to be an area of much interest in the literature at this time, discussions with other regulators and with the pharmaceutical industry indicate that there is no general consensus on the definitions of these terms.  

ICH Q9 refers to the concept of formality in one of its two key principles of QRM…. stating, “the level of effort, formality and documentation should be commensurate with the level of risk.”

A simplistic approach often employed is to regard any risk assessment that is based on the use of a “tool” as being formal, while anything else is informal.  This is probably too simplistic an approach. 

Structured research activities in this area within the GMP environment and with academia have the potential to be of value and address questions such as: 

  • What is meant by informal vs. formal risk assessment? 
  • When each is appropriate to use each?
  • What are the risks and benefits with each?   
  • What safeguards need to be in place for each?
  • What benefits have the more formal approach delivered to date, and likewise, have less formal approaches also added value?  

It would be of great interest retrospectively to study how the concept of formality has been applied in the industry and to understand the relative contributions of each approach. 

3. Utilise Knowledge Management (KM) in QRM

ICH Q10 identifies KM and QRM as twin enablers of an effective pharmaceutical quality system.  However, it seems that the industry is still struggling with ways to turn data and information into real usable knowledge to support QRM work.  This was evidenced by the discussions that took place at an international symposium on KM held in Dublin, Ireland, in March 2015.  This event was attended by international regulators, members of both the ICH Q10 and ICH Q12 teams, knowledge management thought leaders, industry professionals, and KM practitioners; it received broad support for addressing this important topic for the industry.  The symposium, which was hosted by Regulatory Science Ireland (RSI), explored current best practices in the field of knowledge.  It also explored the challenges of enabling knowledge flow, achieving more effective knowledge sharing, and the relationship between KM and QRM, in identifying and addressing product and patient risk.

Some practical suggestions for consideration in this area include:  

  • Repositioning Annual Product Quality Reviews (PQRs) as KM Tools. As anyone in the industry will acknowledge, PQRs involve a lot of work.   However, the value derived from PQRs, in relation to process understanding and the effectiveness of the pharmaceutical quality system, is not always evident.  Writing PQRs sometimes seems to be more of a compliance-led tick-box exercise rather than as a valuable examination of process robustness and validation status.  It is questionable whether the industry is realising the benefits from its extensive PQR work. A solution might be to reposition PQRs as KM tools, where their expected outputs would be increased knowledge about a process and product, including process robustness and validation status.  They could also be repositioned as Risk Review documents, which is essentially what they are – a tool for reviewing a product and a process over a period of time from a risk perspective.   
  • Critiquing what is known about the controls that are put forward in risk assessments as risk mitigating controls.  Just because controls are documented in risk assessment exercises does not mean that they are effective or even relevant in risk control and mitigation as evidenced by some of the case studies presented above.  Key questions that might be asked in every risk assessment might be:
    • What exactly is this measure controlling?
    • What do we know about the effectiveness of this control?
    • What evidence is needed for us to know whether our risk assessment outputs are reliable?

4. More Focus on Science

As discussed above, there is a need for the pharmaceutical industry to apply good science when doing QRM work.  There are some very simple things that can be done to help achieve this in risk assessment activities.  

  • Design GMP-tailored risk assessment tools which do not allow P, S, or D scores to be assigned to failure modes without first considering the existence and effectiveness of GMP controls that influence those scores.  This would enable the application of an evidence basis to inform the risk assessment.   
  •  In recognising that RPNs generated by FMEA-type approaches are based on ordinal number scales with often high levels of subjectivity and are not real values (60-63), move away from using RPN thresholds alone in risk decision-making, especially about whether to implement additional risk controls or not (30). 
  •  Develop more quantitative ways to measure how much risk reduction is achieved via risk control and mitigation activities, and transition away from the current situation where risk reduction is either not measured at all, or is estimated using quite qualitative means (high, medium or low risk ratings etc.)  Other industries have focussed efforts in this area, such as the US aeronautics industry (64-65).

 5. Apply Increased Rigour

Rigour is especially important during formal risk assessment work where the influences of human heuristics and other risk perception factors can have a disproportionately large effect.  There are several very simple measures that can be taken to achieve this.  For example, the industry can design GMP-tailored risk assessment tools that require all GMP controls that are identified as important in risk control or mitigation (current and new) to be assessed under the following criteria:

  • Their actual effectiveness in controlling or mitigating those risks
  • Their qualification or validation requirements, if any.

In addition, one can apply triage methods to filter large numbers of potential failure modes down to the most important ones, and to risk assess the most important in detail and with increased rigour. The Holographic Hierarchical Modelling approach developed by the University of Virginia in conjunction with NASA in the US is one such approach worth exploring for its potential GMP applications.  Here, hundreds of potential hazards are filtered and those that are likely to pose the greatest risk are identified for further study (66). 



This paper reflects a regulator’s perspective on the developments in the application of Quality Risk Management by the pharmaceutical industry since ICH Q9 was finalised in November 2005.  Data on quality defects and recalls, information on inspectional observations and deficiencies, and some personal reflections are presented and commented upon in order to explore where things stand at present.  A number of recommendations are also made for the practical resolution of the problem issues and challenges that have been identified.  

While a significant amount of work has been done by regulators and industry alike since 2005 to embed the principles and concepts of QRM into the GMP environment, indications are that this work has not delivered its expected benefits.  It seems that ICH Q9 has not been implemented as intended, and it is not clear whether medicines are any safer now than they were ten years ago before ICH Q9.

While there is much food for thought for the pharmaceutical manufacturing sector presented here, it is important to also recognise that, in my own opinion at least, many of the problem issues cited in this paper also apply to the GMP inspectorates and their wider National Competent Authorities that regulate that sector.  As regulators work to apply risk-based thinking and approaches in their own day-to-day work activities, the problems of risk perception, subjectivity and uncertainty, and the challenges inherent in measuring how much risk reduction actually achieved via risk control and mitigation activities, are as applicable to them as the industry they regulate. So there is still much to do!   

It is worth noting that within competent authorities, work is underway to address a number of these challenges.  For example, advanced training activities for GMP Inspectors and Investigators on QRM began in December 2014 via an initiative of the PIC/S Expert Circle on Quality Risk Management.  This followed more basic QRM training in 2010.  These training activities were, and are, aimed at inspectors and investigators from as across the world, including Asia, Europe, the Americas, and Africa.   

Perhaps the pharmaceutical manufacturing sector can also consider what advanced training and learning it requires in order for it to achieve the expected benefits for itself and for patients that are envisaged by ICH Q9.

It is within a spirit of openness and reflection that this paper seeks to present these ideas for discussion and debate.



    1. Quality Risk Management (ICH Q9), International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use, November 9th, 2005, available at www.ich.org
    2. Pharmaceutical cGMPs for the 21st Century: A Risk-Based Approach, FDA Press Release, No. P02-28, FDA News, August 21, 2002, available at http://www.fda.gov/bbs/topics/NEWS/2002/NEW00829.html)  
    3. US FDA Guidance for Industry: PAT — A Framework for Innovative Pharmaceutical Development, Manufacturing, and Quality Assurance, September 2004, available at http://www.fda.gov/downloads/Drugs/Guidances/ucm070305.pdf
    4. US FDA Guidance for Industry: Process Validation: General Principles and Practices, January 2011, http://www.fda.gov/downloads/Drugs/.../Guidances/UCM070336.pdf 
    5. The Rules Governing Medicinal Products in the European Community, Volume IV, published by the European Commission, and available at http://ec.europa.eu/health/documents/eudralex/vol-4/index_en.htm
    6. Reflection paper on a proposed solution for dealing with minor deviations from the detail described in the Marketing Authorisation for Human and Veterinary Medicinal products (including biological products), European Medicines Agency, 10 March 2006, EMEA document no. EMEA/INS/GMP/71188/2006
    7. EMA Guideline on Real Time Release Testing (formerly Guideline on Parametric Release), March 2012, EMA/CHMP/QWP/811210/2009-Rev1 
    8. EMA Guideline on process validation for finished products - information and data to be provided in regulatory submissions, February 2014, EMA/CHMP/CVMP/QWP/BWP/70278/2012-Rev1
    9. European Commission Guidelines on the formalised risk assessment for ascertaining the appropriate GMP for Excipients of medicinal products for human use, March 2015, available at http://eur-lex.europa.eu/legal-content/ 
    10. PIC/S Recommendation for Risk-based GMP Inspection Planning, Jan 2012, available at  http://picscheme.org/
    11. WHO Guideline on Quality Risk Management, August 2012, available at http://www.who.int/medicines/areas/quality_safety/quality_assurance/Qual...
    12. ASTM E2500, 2007: A Standard Guide for the Specification, Design and Verification of Pharmaceutical & Biopharmaceutical Manufacturing systems and Equipment, available at http://www.astm.org/DATABASE.CART/HISTORICAL/E2500-07.htm
    13. GAMP® 5: A Risk-Based Approach to Compliant GxP Computerized Systems, February 2008, available from http://www.ispe.org/gamp-5
    14. ISPE Good Practice Guide titled Project Management for the Pharmaceutical Industry, January 2012, available from http://www.ispe.org/
    15. PDA Technical Report No. 54 titled Implementation of Quality Risk Management for Pharmaceutical and Biotechnology Manufacturing Operations, March 2012, available fromhttps://store.pda.org/ProductCatalog/
    16. PDA Technical Report No. 68 titled Risk-based Approach for Prevention and Management of Drug Shortages, January 2015, available fromhttps://store.pda.org/ProductCatalog/
    17. ISPE Risk MaPP Baseline Guide: Risk-Based Manufacture of Pharmaceutical Products, September 2010, available from http://www.ispe.org/baseline-guides/risk-mapp
    18. ISPE Drug Shortages Prevention Plan, Oct 2014, available at http://www.ispe.org/drug-shortages-initiative 
    19. Kay, A., Risk Based Assessment of a Biotechnology Process, paper presented at the 2005 Pharmaceutical Manufacturing Workshop, Dublin, Sept. 27th – 30th, 2005
    20. Mollah, A. H. Application of Failure Mode and effect Analysis (FMEA) for Process Risk Assessment, BioProcess International, November 2005
    21. Tidswell, Edward C., McGarvey, B, Quantitative Risk Modelling in Aseptic Manufacture, PDA Journal of Pharmaceutical Science and Technology, Vol. 60, No. 5, Sept. - Oct. 2006, pp 267-283
    22. O’Donnell, K., Greene, A., A Risk Management solution designed to facilitate risk-based Qualification, Validation & Change Control activities within GMP and Pharmaceutical Regulatory Compliance Environments in the EU, Parts I & II, Journal of GXP Compliance, Vol. 10, No. 4, July 2006
    23. Vesper, J. L., Risk Assessment and Risk Management in the Pharmaceutical Industry, Davis Healthcare International Publishing, 2006
    24. Butler, S., Incorporating Risk Management into the Quality System [at Wyeth Biopharma, Grange Castle], 2007 Qualified Person Forum meeting, School of Pharmacy, Trinity College, Dublin, May 17th, 2007
    25. Cleaning and Cleaning Validation, Volume 2, edited by Paul Pluta, published in March 2013 by the PDA and available from https://store.pda.org/ProductCatalog/
    26. Calnan, N., O’Donnell, K., Greene, A., Enabling ICH Q10 Implementation – Part 1: Striving for Excellence by Embracing ICH Q8 and ICH Q9, PDA J. Pharm. Sci. Technol., November/December 2013, Vol. 67, No. 6, pp 1-20
    27. For information in this regard, see the Quality Defect and Recall sections of the Irish Medicines Board (IMB) and Health Products Regulatory Authority (HPRA) Annual Reports for 2006 through 2014, available from www.hpra.ie.
    28. Waldron, K., Greene, A., Calnan, N, Quality Risk Management: State of the Industry—Part 1. Has the Industry Realized the Full Value of ICH Q9?, Journal of Validation Technology,  Vol. 20, No. 4, January 2015
    29. O’Donnell, K., Greene, A., Failure Modes - Simple Strategies for improving qualitative Quality Risk Management exercises during Qualification, Validation and Change Control Activities, Journal of Validation Technology,  Vol. 13, No. 2, February 2007
    30. The paper by Kelly in this special issue of the Journal in relation to risk analysis and risk rating scales 
    31. Morgan, M. G., Henrion, M., “Uncertainty – A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis”, Cambridge University Press, 1990
    32. Faust, D. Declarations versus investigations: The case for the special reasoning abilities and capabilities of the expert witness in psychology/psychiatry, Journal of Psychiatry & Law, 13(1–2), pp 33–59, 1985
    33. Goldberg, L. R. “The effectiveness of clinicians’ judgements: the diagnosis of organic brain damage from the Bender-Gestalt test”, Journal of Consulting Psychologists, 23:23-33, 1959
    34. Morgan, M. G., Morris, S. C., Henrion, M., Amaral, D. A. L., Rish, W. R., “Technical Uncertainty in Quantitative Policy Analysis: A Sulphur Air Pollution Example”, Risk Analysis, September 4, 1984, 201-216
    35. Rausand, M., Høyland, A, System Reliability Theory; Models, Statistical Methods and Applications" (Second Edition), Wiley, New York, 2004
    36. Pyy, P., Human reliability analysis methods for probabilistic safety assessment, Espoo 2000, Technology Research Centre of Finland, VTT Publications, December 2000, available at www.vtt.fi/inf/pdf/publications/2000/P422.pdf
    37. Kahneman, D., Tversky, A., “On the Psychology of Prediction”, Psychological Review, 80, No 4, 237 – 251, 1973
    38. Kahneman, D., Tversky, A., “Subjective Probability: A Judgement of Representativeness”, Cognitive Psychology, 3:430-354, 1972
    39. Fischhoff, B., Slovic, P., Lichtenstein, S., Fault trees: sensitivity of estimated failure probabilities to problem representation, J. Exp. Psychol. – Human Perception Perf. 4:330-44, 1978
    40. Slovic, P., Peters, E., “Risk Perception and Affect,” Current Directions in Psychological Science, Vol. 15 (2006), pp. 322–325
    41. Savadori, L., Savio, S., Nicotra, E., Rumiati, R., Finucane, M., & Slovic, P. (2004). Expert and public perception of risk from biotechnology, Risk Analysis, 24(5), pp. 1289–1299
    42. Ronteltap, A., van Trijp, J.C.M., Renes, R.J., Frewe, L.J., “Consumer acceptance of technology-based food innovations,” Appetite, Vol. 49, pp. 1–17, 2007
    43. Satterfield, T., et al., “Anticipating the perceived risk of nanotechnologies”, Nature Nanotechnology, Vol. 4 (2009), pp. 752-758
    44. Fischhoff, B., Bostrom, A., Quadrel, M., “Risk Perception and Communication,” Annual Review of Public Health, Vol. 14 (1993), pp.183–203
    45. Costa-Font, J., Rudisill, C., Mossialos, E., “Attitudes as an Expression of Knowledge and ‘Political Anchoring”, Risk Analysis, Vol. 28, No. 5 (2008), pp.1273-1287
    46. Keller, A. Z., Perception and Quantification of Risk, ISPRA courses, Reliability and Data, JRC ISPRA, 21020, Italy, 1984
    47. Hoch, S. J., Availability and Interference in Predictive Judgement, Journal of Experimental Psychology: Learning, Memory and Cognition, 10, No. 4, 1984
    48. Rasmussen, N. C., “The Application of Probabilistic Risk Assessment Techniques to Energy Technologies”, Ann. Rev. Energy, 1981, 6:123-38
    49. Stamatelatos, M., Probabilistic Risk Assessment – What is it and why is it worth performing?, NASA Office of Safety and Mission Assurance, May 4th, 2000, available from www.hq.nasa.gov, accessed November 28th, 2007
    50. Greenfield, M. A., “The Inherent Values of Probabilistic Risk Assessment”, (Presentation), June 19, 2001, NASA Office of Safety and Mission Assurance, available from http://www.hq.nasa.gov/office/codeq/risk/risk_archive.htm, accessed November 29th, 2007
    51. US Nuclear Regulatory Commission, Policy Statement in relation to the use of Probabilistic Risk Assessment by the NRC,  US Federal Register, Vol. 80, pp. 42662, August 10, 1995
    52. Mullen. T. M., Understanding and Supporting the Process of Probabilistic Estimation”, PhD Dissertation, Carnegie Mellon University, Pittsburgh, 1986
    53. Hickman, J. W., et al., “PRA Procedures Guide – A Guide to the Performance of Probabilistic Risk Assessments for Nuclear Power plants”, American Nuclear Society, NUREG/CR-2300, Vols. 1 and 2, January 1983
    54. US Nuclear Regulatory Commission, Regulatory Guide 1.200 for Trial Use: “An Approach for Determining the Technical Adequacy of Probabilistic Risk Assessment Results for Risk-Informed Activities”, February 2004
    55. American Society of Mechanical Engineers, Standard for Probabilistic Risk Assessment for Nuclear Power Plant Applications, ASME RA-S-2002, April 2002
    56. US Nuclear Regulatory Commission, An Approach for Using Probabilistic Risk Assessment in Risk-Informed Decisions on Plant-Specific Changes to the Licensing Basis, NRC Regulatory Guide No. 1.174, Revision 1, November 2002
    57. Kunreuther, H., Novemsky, N., Kahneman, K., “Making Low Probabilities Useful,” Journal of Risk and Uncertainty, Vol. 23, No. 2 (2001), pp. 103–120
    58. MacDonald, J.A., Small, M.J., Morgan, M.J., “Explosion Probability of Unexploded Ordnance”, Risk Analysis, Vol. 28, No. 4 (2008), pp. 825-841
    59. Vose, D., Quantitative Risk Analysis: A Guide to Monte Carlo Simulation Modelling, John Wiley & Sons, New York, 1996
    60. Schmidt, M., W., The Use and Misuse of FMEA in Risk Analysis, Medical Device & Diagnostic Industry, March 2004 
    61. Stamatis, D. H., Failure Mode and Effect Analysis: FMEA from Theory to Execution, 2nd edition, ASQ Quality Press, 2003
    62. Kmenta, S., Ishii, K., Scenario-Based Failure Modes and Effects Analysis Using Expected Cost, Journal of Mechanical Design, November 2004, Volume 126, Issue 6, pp. 1027-1035 
    63. Rhee, J., Ishii, K., Using cost based FMEA to enhance reliability and serviceability, Advanced Engineering Informatics, Volume 17, Issues 3-4, July-October 2003, pp 179-188 
    64. Greenfield, M. A., “Risk Management – Risk as a Resource”, (Presentation), NASA Office of Safety and Mission Assurance, May 14, 1998, available from http://www.hq.nasa.gov/office/codeq/risk/images/risk.pdf, accessed November 28th, 2007
    65. Greenfield, M. A.,  “Risk Management - What have we Learned”, (Presentation at NASA’s May 2001 Symposium on Risk), NASA Office of Safety and Mission Assurance, May 9, 2001, available at:           http://www.hq.nasa.gov/office/codeq/risk/docs/incose.pdf, accessed November 28th, 2007
    66. Risk Anal 200 Risk filtering, ranking, and management framework using hierarchical holographic modeling 2 Apr; 22(2):383-97




The views expressed in this paper are those of the author and do not necessarily represent the views of the Health Products Regulatory Agency, Dublin, Ireland.




The author thanks Cliff Campbell and Deirdre O’Keeffe for valuable discussions on QRM and risk-based validation during the writing of this paper.


Thanks also to Aideen Quigley, Anne Greene, Nuala Calnan and Kelly Waldron for assistance in getting this paper written.





Product Added Successfully

This product has been added to your account and you can access it from your dashboard. As a member, you are entitled to a total of 0 products.

Do you want access to more of our products? Upgrade your membership now!

Your Product count is over the limit

Do you want access to more of our products? Upgrade your membership now!

Product added to cart successfully.

You can continue shopping or proceed to checkout.

Comments (0)

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
  • Use to create page breaks.
Enter the characters shown in the image.
Validated Cloud logo