Decision Rules and Risk

ISO/IEC 17025:2017
Workshop

T&M 2019 Conference and Workshop

16th September 2019
Steve Sidney and John Wilson
1
Introduction

This presentation is presented in conjunction with the

presentation by John Wilson who will cover the Decision Rule
and Statements of Conformity now incorporated in the
ISO/IEC 17025:2017, as well as a summary of what is in the
New ILAC G8:2019 guidance document.

This first section will deal with Risk from a generic point of
view, explain how to go about assessing Risk and then more
specifically Risk in the lab context as well as the requirements
of ISO/IEC 17025:2017. 2
What is risk ?

Uncertainty
what makes achieving an objective uncertain

Level of Risk
takes into account consequences and likelihood of
situations

3
What is risk ?

ISO 17666:2003 – risk

“undesirable situation or circumstance that has both a

likelihood of occurring and a potentially negative
consequence (impact) ……..”

4
 Likelihood of being
bitten

Impact of being bitten

5
Another view

Outcome (Negative) – uncertain – risk

Outcome (Positive) – uncertain – opportunity

6
 Risk and Opportunity
Four phases of risk
Four Phases of Risk
Risk Analysis Risk Evaluation

* Intended user identification * Risk acceptability decisions

* Area wise risk identification
* Risk estimation

Risk Monitoring and Control Post test operation information

* Option analysis * Post-production experience

* Implementation of measures * Review of Risk management
* RESIDUAL RISK evaluation experrience – customer view
* Overall RISK acceptance * Take appropriate action

7
Risk identification

What can happen and why ?

What are the consequences ?

What is the probability of it occurring ?

8
Rating the evaluation – Impact

Low (1) – easy to correct

Moderate (2) – errors occurring again but clear

High (3) – serious errors with possible irreparable

consequences

9
Rating the evaluation – Probability

Low – very rare (1)

Medium – rare (2)

High - frequently (3)

10
Effect of uncertainty (risk)

Risk = Impact * Probability

or

Risk = Consequences * Likelihood

11
Scaling – Version 1

12
Evaluation – 1

Lowest (green) – acceptable

Highest (red) – requires action

Medium (yellow) – decide if still acceptable or what

action/s to take

13
Scaling – Version – 2

14
Mitigation

Factors that mitigate the consequence of the risk

or

Reduce the probability of the risk

15
In ISO/IEC 17025 is the concept new ?

…NOTE 2: Apart from the review of the operational

procedures, the preventive action might involve
analysis of data, including trend and risk analyses and
proficiency-testing results…………. ISO/IEC 17025:2005
Clauses: 4.11, 4.12
The procedure for corrective action shall start with an
investigation to determine the root cause(s) ………..
Corrective actions shall be to a degree appropriate to
the magnitude and the risk of the problem.
16
ISO/IEC 17025:2005 vs 2017

2017: Risk and

1999/2005: Managed Risk
Opportunity Managed
Quality Manual Documented Information
Policies Processes
Procedures Decision Rules
Job Descriptions
Top Management
QM & TM 17
2017 vs 2005

17025:2005 17025:2017
Lab shall have policies and The Lab shall ensure the
procedures to ensure protection of confidential
protection of confidential information.. including
information…incl electronic electronic storage and
storage and transmission of transmission of results
results

18
19
Foreword

the risk-based thinking applied in this edition has

enabled some reduction in prescriptive requirements
and their replacement by performance-based
requirements

20
Introduction

… document requires the laboratory to plan and

implement actions to address risks and opportunities.
Addressing both risks and opportunities establishes a
basis for increasing the effectiveness of the
management system, achieving improved results and
preventing negative effects. The laboratory is
responsible for deciding which risks and opportunities
need to be addressed.
21
Dealing with risk

Relaxation of prescription makes it essential for each

lab to consider the risk for each clause

• Discuss and agree what measures are required

• Implement in a known and controlled way for
consistency
• Review
22
The general case

Risk based approach is where

Breadth and depth of implementation of clause is

varied to suit perceived risk for the particular
laboratory

23
Risk in the context of measurement

uncertainty on meeting the objectives of the

measurement

a) deviation from the expected measurement result

b) outside the calibration and measurement

capability (CMC), testing capabilities or the stated
uncertainties
24
 Laboratory activities

Environmental Technical
Reliability Safety
Internal Financial
External Supply chain
Management Impartiality
Customer Security 25
4.1.4 & 4.1.5 (Impartiality)

Evaluate an on-going basis… include those risks that

arise from its activities, ..its relationships, or
relationships of its personnel.

If a risk to impartiality is identified, the laboratory shall

be able to demonstrate how it eliminates or minimizes
such risk.

26
4.1.4 & 4.1.5 (Impartiality)

NOTE: A relationship that threatens the impartiality of

the laboratory can be based on ownership, governance,
management, personnel, shared resources, finances,
contracts, marketing (including branding), and payment
of a sales commission or other inducement for the
referral of new customers, etc

27
Risks re impartiality

Presence of objectivity
freedom from conflict of freedom from bias
interest
lack of prejudice neutrality
fairness open-mindedness
even-handedness detachment
balance
28
7.8.6.1 (Decision rules)

When a statement of conformity to a specification or

standard is provided, the laboratory shall document the
decision rule employed, taking into account the level of
risk (such as false accept and false reject and statistical
assumptions) associated with the decision rule
employed, and apply the decision rule.
NOTE Where the decision rule is prescribed by the
customer, regulations or normative documents, a
further consideration of the level of risk is not
necessary. 29
Decision rule – definition

Rule that describes how measurement uncertainty is

accounted for when stating conformity with a specified
requirement

Calling a test/calibration result a “pass” when it is really

a “fail” (false accept), or calling something a “fail”
when it is really a “pass” (false reject)

30
7.10.1 (Non-conforming work)

b) actions (including halting or repeating of work and

withholding of reports, as necessary) are based upon
the risk levels established by the laboratory;

31
Risks – NC work and Corrective Actions

Reference standard out of tolerance (CRM, Ref

Standard)
Wrong procedure or out of range
Reagents expired

Likelihood of recurrence
Impact
32
8. Management System Requirements

8.5 (Actions address risks and opportunities)

8.6 (Improvement)
8.7 (Corrective action)
8.9 (Management review)

33
Clause 8.5.1 – Purpose

The laboratory shall consider the risks and opportunities

associated with the laboratory activities in order to:
a) give assurance that the management system can achieve its
intended results;
b) enhance opportunities to achieve the purpose and
objectives of the laboratory;
c) prevent, or reduce, undesired impacts and potential failures
in the laboratory activities; and
d) achieve improvement.
34
8.5.2 & 8.5.3 – Lab shall….

The laboratory shall plan:

a) actions to address these risks and opportunities;
b) how to:
integrate and implement the actions into its management
system;
evaluate the effectiveness of these actions.

Actions taken to address risks and opportunities shall be

proportionate to the potential impact on the validity of
laboratory results. 35
Risk Assessment Example

Evaluate Infrastructure

System or Instruments
Consider
Supportability
Capacity
Efficiency

36
Risk Cube and Risk Scoring

High 5 5 10 15 20 25
Med/High 4 4 8 12 16 20
Consequence

Medium 3 3 6 9 12 15
Med/Low 2 2 4 6 8 10
Low 1 1 2 3 5 5
1 2 3 4 5
Low Med/Low Medium Med/High High
Likelihood
37
Supportability

The condition of the various components including

frequency and cost of repairs, as well as obsolescence
issues.

38
Ranking Consequence Likelihood
System in good working condition, no
Low obsolescence issues <10 %
Medium- System exhibits problems, but can be maintained
10 to 30 %

Supportability
Low Some components at or near end of support

System exhibits problems that cannot be

Medium mitigated 30 to 60 %
More than half components no longer supported

Medium- System routinely unavailable

High Majority of components lack support 60 to 85 %
System non-functional
High System completely obsolete >85 % 39
Evaluation

System 10 years old

Failed and been unavailable 3 times in past 2 years
Each time down unavailable for 4 weeks
Some components still available
Repair service still available
System unavailable for 12 weeks in 2 years = 12 %

Consequence(Medium) criticality raises this to Med/High

Likelihood (Med/Low)
40
Supportability

High 5
Med/High 4 X
Consequence

Medium 3
Med/Low 2
Low 1
1 2 3 4 5
Low Med/Low Medium Med/High High
Likelihood
41
Capacity

The ability of the various components to produce the

required amount of workload in order to meet
customer requirements.

42
Ranking Consequence Likelihood
System capacity exceeds demand requirement
Low No impact to availability <10 %
Medium- System capacity meets current demand
Low Any increase may stress system 10 to 30 %

Capacity
System capacity occasionally fails to keep up
with demand
Medium Some impact on availability 30 to 60 %

System demand exceeds capacity

Medium-
High
Some equipment unavailable resulting in 60 to 85 %
complaints

High Demand exceeds capacity continually >85 % 43

Evaluation

Capable of 4 tests/calibrations per week

Adequate for current workload
One/two months/year demand increases 6 per week
Expansion likely in next 3 years to 10 months of year
Current failure to meet demand 17 %
Future failure to meet demand 83 %

Consequence – Med High

Likelihood – Med/Low High
44
Capacity

High 5 X
Med/High 4
Consequence

Medium 3 X
Med/Low 2
Low 1
1 2 3 4 5
Low Med/Low Medium Med/High High
Likelihood
45
Efficiency

The amount of effort required to conduct the

measurements and the ease of use of the components.

46
Ranking Consequence Likelihood
Low System requires minimal effort to operate <10 %
Medium- System requires some effort, but effort isn’t
Low taxing 10 to 30 %

Efficiency
System requires constant attention and
Medium interaction with analyst/technician 30 to 60 %

System is difficult to operate

Medium-
High
Some measurements have to be repeated to 60 to 85 %
validate results
System tedious to operate
High All measurements made on system impacted >85 % 47
Evaluation

System partly automated

Analyst/technician required to monitor computer to perform manual
functions
– 65 % of their time spent waiting, can’t perform other work

Consequence – Med
Likelihood – High

48
Efficiency

High 5
Med/High 4
Consequence

Medium 3 X
Med/Low 2
Low 1
1 2 3 4 5
Low Med/Low Medium Med/High High
Likelihood
49
Lab System Health Assessment

System Date Supportability Capacity Efficiency Overall Risk Score

A xx/xx/yyyy High Medium Medium Medium 39

B xx/xx/yyyy Low Low Low Low 10
C xx/xx/yyyy Medium Low Medium Medium 28
D xx/xx/yyyy High High Medium High 57

50
Summary

What is risk
How to evaluate
Risk based approach in measurement
Risk in ISO/IEC 17025:2017
An example

51
Thank you !

52