![Optimizing use of business analytics and lab-oriented statistical
software to establish robust and pertinent reference intervals
Ashleigh B. Muenzenmeyer, B.S., C(ASCP)CM
, Edmunds Z. Reineks, M.D., Ph.D.
Cleveland Clinic, Cleveland, OH
CMP Data from
performing lab, dates
12.01.15-12.08.15
Total Protein
“Healthy” Pospulation
N=638 (5%)
Total Protein
N=12,026
• Include biochemically normal
Albumin, Alkaline
Phosphatase, and Alanine
Aminotransferase
• Exclude samples with
biochemically abnormal values
for the above analytes
• Include outpatients
• Exclude inpatient and
emergency room samples
• Exclude cancer center and
dialysis order locations
• Include all other locations
• Include adults
• Exclude all other
age populations
• Include order diagnosis code of
“encounter for general adult
medical examination”
• Exclude all other diagnosis codes
Background
The patient population served by our automated chemistry laboratory has
grown outside our local population due to rapid expansion of our health
system, as well as the increased nationwide geographical footprint of the
Cleveland Clinic’s reference laboratory. Population-based reference
intervals are a set of values classified by upper and lower reference
parameters, which typically represents the central 95% of values from the
reference population of normal, healthy, control subjects. Given our
growing and likely evolving patient population, our existing reference
intervals were re-evaluated for their appropriateness. Various challenges
accompany the process of validating or verifying reference ranges.
Traditional approaches may have some limitations, including a lack of
laboratory resources, inadequate availability of specimens from normal,
healthy subjects (especially for partitioned reference intervals, e.g. due to
patient demographics), insufficient or inefficient access to LIS or EMR
data, or inappropriate starting point reference intervals based on literature
or vendor-provided information. Establishing de novo reference intervals
for common metabolic analytes was preferred (vs. verifying other
intervals) because the new ranges would reflect the actual patient
population being served. The goal of this study was to establish
laboratory-specific de novo reference intervals for 12 common metabolic
analytes by leveraging multiple software tools and existing patient results.
This is an example of how laboratories can efficiently utilize analytics to
drive better patient care.
Methods
This study utilized Altosoft (Kofax, Irvine, CA), “code-free” business
intelligence software to readily identify suitable existing patient samples
where results were stored in our laboratory information system (Sunquest,
Tucson, AZ). Data was exported into Excel (Microsoft, Redmond, WA)
and filtered according to pre-defined, medical director-approved
qualifications (including visit-related diagnosis codes) which resulted in
the datasets used for further analysis. The datasets were evaluated using
an EP Evaluator®
(Data Innovations, South Burlington, VT) statistical
module, entitled “Establish Reference Interval (EST).” This Establish RI
module uses the nonparametric method in accordance with CLSI: C28-A
guidelines to calculate the reference interval (based on central 95% of
results from healthy subjects)
Methods cont. Results cont.
Figure 1: Total Protein’s Data Extraction and Filtration Processes
This figure illustrates the data flowchart for the processes utilized to extract and manipulate data. This example
displays the steps employed in pulling and filtering data for the establishment of Total Protein reference intervals. In
this example, data were extracted using Altosoft for Comprehensive Metabolic Panels during the specified timeframe
(purple). The next set of processes displayed involves filtering the dataset in Excel for pre-defined inclusion and
exclusion criteria (blue). Upon completion of filtering, the analyst possesses a dataset that represents a “healthy”
population (green).
Results
Reference intervals were established using analysis of historical data for 12 common
metabolic analytes. The number of patient results used in establishment (N) far
exceeded the traditionally recommended minimum of 120 samples, and ranged from
540 to 646 for each analyte.
Figure 3: Number of Samples per Reference Interval Study
This figure conveys the robustness of this overall process by displaying the number of samples for each of the 12 common metabolic
analytes. The purple, bolded, horizontal line represents the minimum number of samples in a reference interval study as recommended by
CLSI guidelines, which requires a minimum of 120 samples for establishing a reference interval by nonparametric methods.1
This graph
confirms that the results of these studies yielded significantly higher sample sizes than traditional methods.
Figure 4: Comparison of Established, Package Insert, and Current Reference Intervals
This figure exhibits the similarities and differences between the established, package insert, and current reference intervals for the following
analytes: Aspartate Aminotransferase (AST) package insert3
, Creatinine (Creat) package insert4
, Chloride (Cl) package insert5
, and Total
Protein package insert6
.
Table 1: Comparison of Established, Package Insert, and Current Reference Intervals
This table reveals the results and compares the established, package insert, and current reference intervals for the
following analytes: Albumin package insert7
, Calcium package insert8
, Alkaline Phosphatase package insert9
, Alanine
Aminotransferase (ALT) package insert10
, Blood Urea Nitrogen (BUN) package insert11
, Bicarbonate (CO2) package
insert12
, Potassium (K) package insert5
, and Sodium (Na) package insert5
.
540
274 266
333
305
274 266
285
266
551
646 646
340
306
646
551
638
0
100
200
300
400
500
600
700
Albumin Alk Phos
Male
Alk Phos
Female
AST
Male
AST
Female
ALT
Male
ALT
Female
BUN
Male
BUN
Female
Calcium Chloride CO2 Creat
Male
Creat
Female
Potassium Sodium Total
Protein
Number of Samples Per RI Study
0 10 20 30 40
CR
PI
EST
AST U/L Females
0 10 20 30 40
CR
PI
EST
AST U/L Males
95 100 105 110
CR
PI
EST
Chloride mmol/L
0.5 0.7 0.9 1.1 1.3 1.5
CR
PI
EST
Creatinine mg/dL Females
0.5 0.7 0.9 1.1 1.3 1.5
CR
PI
EST
Creatinine mg/dL Males
6 6.5 7 7.5 8 8.5 9
CR
PI
EST
Total Protein g/dL
Low High Low High
Established 3.9 4.9 Established 8.6 10.0
Package Insert 3.97 4.94 Package Insert* 8.6 10.2
Current 3.5 5.0 Current 8.5 10.5
Low High Low High
Established 32 117 Established 36 108
Package Insert 35 104 Package Insert 40 129
Current 40 150 Current 40 150
Low High Low High
Established 7 38 Established 10 54
Package Insert 0 33 Package Insert 0 41
Current 0 45 Current 5 50
Low High Low High
Established 7 21 Established 9 24
Package Insert* 6 23 Package Insert* 6 23
Current 8 25 Current 10 25
Low High Low High
Established 22 30 Established 3.7 5.1
Package Insert 22 29 Package Insert** 3.4 5.1
Current 23 32 Current 3.5 5.0
Low High Low High
Established 136 144 Established 136 144
Package Insert 136 145 Package Insert 136 145
Current 132 148 Current 135 146
BUN (mg/dL)
Females
BUN (mg/dL)
Males
CO2
(mmol/L)
Potassium
(mmol/L)
Sodium
(mmol/L)
Females
Sodium
(mmol/L)
Males
Albumin
(g/dL)
Calcium
(mg/dL)
Alkaline
Phosphatase
(U/L)
Females
Alkaline
Phosphatase
(U/L) Males
ALT (U/L)
Females
ALT (U/L)
Males
*Age partitions combined.
**Serum/plasma ranges combined.
Conclusions
Data mining tools and real-time analytics utilization was used for robust
establishment of reference intervals that would not be feasible to achieve
with our historical methods. Improvements over our previous methodology
included: 1) Establishing reference intervals for common analytes utilized
large datasets to produce de novo ranges that are truly representative of our
patient population. 2) After gaining familiarity with the analytic tools, the
process proceeded quickly. 3) Readily accessible and exportable data via
the business intelligence software allowed us to bypass our previous
reliance on IT expertise and availability to conduct our data searches. 4)
Although we utilized EP Evaluator®
software in this project, the statistical
analysis of the healthy population dataset could be performed in most
spreadsheet programs with either default functions or via add-in packages.
References
1. CLSI. Defining, Establishing, and Verifying Reference Intervals in the Clinical Laboratory; Approved
Guideline—Third Edition. CLSI document EP28-A3c. Wayne, PA: Clinical Laboratory Standards Institute; 2008
2. EP Evaluator® 10.3.0.556. Establish Reference Interval Report Interpretation Guide. Data Innovations, LLC,
South Burlington, VT; 1991-2013.
3. Aspartate Aminotransferase acc. to IFCC without pyridoxal phosphate activation (AST) [package insert V 4.0
English]. Roche Diagnostics, Indianapolis, IN; May 2014
4. Creatinine plus ver.2 (CREP2) [package insert V 8.0 English]. Roche Diagnostics, Indianapolis, IN; February 2016
5. ISE indirect Na, K, Cl for Gen.2 [package insert V 8.0 English]. Roche Diagnostics, Indianapolis, IN; January 2016
6. Total Protein Gen.2 (TP2) [package insert V 7.0 English]. Roche Diagnostics, Indianapolis, IN; February 2015
7. Albumin Gen.2 (ALB2) [package insert V 6.0 English]. Roche Diagnostics, Indianapolis, IN; March 2015
8. Calcium Gen.2 (CA2) [package insert V 3.0 English]. Roche Diagnostics, Indianapolis, IN; February 2014
9. Alkaline Phosphatase acc. To IFCC Gen 2. (ALP2) [package insert V 5.0 English]. Roche Diagnostics, Indianapolis,
IN; February 2015
10. Alanine Aminotransferase acc. to IFCC without pyridoxal phosphate activation (ALT) [package insert V 5.0 English].
Roche Diagnostics, Indianapolis, IN; July 2014
11. Urea/BUN (UREAL) [package insert V 5.0 English]. Roche Diagnostics, Indianapolis, IN; January 2016
12. Bicarbonate Liquid (CO2-L) [package insert V 9.0 English]. Roche Diagnostics, Indianapolis, IN; September 2014
Established
Package Insert
Current Range
Figure 2: Total Protein’s Statistical Analysis
This figure displays the statistical analysis in the EP
Evaluator’s Establish Reference Interval Module. This
example demonstrates the information generated for Total
Protein’s statistical analysis.
CLSI guidelines (CLSI document EP28-A3c, formerly
numbered C28-A3c) suggest using the nonparametric
method for estimating the reference interval.1
The
nonparametric method makes no assumption about the
shape of the population distribution—it ranks the samples
in ascending order.2 The number of samples in the overall
population determines which results are used to the lower
and upper thresholds—the software calculates 2.5% of the
total number of samples (N) which equals x.2
The
software then counts x samples from the lowest and
highest (while the results are in ascending order) and
assigns those values the lower and upper thresholds,
respectively.2
For example, the Total Protein Central 95%
Index counts from 16 to 623 because N= 638.
The parametric method assumes that the population fits a
Gaussian distribution or can be converted to fit a Gaussian
distribution.2
These conversions may include taking a log
or square root; however, the simplest parametric method
calculates the references limits as the sample mean ± 2
standard deviations.2
The transformed parametric method considers the population distribution shape and evaluates if the data can be scaled to fit a Gaussian
distribution. If so, the mean is computed ± 2 standard deviations and then converted back to the original units (EP Evaluator®
10.3.0.556).2
The confidence ratio determines the ratio average confidence interval width to the reference interval width using the formula:
0.5*(URLU-URLL+LRLU-LRLL)/(URL-LRL).2
The primary factor influencing this ratio is the sample size and the value of 0.10 or less is
preferred.2 In this example, the confidence ratio is acceptable at 0.12 with a sample size of N=638.
TP
ClinicalPathology--RT-PLMI CMP Est Hepatic2
Central 95% Interval
(N = 638)
Lower Upper
Value 90% CI Value 90% CI Ratio
Confidence
Nonparametric (CLSI C28-A) 6.3 6.2 to 6.5 8.0 8.0 to 8.1 0.12
Alternatives
Parametric 6.4 6.3 to 6.4 8.1 8.0 to 8.1 0.06
TransformedParametric -- -- -- -- --
Confidence Limits for Nonparametric CLSI C-28A method computed by exact formula.
Reference Interval Estimation: Combined
SelectionCriteria
Bounds None
Filter None
Statistics
Mean 7.23
SD 0.44
Median 7.20
Range 5.4 to 8.6
N 638 of 638
Distinct Values 27
Zeroes 0
Central 95% Index 16.0 to 623.0
Analyst ABM
ExptDate 12 Jan 2016
Histogram](https://image.slidesharecdn.com/328be7f4-6009-416e-8c69-40fbaf1bcd62-170202004541/85/A-Muenzenmeyer_AACC_2016-1-320.jpg)
A.Muenzenmeyer_AACC_2016
- 1. Optimizing use of business analytics and lab-oriented statistical software to establish robust and pertinent reference intervals Ashleigh B. Muenzenmeyer, B.S., C(ASCP)CM , Edmunds Z. Reineks, M.D., Ph.D. Cleveland Clinic, Cleveland, OH CMP Data from performing lab, dates 12.01.15-12.08.15 Total Protein “Healthy” Pospulation N=638 (5%) Total Protein N=12,026 • Include biochemically normal Albumin, Alkaline Phosphatase, and Alanine Aminotransferase • Exclude samples with biochemically abnormal values for the above analytes • Include outpatients • Exclude inpatient and emergency room samples • Exclude cancer center and dialysis order locations • Include all other locations • Include adults • Exclude all other age populations • Include order diagnosis code of “encounter for general adult medical examination” • Exclude all other diagnosis codes Background The patient population served by our automated chemistry laboratory has grown outside our local population due to rapid expansion of our health system, as well as the increased nationwide geographical footprint of the Cleveland Clinic’s reference laboratory. Population-based reference intervals are a set of values classified by upper and lower reference parameters, which typically represents the central 95% of values from the reference population of normal, healthy, control subjects. Given our growing and likely evolving patient population, our existing reference intervals were re-evaluated for their appropriateness. Various challenges accompany the process of validating or verifying reference ranges. Traditional approaches may have some limitations, including a lack of laboratory resources, inadequate availability of specimens from normal, healthy subjects (especially for partitioned reference intervals, e.g. due to patient demographics), insufficient or inefficient access to LIS or EMR data, or inappropriate starting point reference intervals based on literature or vendor-provided information. Establishing de novo reference intervals for common metabolic analytes was preferred (vs. verifying other intervals) because the new ranges would reflect the actual patient population being served. The goal of this study was to establish laboratory-specific de novo reference intervals for 12 common metabolic analytes by leveraging multiple software tools and existing patient results. This is an example of how laboratories can efficiently utilize analytics to drive better patient care. Methods This study utilized Altosoft (Kofax, Irvine, CA), “code-free” business intelligence software to readily identify suitable existing patient samples where results were stored in our laboratory information system (Sunquest, Tucson, AZ). Data was exported into Excel (Microsoft, Redmond, WA) and filtered according to pre-defined, medical director-approved qualifications (including visit-related diagnosis codes) which resulted in the datasets used for further analysis. The datasets were evaluated using an EP Evaluator® (Data Innovations, South Burlington, VT) statistical module, entitled “Establish Reference Interval (EST).” This Establish RI module uses the nonparametric method in accordance with CLSI: C28-A guidelines to calculate the reference interval (based on central 95% of results from healthy subjects) Methods cont. Results cont. Figure 1: Total Protein’s Data Extraction and Filtration Processes This figure illustrates the data flowchart for the processes utilized to extract and manipulate data. This example displays the steps employed in pulling and filtering data for the establishment of Total Protein reference intervals. In this example, data were extracted using Altosoft for Comprehensive Metabolic Panels during the specified timeframe (purple). The next set of processes displayed involves filtering the dataset in Excel for pre-defined inclusion and exclusion criteria (blue). Upon completion of filtering, the analyst possesses a dataset that represents a “healthy” population (green). Results Reference intervals were established using analysis of historical data for 12 common metabolic analytes. The number of patient results used in establishment (N) far exceeded the traditionally recommended minimum of 120 samples, and ranged from 540 to 646 for each analyte. Figure 3: Number of Samples per Reference Interval Study This figure conveys the robustness of this overall process by displaying the number of samples for each of the 12 common metabolic analytes. The purple, bolded, horizontal line represents the minimum number of samples in a reference interval study as recommended by CLSI guidelines, which requires a minimum of 120 samples for establishing a reference interval by nonparametric methods.1 This graph confirms that the results of these studies yielded significantly higher sample sizes than traditional methods. Figure 4: Comparison of Established, Package Insert, and Current Reference Intervals This figure exhibits the similarities and differences between the established, package insert, and current reference intervals for the following analytes: Aspartate Aminotransferase (AST) package insert3 , Creatinine (Creat) package insert4 , Chloride (Cl) package insert5 , and Total Protein package insert6 . Table 1: Comparison of Established, Package Insert, and Current Reference Intervals This table reveals the results and compares the established, package insert, and current reference intervals for the following analytes: Albumin package insert7 , Calcium package insert8 , Alkaline Phosphatase package insert9 , Alanine Aminotransferase (ALT) package insert10 , Blood Urea Nitrogen (BUN) package insert11 , Bicarbonate (CO2) package insert12 , Potassium (K) package insert5 , and Sodium (Na) package insert5 . 540 274 266 333 305 274 266 285 266 551 646 646 340 306 646 551 638 0 100 200 300 400 500 600 700 Albumin Alk Phos Male Alk Phos Female AST Male AST Female ALT Male ALT Female BUN Male BUN Female Calcium Chloride CO2 Creat Male Creat Female Potassium Sodium Total Protein Number of Samples Per RI Study 0 10 20 30 40 CR PI EST AST U/L Females 0 10 20 30 40 CR PI EST AST U/L Males 95 100 105 110 CR PI EST Chloride mmol/L 0.5 0.7 0.9 1.1 1.3 1.5 CR PI EST Creatinine mg/dL Females 0.5 0.7 0.9 1.1 1.3 1.5 CR PI EST Creatinine mg/dL Males 6 6.5 7 7.5 8 8.5 9 CR PI EST Total Protein g/dL Low High Low High Established 3.9 4.9 Established 8.6 10.0 Package Insert 3.97 4.94 Package Insert* 8.6 10.2 Current 3.5 5.0 Current 8.5 10.5 Low High Low High Established 32 117 Established 36 108 Package Insert 35 104 Package Insert 40 129 Current 40 150 Current 40 150 Low High Low High Established 7 38 Established 10 54 Package Insert 0 33 Package Insert 0 41 Current 0 45 Current 5 50 Low High Low High Established 7 21 Established 9 24 Package Insert* 6 23 Package Insert* 6 23 Current 8 25 Current 10 25 Low High Low High Established 22 30 Established 3.7 5.1 Package Insert 22 29 Package Insert** 3.4 5.1 Current 23 32 Current 3.5 5.0 Low High Low High Established 136 144 Established 136 144 Package Insert 136 145 Package Insert 136 145 Current 132 148 Current 135 146 BUN (mg/dL) Females BUN (mg/dL) Males CO2 (mmol/L) Potassium (mmol/L) Sodium (mmol/L) Females Sodium (mmol/L) Males Albumin (g/dL) Calcium (mg/dL) Alkaline Phosphatase (U/L) Females Alkaline Phosphatase (U/L) Males ALT (U/L) Females ALT (U/L) Males *Age partitions combined. **Serum/plasma ranges combined. Conclusions Data mining tools and real-time analytics utilization was used for robust establishment of reference intervals that would not be feasible to achieve with our historical methods. Improvements over our previous methodology included: 1) Establishing reference intervals for common analytes utilized large datasets to produce de novo ranges that are truly representative of our patient population. 2) After gaining familiarity with the analytic tools, the process proceeded quickly. 3) Readily accessible and exportable data via the business intelligence software allowed us to bypass our previous reliance on IT expertise and availability to conduct our data searches. 4) Although we utilized EP Evaluator® software in this project, the statistical analysis of the healthy population dataset could be performed in most spreadsheet programs with either default functions or via add-in packages. References 1. CLSI. Defining, Establishing, and Verifying Reference Intervals in the Clinical Laboratory; Approved Guideline—Third Edition. CLSI document EP28-A3c. Wayne, PA: Clinical Laboratory Standards Institute; 2008 2. EP Evaluator® 10.3.0.556. Establish Reference Interval Report Interpretation Guide. Data Innovations, LLC, South Burlington, VT; 1991-2013. 3. Aspartate Aminotransferase acc. to IFCC without pyridoxal phosphate activation (AST) [package insert V 4.0 English]. Roche Diagnostics, Indianapolis, IN; May 2014 4. Creatinine plus ver.2 (CREP2) [package insert V 8.0 English]. Roche Diagnostics, Indianapolis, IN; February 2016 5. ISE indirect Na, K, Cl for Gen.2 [package insert V 8.0 English]. Roche Diagnostics, Indianapolis, IN; January 2016 6. Total Protein Gen.2 (TP2) [package insert V 7.0 English]. Roche Diagnostics, Indianapolis, IN; February 2015 7. Albumin Gen.2 (ALB2) [package insert V 6.0 English]. Roche Diagnostics, Indianapolis, IN; March 2015 8. Calcium Gen.2 (CA2) [package insert V 3.0 English]. Roche Diagnostics, Indianapolis, IN; February 2014 9. Alkaline Phosphatase acc. To IFCC Gen 2. (ALP2) [package insert V 5.0 English]. Roche Diagnostics, Indianapolis, IN; February 2015 10. Alanine Aminotransferase acc. to IFCC without pyridoxal phosphate activation (ALT) [package insert V 5.0 English]. Roche Diagnostics, Indianapolis, IN; July 2014 11. Urea/BUN (UREAL) [package insert V 5.0 English]. Roche Diagnostics, Indianapolis, IN; January 2016 12. Bicarbonate Liquid (CO2-L) [package insert V 9.0 English]. Roche Diagnostics, Indianapolis, IN; September 2014 Established Package Insert Current Range Figure 2: Total Protein’s Statistical Analysis This figure displays the statistical analysis in the EP Evaluator’s Establish Reference Interval Module. This example demonstrates the information generated for Total Protein’s statistical analysis. CLSI guidelines (CLSI document EP28-A3c, formerly numbered C28-A3c) suggest using the nonparametric method for estimating the reference interval.1 The nonparametric method makes no assumption about the shape of the population distribution—it ranks the samples in ascending order.2 The number of samples in the overall population determines which results are used to the lower and upper thresholds—the software calculates 2.5% of the total number of samples (N) which equals x.2 The software then counts x samples from the lowest and highest (while the results are in ascending order) and assigns those values the lower and upper thresholds, respectively.2 For example, the Total Protein Central 95% Index counts from 16 to 623 because N= 638. The parametric method assumes that the population fits a Gaussian distribution or can be converted to fit a Gaussian distribution.2 These conversions may include taking a log or square root; however, the simplest parametric method calculates the references limits as the sample mean ± 2 standard deviations.2 The transformed parametric method considers the population distribution shape and evaluates if the data can be scaled to fit a Gaussian distribution. If so, the mean is computed ± 2 standard deviations and then converted back to the original units (EP Evaluator® 10.3.0.556).2 The confidence ratio determines the ratio average confidence interval width to the reference interval width using the formula: 0.5*(URLU-URLL+LRLU-LRLL)/(URL-LRL).2 The primary factor influencing this ratio is the sample size and the value of 0.10 or less is preferred.2 In this example, the confidence ratio is acceptable at 0.12 with a sample size of N=638. TP ClinicalPathology--RT-PLMI CMP Est Hepatic2 Central 95% Interval (N = 638) Lower Upper Value 90% CI Value 90% CI Ratio Confidence Nonparametric (CLSI C28-A) 6.3 6.2 to 6.5 8.0 8.0 to 8.1 0.12 Alternatives Parametric 6.4 6.3 to 6.4 8.1 8.0 to 8.1 0.06 TransformedParametric -- -- -- -- -- Confidence Limits for Nonparametric CLSI C-28A method computed by exact formula. Reference Interval Estimation: Combined SelectionCriteria Bounds None Filter None Statistics Mean 7.23 SD 0.44 Median 7.20 Range 5.4 to 8.6 N 638 of 638 Distinct Values 27 Zeroes 0 Central 95% Index 16.0 to 623.0 Analyst ABM ExptDate 12 Jan 2016 Histogram