Patient-Based Real Time QC

Abstract

Patient-Based Real-Time Quality Control (PBRTQC) uses the statistical characteristics of a particular patient population(s) served by a laboratory using specific analytical platforms. There is a review of this approach in the August 2019 issue of Clinical Chemistry. The power of these techniques is that they offer exquisite customization to provide very sensitive detection of a change in bias. They are not subject to the impact of noncommutability issues and, once set up, are low cost to maintain. However, they do require knowledge of the characteristics of the laboratory patient population(s) and the analytical methods used. PBRTQC needs to be tailored for each measurand in a population, though this is certainly not a major limitation. Optimization requires access to simulation software and patient data from the Laboratory Information System. PBRTQC provides an effective QC system, but it can also be used to provide external quality assessment (EQA) and has a role in postmarket surveillance of in vitro diagnostics. The population medians and the flagging rates (i.e., the number of patients who fall outside the reference intervals) should be stable. Monitoring these parameters over time allows bias to be identified. But even more, these rates can be compared across different laboratories with method-specific data such as calibrator and reagent lot numbers. These data allow the identification of bias introduced by a change in calibrator or lot number across many laboratories. This is the basis of the Noklus Flagger and Percentiler programs available at the Noklus website. The concept, like PBRTQC, is to use the power of the population data to identify bias, globally. The power of the PBRTQC concept to reflect patient population parameters enables changes in bias caused by lot or calibrator variation to be identified across many different laboratories simultaneously. PBRTQC will become the mainstay of QC in laboratories once the profession sees the advantages of this form of process control, and manufacturers and middleware vendors provide the onboard capability. But this will be just the start. The underlying philosophy with PBRTQC is to detect bias in an assay using the specific population characteristics that the laboratory serves, a patient-centric approach. One of the problems with any form of QC now in use is the sheer volume of samples and the requirement for rapid decisions to be made on releasing results. Hand in hand with the implementation of PBRTQC is a need to change the mindset from human decision making to AI approaches to Statistical Process Control. There is a need for large analytical systems to not only use the Hospital Information System to identify patient subgroups, but also for the Laboratory Information System to identify a significant drift, interrogate manufacturers databases regarding calibrator and reagent lot quality, and to initiate recalibration. The days of a human operator controlling the hour-by-hour operation of a large analytical system are disappearing. PBRTQC is a major step to integrating the laboratory into the hospital information system, and to a bigger dataset with the ultimate goal of better patient outcomes. In this Q&A we answer common questions asked about PBRTQC. The panel consists of members of the International Federation of Clinical Chemistry and Laboratory Medicine Working Group on PBRTQC. Huub van Rossum: I work in a hospital-based lab with rather small daily production. In this context, we considered PBRTQC for analytical quality assurance when iQC by measurement of internal control samples alone was deemed to be insufficient. The advantages of PRBTQC are that it enables continuous quality control (QC) if set up to calculate a new PBRTQC value for every newly generated test result, and it has the best error-detection performance for tests with low biological variation. These characteristics perfectly complement iQC, which is by design scheduled and is insufficiently effective in detecting relevant errors for tests with a low ratio of biological variation to analytical variation. We, therefore, have applied PBRTQC for tests whereby iQC is limited, in the sense that we experience a rapid onset of critical errors or temporary assay failures, or for tests with sigma metric values less than or equal to 4. By introducing PBRTQC procedures into the QC plan and by validating the PBRTQC error detection, we also have been able to reduce the number of quality-control sample measurements. The key to achieving this was obtaining objective insights into the error-detection performance of PBRTQC. We did this by generating so-called moving average (MA) validation charts, which display the error-detection performance and the uncertainty thereof for an array of systematic errors. Other laboratories have taken the integration of iQC and PBRTQC a step further by using PBRTQC for a much more extensive test selection to schedule and apply iQC more efficiently. Such an approach is especially interesting for larger laboratories with high production numbers and a relatively low numbers of abnormal test results. For any kind of approach, the key is to obtain objective insights on the error-detection performance of PBRTQC. Andreas Bietenbeck: A straightforward integration is to always verify out-of-control PBRTQC states with iQC and vice-versa. Other methods, such as using patient pools, can also be used in this manner. So-called aggregated Z-values can integrate iQC and (not-moving) medians of patient results. How to integrate iQC and more complex moving PBRTQC mathematically is still an open research question. Alex Katayev: We are a very large clinical reference laboratory with multiple locations and run a substantial test volume for most routine chemistry and immunoassays, with about 80% of our testing volume coming from primary care facilities. Therefore, we have been able to use PBRTQC for most chemistry and immunoassay tests. However, we are still using two layers of QC:iQC and PBRTQC that are nicely complementing each other. We run iQC at the beginning of every shift to make sure that instrumentation is calibrated correctly and maintained and at the end of every shift to close the run that may not be complete (as we are using the “report from the back” approach). In terms of meeting US Regulatory requirements, the most current CLSI guidelines that describe risk-based QC recommendations include a PBRTQC section as a part of their recommendations. However, because we are using a combination of both iQC and PBRTQC, we are meeting current guidelines just for iQC by itself. Tze Ping Loh: Most laboratories will be familiar with iQC and will be looking at PBRTQC as a new tool to improve existing QC practices. To this goal, a risk-based assessment is often helpful in deciding which assay or current QC practice can benefit from the addition of PBRTQC. This can start by examining whether existing iQC provides sufficient error detection capability for the magnitude of error considered clinically important for a particular measurand concentration. This may include the detection of increased analytical imprecision, which generally evades routine iQC. Another error that may avoid iQC detection is bias at low levels that may have a significant clinical impact if it alters the clinical interpretation. An example is a low positive bias on a troponin assay, which increases the risk of an erroneous diagnosis of myocardial injury. For such cases, the use of appropriate PBRTQC can substantially improve error detection capability and minimize patient harm. PBRTQC should also be considered when there are very high throughput assays with a consequent high risk of releasing a large number of erroneous results between iQC samples. In these situations, it is not operationally or economically feasible to perform more iQC. Finally, PBRTQC is helpful for assays that lack commutable iQC material. On the other hand, PBRTQC may be susceptible to changes in the population served by the laboratory. Internal Quality Control can serve as an adjunct source of information in determining whether a breach in PBRTQC is due to patient factors or analytical factors. Tze Ping Loh: While the major advantage of PBRTQC is providing immediate feedback, non-real-time applications can be valuable in assuring the quality of laboratory results. When a significant analytical error is suspected, one can extract a period of laboratory data and calculate simple moving statistics (e.g., MA or moving median) to observe the trend of the results. This approach often can provide helpful insights into the analytical performance of the assay even without sophisticated analysis. It can also provide objective evidence to confirm a hunch from clinical colleagues regarding analytical performance (e.g., ‘I have recently seen a lot of hypercalcemia in the clinic; is there anything wrong with the assay in the laboratory?’) Nevertheless, the output should be interpreted carefully and in conjunction with other forms of quality control (iQC and proficiency testing) data. For such purposes, the use of simple spreadsheet applications (e.g., Microsoft Excel or Google Sheet) is often sufficient. Using this approach will provide a low cost threshold option for laboratories to familiarize themselves with the concepts and behavior of patient-based QC, before investing in more dedicated software. More advanced users can also create macros within the spreadsheet software to automate more sophisticated statistical analysis. Of note, Analyse-It is an add-in software product for Microsoft Excel that is sometimes used by laboratory professionals for compiling test evaluation results and also has built-in moving statistics applications. Alex Katayev: When we started looking at middleware that would allow us to implement PBRTQC the way we wanted, we investigated several different vendors. We choose Data Innovations because, at this time, it was the most flexible platform and allowed us to use customized protocols (including different moving mean/median protocols with and without transformations and “reporting from the back”). This middleware vendor is now using our customizations in their standard commercially available package. However, there may be more offerings in the market as we move forward. Mark Cervinski: At Dartmouth-Hitchcock Medical Center, we use the middleware provided by our major instrument vendor. The ability to use PBRTQC, however, was an additional financial investment that we made to take advantage of the utility of PBRTQC to detect instrument error far in advance of routinely scheduled liquid QC assessment. Many major instrument vendors have made middleware/software options available. I would suggest talking to your vendor to see what options are available. Alex Katayev: Before implementation, we performed an extensive simulation study using internally developed customized Excel-based software and created our PBRTQC rules from “scratch” as we were one of the first commercial laboratories who decided to implement PBRTQC. Those rules were analyte-specific and accounted for distribution of results in the tested population and used error thresholds derived from accepted quality goals (often based on the biological variation of the analyte). Before implementation, we performed rigorous performance verification studies and documented everything. More on this process can be learned from our published manuscript. Nowadays, it may be feasible to rely on other commercially available simulation programs and use pre-set rule packages. Huub van Rossum: Obtaining PBRTQC settings is probably the most challenging aspect of PBRTQC and laboratories often struggle to get their PBRTQC settings right. In general, a PBRTQC procedure consists of making settings to determine (i) which results are included in the PBRTQC algorithm, (ii) the algorithm applied (mean, median, etc. and the variable within the algorithm, e.g., mean of the last 5, 10, 15, or more results), and (iii) the control limits which, when exceeded, trigger an out-of-control alarm. In contrast to iQC, optimal PBRTQC is very laboratory-specific, as it strongly relies on test results obtained over time. Therefore, it is considerably affected by the number of tests, the patient population, and local logistics, including, for example, what times specific samples arrive at the lab or analyzer. An essential requirement for any PBRTQC procedure is that only a manageable number of PBRTQC alarms should be allowed to avoid alarm fatigue. Recently, a few new methods have been described, all of which use realistic PBRTQC error-detection simulations on laboratory-specific datasets to obtain insights into the PBRTQC error-detection performance of that particular laboratory. These procedures allow PBRTQC settings with the best error-detection characteristics to be selected. These methods have allowed several labs to obtain suitable PBRTQC parameters and thus successfully implement PBRTQC. An important development has been that one of these methods uses what is known as ‘bias-detection curves’ for PBRTQC optimization. MA validation charts for PBRTQC validation have become available to laboratories via the online MA Generator application (Huvaros), enabling them to obtain their own optimized and validated PBRTQC procedures. I have used this application to get my laboratory-specific PBRTQC settings. Andreas Bietenbeck: Optimal PBRTQC settings seem to depend very much on the individual laboratory and even small differences can have a substantial effect. For example, my hospital uses a pneumatic tube system for sample transport. Therefore, all samples from our intensive care unit arrive at the same time in our lab and are measured consecutively. This cluster of usual extreme measurement results leads to a signal in PBRTQC algorithms similar to an out-of-control situation and settings have to be adjusted accordingly. For a very similar lab receiving the samples in a different order, other settings might be preferred. The simple copying of settings from published research will likely lead to unsatisfactory PBRTQC performance. Instead I suggest a simulation based on at least a year of historical patient results. Biases can be added to these results to evaluate different PBRTQC settings for the individual laboratory. A simple simulation program performing these tasks accompanies a publication of our group in this journal. Mark Cervinski: This is something that we initially struggled with, as there was no step by step instruction that one could follow. To develop our protocols and get settings, we started by obtaining a large set of historical data from our instruments. This is an essential step as the PBRTQC protocols must take into account the patient population being served as well as the analytical capabilities of the equipment used to generate the data. To develop protocols that would be of value, we modeled the analytic process in MatLab. We used a simulated annealing algorithm available in MatLab to determine the number of patients results to average for each analyte, as well as where to place truncation limits to remove values at an extreme distance from the mean. I would advise doing something similar in any program that is capable of performing simulations. Data analysis can be a daunting task, but you don’t have to attempt this alone. If this is an area for which you do not have the experience, I would suggest reaching out to someone with data analysis skills to help model the process to establish parameters. Tze Ping Loh: Beyond setting up the parameters for PBRTQC, it is equally important to verify its performance using the historical data from the laboratory before routine implementation. This exercise provides objective data on the likely real-world performance of the algorithm. Our workgroup has recently described some recommendations when verifying the performance of PBRTQC. Andreas Bietenbeck: In general, PBRTQC works best with analytes that have a relatively small measuring range. In contrast to other quality control methods, many extra-analytical factors affect PBRTQC. Therefore the patient population should remain stable. PBRTQC also does not work for individual point-of-care tests. Measurements from a glucometer that is used by only one patient reflect only the changes this particular patient and cannot be used for PBRTQC. Alex Katayev: In our experience, these are the major limiting factors in the production use of PBRTQC: i) the test volumes that we run through each analyzer, and ii) the analyte’s “noise” in the tested population versus the chosen error threshold (in other words, “signal to noise” ratio). The lower the test volume is and the lower the signal to noise ratio is, the more difficult it is to employ PBRTQC in practice. Mark Cervinksi: In our experience, PBRTQC methods, more specifically, MAs, do not work very well for analytes such as the sex hormones, tests that have a low/lower daily volume, and those with highly skewed distributions. For skewed distributions, I know that some people have had success in transforming the data before applying a MA, but we’ve not explored that in much detail. Andreas Bietenbeck: PBRTQC a of the testing process than iQC because even changes in the can affect PBRTQC. For example, if increased because of wrong PBRTQC can an alarm. Tze Ping Loh: The place of PBRTQC in the quality system of a laboratory is often seen as a complement or adjunct would that it should be considered the primary QC PBRTQC is in the QC by being the only tool that the performance of the analytical system to clinical It allows of the population shift that is often when using iQC. The from clinical impact can sometimes lead to a of QC While iQC to be an essential tool in quality its are in laboratory practice. The of iQC substantial patient for tests that are high have small for and are by such as is example of a assay, which has been In these PBRTQC should be considered as the PBRTQC is also being explored as a external quality assurance tool that can help laboratories detect systematic errors to their PBRTQC often in the quality system of a and sometimes QC that no are Alex Katayev: At we have to do a lot of and about the use of PBRTQC as it is an different approach and requires a change in We created very standard procedures that described the step by However, a period of about a started to highly this QC because they could see its advantages in and quality Andreas Bietenbeck: PBRTQC is not as as iQC, and there is considerably less experience in this kind of quality can be challenging because there are many different of error to I would advise the laboratory to PBRTQC and to start with only a few when the with these should the use of PBRTQC be Mark Cervinski: This is something that we are to work on this and the for the alarms are to the and someone on that the on This is an and we are to provide for the on all with them of and error detection and to to The goal is to generate standard work that would the to perform a set of In the of the are still by the as they have seen the value of being able to detect an analytic shift before the scheduled QC Tze Ping Loh: PBRTQC is more complex than iQC from a statistical of However, it still in a highly similar as iQC with once it has been set The may from uncertainty with to the error and the with moving statistics using simple there is a significant laboratory error can help the concept to the of to iQC can also reduce the and as many statistical as within the software from the routine will help the Huub van Rossum: factors are essential for PBRTQC implementation and application by the laboratory The first is, of suitable PBRTQC settings, which should and avoid alarm at least in a manageable number of the of the software used to PBRTQC should also critical and to use PBRTQC This the appropriate of PBRTQC that are on the PBRTQC used. In my lab we use to the and out-of-control PBRTQC values an approach that is not difficult to and the software should and a PBRTQC alarm that is simple for to We have a standard on which out-of-control PBRTQC results are and all PBRTQC alarms require immediate The is a for up a PBRTQC alarm. This whether an error by PBRTQC is We use a The first step is to run iQC to obtain a of the of the recently samples are on an system that temporary errors and errors not by iQC can be The step is to review recently samples and patients to other of the PBRTQC such as errors (e.g., using the wrong or patients with extreme test results. are well to perform these they have to the of this and have the significant to the and of or analysis and of or the for of the published and to be for all of the thus that questions to the or of any part of the are investigated and all the of Laboratory of van van van van van van

Topics & Concepts