Daniel M. Baer, M.D., Richard E. Belsey, M.D.
Daniel M. Baer, M.D., Richard E. Belsey, M.D.
Confident use of test information in the diagnosis or management of patient care problems depends on the laboratory's ability to consistently produce reliable (accurate and precise) test results. As with other endeavors, the ability to accomplish this depends on the use of good "tools," i.e., the quality of the instruments, test systems, and even the disposable supplies. In this chapter we deal with how to evaluate the reliability of laboratory instruments and test systems and the importance of using reliable supplies. For example, if you buy the best and most reliable automobile and equip it with the cheapest tires you can find, the reliability and safety of the motor vehicle is undermined by its weakest critical component&emdash;in this case, the tires. Similarly, it is frequently possible to find laboratory supplies at a discount. We are not adverse to saving money, but urge you to be sure that the savings are not illusory. The operating cost of using poor quality equipment, kits or disposable supplies can be evident in the need to retest quality control and patient samples because of reduced precision and accuracy. In the worst case, poor supplies could lead to an erroneous result being used in a patient care decision. Remember, if a deal seems too good to be true...it may be.
Having said this, you will find that wise purchasing of laboratory equipment, supplies and reagents can save money and aggravation for you and your staff. An organized and well thought-out approach is needed.
Daniel M. Baer, M.D., Richard E. Belsey, M.D.
Not all kits for a test are the same. Selection of the best kit for your practice should be based on analysis of a number of factors (Table 1). Your first step should be to make a list of available kits for the test. You can find them listed in a number of directories, such as the Clinical Laboratory Reference, published annually by Medical Laboratory Observer, or in catalogs of the large national laboratory supply companies.
Once you have listed several kits, contact the manufacturer for information. You should look at the kit configuration and components, and try the test before making a selection. We suggest that you follow the worksheet (Figure 1--Adobe Acrobat Reader required) as a guide in the evaluation of the kits. Ask your technical consultant to obtain and review information about the different kits' performance in proficiency survey programs.
Factors to Consider
Method ComplexitySome kits are easier to use than others. Tests with fewer steps and the absence of technically demanding steps such as accurate pipetting are better suited to the office laboratory. Methods that require critical timing steps are prone to error and disrupt work flow of the test operator. Some methods require more operator time and attention than others. Those methods that rate poorly for these factors should be avoided. Some test kits are classified by CLIA '88 in the "waived" category, and are not subject to many of its regulatory requirements. Check with the kit's manufacturer at the CDC web site http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfCLIA/search.cfm if you are in doubt about a kit or method.
Manufacturer Support
We have found major differences in the quality of user's manuals. Some manuals are complex for even an experienced laboratory technologist to follow. In some kits, a well illustrated manual, showing each step in cartoon form, may quickly guide the operator through the procedure. Additionally, a logically arranged kit arrangement, with ready-to-use reagent containers in a rack in the order of use and color coded bottles and caps to prevent the wrong cap being replaced on a bottle, may prove to be more user-friendly. Abbreviated instructions on a card to be used at the testing bench is another positive aid, because it can help avoid errors and speed the test performance. The user's manual should contain a problem-solving section and provide information on obtaining technical support through a toll-free phone number.
Local availability of supplies can be a great convenience in reducing the need for storage of the kits. Be sure that your supplier has a sufficient turnover of stock to avoid delivery of kits with an early expiration date.
Although quality control testing is required, not all manufacturers package quality control materials with their kits, and in some cases do not provide them at all. Quality control materials should provide reactions similar to those you would observe in clinical materials at the decision point of the test. Quality control materials that give strongly positive reactions fail to serve their intended purpose of providing an early warning in degradation of test performance.
Some manufacturers provide a proficiency testing program at no or low charge to users of their reagents. Participation in such a program will not satisfy Clinical Laboratory Improvement Act of 1988 (CLIA '88) rules, but does provide an opportunity for comparison of test results with other laboratories, and will improve the probability of good performance in the CLIA-approved proficiency testing program.
Reliability
The ultimate factor in evaluation of a test method or kit is the ability of the test to accurately provide clinically useful information. In test performance terms, this is expressed as sensitivity and specificity of the test. This information should be provided by the test manufacturer.
Performance in proficiency testing survey programs is one way of judging the reliability of specific test systems. Each of the proficiency testing programs publishes an analysis of the performance of proficiency testing program participants, categorized by test system. From this analysis it is possible to assess the variability of participant results (precision) and the closeness of the mean value of results to the target value (accuracy, lack of bias) of each system. The ability of proficiency testing programs to accurately assess bias, however, is imperfect because some artificial proficiency testing samples do not react in the same way as patient samples, and bias due to "matrix effect" may produce an artifact not seen when actual patient samples are analyzed. Small differences in participant performance should not be given much credence, but large differences in precision are significant in evaluating systems.
Vulnerability of reagents to spoilage is related to packaging and the need to protect the reagents from temperature elevations, moisture, and light. Reagents packaged in individual pouches are less vulnerable to the environment than when they are packaged in a single bulk container.
The quality of the user's manual has a direct relationship to the successful use of the system. Instructions should be clear, well illustrated, and unambiguous. Common operational problems and cautions should be discussed, and there should be a troubleshooting guide. The manual should include illustrations of reactions and end points that are read or interpreted visually.
Clarity of reactions and end points varies between some tests. Color end points are generally easier to read than agglutination. Moreover, agglutination reactions are more sensitive to operator variability than color reactions.
Economy and Cost
The actual cost per patient result of a kit depends on many things, and is often very different from what the vender calculated by dividing the selling price by the number of test units in the package. You need to carefully consider the kit packaging in relation to work load. What is the likelihood that the kit will be outdated before all of the tests have been performed? Not only must you consider the expiration date of the kit, but also the possibility that test strips, slides, paddles, cassettes or plates may deteriorate once the sealed package has been opened.
Storage space represents a cost, especially if the kit is bulky, uneconomically packaged with wasted space in the package, or if it requires refrigeration or freezing.
Look at the manufacturer's quality control and standardization (calibration) recommendations. CLIA requires you to follow these instructions. Having fewer quality control tests and calibration checks is more economical, both in reagent and labor costs. Some kits include quality control samples and standards; others do not. Be sure that you determine if these are an additional cost.
Proficiency testing costs can be considerable if it is necessary to subscribe to an additional program because of an additional test.
Exercise
Select a test that you are considering changing or adding in your laboratory. Obtain information about reagent kits for performing the test and a few sample kits for your evaluation. Using the worksheet in this section. Print one or more copies of worksheet 1 and evaluate each of the kits you have selected, including running several tests (in the privacy of your office in the absence of a salesperson). Obtain a copy of the summary of proficiency testing for this test through your technical consultant. These survey summaries are provided to participants in proficiency testing programs for the analyte.
As part of the evaluation, be sure that the office staff that will be performing the test are involved in the evaluation. They may have important observations about the kits under consideration. They are in a good position to evaluate the complexity of the method, clarity of instructions, and logistical problems of running the test while performing other duties in the office.
Commentary
The number of tests your laboratory will perform per week has a direct influence on which kits to consider. If only a very few tests are performed, perhaps it would be better not to do the test in your office but to send it to a referral laboratory. As a minimum, your laboratory should perform a test once a week to remain proficient.
The laboratory staff's level of technical training and experience must be considered when selecting a kit. Review the kit's instructions. Operators without technical training need instructions that are written in nontechnical language, and preferably include step-by-step illustrations. Reagents should be clearly labeled, with color-coded bottles and closures to prevent reagent use in the wrong order, or mixing of bottles and closures If the test requires accurate pipetting or reagent preparation, it is unsuitable for operators who lack technical training.
Laboratory staff who have other duties in the office may not be able to reliably perform tests that require accurate timing or have multiple steps, because their other duties may interrupt their laboratory responsibilities, potentially resulting in errors. Methods with a single step that can be read after a variable time delay are preferable in this situation.
Some microbiology culture kits provide an incubator, others do not. Be suspicious of a bacteria culture system that does not recommend incubation of the inoculated media at 35°C to 37°C. Remember to include the cost of the incubator in the system's total cost.
The major cost for performing most tests is for staff time. Do not forget to include time for reagent preparation, calibration, quality control testing, and system maintenance.
Quality control testing is required for most tests each day that testing is performed. Generally, either a positive and negative control or controls at two levels are required. Some kits do not provide controls as part of each kit's packaging. Some manufacturers do not provide them at all. Some kits that use immunologic methods on slides or cards include positive and negative controls as an integral part of the slide or card, obviating the need for separate controls.
Compare your office's work load for the test and whether the kit size will result in outdating or reagent wastage. If large numbers of tests are to be performed, you may want reagents in multitest quantities rather than in unit packages for each specimen.
Storage can be a problem. Some kits require refrigeration. Others are packaged in bulky boxes that do not store easily. If a local dealer stocks the kit, you may be able to arrange for more frequent deliveries to avoid keeping more kits than needed on hand, but be careful that the dealer's stock is reasonably fresh and will not outdate prematurely.
Using proficiency testing data, look up the kit and compare its performance with other kits for the same test. Does the kit perform the test reliably without bias or excessive variability? These will be evident by examination of the mean values and coefficients of variation for the kit.
Kit instructions are very important. They must be written in language appropriate for the operator, and be complete and unambiguous. A troubleshooting guide to common testing problems should be included in the instruction manual. Technical support in the form of a toll-free phone number should be available. Companies that provide local training and technical support should get a high score.
The kit's total cost is more than the cost of the kit divided by the number of unit; in the kit. The worksheet calculation will give you the supply cost, but does not include personnel costs, cost of equipment, or the cost of subscribing to a proficiency survey, if an additional one is required because your laboratory is performing the test.
Daniel M. Baer, M.D., Richard E. Belsey, M.D.
Method complexity
Economy
Manufacturer support
Reliability
Cost
Daniel M. Baer, M.D., Richard E. Belsey, M.D.
Depending on the nature of the practice, batch testing, i.e. running all specimens for a single test, may be appropriate in some cases, while in others sequential testing of a number of tests for a single individual is more appropriate. The duties of the available staff must be taken into consideration. If there are staff members available who can devote their full attention to testing, it may be possible to utilize systems that require the undivided attention of the operator for a period of time. On the other hand, if laboratory testing is an ancillary duty that has low priority, it is essential for the system to be tolerant of delays and interruptions, or be a totally automated "walk away system.
The technical ability of the staff will also define the kind of system that is most appropriate. Technically trained staff can perform pipetting, critical timing operations, and preparation of reagents, whereas technically untrained staff cannot.
The expected work load in the setting will define the acceptable reagent packaging and stability requirements. Frequency of quality control and standardization are important factors when the work load is low. On the other hand, if the work load is heavy, attention must be paid to instrument throughput.
Cost is always a factor. Costly equipment cannot be justified in low-volume operations. Attention must be paid to the operational costs related to reagents, standards, and controls. The volume of testing will, to some extent, define the packaging and stability needs and indicate whether wastage is likely to be an important consideration.
Finally, the accuracy and precision characteristics of the system, including the reagent systems and the operator, must be considered. System stability will have an influence on the quality control requirements and the frequency of standardization. A system that provides automated result calculation and printed reports is inherently more accurate than one that does not.
Exercise
Select a test or group of tests for which you are considering the purchase of an analyzer. Select one instrument capable of performing the test or group of tests. Print one or more copies of the instrument selection worksheet, (Adobe Acrobat Reader required) and evaluate the analyzer. You may wish to evaluate two or more analyzers the same way, comparing the evaluations.
Commentary
Meeting the Needs of the PracticeAlthough you have chosen only one test or one group of tests for this exercise, you should think about other analyses that you need to test or may wish to test in the future. Does this analyzer have the capability of doing these tests or will it be necessary to purchase a different analyzer to do these tests?
Analyzers that can perform several different tests sequentially rather than only one kind of test (batch mode) before resetting (and sometimes recalibrating) are more appropriate for the office laboratory, where patients typically present one at a time. The physician usually wants results for several tests as rapidly as possible, so that they can be evaluated before the patient leaves the office.
System Characteristics
In many office laboratories, especially the smaller ones, a fully trained laboratory technologist is not usually available to do the testing. Instead, office staff without technical training may perform laboratory tests in addition to their other duties. These other duties frequently compete with the laboratory duties, and may have a higher priority when there are multiple demands on the staff's time. In this case, it is important to examine those characteristics of the analyzer that relate to the technical skill of the operator and the level of attention that the operator needs to give the analyzer during its operation.
In a previous unit you made a list of tests that are needed in your office. On the worksheet (hyperlink) you have listed the tests available in the test system. Do they match, or are there important tests that the system cannot do?
There are great differences in how different analyzers handle sequential tests from the same patient. In some analyzers, it is necessary to feed each test card or strip into the instrument, wait for a result, then feed the next test strip. This kind of instrument demands the almost constant attention of the operator to operate at a reasonable efficiency. Other instruments can be loaded with several tests, with the operator returning for a printed report of the results when they have all been completed. This type of operation is much more efficient, especially when the operator has other duties in the office laboratory. Test systems that require minimal attention of the operator during the testing process fit better into the small office laboratory, where the staff has many duties to perform and not much uninterrupted time to devote to testing.
Systems that require accurate timing and pipetting can introduce errors into the testing process. Systems using whole blood samples reduce the number of steps in testing, reducing potential errors as a result. Preparation of a serum or plasma sample requires centrifugation, which is time-consuming, requires additional equipment, and exposes the staff to additional handling of blood.
Test systems that produce a printed report have a number of advantages. Printed reports permit the operator to walk away from the analyzer, coming back after several reports have been printed. Printed results can also be used as permanent reports, reducing the opportunity for clerical errors in transcription.
Analyzers differ greatly in their throughput. Some are relatively slow, performing one chemical test every 3 or 4 minutes. Others can analyze a series of tests every 30 seconds; usually there is a delay in receiving the first result, but the following tests are reported in rapid sequence. In another instrument, the running time may be 10 minutes, but several tests can be analyzed at the same time. In all cases, the throughput depends on the way the tests are presented to the instrument.
Selection Process
Staff involvement in the selection process can often result in a better choice. The staff should review the users' manual for clarity of operating instructions and try the instrument. They should comment on the logistics of operation in relation to their other duties and expected work flow. The staff may have a strong preference for a specific test system.
The sales representative may also be helpful in selecting the instrument that best fits the needs of the practice, but remember that selling an instrument is the sales representative's primary interest. Ask the salesperson to leave the instrument in your laboratory for a few days so the staff can try it at their leisure. Some preliminary instruction should be given by the sales representative, as well as a supply of reagents&emdash;at no charge. Trying the instrument does not obligate you to purchase it. Ask about "deals," such as reagent rental programs, in which the instrument is provided at little or no cost and reagent purchase pays for the instrument's use. Such arrangements are common in the laboratory industry.
In all probability, any decision you make about purchasing a laboratory instrument will be one that you have to live with for the next 5 years or longer. The decision should be made carefully and should anticipate your needs for the next few years.
Daniel M. Baer, M.D., Richard E. Belsey, M.D.
General
Practice Requirements
Staff Ability
Work Load
System Characteristics
Quality Management
Cost
Daniel M. Baer, M.D., Richard E. Belsey, M.D.
Several options for selecting a supplier exist. Each has its pros and cons. You can sometimes buy supplies directly from the manufacturer. Advantages are the probability of receiving supplies with the longest dating because of rapid turnover of the product. The disadvantage is that you will deal with an anonymous salesperson via telephone.
Most suppliers are local or regional supply houses. Some are regional branch offices of large national laboratory supply companies such as Allegiance Scientific Products (http://www.cardinal.com/mps/), Fisher Scientific (http://www.fisherscientific.com) or VWR Scientific Products (http://www.vwrsp.com). Some of these have sales representatives with whom you can establish an ongoing relationship. These companies maintain a large inventory of laboratory products for both large and small laboratories. Some even have specialists who understand the needs of physicians' office laboratories. Because of the size of inventories and the rapid turnover of stock, a wide scope of products and fresh reagents with a long remaining shelf life are the rule. Another service provided by these large companies is the Electronic Data Interchange (EDI). Those users with computers and modems can enter orders electronically directly into the company's computer or website, receiving immediate information about the availability of the item and delivery within a day or two.
General medical supply houses also sell some laboratory supplies and reagents. These companies have sales staff that call on physicians' offices; a good relationship between the sales and office staffs often develops. Disadvantages are their general lack of familiarity with laboratory products, and a limited inventory and scope of products.
Another possibility is the purchase of selected supplies and reagents from a local reference laboratory or hospital, some of which have established buying groups or laboratory supply subsidiaries. Advantages are the ability to purchase small quantities, less than usual packaging, such as one or two culture plates; the probability of fresh supplies; rapid delivery, frequently by the laboratory's courier who makes daily visits to the physicians' laboratory; the probability of dealing with a familiar contact at the reference laboratory; and favorable pricing because of the high volume buying power of the large laboratory. The major disadvantages are the limited scope of products and the infrequency of this kind of service being offered by reference laboratories and hospitals.
Evaluating the Reliability of the Supplier
Not all suppliers give good, consistent, and reliable service. Some questions that you should ask when evaluating a supplier are:
Ordering Supplies
Once you have selected one or more suppliers, you need to determine whether to order supplies as needed or establish a standing order. Reagents are usually manufactured in large lots. There is usually some variability in the product from lot to lot, resulting in slightly different test results. This variability is most pronounced in quality control materials and coagulation reagents. Buying a single lot of these reagents that will last for about 1 year improves the stability of results and saves time and money by eliminating the need to overlap the use of old and new lots of certain reagents or frequent recalibrations. Certain reagents such as quality control materials and coagulation reagents, should be purchased as a single lot number to last for 1 year, if at all possible. It is usually necessary to make a commitment with the supplier to purchase a predicted amount. In some cases, when your use is very predictable and constant, the supplier can automatically ship the reagents to you on an arranged schedule. In other cases, when you cannot predict your needs, you can set up a "requirements agreement" in which you predict your total annual use, but notify the supplier when you need a shipment of the supplies.
Establishing a Relationship
More satisfying, if not better, service can be obtained if you have a relationship with one salesperson. In this way, the physicians' office staff and the supplier can get to know each other. If possible, learn the name of the person with whom you place orders, and have the same person in your office interact with the supplier each time. Personal visits to each other's facilities are desirable, but often not possible.
Inventory Control
Keeping an adequate supply of all the supplies and reagents necessary to operate your laboratory takes constant attention and an organized approach. Although many suppliers can respond to emergency orders, they are costly to you and to the supplier, and are disruptive to the laboratory's functioning.
Assign responsibility for maintaining adequate laboratory supplies. One person who performs laboratory testing should be designated to assure that adequate supplies are on hand, and that they are not used beyond their dating period.
We have found that a system using supply control cards (hyperlink) works well to keep track of inventory and to organize the ordering process. Each card contains the following information about the reagent or supply
The supply control card can be kept with the supplies or in a card file. If it is not placed with the supplies, there should be a card or label with the supplies indicating at what point an order should be placed for replenishing the stock.
Placing an Order
There are a number of things that your staff can do that will help assure good service from your supplier. When you place an order, be sure that you provide the correct name and catalog number of the item. To assure proper delivery and billing, the supplier will need to know your account number and name. Allow adequate time to fill and deliver the order&emdash; usually 7 to 10 days is the minimum time needed.
If you are able to consolidate your orders, buying several items at the same time, you can reduce the shipping charges. Consider having one day a week for ordering supplies rather than ordering more frequently.
Exercise
Set up an inventory control system for one reagent that is frequently used. Using a supply control card or other similar system, determine the point at which a new supply should be ordered and in what quantity.
Commentary
The stock level at which a replenishment order should be placed is based on experience, including your usual use of the reagent, the lag time between placing an order and its delivery, and the outdating of the reagent. In general, it is a good idea to keep a minimum of 1 month's supplies or reagents on hand, and to place an order when the stock reaches that level. On the other hand, you should not order so much that the stock is likely to be outdated before it is used up. Experience will tell you what the balance between running out of and wasting the reagent will be. By keeping records of actual use, ordering, and receiving of the supplies, you can improve your chances of striking the right balance.
Daniel M. Baer, M.D., Richard E. Belsey, M.D.
Baer DM, Belsey RE. Assessing an office chemistry instrument. Prim Care. 1986;13:699-711
Inventory Control Systems for Laboratory Supplies; Approved Guideline GP-6A. Wayne, Pa: National Committee for Clinical Laboratory Standards; 1994.