Vol. 00 No.

0 xxx 2012

A Medical Staff Peer Review System in a Public Teaching HospitalAn Internal Quality Improvement Tool
Linda S. Chan, Manal Elabiad, Ling Zheng, Brittany Wagman, Garren Low, Roger Chang, Nicholas Testa, Stephanie L. Hall Peer review of the quality of care of the medical staff in a healthcare delivery system, properly executed and utilized, can bring about changes that can improve the quality and safety of patient care, enhance overall clinical performance, and augment physician education (Antonacci, Lam, Lavarias, Homel, & Eavey, 2009; Keroack et al., 2007; Olcott, Mitchell, Steinberg, & Zarins, 2000). In the past two decades, there have been external forces to monitor the performance of the medical staff. The most obvious one is the Joint Commissions requirement that a healthcare delivery system must have in place a peer review system to monitor the quality of care of the medical staff on an ongoing basis. Although all healthcare facilities are mandated to conduct peer reviews, the process of how they are conducted, reported, analyzed, and utilized varies widely allowing for signicant inconsistencies. Although positive results have been found in some peer review systems (Jarvi, Sultan, Lee, Lussing, & Bhat, 2002; Nolan, Burkard, Clark, Davidson, & Agan, 2010; Owens et al., 2010; Williams, Mechler, & Akins, 2008), many faced multiple challenges impeding the progress in the eld (Livingston & Harwell, 2001; Marren, Feazell, & Padock, 2003; Spaeth, Pickering, & Webb, 2011). Edwards (2009a, 2009b) developed and published a Peer Review Program Self-Evaluation Tool, which can be used to rate current peer review programs. Using this validated tool, it was found that its score is strongly associated with perceived quality impact (Edwards, 2010). Data from 296 acute care hospitals show that the majority of programs rely on an outmoded and dysfunctional process model and that adoption of best practices informed by the continuing study of peer review program effectiveness has the potential to signicantly improve patient outcomes (Edwards, 2011). In 2007, our institution has developed and implemented an electronic Medical Staff Peer Review System (MS-PRS), which has replaced the paper-based system in order to expedite Abstract: Peer review of the quality of care of the medical staff in a healthcare delivery system, properly executed and utilized, can bring about changes that improve the quality and safety of patient care, enhance clinical performance, and augment physician education. Although all healthcare facilities are mandated to conduct peer reviews, the process of how it is conducted, reported, and utilized varies widely. In 2007, our institution, a large public teaching acute care facility, developed and implemented an electronic Medical Staff Peer Review System (MS-PRS) that replaced the existing paper-based system and created a centralized database for all peer review activities. Despite limited resources and mounting known challenges, we have developed and implemented a system that includes 100% mortality reviews, an ongoing random review for reappointment and operative procedures, and morbidity peer reviews. Parallel to the 4-year implementation of the system, we observed a steady, signicant downward trend in the medical malpractice claim rate, which can be attributable in part to the implementation of MS-PRS. In this paper, we share our experiences in the development, outcomes, challenges encountered, and lessons learned from MS-PRS and provide our recommendations to similar institutions for the development of such a system.

the review process and create a database of all peer review activities including outcomes and improvement actions. In this paper, we present the development, implementation, and outcomes of MS-PRS as well as the challenges encountered and lessons learned.

Keywords medical staff peer review peer review system public teaching hospital

Materials and Methods

The Healthcare Delivery System
Founded in 1878, our institution is the largest public teaching hospital in the west coast serving the indigent population. Originally established as a 100-bed hospital with 47 patients, it now is licensed for 724 beds. In November 2008, our Medical Center transferred to its new state-of-the-art facilities. We train approximately 1,500 medical professionals daily, including more than 870 medical residents in nearly all clinical specialties.

Journal for Healthcare Quality Vol. 00, No. 0, pp. 18 C 2012 National Association for Healthcare Quality

Journal for Healthcare Quality

Journal for Healthcare Quality

The Medical Staff Peer Review System (MS-PRS)

Overview The development of the MS-PRS began in July 2006 with the recruitment of a computer engineer whose sole responsibility was the research and development of an electronic peer review system. After development of the peer review guideline and training of all involved parties, the mortality module was fully implemented in January 2007. As of January 1, 2011, the modules for mortality, morbidity, and random peer review are operational including a front-end web-based entry of peer reviews. This paper reports our 4-year experience and ndings from mortality reviews. Processing the reviews The procedure of processing mortality reviews included the following steps:

same specialty serves as the peer reviewer of the case, except his or her own case. The MSPRS blocks all cases that belong to an attending who signs on to review. A standardized peer review form is lled out by the peer reviewer. If the case requires further review by another department, the peer reviewer designates as such in the review form. When all departmental reviews are completed for the case and if the case is deemed to have no opportunity for improvement by all departmental reviews, then the case is closed. No further action is needed. If the case is deemed to have opportunity for improvement in one or more departmental reviews, it will be forwarded to the intermediate-level review. Intermediate-level review The intermediate-level review is performed by the Small Peer Review Committee (SPRC), which consists of a group of voluntary and senior clinicians and surgeons. This intermediate level of review was found necessary to relieve the backlog of cases to be reviewed by the Executive Peer Review Committee (EPRC). The SPRC was formed in July 2010. It meets twice a month. There is a medical leader and a surgical leader. A minimum of three clinicians must be present to have a quorum. All cases deemed to have opportunity for improvement at the departmental level are brought to this committee for review. The SPRC identies opportunities for improvement, evaluates the appropriateness of the level of care provided, and determines whether or not the improvement actions are acceptable. Once a case is completed, SPRC forwards the case to the last level of review and for further follow-up if necessary by EPRC. Final-level review The EPRC serves as the nal decision body for the medical staff peer review process. It meets every 2 months. The committee is composed of the Chief of Staff who serves as the Chair of the EPRC, the Chief Medical Ofcer (CMO), the QI Director, and at least ve physicians from different specialties or departments. The Committees primary responsibilities include: (a) evaluating the appropriateness of follow-up actions taken by clinical and hospital departments and SPRC, (b) identifying opportunities of improvement and forwarding recommendations to responsible parties or departments for system-wide actions, and (c) resolving discrepancies regarding the outcome of peer reviews.

r MS-PRS automatically downloads all der

aths on a weekly basis from the hospital system. Quality Improvement Managers (QIMs) and supportive staff ensure medical records are available either in the paper chart or in the hospitals computer systems. The cases ready for peer review are uploaded to MS-PRS front-end Intranet web page. Peer reviewers perform and record peer reviews online. Cases that require referrals to or consults from other departments are forwarded for further review by the QIMs or support staff. When a case has no opportunity for improvement by all departments, the case is closed. When a case is deemed to have an opportunity for improvement by any one of the departmental reviewers, the case advances to the next levels of review when nal disposition of the case is made.

r r r r r

Peer review levels and bodies Each mortality review progresses through three levels of peer review: departmental, intermediate, and nal levels. Departmental level review The departmental level review begins with the review by an attending of the discharge department. The attendant who is privileged in the

Vol. 00 No. 0 xxx 2012

The EPRC reviews all cases that have opportunities for improvement and makes its nal decisions on preventability and necessary improvement actions. Improvement actions may include but are not limited to conducting rootcause analysis, revising existing, or adding new policies. Letters to the providers are sent out notifying them of the Committees decision.

r Letters to the department Chairs and

providers are generated electronically after decisions from EPRC are made. r Periodic reports summarizing the outcome and process measures are generated by the system both for the Medical Center as a whole and for each clinical department. r The system includes the review of the care provided by residents. Corrective actions for residents are also captured in the system. The mortality and morbidity modules are utilized by departments for discussion of cases at the weekly or monthly M&M conferences for educational purposes. r The system is continuously monitored, maintained, and improved by the QI Department. Condentiality of peer reviews The MS-PRS is developed for the use as an internal QI tool. Therefore, peer review information in the system is only accessible to the persons involved in the QI. Both the name of the provider under review and the name of the peer reviewer are blacked out once the information is entered. Feedback reports are without provider or peer reviewer identications.

Feedback and reporting To enhance the processing, accuracy and utility of the peer reviews, MS-PRS has developed several feedback functions and interim reports to the clinical departments through the QIMs and to all parties involved in processing the reviews. The following are the major built-in functions:

r If a case requires referral to another department, MS-PRS automatically generates a new review form for the receiving department. Multiple reviews by different clinical services for each death are captured and linked in the System. A tracking system is in place to improve the processing of each review including when the medical record is successfully retrieved, when the medical record is given to the peer reviewer, when the peer review is done, and when the peer review information is entered into the System. A list of incomplete reviews requiring further action is produced in real-time in the system so that the staff and the QIMs can access it anytime. The list provides the reasons why the review is incomplete. Weekly reports are generated and uploaded in the system to show the number and percent of reviews by department as a progress report to each department to monitor review activity. Quarterly reports are generated and provided to the departments and leadership regarding the responses to the questions, the level of care provided, the opportunity for improvement, and improvement actions taken. Cases requiring the review of the EPRC are generated for discussion from the system. Decisions by the SPRC and EPRC on the preventability of the death and the acceptability of improvement actions are recorded directly into the system during the meetings.

r r

Stafng The core stafng for the development of the system consists of a full-time computer engineer, half-time project coordinator, and parttime programmer for generating reports from the system. They work under the guidance of the Director of Biostatistics and Outcomes Assessment. This ofce is responsible for the development of the MS-PRS, coordination of the measurements, and compilation of summary reports. The implementation team includes all the QIMs, under the direction of the Director of QI, who are responsible for communicating with the departments regarding the peer review system, reports, follow-up actions, and orientation of new members in the department. There is an administrative support team in the QI Department who assists in entering the peer reviews that are done on paper into the MS-PRS. At this point in time, some peer reviews are still being recorded on paper due to the restriction of the online Intranet web entry capabilities to the main hospital facility only. Medical staff whose ofces are located in other buildings will not be able to access the online Intranet web page.

Journal for Healthcare Quality

Table 1. Mortality Peer Reviews Activities

Opportunities for Improvement Number of mortalities reviewed No opportunity for improvement at level 1 (%) Had opportunity for improvement at level 1 (%) Number of mortalities reviewed at levels 2 and 3 Final status not determined yet (%) No opportunity for improvement (nal) (%) Had opportunity for improvement (nal) (%) 2007 842 798 (94.8) 44 (5.2) 44 0 (0.0) 32 (72.7) 12 (27.3) 2008 785 752 (95.8) 33 (4.2) 33 0 (0) 19 (57.6) 14 (42.4) 2009 726 693 (95.5) 33 (4.5) 33 3 (9.1) 23 (69.7) 7 (21.2) 2010 746 723 (96.9) 23 (3.1) 23 2 (8.7) 19 (82.6) 2 (8.7)

Results
Peer Review Activities Centralized
After 4 years of development and pilot testing, we have in place an electronic MS-PRS to process both random and focused peer reviews. We have created a database that (a) provides accurate data on outcomes to both clinicians and hospital administration, (b) monitors outcome and process measures over time by individual provider or clinical department/division, and (c) tracks the adequacy and timeliness of improvement actions taken. We have observed improvements in educational opportunities and practice patterns of the medical staff, the delivery of healthcare, and hospital environment and policy. Further, MS-PRS has brought together the fragmented peer review activities throughout the healthcare network and allowed peer reviews from all departments to be centralized, standardized, and quantied as a whole.

The Mortality Peer Reviews

The mortality module in MS-PRS has been operational since January 2007 and can provide sufcient information for a more complete picture of the process and outcome. The random review modules were implemented at a much later date in July 2010. Thus, we did not include their evaluation at this point. During 2007 and 2008, mortality reviews were recorded on the paper form by reviewers and later entered into MS-PRS by the supportive staff. Beginning in 2009, the front-end Intranet web entry was operational and the peer reviewers could enter their reviews online into MS-PRS.

Opportunities for Improvement Identied

Over the 4-year period, 2007 through 2010, a total of 3,099 deaths were peer-reviewed, 133 (4%) were considered to have an opportunity for improvement at the rst-level review. Of the 133 cases, 5 (4%) have not been peer-reviewed by the EPRC at this time, 93 (70%) cases were nalized as having no opportunity for improvement, and 35 (26%) cases were considered as having opportunities for improvement (Table 1). Of the 35 cases that had a nal determination to have an opportunity for improvement by the EPRC, 84 improvement actions were taken. Among them, 51 (61%) were provider-related and 33 (39%) were systemrelated. The improvement actions are listed in Table 2.

Reports and Feedback Provided

Feedback to the clinical departments are provided in the form of monthly operational reports and quarterly outcomes reports. The operational reports provide each department the progress of the peer review activities in the different modules in MS-PRS. The quarterly outcomes reports provide the process outcomes in terms of appropriateness or adequacy of the medical history, physical examination, formulation of diagnostic impression and assessment, monitoring of patients progress and follow-up, supervision of residents or medical students, use of tests, medications, blood or blood products, invasive procedures, and documentation. Quarterly reports are also provided for peer review outcomes in terms of level of care met, opportunity for improvement, and improvement actions taken.

Measures of Potential Impact

It is difcult to measure the direct impact of MS-PRS as it is an integral part of the quality improvement effort at our institution. As an overall quality indicator on which MS-PRS might have an indirect effect, we examined the trend for the number of active medical malpractice claims at the beginning of the month per

Vol. 00 No. 0 xxx 2012

Table 2. Improvement Actions Taken

Type of Improvement Actions Total number cases that had opportunity for improvement Total number of improvement actions taken Provider-related subtotal Education including Mortality and Morbidity Conferences Verbal counseling including review of procedures Continuous monitoring of performance Written counseling Improved supervision System-related subtotal Generated new policies Improved hospital environment, safety and cleanliness Improved communication within and between departments Increased stafng Purchased new equipment Increased ICU beds or operating room operations Claried existing policies Generated new form to capture needed information

Number (%)

Number (%)

35 84 51 (61) 18 (35) 17 (33) 7 (14) 5 (10) 4 (8) 33 (39%) 9 (27) 6 (18) 6 (18) 3 (9) 3 (9) 2 (6) 2 (6) 2 (6)

Of the 133 cases that had opportunity for improvement from level 1 review, 93 (70%) were nalized to have no opportunity for improvement, 35 (26%) were nalized to have opportunity for improvement, and 5 (4%) are still pending the last level of review.

100,000 patient encounters during the month from 2006 through 2010 (Figure 1). We used a patient day as an inpatient encounter and an outpatient visit as an outpatient encounter. The total patient encounters for the month is the total of the inpatient days and outpatient visits for the month. It can be observed that there has been a steady signicant downward trend since 2006. To assess the relevance of the decline at our institution, we compared the number of our annual medical malpractice claims with our national and Californias number of medical malpractice payment reports against physicians published in a national database (DHHS, 2011) and with Californias number of complaint cases against physicians published by the Medical Boards of California. It can be observed from Table 3 that the number of medical malpractice claims on our institution declined 58% from FY 06/07 to FY 09/10, signicantly higher decline than that experienced in the United States or in California. Additional measures of impact would be ideal such as serious patient safety events, provider satisfaction with the new process. Efforts will be continued to include these measures in our future evaluations.

Discussion
Despite limited resources and mounting known challenges, we have developed and imple-

mented an electronic MS-PRS that includes mortality reviews of 100% of deaths, an ongoing random review for reappointment and operative procedures, and morbidity peer reviews. We evaluated the quality of our peer review system using the Peer Review Program SelfEvaluation Tool developed and validated by Edwards (2009a, 2009b, 2010). The tool has 13 aspects of evaluation and has a total score of 100. Based on a recent national survey of 330 hospitals, the total scores ranged from 0 to 85 with a mean of 45. Using this tool, our mean score is 69, which was derived from the independent rating by each of the authors. Using this tool, the major challenge we identied is the lack of timely performance feedback. Currently it can take from 1 month to 1 year to complete a mortality review. The standard used in the tool considers a timely review to be completed in less than 3 months. Another challenge we identied is the use of a 7 or more rating scale to measure each performance element. Our current instrument uses different scale for different measures. We use a dichotomous scale for process measures, four levels for quality of care and three levels for opportunity for improvement. Other challenges identied using this tool include standardization of process and review structure and recognition of excellence. We believe that we have made signicant achievements but many challenges lie ahead. Based on our experience we offer

Journal for Healthcare Quality

Figure 1. Active Medical Malpractice Claims at Beginning of Month per 100,000 Patient Encounters During Month (January 2006 through December 2011)

the following lessons learned and recommendations.

r
Lessons Learned and Recommendations

r When budgeting for such a development,

adequate stafng and adequate infrastructure must be included to assure proper implementation, monitoring, and dissemination of the ndings. r Must be supported by a centralized electronic medical records system. Because of the need to pull cases from different hospital information systems, the integrity of the peer reviews is dependent on integrated, complete, and accurate information. In addition, peer reviewers must be provided universal electronic access to record the peer reviews online. r Must have continuous in-service, monitoring, and improvement of the system. All departments must have continuous in-service. A peer review system should monitor the level of care received by the patient, any system breakdown associated with the care of the patient, and the follow-up actions needed to achieve closure for each case. r Must balance the peer review workload among departments. Each department has a dif-

ferent average number of morbidities, mortalities, providers, and operative procedures. All clinical departments must buy in. Without full departmental cooperation, peer review will not be fair or balanced, reports on providers cannot be compared, and followup actions will not be consistent. Must improve the peer review instrument to reduce variation in judging the quality and appropriateness of care and reviewer biases as found by others (Kadar, 2010). Must be endorsed and supported by the medical staff and the institution. The Attending Staff Association must ensure compliance with, and thus utility of, the peer review system. The culture of peer review must be changed from a punitive to a rewarding philosophy. If we truly want to improve quality, it has to be internal, nonthreatening, condential, and yet accurate and effective. Overall, no amount of science, technology, or knowledge is sufcient to bring about good without proper implementation, monitoring, and dissemination; and, no amount of support of a peer review system is sufcient to bring about good without wide endorsement by the medical staff and the institution.

Vol. 00 No. 0 xxx 2012

Table 3. Comparison of the Number of Medical Malpractice Claims Opened During Year Between Our Institution and California and the United States
United States Number of Medical Malpractice Payment Reports Against Physicians 14,006 12,490 11,475 11,021 10,772 0.027 23% (3,234/14,006) 0.004 California Number of Medical Malpractice Payment Reports Against Physicians 1,191 1,075 999 962 1,003 0.011 16% (188/1,191) 0.001 Number of Complaint Cases Opened Against Physicians 1,331 1,182 1,133 1,123 1,312 0.008 1% (19/1,331) <0.0001 Our Institution Number of Medical Malpractice Claims Opened 77 51 56 41 32 58% (45/77) -

Year FY 05/06, 2005 FY 06/07, 2006 FY 07/08, 2007 FY 08/09, 2008 FY 09/10, 2009 p-Value from chi-square test comparing 5-year experience to our institution Percentage decrease from FY 05/06 or 2005 to FY 09/10 or 2009 p-Value from chi-square test comparing the change from 2005 to 2009, to our institution

Data from Medical Board of California and our institution are reported by scal years and data from the National Practitioner Data Bank are reported by calendar year. Data from the National Practitioner Data Bank Combined Annual Report 2007, 2008, and 2009. U.S. Department of Health and Human Services, Health Resources and Services Administration, Bureau of Health Professions, Division of Practitioner Data Bank, September 2011, pp. 6667. Data from Medical Board of California: http://www.mbc.ca.gov/statistics_complaint-investigate.pdf. Bolded results show signicant ndings.

We have addressed the challenges for establishing an MS-PRS in a large teaching hospital. We must also address challenges encountered in measuring the impact of the system. The most direct measures of impact are the number of preventable deaths and the corrective actions taken by the institution as we have reported here. The other measure of impact, the medical malpractice claims, reported here cannot be considered as directly related to the MS-PRS as many other QI programs or management changes have occurred over the same time period. Efforts are underway to evaluate the impact of the system by studying indicators that may be more directly related to the system such as serious patient safety events, provider satisfaction, related readmissions, and related mortalities.

Acknowledgment
This project is supported in part by the Kaiser Permanente Southern California Region Community Benet Program.

References
Antonacci, A. C., Lam, S., Lavarias, V., Homel, P., & Eavey, R. A. (2009). A report card system using error prole analysis and concurrent morbidity and mortality review:

Surgical outcome analysis, Part II. Journal of Surgical Research, 153, 95104. Department of Health and Human Services (DHHS). (2011). National Practitioner Data Bank combined annual report 2007, 2008, and 2009. Retrieved March 2, 2012, from http://www.npdb-hipdb.hrsa.gov/ resources/reports/2007-08-09NPDBAnnualRep.pdf. Edwards, M. T. (2009a). Measuring clinical performance. Physician Executive Journal, 35, 4043. Edwards, M. T. (2009b). Peer review: A new tool for quality improvement. Physician Executive Journal, 35, 5459. Edwards, M. T. (2010). Clinical peer review program selfevaluation for US hospitals. American Journal of Medical Quality, 25, 474480. Edwards, M. T. (2011). The objective impact of clinical peer review on hospital quality and safety. American Journal of Medical Quality, 26, 110119. Jarvi, K., Sultan, R., Lee, A., Lussing, F., & Bhat, R. (2002). Multi-professional mortality review: Supporting a culture of teamwork in the absence of error nding and blameplacing. Hospital Quarterly, 5, 5861. Kadar, N. (2010). Systemic bias in peer review: Suggested causes, potential remedies. Journal of Laparoendoscopic & Advanced Surgical Techniques. Part A, 20, 123128. Keroack, M. A., Youngberg, B. J., Cerese, J. L., Krsek, C., Prellwitz, L. W., & Trevelyan, E. W. (2007). Organizational factors associated with high performance in quality and safety in academic medical centers. Academic Medicine, 82, 11781186. Livingston, E. H., & Harwell, J. D. (2001). Peer review. American Journal of Surgery, 182, 103109. Marren, J. P., Feazell, G. L., & Padock, M. W. (2003). The hospital board at risk and the need to restructure the relationship with the medical staff: Bylaws, peer review and related solutions. Annals of Health Law, 12, 179 234. Nolan, S. W., Burkard, J. F., Clark, M. J., Davidson, J. E., & Agan, D. L. (2010). Effect of morbidity and mortality

Journal for Healthcare Quality

peer review on nurse accountability and ventilatorassociated pneumonia rates. Journal of Nursing Administration, 40, 374383. Olcott, C., IV, Mitchell, R. S., Steinberg, G. K., & Zarins, C. K. (2000). Institutional peer review can reduce the risk and cost of carotid endarterectomy. Archives of Surgery, 135, 939942. Owens, S. R., Wiehagen, L. T., Kelly, S. M, Piccoli, A. L., Lassige, K., Yousem, S. A., et al. (2010). Initial experience with a novel pre-sign-out quality assurance tool for review of random surgical pathology diagnoses in a subspecialty-based university practice. American Journal of Surgical Pathology, 34, 13191323. Spaeth, R. G., Pickering, K. C., & Webb, S. M. (2011). Quality assurance and hospital structure; how the physicianhospital relationship affects quality measures. Annals of Health Law, 12, 235247. Williams, J. R., Mechler, K., & Akins, R. B. (2008). Innovative peer review model for rural physicians: System design and implementation. Journal of Rural Health, 24, 311315.

Ling Zheng, MD, PhD, is an epidemiologist and biostatistician at the Los Angeles County University of Southern California Medical Center who is in charge of the reporting of ndings. Brittany Wagman, BS, was the research coordinator at the Los Angeles County University of Southern California Medical Center during the early years in charge of the implementation and in-service of the system. Garren Low, MS, is the research coordinator at the Los Angeles County University of Southern California Medical Center in charge of the implementation and in-service of the system. Roger Chang, MA, is a doctoral candidate in Biostatistics at the Los Angeles County University of Southern California Medical Center who assists with the integration of data systems and compilation of reports. Nicholas Testa, MD, is the associate medical director for quality improvement at the Los Angeles County University of Southern California Medical Center who oversees the implementation of the system. Stephanie L. Hall, MD, is the chief medical ofcer at the Los Angeles County University of Southern California Medical Center who initiated the project and oversees the development, implementation, and utility of the system. For more information on this article, contact Linda S. Chan at lchan@dhs.lacounty.gov.

Authors Biographies
Linda S. Chan, PhD, is the director of Biostatistics and Outcomes Assessment at the Los Angeles County University of Southern California Medical Center and is professor of Research at the Keck School of Medicine of the University of Southern California. She directs the development and implementation of the medical staff peer review system. Manal Elabiad, MS, is a computer engineer at the Los Angeles County University of Southern California Medical Center who develops and maintains the system.