Job and Task Analysis to Identify Failure Points in Switching Operations

Technical Report

Job and Task Analysis to Identify Failure Points in Switching Operations

1008692

Final Report, October 2004

EPRI Project Manager B. Damsky

EPRI 3412 Hillview Avenue, Palo Alto, California 94304 PO Box 10412, Palo Alto, California 94303 USA 800.313.3774 650.855.2121 askepri@epri.com www.epri.com

DISCLAIMER OF WARRANTIES AND LIMITATION OF LIABILITIES

THIS DOCUMENT WAS PREPARED BY THE ORGANIZATION(S) NAMED BELOW AS AN ACCOUNT OF WORK SPONSORED OR COSPONSORED BY THE ELECTRIC POWER RESEARCH INSTITUTE, INC. (EPRI). NEITHER EPRI, ANY MEMBER OF EPRI, ANY COSPONSOR, THE ORGANIZATION(S) BELOW, NOR ANY PERSON ACTING ON BEHALF OF ANY OF THEM: (A) MAKES ANY WARRANTY OR REPRESENTATION WHATSOEVER, EXPRESS OR IMPLIED, (I) WITH RESPECT TO THE USE OF ANY INFORMATION, APPARATUS, METHOD, PROCESS, OR SIMILAR ITEM DISCLOSED IN THIS DOCUMENT, INCLUDING MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, OR (II) THAT SUCH USE DOES NOT INFRINGE ON OR INTERFERE WITH PRIVATELY OWNED RIGHTS, INCLUDING ANY PARTY'S INTELLECTUAL PROPERTY, OR (III) THAT THIS DOCUMENT IS SUITABLE TO ANY PARTICULAR USER'S CIRCUMSTANCE; OR (B) ASSUMES RESPONSIBILITY FOR ANY DAMAGES OR OTHER LIABILITY WHATSOEVER (INCLUDING ANY CONSEQUENTIAL DAMAGES, EVEN IF EPRI OR ANY EPRI REPRESENTATIVE HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES) RESULTING FROM YOUR SELECTION OR USE OF THIS DOCUMENT OR ANY INFORMATION, APPARATUS, METHOD, PROCESS, OR SIMILAR ITEM DISCLOSED IN THIS DOCUMENT. ORGANIZATION(S) THAT PREPARED THIS DOCUMENT Beare Ergonomics Quality Training Systems

ORDERING INFORMATION
Requests for copies of this report should be directed to EPRI Orders and Conferences, 1355 Willow Way, Suite 278, Concord, CA 94520, (800) 313-3774, press 2 or internally x5379, (925) 609-9169, (925) 609-1310 (fax). Electric Power Research Institute and EPRI are registered service marks of the Electric Power Research Institute, Inc. EPRI. ELECTRIFY THE WORLD is a service mark of the Electric Power Research Institute, Inc. Copyright 2004 Electric Power Research Institute, Inc. All rights reserved.

CITATIONS
This report was prepared by Beare Ergonomics 6243 Parallel Lane Columbia, MD 21045 Principal Investigator A. Beare Quality Training Systems 6418 Dry Barley Lane Columbia, MD 21045 Principal Investigator S. Lutterodt This report describes research sponsored by EPRI. The report is a corporate document that should be cited in the literature in the following manner: Job and Task Analysis to Identify Failure Points in Switching Operations, EPRI, Palo Alto, CA, 2004. 1008692.

iii

REPORT SUMMARY

Though not widely used in the power delivery segment of the commercial electric power industry, linking errors to a formal task analysis is a common technique of human factors engineering that can identify weaknesses in processes and procedures. This study analyzes a collection of switching incidents to determine at which step in the switching process the errors occurred. The report presents a model of how investigation results can be sifted for useful clues about the steps that might benefit from repeated training or a rethinking of established procedure. Appendices in the report contain an outline of the cases and a list of the steps in the switching process. Background EPRIs 1996 report, Field Operation Power Switching Safety (TR-106465), was a milestone that sparked interest across the industry in switching safety. That report led directly to the first switching safety conference held the next year. Since then, the conference has become an annual event with a strong and growing attendance focused on exchanges of incidents and findings from individual utilities as well as EPRI research. The keystone of this project has been a combination of surveys with analysis. Done properly, this approach yields insights for improved practices. This report covers the latest work on the subject: it shows how incidents can be systematically analyzed to reveal where errors are most likely to occur. The proper study of incidents is crucial to understanding and correcting error-prone procedures, and this report shows the power of combining analysis with the structure that was established with an earlier classification report, Generic Job and Task Analysis, EPRI report 1001789. The Job and Task Analysis (JTA) structure provides a framework for analysis and discussion of switching incidents that can be used for training and the assignment of responsibility as well as for the analysis of errors Objectives To document a method for analyzing operating incidents according to the JTA structure To identify the switching steps that are most likely to involve an error To show the value of a standard terminology that allows incidents from different companies to be compared.

Approach EPRI arranged for eight cooperating utilities to provide examples of switching incidents. The project team of experts in the field of accident prevention and accident analysis analyzed these data. The team made an effort to collect all relevant and available details from participating utilities while keeping the source of all individual data confidential to encourage full and complete disclosure. In all there were 168 cases relating to human error with sufficient detail to v

be classified according to the JTA scheme. The team assigned the errors to steps in the switching process and noted the instances in which an error was found at more than one step. Results The report provides useful tables summarizing studied cases and showing the tasks attributed to the errors. The errors recorded in the incident reports were not randomly distributed but tended to cluster on a few tasks in the switching process. The most common error attributed to control center operators involved the step of reviewing documents for accuracy and completeness. The most common error attributed to the authorized switching person was operating the wrong piece of equipment. This error was approximately three times more common in cases of switching for removal than in switching for restoration. A utility can copy the procedure demonstrated in this report to analyze its own incidents and identify specific tasks that are error prone. These specific tasks may benefit from additional emphasis in training and job briefings. EPRI Perspective Safety and reliability have never been more important for a utility. This report identifies the steps that experience shows are the ones most likely to involve an error. If a utility lacks the resources to carry out its own study, it can base a safety program on these findings, using the error occurrences as a guide to where to place emphasis. If a larger utility wants to carry out its own study, this report will serve as a guide to the methodology. A middle approach would be for a utility to construct a database that includes its own cases as well as the cases described in this report. Keywords Safety practices Switching Substations Power system operation Power system control

vi

ACKNOWLEDGMENTS
The following Utilities contributed information for this report. Their interest and support is gratefully acknowledged. American Electric Power BGE Consolidated Edison Company Duke Energy Hydro One Tennessee Valley Authority United Illuminating Western Area Power Administration

vii

CONTENTS

1 INTRODUCTION ....................................................................................................................1-1 2 SOURCE MATERIALS...........................................................................................................2-1 The Incident Reports .............................................................................................................2-1 The Task Analysis .................................................................................................................2-2 3 METHOD ................................................................................................................................3-1 Examples of Task Assignments ............................................................................................3-1 Default Task Assignments ..................................................................................................3-3 Assumptions..........................................................................................................................3-4 Conservatism ........................................................................................................................3-5 Forced fit Task Assignments...............................................................................................3-5 4 RESULTS ...............................................................................................................................4-1 Errors Attributed to Control Center Operators (CCOs)..........................................................4-6 Errors Attributed to Authorized Switching Persons (ASPs) in the Field................................4-7 Errors Attributed to Work Crews............................................................................................4-8 Errors Occurring during Unscheduled Switching ................................................................4-8 Errors Attributed to Relay Technicians..................................................................................4-9 5 DISCUSSION..........................................................................................................................5-1 6 CONCLUSIONS .....................................................................................................................6-1 7 REFERENCES .......................................................................................................................7-1 A LIST OF CONTRIBUTING UTILITIES .................................................................................. A-1 Utilities Contributing Incident Reports for this Study ............................................................ A-1 B LISTING OF HUMAN ERROR EVENTS .............................................................................. B-1

ix

C OUTLINE OF EPRI JOB AND TASK ANALYSIS FOR PLANNED OUTAGES .................. C-1 D FULL TEXT OF TASK DESCRIPTIONS USED IN THIS REPORT...................................... D-1 E INCIDENTS INVOLVING RELAY TECHNICIANS................................................................ E-1

LIST OF FIGURES
Figure 2-1 Overview of the Process of Handling the Planned Outage of a Line or Piece of Equipment ......................................................................................................................2-2 Figure C-1 Overview of the Process of Handling the Planned Outage of a Line or Piece of Equipment ..................................................................................................................... C-1

xi

LIST OF TABLES
Table 4-1 Findings by High-Level Task .....................................................................................4-1 Table 4-2 Tasks in which Errors Occurred.................................................................................4-2 Table E-1 Incidents Involving Relay Technicians (RTs)........................................................... E-1 Table E-2 Summary of Errors in Incidents Involving Relay Technicians (RTs)....................... E-9

xiii

1
INTRODUCTION
Review of operating incident data is useful because it can identify weaknesses in processes and procedures. Linking errors to a formal task analysis is a common technique of human factors engineering that does not seem to have enjoyed wide application within the power delivery segment of the commercial electric power industry. In this report, human errors described in utility incident reports are linked to the tasks that were being or should have been performed, as described in the EPRI Generic Job and Task Analysis Database (Rev1: available in the fall of 2004.) The objective of the present study was to identify those tasks in the switching process in which human errors appear to occur most frequently. Because a task analysis is the basis of the operating procedures and training program, linking errors to a task analysis is of potential benefit to the organization. Specific tasks (e.g., communications, understanding of equipment status) identified as error-likely may benefit from additional emphasis in training and job briefings. Ultimately, we believe the process used in this study can be applied by an individual utility to determine its own points of greatest vulnerability to human error incidents, and hence to develop tools and techniques that will help to reduce the number of human error events experienced. In addition, relating errors to specific tasks is useful to the maintenance and updating of the task analysis itself, as such an effort may identify task elements that are incomplete or missing altogether from the current analysis.

1-1

2
SOURCE MATERIALS
Source materials used for this study were incident reports submitted by participating utilities and a pre-publication copy of Rev 1 of EPRIs Generic Job and Task Analysis (JTA) Database.

The Incident Reports

Eight utilities supplied incident reports for this study. Participating utilities are listed in Appendix A. The incident reports come from a variety of sources. Three utilities supplied incident summaries from computerized databases. One of these sets consisted of all human error incidents from a larger database used to document system disturbances. These reports are concerned with the consequences of human errors, not the exact nature of the errors or how they happened to come about. Five supplied individual incident reports. Although some of them were quite detailed, many were summaries prepared for circulation within the utility, and did not contain as much detail as reports prepared for management or evaluation by an Operations Review committee. Some of the reports submitted were discarded because the errors described did not occur in the process of planning or executing switching instructions (e.g., a backhoe cut into control cables, prints were inaccurate), and thus could not be related to the JTA. Many others were excluded because they lacked sufficient detail to permit any kind of analysis. For example: 138kV MOAB switch flashed over during switching performed by co-op. This was called a switching error, but there was insufficient detail in the report to identify exactly what the error was, or where in the process of planning and execution it occurred. Of the reports reviewed, 168 related to the planning and execution of switching activities, and contained enough detail to identify the subtasks and task steps in which the errors identified in the reports occurred. A brief synopsis of each of the 168 incident reports used in the study is presented in tabular form in Appendix B. This table also lists the tasks involved in the errors identified. In the JTA, tasks are identified with particular classes of personnel: control center operators, switchmen in the field, and members of the work party. Incident reports typically identify the personnel involved by job title, but these vary from one utility to the next. The generic job titles below are used in this report and its Appendices: CCO: Control Center Operator (system operator) whether bulk power or distribution. ASP: Authorized Switching Person, which covers a range of job titles, such as substation technician, substation electrician, lineman, service operator, etc. RT: Relay Technician, used as a generic term for a variety of job titles. 2-1

Source Materials

The Task Analysis

The task analysis used for this report is the EPRI Generic Job and Task Analysis Database (Rev. 1 2004) which is a product of the Power Switching Safety and Reliability Project sponsored by EPRIs Substation Operations and Maintenance Target. Although it is based on EPRI TR-1001789, Generic Job and Task Analysis Handling Planned Outages and Hot Line Work Requests published in September of 2002, the JTA database contains a significant amount of new material and a more detailed breakdown of many of the tasks described in the 2002 technical report. This task analysis (hereafter called simply the JTA) is based on the switching procedures used by over 20 utilities, and has been reviewed for completeness by a committee of individuals who themselves perform or manage the performance of switching on a daily basis. The JTA identifies nine activities to be performed in the process of handling a planned outage for a line or a piece of equipment. These are shown in Figure 2-1.
Task 1 Complete an Outage Request & Submit for Approval Task 2 Conduct an Initial Review of the Outage Request Task 3 Write Switching Instructions for Removal & Return of Lines/Equipment

Task 4 Review Outage Request on the Day of Scheduled Work

Tasks 5 & 6 Implement Switching for Removal of Lines or Equipment

Tasks 7 & 8 Issue/Receive a Clearance

Tasks 9, 10 & 11 Release/Accept a Clearance

Tasks 12 & 13 Implement Switching for Return of Lines or Equipment

Task 14 Review & Close

Figure 2-1 Overview of the Process of Handling the Planned Outage of a Line or Piece of Equipment

2-2

Source Materials

Each of the activities shown in Figure 2-1 is a high-level task, where task is defined as an organized set of activities performed to achieve a defined goal. Because they involve activities performed by different personnel at different locations, four of these, implementing switching for removal and restoration, and issuing and releasing a clearance, are resolved into separate tasks for control center and field personnel, yielding a total of 14 high-level tasks. In the EPRI JTA, each of the 14 highlevel tasks is in turn analyzed into a set of steps, the activities that must be performed to achieve the goal of the high level task. Each step is in turn analyzed into a series of individual substeps required to accomplish it. The first word of each task, step, or substep description is an action verb. At the step level, the verb is often followed by a list of objects, e.g., verify the following: . . . In the hierarchical organization of the task analysis, steps are referenced by two numbers and individual substeps are referenced by three numbers; task, step, and substep. In the JTA database, some substeps are further broken down into a series of elements, a fourth level that is identified with four numbers. A complete listing of tasks from the JTA to the second (step) level is shown in Appendix C. For brevity, abbreviated descriptions of selected substeps and elements (the third and fourth levels of the JTA) are used in the tables in this report. The full text of the task (step, substep, or element) description for each abbreviated listing is provided in Appendix D. Note that in the analysis presented below, the terms task and step are sometimes used interchangeably, but the number of digits associated with the activity (two for step; three for substep; etc.) correctly identifies the level of the activity within the hierarchy.

2-3

3
METHOD
The method used was to review human errors identified in descriptions of operating incidents supplied by utilities participating in EPRIs ongoing Switching Safety and Reliability Project. Brief summaries of each incident containing as many pertinent facts as could be determined from the original report were prepared and reviewed to identify the third (step)- or fourth (substep)level task steps from the JTA which were performed inadequately or sometimes omitted altogether. Because of the structure and completeness of the JTA, this is very easy for cases in which the report tells what was done in error, or omitted, that contributed to the incident.

Examples of Task Assignments

Assignment of tasks on the basis of the information contained in brief incident reports is illustrated in the following examples, identified by the incident number, from Appendix B: Incident # 39 In restoration from an outage, a feeder was transferred to a tie circuit: after about 8 minutes, the feeder was determined to be dead. Upon inspection, a maintenance disconnect was found open [1]: ASP restoring the station had failed to close it. The utility blamed the incident on poor CCO/ASP communications, plus fact that ASP failed to check all associated equipment before beginning switching operations [2]. The CCO failed to observe loss of load promptly due to pace of operations [3]. The brackets in the description above denote three separate errors. These three errors map onto the following task steps from the JTA: [1] 9.1.2 (Work Crew) Verify system equipment returned to same status as found when accepting clearance. [2] 13.4.4 (ASP) Conduct site review of existing conditions. [3] 12.6.5 (CCO) Monitor watt/volt/amp indications while switching instructions executed. These three tasks were identified as follows: [1] The JTA contains a number of steps verifying that conditions are as they should be upon completion of work. Step 9.1.2 above is to be performed by the work crew. The error was attributed to them because the open switch was identified as a maintenance disconnect. There are analogous steps that should be performed by the ASP when switching to remove from service (6.5.10) and when switching to return equipment to service (13.5.8). 3-1

Method

The open maintenance disconnect is the pre-existing condition that set up this incident. The utilitys report does not explicitly identify who left it open (though one would think they would be interested in finding out how it happened). Task 9.1.2, to be performed by the work crew, was chosen instead of one of those performed by the ASP because the incident occurred during restoration from an outage, and the report did not state that the ASP had failed to perform any step in the planned switching as directed, but, rather, that he or she had failed to inspect for such loose ends before performing the restoration switching. Had the same phrase occurred in a report of an incident that occurred during switching for removal, the error of leaving the disconnect open would have been attributed to the last ASP to contact the equipment. In that case, task 13.5.8, visually confirm operations have been completed and that the condition is as it should be would have been selected. Parenthetically, this error is the one that made the other errors consequential. Had the ASP conducted his review as expected, he should have discovered and corrected the open disconnect, after which the switching would have gone off as planned. Similarly, had the disconnect been closed, the feeder would have picked up load whether the CCO was watching it or not. The other two errors would never have been discovered because they would have had no visible effect. [2] 13.4.4 (ASP) was identified as the second task in which an error occurred because the report criticized the ASP for failing to check the equipment status before beginning his switching; the language in the report is very similar to that in the JTA step. [3] Similarly, task 12.6.5 (CCO) was easily identified because the report said the CCO was slow in observing loss of load, and this monitor task is the one in which loss of load would be observed. From the incident reports comment on communications, it is likely that there were errors in additional tasks, probably one or more of the CCO or ASP review tasks that are devoted primarily to communication of detailed system status. However, because no details as to how communications were poor were given in the incident report, it is not possible to identify specific tasks with the faulty communications. Incident # 34 An accident with some equipment damage. Reclosers were properly opened at 3AM for work, but not recorded in the log by CCO due to distraction [1]. Around noon an ASP was sent (by a second CCO) to check recloser status. He reported them open & asked if CCO wanted them on: the CCO said yes. The report identified three errors: initial failure to record recloser status plus failure to establish common big picture between 2 CCOs and the ASP[2], plus failure to follow the companys communication protocol. The utilitys report speculated that having more than one CCO involved may also have contributed. Tasks in which errors contributed to the incident were identified as: [1] 5.6.3 (CCO) Log all actions taken on records and reports. [2] 5.2.3 (CCO) Review (big picture) with field crew leader (switch person) before starting job. 3-2

Method

In the example, task 5.6.3 was identified because the report stated that the first CCO had not recorded the recloser status in the log. Task 5.2.3 was identified because the report explicitly stated that establishing a common big picture among all parties was not done adequately. The tasks are from the remove from service sections of the JTA because the report does not state that the incident occurred in conjunction with restoration: had it so stated, the equivalent step from the restoration section, 12.4.1, would have been assigned. The remove section was used because it was clear the job for which the clearance was issued had not been completed. In this example, no error is attributed to the switchman, who was doing what he thought the CCO had told him to do, though he bears some responsibility for his imprecise communications (some utilities would also fault the CCO for letting him get away with them). It is also likely that the CCO did not take the time to analyze this apparently unexpected request and ask why the reclosers might be open, but nothing about it is mentioned in the report. Based on the times given in the report, it is likely that the job was handed off from one CCO to the other at shift turnover (task 5.6.5 in the JTA), and it may be speculated that this is when the CCOs failed to establish a common big picture between them. However, this task is not cited because the report did not mention shift turnover.

Default Task Assignments

The majority of incident reports were brief summaries intended for a database or distribution within the utility. Such summaries provide only those details their authors consider essential to the incident. Where insufficient detail to support assignment to another task was not provided, the following default task assignments were used: Instructions incomplete or incorrect: 3.1.2 (identify switches to operate). This assumes defect was in the research rather than writingup of the instructions. Whenever typographical errors to be called out in the report, task 3.7.2, Proofread and verify the accuracy of the switching instructions was used. Instructions written in the incorrect sequence: 3.1.4 (write steps in sequence). Failure to catch errors in written instructions in reviews: 3.7.1 (pre-approval review) and 4.1.5 (pre dispatch review). Execution errors: wrong switch operated by CCO (via SCADA): 5.5.1.2 (open) or 12.6.1.2 (close). Execution error: wrong switch operated by ASP in the field: 6.5.6 (open) or 13.5.5 (close).

The last bullet indicates that tasks 6.5.6 and 13.5.5 were used as the task assignment where inadvertent operation of switchgear by the ASP was described, with no additional details given. In cases where a more detailed report stated that the confirmation or verification of the equipment or controls to be operated was inadequate, substep 6.5.6.2 (during switching for removal from service) or 13.5.5.2 (during switching for restoration) was assigned. However, the majority of incident reports do not provide this level of detail.

3-3

Method

The wording, Failed to open, and Failed to close is also relatively common, and suggests failure of intent to perform the action. Such failures could arise from lack of knowledge if the switching is performed from higher-level; general orders that leave it up to the switch person to determine the actual equipment to be operated; or from simply overlooking a step in the instructions prepared by a system operator. However, all distinctions as to the causes of the error disappear when it comes to identifying the task step in which it occurred. Both inadvertent operation of the wrong equipment and failure to operate the correct equipment are errors in the performance of the same step in the JTA. Communication problems are often identified (or, more accurately, alluded to) in incident reports. These may occur during any one or more of a large number of the tasks devoted to information exchange identified in the JTA. Such steps are not cited unless there is sufficient information in the report to identify a particular step. Unfortunately, this level of detail is rare, even in very detailed reports.

Assumptions
Because the majority of incident reports were not very detailed, it was sometimes necessary to make additional assumptions. For example, in the analysis of report # 39, it was inferred (assumed) that the maintenance disconnect was open because someone forgot to ensure that it was in its expected normal position, and further assumed that the responsible party was the work crew rather than the ASP who performed the switching for removal from service. 1. Execution vs. Planning. An error was assumed to be in execution unless a planning defect was specifically identified. 2. Directed switching on poles. This was assumed to follow the same rules and procedures as planned switching of larger equipment. 3. Unscheduled switching. The term unscheduled switching was used to refer to emergency switching (e.g., responding to forced outages) or changes to a job in progress. It was assumed that these situations entail (or should entail) performance of many of the same planning tasks involved in planned outages, although they may be performed on a much abbreviated time scale and without checking-over by a second CCO. For example, when changing switching in the middle of the job the CCO still should consult appropriate documentation to identify the equipment to be switched to achieve whatever result is desired, consider the impact of the switching on other parts of the system, consider the effects of sequence of operations, etc. When responding to a forced outage, the CCO must identify the circuits and equipment affected and should confirm what he or she knows by recourse to circuit diagrams, one-lines, etc. That is, it was assumed that the CCO directing emergency switching is supposed to carefully identify affected equipment in essentially the same way as for planned switching. 4. As-found errors. For cases where switches were found in the wrong positions, it was assumed that an error was made in verifying that equipment was returned to correct status, and this is used as the faulty step of record for switches found in wrong position. This situation also reflects an error in execution, but the final step of verification should have caught it, and only one task was cited as having been performed in error. These errors were assigned to the last people in the station, either a work crew or an ASP restoring upon the completion of previous work, as inferred from the event description. 3-4

Method

Curiously, few of the reports of incidents to which unexpected as-found conditions contributed explained how the abnormal condition came to be in the first place. 5. Review steps. Although virtually all written routine switching is reviewed, review tasks were cited only if mentioned in the report. Thus a report that mentions that a step was omitted from the written instructions would cite task 3.1.2 (identify switches to operate). The review steps in which this error is supposed to be detected and corrected are 3.7.1, pre-approval review, and 4.1.3, pre-dispatch review, and the final reviews by the CCO prior to beginning the work, tasks 5.1.1 and 12.1.5. However, these would be cited only if the report explicitly mentioned that the instructions were not reviewed, or if those reviews failed to detect the error. 6. Pre-switching inspections by ASP. These were not assumed to be required and were not cited unless mentioned in the utilitys write-up. Proper pre-switching inspection might have prevented some of the subsequent execution errors, and errors involving unanticipated preexisting conditions, such as closing into grounds.

Conservatism
Items 4, 5, and 6 above represent a degree of conservatism on the part of the analyst. Any student of errors is well aware of the many dangers posed by assumptions. For this reason with the exception of the assumptions described above the analysis has tried to stick closely to the information supplied in the report (including the utilitys lessons learned, if included in the report) when identifying errors and the tasks within which they occurred. A reader familiar with the kinds of situations described may well surmise that there were other, unreported errors in other tasks involved in the event. However, as illustrated in the discussion of incident # 39 presented above, the analysis identified associated tasks or task steps only in cases where the report provided sufficient information to identify a specific task or step. Note that a consequence of the conservative approach used to assigning errors to tasks in the JTA is probable under-attribution or under-identification of such tasks. That is, errors may be more common than the analysis indicates.

Forced fit Task Assignments

In some cases an error is associated with a task that does not appear to be a good fit for the associated error description. In some cases, the task (step) identified is the one in which the associated action would occur in the task flow recorded in the JTA. For example, coordination/communication problems with other departments and other utilities are covered under tasks 5.1.4 or 12.3.3 (CCO): Verify any prerequisite switching/load transfer has been done. In other cases, the apparent ill-fit occurs because the full text of the task (step) description contains elements or actions that are not included in the abbreviated description given in the table of incidents. See Appendix D for the full text associated with the abbreviated task descriptions used in Appendix B. 3-5

4
RESULTS
The number of errors involving elements of each of the 14 high-level tasks shown in Figure 2-1 is listed in Table 4-1 below. The steps in which deficient performance resulted in or contributed to incidents are shown in Table 4-2, which also presents the number of times each task was involved in an incident, and the associated incident numbers from Appendix B. The boxes referred to in brackets in the third column of Table 4-2 are shaded text boxes containing speculative comments on some of the incidents. These speculations identify tasks that might have been involved in the incident, or whose performance might have prevented it. Tasks speculatively identified in these boxes are not included in task counts given in the text of this report.
Table 4-1 Findings by High-Level Task Task # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Task Description Complete outage request Initial review of outage request Write switching instructions Review on the day of the scheduled work Implement switching for removal control room actions Implement switching for removal field actions Issue a clearance control room actions Receive a clearance field actions Release a clearance field actions Accept release of a clearance control room actions Release a clearance in an emergency Implement switching for return control room actions Implement switching for return field actions Review & close outage request # of Citations 5 1 39 17 61 68 1 7 14 1 0 27 42 0

The tasks and substeps with which errors were associated are listed in Table 4-2. The 78 listed substeps are only slightly more than 1/4 of the 297 third-level substeps listed in the complete JTA for planned switching. 4-1

Results Table 4-2 Tasks in which Errors Occurred # 1. Abbreviated Task Statement # Events (Appendix B) Complete an Outage Request 5 1.1.1 (requestor) Identify what work will be done 1 94 1.1.3 (requestor) Identify the exact location of the work to be 2 141, 160 done 1.2.1 (requestor) Identify type of switching required to perform 1 153 work. 1.2.2 (requestor) Provide a diagram that identifies the work area 1 132 and the lines of equipment to be removed from service Conduct an Initial Review of the Outage Request 1 2.2.6 (preparer) Verify feasibility and determine need for load 1 153 transfer & coordinate with others as needed Write Switching instructions for Removal and Return of Lines and Equipment 39 3.1.2 (CCO) Identify all energy sources that could supply the line 8, 104, 107, 109, 111, 8 or equipment being removed from service 112, 120, 132 3.1.2 (CCO) Identify all energy sources that could supply the line or equipment being removed from service & identify grounds to 3 8, 107, 111 be placed 3.1.3 (CCO) Determine what devices must be switched 3 127, 148, 168 3.2.1 (CCO) Research, document, and verify work location and 2 2, 134 equipment to be returned to service 3.2.3 (CCO) Identify any equipment that needs to be returned to 1 108 a normal configuration as a part of switching 3.2.5 (CCO) Identify ground devices to be removed 1 5 3.3.1 (CCO) If previously-written switching instructions exist... verify accuracy & completeness & suitability for use given 4 92, 94, 147, 167 conditions that are expected to exist, and revise as needed 3.3.3 (CCO) For unusual situations, consult with subject matter 1 159 experts as needed 3.4.1 (CCO) Write switching instructions for removal in sequence, 4 98, 99, 142, 146 including disabling of protective relaying 3.5.1 (CCO) Write switching instructions for restoration in sequence, including re-enabling of devices disabled during 2 5, 6 switching for removal from service 5, 97, 99, 109, 127, 3.7.1.(CCO) Review documents for accuracy and completeness 9 147, 148, 153, 167 3.7.2 (CCO) Proofread & verify accuracy of switching instructions 1 146 Review the Outage Request and Switching instructions on the Day of the Scheduled Work 17 4.1.1 (CCO) Review available logs to determine if there have been any recent changes or other conditions that may impact the 2 128, 165 scheduled work 4.1.2 (CCO) Check for any conditions that could affect the priority 2 90, 95 of the requested outage 4.1.5 (CCO) Review the outage request and switching instructions against maps and one-lines to verify the technical 97, 109, 127, 130, 7 accuracy of the names of devices and the feasibility of the work 134, 142, 167 to be accomplished

2.

3.

4.

4-2

Results

Table 4-2 Tasks in which Errors Occurred (Continued) # Abbreviated Task Statement # Events (Appendix B) 4.1.7 (CCO) Verify that appropriate ground installation(s) have 2 107, 109 been included in the switching instructions 4.2.1 (CCO) Review current status of equipment from all available sources to verify that current status will not adversely 3 102, 105, 108 impact the planned outage 4.2.2 (CCO) Use available tools to study the impact of the Switching and clearance (i.e., power flow, security analysis, 0 [102 box] optimal power flow) 4.2.3 (CCO) Determine effect of planned procedure to local and 0 [6 box] interconnected power systems given current system conditions 4.4.4 (CCO) If the switching instructions need modifications, 1 97 follow the required procedure Implement Switching for Removal of Lines or Equipment Control Room Actions 61 5.1.1 (CCO) Assemble & review copies of Outage Request, 2 3, 53 switching instructions, maps, and one-lines 5.1.4 (CCO) Verify any prerequisite switching/load transfer has 23, 55, 108, 112, 119, 6 been done 149 5.2.3 (CCO) Review the work to be done (big picture) with 6 3, 34, 37, 53, 58, 168 Switch person before starting the job 5.3.2 (CCO) Have the Switch person read the switching 1 156 instructions back to you 5.3.6 (CCO) If there are uncertainties regarding the switching, do 53, 54, 72, 150, 159, 6 not proceed until they are resolved 163 5.4.2.2 (CCO) If field conditions are such that a change must be made to the switching instructions . . . Research proposed 3 3, 9, 99 changes to verify that they are appropriate and will not cause additional problems 5.4.2.3 (CCO) If field conditions are such that a change must be made to the switching instructions . . . Obtain second party 1 99 review and approval of proposed changes 5.4.2.6 (CCO) If field conditions are such that a change must be made to the switching instructions . . . Document each change, 1 69 initial and date change 5.5.1 (CCO) Switch devices by SCADA as required by the 1 150 switching instructions 1, 7, 40, 44, 86, 89, 110, 115, 118, 138, 5.5.1.2 (CCO) Operate the device 13 140, 145, 151 5.5.2 (CCO) Direct Field Crew Leader (Switch person) to proceed with switching and tagging, and grounding according to the 2 53, 124 written instructions 5.5.5 (CCO) Receive and record the time that switching was 1 157 performed 5.5.6 (CCO) Place tags on devices on SCADA and other control 3 7, 152, 165 room maps 5.6.1 (CCO) Update changes to system status on SCADA and 5 29, 51, 62, 64, 69 other control room maps

5.

4-3

Results

Table 4-2 Tasks in which Errors Occurred (Continued) # Abbreviated Task Statement # Events (Appendix B) 5.6.2 (CCO) Verify that all switching steps have been taken, and 2 120, 157 all tags have been placed . . . 5.6.3 (CCO) Log all actions taken on records and reports 6 34, 51, 60, 62, 69, 117 5.6.5 (CCO) If switching has not been completed before the end 2 30, 165 of the shift, review the status of the job with the oncoming shift Implement Switching For Removal of Lines or Equipment Field Actions 68 6.1.4 (ASP) If there are any technical uncertainties regarding the Outage Request and/or associated switching instructions, resolve 1 71 as needed; do not proceed until the uncertainties are resolved 6.2.1 (ASP) Verify job information with the System Operator, including identification of all automatically and remotely controlled 2 47, 71 switches that could cause opened devices to close 6.4.1. (ASP) Verify the technical accuracy of the switching 11, 53, 94, 153, 156, 5 instructions against a one-line and current conditions [18 box] 6.4.2 (ASP) Verify devices to be switched are in the expected 1 4 state before beginning switching 6.4.3 (ASP) Review available logs to determine if there are any 2 58, 71 conditions that may impact the scheduled work 6.4.4 (ASP) Conduct a Site Review of the existing conditions 8, 18, 19, 53, 58, 71, including locations of circuits and equipment, protective grounds, 8 131, 156 and equipment grounding conductors 9, 14, 19, 20, 28, 33, (ASP) Perform switching for isolation according to switching 36, 46, 49, 50, 52, 54, instructions 63, 68, 76, 77, 78, 96, 29 100, 125, 131, 136, Top level description, used where insufficient information to 139, 143, 144, 154, support attribution to one of its sub elements. 161, 163, 164 6.5.6.1 (ASP) Anticipate the results of each switching step 0 [18 box] before taking any actions 6.5.6.2 (ASP) Before operating a device, touch the devices 4, 12, 35, 45, 48, 93, identification label and verify that the correct device has 8 158, 166, [9 box] been selected 6.5.6.3 (ASP) Operate the device according to the switching 1 162 instructions 6.5.6.4 (ASP) Visually confirm that each operation has been 2 32, 72 completed 6.5.8 (ASP) Apply locks/tags to all isolation points 2 65, 83 6.5.11 (ASP) Notify the System Operator (Switching Controller) of 5 77, 101, 153, 162, 163 problems or abnormalities detected during switching 6.6.2 (ASP) Install grounds or grounding devices 1 81 6.7.1 (ASP) Update the Substation Log Book/Records, including description of work performed, any abnormal conditions found, 1 69 any abnormal conditions left Issue a Clearance Control Room Actions 1 7.5.1 (CCO) Log data on clearance issued, including location & 1 123 number of personal grounds placed by clearance holder

6.

7.

4-4

Results

Table 4-2 Tasks in which Errors Occurred (Continued) # 8. Abbreviated Task Statement # Events (Appendix B) Receive a Clearance Field Actions 7 8.1.5 (Work Crew) Conduct a Site Review of the existing conditions including locations of circuits and equipment, 2 160, 167 equipment grounding conductors & protective grounds 8.1.6 (Work Crew) Verify the adequacy of the clearance 8.3.1 (Work Crew) Conduct a Work Briefing with the Crew 2 84, 160 including work procedures & special precautions 8.3.2 (Work Crew) Inspect/prepare tools and safety equipment 1 74 8.4.1 (Work Crew) Test to verify that lines or equipment are de2 141, 160 energized. Release a Clearance Field Actions 14 9.1.1 (Work Crew) Verify that scheduled work has been 1 88 completed 9.1.2 (Work Crew) Verify system equipment returned to same 16, 21, 24, 25, 39,56, 11 configuration as found when accepting clearance 73. 75, 80, 82, 83 9.1.5 (Work Crew) Remove personal grounds installed by the 2 13, 116 crew CCO Accept the Release of a Clearance Control Room Actions 1 10.3.4 (CCO) Update all displays, logs and reports to indicate the 1 113 release of the clearance Release a Clearance in an Emergency 0 This situation not involved in any of the events in this report Implement Switching for Return of Lines or Equipment Control Room Actions 27 12.1.5 (CCO) assemble and review copies of outage request, 5, 121, 60, 62, 64, 87, 7 switching instructions, maps and one-lines 92 12.1.6 (CCO) Review available logs to determine if there have 51, 60, 62, 87, 92, been any recent changes or other conditions that may impact the 8 103, 106, 121 switching instructions 12.1.10 (CCO) Review the current status of equipment from all available sources (mapboard, SCADA, logs, etc.) to verify that 1 103 current status will not adversely impact the switching 12.1.11 (CCO) If there are any technical uncertainties regarding 2 103, 123 the switching instructions, do not proceed until they are resolved 12.3.3 (CCO) Verify any prerequisite switching/load transfer has 2 113, 155 been done 12.4.1 (CCO) Review scope of job with the Field Crew Leader 2 5, 42 (Switch person) before starting the job 12.5.6 (CCO) If there are uncertainties regarding the switching, 1 41 do not proceed until they are resolved 12.6.1.2 (CCO) Switch equipment via SCADA . . . Operate the 2 66, 122 device 12.6.5 (CCO) Monitor watt/var/volt/amp values while switching 1 39 instructions are executed 12.7.1 (CCO) Update changes to system status on SCADA and 1 87 other control room maps

9.

10.

11. 12.

4-5

Results

Table 4-2 Tasks in which Errors Occurred (Continued) Abbreviated Task Statement # Events (Appendix B) Implement Switching for Return of Lines or Equipment Field Actions 42 13.2.2 (ASP) Review switching to be done, including tags & 57, 126 2 grounds to be removed [59 box] 13.4.1 (ASP) Verify accuracy of instructions against a one-line 2 59, 134 and current conditions 13.4.2 (ASP) Review available logs to determine if there are any 26, 29, 43, 51, 59, 61, 8 conditions that may impact the scheduled work 64, 87 13, 17, 39, 51, 61, 64, 13.4.4 (ASP) Conduct a Site Review of the existing conditions 65, 69, 83, 87, 134 including locations of circuits & equipment, protective grounds, 11 [16 box, 116 box, 132 and equipment grounding conductors box] 13.4.6 (ASP) If field conditions do not agree with the outage 1 5 request and/or switching instructions, notify the System Operator 10, 15, 57, 83, 91, 92, 126, 129, 133, 133, 13.5.5 (ASP) Perform switching according to instructions 12 135, 137 13.5.5.3 (ASP) Operate the device according to the switching 1 22 instructions 13.5.5.4 (ASP) Visually confirm that each operation has been completed by verifying change in meters, by seeing a physical 1 43 closure, etc. 13.5.8 (ASP) Visually confirm operations have been completed 67, 69, 85 and that conditions as they should be after all switching [73 box, 82 box, 129 3 completed box] 13.6.1 (ASP) Notify the System Operator of problems during 1 126 switching or abnormal conditions at the site 14. Review and Close Outage Request 0 This situation not involved in any of the events in this report END of Table 267 total # 13.

Inspection of Table 4-2 reveals that multiple errors are often identified in a given incident report. It also is possible that multiple errors are involved in those cases where only a single error is cited. This is consistent with the generally well-defended nature of switching tasks. There are enough checks and balances that any single error will likely be detected before it can have an adverse result: a second error is needed to let the first one do its mischief.

Errors Attributed to Control Center Operators (CCOs)

The tasks in which CCO errors are the most common are: 3.1.2 (CCO) Identify all energy sources that could supply the line or equipment being removed from service. (8 incidents). 3.7.1.(CCO) Review documents for accuracy and completeness (9).

4-6

Results

4.1.5 (CCO) Review the outage request against maps and one-lines to verify the technical accuracy of the names of devices and the feasibility of the work to be accomplished (7). 5.5.1.2 (CCO) If equipment will be switched via SCADA control, operate the device. (13). This substep was the task in which inadvertent operations performed via SCADA, i.e., operation of a component other than the one intended, occurred. 5.6.1 (CCO) Update changes to system status on SCADA and other control room maps (5). 5.6.3 (CCO) Log all actions taken on records and reports (6). 12.1.5 (CCO) Assemble and review copies of outage request, switching instructions, maps and one-lines (7). 12.1.6 (CCO) Review available logs to determine if there have been any recent changes or other conditions that may impact the switching instructions (8).

Deficient performance of tasks 5.6.1 and 5.6.3 tended to occur in pairs: if the CCO didnt update all the maps and displays, he also failed to update all the other records. They contributed to events because the CCOs overseeing subsequent switching were misled by what they didnt see. Note that it was frequently the case that the first CCO recorded the information, but did not update all applicable records, and that the second CCO did not review all possible records as thoroughly as he or she should have, and thus missed the documentation that was updated. For example, two incidents involved cases where crews had, with the approval of the responsible CCO, placed grounds in addition to those called for in the original switching instructions. In one incident the off-going CCO recorded grounds in the log and on the permit, but failed to update the crib sheet, relief sheet, mapboard, or ECS. That is, the additional grounds were recorded, but not in all required locations. A second CCO directing the restoration failed to review all available information prior to ordering switching. The circuit was cut in with the grounds still on when the second CCO restoring the system failed to direct removal the grounds because he or she was unaware of the changes. Except for the selection of the wrong component when performing switching via SCADA, the most frequent CCOs errors (in fact vast majority of all the CCO errors) had to do with the production or use of documentation, i.e., the transmission of information.

Errors Attributed to Authorized Switching Persons (ASPs) in the Field

Field switching tasks in which errors most commonly occurred included: 6.4.1. (ASP) Verify the technical accuracy of the Outage Request and/or switching instructions against a one-line and current conditions (5). 6.4.4. (ASP) Conduct a Site Review of the existing conditions including locations of circuits and equipment, protective grounds, and equipment grounding conductors (8). 6.5.6 (ASP) Perform switching for isolation according to switching instructions (29). Note that this was the default entry for inadvertent operations; errors identified in this way were assigned to this task on the assumption that the plan for conducting the switching was adequate. This entry was not used when the report allowed identification of one of task 5.6.5s substeps, such as 6.5.6.2, below. 4-7

Results

6.5.6.2 (ASP) Before operating a device, touch the devices identification label and verify that the correct device has been selected (8). 13.4.2 (ASP) Review available logs to determine if there are any conditions that may impact the scheduled work (8). 13.4.4 (ASP) Conduct a Site Review of the existing conditions including locations of circuits & equipment, protective grounds, and equipment grounding conductors (11). 13.5.5 (ASP) Perform switching according to instructions (12). (Though worded slightly differently, this step is the equivalent of 6.5.6 for isolating equipment, and has an equivalent same set of substeps.)

By far the most common error was operating a piece of equipment other than intended (tasks 6.5.6, 6.5.6.2, and 13.5.5). This error was approximately three times more common in switching for removal from service than in switching for restoration. For the majority of the cases in which operation of the wrong equipment occurred (tasks 6.5.6 and 13.5.5), it is judged likely that the (omitted or inadequate) verification of the equipment or controls to be operated against the switching instructions was the source of the error. If this is so, then the most common errors attributed to the ASPs were also related to the appropriate use of documentation for collection or verification of information.

Errors Attributed to Work Crews

For incidents attributed to errors by work crews, the tasks most frequently associated with errors were those having to do with leaving equipment in its proper and expected configuration: 9.1.1 (Work Crew) Verify that scheduled work has been completed (i.e., whatever devices you took apart have been put back together) (2). 9.1.2 (Work Crew) Verify system equipment returned to same configuration as found when accepting clearance (10). 9.1.5 (Work Crew) Remove personal grounds installed by the crew (2).

These often result in traps waiting to be sprung when equipment is re-energized. We believe that most such traps can be detected by a diligent pre-switching walkdown of the equipment (task 13.4.4 in switching for restoration). This view is supported by the number of incidents in which the ASP was criticized for not performing such a review. Errors Occurring during Unscheduled Switching Eighteen of the 168 incidents listed in Appendix B involve unscheduled switching, four involving pole or line mounted distribution equipment, the other 12 involving equipment in substations. Four of these (incidents 6, 92, 99, and 167) involved changes or an existing plan during a job in progress. The others (incidents 2, 4, 8, 17, 102, 104, 105, 112, 119, 136, 153, 154, 158, and 168) were responses to trip-outs or equipment problems.

4-8

Results

The majority of identified deficiencies were in information gathering and planning tasks performed by the CCO involving use of documentation, though there were four cases of the most common of all errors, operating the wrong control. In five of the 18 incidents (28%) the utilitys report cited deficiencies in communications. This is almost twice the rate at which faulty communications are cited in reports that did not involve unplanned switching (23/150, or 15%).

Errors Attributed to Relay Technicians

In addition to the incident reports involving switching activities, the event descriptions supplied included 59 incidents involving relay technicians (RTs) engaged in tasks that were usually not addressed by the JTA. Because such errors are a significant source of operating incidents (about 25 % of the incident reports submitted), the RT-related incidents were tabulated and analyzed in Appendix E.

4-9

5
DISCUSSION
The errors recorded in the incident reports reviewed for this study did not appear to be randomly distributed among all the tasks involved in planned switching, but instead tended to cluster in a few tasks. There are many more incidents due to errors in the execution of switching than in its planning. However, this does not necessarily mean that errors are much more common in field activities. Several review steps are built into the process of planning and writing switching, so there are multiple opportunities to detect and correct errors in those activities before they can adversely affect the system and result in an incident report. Such corrected errors seldom if ever find their way into incident reports. This, and the fact they are generally not performed under time pressure, may be the reason they seldom appear to be factors in the incidents examined. The act of throwing a switch has fewer possibilities of interception after the fact, although a switch person can sometimes detect and correct errors by inspecting the switching after completing it but before reporting it completed. However, there is virtually no possibility of intercepting switching performed from the Control Center via SCADA after the command is entered. Errors occur almost three times more frequently in switching for removal of equipment than in switching for restoration: 130 in tasks 5 and 6, vs. 55 in tasks 12 and 13. The bulk of this difference is due to the many more operated wrong device errors in removing equipment from service (51) than in its restoration (13). We believe that this difference is probably due to the tags applied to out of service equipment, which provide an additional visual cue to identification that is lacking when taking equipment out of service. It may also be speculated that operators are more careful when energizing circuits than when de-energizing them. Communications remains a rich source of errors (explicitly mentioned in 28/168, or about 1 in 6 of all incidents). Where communication problems are identified, it is worthwhile to remember that although the task is identified with the ASP or the CCO, there is supposed to be a two-way exchange of information during such sessions. The ASP may tell the CCO one thing, and the CCO hear and write down (or repeat back) another. The ASP is not necessarily at fault if the CCO mis-hears him or her, though most utilities would say the field person bears primary responsibility if he or she does not pickup on the fact the repeat-back was not what was said. Some utilities would also hold the CCO responsible for allowing sloppy (imprecise or not-inaccordance-with-protocol) communications to occur when discussing switching activities. The data presented suggest that most incidents are due to errors in a few critical tasks: Selection of controls to be operated by operators in the field or Control Center when performing switching via SCADA. Operation of the wrong equipment by operators in the field is by far the most common error. 5-1

Discussion

Review of equipment status by operators in the field prior to beginning or resuming a job (referred to in the JTA as conducting a site review of existing conditions). Failure to perform these inspections is cited in many of the incident reports. While it is primarily intended to familiarize the operator with the equipment to be operated, an important function of this task is to detect pre-existing conditions that are unexpected or abnormal. Moreover it is an important safeguard against error or, more accurately, against suffering the consequences of earlier errors that have gone undetected until another change is made to the configuration of the system. Such reviews may be overlooked in the interest of saving time. Updating of all pertinent records. There were several instances where CCOs did not update all required displays/documents, thus leaving the CCO to dispatch the next portion of the job with an inaccurate understanding of the status or configuration of the system. Thorough review of all pertinent records before beginning or resuming a job, meaning in this case reviewing the records of switching performed for isolation prior to beginning the restoration-to-service phase of a scheduled outage. Both CCOs and ASPs should do this routinely.

The following additional observations were made in the course of reviewing the incident reports: Modifications to a pre-approved plan appeared to be a relatively common source of incidents. In three cases this was because the effects of changes were not adequately investigated before they were made (perhaps due to time pressure from a job temporarily on hold). However, the more common problems seemed to be failure to adequately document such changes, or to review those documents in which they were recorded. In emergency switching, gross errors, e.g., misidentification of circuits or stations, seem to be more common than relatively smaller errors (e.g., misidentification of equipment on the correct circuit). Review of changes made during the review process appeared to be problematic in a few cases. If a reviewer makes a change, he or she becomes a co-author of the program. Does yet another truly independent person review the corrections made? Most peoples perceptions are strongly conditioned by their expectations. That is, people often to see what they expect (or want) to see, and fail to notice conditions that they are not explicitly looking for There are reports of crews buzzing lines who attributed indication of voltage to induction (whose dangers may not be properly appreciated) or Static, not the line being still energized. And in inspections, people see what they were told to see. In one incident a breaker was checked closed on three separate occasions before the incorrectly positioned disconnects were discovered. In spite of a near-universal requirement to stop everything and contact the dispatcher if an error is discovered (or more generally if anything is not as expected), ASPs frequently attempt to correct an error before reporting it.

5-2

6
CONCLUSIONS
This study allows us to draw several conclusions beyond the specific results cited in the last section. First, we believe that identifying the tasks associated with errors identified in incident reports is both possible and potentially rewarding, in that it can identify tasks that seem to be the focus of many errors. The analysis in this report demonstrates that incident data can be used with the generic JTA to identify error-likely tasks where continuing training should be given, job aids provided, or additional safeguards built into the routines by which the task is performed. Second, review against operating experience (i.e., incident data) can also be useful for the maintenance and updating of the task analysis, in that it can help identify the appropriate level of detail or additional tasks to be analyzed and trained. Rev 1 (2004) of the EPRI JTA Database incorporates a number of additions and revisions that were identified in an early draft of the present report. Third, for the present analyses, a detailed and fairly extensive narrative of the incident was the most useful part of the report for making task assignments. Due to the large number of reports, no attempt was made to gather additional information on individual incidents. The frequent paucity of detail and the numerous ambiguities in the reports used (and those discarded) does, however, reinforce the importance of detailed and complete reporting if it is anticipated that the reports may be used for additional analyses at a later date. Finally, many of the assumptions and uncertainties in this analysis would be avoided in an inhouse analysis using utility-specific task analyses appropriate to all tasks involved in the work. Interpreting of reports would also involve less guesswork for an in-house analyst who was familiar with company work practices and the intended meaning of terms used in their incident reports. And of course an analyst working with his own companys reports is in an excellent position to obtain any clarification required. The conservative approach used in the analysis reported here was adopted because of the impracticality of obtaining additional information. If the analyst can talk to the investigators or the parties involved in an incident, it is likely that more omitted or improperly performed tasks can be identified.

6-1

7
REFERENCES
1. Generic Job and Task Analysis Handling Planned Outages and Hot Line Work Requests, Electric Power Research institute, Palo Alto, CA: September 2002. TR-1001789. 2. Generic Job and Task Analysis Database, to be published by EPRI in the fall of 2004.

7-1

A
LIST OF CONTRIBUTING UTILITIES
Utilities Contributing Incident Reports for this Study
American Electric Power BGE Consolidated Edison Company Duke Energy HydroOne Tennessee Valley Authority United Illuminating Western Area Power Administration

A-1

B
LISTING OF HUMAN ERROR EVENTS
The incidents reviewed for this report are summarized in the table on the following pages, which gives the event number, a brief synopsis of the incident, the personnel involved, and the tasks in which the errors causing or contributing to the incident occurred. To conserve space in the table, a rather telegraphic style has been used to describe the incidents. For the task assignments, the analysis has tried to stick closely to the utilitys reports. Where an inference as to the task(s) involved has been made, the reasons for the task assignment should be reasonably clear from the description in the Errors and Contributing Factors column, although the descriptions of the incidents are frequently rather sketchy. For example, if the as-found condition was incorrect, the condition has been attributed to an error on the part of the last people in contact with the equipment. In many cases where the reports contain little detail, the wording inadvertently opened and inadvertently closed has led to a default classification of task 6.5.6 (ASP) Perform switching for isolation according to switching instructions & visually confirm that each operation has been completed. Note that errors in multiple tasks are often identified in connection with a single incident report. It is also likely that that multiple errors are involved in those cases where only a single error is cited. Generic job titles are used in the table: ASP = authorized switching person. In the JTA, this term covers a range of job titles, such as substation technician, lineman, service operator, etc. In the table in this appendix the term is occasionally used when the title of the person performing the switching in the field is unknown, on the assumption that members of a work crew may be authorized to perform switching as necessary to perform their jobs. Although there a number of reports in which the as-found position of switches was incorrect, no incident report used in this study indicated that switching had been performed by a person not authorized to do so. CCO = Control Center Operator (system operator) whether bulk power or distribution. RT = Relay Technician. Work Crew = crew performing work other than switching under clearance, or other authorized personnel. In some cases the work crews also perform switching.

Speculations about likely errors not mentioned by the utilitys report, or tasks that might have prevented those errors that did occur, are presented in shaded boxes: the tasks mentioned in these speculations not counted in the tabulations presented in the body of this report. B-1

Listing of Human Error Events

The term unscheduled switching has been used for cases where there was no formal outage request. This term covers emergency switching, trip response, call out, or alarm response. In some cases substation names and other nomenclature have been changed to avoid identifying the contributing utility.

B-2

Listing of Human Error Events Event # 1 2 Errors and Contributing Factors CCO inadvertently opened 7G-7T-2 OCB rather than 7G-1T-2 OCB as intended. Responding to feeder trip-out. Tripped circuit incorrectly identified by CCO. Second CCOs own home was among those de-energized & his wife was the first to call in the outage. Assumed that second CCO knew which circuit his home was on. Failed to confirm on circuit maps that the circuit tentatively identified was the one feeding the people calling in to report loss of service. Switching sequence determined & switching orders written in advance. CCO who dispatched job did not attend planning meetings. When orders faxed to field, field ASP requested change in order in which his part of the work was to be performed. CCO agreed to requested change without checking out possible consequences. CCO criticized for not taking time to thoroughly familiarize himself with the work before beginning it. Initial tailboard with ASP also faulted, in that the importance of the sequence not made clear. ASP responding to outage accidentally opened wrong ABS switch. Checked switch on pole 885, fault indicators showed normal. Dispatched to second location, then directed back to pole 885 to open switch. ASP went to pole 882 and closed that switch. ASP criticized for not verifying pole # and switch position. Personnel CCO CCO Unscheduled Task in which Error Occurred (From TA) 5.5.1.2 (CCO) If equipment will be switched via SCADA control, operate the device 3.2.1 (CCO) Research, document, and verify work location and equipment to be returned to service

CCO Change requested

5.1.1 (CCO) Assemble & review copies of Outage Request, switching instructions, maps, and one-lines 5.2.3 (CCO) Review the work to be done (big picture) with Field Crew Leader or Switch person 5.4.2.2 (CCO) If field conditions are such that a change must be made to the switching instructions . . . Research proposed changes to verify that they are appropriate and will not cause additional problems

ASP Unscheduled

6.4.2 (ASP) Verify devices to be switched are in the expected state before beginning switching 6.5.6.2 (ASP) Before operating a device, touch the devices identification label and verify that the correct device has been selected

B-3

Listing of Human Error Events Event # 5 Errors and Contributing Factors In preparing written switching order, CCO omitted step to remove tagged grounds on circuit prior to closing CB. Not caught by independent checker, nor at meeting of all concerned parties (who made multiple changes to switching order) nor by dispatching CCO. Omission was detected by SS crew leader and corrected before CB closed, avoiding grounding error. Report notes that A detailed and complete pre-switching review of the switching order was not conducted by the System Operations Dispatcher, the Person Responsible for Work, or by the Substation Crew performing the switching. Utility was displeased that field crew leader waited until he was at the affected step to report the condition, rather than reporting it as soon as he discovered it. Personnel CCOs, ASP Grounds Task in which Error Occurred (From TA) Multiple modifications to switching plan 3.5.1 (CCO) Write switching instructions for restoration in sequence, including re-enabling of devices disabled during switching for removal from service 3.2.5 (CCO) Identify ground devices to be removed 3.7.1.(CCO - several reviewers in this case) Review documents for accuracy and completeness 12.1.5 (CCO) assemble and review copies of outage request, switching instructions, maps and one-lines 12.4.1 (CCO) Review scope of job with the Field Crew Leader (Switch person) before starting the job 13.4.6 (ASP) If field conditions do not agree with the Outage Request and/or switching instructions, notify the System Operator (Switching Controller) 6 Emergency load swap was necessitated when planned switching resulted in an overload condition. Dropped part of load when failed to recognize one N.O. switch was open (& so marked on the map), and assumed a second switch was closed. Discussion of necessity of through checking, even with emergency switching plans. CCOs Unscheduled 3.5.1 (CCO) Write switching instructions for restoration in sequence, including re-enabling of devices disabled during switching for removal from service Possible also that the following was not done 4.2.3 Determine effect of planned procedure to local and interconnected power systems given current system conditions CCO 5.5.1.2 (CCO) If equipment will be switched via SCADA control, operate the device 5.5.6 (CCO) Place tags on devices on SCADA and other control room maps CCO, ASP Unscheduled 3.1.2 (CCO) Identify all energy sources that could supply the line or equipment being removed from service Verification error on part of ASP: 6.4.4 (ASP) Conduct a Site Review of the existing conditions including locations of circuits and equipment, protective grounds, and equipment grounding conductors

CCO inadvertently opened unintended CB via SCADA during planned switching. Adjacent switches on SCADA one-line. Criticized for not doing self-verification. Also placed SCADA tag on correct CB, then operated the wrong one. Write-up says he should have operated CB and then tagged it. Emergency switching. Car downed poll. Dispatch sent troubleman to pole 5296, said taps would de-energize eastwest line to downed pole. Trouble shooter confirmed & pulls taps. Downed pole still energized. Sent back to pole 5296. Taps open but control N-S line, not E-W. Dispatcher identified wrong equipment. Additional switching to de-energize downed pole.

B-4

Listing of Human Error Events Event # 9 Errors and Contributing Factors ASPs were at the station to clear the East and West lines from SS to customer. ASPs were following a pre-approved procedure from the CC. When the step to swap the SS (Tap) to the West line the ASP turned the 43S switch to the East To West position. This caused the station to swap to the West line. The 83S switch should have been turned to West Preferred. CCO received a report that a raccoon was on a line structure on the line between CSS and Tap. Line switching was performed to move the feed and de-energize the line from CSS (line CB open) to Tap (line disconnects toward CSS open). During the switching to restore line between CSS and Tap, a set of line disconnects opened earlier to provide clearance were not closed back by ASP in the field when returning the line to normal, resulting in 5 minute outage to customer. ASP was at station to clear up circuit 1203 so that the Sel 251 relay could be changed out. Tech zeroed the regulator and closed the transfer bus disconnect. He then opened the first disconnect, an arc occurred. ASP was thinking that this station had a straight bypass bus, but station has a transfer bus. Personnel ASP Task in which Error Occurred (From TA) 6.5.6 (ASP) Perform switching for isolation according to switching instructions Had correct procedure but operated wrong switch: likely a verification error 6.5.6.2 (ASP) Before operating a device, touch the devices identification label and verify that the correct device has been selected ASP, CCO 13.5.5 (ASP) Perform switching according to instructions

10

11

ASP

If unaware of station configuration, obviously did not perform 6.4.1 (ASP) Verify the technical accuracy of the switching instructions against a one-line and current conditions And also did not have correct set of instructions? (although switching should have been performed according to written instructions, report did not criticize ASP for not following instructions)

12

ASP was sent to station to clear up circuit 12-11, so that the relaying could be changed out. ASP blocked the bank differential and proceeded to the yard to complete the task. He closed the by-pass disconnects on breaker 12-11 and proceeded to open the line side disconnects on the circuit. When he opened the second disconnect the Bank #3 differential operated interrupting the load on Bank #3. ASP had blocked Bank #2 differential instead of Bank #3. A correct switching procedure was being used.

ASP

Somehow lost his place in the task sequence or the station? 6.5.6.2 (ASP) Before operating a device, touch the devices identification label and verify that the correct device has been selected

B-5

Listing of Human Error Events Event # 13 Errors and Contributing Factors ASPs had been transferring the 100kV breakers from the Temporary B Bus back to the Normal A Bus at the station. They had completed all the transfers with the exception of the W-1 CB and the 100kV Capacitor CB. The normal A bus differential had been blocked so that the CTs could be checked for correct polarity. When ASP closed the A bus gang on the W-1 CB, an explosion occurred, knocking both ASPs to the ground. Safety grounds on the 100kV B bus had been inadvertently energized. A breaker crew was changing bushings on Bank 4 LT breaker and a transformer crew was changing bad U bushings on Bank 4. The breaker crew had completed changing the bushings and was manually jacking the breaker closed, checking their clearances. When the auxiliary switch (fingers) changed position, the station Bank HT tripped interrupting the station. Bank 4 has a Bucholtz type gas pressure relay: the contacts make up when oil level is approx. 2 inches below the relay. The transformer crew had removed oil from the bank causing the contacts to make. When breaker crew made the a finger, the trip circuit was completed and the 94T-4 relay operated tripping the station Bk. HT. The ASP had failed to block the 63GP relay. Personnel ASP & Work crew grounding Task in which Error Occurred (From TA) 9.1.5 (Work Crew) Remove personal grounds installed by the crew The following task assignment assumes ASPs failed to inspect for grounds before beginning the switching: 13.4.4 (ASP) Conduct a Site Review of the existing conditions including locations of circuits & equipment, protective grounds, and equipment grounding conductors ASP 6.5.6 (ASP) Perform switching for isolation according to switching instructions Above assumes blocking relay was a part of the instructions

14

B-6

Listing of Human Error Events Event # 15 Errors and Contributing Factors At a new distribution station, load had been placed on circuit 24-10 the day before, and the regulators placed in auto. The ASP had told distribution that an RT would need to be present before load could be placed on the other circuits. Distribution made up the stress cones in the underground vault and wanted to phase circuits. The ASP closed the disconnects and noticed the regulators were on neutral. The Tech charged circuit 24-09 to the underground vault. Distribution phased circuits and told the ASP they were ready to tie the circuits together and pick up load. The ASP blocked the ground relays (in the control house) on circuits 24-09 and 24-10. Distribution tied circuits together and after about twenty seconds both circuits 24-10 and 24-09 tripped out by instantaneous relays. The 24-10 regulator was on L-5 and the 24-09 regulators were on neutral. Failure to have regulators on same step caused circulating current to trip circuits out. Doble crew was in station to Doble circuit 1201. Upon completion of the Doble test, the technicians conferred with each other about the results of the test. It was determined to return the OCB to service. The tech that had been working on 1201 OCB began to return the OCB to service. The second tech observed the ASP doing the switching but did not look at 1201 OCB. The second tech had been away from the OCB looking at the test results and failed to reconnect the leads on the 1201 regulator. When the ASP closed the first disconnect the lead arced to ground and the HT tripped, interrupting the station. ASP responding to callout arrived at the station and found Bank # 2 had tripped by 87T relay. Tech checked relay targets in control house, went to yard, looked at the bank and checked on all capacitor alarms. ASP then went to 12-9 CB and 12-10 CB and tripped the circuits. ASP failed to realize that a bank transfer had taken place and circuits 12-9 and 1210 were carrying load. Personnel ASP Task in which Error Occurred (From TA) Unclear whether ASP failed to perform a step in a written switching instruction or work plan, or was this something that was considered skill-of-the-craft and that he was supposed to know to do. Report does not say step omitted 13.5.5 (ASP) Perform switching according to switching instructions Report does say ASP noted status of regulators (meaning position?): It is possible the 24-10 had changed since he checked them (they were on auto), so his real error was failure to protect himself from this possibility by placing in manual and then to neutral

16

Work Crew, ASP ?

9.1.2 (Work Crew) Verify system equipment returned to same configuration as found when accepting clearance Possible also that ASP failed to inspect station equipment to determine actual status prior to operating it: 13.4.4 (ASP restoration) Conduct a Site Review of the existing conditions including locations of circuits & equipment, protective grounds, and equipment grounding conductors

17

ASP Unscheduled

ASP failed to inspect station equipment to determine actual status prior to operating it: 13.4.4 (ASP) Conduct a Site Review of the existing conditions including locations of circuits & equipment, protective grounds, and equipment grounding conductors

B-7

Listing of Human Error Events Event # 18 Errors and Contributing Factors The 44kV bus at Steam Plant was interrupted when an ASP performed switching to clear AT-2 transformer for a tap change and maintenance. After having cleared AT-2, the ASP was grounding several pot devices located near the area of clearance. He grounded one 44kV pot connected on the transformer side of the 44kV low-tension breaker 4-02. He closed the grounding switches to two pot devices located on the line side of 4-02 breaker, which operated the undervoltage relay causing the breaker feeding the 44kV bus to trip. Personnel ASP Task in which Error Occurred (From TA) 6.4.4 (ASP switching out) Conduct a Site Review of the existing conditions including locations of circuits and equipment, protective grounds, and equipment grounding conductors Report hints this skill of the craft rather than step by step instructions 6.5.6.1 (ASP) Anticipate the results of each switching step before taking any actions Possibly due to 6.4.1 (ASP) Verify the technical accuracy of the switching instructions against a one-line and current conditions 19 ASP was at station to perform a tap change on the transformers. ASP checked with customer to make sure he had dropped his load. ASP checked load in meter test block to make sure customer had dropped the entire load. ASP had CCO block the recloser on the # 1 line out of the station. ASP tripped AB Switch and it flashed, tripping the #1 line. ASP checked station and found he had tripped the AB switch with the capacitor breaker closed. No switching procedure was used. Suggests ASP unsure of status of capacitor breaker 20 ASP was in the process of clearing bus section #2 to repair the bus disconnects on circuit 12-10. When he de-energized the bus section #2 meter pots, which were the voltage source for the transfer relay, the bank transferred to bus section #1. ASP failed to block transfer cut-out switch, which would have blocked the transfer. The ASP had a correct procedure but deviated from it. ASP Intentional shortcut: ASP did not think step was necessary 6.5.6 (ASP) Perform switching for isolation according to switching instructions ASP NOTE: ASPs expected to write own switching procedure for simple jobs & follow it, initial error was: 6.4.4 (ASP switching out) Conduct a Site Review of the existing conditions including locations of circuits and equipment, protective grounds, and equipment grounding conductors 6.5.6 (ASP) Perform switching for isolation according to switching instructions

B-8

Listing of Human Error Events Event # 21 Errors and Contributing Factors Circuits 1203 and 1204 were tied together for the purpose of Doble testing the breakers. Circuit 1203 was cleared and the leads lifted so that it could be Doble tested. The task was completed and the leads were reconnected on the breaker. The ASP closed the disconnects on the line and bus side and 1203 was closed by control handle. Immediately circuits 1203 and 1204 tripped. The ASP closed 1204 back, picking up the customers on 1203 and 1204. An inspection of the breaker determined that the leads on the breaker for circuit 1203 had been crossed. During routine relay test, while using a correct pre-printed switching procedure, load was interrupted on circuits 24-1, 24-2 and 24-4. A step on the switching procedure to close Bank 5 CB was marked through but not accomplished. The bus tie CB was then opened, interrupting the load. The ASP stated that he stopped in the middle of the switching sequence to get a drink of water just prior to tripping the bus tie circuit breaker. Failure to cut out circuit before grounding it. ASP/CCO miscommunication cited, report noted multiple CCOs involved in outage, implied too many. ASP reported finding master stuck breaker switch open & cover on floor: apparent failure to restore properly at conclusion of previous work. ASP reports found trip alarm bypassed in cubicle: apparent failure to restore properly at conclusion of previous work. ASP reports found tap changer motor control switch turned off and breaker maintenance switches in abnormal position. Apparent failure to restore properly at conclusion of previous work, BUT was noted in log, which second ASP failed to consult when restoring. Possibly not properly tagged. Also cabinet markings re switches & fuses not per current standards. ASP bumped Aux Trip panel relay, causing GCB to operate. Personnel Work Crew Task in which Error Occurred (From TA) 9.1.2 (Work Crew) Verify system equipment returned to same configuration as found when accepting clearance

22

ASP

13.5.5.3 (ASP) Operate the device according to the switching instructions

23

ASP, CCO Grounds Comms

5.1.4 (CCO) Verify any prerequisite switching/load transfer has been done 5.2.3 (CCO) Review the work to be done (big picture) with Switch person before starting the job 9.1.2 (Work Crew) Verify system equipment returned to same configuration as found when accepting clearance 9.1.2 (Work Crew) Verify system equipment returned to same configuration as found when accepting clearance 13.4.2 (ASP) Review available logs to determine if there are any conditions that may impact the scheduled work

24

Work Crew

25 26

Work Crew ASP

27

ASP

Not addressed by Task Analysis

B-9

Listing of Human Error Events Event # 28 29 Errors and Contributing Factors ASP inadvertently opened feeder bus breaker. ASP opened switch inadvertently during testing following repairs to station: CCO wrongly informed ASP that operating mechanism was disengaged; ASP did not check status in field. On previous day, order to disengage mechanism could not be executed & was canceled, however status board showing mechanism disengaged was not revised by CCO to reflect true status. CCO controlling job was misled by status board. 13 kV bus inadvertently energized. Blamed largely on poor communications between 2 CCOs involved second SO directed ASP to put automatics back to normal, which closed breaker: utility recommended that CCO who starts a job should finish it rather than passing it off. ASP accidentally brushed Sup Trip Aux Relay, causing it to operate and open feeder, which reclosed automatically. Operating mechanism for MOD not properly disengaged by ASP preparing station for work: failure in execution. Possible ASP knowledge deficiency, too. Tripped during (remote) switching operations when transferring feeder load to aux bus because feeder regulators blocked but not placed in neutral position as required. Execution failure by crew preparing station for performance of relay checks. Reclosers properly opened at 3AM for work but not recorded in log by CCO due to distraction. Around noon ASP sent to check recloser status reported them open & asked if CCO wanted them on: CCO said yes. Multiple errors: initial failure of CCO to log status, plus failure to establish common big picture between 2 CCOs & ASP, plus failure to follow communication protocol. Felt having more than one CCO involved may also have contributed. Personnel ASP ASP, CCO Task in which Error Occurred (From TA) 6.5.6 (ASP) Perform switching for isolation according to switching instructions 13.4.2 (ASP) (ASP) Review available logs to determine if there are any conditions that may impact the scheduled work 5.6.1 (CCO) Update changes to system status on SCADA and other control room maps

30

CCO Comms

5.6.5 (CCO) If switching has not been completed before the end of the shift, review the status of the job with the oncoming shift

31 32

ASP ASP

Simple slip: Not addressed by TA 6.5.6.4 (ASP) Visually confirm that each operation has been completed 6.5.6 (ASP) Perform switching for isolation according to switching instructions

33

ASP

34

CCO Comms

5.6.3 (CCO) Log all actions taken on records and reports 5.2.3 (CCO) Review the work to be done (big picture) with Field Crew Leader or Switch person before starting the job

B-10

Listing of Human Error Events Event # 35 Errors and Contributing Factors ASP inadvertently opened pothead disconnects rather than feeder disconnects. Error in execution. ASP had combined verbal/written switching order: did verbal part OK but skipped first step in written order. All recorded communications OK but ASP criticized for using outside (unrecorded) telephone to report incident to CCO. Error in execution. ASP failed to place relay control switch from Auto to Manual, resulting in tripping of bus. Utility emphasized importance of ASP-CCO communications re feeder automatics in its report. Feeder tripped by relay when cubicle door closed too vigorously: ASP criticized for not following proper closing procedures for doors with relays attached. In restoration from an outage, feeder was transferred to tie: after about 8 minutes, the feeder was determined to be dead. Maintenance disconnect found open: ASP restoring station failed to close it. Blamed on poor CCO/ASP communications plus fact that ASP failed to check all associated equipment before beginning switching operations. CCO failed to observe loss of load promptly due to pace of operations. CCO inadvertently opened tie air break switch. He selected SCADA operated air break to check its nomenclature then a non-SCADA ground switch to change its status. Due to a peculiarity in the way the computer operates, his change of status input operated the last SCADA operated switch selected: CCO cited for lack of knowledge /failure to follow procedures for operation of SCADA system. Feeder outage prolonged when CCOs assumed alarms coming from station were generated by ASP working in station. ASP was in a location where he could not see/hear alarms. CCOs counseled to be more questioning. Personnel ASP Task in which Error Occurred (From TA) 6.5.6.2 (ASP) Before operating a device, touch the devices identification label and verify that the correct device has been selected. 6.5.6 (ASP) Perform switching for isolation according to switching instructions

36

ASP

37

ASP Comms

5.2.3 (CCO) Review the work to be done (big picture) with Field Crew Leader or Switch person before starting the job

38

ASP

Not addressed by TA

39

ASP, CCO Comms

9.1.2 (Work Crew) Verify system equipment returned to same configuration as found when accepting clearance 13.4.4 (ASP) Conduct a Site Review of the existing conditions including locations of circuits & equipment, protective grounds, and equipment grounding conductors 12.6.5 (CCO) Monitor watt/var/volt/amp values while switching instructions are executed

40

CCO

This is something of a forced fit 5.5.1.2 (CCO) If equipment will be switched via SCADA control, operate the device

41

CCO

12.5.6 (CCO) If there are uncertainties regarding the switching, do not proceed until they are resolved

B-11

Listing of Human Error Events Event # 42 Errors and Contributing Factors Feeder inadvertently energized following pole fire. CCO-D requested CCO-T that distribution feeder on affected pole was not to be cut back in. CCO-T did not communicate this to ASP, who proceeded to restore the bus, energizing the feeder. Report confusing on who is considered responsible for incident: also complicated by errors in substation one-lines & bulk power board, unusual station configuration, and lack of control/representation on SCADA system. Circuit switcher opened earlier in day: 2 phases did not reset as they should have by design, and when ASP closed circuit only phase B closed. ASP checked voltmeter meter which happened to be set to phase B & everything looked OK. ASP criticized for failure to verify condition of all equipment before switching. CCO-T inadvertently opened wrong feeder while switching transformer out of service No details given. ASP inadvertently operated OCB and tripped feeder: Grabbed OCB switch handle instead of cubicle door handle. Simple slip? ASP inadvertently Cut out feeder 13917 vice 13907 no additional details. Tripped feeder due to transfer switch being left in Automatic position: interesting because ASP & CCO-T discussed prior to switching. CCO-T correctly stated that it not necessary to place switch in OFF for SCADA switching (because SCADA overrides the automatic transfer scheme). However, switch does have to be placed to OFF to defeat transfer when switching from local panels. ASP opened 33782 breaker instead of 33780. Had just finished discussing next job with CCO, which involved 33782 breaker. Personnel CCOs, ASP Task in which Error Occurred (From TA) Report unclear as to whether the CCO-Ts request not to restore the feeder was a last minute change to a written switching plan 12.4.1 (CCO) Review scope of job with the Field Crew Leader (Switch person) before starting the job

43

ASP

13.5.5.4 (ASP) Visually confirm that each operation has been completed by verifying change in meters, by seeing a physical closure, etc 13.4.2 (ASP) Review available logs to determine if there are any conditions that may impact the scheduled work

44 45

CCO ASP

5.5.1.2 (CCO) If equipment will be switched via SCADA control, operate the device (This is a bit of a stretch.) 6.5.6.2 (ASP) Before operating a device, touch the devices identification label and verify that the correct device has been selected

46

ASP

6.5.6 (ASP) Perform switching for isolation according to switching instructions & visually confirm that each operation has been completed 6.2.1 (ASP) Verify job information with the System Operator, including identification of all automatically and remotely controlled switches that could cause opened devices to close

47

ASP, CCO

48

ASP

6.5.6.2 (ASP) Before operating a device, touch the devices identification label and verify that the correct device has been selected

B-12

Listing of Human Error Events Event # 49 Errors and Contributing Factors ASP inadvertently cut out network feeder while transferring feeder to tie. While grounding Bus OCB, ASP inadvertently tripped tie OCB. Transformer tripped when energized with grounds still applied. CCO decided to ground transformer even though grounding was not on outage request. Entered in electronic log but not hard copy, mapboard, ECS station configuration, or relief sheet. Second CCO checked hard copy & boards, but not electronic copies. ASP did not check station log or review system prior to beginning switching. Multiple errors. Personnel ASP Task in which Error Occurred (From TA) 6.5.6 (ASP) Perform switching for isolation according to switching instructions & visually confirm that each operation has been completed 6.5.6 (ASP) Perform switching for isolation according to switching instructions & visually confirm that each operation has been completed 5.6.1 (CCO) Update changes to system status on SCADA and other control room maps 5.6.3 (CCO) Log all actions taken on records and reports 12.1.6 (CCO) Review available logs to determine if there have been any recent changes or other conditions that may impact the switching instructions 13.4.2 (ASP) Review available logs to determine if there are any conditions that may impact the scheduled work 13.4.4 (ASP) Conduct a Site Review of the existing conditions including locations of circuits & equipment, protective grounds, and equipment grounding conductors 6.5.6 (ASP) Perform switching for isolation according to switching instructions 5.2.3 (CCO) Review the work to be done (big picture) with Field Crew Leader or Switch person before starting the job 5.3.6 (CCO) If there are uncertainties regarding the switching, do not proceed until they are resolved 5.1.1 (CCO) Assemble & review copies of Outage Request, switching instructions, maps, and one-lines 6.4.1 (ASP) Verify the technical accuracy of the switching instructions against a one-line and current conditions 5.5.2 (CCO) Direct Field Crew Leader (Switch person) to proceed with switching and tagging, and grounding according to the written instructions 6.4.4 (the RT) Conduct a Site Review of the existing conditions including locations of circuits and equipment, protective grounds, and equipment grounding conductors

50

ASP

51

CCOs, ASP Grounds

52 53

Transformer not cleared up during planned switching. Inadvertently isolated wrong equipment: ASP used different nomenclature in describing what he did than that used by CCO in ordering switching, but CCO did not pick up on discrepancy. Second error when CCO ordered wrong equipment operated & order complied with as given. Both criticized for failing to review station configuration before beginning. Third error when RT arrived at station & said he accepted station conditions without prior inspection. On his walkdown, this RT later found & reported the discrepant conditions.

ASP ASP, CCO, & RT Comms

B-13

Listing of Human Error Events Event # 54 Errors and Contributing Factors ASP at one station failed to properly clean up transformer (execution error?). ASP at second station told to verify no load before cutting out circuit. ASP reported 25 amps of load. Believing circuit was isolated, CCO attributed to induction & proceeded with switching: loads transferred automatically when loaded circuit opened. At least two errors here, AND second ASP should have called CCO on his decision to go ahead rather than further investigating the source of the indicated load. CCO-T misunderstood CCO-D plan, dispatched ASP to station to cut out feeders before CCO-D had reconfigured feeders outside of station to pick up load. Did not check with CCO-D that his part of plan had been performed prior to ordering switching. Miscommunication & assumptions cited, also criticized for face-to face discussion of plans rather than using (recorded) intercom system. Relay operated because rain entered cabinet through door that had not been properly secured: Failure to restore properly at end of previous job. ASP failed to close C phase feeder disconnect after wildlife proofing job. Switch person who started job agreed to stay over with crew on next shift to complete it. Near the end he was tired & he & checker exchanged roles at the breaker bay where omission occurred. Blamed on interruption & failure to review job & refocus after roles switched. Automatic changeover operated when CCO opened transformer high side circuit switcher via SCADA. Failed to realize switchover scheme was left in automatic when it should have been disabled. CCO & ASP both criticized for not properly identifying station conditions prior to beginning switching (e.g., review of station operating instructions). Personnel CCO, ASP Task in which Error Occurred (From TA) 6.5.6 (ASP) Perform switching for isolation according to switching instructions 5.3.6 (CCO) If there are uncertainties regarding the switching, do not proceed until they are resolved

55

CCOs Comms

5.1.4 (CCO-T) Verify any prerequisite switching/load transfer has been done

56

Work Crew

9.1.2 (Work Crew) Verify system equipment returned to same configuration as found when accepting clearance 13.5.5 (ASP) Perform switching according to instructions. 13.2.2 (ASP) Review switching to be done, including tags & grounds to be removed

57

ASP

58

CCO, ASP

5.2.3 (CCO) Review the work to be done (big picture) with Field Crew Leader or Switch person before starting the job 6.4.3 (ASP) Review available logs to determine if there are any conditions that may impact the scheduled work 6.4.4 (ASP) Conduct a Site Review of the existing conditions including locations of circuits and equipment, protective grounds, and equipment grounding conductors

B-14

Listing of Human Error Events Event # 59 Errors and Contributing Factors Feeder tripped when cut in because ASP failed to remove set of portable grounds. Station configuration unfamiliar to ASP, who also assumed that grounds were applied in bay area only as indicated on the original grounding order. However, record of grounds actually applied was on back of sheet, & in station log. Should have reviewed these & one-lines before beginning work, also should have checked off grounds as he removed them. Personnel ASP Grounds Task in which Error Occurred (From TA) 13.4.1 (ASP) Verify accuracy of instructions against a oneline and current conditions 13.4.2 (ASP) Review available logs to determine if there are any conditions that may impact the scheduled work This is the second one of these where ASP was blindsided by grounds that were NOT a part of the original switching order. You have to wonder if he and CCO did 13.2.2 (Review switching to be done, including tags & grounds to be removed) CCO Grounds 5.6.3 (CCO #1) Log all actions taken on records and reports. 12.1.5 (CCO #2) assemble and review copies of outage request, switching instructions, maps and one-lines. 12.1.6 (CCO #2) Review available logs to determine if there have been any recent changes or other conditions that may impact the switching instructions ASP 13.4.2 (ASP) Review available logs to determine if there are any conditions that may impact the scheduled work 13.4.4 (ASP) Conduct a Site Review of the existing conditions including locations of circuits & equipment, protective grounds, and equipment grounding conductors CCOs Grounds 5.6.1 (CCO #1) Update changes to system status on SCADA and other control room maps 5.6.3 (CCO #1) Log all actions taken on records and reports 12.1.5 (CCO #2) assemble and review copies of outage request, switching instructions, maps and one-lines 12.1.6 (CCO #2) Review available logs to determine if there have been any recent changes or other conditions that may impact the switching instructions

60

Circuit cut in with grounds inadvertently left on. CCO did not notice that grounds had been applied at a second station: was recorded in log, permit, mapbord, & ECS, but not recorded in the operators relief sheet. CCO criticized for assuming this outage the same as previous outages on same circuit, & not reviewing multiple sources to understand the switching to be done. Previous CCO partly at fault for failing to note condition on relief sheet. ATB left closed in the maintenance position. ASP replaced another ASP who had to be taken to hospital in middle of job. Abnormal condition was entered in log book. Switches normally closed only if breaker is open & grounded on both sides, since this was not the case, second ASP assumed switches were in normal position & did not check them. Circuit cut in with grounds on at another station. Off-going CCO recorded grounds in log and on permit, but failed to update crib sheet, relief sheet, mapboard, or ECS. Second CCO failed to review all available info prior to ordering switching.

61

62

B-15

Listing of Human Error Events Event # 63 Errors and Contributing Factors Manual cutout station service switch incorrectly placed in emergency position, deenergizing the oil pumps on the transformers: loss of power to pumps initiates opening of low side ACBs. Crew apparently tried to combine their trips from control house to west yard. However, equipment trips initiated before they got there. Design of station also mentioned as contributing factor in report. Feeder tripped when reenergized due to set of portable grounds left in place. CCO criticized for failure to review all sources of info (grounds were indicated in CC) & failure to properly indicate grounds on ECS: ASP criticized for failure to check conditions, read log book, read grounding order on breaker controller. Personnel ASPs Task in which Error Occurred (From TA) Utility description sounds like a shortcut, i.e., action deliberately performed out of the originally planned sequence 6.5.6 (ASP) Perform switching for isolation according to switching instructions

64

CCO, ASP Grounds

5.6.1 (CCO) Update changes to system status on SCADA and other control room maps 12.1.5 (CCO) (CCO) assemble and review copies of outage request, switching instructions, maps and one-lines 13.4.2 (ASP) (ASP) Review available logs to determine if there are any conditions that may impact the scheduled work 13.4.4 (ASP) Conduct a Site Review of the existing conditions including locations of circuits & equipment, protective grounds, and equipment grounding conductors

65

Previous ASP failed to tag disconnects, second ASP failed to close them when restoring to normal. Operator report says he walked it down three times because of unusual configuration.

ASPs

6.5.8 (ASP #1) Apply locks/tags to all isolation points 13.4.4 (ASP #2) Conduct a Site Review of the existing conditions including locations of circuits & equipment, protective grounds, and equipment grounding conductors Walkdown performed but failed to detect unusual condition. (One assumes that disconnects would be on switching order, but utility report does not mention overlooking a step in the order.)

66

CCO cut out feeder inadvertently via SCADA during restoration switching.

CCO

12.6.1.2 (CCO) Switch equipment via SCADA . . . Operate the device

B-16

Listing of Human Error Events Event # 67 Errors and Contributing Factors Pipe cable pump controller found in OFF position, (should be AUTO: reduced pressure switch found in ON, should normally be OFF. Operator said he probably miss-set controls while doing training. Personnel ASP Task in which Error Occurred (From TA) 13.5.8 (ASP) Visually confirm operations have been completed and that conditions as they should be after all switching completed The item that fits best here is 9.1.2 for work crews: this designation used for all persons in station for reasons other than switching, and the switches were moved by a person acting in the capacity of trainer rather than switchman. However, since much switching training is onthe-job, between journeyman & apprentice, switching is often combined with training. We therefore attribute this incident to an ASP task ASP 6.5.6 (ASP) Perform switching for isolation according to switching instructions & visually confirm that each operation has been completed 13.4.4 (ASP) Conduct a Site Review of the existing conditions including locations of circuits and equipment, protective grounds, and equipment grounding conductors 13.5.8 (ASP) Visually confirm operations have been completed and that conditions as they should be after all switching completed 5.4.2.6 (CCO) If field conditions are such that a change must be made to the switching instructions... document each change 5.6.1 (CCO) Update changes to system status on SCADA and other control room maps 5.6.3 (CCO) Log all actions taken on records and reports 6.7.1 (ASP) Update the Substation Log Book/Records, including description of work performed, any abnormal conditions found, any abnormal conditions left 70 Operator opened breaker instead of placing ManualAutomatic switch in Automatic. ASP 6.5.6 (ASP) Perform switching for isolation according to switching instructions

68

Feeder inadvertently cut out.

69

Dropped load in routine switching: air break switch mechanism had been disengaged and tagged one month earlier & did not close on command. Report indicated a failure to walk down equipment to be operated before beginning on the switching order & also upon completion of switching. Report implies that disengaging mechanism was a departure from plans thought up by the CCO directing the earlier switching. There seems to also have been a failure in documenting the status of the switch for future reference, as neither the CCO or the ASP was criticized for overlooking the records, but ASP was criticized for not walking down verbal switching order before performing it.

ASP, CCO ?

B-17

Listing of Human Error Events Event # 71 Errors and Contributing Factors Operator directed to close the De-Ion breaker & the DC output switches for station battery charger. Believed them to be same switch, did not discuss with CCO, but in retrospect his report to CCO indicates that he only operated one switch. ASP criticized for failure to read station log, failure to perform pre job walkdown, & not stopping and clearing up his confusion with CCO; CCO criticized for not listening closely to his report, failure to use three way communications. Personnel ASP, CCO Comms Task in which Error Occurred (From TA) 6.2.1 (ASP) Verify the job information with the System Operator (Switching Controller) 6.5.6 (ASP) Perform switching for isolation according to switching instructions 6.4.3 (ASP) Review available logs to determine if there are any conditions that may impact the scheduled work 6.4.4 (ASP) Conduct a Site Review of the existing conditions including locations of circuits and equipment, protective grounds, and equipment grounding conductors 6.1.4 (ASP) If there are any technical uncertainties regarding the Outage Request and/or associated switching instructions, resolve as needed; do not proceed until the uncertainties are resolved 5.3.6 (CCO) If there are uncertainties regarding the switching, do not proceed until they are resolved 72 ASP did not engage the mechanism properly before closing L/I switch, switch did not close. Engaging mechanism was explicit part of order. Outage occurred when he then opened second L/I switch. Criticized for failure to confirm success of first operation before proceeding with second switch. Pump house door found open & low temperature alarm activated. Failure to properly secure door by someone previously in station. ASP 6.5.6.4 (ASP) Visually confirm that each operation has been completed

73

Work Crew

9.1.2 (Work Crew) Verify system equipment returned to same configuration as found when accepting clearance Though a work crew seems more likely to have been in the pump house than a switchman, it could have been the ASP: 13.5.8 (ASP) Visually confirm operations have been completed and that conditions as they should be

74

Test lead contacted ground, blew fuse, caused feeder transfer. Rubber boot around clip was not tight, suggested more thorough inspection of test equipment prior to use. Vent cap left off top of transformer, moisture got in & caused failure.

Work Crew

8.3.2 (Work Crew) Inspect/prepare tools and safety equipment 9.1.2 (Work Crew) Verify system equipment returned to same configuration as found when accepting clearance

75

Work Crew

B-18

Listing of Human Error Events Event # 76 Errors and Contributing Factors Inadvertently operated wrong stuck breaker protection for transformer while installing new relays for future feeder. Went to wrong control panel & pulled (removed) relay, causing feeder to trip. Apparently did so without communicating his intentions to CCO. Personnel ASP Probably Work Crew ASP Task in which Error Occurred (From TA) 6.5.6 (ASP) Perform switching for isolation according to switching instructions 6.5.6 (ASP) Perform switching for isolation according to switching instructions 6.5.11 (ASP) Notify the System Operator (Switching Controller) of problems or abnormalities detected during switching ASP 6.5.6 (ASP) Perform switching for isolation according to switching instructions

77

78

ASP given order to open feeder-side disconnects on 6 feeders inadvertently opened single phase disconnect on live feeder. Criticized for not following work practice to visually confirm breaker open, open associated disconnects, & repeat with each breaker. Checked appropriate breakers all open & then started on disconnects. On rainy night, operating a switch from a stepladder, ASP slipped & pulled switch handle to ground position, grounding live feeder. (This is a minor accident, not an error in the conventional sense.) Technician noticed doors opened on microwave building, secured same without incident. Apparently left open by previous persons at station. ASP hit live AF stirrup when trying to apply grounds to OCB side. Follow-up missing. Classic slip? ASP responding to general alarm, found sump-pump unplugged & pit full, which caused high-water alarm. Unplugged by persons in station previously?

79

ASP

6.5.6 (ASP) Perform switching for isolation according to switching instructions

80

Work Crew

9.1.2 (Work Crew) Verify system equipment returned to same configuration as found when accepting clearance 6.6.2 (ASP) Install grounds or grounding devices 9.1.2 (Work Crew) Verify system equipment returned to same configuration as found when accepting clearance Again, this seems less likely to have been done by someone performing switching 13.5.8 (ASP) Visually confirm operations have been completed and that conditions as they should be

81 82

ASP Work Crew or ASP

B-19

Listing of Human Error Events Event # 83 Errors and Contributing Factors Working on mobile unit, ASP left disconnect closed after test of mechanical operation. Portable protective grounds installed on far side of disconnect. Second ASP attempted to energize unit, which tripped & locked out. First ASP criticized for failure to return station to its as-found condition before reporting off job. Disconnects not tagged out of service, though that was the plan. Second ASP failed to check condition of station equipment before attempting to energize station. Personnel ASPs, Work Crew Grounds Task in which Error Occurred (From TA) (These actions not performed in the context of switching out for scheduled work) 13.5.5 (ASP #1) Perform switching according to instructions 9.1.2 (ASP #1/Work Crew) Verify system equipment returned to same configuration as found when accepting clearance 6.5.8 (ASP #1) (ASP) Apply locks/tags to all isolation points 13.4.4 (ASP #2) Conduct a Site Review of the existing conditions including locations of circuits & equipment, protective grounds, and equipment grounding conductors 84 Meter Engineering shorted out secondary wire while removing recorder. Employees not aware of recently developed procedures for this type of work. Communications failure blamed. Switch found in abnormal condition. Probable switching incident but cannot determine nature of error from report. CCO inadvertently opened high-side transformer breaker via SCADA. Feeder tripped because set of portable grounds left on circuit inside station. Two jobs in progress: line work required grounds, relay work did not. Line work completed first, left grounds on until relay work completed. CCO(s) criticized for failure to check map board, relief sheet, & switching log, also for failure to update ECS system configuration. ASP criticized for failure to read station log, grounding order on breaker controller, & failure to determine condition of station equipment prior to beginning switching. Work Crew Comms ASP 13.5.8 (ASP for earlier switching) Visually confirm operations have been completed and that conditions as they should be after all switching completed 5.5.1.2 (CCO) If equipment will be switched via SCADA control, operate the device This is another problem with coordination 12.1.5 (CCO) assemble and review copies of outage request, switching instructions, maps and one-lines 12.1.6 (CCO)) Review available logs to determine if there have been any recent changes or other conditions that may impact the switching instructions 12.7.1 (CCO) Update changes to system status on SCADA and other control room maps 13.4.2 (ASP) Review available logs to determine if there are any conditions that may impact the scheduled work 13.4.4 (ASP) Conduct a Site Review of the existing conditions including locations of circuits & equipment, protective grounds, and equipment grounding conductors 8.3.1 (Work Crew) Conduct a Work Briefing with the Crew including work procedures & special precautions

85

86 87

CCO CCO, ASP Grounds

B-20

Listing of Human Error Events Event # 88 Errors and Contributing Factors Transformer differential relay operated. B-phase transformer was in service so maintenance could be performed on the spare transformer. When new transformer placed in service, 161kV current transformer shorted in transformer CT box. Cause was a shorting screw left in place on the differential CT. Transformer bank inadvertently interrupted by SCADA control, resulting in 60kV voltage swing on 161kV system. Failure to verify correct equipment selected. Opened 69kV breaker due to using pre-planned switching with abnormal system conditions. Dispatching failed to recognize an abnormal system configuration outside the substation. Issued preplanned switching procedure to bypass & clear 69kV breaker. Electrician reported no 3-phase load on spare breaker, and COO ordered him to back out of the switching, which he did by executing planned switching to date in reverse order. However, ASP failed to reverse one step, and tripped on unbalanced load when a selector switch was placed in the bank position. ASP had forgotten to verify bank differential relay contacts were open before closing the bank differential relay trip cutout. Used preplanned switching to restore breaker following its outage for repair. CCO did not have functioning SCADA screen, so he could not accurately determine current system configuration. The ASP suggested using pre-planned switching for the breaker. He and CCO reviewed switching & thought it acceptable. However, the current configuration did not match that assumed by planned switching, and the ASP had to n/a two steps that did not match, but did not stop when he came to the non-matching parts. Write-up suggests CCO did not review other job documentation before authorizing use of pre-planned switching. Personnel Work Crew Task in which Error Occurred (From TA) Maintenance error 9.1.1 (Work Crew) Verify that scheduled work has been completed

89

CCO

5.5.1.2 (CCO) If equipment will be switched via SCADA control, operate the device 4.1.2. (CCO) Check for any conditions that could affect the priority of the requested outage

90

CCO

91

ASP

13.5.5 (ASP) Perform switching according to instructions Write-up seems to indicate some CCO responsibility, too

92

ASP, CCO Unscheduled

3.3.1 (CCO) If previously-written switching instructions exist... verify accuracy & completeness & suitability for use given conditions that are expected to exist, and revise as needed 12.1.6 (CCO) Review available logs to determine if there have been any recent changes or other conditions that may impact the switching instructions 13.5.5 (ASP) Perform switching according to switching instructions 12.1.5 (CCO) Assemble and review copies of outage request, switching instructions, maps and one-lines

B-21

Listing of Human Error Events Event # 93 Errors and Contributing Factors Electrician went to wrong panel and switched wrong A switch and breaker. Did not take instructions with him to the panel. Did not touch & verify the nameplate, & recorded the action incorrectly. Criticized for not using six steps process. Breaker failure relay initiated, clearing sections of 500kV bus. Engineer had submitted written isolation and test plan but dispatchers had not seen it. He faxed them another and they did not see that one either. Outage request vague: did not mention relay work. Dispatcher issued pre-planned switching that did not include removal of the breaker failure relay from service. Field personnel did not verify that the switching included removal of the breaker failure relays. Indication that the pre-planned switching was not totally adequate. Dispatcher failed to realize that planned switching activity to test bus had been compromised by other planned work performed earlier, but not covered as part of the daily work activity meeting. Write-up says CCO failed to realize his scheduled work had been compromised by other planned work that had not received proper review. Bagged? Electrician skipped a step to open a switch. Expected to open it, but it was in a different place in the sequence than he expected it. When he went to open it, he was thinking about another switch. Obviously familiar with the station, but the switching procedures had been changed from what had been done many times previously. Did not rely as much on the written order as he should have. Rely on instinct only when checking the switching. Once you are satisfied with what is written rely on the switching. Personnel ASP Task in which Error Occurred (From TA) 6.5.6.2 (ASP) Before operating a device, touch the devices identification label and verify that the correct device has been selected Breakdown of work approval & planning process. Very early in train: preparation of outage request 1.1.1 (requestor) Identify what work will be done 3.3.1 (CCO) If previously-written switching instructions exist... verify accuracy & completeness & suitability for use given conditions that are expected to exist, and revise as needed 6.4.1 (ASP) Verify the technical accuracy of the switching instructions against a one-line and current conditions CCO This plan would have been OK, but additional work changed system configuration and it had not been properly reviewed for impact on other scheduled work (i.e., this job) 4.1.2 (CCO) Check for any conditions that could affect the priority of the requested outage ASP 6.5.6 (ASP) Perform switching for isolation according to switching instructions

94

Engineer, ASP, CCOs

95

96

B-22

Listing of Human Error Events Event # 97 Errors and Contributing Factors Switching to remove the station 161kV bus was in error: the spare relays should have been taken out of service prior to de-energizing the main bus. Accurate preplanned switching was available and checked by two CCOs. Final CCO then rewrote the preplanned switching, which would have worked fine. He did not have good understanding of the configuration. Write-up hints that the rewritten switching was not checked prior to dispatching, a violation of policy. A161/69/13kV station with a 13kV capacitor bank was connected to the 161kV system at a tap located 9 miles from the station. On the electronic maps used by the dispatcher to write the switching, the tap and the station appear to be in the same location, and the switching was written under the assumption that the same person would open the cap bank, 69kV breakers, and the line switches. The switching was rewritten to have one person switch at the tap and one at the station. The re-written switching changed the original sequence, and had the tap switches opened before the cap bank and 69kV breakers at the station. When this was done the station was de-energized with the tap disconnect with a load of 145 amps feeding from one of the 69kV lines into the station, and the capacitors still in service. During a routine outage on the 161/69/13kV transformer, the plant requested an outage on the 69kV main bus breaker, with the understanding that it would be available for use when the transformer was returned to service later in the day. However, the breaker tested bad and the plant was insistent on getting the bus back in service. Switching to re-energize the bus was re-written to use the 69kV spare breaker. A PK block that had to be removed when using the spare breaker in place of the normal bus breaker was overlooked. The next day the bus differential relay operated, causing a 5-minute outage for one customer. The write up suggested that this was a knowledge problem, compounded by the fact that the change was not properly reviewed before the switching was issued. Personnel CCO Task in which Error Occurred (From TA) 4.4.4 (CCO) If the switching instructions need modifications, follow the required procedure 4.1.5 (CCO) Review the outage request and switching instructions against maps and one-lines to verify the technical accuracy of the names of devices and the feasibility of the work to be accomplished

98

CCO

Sequence was critical to this one: write-up suggests it was a knowledge problem 5.4.2.2 (CCO) If field conditions are such that a change must be made to the switching instructions . . . Research proposed changes to verify that they are appropriate and will not cause additional problems 3.4.1 (CCO) Write switching instructions for removal in sequence, including disabling of protective relaying Same utility as above but no mention of second party review of the changes

99

CCO Unscheduled

5.4.2.2 (CCO) If field conditions are such that a change must be made to the switching instructions . . . Research proposed changes to verify that they are appropriate and will not cause additional problems 3.4.1 (CCO) Write switching instructions for removal in sequence, including disabling of protective relaying 5.4.2.3 (CCO) If field conditions are such that a change must be made to the switching instructions . . . Obtain second party review and approval of proposed changes

B-23

Listing of Human Error Events Event # 100 Errors and Contributing Factors One step in the switching to remove a 161kV bus from service was to remove several relays from service to prevent their operation due to loss of potential. While removing the cover to one of the relays, the target reset lever contacted the relay seal coil contacts, causing the relay to operate. Blamed on the ASPs inattention to detail. Personnel ASP Task in which Error Occurred (From TA) Seems like a simple slip? Real problem is a less than forgiving design Error occurred in 6.5.6 (ASP) Perform switching for isolation according to switching instructions Inattention to detail in this case may refer to 6.5.6.1 (ASP) Anticipate the results of each switching step before taking any actions 101 ASP in station to investigate why a customer had been unable to reset two HEA relays. Actually they had been able to do so, but the write-up states they had not been successful before the ASP was called out. The write-up suggests that the ASPs conversation with the CCO left him with the impression that the relays were still out. He attempted to turn the reset switch but could not. He then went behind the panel to investigate why the switch would not turn. In removing the cover from the relay, he knocked it off latch, tripping the 161kV main bus. He then reset the HEA and restored the station to service. Only then did he call the CCO to report what had happened. ASP criticized for not following proper switching procedures. Attempted to de-energize a 69kV line with a disconnect. CCO had established parallel conditions but did not adequately take into account the current across the switch at the time it was to be operated. The switch was damaged when used to break load. New CCO had a qualitative understanding that load could be broken by a disconnect under parallel conditions, but did not understand that present load conditions exceeded the quantitative limitations on such switching. ASP Unscheduled Again a simple slip (or maybe an overly sensitive design) 6.5.11 (ASP) Notify the System Operator (Switching Controller) of problems or abnormalities detected during switching

102

CCO

4.2.1 (CCO) Review current status of equipment from all available sources to verify that current status will not adversely impact the planned outage Probably should have done 4.2.2 (CCO) Use available tools to study the impact of the Switching and clearance (i.e., power flow, security analysis, optimal power flow)

B-24

Listing of Human Error Events Event # 103 Errors and Contributing Factors Generating station operator called CCO to verify that disconnect G2E was still open & locked. CCO determined that it was. Station operator then asked to have Ground # 52 removed. CCO said that records showed Ground 52 was already removed. Station began pre-warmup of unit. During warmup, station discovered ground was still in place. The ground was removed & station brought on line without incident. CCO criticized for not understanding station request, failure to follow communication protocol. Communication error: CCO apparently thought # 52 ground was same as disconnect. Assumptions made vice re-checking of information. West Substation breakers 4E and 5E opened automatically, de-energizing feeders 501 and 502. Feeder 501 supplies East State Street Stations 1 and 2 and Riverdale. Feeder 502 supplies Glendale substation. The CCO should have checked these stations to determine if the associated transformers had also opened with the initial trip out. He viewed the discrete alarm that came up for West Substation breakers 4E and 5E and thought the alarm was for Bryant breakers 4E and 5E. He had responded to a trip out at Bryant 4E, 5E on Monday, leaving Hamilton Street bank 4 alive on back feed. In his mind, this was the same scenario. He therefore proceeded to check Hamilton Street substation transformers instead of Glendale. At Hamilton Street he saw that all the transformers, except for the spare bank, were in service and opened breakers 4TN and 4TS and cut in the spare bank, thinking that bank 4 should have cleared when the trip out occurred. Hamilton Street was not involved in the trip and transformer 4 should have remained in service. The CCO went to the wrong station to check and clear banks alive on back feed. Only recommendation had to do with improved communications between CCOs for different districts following automatic operations. Inadequate verification of information cited. Personnel CCO Grounds Comms Task in which Error Occurred (From TA) 12.1.6 (CCO) Review available logs to determine if there have been any recent changes or other conditions that may impact the switching instructions 12.1.10 (CCO) Review the current status of equipment from all available sources (mapboard, SCADA, logs, etc.) to verify that current status will not adversely impact the switching 12.1.11 (CCO) If there are any technical uncertainties regarding the switching instructions, do not proceed until they are resolved

104

CCO Unscheduled Comms

3.1.2 (CCO) Identify all energy sources that could supply the line or equipment being removed from service Notice that some equipment naming conventions lend themselves to this kind of error

B-25

Listing of Human Error Events Event # 105 Errors and Contributing Factors State Street 345kV breakers 2,7,8, and 14 opened automatically, along with 138kV breakers 4W and 5W. The 345kV breakers were out of service for maintenance. The CCO was asked to recall work on breaker W98 to secure the State Street bus. In the course of the recall, CCO directed ASP to prepare breakers 16 and 18 for service. ASP called back to say he could not because a bus section ground was applied. CCO rescinded previous order and requested ground be removed. Assumptions, inadequate verification, deviation from known process. 106 Feeder failed an ammeter test. Inspection revealed that a ground was applied at a different location. The computer log showed the ground had been removed and replaced the day before. Loss of focus cited. CCO misunderstood scope of work and switching instructions did not include adequate grounding for protection of cable splicers. Criticized for not thoroughly reviewing all associated paperwork. Called inadequate verification of information. Utility general grounding protocol was changed as a result of this incident, so inadequacy of protection may not have been so clear-cut as appears from report. 108 Following work on a feeder, its automatic transfer switch remained in manual & stop tagged because a phase check was required. The phase check was never ordered & the ATS remained in manual. Four days later, customer was dropped when second parallel feeder was cut out for work and load did not transfer. CCO was aware of status but apparently failed to realize that ATS had to be returned to auto. CCO 4.2.1 (CCO) Review current status of equipment from all available sources to verify that current status will not adversely impact the planned outage 3.2.3 (CCO) Identify any equipment that needs to be returned to a normal configuration as a part of switching knowledge 5.1.4 (CCO) Verify any prerequisite switching/load transfer has been done CCO Grounds 12.1.6 (CCO) Review available logs to determine if there have been any recent changes or other conditions that may impact the switching instructions 3.1.2 (CCO) Identify all energy sources that could supply the line or equipment being removed from service & identify grounds to be placed 4.1.7 (CCO) Verify that appropriate ground installation(s) have been included in the switching instructions Personnel CCO Unscheduled Grounds Task in which Error Occurred (From TA) 4.2.1 (CCO) Review current status of equipment from all available sources to verify that current status will not adversely impact the planned outage Failed to determine status of all equipment involved in his switching plan

107

CCO

B-26

Listing of Human Error Events Event # 109 Errors and Contributing Factors Feeder was given out to work without proper protection. The CCO did not provide the closest ground to the working point. Planned protection used an SF6 switch for isolation and overlooked the closest bank. CCO preparing switching plan overlooked bank and second CCO dispatching the job did also. Utility cited inadequate verification of information. Personnel CCOs Task in which Error Occurred (From TA) 3.1.2 (CCO) Identify all energy sources that could supply the line or equipment being removed from service 3.7.1.(CCO) Review documents for accuracy and completeness 4.1.5 (CCO) Review the outage request and switching instructions against maps and one-lines to verify the technical accuracy of the names of devices and the feasibility of the work to be accomplished 4.1.7 (CCO) Verify that appropriate ground installation(s) have been included in the switching instructions 110 Following a written procedure, CCO inadvertently opened wrong breaker at a substation via SCADA. Failed to verify that breaker was associated with the feeder he wanted to deenergize. Loss of focus, inadequate verification of information during actual performance of switching. 111 CCO failed to identify all connections to line out for service: Splicer working on line experienced stray voltage, probably from backfeed. Additional ground was applied and job completed without further incident. Inadequate verification of information during planning for clearance, even though CCO followed company protocol in tracing out the line CCO CCO 5.5.1.2 (CCO) If equipment will be switched via SCADA control, operate the device Note that CCO did contact ASP at station to check the clearance and notify him of impending SCADA operation, as per task 5.5.1.1, but then selected and operated incorrect breaker 3.1.2 (CCO) Identify all energy sources that could supply the line or equipment being removed from service & identify grounds to be placed

B-27

Listing of Human Error Events Event # 112 Errors and Contributing Factors Supervisor at SS called CCO requesting PAR#1 at his station be isolated immediately because of visible arcing on top of the bank. CCO initially attempted to isolate PAR#1 at a different station. He also had second CCO contact neighboring utility generating station to de-load feeder associated with the PAR at the (incorrect) station: the station (500MW) tripped off line. To make matters worse, the station contacted by CCO # 2 was not the station requested by CCO # 1. Second CCO faulted for contacting station directly rather than working through neighboring utility control center (though the station did honor his request) and should have realized he had contacted the wrong station when ASP at station reviewed switches to be opened with him. Utility said none of the parties involved, except the substation supervisor who made the first call, was listening to what the other was saying. Assumptions made, inadequate verification of information. 113 CCO called CCO from neighboring utility and requested he close the disconnects for two breakers and prepare the breakers for service. Second utilitys CCO repeated that he was to close disconnects and, once confirmed, close associated breakers and then call first CCO back. First CCO confirmed this statement: Neighboring utility CCO then did what he said he would, leading one of the lines to be energized while still being prepared for service at a connected substation. Poor communication during task: CCO at second utility thought he was to close disconnects and breakers; first CCO intended neighboring utility to close disconnects now and wait for his call to close the breakers. CCO Comms 12.3.3 (CCO) If load transfers or any other switching is a pre-requisite to the switching, coordinate switching or verify that it has taken place Personnel CCOs Unscheduled Comms Task in which Error Occurred (From TA) 3.1.2 (CCO #1) Identify all energy sources that could supply the line or equipment being removed from service 5.1.4 (CCO #2) Verify any prerequisite switching/load transfer has been done

B-28

Listing of Human Error Events Event # 114 Errors and Contributing Factors CCO # 1 recorded application and removal of grounds at two locations. Later, it was reported to CCO # 2 that both grounds were still on. CCO # 2 marked the removal of grounds from location # 1 as incomplete (i.e., ground still on), but recorded the removal of the ground at location # 2 as having been completed. When restoring upon completion of the job, the ground at the second location was left in place because it had been previously recorded as removed. Communications indicate CCO # 2 understood that both grounds were still on, but recorded the status of the second ground incorrectly. 115 Breaker RS5 was requested out of service to perform load checks. CCO contacted station and notified ASP he would be opening RS5 for testing and requested clearance to operate breaker RN5 from the ASP. ASP gave clearance and CCO opened RN5 instead of RS5 as intended. Report cited loss of focus, poor communications. 116 Energized bus with portable grounds still attached to one of its breakers. Write-up does not describe how exactly it happened: appears to be communication error from recommendations. Work Crew, CCO Grounds Comms 9.1.5 (Work Crew) Remove personal grounds installed by the crew Possibly could have been prevented by 13.4.4 (ASP) Conduct a Site Review of the existing conditions including locations of circuits & equipment, protective grounds, and equipment grounding conductors CCO Comms 5.5.1.2 (CCO) If equipment will be switched via SCADA control, operate the device Personnel CCO Grounds Comms Task in which Error Occurred (From TA) 10.3.4 (CCO) Update all displays, logs and reports to indicate the release of the clearance Poor performance of update records task as performed during course of a job

B-29

Listing of Human Error Events Event # 117 Errors and Contributing Factors ASP reviewed status of grounds placed at his station and requested orders to place 3 additional grounds and remove one that was already in place. CCO issued requested order. ASP later called back to say that the ground requested off had been removed and one of the three additional grounds requested had been applied. COO repeated back status as reported by ASP and marked the additional ground applied as complete, the two not applied as incomplete, and the action to remove the one ground already in place as incomplete, which indicated that it was still in place. After work completed the ASP was directed to remove 13 field grounds. He reported back that they had removed 12, but one was found in the open position. CCO inadvertently operated circuit switcher on spare capacitor bank rather than the one intended (via SCADA) During emergency switching, CCO cut out breaker to deenergize feeder, resulted in associated transformer being backfeed. Apparent communication problem, as he believed CCO at different center had cleared both transformers. Feeder released for work without proper protection applied. Feeder was at the time out of service for another job. Operator criticized for assumptions. Personnel CCO Grounds Task in which Error Occurred (From TA) 5.6.3 (CCO) Log all actions taken on records and reports

118 119

CCO CCO Unscheduled Comms CCO

5.5.1.2 (CCO) If equipment will be switched via SCADA control, operate the device 5.1.4 (CCO) Verify any prerequisite switching/load transfer has been done

120

3.1.2 (CCO) Identify all energy sources that could supply the line or equipment being removed from service 5.6.2 (CCO) Verify that all switching steps have been taken, that all tags have been placed, and that the resulting physical clearance will provide the Clearance Holder with a safe work distance for the purpose of the job

121

CCO energized feeder into installed 3-phase grounds at station. Utility said error occurred because operator restoring was also the one who took it out of service. Situation changed since then: he criticized for not being vigilant enough in checking all records prior to re-energizing. Opened breaker on feeder when attempting to switch out Cap bank. Breakers next to each other on SCADA display.

CCO Grounds

12.1.5 (CCO) assemble and review copies of outage request, switching instructions, maps and one-lines 12.1.6 (CCO) Review available logs to determine if there have been any recent changes or other conditions that may impact the switching instructions

122

CCO restore

12.6.1.2 (CCO) Switch equipment via SCADA . . . Operate the device

B-30

Listing of Human Error Events Event # 123 Errors and Contributing Factors Feeder failed ammeter test due to field ground applied to on new bank. CCOs did not record grounds on computer because expected installed ground had defective handle & ground move was not recorded at beginning of sequence. Criticized for not being alert to unusual request from field for combination lock number. Failed to record field grounds (possibly because the system used did not anticipate that they might be necessary?). Also clearly sub-optimal communications. 124 During switching to isolate a 138kV bus, a 138kV MOAB Switch operated successfully via SCADA, but a relay was showing incorrect indication. P&C directed ASP to bring up a green flag on the old 101 control switch (101 was the manufacturers designation for the type of switch). ASP then opened Breaker 101 control switch. ASP directed to open breaker 16 opened 17 instead. Open 12kV loops with transmission tags on the switches were closed by distribution linemen in conjunction with work in the station. Done without approval/notification of transmission dispatcher. Clearance violation. ASP Comms 5.5.2 (CCO) Direct Field Crew Leader (Switch person) to proceed with switching and tagging, and grounding according to the written instructions Communication from P&C personnel to the ASP used a jargon that unfortunately coincided with one of the breaker designations, so the ASP operated a control other than the one intended 6.5.6 (ASP) Perform switching for isolation according to switching instructions 13.2.2 (ASP) Review switching to be done, including tags & grounds to be removed 13.5.5 (ASP) Perform switching according to instructions Failed to communicate with switching authority 13.6.1 (ASP) Notify the System Operator of problems during switching or abnormal conditions at the site Personnel CCO Grounds Comms Task in which Error Occurred (From TA) 7.5.1 (CCO) Log data on clearance issued, including location & number of personal grounds placed by clearance holder 12.1.11 (CCO) If there are any technical uncertainties regarding the switching instructions, do not proceed until they are resolved

125 126

ASP ASP

B-31

Listing of Human Error Events Event # 127 Errors and Contributing Factors Several items isolated to enable repairs on hot spots in disconnects. Involved transfer of load & coordinated switching between Transmission & Distribution. Testing prior to performing work revealed that a line switch had not been opened and disconnect(s) de-energized as requested. Oneline of station used to write request was not accurate. Some discrepancies apparent in distribution switching instructions given to transmission. In resolving these discrepancies, neither side listened to the other very well. Distributions switching instructions did not include line switch. This omission not caught by checkers. Following this incident, one recommendation was to give field crews a copy of the outage request 128 Inadvertently dropped substation when opened its only remaining feed. Criticized for not more closely examining system conditions prior to issuing switching to field personnel. 138kV Cap bank supervisory control switch left in local position upon completion of work: detected by alarm. Switching order required restoration of switch to remote position. CCO 4.1.1 (CCO) Review available logs to determine if there have been any recent changes or other conditions that may impact the scheduled work 13.5.5 (ASP) Perform switching according to instructions Performing the task below might have prevented this incident 13.5.8 (ASP) Visually confirm operations have been completed and that conditions as they should be 130 CCO switched out 69kV breaker for maintenance without bypassing it and dropped 3 subs. Recommendations suggest that he was unaware that the line was a radial feed, but description says other end of line is normally open. Report does not mention results of pre switching reviews. Recommendations have some good ideas for making radial feed situations more obvious, including placing SCADA hold tag on any (closed) breakers where switching has created a radial feed situation in which they are the only remaining feed. CCO 4.1.5 (CCO) Review the outage request and switching instructions against maps and one-lines to verify the technical accuracy of the names of devices and the feasibility of the work to be accomplished Failed to be aware of system configuration Personnel CCOs Comms Task in which Error Occurred (From TA) 3.1.3 (CCO) Determine what devices must be switched 3.7.1 (CCO) Review documents for accuracy and completeness 4.1.5 (CCO) Review the outage request and switching instructions against maps and one-lines to verify the technical accuracy of the names of devices and the feasibility of the work to be accomplished

129

ASP

B-32

Listing of Human Error Events Event # 131 Errors and Contributing Factors ASP failed to place relay selector switch in proper position (per switching orders). Recommendations suggest that failure to do proper walkthrough before switching was responsible. Personnel ASP Task in which Error Occurred (From TA) 6.4.4 (ASP) Conduct a Site Review of the existing conditions including locations of circuits and equipment, protective grounds, and equipment grounding conductors 6.5.6 (ASP) Perform switching for isolation according to switching instructions 132 Central dispatch was forwarded plans for emergency installation of a mobile transformer. These appear to have contained no diagram of the installation. A temporary structure housing a bus and disconnects was a part of the installation. Subsequently when restoring the station, switching instructions issued by central dispatch did not include step to open the disconnects at the temporary structure. When disconnecting cables attached to bus risers of CB, the crew experienced electrical flash because cables were energized through the closed switches at the temporary structure. ASP manipulated Control Switch & supervisory control switch for CB-B rather than CB-E. When customer attempted to close CB-E via remote, it did not work. CCO requested ASP to operate CB-E manually, & again he performed this operation on CB-B, which closed it. ASP did not have pre prepared switching instructions: CCO & ASP modified old set of instructions on file at station. Job was to isolate a portion of a bus to reconnect two metering current transformers that had been removed from service on a prior pre-arranged job. Switching step to close a customer owned air break switch was omitted from switching order. Report notes that omission also not detected by ASPs. ASPs expected to review & walk-down switching. Good suggestion: customer contact person to review prearranged switching that covers their equipment. CCO, ASP? 1.2.2 (Requestor) Provide a diagram that identifies the work area and the lines of equipment to be removed from service 3.1.2 (CCO) Identify all energy sources that could supply the line or equipment being removed from service It is possible this might have been prevented by 13.4.4 (ASP) Conduct a Site Review of the existing conditions including locations of circuits & equipment, protective grounds, and equipment grounding conductors

133

ASP

13.5.5 (ASP) Perform switching according to instructions Made the same error twice!

134

CCO, ASP

3.2.1 (CCO) Research, document, and verify work location and equipment to be returned to service 4.1.5 (CCO) Review the outage request and switching instructions against maps and one-lines to verify the technical accuracy of the names of devices and the feasibility of the work to be accomplished 13.4.1 (ASP) Verify accuracy of instructions against a oneline and current conditions 13.4.4 (ASP) Conduct a Site Review of the existing conditions including locations of circuits & equipment, protective grounds, and equipment grounding conductors

B-33

Listing of Human Error Events Event # 135 Errors and Contributing Factors ASP failed to reset two HEA relays prior to attempting to close 69kV CB. Was distracted by being interrupted by questions from two major customers as he traveled to the control house. Realized his omission when the breaker retripped as soon as he closed it. During emergency switching, ASP inadvertently tripped 138kV CB M instead of turning up a green flag on the controls to CB A. Failure to check equipment name tags (labels) thoroughly? 137 Distribution line serviceman opened incorrect breaker during switching to restore station. Error realized immediately & reclosed within 2 minutes. When clearing Washington CB 61A Riverside CB 65 A, line section 66-104 via SCADA, CCO opened Washington CB 65A by mistake clearing Washington South Fork line section 66-144. The 65 A breaker was on the affected line. Interesting solution: on the night before switching, night CCO will place SCADA tags on all items to be switched: day operator removes them as he switches. 139 Belt on compressor motor broke and motor fell into air tank still running, was drilling a hole in tank. ASP told to trip CB to motor. Not familiar with sub and (station load) breakers not marked. Opened breaker labeled CB 69-B motor which tripped CB 69-B. Opened CB 1301 in error via SCADA while clearing CB 3401A. Distribution crew request for transmission line clearance identified wrong line to be de-energized. Did not test line deenergized. Arc occurred when attempting to ground line. Crew was working from an engineering print that stated the line was 138kV line. The line was built to 138kV but run at 69kV. ASP Personnel ASP Task in which Error Occurred (From TA) 13.5.5 (ASP) Perform switching according to instructions Loss of focus

136

ASP Unscheduled Comms ASP

6.5.6 (ASP) Perform switching for isolation according to switching instruction This is another incident in which use of jargon rather than official nomenclature may have played a role, though it doesnt explain the selection of the wrong breaker 13.5.5 (ASP) Perform switching according to instructions

138

CCO

5.5.1.2 (CCO) If equipment will be switched via SCADA control, operate the device Simple slip or cognitive confusion? Again, this probably wouldnt have happened the way it did if each device had a unique number

6.5.6 (ASP) Perform switching for isolation according to switching instructions The above attribution is something of a stretch. Switching (on electric system) was never intended

140 141

CCO ASP

5.5.1.2 (CCO) If equipment will be switched via SCADA control, operate the device 1.1.3 (Requestor) Identify the exact location of the work to be done 8.4.1 (Work Crew) Test to verify that lines or equipment are de-energized

B-34

Listing of Human Error Events Event # 142 Errors and Contributing Factors Switching order written incorrectly, with steps out of sequence. CC noticed the error after the order was issued but before switching was done. However, not able to contact the field switchman to make the necessary changes. Personnel CCO Task in which Error Occurred (From TA) 3.4.1 (CCO) Write switching instructions for removal in sequence, including disabling of protective relaying 4.1.5 (CCO) Review the outage request and switching instructions against maps and one-lines to verify the technical accuracy of the names of devices and the feasibility of the work to be accomplished ASP ASP 6.5.6 (ASP) Perform switching for isolation according to switching instructions 6.5.6 (ASP) Perform switching for isolation according to switching instructions 5.5.1.2 (CCO) If equipment will be switched via SCADA control, operate the device Apparent slip/or confusion in execution phase (was following written plan) CCO 3.4.1 (CCO) Write switching instructions for removal in sequence, including disabling of protective relaying 3.7.2 (CCO) Proofread & verify accuracy of switching instructions

143 144

Did not follow switching order as written: opened BSD 1338 instead of LSD 1339. Discovered by maintenance crew. ASP turned off recloser and opened incorrect 69kV CB, then proceeded to open the disconnect for the breaker he should have opened under load. Dispatcher using SCADA opened PL562 breaker vice HL562, which had been previously opened. Restored in 2 min. Operator on overtime with 6 or more outages in progress, short staffed. No (Pre-prepared) switching plan was available, so CCOs made work plan using info from SCADA displays. CCO reviewing it concentrated on one aspect of the switching, overlooked switch nomenclature error. Opened incorrect switch as specified in the incorrect plan. Excessive workload cited, outage late by one hour due to the workload. Manning levels not sufficient to support 2-person switching mentioned. Error in planning & in subsequent second-party checking. Following previously prepared switching plan that had been checked by two people, CCO did switching to prepare for an outage. Maintenance person instructed to lock disconnects in open position discovered them still closed. Use of incorrect switching plan cited. (unclear as to whether the plan used was the intended plan, even though it was inaccurate, or not the correct plan for the job). Plan incorrect, does not say how critical actions were omitted or not picked up by one of the CCOs who reviewed it (may have been as many as four involved: one to prepare, two to review, & one to direct it). Not good incident report.

145

CCO

146

147

CCOs

3.3.1 (CCO) If previously-written switching instructions exist... verify accuracy & completeness & suitability for use given conditions that are expected to exist, and revise as needed 3.7.1 (CCO) Review documents for accuracy and completeness

B-35

Listing of Human Error Events Event # 148 Errors and Contributing Factors Item omitted from switching orders, work permit issued while another permit on same equipment still in effect (tags still on equipment). Switching instructions reviewed by two operators: first found several (but evidently not all) omissions & added them to order, second OK it. Deficiencies in procedures and training identified in analysis. 149 On basis of discussion of planned switching between two Control Centers, second CC was under the impression that first CC had given permission to proceed with the switching. This was not the case. Imprecise communications cited. Order previously prepared. Checked against Station print. Closed feeder tie & operator (ASP) reported no amps flowing through it. CCO dismissed it as a metering problem. When opened second breaker load was not picked up. Inspection revealed open disconnect. Stated that it was a busy morning. Not entirely clear whether field or CC failed to close disconnect, but language suggests CC. Also not clear whether the prints used were accurate, but assume they were because no fuss made about it. 151 Capacitor breaker inadvertently opened while making tap change. Contributing factor cited was waking up at 4AM repeatedly, after switching over from night shift, & information overload. Conductor being pulled fell on energized 115kV line. Linemen sent to substation to check on relay targets noticed that appropriate Hot Line Order Targets were not lit. Investigation revealed that reclosers had been turned off via SCADA but the Hot Line tag had been improperly placed, resulting in an invalid Hot Line Order. Based on incident notification: no follow-up. CCO 5.5.1.2 (CCO) If equipment will be switched via SCADA control, operate the device Simple slip CCO 5.5.6 (CCO) Place tags on devices on SCADA and other control room maps CCO Comms CCO 5.3.6 (CCO) If there are uncertainties regarding the switching, do not proceed until they are resolved 5.5.1 (CCO) Switch devices by SCADA as required by the switching instructions 5.1.4 (CCO) Verify any prerequisite switching/load transfer has been done Personnel CCOs Task in which Error Occurred (From TA) 3.1.3 (CCO) Determine what devices must be switched 3.7.1 (CCO) Review documents for accuracy and completeness

150

152

B-36

Listing of Human Error Events Event # 153 Errors and Contributing Factors Electrician responding to alarm noticed hydraulic fluid leaking from 230kV breaker, was unsure if it was fully closed, or going to stay closed. Requested CCO to take the load off the breaker: CCO opened other end of line. ASP then requested switching to remove breaker from service for repairs. ASP did not indicate the breaker could not be operated and would have to be isolated in the closed position. CCO wrote switching for normal removal from service (open breaker & then disconnects). Other duty CCO had left & switching was not checked by second party prior to dispatching. When ASP informed him that breaker would not open, CCO & ASP agreed to delete the steps to open the breaker & proceeded to try to open disconnects (that is, they altered the switching program on-the-fly after the initial steps had been implemented). Did briefly discuss vars across the breaker with line open at other end. Disconnect arced when ASP opened it & arcing tripped breaker. CCO criticized for not bypassing the breaker before attempting to isolate it (lack of knowledge of power system cited). ASP criticized for not including information that he thought breaker would not open in his initial request for the switching, not reviewing the switching as-issued against the one-line, & for not calling CCO when the arcing occurred. Switching in progress to bypass a faulted breaker. ASP was diverted to perform this switching while on rout to a scheduled job. ASP opened disconnect 1545 instead of 1245 as instructed, but reported 1245 open. ASP discovered error when CCO asked him to check the position of the disconnect he had closed in error. Personnel ASP, CCO Unscheduled SARAH to check 2.2.6 as task for bypassing breaker Task in which Error Occurred (From TA) On-the-fly alteration of switching, which was a semiemergency to begin with 1.2.1 (requestor: ASP in this case) Identify type of switching required to perform work 3.7.1 (CCO) Review documents for accuracy and completeness 2.2.6 (preparer) Verify feasibility and determine need for load transfer & coordinate with others as needed 6.4.1 (ASP) Verify the technical accuracy of the Outage Request and/or switching instructions against a one-line and current conditions 6.5.11 (ASP) Notify the System Operator (Switching Controller) of problems or abnormalities detected during switching

154

ASP Unscheduled

6.5.6 (ASP) Perform switching for isolation according to switching instructions

B-37

Listing of Human Error Events Event # 155 Errors and Contributing Factors As a part of restoring a bus after work had been completed, the CCO was to call adjoining companies and request them to open 69kV loops in sequence. He instead called the wrong company first, and requested that they open their loop, which they did, de-energizing four 41.8 and 12.5kV circuits. CCO criticized for not following exact sequence of program as written, using slang hook-up, rather than the proper nomenclature, interconnect, in his request, and not verifying that he had contacted the intended party. The other utility dispatcher identified himself by name but did not identify his utility. Both criticized for a discussion about a minor frequency excursion when they should have been focused on the switching. While performing switching to bypass a breaker for relay replacement, ASP closed disconnect 565 rather than 965 as per the switching instructions. ASP Called in and reported that he had closed the wrong disconnect, 665 (rather than the one he in fact did close, 565). ASP Has problems with transposing numbers, did not perform station inspection or review of oneline. Sloppy communications also cited: ASP did not repeat what he was to do, CCOs did not require him to. Request changed from clearance to general switching, CCOs very busy that day but rewrote part of switching in keeping with new classification, possibly giving more concentration to the paperwork than the actual switching. While performing switching to remove a 69kV line from service, an ASP was given instructions to check three breakers open (they were opened via SCADA). ASP interpreted his instructions as not to proceed with the switching until the CCO called him back. The rest of the switching was performed and a clearance issued. Clearance issued in error (before CCO had confirmation that the three breakers were in fact open). The ASP phoned the CCO and asked when the CCO wanted him to proceed. Work had been delayed and there was a sense of urgency about getting on with it. Also there were many people and much noise in the control room (at the sub) which impeded communication. Personnel CCOs Comms Task in which Error Occurred (From TA) 12.3.3 (CCO) If load transfers or any other switching is a pre-requisite to the switching, coordinate switching or verify that it has taken place

156

ASP CCOs

6.5.6 (ASP) Perform switching for isolation according to switching instructions 6.4.1 (ASP) Verify the technical accuracy of the switching instructions against a one-line and current conditions

Comms Spot revision

6.4.4 (ASP) Conduct a Site Review of the existing conditions including locations of circuits and equipment, protective grounds, and equipment grounding conductors 5.3.2 (CCO) Have the Field Crew Leader/ASP read the switching instructions back to you

157

CCO, ASP Comms

5.5.5 (CCO) Receive and record the time that switching was performed 5.6.2 (CCO) Verify that all switching steps have been taken, and all tags have been placed . . .

B-38

Listing of Human Error Events Event # 158 Errors and Contributing Factors A 115-kV line relayed to lockout during a snowstorm. Load was interrupted for 2 hours and 35 minutes due to a broken cross arm on a section of the 115-kV line. While performing general switching at a tap, the ASP was instructed to open switch 1963, but opened 163 instead. The error was not discovered. In subsequent switching, when instructed to close 1963, the ASP closed 163 instead. Criticized for not carrying the switching order and checking for right location and right sequence. For switching to remove a breaker from service for painting, the dispatcher modified existing pre-written switching. Two prewritten programs were used, both modified by the deletion of some steps. The restore switching involved turning a bus differential cutout switch from position 1 through position 2 to position 3. The first dispatcher believed that the cutout switch in question was a pull-to-operate switch and passing through position #2 would have no effect so long as the switch handle was pulled out. The second dispatcher accepted his identification as to the type of switch. Although the acceptability of the modifications to the switching plans hinged on the kind of switch used for the cutout, they did not ask the switchman at the station to confirm the type of switch. It was not a pull-to-operate switch, and passing through position #2 caused a lockout relay to operate, de-energizing a bus, a 115/69kV transformer, and three 69kV circuits for four minutes. Personnel ASP Unscheduled Task in which Error Occurred (From TA) 6.5.6.2 (ASP) Before operating a device, touch the devices identification label and verify that the correct device has been selected

159

CCO(s)

3.3.3 (CCO) For unusual situations, consult with subject matter experts as needed Although this task assignment may be a bit of a forced fit, the fact that the two dispatchers discussed it suggests it was an issue that should have been resolved by recourse to documentation or additional expertise But if not this, then certainly the below 5.3.6 (CCO) If there are uncertainties regarding the switching, do not proceed until they are resolved The report did not criticize the switchman for failing to pick up on the error in the written instruction

B-39

Listing of Human Error Events Event # 160 Errors and Contributing Factors A section of a 115kV line was requested out to allow work on a support structure. The outage request was verbal rather than written. The section de-energized was not that required for the work. Specific Clearance boundaries were discussed/agreed to prior to the crew Foremans acceptance of the clearance, but not discussed in the tailgate or verified immediately before beginning of the work. The crew fuzzed the conductor with a hot stick and received a small audible indication. The crew attributed this to line induction from a parallel 230-kV power line (and did not recognize that they were grounding an energized circuit). When they attempted to apply grounds (using a hot stick), the still-energized circuit flashed to the ground cable. Two crew members suffered minor flash burns to the eyes. Incident was blamed on verbal miscommunication that could have been prevented with a written outage request. Report also recommended use of direct read vice audio/visual voltage detectors. Report implies that the location of the required outage was muddled in the verbal transmission of the outage request, rather than incorrectly specified by the requestor. 161 One of the steps in switching to return a breaker to service was to remove tag, close, check closed & lock a disconnect. The ASP wrote down the time, unlocked the lock, removed the tag, and then re-locked the control without operating it. He then went to the control house to report the times. When performing additional steps in the sequence, he drew and arc when operating another disconnect. He then realized he had not in fact closed the first disconnect, and immediately phoned the CCO to inform him of the error. Crew distracted by concerns for a member who had not reported to work. Criticized for not following the 6-steps. ASP Personnel CCOs, Work Crew Comms Task in which Error Occurred (From TA) 1.1.3 (Requestor) Identify the exact location of the work to be done 8.1.5 (Clearance holder/Work Crew) Conduct a Site Review of the existing conditions including locations of circuits and equipment, equipment grounding conductors & protective grounds 8.3.1 (Work Crew) Conduct a Work Briefing with the Crew including work procedures & special precautions 8.4.1 (Work Crew) Test to verify that lines or equipment are de-energized Note that the absence of a written request makes several of the checking steps in the Task Analysis impossible to perform: the investigators stated that the incident would not have occurred had there been a written request. These steps are not cited here because they could not be performed, rather than performed incorrectly

6.5.6 (ASP) Perform switching for isolation according to switching instructions

B-40

Listing of Human Error Events Event # 162 Errors and Contributing Factors When performing switching to bypass a breaker, the ASP inadvertently placed a Range change control switch in position 5 instead of position 4 as per the switching instructions. Position 5 was unwired: the switch had 12 marked positions, of which only positions 1-4 were used. The switch was of the pull-to-operate type, and has caused switching problems in the past, as the exact position is difficult to see while grasping the switch handle. ASP realized that something was wrong when he discovered a breaker which should have been closed was open. He then checked the position of the selector switch, found it in the wrong position, placed it in the correct position and called dispatch to report the error. ASP criticized for not completing 6 steps and for correcting the error before he called into dispatch, which, by policy, he should have done when he discovered the breaker in an unexpected position. In switching to bypass a breaker, the ASP apparently opened and locked disconnect 1465 vice 1565, as instructed. Was not discovered until one month later, when performing switching to bypass the same breaker, the ASP was instructed to close 1565. He found it closed and wrote down the time. He did not report that it had been found in an unexpected position. He only realized something was wrong when performing switching to restore the breaker to normal he drew an arc when he opened disconnect 1565. First ASP criticized for not following the 6 steps, second ASP for not stopping & reporting when he discovered the disconnect in an unexpected position. Report also mentions that the CCOs should have noted that their EMS displays were showing 0 MW and 0 Mvar across breaker 1462 (because its disconnect was open): this was attributed to a transducer problem rather than being properly investigated, apparently because there were other metering problems at the substation. Report also noted that breaker 1462 was checked closed on three separate occasions before the incorrectly positioned disconnects were discovered, yet nobody noticed the disconnect was open. Personnel ASP Task in which Error Occurred (From TA) 6.5.6.3 (ASP) Perform switching: operate the device according to the switching instructions 6.5.11 (ASP) Notify the System Operator (Switching Controller) of problems or abnormalities detected during switching Note: the investigation report identified the root cause as a human factors problem with the switch (such pull-to-operate switches have been involved in a number of this utility incidents, though this was apparently unknown to the investigators)

163

ASP(s) CCO(s)

6.5.6 (ASP) Perform switching for isolation according to switching instructions 6.5.11 (ASP) Notify the System Operator (Switching Controller) of problems or abnormalities detected during switching 5.3.6 (CCO) If there are uncertainties regarding the switching, do not proceed until they are resolved

B-41

Listing of Human Error Events Event # 164 Errors and Contributing Factors When performing switching to restore a breaker, the ASP was instructed to open, check open and lock disconnect 2215. He opened switch 2315 instead. He recorded the time and reported to the dispatcher that he had performed the switching as requested. When the ASP read back the switching step to the CCO, he realized that he had mistakenly opened disconnect 2315 and informed the CCO of the situation. ASP criticized for not following the 6 steps. He claimed that he did, but the fact the switch number on the nameplate did not match the switching instructions just didnt register until he reviewed the switching with the CCO. PCB 666 was out of service on a clearance, with disconnects open. A CCO reporting to work after 6 days off was informed he would have to fill in for the transmission CCO, who had called in sick. The fill-in had not worked transmission in over 7 months. He did not review the CC log. The off-going transmission CCO did not mention that the breaker was open & under clearance. The breaker was not tagged on SCADA. The CCO noticed a SCADA alarm indicating the breaker was open. He believed that the breaker had tripped but did not perform a test. The CCO asked a trainee he was supervising to call the customer and inform that the breaker was open and that he was going to close it for a test. The customer CCO did not tell the trainee that the breaker was open under clearance. CCO closed it breaker via SCADA. The Generation CCO then noted that disconnects were open. CCO checked SCADA one-line & found disconnects were indeed open. He then checked the log and found that the breaker was out on clearance. Personnel ASP Task in which Error Occurred (From TA) 6.5.6 (ASP) Perform switching for isolation according to switching instructions Note that the step-by-step post-switching readback of the steps executed was an effective error detection device in this case

165

CCOs Comms

4.1.1 (CCO) Review available logs to determine if there have been any recent changes or other conditions that may impact the scheduled work 5.6.5 (off-going CCO) If switching has not been completed before the end of the shift, review the status of the job with the oncoming shift 5.5.6 (off-going CCO) Place tags on devices on SCADA and other control room maps Report noted several deficiencies in the utilitys practices and procedures: The CC needed a white board for showing all current clearances It was permissible for work crews to NOT call in when entering reporting for duty yards and stated that such calls should be made mandatory There were no formal policies/standards for briefing at shift turnover

B-42

Listing of Human Error Events Event # 166 Errors and Contributing Factors While performing switching to establish a clearance, the ASP was instructed to open and tag disconnect 2141. After verifying that the breaker was open, he went to the control building to get a hotstick with which to open the stick-operated single-phase disconnects. When he returned with the stick, he went to the wrong breaker bay and opened the disconnect 2241on an adjacent, energized breaker. ASP criticized for checking only the last two digits of the switch number rather than all four. Report also suggested that it would be a good idea to check the status of any breaker immediately before opening its disconnects, as it is possible that it could have closed while he went to get the hotstick. A contractor had a clearance to install drops to a new 500/230kV transformer installation. The work was protected by 2 pairs of open & locked motor operated disconnects (MODs). One of the disconnects was also tagged open on a prior clearance. It was decided to expand the clearance area for the transformer work. In doing so, MOD 485, the common clearance point, was somehow written by the outage coordinator as MOD 483 on the instructions (which were checked by 2 CCOs and five additional people in the course of the work, none of whom noticed the error). The prior clearance was released and the clearance tag on MOD 485 removed six hours after the clearance was issued for the transformer work. The job supervisor did not perform a walkdown of his clearance points prior to accepting the clearance. A third CCO discovered the error. Transmission CCO criticized for dispatching the order without thoroughly reviewing it. The clearance holder saw a tag on MOD 485 but did not verify that the tag was for his clearance. Personnel ASP Task in which Error Occurred (From TA) 6.5.6.2 (ASP) Before operating a device, touch the devices identification label and verify that the correct device has been selected

167

CCOs Unscheduled

Multi-clearance Failed in several checking tasks 3.3.1 (CCO modifying instructions) If previously-written switching instructions exist... verify accuracy & completeness & suitability for use given conditions that are expected to exist, and revise as needed 3.7.1 (CCO) Review documents for accuracy and completeness 8.1.5 (Clearance holder/Work Crew) Conduct a Site Review of the existing conditions including locations of circuits and equipment, equipment grounding conductors & protective grounds 4.1.5 (CCO) Review the outage request and switching instructions against maps and one-lines to verify the technical accuracy of the names of devices and the feasibility of the work to be accomplished

B-43

Listing of Human Error Events Event # 168 Errors and Contributing Factors Personnel investigating reports of blinking lights discovered an arcing 34kV disconnect at a substation. A line foreman sent to investigate thought the disconnect could be re-seated if the circuit were momentarily de-energized. The CCO agreed, and opened a breaker via SCADA command. The CCO realized it was not the correct breaker when the foreman (ASP) reported the arcing was still present. Dispatcher was criticized for not tracing out the circuit on the appropriate drawings before selecting the breaker to operate. Neither the CCO nor the ASP ever stated the number of the breaker that should be opened. Personnel CCO, ASP Unscheduled Comms Task in which Error Occurred (From TA) CCO felt time pressure, though, objectively, there was none 3.1.3 (CCO) Determine what devices must be switched 5.2.3 (CCO) Review the work to be done (big picture) with Field Crew Leader or Switch person before starting the job

End of table

B-44

C
OUTLINE OF EPRI JOB AND TASK ANALYSIS FOR PLANNED OUTAGES
There are fourteen tasks in the process of handling a planned outage for a line or a piece of equipment as shown in Figure C-1. While there are great variations in the steps and sub-steps associated with each of these tasks, all utilities that supplied data for this project perform these fourteen tasks.

Task 1 Complete an Outage Request & Submit for Approval

Task 2 Conduct an Initial Review of the Outage Request

Task 3 Write Switching Instructions for Removal & Return of Lines/Equipment

Task 4 Review Outage Request on the Day of Scheduled Work

Tasks 5 & 6 Implement Switching for Removal of Lines or Equipment

Tasks 7 & 8 Issue/Receive a Clearance

Tasks 9, 10 & 11 Release/Accept a Clearance

Tasks 12 & 13 Implement Switching for Return of Lines or Equipment

Task 14 Review & Close

Tasks in Handling a Planned Outage

Figure C-1 Overview of the Process of Handling the Planned Outage of a Line or Piece of Equipment

C-1

Outline of EPRI Job and Task Analysis for Planned Outages

1. Complete an Outage Request Research and Record Administrative Data Research and Record Technical Data Submit the Outage Request Form for Review and Approval Review the Outage Request for its Accuracy and Completeness Review Outage Request for its Feasibility and Possible Combination with Other Work Needing To Be Performed Coordinate/Notify Others Involved with Outage Request (Internal and External) Make revisions, Decide to Approve, Postpone, or Cancel Handle Administrative Aspects of Outage Request Decide What Needs to be Switched, Tagged and Grounded for Removal of Lines or Equipment Decide What Needs to be Ungrounded, Untagged and Switched for Return of Lines or Equipment Research switching instructions Write switching instructions for Removal of Lines or Equipment Write switching instructions for Return of Lines or Equipment Develop Support Documents Review and Assess Technical Merits of Assembled Documents Submit switching instructions for Review and Approval

2. Conduct an Initial Review of the Outage Request

3. Write Switching instructions for Removal and Return of Lines or Equipment

4. Review the Outage Request and Switching instructions on the Day of the Scheduled Work C-2 Read All Documents for Understanding Review Against Current Technical Conditions Review Against Current Non-Technical Conditions Recommend/Decide Whether or Not to Proceed Make Notifications Prepare to Coordinate Switching Exchange Job-Related Information Exchange switching instructions Exchange Information on Field Conditions

5. Implement Switching For Removal of Lines or Equipment Control Room Actions

Outline of EPRI Job and Task Analysis for Planned Outages

Switch, Ground and Tag Equipment Complete Documentation and Notifications Prepare to Perform Switching Exchange Job-Related Information Exchange switching instructions Conduct Site Review and Report Findings Perform Switching and Tagging Test for Dead and Apply Grounds Complete Records Prepare to Issue Clearance Exchange Technical Information Authorize Placement of Personal Grounds Issue Clearance Update Records and Reports Transfer Clearance if Needed Prepare to Receive Clearance Exchange Technical Information Prepare Crew Test for Dead and Place Personal Grounds Perform Work and Report Status Transfer Clearance if Needed Update Records and Reports Prepare to Release Clearance Exchange Technical Information Release Clearance Update Logs/Records/Reports Exchange Technical Information

6. Implement Switching For Removal of Lines or Equipment Field Actions

7. Issue a Clearance Control Room Actions

8. Receive a Clearance Field Actions

9. Release a Clearance Field Actions

10. Accept the Release of a Clearance Control Room Actions

C-3

Outline of EPRI Job and Task Analysis for Planned Outages

Accept Release of Clearance Make Notifications and Update Records and Reports Determine Status of Lines or Equipment Obtain Approval from Either the Clearance Holder or the Clearance Holders Supervisor to Proceed with the Release of a Clearance Review Against Current Technical Conditions Review Against Current Non-Technical Conditions Recommend/Decide Whether or Not To Proceed Exchange Job-Related Information Exchange switching instructions Authorize Removal of Tags and Switching Update Records and Reports Prepare to Perform Switching Exchange Job-Related Information Exchange switching instructions Conduct Site Review and Report Findings Perform Switching and Remove Tags Make Notifications and Complete Records Monitor Lines and Equipment Verify Maps and Diagrams Reflect System Changes Update/Store Records

11. Release a Clearance in an Emergency

12. Implement Switching for Return of Lines or Equipment Control Room Actions

13. Implement Switching for Return of Lines or Equipment Field Actions

14. Review and Close Outage Request

C-4

D
FULL TEXT OF TASK DESCRIPTIONS USED IN THIS REPORT
# Abbreviated Task Statement Full Text of Task Statement

1. Complete an Outage Request 1.1.1 (requestor) Identify what work will be done 1.1.3. (requestor) Identify the exact location of the work to be done 1.2.1 (requestor) Identify type of switching required to perform work. 1.2.2 (requestor) Provide a diagram that identifies the work area and the lines of equipment to be removed from service 2. Conduct an Initial Review of the Outage Request 2.2.6 (preparer) Verify feasibility and determine need for load transfer & coordinate with others as needed 2.2.6 Verify feasibility and determine the need for load transfer and coordinate with others as needed 1.1.1 Identify what work will be done 1.1.3 Identify the exact location of the work to be done (include substation name, line number, device name and number, etc.) 1.2.1 Identify the type of switching required to perform work (work with de-energized lines or equipment) 1.2.2 Provide a diagram that identifies the work area and the lines or equipment to be removed from service

3. Write Switching instructions for Removal and Return of Lines and Equipment 3.1.2 (CCO) Identify all energy sources that could supply the line or equipment being removed from service 3.1.2 (CCO) Identify all energy sources that could supply the line or equipment being removed from service & identify grounds to be placed 3.1.3 (CCO) Determine what devices must be switched 3.2.1 (CCO) Research, document, and verify work location and equipment to be returned to service 3.2.3 (CCO) Identify any equipment that needs to be returned to a normal configuration as a part of switching 3.2.5 (CCO) Identify ground devices to be removed 3.1.2 Identify all energy sources that could supply the line or equipment being removed from service Identify grounds to be placed Identify safety tags to be issued

3.1.3 Determine what devices must be switched to safely isolate equipment 3.2.1 Research, document and verify work location and equipment to be returned to service 3.2.3 Identify any equipment (capacitors, transformer tap changers, regulators, etc.) that need to be returned to a normal configuration as a part of switching 3.2.5 Identify ground devices to be removed

D-1

Full Text of Task Descriptions used in this Report # Abbreviated Task Statement 3.3.1 (CCO) If previously-written switching instructions exist... verify accuracy & completeness & suitability for use given conditions that are expected to exist, and revise as needed 3.3.2 (CCO) If switching instructions do not exist, compile information needed to write switching instructions Full Text of Task Statement 3.2.1 If previously written switching instructions exist for removal and return of lines or equipment from service, check the technical accuracy and completeness of the instructions, given the conditions expected to be present at the time of the job, and revise as needed 3.3.2 If switching instructions do not exist for removing and returning lines or equipment from service, compile information needed to write switching instructions using the information gathered 3.3.3 For unusual situations, consult with subject matter experts from substation test, system protection, telecom 3.4.1 Write switching instructions in sequence, using the name and/or number of each device to be switched or operated; include: Special notifications Verification that switches, circuit breakers, transformer tap changers, voltage regulators are in expected position before switching begins Disabling automatic controls on equipment such as capacitors, transformer tap changers, regulators, etc. Shifting loads Known cautions at appropriate switching step(s) Disabling protective relaying that could automatically energize lines or equipment included in the Outage Request The placement of substation and line grounding devices

3.3.3 (CCO) For unusual situations, consult with subject matter experts as needed 3.4.1 (CCO) Write switching instructions for removal in sequence, including disabling of protective relaying

3.5.1 (CCO) Write switching instructions for restoration in sequence, including re-enabling of devices disabled during switching for removal from service

3.5.1 Write switching instructions in sequence, using the name and/or number of each device to be switched or operated; include: Special notifications Verification that switches, circuit breakers, transformer tap changers, voltage regulators, etc. are in their expected position before switching begins Enabling automatic controls on equipment such as capacitors, transformer tap changers, regulators, etc. Enabling of protective relaying that had been disabled as a result of this Outage Request Removal of substation and line grounding

D-2

Full Text of Task Descriptions used in this Report # Abbreviated Task Statement Full Text of Task Statement devices that have been placed as a result of this Outage Request 3.7.1.(CCO) Review documents for accuracy and completeness 3.7.2 (CCO) Proofread & verify accuracy of switching instructions Known cautions at appropriate switching step(s) Verifications that the system has been returned to the normal or as found state

3.7.1. Review documents for accuracy and completeness 3.7.2 Proofread and verify the accuracy of the switching instructions

4. Review the Outage Request and Switching instructions on the Day of the Scheduled Work 4.1.1 (CCO) Review available logs to determine if there have been any recent changes or other conditions that may impact the scheduled work 4.1.2 (CCO) Check for any conditions that could affect the priority of the requested outage 4.1.1 Review available logs to determine if there have been any recent changes to equipment, any recent unexplained alarms, or other conditions that may impact the scheduled work 4.1.2 Check for any conditions that could affect the priority of the requested outage, including: Recent unplanned system events 4.1.4 (CCO) Ensure the Outage Request is clear and the work to be done is understood 4.1.5 (CCO) Review the outage request and switching instructions against maps and onelines to verify the technical accuracy of the names of devices and the feasibility of the work to be accomplished 4.1.7 (CCO) Verify that appropriate ground installation(s) have been included in the switching instructions 4.2.1 (CCO) Review current status of equipment from all available sources to verify that current status will not adversely impact the planned outage 4.2.2 (CCO) Use available tools to study the impact of the Switching and clearance (i.e., power flow, security analysis, optimal power flow) 4.2.3 (CCO) Determine effect of planned procedure to local and interconnected power systems given current system conditions Other clearances already issued Any future planned events Power generation problems or power transactions

4.1.4 Ensure the Outage Request is clear and the work to be done is understood 4.1.5 Review the Outage Request and switching instructions against maps and one lines to verify the technical accuracy of the names of devices, and the feasibility of the work to be accomplished 4.1.7 Verify that appropriate ground installation(s) have been included in the switching instructions 4.2.1 Review the current status of equipment from all available sources (mapboard, SCADA, logs, etc.) to verify that current status will not adversely impact the planned outage 4.2.2 Use available tools to study the impact of the Switching and clearance (i.e., power flow, security analysis, optimal power flow) 4.2.3 Determine effect of planned procedure to local and interconnected power systems, given current system conditions( (load conditions, weather, available protection, outage duration, equipment data, conflicting field activities)

D-3

Full Text of Task Descriptions used in this Report # Abbreviated Task Statement 4.4.4 (CCO) If the switching instructions need modifications, follow the required procedure Full Text of Task Statement 4.4.4 (CCO) If the Switching instructions need modifications, follow the required procedure

5. Implement Switching for Removal of Lines or Equipment Control Room Actions 5.1.1 (CCO) Assemble & review copies of Outage Request, switching instructions, maps, and one-lines 5.1.3 (CCO) Make notifications: CC supervisor, customers if needed, other entities as needed 5.1.1 Assemble and review copies of Outage Request, switching instructions, maps, and one lines 5.1.3 Make notifications: 5.1.3.1 Advise Control Room Shift Supervisor that switching will proceed as scheduled 5.1.3.2 If customers will be impacted, either notify customers directly or through corporate representatives 5.1.4 (CCO) Verify any prerequisite switching/load transfer has been done 5.1.4 If load transfers or any other switching is a prerequisite to the switching, coordinate required switching and verify that these actions have taken place 5.2.3 Review and verify the following basic information with the Field Crew Leader (Switch person) before starting the job: Communication tool to use (phone number, radio channel) 5.3.2 (CCO) Have the Switch person read the switching instructions back to you Purpose of the Outage Request Work location(s) including the name and location of circuits or equipment involved Time of the activity Any abnormal situations that may impact the work to be accomplished Sequence of switching against maps and one-lines Minimum clearance distance needed for safety Isolation points and associated tags Location and type of grounds and grounding devices to be installed

5.2.3 (CCO) Review the work to be done (big picture) with Switch person before starting the job

5.3.2 Have the Field Crew Leader (Switch person) read the switching instructions back to you, placing a check mark beside each instruction as it is read 5.3.6 If there are uncertainties regarding the switching, do not proceed until they are resolved

5.3.6 (CCO) If there are uncertainties regarding the switching, do not proceed until they are resolved

D-4

Full Text of Task Descriptions used in this Report # Abbreviated Task Statement 5.4.2.2 (CCO) If field conditions are such that a change must be made to the switching instructions . . . Research proposed changes to verify that they are appropriate and will not cause additional problems 5.4.2.3 (CCO) If field conditions are such that a change must be made to the switching instructions . . . Obtain second party review and approval of proposed changes 5.4.2.6 (CCO) If field conditions are such that a change must be made to the switching instructions . . . Document each change 5.5.1 (CCO) Switch devices by SCADA as required by the switching instructions 5.5.1.2 (CCO) Operate the device 5.5.1.3 (CCO) Compare watt/var/volt/amp values to verify that switching achieved the expected results 5.5.2 (CCO) Direct Field Crew Leader (Switch person) to proceed with switching and tagging, and grounding according to the written instructions 5.5.5 (CCO) Receive and record the time that switching was performed 5.5.6 (CCO) Place tags on devices on SCADA and other control room maps 5.6.1 (CCO) Update changes to system status on SCADA and other control room maps 5.6.2 (CCO) Verify that all switching steps have been taken, and all tags have been placed . . . Full Text of Task Statement 5.4.2.2 Research proposed changes to verify that they are appropriate and will not cause additional problems

5.4.2.3 Obtain second party review and approval of proposed changes

5.4.2.6 Document each change, initial and date change 5.5.1 Switch devices by SCADA as required by the switching instructions 5.5.1.2 Operate the device 5.5.1.3 Compare watt/var/volt/amp values before and after switching to verify that switching achieved the expected results 5.5.2 Direct Field Crew Leader (Switch person) to proceed with switching and tagging, and grounding according to the written instructions 5.5.5 Receive confirmation from the Field Crew Leader (Switch person) that field switching has been completed and record the time 5.5.6 Place tags on devices on SCADA and other control room maps 5.6.1 Update changes to system status on SCADA and other control room maps 5.6.2 Verify that all switching steps have been taken, that all tags have been placed, and that the resulting physical clearance will provide the Clearance Holder with a safe work distance for the purpose of the job 5.6.3 Log all actions taken on records and reports, including the following information: System Operator (Switching Controller)s name Station/location Circuit/equipment out on Clearance (include name and number) Voltage class Switching time Clearance number/version Purpose of clearance Field Crew Leader (Switch person)s name

5.6.3 (CCO) Log all actions taken on records and reports

D-5

Full Text of Task Descriptions used in this Report # Abbreviated Task Statement 5.6.5 (CCO) If switching has not been completed before the end of the shift, review the status of the job with the oncoming shift Full Text of Task Statement Number and type of grounding devices and their location Time actions were accomplished Special instructions pertaining to equipment involved in the clearance

5.6.5 If switching has not been completed before the end of the shift, document all work that has been completed before leaving the shift and review the status of the job with the oncoming shift 6.1.1 Call the System Operator (Switching Controller) upon arrival at the job site 6.1.4 If there are any technical uncertainties regarding the Outage Request and/or associated switching instructions, resolve as needed; do not proceed until the uncertainties are resolved 6.1.5 Verify the following with the System Operator (Switching Controller): System Operator (Switching Controller)s name Outage Request/Switching instructions number/version Purpose of work Identification of all energy sources Description of work Identification of all automatically and remotely controlled switches that could cause opened devices to close

6. Implement Switching For Removal of Lines or Equipment Field Actions 6.1.1 (ASP) Call the system operator upon arrival at the job site 6.1.4 (ASP) If there are any technical uncertainties regarding the Outage Request and/or associated switching instructions, resolve as needed; do not proceed until the uncertainties are resolved 6.2.1 (ASP) Verify job information with the System Operator, including identification of all automatically and remotely controlled switches that could cause opened devices to close

6.4.1. (ASP) Verify the technical accuracy of the switching instructions against a one-line and current conditions 6.4.2 (ASP) Verify devices to be switched are in the expected state before beginning switching

6.4.1. Verify the technical accuracy of the Outage Request and/or switching instructions against all available sources including a one line and current conditions 6.4.2 Verify that he following are in the expected state: Each device to be operated Each device to be impacted by switching Each supply to the devices being operated

6.4.3 (ASP) Review available logs to determine if there are any conditions that may impact the scheduled work

6.4.3 Review available logs to determine if there have been any recent changes to equipment, any recent unexplained alarms, or other conditions that may impact the scheduled work

D-6

Full Text of Task Descriptions used in this Report # Abbreviated Task Statement 6.4.4 (ASP) Conduct a Site Review of the existing conditions including locations of circuits and equipment, protective grounds, and equipment grounding conductors Full Text of Task Statement 6.4.4 Conduct a Site Review of the existing conditions including: Nominal voltages of lines and equipment and maximum switching transient voltages Presence of hazardous induced voltages Presence and conditions of protective grounds and equipment grounding conductors Condition of poles Environmental conditions relative to safety Locations of circuits and equipment, including power and communication lines and fire protective signaling circuits

6.5.6 (ASP) Perform switching for isolation according to switching instructions Used as default task for inadvertently opened in switching for removal of equipment, where one of its subtasks (below) is not be identified in the incident report. 6.5.6.1 (ASP) Anticipate the results of each switching step before taking any actions 6.5.6.2 (ASP) Before operating a device, touch the devices identification label and verify that the correct device has been selected 6.5.6.3 (ASP) Operate the device according to the switching instructions 6.5.6.4 (ASP) Visually confirm that each operation has been completed 6.5.8 (ASP) Apply locks/tags to all isolation points 6.5.11 (ASP) Notify the System Operator (Switching Controller) of problems or abnormalities detected during switching

6.5.6 Perform switching for isolation according to the System Operator (Switching Controller)s instructions

6.5.6.1 Anticipate the results of each switching step before taking any actions 6.5.6.2. Before operating a device, touch the devices identification label and verify that the correct device has been selected 6.5.6.3 Operate the device according to the switching instructions 6.5.6.4 Visually confirm that each operation has been completed by verifying changes in meters, by seeing a physical opening, etc. 6.5.8 Apply locks and/or tags to all isolation points 6.5.11 Notify the System Operator (Switching Controller) of the following Each step (or sequence of steps) that was completed Problems or abnormalities detected during switching Any abnormal conditions remaining at the site

6.6.2 (ASP) Install grounds or grounding devices

6.6.2 Install grounds or grounding devices

D-7

Full Text of Task Descriptions used in this Report # Abbreviated Task Statement 6.7.1 (ASP) Update the Substation Log Book/Records, including description of work performed, any abnormal conditions found, any abnormal conditions left Full Text of Task Statement 6.7.1 If switching took place in a substation, update the Substation Log Book/Records, including as many of the following as applicable: Name of Field Crew Leader (Switch person) 7. Issue a Clearance Control Room Actions 7.5.1 (CCO) Log data on clearance issued, including location & number of personal grounds placed by clearance holder 7.5.1 Log the following: Name of the clearance holder 8. Receive a Clearance Field Actions 8.1.5 (Work Crew) Conduct a Site Review of the existing conditions including locations of circuits and equipment, equipment grounding conductors & protective grounds 8.1.5 Conduct a Site Review of the existing conditions including: Nominal voltages of lines and equipment and maximum switching transient voltages Presence of hazardous induced voltages Presence and conditions of protective grounds and equipment grounding conductors Condition of poles Environmental conditions relative to safety Locations of circuits and equipment, including power and communication lines and fire protective signaling circuits Time clearance was issued Purpose of the clearance Location and number of all personal grounds Date and time of arrival Name of System Operator (Switching Controller) notified of entrance Purpose of entrance into substation Switching instructions Outage Request number and version Description of work performed Any abnormal conditions found Any abnormal conditions left Name of System Operator (Switching Controller) notified of abnormal conditions Departure time Signature and date (to indicate the work has been completed)

D-8

Full Text of Task Descriptions used in this Report # Abbreviated Task Statement 8.1.6 (Work Crew) Verify the adequacy of the clearance Full Text of Task Statement 8.1.6 Verify the adequacy of the clearance including the following: Clearance boundaries match the request 8.3.1 (Work Crew) Conduct a Work Briefing with the crew including work procedures & special precautions Equipment is properly isolated Clearance will provide a safe distance for the work to be done

8.3.1 Unless all crew members were present for a previous Work Briefing for the same job at the same work location, conduct a Work Briefing with the Crew that includes the following: Review the current circuit conditions Location of all grounds and grounding devices Note that a test was done to confirm deenergization Hazards associated with the job Work procedures involved Special precautions Energy source controls Personal protective equipment requirements

8.3.2 (Work Crew) Inspect/prepare tools and safety equipment 8.4.1 (Work Crew) Test to verify that lines or equipment are de-energized 9. Release a Clearance Field Actions 9.1.1 (Work Crew) Verify that scheduled work has been completed 9.1.2 (Work Crew) Verify system equipment returned to same configuration as found when accepting clearance

8.3.2 Inspect/prepare tools and safety equipment 8.4.1 Test to verify that lines or equipment are de-energized

9.1.1 Verify that scheduled work has been completed 9.1.2 Verify that electric system equipment is returned to the same configuration as found when accepting the clearance (i.e., breakers and switches are left in the same position as found) 9.1.5 Remove personal grounds installed by the crew as follows: 9.1.5.1 Use a live line device to remove the ground from the line or equipment 9.1.5.2 Then remove the ground-end connection

9.1.5 (Work Crew) Remove personal grounds installed by the crew

10. CCO Accept the Release of a Clearance Control Room Actions 10.3.4 (CCO) Update all displays, logs and reports to indicate the release of the clearance 11. Release a Clearance in an Emergency This situation not involved in any of the events in this report 10.3.4 Update all displays, logs and reports to indicate the release of the clearance

D-9

Full Text of Task Descriptions used in this Report # Abbreviated Task Statement 12.1.5 (CCO) Assemble and review copies of outage request, switching instructions, maps and one-lines 12.1.6 (CCO) Review available logs to determine if there have been any recent changes or other conditions that may impact the switching instructions 12.1.10 (CCO) Review the current status of equipment from all available sources (mapboard, SCADA, logs, etc.) to verify that current status will not adversely impact the switching 12.1.11 (CCO) If there are any technical uncertainties regarding the switching instructions, do not proceed until they are resolved 12.3.2.3 (CCO) If other entities will be impacted or involved, notify them that the work is/is not proceeding as scheduled 12.3.3 (CCO) Verify any prerequisite switching/load transfer has been done Full Text of Task Statement 12.1.5 Assemble and review copies of Outage Request, switching instructions, maps, and one lines 12.1.6 Review available logs to determine if there have been any recent changes to equipment,, any recent unexplained alarms, or other conditions that may impact the switching instructions 12.1.10 Review the current status of equipment from all available sources (mapboard, SCADA, logs, etc.) to verify that current status will not adversely impact the switching 12.1.11 If there are any technical uncertainties regarding the switching instructions, do not proceed until the uncertainties are resolved 12.3.2.3 If other entities will be impacted or involved, notify them that the work is/is not proceeding as scheduled 12.3.3 If load transfers or any other switching is a pre-requisite to the switching, coordinate switching associated with load transfers and verify that load transfers have taken place 12.4.1 Review all required information with the Field Crew Leader (Switch person) before starting the job: Verify that the Field Crew Leader (Switch person) receiving the instructions is the person who was designated to receive the instructions Confirm communication tool to use (phone number, radio channel) Confirm possession of a copy of the Outage Request and/or associated switching instructions Review the purpose of the switching activity Verify work location(s) including the name and location of circuits or equipment involved Verify the time of the activity Review any abnormal situations that may impact the switching Review the sequence of switching against maps and one-lines Verify the minimum clearance distance needed for safety Verify isolation points and associated tags

12. Implement Switching for Return of Lines or Equipment Control Room Actions

12.4.1 (CCO) Review scope of job with the Field Crew Leader (Switch person) before starting the job

D-10

Full Text of Task Descriptions used in this Report # Abbreviated Task Statement 12.5.6 (CCO) If there are uncertainties regarding the switching, do not proceed until they are resolved. 12.6.1.2 (CCO) Switch equipment via SCADA . . Operate the device 12.6.5 (CCO) Monitor watt/var/volt/amp values while switching instructions are executed 12.7.1 (CCO) Update changes to system status on SCADA and other control room maps Full Text of Task Statement to be removed Review the location and type of grounds and grounding devices to be removed

12.5.6 If there are uncertainties regarding the switching, do not proceed until they are resolved. 12.6.1 Switch equipment via SCADA as required by the switching instruction 12.6.1.2 Operate the device 12.6.5 Monitor watt/var/volt/amp values while switching instructions are executed 12.7.1 Update changes to system status on SCADA and other control room maps

13. Implement Switching for Return of Lines or Equipment Field Actions 13.2.2 (ASP) Review switching to be done, including tags & grounds to be removed 13.2.2 Review the following: Switching to be done 13.4.1 (ASP) Verify accuracy of instructions against a one-line and current conditions Any unusual conditions that may affect switching Location of all isolation points Tags to be removed Grounding devices to be removed

13.4.1 Verify the technical accuracy of the switching instructions against all available sources including a one line and current conditions 13.4.2 Review available logs to determine if there have been any recent changes to equipment, any recent unexplained alarms, or other conditions that may impact the scheduled work 13.4.4 Conduct a Site Review of the existing conditions including: Nominal voltages of lines and equipment and maximum switching transient voltages Presence of hazardous induced voltages Presence and conditions of protective grounds and equipment grounding conductors Condition of poles Environmental conditions relative to safety Locations of circuits and equipment, including power and communication lines and fire protective signaling circuits

13.4.2 (ASP) Review available logs to determine if there are any conditions that may impact the scheduled work

13.4.4 (ASP) Conduct a Site Review of the existing conditions including locations of circuits & equipment, protective grounds, and equipment grounding conductors

13.4.6 (ASP) If field conditions do not agree with

13.4.6 (ASP) If field conditions do not agree with

D-11

Full Text of Task Descriptions used in this Report # Abbreviated Task Statement the outage request and/or switching instructions, notify the System Operator) 13.5.5 (ASP) Perform switching according to instructions 13.5.5.3 (ASP) Operate the device according to the switching instructions 13.5.5.4 (ASP) Visually confirm that each operation has been completed by verifying change in meters, by seeing a physical closure, etc. 13.5.8 (ASP) Visually confirm operations have been completed and that conditions as they should be after all switching completed 13.6.1 (ASP) Notify the System Operator of problems during switching or abnormal conditions at the site Full Text of Task Statement the Outage Request and/or switching instructions, notify the System Operator (Switching Controller) 13.5.5 Perform switching according to the System Operator (Switching Controller)s instructions 13.5.5.3 Operate the device according to the switching instructions 13.5.5.4 Visually confirm that each operation has been completed by verifying change in meters, by seeing a physical closure, etc. 13.5.8 Visually confirm operations have been completed and that the condition is as it should be 13.6.1 (ASP) Notify the System Operator (Switching Controller) of the following: Each step (or sequence of steps) that was completed 14. Review and Close Outage Request This situation not involved in any of the events in this report End of Table All tags that were removed The number and type of grounding devices that were removed Problems or abnormalities detected during switching Any abnormal conditions remaining at the site

13

1 1

D-12

E
INCIDENTS INVOLVING RELAY TECHNICIANS
The tasks performed by relay technicians (RTs) are not addressed by the generic JTA. However, errors in these tasks are a source of many incidents involving switch operations triggered by relays, and are usually included in any collection of switching incidents. While these may or may not be considered switching errors they are errors that have an impact on system configuration, and may result in unplanned switching to restore the system to its intended state. It is therefore useful to examine the RT errors. The relay incidents in the data reviewed for this study are presented in Table E-1, below. In Table E-1, the numbers in brackets refer to the numbered category to which the incident was assigned in Table E-2 (page E-7), which summarizes the common errors.
Table E-1 Incidents Involving Relay Technicians (RTs) Event # R1 [2] Errors and Contributing Factors While performing routine relay test at Tie the Bank 1, 2, 3 HT was inadvertently tripped. While attempting to by-pass Bank 3 L.T. CB the wrong Bank differential was blocked. The ASP blocked Bank 1 differential and should have blocked Bank 3. He was on the wrong panel. Failed to close H-tie OCB & inadvertently opened feeder breaker while performing relay checks. Feeder interrupted: RT directed by CCO to perform control checks, placed VCB in test position without prior setting of Automatic switch to Manual. Report mentioned failure to review station instructions & also possibility that more detailed switching orders might have prevented incident. Error & Task in which Error Occurred 6.4.5 (ASP) Perform switching for isolation according to switching instructions & visually confirm that each operation has been completed

R2 [7] R3 [1], [3]

UNCLEAR

6.1.5 (ASP) Verify the job information with the System Operator (Switching Controller), including identification of all automatically and remotely controlled switches that could cause opened devices to close (ASP) 6.3.1 (ASP) Verify the technical accuracy of the Outage Request and/or switching instructions against a one-line and current conditions(ASP) 3.4.1 (CCO) Write switching instructions in sequence, including disabling of protective relaying (CCO)

R4 [1], [3]

Same feeder, same crew, same day, feeder again interrupted due to failure to properly set test switches. Blamed on haste, failure to review prints [1].

6.3.1 (ASP) Verify the technical accuracy of the Outage Request and/or switching instructions against a one-line and current conditions (ASP)

E-1

Incidents Involving Relay Technicians

Table E-1 Incidents Involving Relay Technicians (RTs) (Continued) Event # R5 [4] R6 [4] R7 [5] R8 [2] R9 [1] Errors and Contributing Factors Test equipment connected incorrectly, causing fuse to blow. Lifted wire inadvertently contacted control wire to trip relay caused OCB to open. Improper relay setting due to use of improper calibration data. Inadvertently tripped transformer OCB when performing relay check: no details. RT inadvertently failed to reopen switch closed for testing: tripped several feeders. Error & Task in which Error Occurred Not addressed by TA Not addressed by TA Not addressed by TA Not addressed by TA 9.1.4 (Work Crew) Verify system equipment returned to same configuration as found when accepting clearance (Work Crew) Not addressed by TA

R10 [4]

Test lead clip inadvertently contacted second terminal after initial terminal was clipped, tripping transformer & whole station because other transformer already out. Terminals apart, & design at least partly to blame. While removing test lead, accidentally shorted it to ground, blowing fuse & activating relay (report says the station did not need the elaborate protection that it had, so they recognize a design error in here too, but the immediate cause of event was the short). RTs did not completely isolate output of CT under test, current operated over current relay. Gigged for not reviewing control wiring print in transformer control cabinet, which showed field wiring responsible for problem. Note that the term field wiring implies wiring was not normal. Automatic transfer switches left in ON position following switching by SCADA. RT gigged for failing to check positions prior to starting work. Report notes CCO & RT failed to discuss the automatic transfers & restoration switching. (Note switches were left in abnormal position by someone who preceded him in the station, but report does not mention investigating how switches got to be in abnormal position). Feeder tripped while installing under frequency relays. Prints used found to be incorrect.

R11 [4]

Not addressed by TA

R12 [2], [3]

Not addressed by TA

R13 [1]

ASP, CCO, RT 6.4.6 (ASP) If SCADA controlled equipment was placed in local control, return to remote control 8.3.1 (Work Crew) Conduct a Work Briefing with the crew including work procedures & special precautions (receiving clearance) Not addressed by TA (in any event, this was a case of a trap in the form of incorrect prints)

R14 [6]

E-2

Incidents Involving Relay Technicians

Table E-1 Incidents Involving Relay Technicians (RTs) (Continued) Event # R15 [5] Errors and Contributing Factors Tripped due to incorrect circuit wiring when DMF relay replaced 9 mo earlier. Protection not load-checked at time of installation: should have been. Implies relay checking at one station triggered ACB operation at second station. Cant identify specific error from report. RTs attached jumper to wrong points when restoring installation after CT test. Did not use prints when doing restoration: believed that length & formation of (old) wires prevented incorrect reinstallation. RTs failed to isolate one (of several) trip paths when testing stuck breaker trips. Report noted that new relays had test switches in the transformer relay cabinet rather than available test poles in the in the bus differential relay cabinet, where all other test switches are located. Report also noted that technician fatigue might have been a contributing factor. Noted also that there was an error in the timers, which made them operate very quickly. RTs failed to disable stuck breaker protection prior to testing feeder overcurrent relays: tripped out mobile transformer. Wiring error caused stuck breaker timer to actuate when tie breaker Auto/Non-Auto switch placed in Auto. Report recommended that stuck breaker protection be disabled when working on potentially sensitive equipment. Also noted severe time pressure from contiguously scheduled outages & crew working long hours. Transformer breakers opened automatically when paralleled to circuit. New relay not wired to CTs properly. Corrected problem & same thing happened second time. First blamed on failure of RTs to recognize required changes to wiring: second blamed on failure to perform thorough check of circuit before attempting cutin. Note that breaker phase orientation not shown correctly on drawings from which they were working, & not recognized by RTs to be incorrect. Feeder tripped due to incorrect setting on backup feeder overcurrent relay. No detail on how this occurred. N/A outside scope of TA Error & Task in which Error Occurred N/A: out of scope of TA

R16 [3?] R17 [3], [4]

UNCLEAR

13.4.1 (ASP) Verify accuracy of instructions against a one-line and current conditions

R18 [2]

UNCLEAR

R19 [2] R20 [5]

R21 [5], [6]

N/A outside scope of TA

R22 [5]

N/A outside scope of TA

E-3

Incidents Involving Relay Technicians

Table E-1 Incidents Involving Relay Technicians (RTs) (Continued) Event # R23 [1], [3] Errors and Contributing Factors Transformer ABS opened during relay checks on second transformer. Crew failed to open appropriate test switches for conditions. Crew apparently assumed that test switches did not have to be opened because the transformer whose protection they were testing was out of service and isolated. Apparently failed to thoroughly review appropriate circuit diagrams. Tie breaker inadvertently tripped while performing relay checks on feeder relaying. Failed to realize that closed tie breaker created unusual conditions in control scheme. Restored promptly, apparently before contacting CCO. Error & Task in which Error Occurred 13.4.1 (ASP) Verify accuracy of instructions against a one-line and current conditions

R24 [2]

Possibly result of incomplete review of unusual conditions, e.g., 6.1.5 (ASP: Verify the job information with the System Operator (Switching Controller), including identification of all automatically and remotely controlled switches that could cause opened devices to close), but not gigged for it Not addressed by TA 13.4.1 (ASP) Verify accuracy of instructions against a one-line and current conditions Note: JTA does not include an equivalent of 5.3.5 (CCO: If there are uncertainties regarding the switching, do not proceed until they are resolved) for the ASP or the work crew Not addressed by TA

R25 [8] R26 [1], [6]

RT bumped controller for breaker, causing it to open. RTs failed to properly identify circuit that would operate the opposite feeder breaker, & thus did not open the appropriate test switch. Noted problems with some of the drawings used for job. Gigged for not stopping & resolving problem (questioning attitude-STAR) when they became aware of deficiencies in the drawings, & not conducting final review of all test switches prior to beginning testing. RT inadvertently executed wrong point on RTU, causing feeder OCB to open: used wrong stations RTU point assignment sheet when performing the work. Meter Engineering shorted out secondary wire while removing recorder. Employees not aware of recently developed procedures for this type of work. Communications failure blamed. Improperly adjusted stuck breaker timer? Cant identify error from report. Unexpected breaker operation at power plant while commissioning newly installed carrier set and relay. The activity bridged positive voltage to the breaker trip circuit. Red cutout switches did not completely isolate relays (a design peculiarity).

R27 [3]

R28 [9]

8.3.1 (Work Crew) Conduct a Work Briefing with the Crew including work procedures & special precautions UNCLEAR Design or documentation error? Report did not fault the technicians on the spot.

R29 [5] R30 [10]

E-4

Incidents Involving Relay Technicians

Table E-1 Incidents Involving Relay Technicians (RTs) (Continued) Event # R31 [1] Errors and Contributing Factors While performing functional test on new relays, one operated and interrupted bus. Crew failed to open appropriate trip cut-out switches before performing test. Engineer had all switches open while wiring his relays, assumed they were left open after application of relay settings & did not verify their positions. Also suggests work plan not appropriately detailed IAW company policy. R32 [11] While performing routine trip checks, the technician accidentally cycled a 500kV breaker at a power plant. In troubleshooting the cause for a breakers failure to close during the test, the technician removed and reconfigured the test block. However, instead of re-inserting the test block cover, he mistakenly inserted the normal cover for the trip cutout block. When he initiated a test of the breaker he was testing, relay operation caused the second breaker to cycle. The work was performed according to a detailed written test plan. The utility cited inadequate plan as the cause of the incident. During a functional test of relays at a generating plant, a maintenance technician manually initiated an overcurrent trip on a 169kV bank breaker. He intended to trip the generator breaker. Point out 1 initiates the generator breaker trip. He selected instead point Relay 1 which initiated out 3 which tripped the bank breaker as well as the generator breaker. The technician had no experience in performing trip checks on the type of relays being checked. The utility faulted supervision for not performing an independent review of the written work plan prepared by the technician, and changed the guidelines for such plans to require independent review. They also revised the DC schematics to clarify the relationship between relay numbers and output numbers. During continuity test, Relay Dept placed jumper from terminal block to ground. Caused inadvertent operation of bus differential. May not have been designed & wired in accordance with standard practice. Fix was design change. P&C personnel disabled one transfer trip, but customer had a redundant system apparently unknown to the RTs and not standard on company equipment. Called operating error, probably really design error Simple slip? Error & Task in which Error Occurred ??

R33 [2], [6]

R34 [10]

R35 [10]

Bagged?

E-5

Incidents Involving Relay Technicians

Table E-1 Incidents Involving Relay Technicians (RTs) (Continued) Event # R36 [2] R37 [2] R38 [4] R39 [2] Errors and Contributing Factors RTs inadvertently tripped bus differential. Reported knife switches to relays not opened (by people preparing station for work?) RTs working on calibrating relays forgot to open knife switches, tripped 138kV breaker in another station. RT accidentally tripped 138kV CB when cleaning contacts. P&C tech was having trouble getting breakers to open via manual trip lever. He noticed transformer was not humming and decided to open the transformer MOAB switch while it was de-energized. It became energized while he was doing this and the MOAB interrupted the load. Technician did not ask for permission to use the MOAB and did not inform transmission switching center of his intention to do so. P&C techs were performing trip tests. In the course of the work, they discovered a defective light socket and called an electrician to fix it. While waiting for him to arrive, they decided to continue with their trip tests. One isolated the appropriate circuits and then opened a relay other than the one he had just cleared for testing. This tripped the station. Technician then closed breaker to restore the station on his own authority (apparently the company allows this under special circumstances). Two engineering techs were calibrating relays. Apparently checked prints for circuit of CB-I instead of CB-J. Second tech inserted current installed in a GE test plug. Writeup says techs failed to realize that the particular relays they were working on operated the breaker and were not isolated by the GE test plug. Performing planned relay trip testing on 138kV CB, RTs had set up a simulated breaker. Closed knife switch but tripped real rather than simulated breaker. Blamed in part on the labeling and layout of the knife switches. P&C personnel were removing old control cables associated with transformer that had been removed. Accidentally cut a DC cable and a cable to an active trip circuit breaker cable, causing the breaker to trip & interrupt one industrial customer (restored within 30 seconds). Not planned switching 6.3.5 (ASP) document changes? Not good fit +6.4.5 Error & Task in which Error Occurred ASP & RT Unclear But: 6.4.5 tentative, since no mention of plan

R40 [12]

Work crew error, operated switch other than intended

R41 [3]

Work crew error, maybe knowledge related. Different kinds of relays had different circuits? See p-18 of write-ups

R42 [2]

R43 [13]

Slip? Inattention?

E-6

Incidents Involving Relay Technicians

Table E-1 Incidents Involving Relay Technicians (RTs) (Continued) Event # R44 [2] Errors and Contributing Factors P&C personnel reached into an open relay housing to reset a dial, and inadvertently moved the dial to the point where contacts made up and tripped the breaker. Did not pull the relay plug before reaching into it. RT operated wrong breaker while testing CB. 138kV CB at station operated once due to substation relay testing. RTs transfer tripped customer (supplier?) during relay test. While placing new GCB in service, failed to cut off the North Differentials when phasing in North simi-dif relays. 12kV CB tripped open. While removing relay from service, RT dropped it causing it to operate. Substation electrician was to test trip a 238kV CB. Test tripped the breaker that was feeding the station rather than the one that had been opened for testing. Jury-rigged drain shield intended to direct water dripping from a frozen air conditioner away from the relay panel fell and hit the breaker failure timer rely, causing a transfer trip on a 245kV line and tripping one unit at the plant off line. Plant was testing relays, caused CB to trip open. When performing a transfer trip test, technician failed to put control handle in test position. CB tripped while RT doing trip testing. RT had disabled the incorrect aux relay because the relay diagram print was incorrect. RT did not disable proper relays before performing maintenance on bus PTs. One 138kV breaker opened & reclosed automatically. RTs had failed to close a carrier relay test switch. When putting 345kV line back in service, one CB tripped back out when it was closed. When testing a breaker that had failed to open on command, an adjacent breaker tripped. Tecs had not pulled necessary leads to the other breaker and when the breaker failure was simulated it caused the second breaker to trip. Error & Task in which Error Occurred Slip, made consequential by sloppy procedure of not disabling relay before placing hands into it

R45 [12] R46 R47 R48 [2] R49 [8] R50 [12]

6.4.5 (ASP) Unusable, no detail Unusable, no detail Check report

Simple slip 6.4.5 (ASP) default

R51

Unusable, not related to switching

R52 R53 [2] R54 [6] R55 [2]

Unusable, no detail

Bagged by incorrect print

R56 [1] R57 [2]

E-7

Incidents Involving Relay Technicians

Table E-1 Incidents Involving Relay Technicians (RTs) (Continued) Event # R58 Errors and Contributing Factors During an operational test of an Intertie Transfer Tripping scheme, incomplete notification and coordination resulted in several breakers tripping at a substation, which subsequently tripped one unit at 136 MW and disabling one 230kV transmission line. R59 When taking break off MOAB to remove DC voltage, also removed voltage from a relay that [10] then tripped five 138kV breakers. Relay wiring subsequently found to be inconsistent with standard design drawings: was rewired to be consistent. End of table Error & Task in which Error Occurred Unusable, insufficient detail, no followup report

Unusable, Bagged

The RT errors listed above are summarized in the table below. The numbers in the Examples column identify the individual incidents from the table above, which provides a more detailed description of each incident. Again, note that a single incident may involve multiple errors. Items 1, 2, and 3 in Table E-2 accounted for the majority of RT errors. In a sense, all three could be said to relate to preparation for the job, or information processing, as opposed to actually performing the motions required. In contrast, items 4, 8, and 13 (about 1/6 of the total number of incidents examined) seem to be simple slips, and may be harder to guard against by traditional means such as increased training or supervisory oversight. It is of interest that Items 5, 6, and10, which account for 15 (about ) of the 59 incidents examined involved things that were not in the control of the individuals performing the work at the time the reported incident occurred. That is to say, a large proportion of the accidental relay activations in this sample were the result of traps. We did not examine a task analysis of the activities performed by relay technicians. However, we believe that a comparison of errors to tasks would identify a small set of tasks that could benefit from increased training or perhaps a revision of the tools or techniques with which the tasks are preformed, for example in the case of inadvertent shorts.

E-8

Incidents Involving Relay Technicians

Table E-2 Summary of Errors in Incidents Involving Relay Technicians (RTs) Error type [1] Failed to open/close proper test switches (may actually be the same as next item, different level of specificity) [2] Improper isolation of equipment from protective scheme prior to operation for tests [3] Failed to consult circuit diagrams/used wrong circuit diagram for equipment in question [4] Inadvertent shorts, faulty test equipment connections [5] Incorrect wiring or relay settings [6] Circuit diagrams or other documentation contained errors or was not clear [7] Failed to perform switching needed to maintain continuity of service while testing equipment [8] Bumped control switch or sensitive equipment, causing it to operate [9] Failed to follow appropriate procedure for the kind of work being done [10] Design peculiarities (e.g., not in accordance with companys standards or usual layouts) [11] Used wrong test block for isolation of equipment [12] Operated wrong switch [13] Physically damaged equipment, resulting in activation of relays Examples from Table E-1 113, 23, 26, 31, 56 # 5

1, 3, 4, 8, 9, 12, 18, 19, 24, 33, 36, 37, 39?, 42, 44, 48, 53, 55, 57 3, 4, 12, 16?, 17, 23, 27, 41

19

5, 6, 10, 11, 17, 38

7, 15, 20, 21, 22, 29 14, 21, 26, 33, 54

6 5

25, 49

28

30, 34, 35, 59

32 40,45, 50 43

1 3 1

E-9

Export Control Restrictions Access to and use of EPRI Intellectual Property is granted with the specific understanding and requirement that responsibility for ensuring full compliance with all applicable U.S. and foreign export laws and regulations is being undertaken by you and your company. This includes an obligation to ensure that any individual receiving access hereunder who is not a U.S. citizen or permanent U.S. resident is permitted access under applicable U.S. and foreign export laws and regulations. In the event you are uncertain whether you or your company may lawfully obtain access to this EPRI Intellectual Property, you acknowledge that it is your obligation to consult with your companys legal counsel to determine whether this access is lawful. Although EPRI may make available on a case by case basis an informal assessment of the applicable U.S. export classification for specific EPRI Intellectual Property, you and your company acknowledge that this assessment is solely for informational purposes and not for reliance purposes. You and your company acknowledge that it is still the obligation of you and your company to make your own assessment of the applicable U.S. export classification and ensure compliance accordingly. You and your company understand and acknowledge your obligations to make a prompt report to EPRI and the appropriate authorities regarding any access to or use of EPRI Intellectual Property hereunder that may be in violation of applicable U.S. or foreign export laws or regulations.

About EPRI EPRI creates science and technology solutions for the global energy and energy services industry. U.S. electric utilities established the Electric Power Research Institute in 1973 as a nonprofit research consortium for the benefit of utility members, their customers, and society. Now known simply as EPRI, the company provides a wide range of innovative products and services to more than 1000 energyrelated organizations in 40 countries. EPRIs multidisciplinary team of scientists and engineers draws on a worldwide network of technical and business expertise to help solve todays toughest energy and environmental problems. EPRI. Electrify the World

Program: Substations

1008692

2004 Electric Power Research Institute (EPRI), Inc. All rights reserved. Electric Power Research Institute and EPRI are registered service marks of the Electric Power Research Institute, Inc. EPRI. ELECTRIFY THE WORLD is a service mark of the Electric Power Research Institute, Inc. Printed on recycled paper in the United States of America

EPRI 3412 Hillview Avenue, Palo Alto, California 94304 PO Box 10412, Palo Alto, California 94303 USA 800.313.3774 650.855.2121 askepri@epri.com www.epri.com