A Road To Successful Integrated DCS Upgrades.

Tim Isaacs San Onofre Nuclear Generating Station (SONGS) Southern California Edison

Preview (why we are here).

Description of the SONGS DCS Upgrade. SONGS DCS Team. Maintenance DCS Team.
Planning & implementation. Procedure Development. Personnel training and qualification.

Lessons learned.

SONGS Controls upgrade

The Feed Water Control and Turbine Control systems at SONGS were becoming unreliable due to component aging and replacement parts were difficult to procure. An Ovation DCS was selected replace the existing Feed Water Control, Main Turbine Control and Turbine Trip systems.

SONGS FWCS and TCS Upgrade

Install an Ovation network backbone and Ovation Operator and Server Drops. Feed Water Control System:
Remove the legacy control cabinets. Install Ovation cabinets (controller and extended I/O). Rework control boards to remove legacy control stations and indication. Install touch screens and SLIMS. Modify the existing plant instruments to provide isolated redundant inputs to the DCS.

SONGS FWCS and TCS Upgrade

Turbine Control System.
Gut existing cabinets (5). Install Ovation Controllers and I/O in three of the cabinets. install Fan-Out network hardware for Unit DCS in one cabinet. Perform Control Room modifications.

The SONGS DCS Team

The scope of this control upgrade was such that SONGS managers determined that key players and end users should be involved in all phases of the project, from concept to installation. Previous projects had suffered from the effects of Silos. Management thought that a team structure would help to bring together isolated groups and provide for a more efficient high quality project.

The SONGS DCS Team

The SONGS DCS Team consisted of representatives of the following groups:
Design Engineering. Maintenance Engineering. Procurement. Maintenance. Operations. Vendor (Westinghouse Nuclear).

The SONGS DCS Team

The SONGS DCS Team consisted of representatives of the following groups:
Project Management and Engineering Services. (Construction). Plant Management. Maintenance. Computer Engineering. Training.

DCS Team members served as the lead for their respective group in project activities.

Maintenance DCS Team

Team mission:
Provide error free implementation of the upgrade. Ensure that I&C personnel are qualified to install, operate and maintain the DCS equipment. Perform all support operations for the project; planning, procedure development, document review and engineering support.

Maintenance DCS Team

Team mission:
Perform all on and offsite testing. Provide timely feedback to maintenance management. Provide for a transparent product. Provide clear and open communications with all other groups in the SONGS DCS Team. Provide a core group to allow for future growth of maintenance personnel.

Maintenance DCS Team

Members:
Supervisor (Team Lead). 6 I&C Technicians
4 assigned for full time from outage M-6. 2 assigned part time.

Procedure writer.
Part time assignment to mentor the Maintenance Team in procedure development.

Maintenance Team Challenges

First time evolution.
No experience in performing an upgrade of this scale. Industry experience was very limited (nuclear). The Technicians were not planners or procedure writers.
We had to learn as we went on everything.

Maintenance Team Challenges

Team approach was unfamiliar to the Maintenance Team.
True team effort everyone supports everyone else. Most decisions to be made at the lowest possible level. First time not being told what & how to do it. Members given authority commensurate with responsibilities.

Maintenance Team Challenges

High Risk Evolution.
New team was given broad responsibility and authority, failure to meet objectives could have lead to project delay. SONGS Maintenance management took the risk of this new approach. High work load on the team, coupled with inexperience was not indicative of success.

Maintenance Team Challenges

Why did SONGS managers take this risk?
Previous upgrades to plant equipment were performed by a Construction group utilizing contract personnel.
In general these tasks were performed in a high quality manner. The expertise gained in system installation left site with the contractors.

Maintenance Team Challenges

Why did SONGS managers take this risk?
In general training and procedure development were not considered core requirements to project completion. Many systems were still lacking procedures and training long after startup.

Maintenance Team Challenges

Why did SONGS managers take this risk?
A project with the scope and significance of the DCS upgrade, would require that organic SONGS personnel, have the skills and knowledge to maintain the systems.

Maintenance Team Challenges

SONGS Management determined that SONGS organic personnel performing the upgrade would:
Provide a skilled core group of maintenance personnel to maintain the system after startup. Provide for Integrated implementation.

Maintenance Team Challenges

Why did SONGS managers take this risk?
SONGS Management determined that SONGS organic personnel performing the upgrade would provide for integration of:
Training. Qualification. Procedures. Preventative Maintenance Programs.

Lessons Learned.
Communicate. Ovation Certification of key players early in the program leads to success. An offline system is essential. Put Point IDs for I/O on Loop Drawings / Wiring Diagrams, this provides a tie between Ovation and the plant. Talk to those who have done it before.

Lessons Learned.
Configuration Control is job 1. Your Vendor is your friend (if not, well I am not going there). Keep responsibility commensurate with authority. Establish Roles and Responsibilities early and do not be afraid to change them if necessary.

Lessons Learned.
Cyber-Security is important. Really important. Open communications with the CS group is essential. Use the Factory Acceptance Test Free Form or Customer testing. Build Human Performance into procedures.

Contact
Tim Isaacs
I&C Supervisor isaacstm@songs.sce.com 949-368-7937

Mike Root
Maintenance Engineering rootrm@songs.sce.com

Cheryl Atooli
Design Engineering atoolica@songs.sce.com

Q and A