Você está na página 1de 29

57

th
Annual ISA Power Industry Division Symposium
2-4 June 2014, Scottsdale, Arizona
Hilton Scottsdale Resort 1 1
David Herrell and Kyle Dittman
MPR Associates

Bob Cardwell
Southern Nuclear
Replacing an Obsolete Software-
Based Module with an FPGA-
Based Module
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
2
Author Short Biography Slide
David Herrell is an Executive Engineer at MPR Associates,
Alexandria, VA with 35+ years of nuclear digital I&C experience
Part of MPRs senior I&C technical staff, he works with
suppliers and nuclear power plants around the world
Worked for a system supplier, as a seconded contractor, and
as a utility employee at Salem and Hope Creek prior to MPR
Bachelors and Master degrees in Electrical Engineering
Member of IEEE Nuclear Power Engineering Committee
(NPEC), member and current chair of Subcommittee 6 on
Safety Systems, and member of Working Group 6.4
responsible for IEEE Std. 7-4.3.2
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
3
Background
In the 1980s, Edwin I. Hatch
Nuclear Power Plant
replaced the electromagnetic
timers on the Unit 2 EDGs
with commercial software-
based equipment
Three cabinets of equipment
were installed, each
containing 1 dc-to-dc
converter, terminal blocks, 2
control modules, 2 alarm
relay outputs, and 2 counters
with relay interfaces to the
counters
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
4
Background (contd)
Multiple failures and obsolescence concerns of Rochester
Instrument Systems (RiS) modules initiated a project to
generate form, fit, and function replacements for the control
modules
SNC awarded a contract to MPR to re-engineer and provide
replacement modules as basic components
MPR and SNC decided to base the replacement module
architecture on a field programmable gate array (FPGA)
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
5
Background (contd)
The equipment was reverse-engineered and implemented on
an FPGA-based module
The Product Design (PD) Group has been developing FPGA-
based designs for medical devices under FDA regulations
The Nuclear Group used the PD Groups capabilities, by
adapting the PD Group plans, procedures, and instructions
into a safety-related Programmable Logic lifecycle process
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
6
Background (contd)
Early in the project, MPR considered the possibility of relying
solely on 100% testable and tested logic, but decided to
continue with IV&V as good engineering practice
MPR had Gavial Engineering and Manufacturing procure
components, assemble, solder, and preliminary test the
modules under their 10CFR50 Appendix B compliant Nuclear
QA program
The use of commercial components required the performance
of commercial grade dedication activities on the fabricated
module
Commercial grade dedication was performed per our 10CFR50
Appendix B compliant Quality Assurance Program


57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
7
Existing Design

57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
8
Existing Chassis


57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
9
Analysis of Existing Design
Only limited design documents for the original modules were
available, including cut sheets, limited functional design
descriptions, and schematics from Rochester Instrument
Systems
Having schematics for modules and for cabinet wiring
eliminated the need for extensive wire tracing and generation
of replacement prints
Verification (as-built) of the cabinet schematics was performed
The functional design descriptions were adequate to
determine how the module worked in sufficient detail to avoid
the need to disassemble the Fairchild F8 microprocessor
software

57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
10
Analysis of Existing Design (contd)

SNC provided a spare working cabinet for use in the design
activities
Much of the software documentation (e.g., flowcharts) had
little value other than showing how the module worked
FPGA implementation does not include problematic software
features such as sequential instructions, jumps, multitasking, and
hardware interrupts
It was determined that building a generic replacement was not
appropriate
Replacement module functions were customized
Features of the OEM module were not needed
DG safety function could be simplified

57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
11
Re-Engineered Requirements
Unused inputs and outputs (12 inputs to 3, 12 outputs to 10),
were eliminated
Reduced complexity,
Increased reliability (fewer parts),
Reduced power consumption, and
Allowed for enhanced diagnostics
Diagnostics enhanced to verify that the output relay coils have
continuity, rather than just checking that the output switch
turns on or off
Diagnostics use the inductive characteristics of the relay coil to
check continuity
Diagnostics considered active (inject current) or passive
(monitor voltage)
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
12
Re-Engineered Requirements (contd)
System/Plant external wiring limits the extent of diagnostics
Existing OEM module was designed for minimal EMC
Design constraints for new EMC requirements; meets United
States Nuclear Regulatory Commission Regulation Guide
1.180 requirements including:
Electrically fast transients requirements,
Electrostatic discharge requirements, and
Surge withstand requirements
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
13
System, Hardware, and Software
Requirements
A single document was written for both module hardware and
Programmable Logic requirements
Many of the detailed design decisions made during the
original modules design are now constraints on the
replacement module (e.g., module size, electrical connection,
and pinout; front panel size; chassis arrangement and wiring)
Requirements were created to address issues with the original
design (e.g., weak ground connection from module to chassis,
minimal distance from module to chassis necessitated paper
insulator to protect OEM module contact with chassis)
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
14
System, Hardware, and Software
Requirements (contd)
There was no attempt to recreate the original generic module
requirements, as the module is being used for a single
purpose in a single plant
Programmable Logic architecture created, showing how the
parallel action embedded in the logic actually functions
Requirements and detailed design iterated to point where
VHDL code implementation and module schematic could be
started
Design iteration continued through completion of
implementation, with final passes to resolve any remaining
IV&V clarity issues
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
15
System, Hardware, and Software
Requirements (contd)
MPR performed hazards analysis throughout life cycle, to
inform the design, implementation, and V&V processes
hazards external to the replacement module could not be
resolved
Hazards that could not be resolved involved constraints in the
existing design, which were present from the initial installation;
no new hazards were added
Hazards analysis activities augmented the testing program by
verifying that all hazards for which testing could be performed
were included in the testing program, and that those which
could not be tested were reviewed independently
Routine surveillance testing covers the external hazards to the
replacement module
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
16
Software Tools
Software tools are at least equally important for FPGAs as for
software-based devices
Programmable logic requires evaluation of internal FPGA
signal timing, which cannot be externally measured
Only way to evaluate internal timing is by use of simulation
and timing evaluation software tools
Internal timing verification cannot depend on testing, since
hardware may work while violating vendor internal timing
constraints (e.g., setup, hold)
Most vendors provide frequent updates to their software tools,
which should be considered for use, as tool errors are
corrected (important) in addition to new FPGA support

57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
17
Software Tools (contd)
MPR uses National Instruments LabVIEW for IV&V testing
and for equipment qualification testing
Custom LabVIEW application designed, verified, and validated
as a means of stimulating the module and measuring,
recording, and analyzing the modules response
MPR also uses a tool to generate requirements traceability
matrices
Automatic generation of RTMs based on metadata tags
embedded in documents eliminates the pain associated with
manual generation and correction of generation errors
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
18
Prototype Redesign


57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
19
Design Enhancements


As a basic component form, fit, and function replacement,
enhancements have to fit within the module and cannot
require external change; sensible enhancements are not
precluded
Increase in computed MTBF based on the design changes
Original module required replacement of EPROM for each
unique program and timing sequence
Replacement module has switch selectable sequence, one
module with three selectable sequences

57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
20
Design Enhancements (contd)
Original module had 3 LEDs Power, Running,
and Failed
Replacement keeps Power and Failed, eliminates
Running as there is no equivalent in FPGA space
Adds LEDs for 3 external field contact states
Adds LEDs for 10 demanded relay output state
choice made to show demanded rather that actual
state
Adds 2 numeric LEDs to provide:
Replaces external obsolete external counters
(abandoned in place)
Added display of FPGA failure status code
Added display of selected sequence

57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
21
Design Enhancements (contd)
Enhanced diagnostics for Hatchs specific application:
Can now diagnose limited amount of relay and internal wiring
failures
Did include separate watchdog timer, such that FPGA does
not annunciate its own failure
Diagnostics driven by hazards analysis

57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
22
Process and Implementation
Considerations
All activities were performed under our 10CFR50 Appendix B
compliant Nuclear Quality Assurance Program, including
Programmable Logic life cycle
Modifications were required to fit software life cycle to VHDL
Many design and review topics for software were not
applicable to VHDL (e.g., interrupts, multi-tasking, constraints
of sequential execution, loops, jumps, memory
allocation/deallocation, paging, etc.)
Few new topics were added, including evaluation and checks
of logic signal timing internal to the FPGA
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
23
Process and Implementation
Considerations (contd)
Since no mathematical functions, there are no typed variables;
everything is either a bit or a collection of bits
There are no widespread industry consensus guidelines for
good coding practices, as there are for software
MPR did not apply the equivalent of a static analyzer to the
VHDL
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
24
Process and Implementation
Considerations (contd)
VHDL does allow for sharing signals between modules, with
only explicit definition of the sharing
Program instrumentation (e.g., printf() in C) is simple in
software; more complex in FPGAs and requires interesting
logic to support simple scanning
Can still implement stubs and drivers for VHDL code for
testing stimuli, just like for procedural languages
Software tools exist to simulate the internal logic, including
delay times based on the placed and routed VHDL, which may
be necessary to resolve timing issues
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
25
Process and Implementation
Considerations (contd)
With self-implemented math, precision and accuracy are still
concerns
VHDL code must still initialize memory prior to use
Exception handling still must be designed in to the VHDL, with
inputs and outputs checked for reasonability
Designers still make the same mistakes (e.g., bad
assumptions, missing punctuation, and erroneous but
compliable syntax, incomplete switch statements, etc.)
Generating good, complete unit tests is still as complex
For both software and VHDL, the quality of the product is a
function of the designers experience and capabilities
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
26
QA Plans, Procedures, Processes
MPR has extensive experience with FPGAs in medical devices,
under FDA rules
MPRs Nuclear QA program requires generation of a task-
specific QA plan for safety related projects, explaining how the
project will work under 10CFR50 Appendix B constraints.
With the FDA-compliant processes tailored to 10CFR50
Appendix B vocabulary, work performed in accordance with our
Nuclear QA program
57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
27
Verification and Validation
vs. 100% Test
Intended to use the 100% testable and tested approach
With all diagnostics and multiple state machines considered,
the complexity required to apply all possible input combinations
to all possible states becomes unreasonable
State Machines include: main state machine (8 states),
sequencing step counter (62 states), field contact input debounce
state machines (3 sets of ~22 states), active diagnostics
External and internal inputs include: 47 diagnostic failures, 3 field
contact inputs, and front panel reset switch, or 2
51
combinations)
MPR used a traditional IV&V process, and notes that IV&V
found design errors that testing would not have uncovered


57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
28
Conclusions
Redesigning obsolete digital equipment is possible with no
design documentation, but even a little documentation
simplifies the process
Replacing an analog device is simpler than replacing a digital
device
Replacing software-based devices successfully with FPGA-
based devices requires thought, understanding of the original
equipment, and familiarity with both software and FPGAs
Consider and implement modular reuse of VHDL code
Licensing an FPGA-based replacement is not significantly
different than licensing a software-based device

57th Annual ISA POWID Symposium, 2-4 June 2014, Scottsdale, Arizona
29





Questions?