Risk Assessments for Critical Operations Power Systems

Tag : limiting switch
New NEC Article 708 Critical Operations Power Systems leaves muchto the judgment of the engineer designing the COPS. By Michael A. Anthony, PE, Robert Arno, Robert Schuerger, PE, andEvangelos Stoyas, PE -- Consulting-Specifying Engineer, 6/1/2008
Engineers are asking many questions about the new Article 708,Critical Operations Power Systems (COPS), in the 2008 NationalElectrical Code (NEC). For example: How do we comply with thedocumented risk assessment required in Section 708.4? Unlike theother special systems of Chapter 7, there is subtext to the newarticle that may depend upon mandates from state or federalauthorities.
Fine Print Note No. 1 reads: “Critical operations powersystems are generally installed in vital infrastructure facilitiesthat, if destroyed or incapacitated, would disrupt nationalsecurity, the economy, public health, or safety, and where enhancedelectrical infrastructure for continuity of operation has beendeemed necessary by governmental authority.”
To NEC traditionalists who believe the NEC should remain aprescriptive installation code for electrical safety in buildingpremises wiring only, this may be unsettling. They want clear,bright-line code that makes their point at construction board ofappeals meetings. Even though there are a few sections of the codethat address design, along comes Article 708, placed in the specialsystem chapter, with all of these subtleties not seen in the otherChapter 7 articles that govern backup power systems commissioning,maintenance, and testing. It has annexes with new words that seemto make paperwork as important as wiring (see COPS terms of art, atright).
It also raises issues of how these COPS requirements will befinanced. You can build the cost of a generator into the first costof a building on the basis of NFPA 101. However, is ensuring 72hours of fuel for it (as required in 708.22-C)—because theCOPS is an integral part of the building—considered anoperating expense? Conversely, can a homeland security grant helppay for an infrastructure upgrade where the designated criticaloperations area (DCOA) is an embedded system in a multi-functionbuilding? Does testing to compliance with 708.6 mean thatthird-party agencies need to be paid from a county homelandsecurity grant or from the same municipal budget that coversinspections for general commercial occupancies?
These are questions that will be answered as 708 is integrated intoother standards and codes. Article 708 cross-references six otherNFPA standards within itself. Progress in the integration processwill be measured by how fast we see reciprocal referencing of 708in other NFPA, International Code Council, Federal EmergencyManagement Agency (FEMA), and Dept. of Homeland Security documents;and how fast precedents track in white papers and advisories ofpublic utility commissions and county emergency managementagencies.
Here, we describe the characteristics of an NEC-compliant riskassessment for a single designated critical operations area. Weassume it does not trigger a multi-trade infrastructure upgrade ofthe larger facility within which the COPS is embedded. We borrowheavily from the technical foundation provided by the U.S. ArmyCorps of Engineers (USACE) Power Security Enhancement Program whosetechnical manuals, used in the design and operation of Command,Control, Communications, Computer, Intelligence, Surveillance, andReconnaissance facilities have been released for civiliandistribution.¹
Because the geometry of regional critical infrastructure typicallyis widely scattered, keep in mind that the risk assessment methodsdescribed here can be scaled outward to encompass an entire city,county, or state. The emergency operations plan required in 708.64should reflect a comprehensive understanding of critical operationsas a system-wide entity that may encompass several locationsnetworked together as a single operation. STEP 1: DETERMINE LOCATION(S)
A template for a regional risk assessment was introduced before the2008 NEC went to print, before it had been established that thetechnical panel assigned to the task was dealing with a Chapter 7special system rather than a Chapter 5 special occupancy. Thearticle included a qualitative discussion of how jurisdictionsmight rank regional risks. Simply put, DCOAs had to be scaledaccording to the relative likelihood of natural or human-madedisasters.²
In a follow-up article, a nominal prioritization procedure thatresembled a mode criticality “multi-voting” techniquepromoted by the American Society for Quality was used to produce adisciplined regional risk assessment. An example ranking thelikelihood of earthquakes, convective weather, and pipelineaccidents was tabulated for a southeastern Michigan county.³
In identifying candidate locations that can meet 708.5 requirementsfor physical security, it is likely that adopting jurisdictionswill have a choice of existing emergency response/disastermanagement centers. After all, there has been a great build-out ofemergency management facilities—even before Article 708 camealong—and we have to assume that they are able to communicatewith one another. It's important to keep communication channelsopen, so the writers of the 2008 version of Article 708 appendedreferences to the signaling between these agencies in Annex G:Supervisory Control and Data Acquisition. The same topic is coveredin substantial detail elsewhere. 4
Ideally, COPS should not contain a single point of failure thatwould allow both the normal electrical service and the emergencybackup power to be affected by a single incident. This is moredifficult to achieve than it first appears with only one transferswitch.
Figure 1 shows a very common one-line diagram for a small facilitysupplying telecommunication, security, and computer equipment inwhich we have assumed that all the architectural and site securityrisks have been mitigated.
The normal design for uninterruptibe power supply (UPS) batteriesis 10 to 15 min, providing power continuously to the UPS outputwhile the generator starts and comes on line. In Figure 1, afailure of the automatic transfer switch (ATS) puts the UPS moduleon batteries—without any input power. For a facility that haspersonnel on-site 24/7, providing a means to either manuallytransfer the switch under no load (which most ATS have built-in) orbypassing the switch would solve this single point of failure. Foran unmanned site, redundant components would be required. Thesekinds of failures can be identified by a fault tree analysis or afailure modes and effects analysis.
Emergency and normal electrical equipment should be installedseparately, at different locations, and as far apart as feasible.Beware of co-locating essential or redundant feeders with othermajor utilities. Some types of facilities—where tornadoes orblasts rank high on the regional risk assessment—needredundant electrical distribution to critical areas. Centralutility shafts may be subject to damage.
One qualitative method of evaluating the architectural, utility,and site layout aspects of this part of the risk assessment isfound in the “Building Vulnerability AssessmentChecklist” developed by the Department of Veterans Affairsand described in detail in a FEMA manual. 5 The reference takes the user through a consistent securityevaluation of designs at various levels.
There are other methods, such as those used by USACE in theevaluation of the utility systems for prospective C4ISR facilities(see Risk assessment: determining location, at right). 6
Keep in mind that if a DCOA remains embedded within an existinggeneral occupancy building, the COPS for the DCOA will have to beprotected from changes to the mechanical and electricalinfrastructure associated with the general occupancy parts of thebuilding. STEP 2: DETERMINE REQUIRED AVAILABILITY
Availability is the percentage of time the system is in operationor is available for operation. Suppose the state emergencymanagement agency issues a directive that all city fire departmentsmust have backup power available to at least “threenines” (99.9%), less than the 72 hours required in 708.22-C.Three nines would not be an unfamiliar metric: the power providedby most U.S. utilities is available out to three and four nines.The agency is asking for utility-like availability at the DCOAusing the power sources listed in Part III of 708.
Identifying an acceptable risk is one of the most difficultdecisions for higher governmental authorities to make. While thecost of additional nines is a well-known parameter in the businesscontinuity industry guided by NFPA 1600, “Standard onDisaster/Emergency Management and Business ContinuityPrograms,” these agencies would have to look at the economicsof additional nines to public service organizations with differentmissions.
Getting availability out to three nines could be costly if fuelstorage or emissions restrictions apply to the site. As theavailability table in Annex F shows, three nines of availabilitytranslates into a potential 8.76 hours of downtime in a year. Ifgetting to three nines means periodic testing that triggers a localcap on emissions, designers may have to investigate other feasiblesites. They could develop vehicle-mounted generators covered in708.20(F)(6) as an option, along with fuel-handling logistics.
Administrative options should not be overlooked in the hunt forsavings. Limiting the use of parts of the overall facility providesa workaround, sometimes as simple as scaling the size of the DCOAto the number of people or the level of training of the people whowill use it.
To achieve 708 objectives with a single engine-generator setsuggests a considered design in which administrative procedures andpower chain hardware is balanced carefully. One characteristic ofthe reliability engineer's art is how administrative options aretranslated into the system model. Translation of system needs orrequirements to reliability numeric is critical in establishing afacility meeting the 708 requirements. Caution should be exercisedhere not to fall into a “cookie cutter” approach. STEP 3: BASE CASE: ESTABLISH AVAILABILITY POTENTIAL
Several different tools exist to model the availability potentialof the COPS design. In general, reliability engineers use modelingtools with the capability of statistical simulation to incorporatevariations in component capacities associated with consumables suchas water and fuel. Failure rate and repair time data play animportant role in the analysis of a system to determine whether itmeets requirements.
Figure 2 is a tabulation of COPS components along the power chainof our simple study system. Each component in the tabulation has anavailability derived from an extensive data collection effort andis expressed (usually as a decimal) in terms of potentialavailability per year.
IEEE reliability experts began collecting data on failure rate andrepair more than 20 years ago, published in the Gold Book 7 . This effort ran parallel to an international trend in totalquality management (TQM) that employed statistical andprobabilistic methods to improve component and system quality. Areference to TQM also shows up in Annex F. Realizing the importanceof gathering failure rate and repair cost data to its own missionfor civilian infrastructure security, the USACE funded the mostcomprehensive reliability data collection effort to date. Thefailure data used in doing any DCOA/COPS study should be includedin the appendix of any risk assessment.
Unlike the cable reactance data in the wiring tables of Chapter 9of the NEC—some of which is the better part of 100 years oldnow—power chain component reliability data are far moredynamic, reflecting improvements to “commodity”components of a COPS such as engines, generators, transferswitches, and UPS systems.
None is a commodity in a strict economic sense. They arecomponents, lumped as parameters within a system, and manufacturersare always improving them. Because they tend to befactory-assembled and can be shipped to the system site, they canbe treated as interchangeable commodities in a COPS. The core valueof a COPS lies in the proper application of those commoditycomponents to produce an initial and continually verifiablenameplate availability.
Skilled reliability engineers will apply their judgment inreconciling the competing requirements for redundancy andsimplicity. They will hit the availability target with the leastnumber of components, concentrating dollars on the components thatwill yield the largest value. Construction and maintenanceactivities are reciprocal partners in that effort, so thereliability engineer will balance attention to both the specificsand the interconnectedness of the first-cost/long-term O&M costconundrum.
With commercial software, one can enter the failure and repair datafrom user-defined libraries into branch and node input data screenssimilar to the manner in which short circuit or load flow studiestake in system data. The output is a reliability block diagram asshown in Figure 3. STEP 4: HITTING THE TARGET
To focus on the areas needing the most attention in the systemimprovement process another set of useful tools are Failure Modesand Effects Critically Analysis (FMECA) and Fault Tree Analysis(FTA).
Each method is described in Annex F, Part I, and each employs adifferent probability distribution equation, but both accomplishthe identification of weak members of in a system that contributeto prospective unavailability. These procedures focus efforts tocritical branches and nodes of the system needing the mostimprovement.
These quantitative methods are new to the NEC, but are not new toother NFPA documents. For example, NFPA 1600 mentions FTA in itsAnnex A. Annex A of the 2006 “Vehicular Fuel Systems Code– NFPA 52” also mentions FMECA as a hazard analysismethod.
FMECA and FTA methods are standard operating procedure in manyindustries that continually seek to improve designs for productsand processes. They are required to comply with safety and qualityrequirements, such as ISO 9001, QS 9000, ISO/TS 16949. Substantialhow-to information specific to mission critical power systemsappears in another technical manual. 8
Figure 4 is a summary tabulation of the input and output from acommercial reliability program. The mean time before failure (MTBF)and mean time to repair (MTTR) are calculated from all of theupstream components of the sample system. One can verify theavailability by using the equation:
Availability = MTBF/(MTBF+MTTR) = 115,900.10/(115900.10+1.55) =0.999987
Note that four-nines availability exceeded the initial target ofthree nines. If four nines is too expensive, the reliabilityengineer might try to “value engineer” the COPS withless expensive components at initial construction to be compensatedwith a more robust testing plan in the long run.
Design engineers also can run cases to demonstrate to the inspectorhow COPS availability erodes over time unless maintenance tasks areperformed, or how reliability might grow as commodity componentsburn-in. Other cases could be run to show how too-frequent use,coupled with over-testing, would trigger a longer repair time in ahigh-speed generator due to manufacturer recommended engineoverhaul. STEP 5: PERFORMANCE TESTs
The development and implementation of functional performance tests(FPT) described in Part II of 708 applies an operationalperspective to system design. Functional tests are developed duringdesign and performed after construction to demonstrate that theCOPS system will function according to the desired nameplateavailability. Since FPTs are based on the actual installation, ifthe system changes, the tests may need to be modified.
Although baseline test results are required in 708.8, the FPTdescribed in Annex F is optional material, “included forinformational purposes only.” The FPT would be not justcommissioning of components or subsystems, but a full system-wideperformance test exercising as many functions as possible. Itshould be a multilevel simulated failure exercise to ensure systemreadiness. Ensuring its nameplate availability throughout thelifecycle of the COPS can be accomplished by applying NFPA 70B,Recommended Practice for Maintenance of Electrical Equipment.
Business continuity companies are seeking business models whereinthe commodity portion of sophisticated systems (UPS, on-sitegenerators, transfer switches) can be released turnkey to asupplier who can build to availability specification and manage thesupply chain to keep it there. Emerging specifications forconnectability and maintainability allow these systems to integratefaster with broader business continuity networks. Installing thebest COPS will require the establishment of new partnerships andthe development of new supply chains. The more economically we canbuild these systems, the more systems we can build, which is betteroverall. THE BOTTOM LINE
The subject of electric power security is a minefield ofsensitivities about boundaries and budgets, risk, and civilreadiness. Interdependent systems that support electricity supplyare not perfect and institutional mechanisms to supportreliability, security, and survivability need to strengthen at thebuilding premises level. NFPA understood the need and found themeans to convey the best practices of the business continuityindustry into public sector emergency preparedness.
Article 708 looks a lot like performance-baseddesign—something the building safety community still tends toput at a distance. The science involved in developing a COPS is atleast as sophisticated as the multi-disciplinary science advancedby the Society of Fire Protection Engineers and described inChapters 5 of the Life Safety Code (NFPA 101), the Uniform FireCode (NFPA 1), or the Uniform Building Code (NFPA 5000).
Despite NFPA's extensive coverage of performance-based practice,most jurisdictions still regard performance-based designs as theexception rather than the rule. There are at least three reasonsfor this, each loosely related:
• The difficulty in verifying the claims of substantial equivalencyamong complex systems . It is easy to compare two nameplates, but hard to compare twobinders full of facility engineering documentation—even ifyou could find it, and even if it were up-to-date. The prospect ofsplit functionality of an engine-generator system for an emergency,legally required, and COPS, exacerbates the problem.
• An aversion to anything that cannot be counted . Prescriptive methods such as “one smoke detector every 30ft. down an egress corridor” or “two sources are alwaysbetter than one,” while sometimes wasteful, can be verifiedby the naked eye. The price we pay to for visibility andstandardization is that we overbuild.
• The insurance company wants to see a prescriptive solution. Enough said.
We have to be careful about what could be perceived as regulatoryexcess. We do not want to under-do COPS, but we cannot overdo themeither. Jurisdictions always have the option of delaying adoptionor even ignoring any NEC requirement. Experienced electricalprofessionals know that the cheapest time to build backup powersystems is the day before the next major regional contingency.
The prospect of stepped-up regulation frequently stimulatesinnovation to avoid it, creates opportunities for unproventechnologies, or creates improvements in commodity components.Unless the jurisdiction can afford the multiple-use of generatingsources contemplated in 708.22(B), we know that backup systems arenot perceived to have value until they are needed.
Nevertheless, a vast industry process is just booting up.Innovation in fast-turn design and constructiontechniques—including partnerships that span the power supplychain—have emerged. Other ways of steering capital to COPS,though not necessarily cheaper, may lie in integrating them intoregional distributed power regimes. While these are typically moreexpensive than centralized systems, in a distributed power regime,COPS would become more common so we could reach the distributedgeneration “tipping point” envisioned by alternativeenergy futurists. The concept is already tracking at the FederalEnergy Regulatory Commission.
For the moment, lack of electricity should not be among theproblems of first responders and disaster recovery teams. Insteadof trying to manage a crisis within a crisis, Article 708establishes the framework for managing a plan within a crisis.
Anthony is senior electrical engineer at the University ofMichigan, Ann Arbor. Arno is director of the C4ISR group at EYPMission Critical Facilities. Schuerger is director of riskassessment at EYP Mission Critical Facilities. Stoyas is aconsulting engineer who was a member of Code Panel 20 thatdeveloped Article 708 and a member and former chair of thetechnical panel that writes NFPA 70B.