The URL can be used to link to this page

Home My WebLink About05.a. Receive update on failure analysis of variable frequency drives (VFDs) at Central San Page 1 of 10 Item 5.a. CENTRAL SAN Y-109-ITMEN ME November 3, 2020 TO: ENGINEERING AND OPERATIONS COMMITTEE FROM: NEIL MEYER, PLANT MAINTENANCE DIVISION MANAGER REVIEWED BY: ROGER S. BAILEY, GENERAL MANAGER SUBJECT: RECEIVE UPDATE ON FAILURE ANALYSIS OF VARIABLE FREQUENCY DRIVES (VFDS)AT CENTRAL SAN Staff will provide an overview of the attached presentation pertaining to VF failure analysis and the replacement program. Strategic Plan re-In GOAL FIVE: Infrastructure Reliability Strategy 1—Manage assets optimally to prolong their useful life, Strategy 2—Execute long-term capital renewal and replacement program, Strategy 3—Protect personnel and assets from threats and emergencies ATTACHMENTS: 1. Presentation November 3, 2020 EOPS Regular Committee Meeting Agenda Packet- Page 16 of 41 Page 2 of 10 1� ROOT CAUSE ANALYSIS OF LOWER ORINDA PUMPING STATION (LO PS) VFD FAILURES Engineering and Operations Committee November 3, 2020 Presented by Neil Meyer Plant Maintenance Division Manager Slide 1 OUTLINE • Review LO PS Variable Frequency Drives (VFD) incident. o Background o Sequence of events o Root causes of failure o Summary of corrective and improvement actions • Update on status of other VFD's. o VFD Inventory 6o Spares status and standardization o Current O&M practices oRepair or replacement plans ,6 Slide 2 - November 3, 2020 EOPS Regular Committee Meeting Agenda Packet- Page 17 of 41 Page 3 of 10 BACKGROUND LOWER ORINDA PUMPING STATION (BOX 11 cancare " ' O ■ .,.. r,.., a.rkalryr �on... . Oakland ® rj .4ym N SlideFEW 3 CENTRALSAN LOWER ORINDA PS OPERATION DETAILS ' • Originally built in 1948. it • Major renovation in 2004. o Converted to all electric ➢ Three 4900 GPM(7.05 MGD)250-hp wet y�� . • weather pumps with VFD and bypass Wet weather flow:7.5 MGD a Peak wet weather: 17.64 MGD ➢ Two 1900 GPM (2.74 MGD)75-hp dry weather pumps with VFD • Dry weather flow: 1.3 MGD - • 850-kilowatt emergency diesel generator. • Controls include a relay logic with float to operate pumps across the line automatically bypassing the VFD in case of failure (hard-wired backup pump). Slide 4 November 3, 2020 EOPS Regular Committee Meeting Agenda Packet- Page 18 of 41 Page 4 of 10 SEQUENCE OF EVENTS February 19,2019 • A single WW pump VFD(Pump 105)failed and could not be repaired. Power locked out due to burned parts. • Engineering was in the process of designing replacement VFD under Uniform Public Construction Cost Accounting Act with an electrical consultant. New design resized VFD to 300-hp. January 16,2020 • A second WW pump VFD(Pump 103)failed. • Emergency declared by Board of Directors on January 16,2020. • Expedited purchase of equipment to get project completed. • Central San Electrical Technicians repaired VFD 103,borrowing parts from VFD 105. • Hired drive expert to assess and repair if possible remaining failed VFD. c Aground fault was discovered and repaired. March 5,2020 • Emergency lifted. September 1,2020 • Construction finished on VFD replacement. c Few minor punch list items to be completed November 2020. Slide 5 CURRENT ROBICON VFD ISSUES • Robicon 454 GT VFD's were state-of-the-art when installed in 2005, DP 5448. • Robicon agreed to sell its assets to Siemens Energy & Automation Inc. in June 2005. • Robicon VFD's became obsolete in 2010, making new replacement parts unavailable. • Few new or used parts are currently available on eBay and from used parts resellers. Difficult to find service providers. L Slide 6 I' November 3, 2020 EOPS Regular Committee Meeting Agenda Packet- Page 19 of 41 Page 5 of 10 ROOT CAUSE OF FAILURE 1st VFD Failure - Insulated Gate Bi-polar Transistor(IGBT) Failure • Did not find conclusive evidence on why the IGBT failed. • Possible reasons for damage from overheating include: o A poor connection to a heat sink o Overloading the VFD o Misfiring of a gate driver board o Dust buildup o A poor wiring connection 2nd VFD Failure -Gate Driver Board Failure • Common feature caused by a bad electrolytic capacitor on the printed circuit board. • Proactively reconditioned to extend lifespan, but with limited supply, not all have been reconditioned. Slide 7 AW10 i= ACTION ITEMS AND CURRENT STATUS Corrective Actions • Reviewed the incident with all PS staff to increase awareness of recent VFD failure and continue performing daily checks of PS equipment and systems (including drive equipment and systems)—Complete • Reviewed the incident with all E&I staff to increase awareness of VFD failures and heightened alert when the Electrical and Instrument Shops assigned reactive and PM's for VFD's and MCC's—Complete • Obtained refurbished spares from Jeff Fiore —Complete • VFD replacement project—Substantially complete (minor punch list items remain) • Salvage spares from the replacement equipment at _ LO PS—Complete Slide 8 ` 11540THITNIII11W.M6 November 3, 2020 EOPS Regular Committee Meeting Agenda Packet- Page 20 of 41 Page 6 of 10 ACTION ITEMS AND CURRENT STATUS Improvement Actions • Establish a contract with Fiori Industrial Services for future service troubleshooting and repairs as needed—Complete (contract in place) • Engineering and Maintenance to review replacement plan for VFD's. Ongoing replacement plan in place. Part of a continuous process to review and add equipment on replacement schedule. • Obtained spare drive as part of LO VFD project. • Tighter specifications on VFD requirements has resulted in Central San moving to standardization of VFD's, ensuring a higher standard of equipment to increase equipment reliability, performance, and serviceability. Slide 9 CURRENT VFD INVENTORY • Between PS and the Treatment Plant(TP),there are 125 VFD's in the Asset Register with 16 different manufacturers ranging in size from a fraction of 1-hp to 1,000-hp. This requires a larger variety of spare parts and technical knowledge of brands. o Manufacturers: ➢ ABB- 13, Allan Bradley-9, Dynamic-4, Eaton-27, Emerson - 12, Robicon- 19, Siemens-2, Westinghouse- 11, and others. • Life Expectancy of a typical VFD is 10-15 years. J,o 53%at Central San are estimated to be past their average service life. Slide 10 11THITNI1111W.M6 November 3, 2020 EOPS Regular Committee Meeting Agenda Packet- Page 21 of 41 Page 7 of 10 CURRENT VFD INVENTORY • Two of the most failed drives are currently being phased out. o Robicon 454GT Gate Driver Boards issues requires constant repairs. o Westinghouse A4077-4PF-10269 Keypads failures-no longer serviceable. • Certain VFD's are equipped with bypass in case of a VFD failure. o The pump is limited to pumping at full capacity when in bypass mode making them difficult to control. o Ideally, bypass mode is used for flows higher than the pump's capacity, allowing for a second VFD-driven pump to control flow. Slide 11 SPARES • Spare parts for VFD's include relays,transformers, control boards,fuses... over 45 items for some manufactures(i.e., Robicon,Emerson). o Built spare database to search by TP or PS or all. ➢ Lists spare parts and spare assets. o Some manufactures'(such as Robicon)parts are obsolete,so they are sent out to be rebuilt,and salvage parts are used. o Eight boards salvaged from recent decommissioned Robicon PS VFD's. ➢ Purchased two additional refurbished boards from Jeff Fiore. o Newer VFD's have fewer replaceable parts and are more modular,simplifying replacement and inventory requirements. • Spare VFD's are considered for certain instances depending on type,size,and availability. o Critical one-offs and VFD's with long lead times are kept in stock. ➢ Examples:LO PS wet weather VFD's and Cogen NO,pump VFD. Slide 12 November 3, 2020 EOPS Regular Committee Meeting Agenda Packet- Page 22 of 41 Page 8 of 10 CURRENT O&M PROGRAM Pumping Station Operators • Perform daily checks of PS equipment and systems. • Prepare for WW with additional functional checks on equipment. • Routinely perform numerous PM's on equipment. Electrical and Instrument Shops • Perform routine maintenance PM's including VFD and MCC PM's. o PM's include a visual inspection; IR scan;checking and tightening connections;cleaning,and/or replacing contacts in motor starters; performing a megger test of connected motors;and cleaning of buckets,drives,and other devices in MCC lineup. Controls and Mechanical Shops • Provide support, perform corrective actions based on routine inspections,and upgrade and repair needs. Slide 13 OPERATIONS AND ENGINEERING COORDINATION Planning • Continuously review assets for replacement and schedule. • Maintenance shops attend Engineering Planning, Design (30/60/90) Review Meetings. • Bi-annual Coordination Meetings between Operations Department and Engineering and Technical Services Department ensure equipment replacement is captured on projects. Slide 14 November 3, 2020 EOPS Regular Committee Meeting Agenda Packet- Page 23 of 41 Page 9 of 10 OPERATIONS AND ENGINEERING COORDINATION Redesign efforts • New VFD's are more compact, making retrofits possible utilizing the existing cabinet, power, and control wiring. • Central San staff has completed several retrofits, examples include: o Concord Industrial PS sewage pumps 2014 o Maltby and Fairview PS sewage pumps 2015-2016 0 3whp VFD in 2018 • Influent Pump VFD#1 was similarly overhauled with a retrofit utilizing existing cabinets and wiring by a contractor in 2018. Slide 15 VFD REPLACEMENT PROGRAM • Out of the 125 VFD's in the Asset Register: 0 25% have been replaced with new ones in the last 5-6 years. ➢ Combination of in-house and outside contractors. 0 65%are scheduled to be replaced on future construction projects in the next 5 years. 0 10%are under evaluation including: ➢ 2 LO PS Dry Weather Robicon 454GT's. ➢ 7 Critical Equipment VFD's in the TP. ➢ These drives are evaluated for projects or being looked at for retrofit in-house. Slide 16 November 3, 2020 EOPS Regular Committee Meeting Agenda Packet- Page 24 of 41 Page 10 of 10 VFD REPLACEMENT PROGRAM • VFD's being replaced have tighter specifications to ensure Central San receives a trusted product that can be supported for years to come,that is familiar to Operations and Maintenance, with ease of maintenance and repairs. • Equipment replacement timing is a continuous process that considers age, condition,work order history, serviceability, obsolescence, redundancy, and reliability—including availability and stocking of spares. Slide 17 Questions? Slide 13 November 3, 2020 EOPS Regular Committee Meeting Agenda Packet- Page 25 of 41