HomeMy WebLinkAbout08.a.1) Update on water damage to the District’s Information Technology (IT) Data Center.Agenda Item 8.a (1)
IT Data Center
Water Damage Update
March 5, 2015
Presenter
David Heath, Director of Administration
..,c = CD Amrow
Protecting Public Health and the Environment
Timeline
• Late Friday evening February 20, water
leaked into the data room from the kitchen
above damaging a rack of data server,
controller and storage devices.
• Saturday was a scheduled system outage
for disaster recovery testing as new
equipment had arrived.
• Water damage was discovered late
Saturday afternoon while staff was
performing the testing.
2/27/2015
2/27/2015
2/27/2015
Floor and wall damage
— includes damage
from previous
overflows. Unable to
discern what was
latest incident.
D
7lei{
t
2/27/2015
5
2/27/2015
Backup /Production Servers
Water Damage Plan
Nutmix Production "Pool"
MTZ -20 Tb
CSO — 20Tb
�a
aw
Quantum Tape Drive
Backup Device Nimble Backup /Production Server
DistGroup ( "N" Drive) -production: - GIS
DistAV - Multiple VM Servers
HOB3 - Backup: - Exchange(Mail)
DistGroup ( "N" Drive)
DistAV
-Hob3 to
Dell MD Unit Production "Pact'
"3200"
-Tier 1"- Exchange
- D1stAV
"1200"- Tier 2 — DistGroup
( "N" Drive)
HOB3
"1200" -Tier 3 — DistAV
Dell Equal Log. dk.. Server
Differentials: Damaged Rack
- DistAV Equipment
- DistGroup ( "N" Drive)
Plan "A!'- Bring Dell M D "1200" Hard Drive Un Its on -line to make DistGroup, DistAV, and HOB -3 availab le to end -users
- Utilize Dell MD "3200" Hard Drive Unit and move Exchange (Mail) to Nlm ble Backup /Production Server (a more stable environment)
• Can't start process to try and bring up the "1200'x' until S:OOPM 2126/15 because it would impact the Exchange (Mail) Server
- "1200's may not come back on -line
- Fastest process
Plan "B "- Utilize Dell Equal Logic Backup Server to restore DistGroup and DistAV Differentials Nutan& Production Pool
Plan "C" - Utilize Quantum tape drive backup information to restore DistGroup, DistAV, and HOBS to N imble Backup /Production Server
Conclusions
• The current data center location has
experienced multiple instances of water
infiltration overtime.
• Engineering has designed an interim plan
to shield IT Equipment using metal
collectors and gutters.
• Moisture sensors are on order to be
installed to provide early detection.
• Relocating the data center is currently
under evaluation by the Engineering Dept.