Flagami Outage
Home Up

 

The Cause:

On Tuesday, Feb. 26, 2008, much of Florida suffered a power outage.  I'm calling it the Flagami outage since that's the name of the FPL substation where the outage started.   Ironically, this substation is right next door to the FPL Headquarters at 92nd Ave and Flagler Street in Miami.  At 13:08 there was a small fire on insulator at this substation.  At 13:09 the power went off for 3 million customers in the State of Florida.  While the fire was the trigger for the outage, it's still not clear why this caused so many generators to go off line.  Also important to note, not all of Florida or even all of South Florida was impacted.  In many areas, there was either no outage or only a small glitch. 

Coverage of the outage:   WPLG  Washington Post  FPL's Conference Call  Miami Herald

Observations:

I was driving on SR836 when I heard WQAM go off the air for a few seconds.   At about the same time I got multiple emails on my Blackberry from UPSs around Miami Dade County.  I knew there was a major power hit.   Since the UPSs are backed up by generator, I got the power restored messages a few seconds later.  After a call from another telecom professional, I found out that the power was out for many areas.  Since I was on a highway during the day, there was no way to see that the power was out.  I checked our local EAS (Emergency Alert System) Primary radio news station, WIOD, for local news updates and instead found Rush Limbaugh.  

As a part of my job, I proceeded to the Miami Dade 911 Center.  For the next few hours I tracked the progress of the outage, the recovery and the impact on our telecom systems.   As FPL generators came back on-line, power was restored in different areas.  By about 17:30, almost all areas were back up.   The Miami Dade Emergency Operations Center was activated to a Level 2 (out of 3, 1 being normal operations).   There were hundreds of additional calls for Fire Rescue service in Miami Dade due to mostly people stuck in elevators and false alarms from Fire Alarms that had lost power.

Telecom Impact:

Lucky for us, the length of the outage was short enough to avoid failures of Remote Terminals.   We only had one location fail due to the power outage.  This email documents the time and duration of the T1 outage:

A MultiPing alert has fired for the following period(s):
2/26/2008 4:14:11 PM to 2/26/2008 4:46:12 PM

Monitoring: Sta 25 70DHZX521835 10.18.197.254

From a few previous outages, I knew that this Remote Terminal was located in the closet of a private aviation company.   It had been out for many days after Hurricane Wilma.  This same RT serves the FAA Tower at Opa Locka airport and the USCG base.

Since we know the power went off at 13:09, the batteries at this RT lasted 3 hours and 5 minutes.   The outage lasted 32 minutes.   I'm assuming the outage ended when FPL restored power to the Airport.  Out of the 80+ locations I'm responsible for, this is the only one that had a telecom outage.

The Potential:

It is my opinion that we would have had many more such telecom outages if the power outage had lasted more that 4 hours.  Four hours is the design goal, and now FCC requirement, for Remote Terminal run time during a power outage.  This power outage could be a good drill to see the best possible scenario for RT Generator deployment.   It was the middle of a typical business day so I expect ATT was fully staffed and few techs had to be called in to work.   The weather was clear except for one small part of Broward County.   Since this outage was not storm related, there were no unsafe areas or areas that had to be cleared by the power company for access.   No curfews were invoked.  Clearly, there was no better time for an outage.  Other power outages would have been far more difficult to respond to.   While many parts of ATT's service area were impacted by this power outage, not all of it was.   If the cascading effect had continued to more generators, the outage could have been 100% in South Florida or even all of Florida.

Information to evaluate response:

Questions I hope someone will ask ATT (former BellSouth):

bulletWhat was the time line for their detection of the problem.
bulletHow many RTs lost utility power?
bulletHow many RTs got power back within seconds from co-located backup generators?
bulletDid any RTs fail instantly from defective batteries?
bulletHow many portable generators left equipment yards to go to critical RT locations?  
bulletHow long did it take to get these generators to the RT sites?  
bulletHow did the lack of traffic signals and the resulting traffic jams effect the generator transport time?
bulletWhat percentage of the effected RTs got a generator hooked up before the power came back on? 
bulletHow many RTs shut down due to battery exhaustion?   What was the run time before the exhaustion?
bulletHow many telephone lines and circuits were out of service due to the battery failures?  Note that I'm not saying the number of customers or the number of lines reported.  I'm looking for the actual number of lines taken out of service during the outage.
bulletHow many effected RTs would have had running generators connected before the 4 hour battery life specification? 
bulletGiven the number of RTs set up and running with portable generators during this outage, project how long it have taken to get all effected RTs on portable generators.   Include the travel time for each generator that would have been needed. 
bulletAs RTs failed, how was the area effected relayed to 911 centers?  Without telephone service, these areas had no way to call for help.

Conclusion: 

In my opinion, we got lucky.   If this outage had been more like the FPL outages of the 1970s and 80s, or Hurricane Wilma, or the recent outages in the Northeast, we would have had significant telecom outages.   Even another hour of power outage could have resulted in tens of thousands of people being without any telephone or datacom service, including cellular service or even calling 911 from a landline.  Hopefully ATT and their regulators (FCC and the Florida Public Service Commission) will use this event as a learning experience and come up with some realistic projections based on the data gathered.    Clearly we have been reminded that it is very easy to have a huge part of South Florida without power, even during a perfect day.   Only because a limited number of generators were effected was FPL able to get the area back on in less than the magic 4 hour period.   Any longer and this would have been far more devastating to telecommunications.  I feel that after seeing the projections regulators will seriously reconsider the 4 hour backup requirement and increase it to a realistic time period to get all RTs in a large geographic area on generator power before the batteries are exhausted. 

 

02/27/2008

Except for portions owned by others, Copyright: Ray Vaughan, 2008

Comments/Questions