They say a chain is only as strong as its weakest link, a lesson paging carriers learned in a painful way last Tuesday evening when an estimated 90 percent of all paging subscribers in the country found themselves without service due to a faulty satellite.
While much paging service was restored before the end of the week, the paging industry could continue to see repercussions from the incident.
Just after 6 p.m. EST, the Galaxy 4 satellite-a $250 million, five-year- old communications satellite owned and operated by Greenwich, Conn.-based by PanAmSat Corp.-spun out of control. PanAmSat, which is 81-percent owned by the Hughes Communications Co. subsidiary of General Motors Corp., declared the satellite unrecoverable last Wednesday.
Every paging carrier in the nation relying on the satellite for their primary direct satellite broadcast system-virtually all of them-lost service, affecting about 40 million customers. Companies like Paging Network Inc., PageMart Wireless Inc., AirTouch Paging and Cellular Inc. and TSR Wireless Inc. lost service to all of their customers. Thirty-seven percent of MobileMedia Corp.’s 3.3 million subscribers and half of Metrocall Inc.’s more than 4 million customers experienced the same service outage.
Carriers worked 24 hours a day to realign their transmitters to a backup satellite, beginning with the major markets and continuing to smaller markets through the end of the week. Some customers regained service within hours, while others waited days. Certain rural markets were not expected to receive service until this week.
Carriers helped each other out in unprecedented ways on Wednesday, with technicians from one carrier volunteering to reconfigure the receivers on another carrier’s transmitter. They all knew that what affected one business affected the industry.
“A day like [Wednesday] is not the day you want to try to assert some kind of competitive advantage,” said a spokesman from one carrier.
Faced with a service outage unprecedented in the history of paging, the industry tried to put a positive spin on it.
“It was an unfortunate situation, but Tuesday’s failure of the Galaxy 4 satellite was a wakeup call to the American people,” said Jay Kitchen, president of the Personal Communications Industry Association. “It illustrated the profound role paging, and in fact, all of wireless, plays in our everyday lives.”
Spin doctoring aside, the situation was more likely a wakeup call to the paging industry, highlighting the serious lack of redundancy in direct broadcast satellite networks. That 90 percent of the country’s paging subscribers could lose service for at least six hours and as long as three days because of a breakdown in just one link that most networks share is a major concern.
When a page is sent, it travels along the wireline phone network to the paging terminal. The terminal takes the page and sends it to a satellite uplink station, which sends the page to a satellite. The satellite redirects the page back to a satellite receiver on a transmission site, which then broadcasts the page.
The root of the problem was the critical role this one satellite played in the nation’s advanced paging networks-the top 10 carriers in the country relied on the same satellite for their direct broadcast satellite systems and had no immediate backup system in place in the event that satellite went down.
“Why does 80 percent of the (U.S.) paging industry depend on one satellite?” asked Jeanine M. Oburchay, associate director and paging analyst for Bear, Stearns & Co. Inc., New York. “Paging didn’t rely on satellites when it was terrestrial-based and on a lower part of the spectrum … Then it moved into the 900 (MHz) range and needed a satellite for backhaul … PageNet was the first to get up there and choose a satellite, and everyone wanted to be PageNet, so they followed.”
The only hope for immediate recovery was the G4 regaining navigational control and beginning transmissions again. Once it became clear that wouldn’t happen, PanAmSat migrated its paging carrier customers to its Galaxy 3 and SBS6 satellites, while carriers switched to their own backup satellites.
“It’s one of the risks you take when you use satellites as your key feed,” said Steve Virostek, paging analyst at the Strategis Group. “They’re great for coverage and the cost is right, but they’re 22,000 miles in the sky. You can’t just send a technician up there when something goes wrong.”
Although there were other satellites made available to handle the paging traffic, much more needed to be done to get the nation’s networks back on line. Directing satellite uplinks to the new satellite was a relatively simple procedure, but much more difficult and time-consuming was reconfiguring each satellite dish on each transmitter to the new satellite on a city-by-city basis. An army of technicians had to reconfigure each of these transmitters by hand. PageNet has 9,500 transmitters that needed reconfiguration, Metrocall 5,000, MobileMedia 2,500, PageMart 2,000 and AirTouch 1,500.
Despite the time and manpower it took to implement this contingency plan, paging carriers and PCIA said they felt the salvage effort was a success.
“The industry is to be commended for its response to restore paging and messaging services to its customers,” Kitchen said. “Carriers have come together, sharing resources, including technicians, to quickly bring their systems back online.”
While the affected carriers patted themselves on the back, the fact that thousands of customers lost service for days begs the question of whether carriers could have been more prepared. Many industry insiders weren’t impressed. A backup plan that takes more than 48 hours to complete didn’t sound like much of a plan to most because the damage was already done.
“You got to wonder if it was planned or did we get lucky?” asked Bob Egan, research director at the Gartner Group. “Given the importance of paging-being that if you’re carrying a pager, there’s a sense of urgency-if you go 12 hours without service … it doesn’t look good.”
SkyTel unscathed
A true backup plan would have meant less service interruption, like the plan that SkyTel Corp. used.
“(SkyTel), who we all laughed at back in 1995 because they wanted (satellite) redundancy for their premium service, was back up in 10 minutes,” Oburchay said. “You may see other companies building in contingency plans now, (although) there has been only one satellite failure in 35 years.”
According to Marc Kuykendall, SkyTel’s director of corporate communications, messages on SkyTel’s one-way network are first sent to the G4, but if the satellite doesn’t respond after 72 seconds, messages go to the SpaceNet 4 satellite. SkyTel’s transmission towers are configured to receive signals from either satellite without the need for reconfiguration.
“When everybody was scrambling around (Wednesday), our one-way system was operating fine,” he said.
“I think SkyTel was the only one that had a backup plan,” Egan said.
While SkyTel’s two-way network did not contain that same level of redundancy, the advantage of a two-way network is that it can handle such a failure in better stride. Since it didn’t have a backup satellite, the company instead sent messages out over its return path, which is entirely landline. Although this slowed message delivery and reduced coverage in many areas, service continued for the most part.
“It’s not the way you always want to do things,” Kuykendall said, “but it can get you through a rough day or two.”
Quite unexpected
Few ever expected the satellite link to fail, given satellite communications’ otherwise exemplary record, explaining why other carriers didn’t have a backup satellite system like SkyTel’s or implement a network-wide, terrestrial-based system to increase network redundancy.
“This type of disruption in satellite service is extremely rare,” said Federal Communications Commission Chairman Bill Kennard in
a written statement. “I understand that, over the last 10 years, the average satellite failure rate, in many cases involving just partial failure, has been approximately 1 percent.”
In light of the disaster, Kuykendall said SkyTel feels the cost of its redundancy system was money well spent.
“I think it’s part of your commitment to be a premium service provider,” he said. “Part of that is making sure your links are protected. If you’re a carrier, what else do you have to (offer) than your network?”
Has the industry learned from this disaster?
“Sure we’ll learn from it,” Kuykendall said, “it would be foolish not to. I think it’s important we learn some new ways to communicate with customers and tell them we’re building redundant networks so this device remains reliable.”
“I think that carriers without a backup plan will now have one in place, and those that had a backup plan … will probably want to come up with one that’s a little quicker,” said Krista Grossman, spokeswoman for MobileMedia.
“Even back in the caveman days, the campfire went out every once in a while,” said PCIA’s Kitchen. “The result of this unfortunate situation will be a more reliable system in the future.”
Because carriers last week were so focused solely on restoring service to customers, there are several issues that have not yet been addressed, such as how much restoring service to customers cost the industry and whether carriers plan to reimburse customers for the inconvenience.
One issue not expected to be a factor is litigation. According to Mark Tauber, a lawyer specializing in communications with the law firm of Piper & Marbury L.L.P., provisions in the service contracts between the carriers and PanAmSat, as well as those between the carriers and their customers, likely will rule out any real damages claims. The only recourse carriers have to recoup the costs incurred in the massive effort to return service is if they had some type of business or service interruption insurance.
“In the absence of insurance, they’ll probably just eat the cost,” he said.
Tauber said he felt the only claim paging customers will have is for a service refund and doubted carriers will be held liable for any damages claims.
“Clearly the paging operators are going to argue that this was out of their control,” he said. “The customer would have to demonstrate a willful and wanton negligence … which is a pretty high standard … I wouldn’t be surprised if there are some cases coming out of this, but most won’t survive motions to dismiss.”
The most carriers have to fear, Tauber said, is customers churning to other providers that offer a better-protected system.
“You’re going to have a greater demand for backup now,” he said.
