With the fall and winter openings of personal communications services in New York City, the issue of network control will become highly visible. Nowhere else is there likely to be so complex and difficult an installation or one that will require operation so well controlled. In short, nowhere else will the reasons for the use of Network Operations Centers be so clear and commanding.
Wherever the site of the network, however, the reasons to design and construct NOCs (sometimes called Network Management Centers) remain in force. To begin with, NOCs are necessary for efficient, effective network emergency management. Second, they are important for day-to-day network maintenance and operation. Third, they are extremely useful marketing tools. And, finally, they are excellent business strategic planning tools. Let’s look at these uses one at a time.
What is a NOC?
For those unfamiliar with the term, a NOC is a space devoted to surveillance of a network’s operation, monitoring equipment for functionality-and especially for malfunction-and the center from which adjustments or repairs may be made directly to network components or from which information and directions can be sent to field technicians.
In the room are large-screen displays (often six feet high or more) displaying network maps and status information, as well individual consoles with computer workstations, each with two or three monitors. In addition, the consoles govern incoming signals from video, audio and short-wave sources, as well as public emergency radio equipment.
Emergency responses
The most dramatic use for the NOC is emergency response and management when, for example, a switch fails or a software bug creates a problem, as happened so visibly to America Online this past summer or, on an even broader basis, in the event of an earthquake, hurricane or other natural disaster. Wireless companies are linked with governmental disaster relief systems and are used to help restore services.
Such a failure will show up not only on the individual monitors, but on the large screen, showing the location and nature of the problem. Because it can be seen by all personnel, the appropriate steps can be taken by “drilling down” from the generalized location (such as a service area) to the particular rack or individual piece of equipment or even the specific malfunctioning circuit board.
The capacity to drill down is inherent in the sophisticated combinations of hardware and software of the Operation Support Systems. But it can be put to use only when one knows where to focus. The design of the NOC is the key to such use. Placing the large-screen displays in view of everyone ensures that no one will become so involved with individual concerns as to lose sight of what is occurring in the enterprise as a whole. In effect, seeing the maps allows the monitoring personnel to establish teamwork patterns to conduct a sort of electronic triage and deal with problems in order of importance, thereby minimizing, perhaps even eradicating, customers’ failed attempts through the network. No one expects disaster, but if it happens, the NOC directs and speeds the resumption of business. The use of television and radio provides important data regarding emergency conditions, and radio communications with involved agencies are critical to emergency response.
Daily maintenance
Less dramatic, but even more important in the overall operation of the network, on a day to day basis the NOC manages environmental conditions to ensure constant, transparent network functions.
It warns of hazards like water building up under the computer floor or the draining of batteries if electric service goes out; it records whether the standby generator starts each day in the programmed routine maintenance cycle. It looks at high traffic areas of the network for exceptions and problems.
When problems cannot be fixed by direct communication with switches and computer hardware, the NOC contacts appropriate service technicians to direct them for repairs. It also notifies the customer service department with information to give to customers regarding what has gone wrong and how long the system will be down.
An example of this oversight function occurred shortly after the command and control center for Pacific Bell’s Internet division was completed.
In a control room, with 10 people and one large screen, on its first day of operation, someone looked up at the large-screen display and noticed a fault in the BP system. This was reported to the parent company because the fault was not in the Internet division’s system itself.
In time someone in Pacific Bell would have found the problem, but the control center enabled Pacific Bell to make the repair and improve service more quickly and more easily.
It is useful to point out that by isolating the network from outside carriers, the NOC can also determine when a problem is not within its network. This permits the network to notify its affected customers about the problem’s outside source and see to it that the external carrier is informed of the need for repair. (In this sense, a properly designed and functioning NOC is a sort of insurance against blame.)
Planning: proximity to data center
Proximity of the data center to the NOC is essential. There are many cables and connections between the NOC and the data center, so both simplicity of installation and cost are affected by distance. Furthermore, although the NOC and data center are heavily integrated electronically, not all “repairs” or corrections can be made via computer instruction. Both when setting up a network and when operating it, some jobs need direct human intervention.
It’s literally necessary at times to run from one space to the other. If the architecture/technical design team is aware during the planning phase of the functional relationship between the NOC and the data center, planning proper adjacencies isn’t terribly difficult.
Design: teamwork
Clearly the quality of NOC design is crucial, and good design demands effective teamwork among the architect, engineer, technical consultants and the other members of the team. Experience and successful previous working relationships are important.
In no other area is this more applicable than in the design and construction of NOCs. The team will have to include members who understand the need for flexibility, for interrelationship with the marketing group, as well as the needs for power and a/c; the team will have to cope with architectural design, mechanical, electrical and plumbing engineering, audio/visual and acoustical design, lighting, console design and perhaps other factors.
The handling of building technology is extremely important, Not only must the space contain the necessary equipment, but it must do so in the right way. As just one example:
Minimum NOC ceiling height is determined by the height above consoles needed for good sightlines to the large display screens plus the height of the screens themselves. Here teamwork is essential. Where are the consoles to be placed?
What are the important proximities for workstation functions? How many consoles and workstations should there be? What about lighting and a/c demands for both the large screens and the individual consoles? Such questions, and their answers, will determine not only the parameters for the large screen placement, and therefore ceiling height, but often the location of the NOCs and data centers themselves, which need not be near administrative offices.
Again an example: one installation we designed was put in a building previously used by IBM. The top floor had been used for a data center. In part the building was chosen because with the raised floor, redundant power and air handling capacity already in place, both construction time and cost could be held down.
Building choice
The importance of finding the right location and getting the right building cannot be too strongly emphasized.
For those who construct new buildings, the problems are lessened. For those who intend to lease-and they are a majority-finding the site and then getting the lease worked out and signed will usually take more time than any other part of the process. Especially when leasing, therefore, we recommend consultation with the architectural/engineering/technical team before shopping for a building. Be sure to employ a team that has had experience in NOC projects. You don’t want a barber learning to shave on your face! Begin together early with programming, site selection, and fit-up design. Choose a building only after the design team determines what your equipment needs-and therefore your space needs-will be. When the building has been selected, the architect can begin the specific design by doing his/her needs analysis to ensure that the design agrees with the intended functionality, determined by a needs analysis related to computers, video, audio, control systems, etc.
This analysis should contain interviews with both the management, for its projections of service demands, and the people who will actually be working in the control center. While these analyses are performed, the details of the lease can be settled; when that is settled, the actual process of construction will be able to move ahead swiftly.
Cost benchmarks
The cost of a NOC facility is incremental, dependent upon too many factors to predict specifically here. One should assume that the NOC is going to cost anywhere from $350,000 to $1 million, depending on the number of operators and screens and the capabilities proposed.
Marketing use
The design of the NOC should consider its potential marketing use. We recommend locating a large conference room adjacent to the NOC, in which there will be presentation systems to sell the company and demonstrate customization capabilities. This adjacency is especially useful when working with potential corporate clients. Nothing will impress such a PCS user more than a view of how everything is under control. One device for such assurance that we have used for several clients-among them AT&T and SP Telecom-is having a wall dividing the NOC and the marketing conference room made of electroluminescent “snap glass.” At the appropriate point in the discussion of the network’s capabilities, a remote switch turns the wall instantly from frosted glass to clear, revealing the control room in operation. As a new technology looking for market share, PCS networks need the reassuring message that things are under control and aren’t likely to go wrong-but if they do, the NOC has everything needed to manage the resumption of service.
Planning uses
For the PCS provider, the NOC is also a valuable internal communications tool. It serves as a central communications point for all information regarding the network-and therefore acts as a constant mechanism of control by determining whether there are problems of system capacity-either excessive or insufficient-and making it possible to fine tune business goals and procedures. The NOC has several “spokes of communication,” as we call them:
To engineering-for technical monitoring and an ongoing update of available new technology.
To maintenance-for constant oversight of equipment.
To the customer service call center-for information about customer needs and demands.
To marketing-for projections of short-term customer growth.
To strategic planning-for a synthesis of all the above as business plan.
From the outset in the process of designing a NOC, these spokes feed into the decisions regarding what to do and how to do it. By beginning with a “vision session,” a time of establishing the company’s goals, and then going on to compare the needs to the price tag, management can determine what to do immediately, what to leave to staged development, and what to watch as business develops. When used well, the NOC will be a central instrument for making the vision a reality.
John Whitcomb and David Person are consultants for Paoletti Associates, a member firm of the Strategic Team of Allied Resources (STAR), based in San Fransisco.
