HNM Calling Center Central Office International Circuits Local Exchange Shelterized Local Exchange Fiber Optic Cable Metallic Cable Environmentals Switching Microwave Satellite Loop Test System Wireless Local Loop
Network and Facility Management: Needs, Challenges and Solutions
Graham Jones
Harris Broadcast Communications
[email protected]
New tools are needed to enable broadcasters to efficiently manage the modern broadcasting station and network environment. Broadcasters are required to introduce increasingly sophisticated systems and equipment, at the same time making more efficient use of highly trained staff, and reducing system downtime. Many station groups and state networks are planning to consolidate control and monitoring of multiple stations in centralized locations. This paper looks at the requirements for sophisticated system-wide and integrated network management solutions for broadcast applications, identifies some of the challenges, and indicates some of the factors to consider in choosing and implementing a network management system.
What is Network Management?
Network management may be defined as the management of network resources in order to meet service-level objectives effectively and efficiently. A network management system (NMS) typically includes tools for gathering data on network elements, tools for data storage, analysis and prediction, tools for configuration and control of network elements, and tools for performance and system planning management.
LAN / WAN Management
NMS tools first evolved for the management of large local area computer networks and today are used for every type of network, from local area networks (LAN) to wide area networks (WAN) and the Internet. They enable network element configuration, for example Ethernet router set-up, monitoring of faults and alarms, measurement of network performance, and status monitoring such as computer disk space, or an application status. SNMP (simple network management protocol) is a major component of computer network management.
Telecom Network Management
Telecommunications networks incorporate a wide range of equipment
and a variety of services for consumer and business customers. Network management is widely used for such systems and, using a variety of protocols, typically provides monitoring and configuration of all elements. These may include everything from telephone switches to microwave links, and satellite ground stations to fire alarms. Other management capabilities include network optimization, service activation, automatic testing, trouble ticketing and billing mediation. Rather than simply telling the operator that there is a problem, the NMS improves the service being provided. A full-featured telecom NMS will comprise a whole range of software modules, working together as an integrated Operational Support System (OSS).
Broadcast Networks
Television, radio and data broadcast networks comprise major sub-subsystems for production, contribution, distribution and transmission in one form or another. Today they virtually all use computer LAN/WAN technology for some parts of their operations or support infrastructure, and they also have some of the same characteristics as telecom networks. At the most basic level this means they include facilities, equipment, communication links and delivery of services to an end-user customer. This suggests they may be candidates for network management. Some sectors, such as satellite television networks, the new satellite radio networks and the emerging data broadcast networks, are already using NMS, in some cases for management of program chain equipment as well as for their LANs and WANs. Others—the national networks, program originators, state networks, station group owners and individual stations—are just awakening to the possibilities. For convenience, throughout this paper the term
network management or NMS is used, but the systems described are intended to cover the management of complete
end to end systems including all aspects of the facility. This paper presents the perspective of the over-the-air broadcaster, but much of it is equally applicable to satellite and cable networks.
Why do we Need Network Management?
Changing Environment in BroadcastingThe world of broadcasting is changing with the introduction of digital television and with competition from alternative means of distribution. All broadcast stations will soon have an additional output channel for DTV, possibly with HD (high definition) or multiple channels of SD (standard definition), they may already have more than one analog channel, for direct cable access or LMAs (local marketing agreements). Data broadcasting offers a whole new range of services and may bring additional complexities of operation.
As this is going on, there is a consolidation in local station ownership, with more stations in the hands of larger groups. They are looking at ways to improve efficiency, including more centralized operations, both for programming operations and engineering support. Many public broadcasting state networks have the same ends in mind. As well as the drive to reduce costs, another factor in this trend is the difficulty of finding suitably qualified engineering staff.
Changing Technology
An increasing amount of equipment is software-based, utilizing computer technology, with GUIs (graphical user interface), serial ports and LAN connections and SNMP (simple network management protocol) built-in. Stations and networks rely on LANs and WANs, both as a means of program material distribution and as a support infrastructure. Traditional methods of fault diagnosis are frequently not applicable, and tracing the root cause of service failure may be a difficult task.
Current Monitoring and Control
Stations today typically have multiple independent monitoring and control systems, including those for: transmitter site, satellite receiving and up-link, ENG receive site, building power, HVAC, fire alarms, security, and for the important on-air operations automation system. Each of these will usually have a dedicated status and configuration terminal. In addition, many individual items of equipment now require a separate PC (personal computer) for monitoring and configuration. The result of this is an explosion in the number of PCs in the master control room and equipment area.
What Do These Changes Mean for the Broadcaster?
There are major implications for the broadcaster:
More equipment to be monitored and controlled by the same or fewer staff More remote operational procedures
More problems in diagnosing faults and taking corrective action Push for greater efficiency
All of which mean more stress for the operators and engineers, and implies a need for network management tools.
User Needs
This section identifies some of the needs and desires of broadcasters that a network management system can help to fulfil, based on feedback received from senior level broadcast engineers over the past 12 months. Of course the specifics will vary somewhat for different users depending on individual circumstances.
High Level
At the highest level the main desires are to: Improve system availability Improve quality of service Decrease trouble response time Reduce maintenance costs
Reduce staffing requirements and training levels Reduce operating expenses
Increase operating efficiency
Functionality
The functionality of the NMS system should:
Provide control, configuration, status, alarm management and fault diagnostics for the majority of the technical plant
Allow multiple sites and multiple stations to be monitored and controlled from a single or multiple points Consolidate current monitoring and control activities into a single system, with more capability
Provide intelligent support for systems operations and maintenance Allow for additional system management functions including:
- Automation confidence monitoring and log reconciliation - Resource management
- Bandwidth allocation / Scheduling / Traffic / PSIP - Data Broadcasting
Needs in Detail
The NMS should provide:
Status and alarms in near real-time - Events viewer and log
- Graphical displays with custom screens
- Screens with geographic, physical, flow and logical views of the system and equipment Escalating alarm functions
- Notification systems—screen display, audible, page, telephone, e-mail Configuration and control (not expected to be frame accurate)
Ability to work with new and existing equipment Ability to work with local and remote equipment Ability to work with existing element managers Ability to integrate with automation system
Data analysis and reporting - Trending
- Statistics
Smart, rules-based, fault analysis and executive action Automatic equipment configuration
Auto-discovery
Industry standard open system architecture Software hooks for third-party applications
Users and Operators
The NMS should provide secure access, with individual profiles, over internal networks, intranet, internet, and dial-up lines. Users or user-grodial-ups should have personalized system views and control privileges dependent on their authority and operational needs.
Operator positions may typically be required as follows:
Within the plant
Master Control Engineering Operations Director of Engineering Building Manager
External to the plant
Director of Engineering at home or on the road Group or Network – NOC
Manufacturer’s technical support center Third party NOC
Challenges
In designing a network management system for broadcast use there are many challenges that must be addressed, the more important ones are set out below.
One Size Does Not Fit All
Flexibility
The network management system should be flexible enough to cope with the many variations of size and complexity of broadcast stations and networks. Hardware and software options should allow a system to be sized appropriately for the intended purpose. The system architecture should be scaleable and expandable to allow the management system to grow as the user needs grow. Preferably it should allow for multi-level systems, with a hierarchical architecture, and with the option of either a centralized or distributed configuration. There should be provision for back-up capabilities with full redundancy when required. Similarly with the feature set: the software should be modular in design to allow the use to select only the functions that are required.
Hierarchical Architecture for groups and networks
For station groups, state networks, or national networks, there is a great advantage in having a hierarchical system architecture. This allows the use of stand-alone systems at local broadcast stations that provide all the benefits of network management to that station itself. Multiple stations can then be overseen by a supervisory system at a centralized location that can ultimately reduce or even eliminate the need for some staff at the local station – perhaps at certain times of the day.
Equipment Interfaces
There is a considerable challenge with introducing network management into the broadcast world due to the large number of different device types, manufacturers and models for equipment that are in use by broadcasters in the US and around the world. These have many different interfaces for control and monitoring, including:
SNMP network interface Serial interfaces
- Multiple electrical standards - RS-232, RS-422, RS-485 - Multiple protocols IEEE-488 GPI - Contact closures - TTL Analog
The network management system should have tools for developing and configuring all device drivers, including those for proprietary protocols, in a straightforward, timely and cost efficient manner.
Legacy Equipment
The issue of interfacing to devices is significant in the case of legacy equipment, in this context defined as that without SNMP interface capability. While it would be desirable for stations to replace all old equipment as they transition into the DTV age, business reality means that legacy equipment will be around for many years to come. To be useful to a broadcaster it may be essential for the network manager to communicate with the following types of equipment that may not have SNMP capability:
Master control switchers Video tape machines and carts Video servers
Character generators and graphics systems Test and monitoring
Studio-transmitter links Transmitters
Satellite receive and up-link systems Studio production and post-production Weather radar system
Facility power, generator, UPS, HVAC
Facility fire, smoke detection, elevators, security
The installed base of different types of such equipment means that many hundreds, if not thousands, of variations of interface drivers may be required, ranging from the most simple two-state device to extremely complex equipment. In addition to the device driver, for systems using a graphical interface there may be significant effort involved in producing custom screens related to the device, depending on the complexity of the device and the desired user interface for showing the equipment.
Element Managers
The challenge of interfacing to existing element managers is not as severe as to legacy equipment direct since there are far fewer types of element managers currently deployed. However, to achieve maximum benefits from a new network management system it may be necessary to integrate it with an existing element manager that was not designed to work with a supervisory management system. This may require additional systems engineering effort, use of such interfaces as do exist (for example a printer port), or possibly modifications to the third party system itself. Element managers that may need to be integrated include:
Satellite receive and uplink control systems ENG microwave receiver site control systems Encoder control
Routing switcher control Video server control Remote test equipment HVAC control systems Fire and Security systems
Web Browser Element Managers
Where an item of equipment has a web-browser interface available for remote control and monitoring, then it may be possible to use the network manager as a gateway to the browser screens. This eliminates the need to develop custom screens for that item of equipment. In this case, however, there is the challenge of ensuring that an alarm or status change in the equipment is brought to the notice of the network management system for display and action; and also of transferring any data that may be required to be logged from the device to the NMS.
SNMP Equipment
Any item of equipment with an SNMP interface can be readily integrated into a network management system. Depending on the complexity of the device and the desired user interface for showing the equipment, there still may be significant effort involved in producing custom screens related to the device. There is an increasing amount of equipment now available with SNMP, including:
Computer-based systems (depending on operating system) Network elements
- Routers, bridges, gateways, etc. Newer broadcast equipment
- DTV encoders - PSIP generators - Data encapsulators - Data servers - Video servers - Format converters - Test equipment - Smart frames - Automation systems
Any item of equipment with a provided SNMP proxy agent
SNMP
For those who are new to this topic, this section provides a very brief overview of SNMP, the simple network management protocol. “Simple” evidently because it provides a limited set of management commands and responses. It was developed in the 1980s to monitor and maintain large Ethernet computer networks. SNMP runs on top of TCP/IP and enables communication between an SNMP Manager and an
Agent to determine and report status
of network elements. SNMP is the
SNM P SNMP Devices Proxy Agents Proxy Agents Proxy Agents Proxy Agents Proxy Agents Proxy Agents Storage SNMP Manager SNMP Agent Device
de-facto standard operations and maintenance protocol for the Internet, and is probably the key technology that enabled the growth of the Internet. It is rapidly becoming the broadcast industry standard interface for remote monitoring and configuration on new equipment. One of the advantages of SNMP is that it places a low stress on the network on which it is running. One disadvantage is that it is not suitable for commands and controls requiring frame-accurate time implementation.
The SNMP Manager is part of the network management system; it provides the user interface and holds the database of device information. The SNMP Agent for the network element is the device’s point of contact to the SNMP Manager and determines what information can be relayed between the device and the manager. If the agent is not resident in the device, then it is known as a Proxy Agent. The manager and the agent work with a Management Information Base (MIB). The MIB acts as the translator between SNMP and the device-specific data being managed, and documents the set of available hierarchical information.
The SNMP Agent populates the MIB with attributes for the managed device, which are identified with unique Object IDs (OIDs), it also sets up a polling cycle (solicited information) for the SNMP Manager to query the agent and perform a MIB walk to get variables and build a network tree. The Agent maintains the current value of the variables and sends an unsolicited alert message when an attribute changes in the MIB. SNMP exchanges data through PDU (protocol data unit) messages for reading and setting data and monitoring network events, SNMP v.1 defines five basic commands (primitives) for status monitoring and control of the device as shown below. SNMP versions 2 and 3 have slightly different sets:
Get retrieves an object instance from the agent GetNext retrieves the next object instance from the agent Set sets values for object instances within an agent Get Response response from the agent after receiving a get command Trap unsolicited alert message from the agent to the manager
Integration with Automation
One challenge that requires special mention is what happens when we add network management to a station with an existing automation system that is controlling legacy equipment. In this case we will assume that the automation system interface to the SNMP manager is provided through an SNMP proxy agent running on a proxy client computer (Link 3 in the figure below).
The proxy client gathers information on the status of the automation system, communicating with the automation system device server through the automation system LAN (Link 2), also communicating with the file server and client work stations. The control interface from the automation system device server to the majority of automation-controlled legacy equipment (Link 1) is typically implemented through RS-422 serial ports. In order to independently gather detailed information on the status of the controlled equipment it would be preferable for the NMS to separately communicate with the station devices as shown in Link 4 on the drawing. The problem issue is that the majority of legacy devices have only one serial port. If that is in use by the automation system, then it is not available for the network manager.
One solution to this dilemma requires enhancement of the automation interface to the device to allow a full set of status information to be gathered by the automation device server and placed in the automation system MIB. In that case, care must be taken that the critical frame-accurate performance of the automation system is not degraded by the extra workload placed upon it. An alternative solution would require modification of the legacy device to provide more than one simultaneously available remote control and monitoring port. This is not in general an attractive option, but may be possible in some cases, for example with a master control switcher where tally contacts may be available to provide remote indications of some device statuses.
Solutions – Design Factors to Consider
It is outside the scope of this paper to discuss specific network management products that are available, but in making a choice of system and vendor there are various factors and design features that should be considered. Apart from the obvious issues of functionality discussed in the User Needs section above, other things to look for include:
System Architecture
For use in the broadcasting environment the following system architecture is desirable: Highly reliable distributed client-server architecture
Provision for full redundancy with automatic database synchronization Multi-user, multi-tasking system
Works in multi-protocol, multi-vendor environment Scaleable and expandable
Software Design
There are advantages in having a modular software architecture, where you only pay for what you need. Modules may include:
Core network management
Advanced data analysis and reporting Automatic rules-based executive action Performance management
Trouble ticketing
Provisioning and service activation
APIs should be provided to allow integration with third party applications. The use of tools such as CORBA (Common Object Request Broker Architecture) provides efficient management of links between software modules and of system objects in a distributed environment.
Software Architecture
Software design should comply with Open System Standards, using a layered architecture such as the one set out in TMN – Telecommunications Management Network Model as shown here.
Solutions – Implementation
Typical Network DiagramExamples of Graphics Screens
Business Management Service Management Network Management Element Management Network Elements Corporate WAN HBM Server Network Supervisor Network Router Modem FlexiCoder HBM Client News Operations
PSIPplus DataPlus SwitcherRouting EquipmentTest Format
Converter ServerVideo
Terminal Server
Other Serial Interface Devices GPI and Analog
Interfaces Other SNMP Devices HBM Server Master Control Network Router Modem
TYPICAL HBM SYSTEM DIAGRAM LOCAL STATION STUDIO LOCATION
CORPORATE NETWORK OPERATIONS CENTER
OTHER LOCAL STATIONS
TCP/IP Ethernet Local Area Network
LOCAL STATION TRANSMITTER SITE
HBM Client Chief Engineer Terminal Server Transmitter Test Equipment RS-232 Public Telephone Network ARX-H200 Receiver RS-232 RS-232 RS-232 Data Server STL STL Other Serial Interface Devices SNMP Devices Remote Terminal Unit Smart Frame
TCP/IP Ethernet Local Area Network TCP/IP Ethernet Local Area Network
RS-232
SNMP SNMP SNMP SNMP
SNMP SNMP SNMP SNMP
GPI and Analog Interfaces HBM Client Corporate User Network Router Automation System SNMP Subsystem Element Manager Remote Terminal Unit
Geographic Geographic Region Equipment Rack
Network Operations Center
Conclusion
Network management systems can help broadcasters provide more complex services efficiently, with fewer staff and at lower cost. In selecting a network management system, careful analysis of user needs and customization of device interfaces and graphics screens is required to achieve maximum benefits. Suitable management products are now becoming available and will most likely become a commonly used tool in the broadcast station, group and network.
