Data and Voice Network Infrastructure
Denis Levesque Mike Milne
Published by APPA:
APPA is the association of choice serving educational facilities
professionals. APPA's mission is to support educational excellence with quality leadership and professional management through education,
research, and recognition.
Reprint Statement:
Except as permitted under copyright law, no part of this chapter may be reproduced, stored in a retrieval system, distributed, or transmitted in any form or by any means - electronic, mechanical, photocopying, recording, or
otherwise - without the prior written permission of APPA.
From APPA Body of Knowledge
APPA: Leadership in Educational Facilities, Alexandria, Virginia
This BOK is constantly being updated. For the latest version of this chapter, please visit
www.appa.org/BOK .
APPA
1643 Prince Street
Alexandria, Virginia 22314-2818 www.appa.org
Data and Voice Network Infrastructure
Introduction
Networking technologies serve as the telecommunications backbone of most universities and colleges, and have a critical role in their operations. The design, installation, and operation of network systems was initially treated as an administrative overhead function and was not typically considered a priority function.
The following factors have caused an evolution in how network technologies are used and how academia relies on them:
Distribution of data. Most end-user workstations do not have a local
copy of all necessary data required by the end user. Institutions commonly use a network operating system to mount “virtual disks.” Although this solves many problems, such as the ability to archive data, an network outage removes the ability of the user to access this data, and poor network performance, such as congestion and latency, will adversely affect users.
Distribution of applications. End users rely heavily on e-mail, and
business systems are typically architected using a client-server model, or more recently, a web-based model. All such applications require a functional network.
Convergence of data, voice, and video services. Many universities
and colleges have deployed VoIP (Voice over Internet Protocol) systems to replace legacy telephony systems, and the use of video technologies is growing.
Integration of “stovepipe” networks. Most core infrastructure
systems that previously operated on a private “stovepipe” network are being migrated onto the converged networks. These systems include building automation, access control, intrusion, and fire alarms.
Reliance on Internet connectivity. In addition to the factors
detailed, organizations increasingly rely on the Internet. Reliable and robust communications must now exist both internal and external to the organization.
VoIP as a Driver of Converged Networks
Traditionally, large organizations such as colleges and universities have operated dual different network infrastructures, one that provides traditional POTS (Plain Old Telephone Service) and another that provides traditional
data-networking services. The POTS network typically used lesser quality wiring such as EIA/TIA Category 3, and connected POTS telephone sets in the user environment to one or more PBX (Private Branch Exchange)
systems. Cabling to support data networking was usually installed as an afterthought. Before the widespread availability of commercial-grade VoIP systems, this separation resulted in the expense of installing and operating two different networks.
Today, VoIP is the main factor driving converged networks. For new buildings in the planning phases, the major benefit is that both voice and traditional data traffic can be supported over the single physical cable plant designed to carry data traffic. As existing POTS voice equipment reaches its end of life, many organizations are replacing their existing POTS
systems with VoIP.
In addition to the reduced capital cost building wiring, additional features are present in VoIP systems, such as integration with campus directory services and advanced ACD (Automatic Call Distribution). Many VoIP systems also have a programmatic interface that allows organizations to integrate their own applications into the telephony world.
Network Architectural Model
Networks tend to grow through "osmosis," and, as additional requirements develop, the network is expanded with whatever technologies are "kicking around" or whatever can be purchased at the lowest cost. Although this approach may meet immediate requirements, the net result is that over time, the overall topology of the network will resemble spaghetti.
Suboptimal routing paths and congestion are likely to occur, and the ability to sustainably grow the network will diminish.
These problems can be avoided by designing and adhering to a network architectural model. Although this may seem like additional work, in the long term it will allow better growth potential, as well as reduce installation and support costs.
The "Classic" Network Architectural Model
The "classic" model has three distinct layers:
Access layer. The purpose of the access layer is to serve as a point
of connection for end-user systems. Network switches are typically used to provide connectivity using 100 Mbps Ethernet over UTP
(Unshielded Twisted Pair) wiring, but 1,000 Mbps (Gigabit Ethernet) is also used and is likely to grow over time. The access layer is typically
located in a TR (telecommunications room), and there is usually at least one TR on each floor in a building. UTP cables run horizontally from the TR out to the end-user workstations. Uplink cables from the access layer to the distribution layer run vertically through the
building, and can use either UTP or fiber-optic cables, depending on the distances and speeds involved.
Distribution layer. The purpose of the distribution layer is to serve as
a connection point to aggregate connections from all of the access layers, as well as to link them to the core layer. There is usually one distribution layer per building. Uplink cables from the distribution layer to the core layer typically use fiber-optic cables, as the distance and speed requirements typically exceed the capabilities of UTP wiring.
Core layer. The purpose of the core layer is to serve as the network
"backbone" by interconnecting all distribution layers together. Interconnect cables between core layer components typically use fiber-optic cables. Devices that form the core layer are usually connected to each other in a fully meshed topology to provide fault tolerance.
Figure 1. An Example of a "Classic" Network Architecture
In the design in Figure 1,
distribution devices will not cause a network outage.
The failure of a distribution device will cause an outage to the whole building served by the failed device.
The failure of an access layer will result in an outage to the floor served by the failed device.
Resiliency Improvements
Network fault tolerance can be improved by pairing adjacent buildings (i.e., buildings 1 and 3), and adding resilient connections from the access layer of building 1 to the distribution layer of building 3 and vice versa. In the diagram in Figure 2, buildings 1 and 3 use a dual-homed access layer that now protects both buildings against a failure of the distribution layer. The resilient connections are shown in magenta. Network path selection would be done by the IEEE 802.1 Spanning Tree Protocol, and in the event of a failure, a failover to the alternate path would occur. Note that building 2 has not had this improvement made and its distribution layer is still a single point of failure.
Figure 2. An Example of a Resilient Network Architecture
Resiliency is not normally designed into the access layer, as most end-user workstations have only a single network interface card.
Organizations that do not have a large number of buildings or users may benefit from a "collapsed backbone" design, which contains only access and core layers.
Figure 3. An Example of a "Collapsed Backbone" Network Architecture
The primary benefit is the reduced cost, as a distribution layer is not required for each building. The access layers in each building are dual homed to two different core layers for resiliency. This design can also be easily modified to be a classic three-tiered design in the future by simply adding a distribution layer to each building.
Structured Cabling Systems
Structured cable systems were introduced in the 1980s in an attempt to standardize how building wiring was installed. This standard model keeps cables neat and tidy, prevents them from accidental damage, and creates “design-once, build-many” model that can be easily replicated as new buildings are constructed.
Figure 4. Structured Cabling — Access Layer to Workstation
Both the 100- and 1,000-Mbps Ethernet standards, as well as the evolving 10-Gbps Ethernet standard, specify that the maximum length of a cable run from the switch port to the end-user computer cannot exceed 100 meters (325 feet). This cable run length is inclusive of all patch cables. Cables longer than the 100-meter standard will cause excessive signal attenuation and degradation, and ultimately will result in data transmission errors.
UTP Wiring Selection
Without exception, the common standard used for end-user workstations is Ethernet delivered over UTP wiring; 100 Mbps is the most commonly used speed, and a small and growing amount of 1,000-Mbps workstations are used. UTP cable selection must consider both existing speeds, as well as potential growth and new technologies.
EIA/TIA Category 5 wiring will support speeds up to 100 Mbps Ethernet
EIA/TIA Category 5e wiring will support speeds up to 1,000 Mbps EIA/TIA Category 6 wiring will support speeds up to 1,000 Mbps
Wiring is usually installed during the construction of a building and rewiring a building is possible only with a major renovation. Logic would dictate that any new installations should consider Category 5e wiring as a minimum, as it can support Gigabit Ethernet to the desktop. The standards for 10-Gbps Ethernet are still evolving, but it is unlikely that it will support Category 6 wiring.
Adherence to Local Building and Fire Codes
The correct cable jacket composition type must be used in accordance with local fire regulations. Cables are designated with a “FT” rating that
indicates its ability to withstand fire. Cables that run through plenum
spaces designed for air movement within a building are normally required to use an FT-6 fire-rated jacket, whereas nonplenum cables can generally
use a lesser FT-4 rating. In addition to the ability to withstand fire, cables with a higher FT rating are less susceptible to act as combustible material that would cause a fire to spread, and may emit a lower level of toxic materials when burned. Readers are cautioned that this information is for example purposes only, and they should consult their local fire and building codes for definitive information.
Termination at the Workstation End
The cable run is normally terminated at the workstation end in the end-user space with an RJ-45 Jack. These jacks can be wall mounted into
drywall/sheetrock walls, mounted against baseboards, installed in “Jiffy Poles,” or perhaps installed in modular cubicle furniture.
The end-user workstation is normally connected to the jack with a patch cable. A jack-numbering and -labeling nomenclature should be developed, and each cable run should be clearly labeled both at the user connection as well as in the telecommunications room (TR). Commercial database software products exist that can assist in accurate record keeping.
Termination in the Telecom Room
Cable runs are normally terminated in the TR onto a patch panel. Patch cords are then used to connect individual cable runs to the switch ports. Numerous patch panel standards exist, including the following:
Those based on “BIX” patch frames traditionally used by telephone systems
Patch panels that terminate the cable run in a standard RJ-45 connector
Proprietary methods such as AT&T Systemax.
Protection of the Cable Run
Cable runs from the TR to the end-user workstation should be as protected as possible, within reason. If the environment is susceptible to vandalism, enclosed cable trays or conduits should be used. Care must be taken to avoid routing UTP cables near sources of electromagnetic or radio
frequency interference, and cables should not be close to AC power cables, transformers, electric motors, photocopiers, and the like. Wiring testers from companies such as Fluke Instruments or Tektronix can determine if external interference is causing data errors.
Cable Plant Certification
Some organizations find it desirable to adhere to an industry or vendor standard such as NORDX/IBDN or AT&T Systemax and build their cable
plants to achieve these vendor certifications. If this certification is desired, additional precautions must be taken. This certification can be used to provide a high degree of faith in your physical cable plant, but the cost of achieving this certification can outweigh the benefits.
Location of the Telecom Rooms
Several items should be considered when deciding where to locate TRs: The TR should be located such that all horizontal cable runs do not exceed the 100-meter length restriction. In large buildings, this may not be possible and may require the creation of more than one TR per floor.
The TR footprint should be sized based on the number of end users that will be connected. Consideration should be given to provide additional space to allow for growth.
In general, conduits that run vertically in a building are usually located in the center of the building near elevator shafts or stairwells. Ideally, the TRs should be located such that the vertical building conduit runs through the TR. In addition, it is advantageous to “stack” the TRs such that they are vertically aligned.
To protect against equipment theft, tampering, and unauthorized connections, TR locations should be lockable with restricted access. Intrusion alarms should be considered if the value of the equipment is high or if a history of theft or vandalism exists.
An electrical supply adequate to power all installed equipment must be present. Overcapacity should be considered to account for growth as well as allow for equipment that may be used during
troubleshooting.
Sufficient ventilation and/or cooling must be present to ensure the room temperature does not exceed the environmental capabilities of the equipment. Both electrical-consumption and heat-generation statistics can be obtained from data sheets provided by the
equipment manufacturers. Generally speaking, TRs containing only access layer equipment require only air movement and ventilation, whereas the equipment comprising the distribution and/or core layers will require air-conditioning.
If necessary, the use of UPS (Uninterruptable Power Supplies)
systems should also be considered to ensure the equipment remains powered in the event of a loss of building power.
Telecom rooms do not normally house equipment other than network switches or routers, and once a room is built only small incremental growth (i.e., the addition of a new switch for additional port capacity) is typically experienced. If the rooms will also be used to house
servers or other equipment then both the electrical supply and cooling capacities must be sized to accommodate all equipment. The design should also consider potential future growth in terms of rack mounting space, as well as electrical and cooling capacity.
Location of Telecom Rooms Containing Core Layer Equipment
As outages caused by power failure, construction damage, and natural disasters tend to be localized, TRs containing core layer equipment should be located in different buildings.
Uplink Cables Between Access and Distribution Layers
Cables between the TRs on each floor and the distribution layer can use either UTP or fiber-optic cables. With existing technologies, if UTP cables are used, the maximum length is 100 meters and speeds exceeding
Gigabit Ethernet are not possible. In most cases, links between the access and distribution layers operate as Layer 2 switched trunks, with routing functions being performed by the distribution devices. This allows different VLANS to be deployed in each access layer TR without incurring the
expense of Layer 3 routing capabilities in the access layer.
Most organizations use fiber-optic cables to link the access layers to the distribution layer. This allows cable lengths to exceed the 100-meter restriction and also provides the ability to migrate to a higher-speed interconnect link such as 10-Gbps Ethernet in the future.
Two types of fiber-optic cables exist. Multimode fiber can usually operate up to 2,000 meters in length, whereas singlemode fiber can operate up to 10,000 meters in length. The specific maximum distance requirements that can be used with either type of fiber-optic cable are different depending on the manufacturer.
Regardless of the type of cables installed, the cable runs should be clearly labeled at both ends. The fiber-optic cables are typically terminated on patch panels at either end, and patch cords are used to connect the equipment.
Location of Telecom Rooms for Distribution and Core Layers
The same general considerations apply in the selection of Telecom Rooms for the Distribution and Core Layers as for the Access layer.
It is common practice within one building to have the Distribution Layer for a building and the access layer for the floor in the same room. In addition, a Core layer could also be resident in the same room.
Uplink Cables Between Distribution and Core Layers
As the distribution and core layers are typically in different buildings,
fiber-optic cables are typically used. To maximize future flexibility in terms of length and speed, singlemode fiber is commonly used.
In large campus environments, consideration should also be given to the physical cable layout interconnecting the buildings. Rather than creating a full-mesh connection (running cables from every room directly to every other room), it is common practice to create two or more interconnect junction points, and then connect each building to these common
locations. With the appropriate number of patch cords, a full any-to-any connection is easily constructed if necessary.
The two distribution uplink cables from each building should also take a diverse physical path to protect against cable cuts.
Network Management Considerations
Although network management is a large topic, most organizations find that at minimum they need the following knowledge about their networks to deliver reliable service:
Knowledge of what network components are functioning properly (i.e., up or down)
Knowledge of the traffic levels at key connection points in the network Real-time alerts to inform staff when a device failure occurs or when traffic levels exceed performance thresholds.
Most network devices (i.e., switches and routers) support the SNMP (Simple Network Management Protocol). The SNMP is enabled on all network devices, and an SNMP network management workstation then periodically polls the status of each device to determine its availability and performance levels.
Many software applications exist for availability monitoring, ranging from open-source (free) tools, to fairly low cost commercial products such as What's Up Gold or Intermapper, through to high-end packages, such as Hewlett Packard Openview.
The same wide range of tools exists for traffic and performance monitoring, and most academic organizations use open-source tools such as MRTG (Multi Router Traffic Grapher, www.mrtg.org), or CACTI (www.cacti.net). An example screen from a CACTI system is shown in Figure 5.
Figure 5. An example screen from a CACTI system
Thanks to CACTI for use of system. Readers are encouraged to visit www.cacti.net for more information on this product.
For more information
These technologies continue to evolve at a fairly rapid pace and readers are encouraged to consult industry standards. Wiring standards for both copper and fibre based wiring are maintained by TIA/EIA and are available at www.tiaonline.org/standards/. Standards for Ethernet technologies are maintained by the IEEE and can be obtained at
http://ieeexplore.ieee.org/xpl/standards.jsp.
Summary
It is highly recommended that APPA members at universities and colleges develop a solid relationship with the information technology organizations that provide computing and communications services to their respective campuses. Clearly, convergence of data, voice, and video services is under way. Relationships with other campus organizations that are responsible for building automation or access control—in fact, any communications connectivity—should also be nurtured. Facilities
organizations need to keep abreast of emerging trends and their impact on the campus physical plant infrastructure.
