9 Switch Fabric Cards
SOFTWARE FUNCTION SET IN LOCAL SWITCH FABRIC CONTROL
One could—and should—argue that on a switch fabric card, an additional local control processor not in the data path does not have a significant cost impact and it provides additional capabilities for local control and bootstrapping, control interface hardware to the OAM&P card, and means for statistic traffic monitoring. While the cost and surface area are immediately clear, the software effort is not. Typically, the software development effort for these local control processors is in the order of hardware development cost of the digital portion of the board—but this is not the whole story.
If there is some local control on the switch fabric card, then the centralized OAM&P card can be divested of a few functions that do not need to be processed device-globally. As a result, the OAM&P card’s software can be reduced in complexity and improved in availability and robustness without impact on functionality.
Software on the switch fabric card is not in the data path. As a result, there is no requirement for real-time response of any microprocessor or microcontroller software on the switch fabric card. However, this does not mean that the robustness and the fault tolerance of the microprocessor or microcontroller software on the switch fabric card are not important. The opposite is true. First of all, the software
must setup and configure the hardware in its functions and features. The software also must perform statistic traffic data collection for traffic monitoring and traffic engineering.
It is of no use if the hardware by itself is crash-proof but the software running on it frequently crashes or must reboot due to software faults. That means that the software must be capable of surviving some hardware errors. Garbage collection must be flawless. In case something goes wrong anyway—like spurious hardware faults or nested Interrupt Service Routines that terminate because of run-time issues—a planned shutdown and restart (recovery) must be implemented. However, rolling recoveries must be avoided.
Functions that are typically executed on a local control processor on the switch fabric card are bootstrapping, setup and configuration of the switch fabric chip set, statistic traffic monitoring, error statistics and heuristics, and sometimes even more complex traffic analysis of data provided by the switch fabric chip set, such as:
number of routed packets or cells over a preset period of time per priority class;
dropped cells within that period of time per priority class; the minimum, maximum or average number of queued cells or packets in the switch fabric per priority class;
the minimum, maximum and average duration of queue or switch fabric traversal time or even scheduler information per priority class. BERs and cell or packet error rates can be provided by the switch fabric chip set as well and in that case will be collected by the local control processor for completion of the statistic traffic moni-toring to be forwarded to the centralized OAM&P card. With information like that, it is fairly simple to prove QoS fulfillment for true IPv6 compliance.
The software in the local switch fabric controller must be able to carry out the following most basic tasks:
• Setup
• Initialization
• Statistic Traffic Data Collection and summarizing the individual results from Queue Manager and Switch Elements
• Local error and status collection
• Local event handling within administrative statuses
• Status supervision and assignment (ACT, STB, DEF, UNA, and MBL)
• Communication with and reporting into OAM&P card
• Watchdog timer operation for local control
Therefore, it requires that the local controller communicates with the OAM&P card and can parse its messages and react upon them. It will also require a watchdog timer in order to make sure that the software can automatically be restarted in case of a software crash. There also should be a set of setup commands or APIs that enable the user of the software to set up the switch fabric in a few predefined robust default and standard configurations. One of these could be 16-port 40-Gbit/s configuration with strict priority in the queue manager and round robin in the crosspoint switch, and additional commands that could be used to change the setup in a finer granularity. The setup also should include an error notification system and basic functions to communicate with the OAM&P card.
The interface between the line card and the switch fabric card is restricted to the combined data path and control path for the payload and the header or LCI.
There are no commands sent from line cards to the switch card. In other words, the switch card does not terminate any traffic. Commands into the switch card originate only from the OAM&P card. Statuses are sent from the switch card into the OAM&P card. If necessary, the OAM&P card sends statuses or commands to the line cards, but it does it out of band. The switch fabric card does not act as a host; it is the client.
There is no API required in the line cards to handle switch fabric card messages, statuses, or commands directly. The OAM&P card or the OAM&P task on one line card will handle all this and forward it to the switch fabric card if necessary, when appropriate.
The local switch fabric controller software must support all OAM&P functions.
This is necessary to enable Traffic Management and Engineering. The OAM&P functions are run and carried out by the OAM&P card; they consist of functions that collect statistic data, functions that enable the switch to operate, and functions that handle errors and exceptions. OAM&P functions are crucial in a router or a switch. These functions typically are implemented in a centralized OAM&P card, but not on the switch fabric card.
In addition to these, the local control processor and its software must be able to set and recognize and react accordingly to the following statuses:
• ACT (active)
• STB (standby)
• DEF (defective)
• UNA (unavailable)
• MBL (maintenance blocked)
All status transitions are received from the OAM&P card, and the switch fabric status can be sent back. However, the switch fabric controller is not supposed to change the administrative status by itself. It is only allowed to receive set status commands, verify them, and place the switch fabric plane into the appropriate status.
It also should diagnose the switch fabric chips and send notifications to the OAM&P card. ACT is the active state, and the OAM&P card can send a set status command from ACT to any other status. There is only one ACT switch fabric card; the others are in any of the other statuses. A transition from UNA or DEF to ACT is not allowed. DEF or UNA must be transitioned to MBL first, then to STB. Transitioning one switch fabric plane from STB to ACT means that the other plane that was previously in ACT will automatically be placed into STB unless it goes into DEF.
It is required for the switch fabric controller to hold its status in local memory, compare incoming status transition commands with the allowed set of transitions, implement it, and send a status update message back to the OAM&P card.
There must be APIs or function calls that easily handle these statuses and the transitions. Additionally, there must be a signal for the switch fabric controller to the OAM&P card, used for notifications in case of locally reported errors. This can be done using messages or optionally by asserting an IRQ towards the OAM&P card.
In addition to these functions, there should be functions or APIs to reset indi-vidual links, queue managers, and the switch elements.
Finally, the watchdog timer should supervise CPU activity. If the CPU does not reset the timer within a predefined period of time, then the timer expires and resets the CPU via its RESET signal so that the software can reboot. It may make sense that the software can be reset via command, too.
CONCLUSION
Switch fabric applications differ drastically based on their intended use. In some applications, changes of the setup are rare and switching can be considered static or quasi-static. Under those circumstances, a crosspoint switch or an optical cross-connect can be the perfect solution.
Switch fabric applications requiring high bandwidths while maintaining low latency, QoS awareness, and low cell losses are typically found in TelCo-grade datacom environments. Data communications over these devices have requirements that match or exceed NEBS standards in terms of performance, throughput, reliabil-ity, and system availabilreliabil-ity, upgradeabilreliabil-ity, and scalability. Throughput must not be compromised under any circumstances; Quality of Service must be maintained in order to fulfill the Service Level Agreements.
These requirements are reflected in current switch fabric architectures, which are at the heart of the current and future generations of routers and ATM switches. The predominant switch fabric architecture that has emerged out of this is the Combined Virtually Output Queued (CVOQ) switch fabric. It is an Output Queued (OQ) and Virtually Output Queued (VOQ) cell-based switch fabric with queue managers, cross-bar switches and other components that provide the memory and the scheduling or arbitration, together with Serializer/Deserializer (SerDes) components. Some imple-mentations of this architecture combine multiple functions into one single chip; others separate these physically or logically. Mostly, the queue managers incorporate the schedulers and the memory required for the OQ and the VOQ functions. In some instances, the queue managers incorporate High Speed Serial Links (HSSLs) to connect to other line card components and the crossbar switches.
In the above-mentioned CVOQ switch fabrics, the queue managers can be placed either on the line cards or on the switch fabric cards. Both placements seem to have good reasons. However, one placement makes more sense than the other.
We have shown that in router architectures with VOQ, OQ or CVOQ architec-tures work better with switches that have the queue managers and the crossbar switches combined on the switch fabric card. They offer better throughput, lower latency, higher levels of link rate utilization, and distribute power dissipation better and more evenly over the entire device. They therefore avoid producing hot spots.
This also contributes to the possibility of incorporating line cards with combined throughput rates that are lower than the native queue manager line rate. Putting more logic and memory onto the switch fabric card significantly reduces the amount of logic and buffers on the line cards. Since there are more line cards than switch fabric cards, this additional effort on the switch fabric card will be overcompensated by the reduced amount of logic and memory on the line cards.
177