Single Control Card + Multiple Line Card Architecture

An example of the single control card + multiple line card architecture is shown in Figure 8.4. There is one control card, with four line cards and a switch fabric card. The switch fabric and the line cards are connected through high-speed serial links. The control card and the line card can be connected by PCI or Fast Ethernet. The latter is sometimes known as the internal Ethernet or Ethernet on the backplane. The control card– to–line card interface is used for messages that need to be processed by the control card’s processor(s) and for messages originated by the control card to be sent to the physical ports. The bandwidth requirement on this control card–to–line card interface is not as high as the line card–to–switch fabric interface. However, when systems with a large number of line cards are built, the traffic on this interface can become an issue. System designers often address this by using a high-speed switched connection like Switched Gigabit Ethernet or Infiniband.

8.3.1 Line Card–Line Card Communication

Line card–to–line card data transfer is through the switch fabric. The switch fabric card provides the switching function for the payload sent by the line cards. In the IPS, if a frame is forwarded from Port 1 of Line Card 1 to Port 3 of Line Card 2, it is sent on the serial link from Line Card 1 to the Switch Card and then from the Switch Card to Line Card 2. A header on the frame indicates the port number on Line Card 2 where the frame is to be sent. The line card (switch fabric electronics) usually has two parts—a line card component and a switch fabric component. Second, the frames are sliced into fixed-length cells and sent on the link. Switching is done faster with fixed-size cells, and the delay is

predictable.

8.3.2 Line Card–Control Card Communication

A messaging interface is used for interactions between the tasks on the control card CPU and the tasks on the line card CPU, as shown in Figure 8.3. The types of interactions in Table 8.1 take place on this messaging interface.

Table 8.1: Messaging interactions.

Type of Message/Function Direction of Communication change of a port, response for a statistics query from Control Card

Payload message after CPU processing

Control Card to Line Card

MAC frame forwarded on all ports if destination address is unknown

Designing Embedded Communications Software

by T. Sridhar ISBN:157820125x

CMP Books © 2003 (207 pages)

With this foundation, you explore a development model that addresses the complete range of issues in the design of embedded communications software, including real-time operating systems, hardware and software partitioning, layering, and protocol stacks.

Chapter 2 - Software Considerations in Communications Systems Chapter 3 - Software Partitioning

Chapter 4 - Protocol Software

Chapter 5 - Tables and Other Data Structures Chapter 6 - Buffer and Timer Management Chapter 7 - Management Software

Chapter 8 - Multi-Board Communications Software Design Chapter 9 - Going About the Development

Appendix A- Examples from Commercial Systems Glossary of- Common Terms and Acronyms References

Heartbeat Message Control Card to Line Card

Used by Control Card to test whether the line card is functional and also for testing the link between the two. Used for redundancy

Heartbeat Response Message

Line Card to Control Card

Sent by line card to indicate the current status

As the table indicates, each message type from the control card to the line card has an equivalent message type in the reverse direction. A heartbeat message is typically sent as a request from the control card to the line cards. The line card has to respond to this heartbeat request with a response in the reverse direction. If the control card has not received responses to a series of heartbeat messages, it can timeout the line card and flag it as down—note that the control card can only report this as an error but not determine if the line card has failed or if the link has failed.

Protocol messages destined for tasks on the control card are forwarded by the line card. In the IPS, these could be control plane packets like routing protocol PDUs and packets destined to the IPS, such as TCP/telnet packets The line card will make the distinction by comparing the destination IP address in the packet with the IP address of the switch ports or the broadcast/valid multicast address. If there is a match, the packet will be passed on to the CPU (i.e., control card)—the typical IP end node

operation.

Figure 8.3: Single Control Card + Multiple Line Card Architecture.

8.3.3 Message Types and Protocols

Table 8.1 indicates a simple summary of message types required on the interface. In more complex scenarios, the control card can affect the forwarding behavior of the line card. This can require additional types of messages from the control card. One scenario is the modification of the traffic management parameters on the line card. This can be the changing of the queue sizes and/or algorithms for packet discard. This area is being investigated by the Forwarding and Control Element Separation (ForCES) Working Group of the IETF. ForCES is specifying the protocol and the format and semantics of the messages between the control element and the forwarding element in a communications system with plane separation.

The message protocol used between the control card and the line card is usually proprietary since it is only used “inside the box.” However, some systems do run protocols such as TCP to provide reliable communication between the control card and the line card, but this is inefficient since TCP is a general-purpose protocol.

While TCP (or a proprietary protocol) may be used for transport, there is a need to have another protocol for card-to-card communication, Inter-Card Communications Protocol (ICCP), which will run over the reliable transport protocol, like TCP. This protocol will identify the type of message, specify

Designing Embedded Communications Software

by T. Sridhar ISBN:157820125x

CMP Books © 2003 (207 pages)

With this foundation, you explore a development model that addresses the complete range of issues in the design of embedded communications software, including real-time operating systems, hardware and software partitioning, layering, and protocol stacks.

Table of Contents

Designing Embedded Communications Software Foreword

Preface

Chapter 1 - Introduction

Chapter 2 - Software Considerations in Communications Systems Chapter 3 - Software Partitioning

Chapter 4 - Protocol Software

Chapter 5 - Tables and Other Data Structures Chapter 6 - Buffer and Timer Management Chapter 7 - Management Software

Chapter 8 - Multi-Board Communications Software Design Chapter 9 - Going About the Development

Appendix A- Examples from Commercial Systems Glossary of- Common Terms and Acronyms References

Index List of Figures List of Tables List of Listings List of Sidebars

the command or response, contain parameters related to the messages, and so on.

8.3.4 Partitioning Software Between Control Card and Line Card

Chapter 3 detailed the functional architecture of a Layer 3 switch or router. This was a

single-processor design in which tasks interfaced with each other using Inter-Process Communication (IPC), and the driver programmed the hardware controller as well as received and transmitted frames from the Ethernet links (see Figure 3.5). In this section, this basic design is expanded into a distributed architecture using a control card with its control processor and a line card with multiple Ethernet ports.

Figure 8.4: Software Partitioning on multiboard router

In Figure 8.3 (139 ), the routing tasks such as OSPF, BGP and SNMP are now located on the control card. These tasks build routing tables and provide information to a Route Table Manager (RTM). The RTM has the following functions:

It is responsible for route leaking or route redistribution, based on a configured policy—e.g., which BGP routes are to be known to OSPF.

It uses the “raw” routing table/Routing Information Base (RIB) information from the various routing protocols and constructs a forwarding table/Forwarding Information Base (FIB).

It distributes the FIB to the individual line cards.

The RTM `is responsible for providing the FIB to the individual line cards. It is also responsible for FIB modifications. Moving from the single-processor architecture of xrefparanum to the control card + line card architecture of Figure 8.3, we note that the IP Switching Task in xrefparanum is now on the line card. This task uses the RTM-provided FIB information to program the hardware on the line card for forwarding, including building the hardware forwarding tables. The details vary with the type of hardware- forwarding or -switching silicon devices.

8.3.5 Partitioning Abstraction

Abstraction implies that the higher layer software does not know task distribution across multiple

Designing Embedded Communications Software

by T. Sridhar ISBN:157820125x

CMP Books © 2003 (207 pages)

With this foundation, you explore a development model that addresses the complete range of issues in the design of embedded communications software, including real-time operating systems, hardware and software partitioning, layering, and protocol stacks.

Chapter 2 - Software Considerations in Communications Systems Chapter 3 - Software Partitioning

Chapter 4 - Protocol Software

Chapter 5 - Tables and Other Data Structures Chapter 6 - Buffer and Timer Management Chapter 7 - Management Software

Chapter 8 - Multi-Board Communications Software Design Chapter 9 - Going About the Development

Appendix A- Examples from Commercial Systems Glossary of- Common Terms and Acronyms References

boards and treats the software as if it were running on a single CPU system. Higher layer software tasks continue to use IPC to communicate with another task, except that the task may be running on another board. The communication mapping to another board is done in the IPC implementation and does not affect the two communicating tasks.

As shown in Figure 8.4, the routing protocol tasks perform as if they are on a single board system with 8 ports. This function is provided in the abstraction layer, which will be the only component changing when moving from a single-board to a multi-board system. In Figure 8.4, we call this a Distribution Abstraction Layer (DAL). Note that the DAL on the control card has a corresponding DAL on each of the line cards.

The DAL needs to perform the following functions:

Maintain mapping of interfaces to board number + port number

Maintain mapping of task and resource names so that distribution is hidden from the higher layer tasks. Application tasks only use the name for the resource—mapping is done at the DAL.

Provide IPC services across the cards using system services as appropriate