Distributed Systems
3. Network Quality of Service (QoS)
Paul Krzyzanowski pxk@cs.rutgers.edu
What factors matter for network performance?
• Bandwidth (bit rate)
– Average number of bits per second through the network
• Delay (latency)
– Average time for data to get from one endpoint to its destination
• Jitter
– Variation in end-to-end delay
• Loss (packet errors and dropped packets)
– Percentage of packets that don’t reach their destination
Why do we care?
• We want to provide users with
– Good interactive performance
– Glitch-free, delay-free streaming video – Good quality IP telephony
Quality of Service Goals
• Provide applications with the data flow performance they need:
– Bandwidth, delay, jitter, loss
• Examples
– VoIP: low bandwidth, low latency (<150 ms), low jitter (<30 ms) – Video streaming: high bandwidth, longer latency OK
– Remote login: low bandwidth, relatively low latency, high reliability – File transfer: high bandwidth, high latency OK, high jitter OK , high
reliability
Why do we have QoS problems?
• We don’t have unlimited resources!
– On-host resources
• Multiple processes may want to send packets
– Network
• Available bandwidth
• Bandwidth mismatch
– Network routing & switching
• Processing delays
• Buffer space in routers & switches
– Too many queued packets → queue overflow & dropped packets → congestion
• Address quality of service with resource allocation & prioritization
Contributors to congestion
Bandwidth mismatch
Aggregation
Extra traffic resulting from
– Faulty hardware/firmware
– Excess retransmission from data corruption
– High-traffic apps or misbehaving hosts (e.g., malware generating floods)
Congestion leads to packet loss
10 Gbps 50 Mbps
1 Gbps 1 Gbps
Contributors to latency
• Packetization on host
– Create IP & MAC packets
• Serialization
– Send data onto the network
– One packet at a time – others are queued
• Propagation delay
• Processing delay on routers
– Input queue → output queue and packet inspection
• Queuing delay
– Wait for interface to be available
Contributors to jitter
• Packets arrive at a router or switch at the same time
– Used to lead to collision on a shared link – Buffers store packets
– Transmitted FIFO, not all at once!
– Dequeuing creates jitter
• Dynamic re-routing
• Transport layer
– Caching out of order packets → delayed delivery
• Out-of-order packets need to be buffered
• May need to be retransmitted – Retransmission
• Transport-layer retransmission
• End-to-end (huge!) jitter
FIFO output queue
Contributors to packet loss
• Buffer overrun on routers/switches (queue full)
• Bad transmission lines (e.g., signal degradation & noise)
• Faulty hardware
QoS Aware Networks
• A QoS-aware network will allow an application to specify its needs and will reserve sufficient capacity in the network to meet those goals
• Admission Control
– Applications request a particular quality of service from the network – If the request is granted, the application has the contract
• Traffic Control
– Classifies, queues, and schedules traffic within the network
Service Models for QoS
• No QoS (best effort)
– Default behavior for IP with no QoS – No preferential treatment
– Host is not involved in specifying service quality needs
• Soft QoS (Differentiated Services)
– No explicit setup
– Identify one type of service (data flow) vs. another – Certain classes get preferential treatment over others
• Hard QoS
– Network makes commitment to deliver the required quality of service – Host makes a reservation
• Traffic flows are reserved
Approaches to Network Resource Allocation
• Router-based vs. Host-based
– Router-based
• Each router decides which packet to forward & when
– Host-based
• Each host observes network conditions and adjusts its behavior
• Reservation vs. feedback-based
– Reservation-based
• Host asks for specific capacity from the network
• Each router allocates buffer space for that flow to satisfy the request.
• If a router cannot grant, reservation is denied
– Feedback-based
• Hosts sends data without reserving resources
• Host gets congestion feedback from a router or detects packet loss &
adjusts its transmission rate
Congestion Control
• Congestion:
– Caused by more incoming traffic than we can route
• Three approaches
– Admission control (input)
• Prevent congestion that has been detected from getting worse
• Do not allow packets for a specific flow, new flows, new virtual circuits
– Choke packets (feedback) – Queue management (output)
Congestion Control: Choke Packets
• If a router determines congestion, send back a “choke packet”
– Packet contains destination address identifying the port that is congested – This is a request from the router not to send packets to that destination
(voluntary)
• IP handles congestion by dropping packets
• Explicit Congestion Notification (ECN) allows end-to-end notification of congestion (RFC 3168)
Controlling Jitter
• Multiple queues per output port
• Queue based on priority
• Each packet is assigned to a queue
– Layer 2: 802.1p priority bits, MPLS EXP field
– Layer 3: IP ToS bits, protocol type, source/destination address – Layer 4: port numbers
– Any combinations
output queues
Classifier Scheduler
Incoming packet Transmitted
packet
Controlling Jitter: scheduling packets
• Simplest was a single FIFO queue
• Round robin from multiple queues
– Each class of service gets a fair share – not what we want
• Priority
– If there are always packets in a high priority queue, we get head of line blocking (simply packets holding up other packets)
• Weighted Fair Queuing (WFQ)
– Each queue gets a priority and a minimum % of link speed
output queues
Classifier Scheduler
Incoming packet Transmitted
packet
Flow Detection
• Flow: stream of related packets between hosts
– In IP, set of packets from one address:port to another address:port with same protocol
• Soft state
– Keeping state for a flow makes resource allocation easier (e.g., route, average packet size, packet arrival rate)
– This is called a soft state. There is usually no signaling information that is used to create these states.
• Flow detection in routers:
– Network controls flow rate by dropping or delaying packets – Define rules for traffic queuing, shaping, and policing
• drop TCP packets over UDP
• Discard specific UDP flows to ensure QoS for other flows
Queue Management With Flows
• Flow-based weighted fair queuing
– Per-flow queues
– Each queue is assigned a weight (volume)
• Class-based weighted fair queuing
– User-defined classes
• Protocols
• Input interfaces
• Specific source and/or destination IP addresses
• Each class is assigned
– Bandwidth
– Queue limit (# packets)
– Weight
Flow 1
Flow 2 Flow 3 Flow 4
Bandwidth Management
• Traffic Shaping
– Goal: regulate average rate of data transmission per flow – Queue packets during surges and release later
– Example: high-bandwidth link to low-bandwidth link
• Traffic Policing
– Goal: Monitor network traffic and discard offenders – Discard traffic that exceeds allotted bandwidth
Traffic Shaping: Leaky Bucket
• Visualization
– Bucket with a hole
– Filled up at a varying rate
– Water leaks at a constant rate
• Bucket = packet queue buffer
• If a packet comes in and bucket is full, discard packet
– Buffer overrun
• If there is nothing to transmit (bucket is empty)
– Buffer underrun
• Convert an uneven flow of packets into an even flow
Traffic Shaping: Token Bucket
• Bucket holds tokens that are generated at a certain rate
• To transmit a packet
– The bucket must hold and destroy a token(s)
• The token bucket allows a host to save up permission to send large bursts later
– Bucket size determines maximum burstiness
Traffic Shaping: Token Bucket
Desired average rate: r bytes/second
Add a token every 1/r seconds
If # tokens > b (bucket is full), discard the token
When packet arrives (size = n bytes):
if # tokens is < n
Traffic shaping: queue (delay) the packet until there are enough tokens
Traffic policing: drop the packet else
transmit the packet and remove n tokens
(in implementation, the “tokens” are just one number, not a collection)
Token bucket vs. Leaky bucket
• Token bucket: may be bursty
• Leaky bucket: cannot be bursty
Quality of Service in IP
• IP was not designed with QoS controls in mind
• IP cannot (normally) take advantage of QoS controls offered by an underlying network (LAN)
– You can’t count on a specific underlying network for end-to-end service
• Some QoS mechanisms created as add-ons
Quality of Service Problems in IP
• Bandwidth mismatches and aggregation
– Lead to congestion
• Inefficient packet transmission (not really a QoS issue)
– 59 bytes to send 1 byte in TCP/IP!
– 20 bytes TCP + 20 bytes IP + 18 bytes ethernet
• Unreliable delivery
– Software to the rescue – TCP/IP – But leads to jitter
• Unpredictable packet delivery
– No controls on bit rate, delay, jitter
– Packets may take different routes, resulting in different performance
Inefficient Packets
• Some software generates lots of tiny packets
– Can lead to head-of-line blocking – Nagle’s algorithm:
• Buffer new data if unacknowledged TCP/IP data outstanding
• Header/packet compression
– Link-to-link
– Header compression (RFC 3843) – Payload compression (RFC 2393)
– Trade-off: $ delivery vs. $ compression – Routers have to be aware
Differentiated Services (soft QoS)
• Treat some traffic as better than other
– Statistical - no guarantees
• Identify class of service & priority
– Router can use this data to make scheduling/dropping decisions
• Use on Internet (especially across ISPs) limited due to
peering agreement complexities
Differentiated Services (DiffServ)
• ToS field in IP header (bits 9…16)
– Differentiated Services Control Point (DSCP): first six bits
p p p x x x - -
Priority: 0-7
RFC 2474, 2475, December 1998
Differentiated Services (DS) field
DSCP value identifies class of service
Priority is used if these three bits are 0
Each router uses this value to determine how to prioritize the packet – it affects “per hop”
behavior of the router
Hard QoS: Integrated Services
• IntServ: Integrated Services (RFC 1633)
– End-to-end reservation of services
• Traffic specification (TPSEC)
– Define token bucket: rate & size
• Request specification (RPSEC)
– Specify levels of assurance
• Best effort (no reservation)
• Controlled Load: soft QoS – data rates may increase or packet loss may occur
• Guaranteed: hard QoS – tight bounds on delay
• Underlying mechanism
– Resource Reservation Protocol (RSVP)
Reservation & Delivery Protocol
• RSVP: ReSerVation Protocol
– Each unidirectional data stream is a flow
– Hosts request specific quality of service per flow – Routers reserve resources
– RFC 2205
• Every device through which data flows must support
RSVP
Other mechanisms
• Multi-Protocol Layer Switching (MPLS)
– Make packet routing more efficient … more like layer 2 – Each packet is assigned a routing label
(based on destination & packet priority)
– Routers make queuing and forwarding decisions based on this label
– MPLS routers can create end-to-end circuits using any protocol – QoS parameters
• Traffic Class
• ECN (Explicit Congestion Notification)
Other mechanims
• 802.1p: control QoS at the MAC layer (layer 2)
– Eight traffic classes defined in an ethernet frame – This is not an IP QoS mechanism
Priority Traffic
0 Background
1 Best effort 2 Excellent effort 3 Critical
4 Video <100 ms latency 5 Voice <10 ms latency 6 Internetwork control 7 Network control
Media Delivery Protocols
• Real-Time Control Protocol (RTCP)
– Provides feedback on QoS (jitter, loss, delay) – RFC 3550
• RTP: Real-Time Transport Protocol
– Not a routing protocol – No service guarantees – Provides:
• Payload identification
• sequence #
• time stamp
• RTP/RTCP do not provide QoS controls
Virtual Circuit Networks
• Avoid having to choose a route for every packet
• Route established when a connection is set up
• All traffic from source to destination flows on the same route
• Packets only need to contain a virtual circuit number, not destination address
• IP networking does not work this way
– TCP/IP’s virtual circuit is modeled entirely in the TCP driver in the OS – Routers are unaware of the presence of a virtual circuit
ATM: Asynchronous Transfer Mode
• Late 1980’s
• Goal: Merge voice & data networking
low but constant bandwidth
high but bursty
bandwidth
ATM
• Traditional voice networking
– Circuit switching
• Too costly
• Poor use of resource
• Does not lend to multicasting
• ATM: wide area networking with QoS controls
– Based on
fixed-size packets over virtual circuits
– Virtual circuit must be established before any traffic is sent
• Route is established and all participating switches commit to QoS
– Fixed-size cells provide for predictive scheduling
– Large cells will not hold up smaller ones – Rapid cell switching
ATM
• ATM cell
– 53-byte cell: 48-byte data, 5-byte header
• Sender specifies traffic type on circuit setup:
CBR Constant bit rate bandwidth Uncompressed video, voice
VBR Variable bit rate Avg, peak bandwidth
Compressed video, voice
ABR Available bit rate Minimum bandwidth
web access
UBR Unspecified bit rate – ftp
ATM
• Small cells lots of interrupts
– 622 Mbps link: 12 million interrupts per second
• ATM hardware supports an ATM Adaptation Layer (AAL)
– Converts cells to variable-sized (larger) packets:
AAL 1: for CBR AAL 2: for VBR AAL 3/4: ABR data
AAL 5: ABR data, simplified AAL 6: MPEG-2 video
