Previous versions and working principles

Previous implementations of TED already existed for the Linux Kernel. After being developed by Vittorio Ghini and presented in 2011 as part of the ABPS architecture[53], it has been reviewed and modified several times as the Linux Kernel was evolving and being updated. The last working version of TED before my contribution was developed for version 4.0.1 of the Linux Kernel by Gabriele Di Bernardo and Alesssandro Mengoli in their bache- lor’s thesis works[54][52]. They introduced support for IPv6 and based the interlayer information passing mechanism on the sk buff kernel structure. Simplifying, an allocation of the sk buff structure contains the headers and the payload of a packet together with some additional information that the kernel may use to correctly manage it. We can then think of a direct re- lation between packets and sk buff structures. When a user space process sends a packet through a TCP or UDP socket, the kernel will create a new sk buff structure and link it to the transport layer header and the payload

40 5. Project development

of the packet. Then the sk buff structure is passed to the IP management module of the kernel. At this stage if the payload of the transport layer packet is bigger than the MTU then the kernel splits the sk buff structure in as many sk buff fragments as it needs. That is, each new IP packet fragment correspond to a separate sk buff structure. These new sk buff structures are stored in a linked list for future retrieval. Each new sk buff structure, or the old one if fragmentation is not required, are passed to the layer 2 module of the selected transmitting NIC. In case the selected NIC is a SoftMAC WiFi device, sk buff structures land on the mac80211 mod- ule which manage 802.11 frames and where the TED core logic resides. For each sk buff structure mac80211 creates a data-link-layer frame which is then passed to the NIC device driver. Just before the actual transmission, TED stores in internal ted info structures some information about every new created frame. ted info structures are collected in a linked list and in- dexed through frame sequence numbers. When the corresponding AP sends an acknowledgement frame back to the station or when the station gives up after a certain amount of retransmissions, TED retrieves the corresponding ted info structure from its list, enriching it with the delivery status infor- mation and the retransmission count.

The following snippet shows the ted info structure.

s t r u c t t e d _ i n f o { _ _ l e 1 6 m a c _ f r a m e i d ; /* 8 0 2 . 1 1 f r a m e id */ u i n t 3 2 _ t t r a n s p o r t _ p k t i d ; /* UDP d a t a g r a m id */ /* 8 0 2 1 1 l a y e r info */ u8 a c k e d ; u8 r e t r y _ c o u n t ; /* n e t w o r k l a y e r f r a g m e n t info */ u16 f r a g m e n t _ d a t a _ l e n ; u16 f r a g m e n t _ o f f s e t ; u8 m o r e _ f r a g m e n t ; s t r u c t t i m e s p e c t x _ t i m e ; s t r u c t t i m e s p e c r x _ t i m e ; s t r u c t t e d _ i n f o * next ; };

5.1 TED 41

Some of the information contained in the ted info structure will be delivered to the user space process which started the send operation.

Currently TED only works with UDP transport layer packets.

To let the user space process receive TED notifications it has to send datagrams using the sendmsg() function. This function allows sending ad- ditional control information to the kernel along with the payload. Such piece of information consists of a pointer address to an internal integer variable (the UDP datagram id).

This is done through the msg header structure which must be passed as argument to sendmsg(): m s g _ h e a d e r . m s g _ i o v = iov ; /* C o n t a i n s the d a t a g r a m */ m s g _ h e a d e r . m s g _ i o v l e n = 1; m s g _ h e a d e r . m s g _ c o n t r o l = cbuf ; m s g _ h e a d e r . m s g _ c o n t r o l l e n = s i z e o f ( cbuf ) ; cmsg = C M S G _ F I R S T H D R (& m s g _ h e a d e r ) ; /* C o n t a i n s ctl info */ cmsg - > c m s g _ l e v e l = S O L _ U D P ; cmsg - > c m s g _ t y p e = T E D _ C M S G _ T Y P E ; cmsg - > c m s g _ l e n = C M S G _ L E N ( s i z e o f ( i d _ p o i n t e r ) ) ;

/* Copy the a d d r e s s of our user s p a c e p o i n t e r ( i d _ p o i n t e r ) * into the c msg data . L a t e r the k e r n e l will put a new id * d i r e c t l y in our user s p a c e p o i n t e r a c c e s s i n g cmsg data . */

m e m c p y (( u i n t 3 2 _ t *) C M S G _ D A T A ( cmsg ) , & i d _ p o i n t e r , s i z e o f ( i d _ p o i n t e r ) ) ; m s g _ h e a d e r . m s g _ c o n t r o l l e n = cmsg - > c m s g _ l e n ;

s e n d m s g ( sd , & m s g _ h e a d e r , M S G _ N O S I G N A L | M S G _ D O N T W A I T ) ;

In this way when the kernel takes control of the datagram to be sent, it generates a datagram identifier and copies it directly to the user space variable. Such identifier is also stored in the sk buff structure in order to let it pass through the transport and network layers, finally allowing TED to store it in the transport pktid variable of the corresponding ted info structure(s). Let us notice that since the transport layer datagram may be fragmented, more ted info structures can refer to the same datagram. After TED is aware of the delivery status and the retransmission number of a transmitted frame it can send a notification the user space process. It achieves that by putting a message in the socket error queue containing deliv- ery status, retransmission count and fragment information. In practice TED

42 5. Project development

creates a new sk buff, which corresponds to the notification message and passes it as argument to the sock queue err skb() function. Such func- tion also requires the pointer to the sock structure that identifies the socket which owns the original sent datagram as argument. This latter pointer to the socket structure is stored in every sk buff. Clearly, the user space pro- cess must have enabled error notifications through the IP RECVERR socket option during the socket creation and should be listening for error messages from the socket after sending datagrams. From the asynchronous nature of this approach comes the importance of a shared identifier between kernel and user space. For instance the user space process may receive TED no- tifications referring to old sent datagrams and without identifiers it may be hard to know which datagram the notification refers to. This also applies to fragmentation information: even if the user space process identifies the correct datagram, the latter may be composed of many fragments. How the user space process uses such information is application dependent but take for instance the case of the client proxy: it may retransmit a non delivered datagram through the second NIC. We will see in the next section that at the moment the client proxy does not perform datagram retransmission but relies on TED notifications to decide when to start transmitting the follow- ing datagrams to the second NIC too. After the efforts of Mengoli and Di Bernardo, TED provided a good implementation of the ABPS “early-packet- loss-detection” mechanism. Hower it did lack IPv6 fragmentation support thus my first contribution in TED was implementing such functionality.