To assess the performance of the ADR algorithms, we define the following four met- rics: optimality, energy consumption, convergence speed, and stability.
5.5.1 Optimality
An ADR algorithm should optimise a link, which is the configuration that minim- ises energy consumption per data transfer, given the channel state. Ideally, to assess the performance of an ADR algorithm, we would want to see how close it gets to the optimal configuration. However, determining the optimal setting is hard, as this re- quires global knowledge of the channel state that varies over time. With the packet traces collected in section 5.6, however, we do have this global knowledge, and can calculate the optimal setting. The optimal setting we can determine from our data set is the lower bound for the ADR algorithms.
In case we do not have this information, we can compare it to the static ADR algorithm instead. The static ADR algorithm gives the upper bound, and how much the other ADR algorithms improve on this, but it does not give us an idea on how close we are the to the optimal, and how much more room there is for improvement. To be able to compare the different configurations, we use EKB, which is the joule per kilobit of effective data transferred. We do need the PRR to calculate EKB. In a live system, where we do not have the PRR for every single setting, we cannot use EKB to evaluate the performance of an ADR algorithm. We could calculate the joule per kilobit for every setting, without taking the PRR, or effective bitrate in consid- eration. This would however bias the optimality towards the highest data rate and lowest transmit power, which could have a fairly low PRR. For example, in our exper- imental data, a setting of SF7, BW500, CR1, TP10, would give an energy consumption of 4.25 mJ/kbit, while it has a PRR of 0.85, and therefore an EKB of 5.00 mJ/kbit. A setting of SF7, BW500, CR1, TP13 would be much better, as it has a PRR of 0.98 and an
5.5. ADR Performance Metrics 115 EKB of 4.90 mJ/kbit, although its energy consumption would be 4.80 mJ/kbit. Purely
based on the energy consumption, the first configuration would be better than the second one. However, taking in consideration the actual performance, the second configuration is more energy efficient.
Using EKB does assume that the MAC protocol does take some effort in delivering packets, like the usage of ACKs or repeated transmissions. A MAC protocol that does not do this, but does want some form of guaranteed delivery would be better off in using the PRR as optimality metric. Or alternatively, add an extra constraint for the minimal PRR desired.
5.5.2 Energy Consumed
To get to the optimal setting, an ADR algorithm needs to send packets to assess the channel state and make a decision on what setting to move to. These channel assessing
packets, or probe packets, can be either passive or active. Passive probing is where the
channel state is assessed from the regular traffic a node produces. This means there is no additional cost in running the ADR algorithm. In the case of active probing, whereby a node actively sends packets to assess the channel state, running the ADR algorithm does come with additional energy cost.
The other way in which energy consumption of an ADR algorithm is calculated is from how quickly it approaches the optimal setting. Ideally an ADR algorithm should quickly move to the lower energy consuming configurations, instead of lingering on a high power setting for a long time. Similar to the optimality metric, this favours high date rate configurations with potentially low PRR, in this particular case we also favour aggressively moving in to the high data rate, low transmit power configurations, even though it may be too much.
To compare the energy consumption, we calculate the energy consumed for trans- mitting packets in joules from the start of the algorithm up until a stable state has been
reached. For some algorithms (Probing and Optimistic Probing) this will be when the search space has been exhausted, for other algorithms (TTN) this will be the last time when the algorithm changed configurations after running for an extended period of time.
This metric favours algorithms that use the minimal number of (extra) packets and are more aggressive to move towards the optimal setting.
5.5.3 Convergence Speed
Similar to how much energy an ADR algorithm uses, it is also important to consider its convergence speed. That is, how long it takes before it approaches the optimal or stable setting. Convergence speed is related to energy consumed, in that a high convergence speed requires less transmitted packets, and thus less energy is consumed.
We define convergence speed as the total airtime of the packets transmitted from the start of the algorithm up until it reaches a stable state. The definition of stable state is the same as with the energy consumed; either the last time the algorithm changed configurations when running for an extended time, or when the search space has been exhausted.
This metric favours aggressive algorithms, that use the minimal amount of packets and quickly move towards high data rate (but not necessarily low transmit power).
5.5.4 Stability
Over time, the channel state changes, and the optimal configuration as found by the ADR algorithm may not be the best any more. Either our current configuration per- forms much worse than a potential better one, or there is more room for picking a better configuration.
An ADR algorithm should be able to detect the channel state has changed, and choose the appropriate course of action. It can either be that an ADR algorithm picks
5.6. Gathering Real-World Link Data 117