Black Book dition. Critical Testing for Wireless Client Devices. Edition April September 2015 ADVANCED MPLS

(1)

Black Book

dition

Edition 10

Wi-Fi Device Testing

(2)

(3)

Your feedback is welcome

Our goal in the preparation of this Black Book was to create high-value, high-quality content. Your feedback is an important ingredient that will help guide our future books.

If you have any comments regarding how we could improve the quality of this book, or suggestions for topics to be included in future Black Books, please contact us at [email protected].

Your feedback is greatly appreciated!

This publication may not be copied, in whole or in part, without Ixia’s consent.

RESTRICTED RIGHTS LEGEND: Use, duplication, or disclosure by the U.S. Government is subject to the restrictions set forth in subparagraph (c)(1)(ii) of the Rights in Technical Data and Computer Software clause at DFARS 252.227-7013 and FAR 52.227-19.

Ixia, the Ixia logo, and all Ixia brand names and product names in this document are either trademarks or registered trademarks of Ixia in the United States and/or other countries. All other trademarks belong to their respective owners. The information herein is furnished for informational use only, is subject to change by Ixia without notice, and should not be construed as a commitment by Ixia. Ixia assumes no responsibility or liability for any errors or inaccuracies contained in this publication.

(4)

(5)

How to Read this Book

The book is structured as several standalone sections that discuss test methodologies by type. Every section starts by introducing the reader to relevant information from a technology and testing perspective.

Each test case has the following organization structure:

Overview Provides background information specific to the test case.

Objective Describes the goal of the test.

Setup An illustration of the test configuration highlighting the test ports, simulated elements and other details.

Step-by-Step Instructions Detailed configuration procedures using Ixia test equipment and applications.

Test Variables A summary of the key test parameters that affect the test’s performance and scale. These can be modified to construct other tests.

Results Analysis Provides the background useful for test result analysis, explaining the metrics and providing examples of expected results.

Troubleshooting and Diagnostics

Provides guidance on how to troubleshoot common issues.

Conclusions Summarizes the result of the test.

Typographic Conventions

In this document, the following conventions are used to indicate items that are selected or typed by you:

 Bold items are those that you select or click on. It is also used to indicate text found on the current GUI screen.

(8)

Dear Reader

Ixia’s Black Books include a number of IP and wireless test methodologies that will help you become familiar with new technologies and the key testing issues associated with them.

The Black Books can be considered primers on technology and testing. They include test methodologies that can be used to verify device and system functionality and performance. The methodologies are universally applicable to any test equipment. Step-by-step instructions using Ixia’s test platform and applications are used to demonstrate the test methodology.

This tenth edition of the Black Books includes twenty-five volumes covering key technologies and test methodologies:

Volume 1 – Higher Speed Ethernet Volume 2 – QoS Validation Volume 3 – Advanced MPLS

Volume 4 – LTE Evolved Packet Core Volume 5 – Application Delivery Volume 6 – Voice over IP

Volume 7 – Converged Data Center Volume 8 – Test Automation

Volume 9 – Converged Network Adapters Volume 10 – Carrier Ethernet

Volume 11 – Ethernet Synchronization Volume 12 – IPv6 Transition Technologies Volume 13 – Video over IP

Volume 14 – Network Security Volume 15 – MPLS-TP

Volume 16 – Ultra Low Latency (ULL) Testing Volume 17 – Impairments

Volume 18 – LTE Access

Volume 19 – 802.11ac Wi-Fi Benchmarking Volume 20 – SDN/OpenFlow

Volume 21 – Network Convergence Testing Volume 22 – Testing Contact Centers Volume 23 – Automotive Ethernet Volume 24 – Audio Video Bridging

Volume 25 – Wi-Fi Client Device Testing

A soft copy of each of the chapters of the books and the associated test configurations are available on Ixia’s Black Book website at http://www.ixiacom.com/blackbook. Registration is required to access this section of the web site.

Ixia is committed to helping our customers’ network perform at its highest level, so that end users get the best application experience. We hope this Black Book series provides valuable insight into the evolution of our industry, and helps customers deploy applications and network services—in a physical, virtual, or hybrid network configuration.

(9)

Critical Testing for Wireless

Client Devices

Test Methodologies

Wi-Fi has become the de-facto standard of communication for local area networks. It’s fast, flexible and cheap, which has led to a proliferation of Wi-Fi devices in the market. While this trend is overall a good thing for the market, a large number of devices still exhibit issues resulting in poor experience for the end user.

This booklet aims to address this quality gap by proposing various test methodologies to verify the performance, functionality and security resiliency of Wi-Fi client devices.

(10)

(11)

Introduction

Over 10 Billion Wi-Fi enabled devices have been shipped to date, and this number is projected to grow at 10% for the years to come. Although a majority of the Wi-Fi device shipments today are made up of smart phones, tablets, e-readers and laptops, there is a growing trend of Wi-Fi becoming the access technology for several application specific devices, including:

 Home – security cameras, set-top-boxes and media players, thermostats etc.

 Hospitals – patient monitors, infusion pumps, oxygen monitoring devices etc.

 Industry – machine diagnostics, sensors, smart grids etc.

For most of these devices, good Wi-Fi connectivity is critical to their functioning and quality of experience delivered to the end-user.

(12)

Wi-Fi Device Issues

Wi-Fi devices have been around for over a decade now; however their use-cases have come a long way from their initial days. These new use-cases impose several requirements that the technology was not originally designed to address. Wi-Fi technology has been evolving to keep up with these new use-cases, but it’s not immune to issues. From a test perspective it’s these issues that need to be targeted comprehensively.

 Unlicensed frequencies – As Wi-Fi operates in unlicensed frequencies, devices typically have to contend with interference issues coming from other devices operating in the same frequencies. This can come from Bluetooth, Microwave, DECT and other Wi-Fi devices.

 Lack of deployment standards – There are no standard deployment models. This leads to lot of variation between each deployment. Devices have to cope with these variations when they operate.

 Legacy devices – Wi-Fi standards evolve pretty fast. This model is possible because the standard imposes backwards compatibility on devices. Often Wi-Fi devices have to operate in an ecosystem with several legacy devices. This causes a range of issues, because legacy devices operate at different PHY rates and usually occupy the medium for much longer periods.

 Roaming issues – Handover/Roaming is a relatively new concept in Fi. Originally Wi-Fi was designed for fixed or nomadic wireless access; however due to its recent popularity in the enterprise several new use-cases now require a seamless handover. Moreover, the devices are completely responsible for planning and executing the handover. This creates an extremely challenging scenario for devices.

 AP/Device Interoperability Issues – Interoperability issues exist in any technology; Wi-Fi is not immune to it either.

 QoS – Wi-Fi networks mostly carried best effort data traffic in the beginning; however, with its surge in popularity, several new use-cases require QoS. This is extremely challenging in Wi-Fi networks because of a lack of centralized control.

 Radio Resource Management – Transmitting data, and transmitting data efficiently are different concepts. Efficient transmission is increasingly becoming a focal point as it leads to overall better network utilization and minimal impact on the battery.

 Battery Performance – Most Wi-Fi devices are battery operated. Devices that don’t optimize transmission algorithms will find that they spend too many cycles in transmissions and re-transmissions. Optimizing transmission is key to better battery performance.

 Radar Compliance – Regulatory bodies impose restrictions around usage of certain frequencies at certain times. This functionality is also referred to as Dynamic Frequency

(13)

Selection. Devices exhibit many issues when it comes to DFS, because it involves dynamically switching from one channel to another.

 Antenna design – Antenna design is a complex subject: if not done right, device performance can vary vastly, depending on its orientation or its interaction with the environment.

(14)

Wi-Fi Device Test Challenges

Tools and equipment

Prior to Ixia’s foray into device testing, Wi-Fi device test labs had to use several test tools from various vendors.

 Real APs – Access Points that DUT can connect to

 Traffic generation tools – Tools to generate different types of traffic.

 Programmable attenuator – Controls attenuation to set different path loss.

 Channel emulator – Emulates different channel conditions like home, small/large office, outdoor etc. The channel conditions vary in each of these deployment conditions.

 Packet capture/sniffing and protocol analyzer – Captures wireless traffic and helps decode and analyze it.

 Signal and Spectrum analyzer – Tools to analyze the Radio Frequency transmitted. This hodge-podge solution had several issues:

 Complexity – imagine working with 6 vendors!

 Usability – users have to train themselves in all of these different products with different interfaces

 Predictability – accounting for failure is 6 times harder when dealing with different pieces

 Time – the time-cost to put together and maintain these different pieces

 Cost – and finally $$$

(15)

Testing Methodology and Focus

Unlike cellular technologies, in Wi-Fi there are no test-focused specifications for Wi-Fi devices. This whole space is relatively new compared to the cellular world, where there have been several generations of technologies released, and the market has evolved to standards-based test approaches.

When testing Wi-Fi Devices, three main technology areas should be assessed: PHY Layer

This is responsible for converting information into RF signals, and for transmitting them between the source and the destination. To ensure successful communication, the transmitter and receiver of the device need to perform well and also conform to all necessary specifications. On the functional side, the transmitter needs to make sure that the RF energy transmitted is confined to the allowed spectrum mask within the frequency band of operation. It also needs to ramp up and ramp down power within spec to ensure any given transmission does not interfere with the next one. Because ensuring a proper PHY layer proves critical for ensuring end-user quality of experience on a Wi-Fi device, RF testing is an essential step. On the performance side, the transmitter needs to have proper transmitter modulation accuracy at different modulation rates, while the receiver needs to have a low Error Vector Magnitude (EVM) when receiving at various data rates and power levels.

MAC Layer:

Unlike other wireless access technologies such as LTE, where the base station makes most of the decisions for the device, the Wi-Fi protocol requires the device to make lots of decisions. The device must decide:

 When to transmit

 How to contend and acquire the channel

 How to roam between access points

 How and when to rate adapt

 How and when to use power-save mechanisms

A typical Wi-Fi device is expected to be a lot smarter, and needs to implement several complex algorithms at the MAC layer. Hence the MAC layer on a device needs to be tested thoroughly for both functionality and performance.

On the functional side, the device needs to be tested to make sure it can roam, rate adapt, connect to the AP using proper security mechanisms; and that it can only connect to APs with matching credentials. On the performance side, it’s important to test that the device can optimize its

(16)

Application Layer

This is what the user sees and interacts with. Here, we need to look at several aspects of application performance, including issues such as seamless LTE to Wi-Fi handover, delay-sensitive Unified Communications (UC) applications, high-definition video streaming over Wi-Fi, and the like. One very important point to note is that bad RF performance or bad MAC layer performance will result in a bad user experience with an application. It’s important to thoroughly test and harden a device at each one of these layers in isolation, and then test the system as a whole.

Ixia recommends that testing is best conducted using a staged approach. It is important to baseline the performance of the Device under Test (DUT) under ideal conditions, and to find and fix issues. Testing should then progress by introducing one variable at a time, moving from the most deterministic to the most realistic test conditions.

(17)

Design/Development/QA Testing – Stage 1

During Stage 1 of testing – Dev Test and QA – configurability, repeatability, stress, and automation are very important to achieving maximum test coverage in the minimum amount of time. This stage of testing is best addressed by using a piece of test equipment that can simulate a Wi-Fi access point.

The key tests that need to be run in Stage 1 include measuring radio performance, validating device connectivity, measuring raw throughput, and ensuring protocol conformance. Stage 1 also includes other MAC and PHY protocol-related aspects such as roaming, rate adaptation, power-save protocols, and security. Extensive test coverage is critical during this stage, and covering numerous test cases in a small amount of time is essential for an on-time release of a high-quality product in a highly competitive market.

The WaveDevice Golden AP solution combines hardware and software. The hardware includes an 802.11a/b/g/ac Golden AP emulator, full line-rate traffic generator, channel and distance emulator, line-rate real-time protocol sniffer, and line-rate, real-time signal generator and analyzer.

The test hardware connects to an RF enclosure using RF cables; the device under test is placed inside the RF enclosure. The DUT runs simple endpoint software – called the “WaveAgent” - that sits at the transport layer on the device. The WaveAgent receives commands from the Ixia WaveDevice hardware to send/receive different types of traffic at different rates, making precise performance measurements.

Interoperability Testing – Stage 2

Stage 2 of testing can begin during the post-production phase of the development life-cycle. The device is now fully developed and must be tested as a system prior to release. During this stage, it is important to subject the device to more realistic test conditions, including testing against real APs and testing over-the-air

(18)

In Stage 2, the client device under test has to be tested against the most common real APs to make sure that the device can work well with those APs in the field. The key tests in Stage 2 include TCP/UDP/VOIP Upstream/Downstream performance at different frame sizes and rates, and on different frequency channels with different settings on the AP and the client.

The client device under test is connected using RF cables to the AP through the IxVeriWave RF Management Unit. Both the AP and the Client are placed in separate RF enclosures to create an isolated, fully controllable, and repeatable test environment.

The testbed also includes the IxVeriWave WT-90/92 that houses two 802.11ac wireless cards that can capture all the traffic between the AP and the Client on the wireless interface and perform expert analysis to isolate and identify PHY/MAC-level issues with the AP and the Client device. The WT-90/92 chassis also includes an Ethernet card that is connected to the Ethernet interface of the AP and acts as one of the endpoints for traffic. The second endpoint is the WaveAgent software installed on the device under test. The WaveDevice software application can create TCP/UDP/VOIP traffic in both upstream and downstream directions and measure end-to-end Key Performance Indicators (KPIs).

The RF Management Unit can also programmatically simulate distance between the AP and the client and thus allows the software application to run performance over distance tests.

All the components of the testbed are nicely integrated and can be controlled from an easy-to-use GUI. The easy-to-user can run automated tests with various combinations of test settings and watch the results in real-time.

Field Testing – Stage 3

Stage 3 occurs when the device is deployed in a live network. Here, it is important to characterize the behavior of the device in real-world conditions, and to find and fix the small percentage of issues that fell through the cracks during lab testing. IxVeriWave users can also validate that the network into which the client device is being deployed has no major issues and can support the reliable operation of the device.

(19)

In Stage 3 of testing, the goal is to evaluate the device’s performance in the field. Stage 1 and Stage 2 testing provide an excellent platform for the test engineers to do everything possible in the lab and ship an excellent product. However, there will almost certainly be some issues that only show up in the field.

While testing Wi-Fi networks and devices in the field, the common misconception is: good RF coverage means happy users. A device could be getting excellent signal strength at all the locations on the floor but still have poor performance. There could be several reasons for this: maybe the device is getting excellent signal in general but is not connected to the best available AP; maybe the traffic load is not balanced across all the APs and thus resulting in low throughput on the device; maybe all the neighboring devices are communicating only on the 2.4GHz band even though there is ample free bandwidth available on the 5GHz band; or maybe it is not even a wireless problem, and there is some policy/role based misconfiguration on the wired network that is causing poor performance.

Ixia’s WaveDeploy test tool allows customers to run active site assessments from real devices using the same WaveAgent software used in Stage 1 and Stage 2 testing. These assessments measure the voice, video, and data performance of real devices at various locations on the deployment floor under very real usage conditions.

From extensive testing conducted in the field over several sites, it becomes clear that the traditional RSSI-based survey kind of testing is not sufficient. Coverage doesn’t mean capacity. It is very important to:

(20)

Only then can the users find the issues in the field that were not found in the Stage 1 and Stage 2 lab testing.

Device Test Methodology – Summary

 Start with a standard solution offering such as a reference design from a silicon manufacturer, or an embedded Wi-Fi module.

 Customize the generic solution to hit the performance, power, reliability, and physical requirements of your client device.

 Exhaustively test the behaviors of the device to ensure that the Wi-Fi customizations have not compromised functionality or performance. Identifying and addressing a handful of issues at this stage can save a lot of angry phone calls and customer support trips after deployment.

 Tune the design: test, then adjust, then test again. Continue until you reliably achieve the behavior you need.

 Once the device is rock-solid on its own, ensure that it interoperates with the network under all possible realistic environments while testing in the lab.



Finally, test the end user’s network while performing a major rollout to ensure that the user’s network can support a high-quality client transaction and that there are no site-specific issues.

(21)

Test Case 1: Throughput Benchmarking Test

Overview

Devices come in various shapes and sizes, but one function common among all of them is the ability to transmit, receive, and process traffic. The throughput benchmark test provides a concise sketch of the overall performance of the device. It indirectly validates the radio HW, RF signal chain, device driver, OS and application level performance, all in a single test.

In Wi-Fi, the overall throughput of a device can be impacted by several factors; some of them are listed below:

 Transmit power

 Ecosystem traffic

 Packet sizes

 Frame Aggregation

 Number of spatial streams – SISO/MIMO

 TCP/UDP

To be able to make sense of the results, it’s essential to keep the test variables to a minimum and under control during each trial. A test-bed like Ixia’s WaveDevice Golden AP gives users this control while enabling them to execute various tests.

Objective

Benchmark the maximum Downstream & Upstream throughput of a given device.

Setup

The setup here is made up of AP with traffic generation and control capabilities, a chamber to isolate the RF environment from external RF sources and DUT with an agent that can be controlled to generate data traffic.

(22)

Step-by-step Instructions

1. Launch IxVeriwave Golden AP WaveDevice. The workflow for configuring a test is outlined in the left frame of the GUI – System (chassis and port assignment), Access Points (AP configuration), Devices (Devices and Tests), Analysis (Results analysis). Please refer to the user guide to familiarize yourself with WaveDevice GUI

2. Enter the IP address of the chassis that hosts the Golden AP card. Click on connect when done.

3. Select the Golden AP card to be used for the simulating the AP and click Reserve at the bottom of the screen

4. Set the Channel information for the simulated AP. Note: channel selection can have an impact on AP configuration parameters like – AP Type, Bandwidth.

(23)

5. Switch to Access Point configuration to configure the simulated AP. You can leave most of the parameters in their default values.

 General Tab

o Port - <Make sure this matches, what you’ve reserved>

o SSID - Blackbook_Exercise

o Default Tx Power - <leave as default to start with, adjust based on RSSI feedback from device>

 Data and Beacon PHY rates tab

o This is the screen to set max. supported Data and Mgmt. PHY rates of the simulated AP. The DUT used for this exercise is an Apple iPhone 6, which is a SISO device that supports 256QAM. Therefore the AP will be configured to support MCS 8 and 9.

o Note: Disable Antennas 2, 3 and 4 in the configuration screen, if they are not connected to the device

(24)

Note: Setting a low beacon PHY rate will impact the maximum throughput the device under test can achieve. If you are sure your DUT supports higher management PHY rates, you can override this setting.

o Under VHT Rates, set

 NSS 1 - MCS 0-9

 NSS 2 - Not Supported

 Advanced

o Leave all remaining parameters in their default values.

o Aggregation parameters will be discussed in more detail in a separate test case.

6. Activate the AP and switch to Devices page

Clicking Activate AP will begin beacon transmission from AP. The Beacons will be transmitted at Tx Power level set in AP config screen (default value 15dBm).

(25)

8. Select the DUT from the devices that show up in the summary screen

9. Pick the “GDPT” option from Test Type drop down menu and configure it with the following

parameters to drive maximum throughput. The General Data Plane Test is designed to characterize the performance of a device by subjecting it to different types of traffic. GDPT Test is an ideal test to benchmark throughput performance of a device under test.

 Traffic Type: UDP and TCP

 Traffic Direction: Downstream and Upstream

 Frame Size: Set to MTU value 1518, for best results

 Frame rate: Set 100% of theoretical frame rate. Theoretical frame rate is derived based on several configuration parameters like channel bandwidth, max. data PHY rate, guard timer, aggregation settings. For an 80Mhz, AC SISO device with support for 256 QAM modulation (MCS 9), Short Guard timer – the theoretical frame rate works out to be 433.3Mbps

 Under options set the trial duration to 60 secs. Trial duration can be increased for long duration and stability tests.

(26)

10. Start the test by clicking on the “Start Test” icon in the ribbon on top.

Result Analysis

When test starts executing, monitoring stats will begin populating simultaneously. Monitoring stats are retrieved from IxVeriwave cards like (RFA/WBA) as well as the WaveAgent running on DUT. Retrieved stats are presented in WaveDevice GUI in 3 categories

 Flow Stats - stats measuring the active traffic flows

 Client Stats - stats pertaining to the each client or DUT

 Port Stats - stats measured at port which include all clients and APs using the specific IxVeriwave HW port

Stats are also stored away as CSV files in the hard disk. There is an analysis module called “View

Measurement” that can analyze results by co-relating stats and presenting results as bar or line graphs.

GDPT tests go by trials, for each trial the set of key configuration parameters are highlighted as the trial progresses. In this test there are 4 trials:

 UDP Downstream with packet size of 1518 bytes

 UDP Upstream with packet size of 1518 bytes

 TCP Downstream with packet size of 1518 bytes

 TCP Upstream with packet size of 1518 bytes

When each trial starts executing, the first set of stats to look at would be Offered Load and

(27)

Statistics Terminology Intended Load

Intended Load is the throughput intended to be generated. For tests like GDPT and RvR it’s computed as a percent of theoretical PHY rate, and it is configurable in the test. Theoretical frame rate

Theoretical frame rate is an estimate of maximum PHY throughput achievable, based on the current GoldenAP configuration. It is derived from several configuration parameters like channel bandwidth, maximum data PHY rate, guard interval, and few more. This value is an estimate, because the exact value cannot be determined without taking into account clients and their behavior.

Offered load

Downstream

In the downstream direction, this stat represents the L4 traffic load generated at the simulated distribution system. As the L4 traffic is generated in the same hardware as simulated golden AP, this stat also represents the throughput load at L2 of AP. Moreover, because of the way Wi-Fi MAC works, the system will limit the load value to traffic successfully ACK’ed by the L2 of DUT. So offered load is the L4 traffic load generated at the simulated AP that is also successfully ACK’ed at L2 of the DUT. The difference between offered load and Intended load can be attributed to system overhead (management frames, contention, retransmission etc) and receiver performance. Upstream

In the upstream direction, this stat represents the L4 traffic generated by Waveagent. The Waveagent relies on DUT’s Operating system TCP/IP stack to send the traffic out. If for whatever reason there is a bottleneck in the transmission path and the OS is unable to keep-up with traffic generated by WaveAgent, it will reflect in a lower offered load. Forwarding rate

Downstream

In the downstream direction, this stat represents the effective traffic that reaches the WaveAgent. If any traffic is dropped between L2 and L4 of the DUT, it will be reflected between the delta between Forwarding Rate and Offered Load.

Upstream

In the upstream direction, this stat represents the traffic received by the simulated AP (L2). Because the entire simulated AP sub system is in the same HW, there are no resulting packet losses between layers of the simulated AP. Therefore, this can also be interpreted as L4 traffic at the distribution system.

(28)

Trial 1 - UDP Downstream with packet size of 1518 bytes

To begin with, check the offered load and forwarding rate for the given trial. Also take note of medium utilization, Failed ACK frames (L2 Frame error in DL) and FCS errors (L2 Frame errors in UL).

Observation 1 - Throughput

(29)

The difference between offered load and forwarding rate is generally attributed to packet loss across the various interfaces of the stack. Note that L2 frame errors are all only in 1 direction. That’s because traffic is only in downstream direction, and even though the client transmits some management frames in upstream, it doesn’t result in any L2 errors.

Observation 2 – Packet loss

Packet loss and corresponding L2 Frame errors are low – 0.23% and 1.8% respectively. L2 Frame errors can be related to DUT not being able to keep up with received traffic rates, processing aggregated frames (AMPDUs), or MCS decoding error. Minimizing L2 frame error will improve overall throughput performance.

(30)

Observation 3 – Aggregation

In the downstream direction the simulated AP will aggregate packets-based negotiations with DUT. Aggregating MPDUs (mac layer protocol data unit) results in overall higher throughput, as there is lesser overhead in acquiring the medium for transmitting the same amount of data. As we can see from the result below, aggregation performance was quite good; all Tx Packets were aggregated in the max aggregation bucket, which is 32-64.

(31)

Trial 2 - UDP Upstream with packet size of 1518 bytes

To begin with, check the offered load and forwarding rate for the given trial. Also, take note of medium utilization, failed ACK frames (L2 frame error in DL), and FCS errors (L2 frame errors in UL).

Observation 1 - Throughput

(32)

The difference between offered load and forwarding rate is generally attributed to packet loss across the various interfaces of the stack. Note that L2 frame errors are all only in 1 direction, that’s because traffic is only in upstream direction and even though the Golden AP transmits some management frames in upstream, it doesn’t result in any L2 errors.

Packet loss and corresponding L2 frame errors are pretty low – 0.31% and 0% respectively. In the upstream direction, L2 frame errors occur due to transmitter issues or low signal quality. Packet loss is measured as the number of packets lost between transmitter and receiver, as L2 takes care of retransmission in case of errors, packet loss equals the packets lost between layers of the DUT (L4-L2).

(33)

Observation 3 – Aggregation

In the upstream direction the DUT has to aggregate packets. Aggregating MPDUs (mac layer protocol data unit) results in overall higher throughput, as there is lesser overhead in acquiring the medium for transmitting the same amount of data.

As we can see from the result below, aggregation performance was OK, but it was not maximized. Each AMPDU had roughly 32 MPDUs aggregated. Effective throughput could have been higher if more MPDUs were aggregated. In real-life though, DUTs have to balance high throughput with overheads associated with retransmission; from that angle, aggregating fewer MPDUs could be seen as a balancing act to optimize throughput.

(34)

Trial 3 - TCP Downstream with packet size of 1518 bytes

To begin with, check the offered load and forwarding rate for the given trial. Also take note of medium utilization, failed ACK frames (L2 frame error in DL) and FCS errors (L2 frame errors in UL).

Observation 1 - Throughput

Offered load and forwarding rate match - 216.365 Mbps. But the overall throughput is lower compared to UDP. This is expected for TCP. Note that the L2 errors exist in both directions (unlike UDP). This is because in TCP the ACKs coming back also take up resources.

(35)

Packet loss and corresponding L2 frame errors are low – 0.23% and 1.8% respectively. L2 frame errors can be related to DUT not being able to keep up with received traffic rates, processing aggregated frames (AMPDUs), or MCS decoding error. Minimizing L2 frame error will improve overall throughput performance.

(36)

Observation 3 – Aggregation

In the downstream direction the simulated AP will aggregate packets based negotiations with DUT. Aggregating MPDUs (mac layer protocol data unit) results in overall higher throughput, as there is lesser overhead in acquiring the medium for transmitting the same amount of data. As we can see from the result below, aggregation performance was quite good; all Tx Packets were aggregated in the max aggregation bucket, which is 32-64.

(37)

Trial 4 - TCP Upstream with packet size of 1518 bytes

This trial is not analyzed here, as the same techniques described in prior trials can be applied to analyze results from this trial.

Troubleshooting and Diagnostics

Symptom Diagnosis Comments

Low throughput (DL)

1. Check for L2 frame errors, high L2 frame errors implies any one of the following

 DUT unable to keep up with high traffic rates

 DUT framing issues with aggregated packets

 DUT unable to decode specific MCS at certain power levels (even though this is a L1 issue, its reported as a L2 frame error) 2. Check packet loss; packet loss can be result of

inter-layer processing errors

Assume testing is done in RF isolated chamber, so interference is not an issue Low throughput (UL)

2. DUT Transmitter Issue

3. Tx power level below Golden AP’s receiver sensitivity

4. Check packet loss; packet loss can be result of inter-layer processing errors

5. Check aggregation performance

Assume testing is done in RF isolated chamber, so interference is not an issue

Test Variables

 Different packet sizes

(38)

Conclusion

Based on analysis done above we conclude that maximum throughput for this device has been:

 UDP

o Downstream – 375 Mbps o Upstream – 325 Mbps

 TCP

o Downstream – 216 Mbps

By using the Golden AP and a conductive setup, we were able to get consistent results with regards to maximum throughput.

(39)

Test Case 2: Performance Characterization over Packet Sizes

Overview

Several wireless devices are designed for specific applications like medical devices, point of sale, VoIP phones etc. Traffic pattern in such devices is generally pretty deterministic, these devices need to be benchmarked with specific traffic profiles. General purpose devices (laptops, tablets etc.), on the other hand, deal with wide-ranging traffic patterns, and they too need to be benchmarked with different traffic profiles. There have been many instances where device performance deteriorates for specific packet sizes. It’s therefore important to characterize a device performance with different traffic characteristics.

Objective

Characterize device performance over different packet sizes.

Setup

1. Follow steps 1-8 defined in TC1 to configure a simulated AP.

2. Pick the “GDPT” option from Test Type drop down menu and configure it with the following

parameters. The General Data Plane Test is designed to characterize the performance of a device by subjecting it to different types of traffic. GDPT Test is an ideal test to benchmark performance of a device under test.

 Traffic Type: UDP and TCP

 Traffic Direction: downstream and upstream

 Frame Size: Set to MTU value 128, 256, 512, 1024, 1518, for a full sweep of different packet sizes

(40)

 Frame Rate: Set 100% of theoretical frame rate. Theoretical frame rate is derived based on several configuration parameters like channel bandwidth, max. data PHY rate, guard timer, aggregation settings. For an 80Mhz, AC SISO device with support for 256 QAM modulation (MCS 9), Short Guard timer – the theoretical frame rate works out to be 433.3Mbps

Under Options, set the trial duration to 60 secs. Trial duration can be increased for long duration and stability tests.

3. Start the test and begin monitoring statistics.

Result Analysis

Tests are broken into trials, which basically lock the configuration down during execution. For this testcase there are 10 trials. For sake of brevity, we will focus our analysis to only UDP traffic. Observation 1 – Offered Load

From the graph below, the first thing that becomes evident is that the offered load decreases with decreasing frame size. Offered load represents the traffic successfully transferred between the L2 (mac) AP and DUT. Refer to <> for more information on offered load.

The second point that becomes evident is that offered load is higher in downstream compared to upstream.

(41)

Observation 2 – Forwarding rate

Overall forwarding rate also tracks offered load pretty closely with a small percentage of packet loss, which we will analyze next.

(42)

Observation 3 – L2 Frame Errors and Packet loss

L2 frame errors – represents transmission issues between the L2 of AP and DUT. These errors are an overhead for the system as they increase retries and slow down the overall system. Note: L2 frame errors don’t directly result in packet loss, as the 802.11 stack takes care of re-transmissions.

In the run below, it’s evident that L2 errors increase drastically – almost 40X in downstream – as the frame size decreases.

Packet loss on the other hand is nominal. In the downstream direction Packet loss is highest for the biggest frame size (1518 bytes) and in the upstream direction Packet loss is pretty even across the board.

(43)

Observation 4: Aggregation

Aggregation stats represent the number of MAC layer PDUs aggregated by the system. It’s presented in both upstream and downstream directions. From the graph below, it becomes evident that in the upstream direction the DUT aggregates MPDUs differently for different packet sizes. In the downstream direction, the simulated AP always aggregates at a constant rate of 64 MPDUs. This could be one of the reasons why the L2 frame errors are high in downstream direction, especially for smaller frame sizes.

(44)

Troubleshooting and Diagnostics

Symptom Diagnosis Comments

Low throughput (DL)

 DUT unable to decode specific MCS at certain power levels (even though this is a L1 issue, its reported as a L2 frame error)

Assume testing is done in RF isolated chamber, so interference is not an issue Low throughput (UL)

 DUT Transmitter Issue

 Tx Power level below Golden AP’s receiver sensitivity

2. Check packet loss: packet loss can be a result of inter-layer processing errors

3. Check aggregation performance

(45)

Test Variables

 Frame rates

 More packet sizes

 Longer duration tests

Conclusion

Based on analysis done above, we conclude that for the given DUT:

 The bigger the frame size, the better the throughput performance

 Number of L2 frame errors is much lower for bigger frame sizes compared to small frame sizes. This translates to more overhead for lower frame sizes

 Bigger frame sizes have better forwarding rate to intended load, compared to smaller frame sizes

(46)

Test Case 3: Performance Characterization over Distance – Rate vs

Range

Overview

Wi-Fi technology is very commonly used in residential and enterprise scenarios to carry deal sensitive and high bandwidth real-time voice and video traffic. Since the devices connected to the Access Point are wireless, they can be located at different distances from the AP. It’s important to make sure that users get good quality of experience at different distances from the Access Point.

Objective

Characterize the performance of device over various distance profiles.

Setup

1. Follow steps 1 - 5 of Test Case 1.

 General Tab

o Port - Select Golden AP port reserved in ports page.

o Default Tx Power - Set this value to default and adjust if required, depending on feedback from device.

(47)

o This is the screen to set max. Supported Data and Mgmt. PHY rates of the simulated AP. The DUT used for this exercise is a Nexus 6, which is a MIMO 2x2 device that supports 256QAM. Therefore the AP will be configured to support Nss 1, Nss 2 with MCS 0 - 9.

o Disable Antennas 3 and 4 in the configuration screen, if they are not connected to the device

o Under OFDM, leave the default settings as is, as these are mandatory supported rates according to 802.11 specification. User has the privilege to change these settings.

 NSS 1 - MCS 0-9

 NSS 2 - MCS 0-9

(48)

 Advanced

Clicking Activate AP will begin beacon transmission from AP. 4. From DUT join the wireless network and start WaveAgent

(49)

Note: RSSI value shown for each device is reported by WaveAgent endpoint installed on DUT; initial path loss can be measured using Tx Power configured in Golden AP and RSSI value reported by WaveAgent: TxPower - RSSI

6. Pick the “Rate vs. Range Test” from “Test Type” drop down menu and configure test with the following parameters. The Rate vs. Range test is designed to characterize the device receiver performance at different distances from the Access Point while locking the data rate. This provides a fair idea about receiver performance for the given Modulation and Coding Scheme/data rate. This test can be configured with multiple data rates so user can benchmark the receiver performance for different Modulation and Coding Scheme/data rate at different distances in single click.

Golden AP uses Tx Power to simulate distance. The Tx power can be configured with minimum of 1 dB step difference.

(50)

Test Configuration

 Tx PHY. Rates: Nss 1 MCS 0, Nss 1 MCS 9, Nss2 MCS 3 and Nss2 MCS 6

Note: Ideally this test case should be run with all possible MCS values to fully sweep and identify any issues. However, to keep the content relevant for this Black Book, this test has been configured to target certain key MCS indexes.

 Tx Powers: 0 dBm to -40 dBm with step size -2 dBm

Note: Tx Power value can configured with minimum of 1 dB step size in the range of +15 dBm to -50 dBm

 Traffic Type: UDP

 Traffic Direction: Downstream

 Frame Size: Set the Frame Size value to 1518

 Frame Rates: Set 50% of theoretical frame rate. Frame rate has been set as 50% of theoretical, to target reasonable throughput values.

7. Start the test by clicking on the “Start Test” icon in the ribbon on top

Result Analysis

Path Loss in setup: It is always recommended to measure the path loss between AP and DUT before measuring the receiver performance. Initial estimated path loss can be measured using TxPower configured in Golden AP and RSSI value reported by WaveAgent. In this example the initial estimated path loss is 44 dB.

When test starts executing, monitoring stats will begin populating simultaneously. Monitoring stats are retrieved from IxVeriWave cards like (RFA/WBA3601) as well as the WaveAgent running on DUT. Retrieved stats are presented in WaveDevice GUI in 3 categories

 Flow Stats - stats measuring the active traffic flows

 Client Stats - stats pertaining to the each client or DUT

 Port Stats - stats measured at port which include all clients and APs using the specific IxVeriWave HW port

(51)

Stats are also stored away as CSV files for each trial in the host PC. There is an analysis module called “View Measurement” that can analyze results by co-relating stats and presenting results as bar or line graphs. Along with other UI graphs, trial results are also available in table format with a great deal of information.

RvR tests go by trials; for each trial, the set of key configuration parameters is highlighted as the trial progresses. In this test there are 84 trials. The number of trials will be derived from number of Tx PHY Rates times number of Tx Powers times frame rates.

For sake of brevity, we will pick some key trials for analysis. This should be sufficient to give an idea of how to go about analyzing results from the RvR test.

When each trial starts executing, the first set of stats to look at would be Offered Load and

Forwarding Rate, L2 Frame Errors and Medium Utilization.

The following chart shows the typical receiver sensitivity for 802.11ac modulation coding schemes with channel width20/40/80 and 160 MHz. Most receivers today exceed these performance metrics quite comfortably.

If the DUT does not have any specifications on its receiver sensitivity, this reference chart should provide some guidance on how to evaluate the device.

Observation 1 – Throughput Analysis

Note: Please refer to Test Case 1 - Statistics Terminology for more info on terms used here. Use View Measurement module to plot Offered Load and Forwarding Rate graphs over distance (represented as decreasing TxPower)

 First observation is that Offered Load maps close to Intended load. This implies the DUT is receiving the target throughput from L2 perspective.

(52)

 Second, DUT L2 performance of NSS-2-MCS-6 is slightly better NSS-1-MCS-9, this matches our expectation.

(53)

 Bingo – key issue noticed. Forwarding rate drops close to 0, for NSS-2-MCS-6. Offered load for the same modulation was 250Mbps; this implies that at L2 of DUT the success rate was close to 100%. However, between L2 and L4 of DUT, there was over 99% packet loss. These packets will not be retransmitted, as the test was using UDP packets. This translates to poor Quality of Experience for the user.

 At -16dBm Tx power, there is again a dip in Forwarding Rate, for NSS-2-MCS-6.

 NSS 1 MCS 9 and NSS 2 MCS3 also shows drops in forwarding rate at certain Tx powers, and again shoots up at adjacent lower power values.

(54)

 Finally, when comparing results with packet loss graph, it’s once again clear there is an issue for NSS-2-MCS-6 at certain power levels.

Observation 2 - L2 Frame Errors

L2 Frame errors are related to DUT not being able to keep up with received traffic rates, processing aggregated frames (AMPDUs), or MCS decoding error at certain Tx power.

 L2 errors are yo-yo’ing up and down for a range of power levels, after which they go all the way up. This behavior clearly demonstrates that DUT is not able to decode/acknowledge frames with specific Tx Power and Tx PHY rate. This behavior will result more Layer 2 retries, and will finally increase the cost of throughput by using additional air time. Since Wi-Fi is shared medium, this will not only impact the device performance but overall Wi-Fi network performance.

Observation 3 – Jitter

Increased Path Loss may result in high jitter values. The high variation in jitter across different power levels can cause degraded Voice/Video quality at different distances from the Access Point.

 Observed high jitter values with Nss 1 MCS 0, which is lowest Tx PHY Rate configured irrespective of Tx Powers

 Higher Tx PHY rates resulted in low jitter values, which shows that the device is receiving a contentious stream.

(55)

Troubleshooting and Diagnostics

Symptom Diagnosis Comments

Low Forwarding Rate

Check for L2 frame errors, high L2 frame errors implies any one of the following

 DUT unable to decode specific MCS at certain power levels (even though this is a L1 issue, its reported as a L2 frame error) Check Packet loss: packet loss can be a result of inter-layer processing errors

Test Variables

 Modulation and Coding Scheme/Tx PHY Rates

 Tx Power

(56)

Conclusion

Based on the analysis done above, we can conclude that the DUT clearly exhibited receiver issues at some modulation rates and power levels. This calls for deeper analysis to understand and remediate the issues.

Forwarding rate of device at different distances can be heavily influenced based on several factors:

 Modulation and Coding Scheme/Tx PHY Rate

 Path Loss between AP and Client

 Device receiver sensitivity

 Device ability to process the frames at receive data rate

There can be other factors as well, but assuming the test bed setup is isolated and left in optimum conditions, the above factors are key influencers.

(57)

Test Case 4: Cost of Throughput Analysis

Overview

Wi-Fi devices operate in a shared access medium, and they have to co-operate and co-exist with several other devices. Wi-Fi, like Ethernet, uses a distributed access scheme with only a small difference: that it uses a CSMA/CA scheme (carrier sensing multiple access / collision avoidance) to control access to the medium. However, collisions still occur. Moreover, there are several other challenges that MAC layer has to deal with that add to the overhead.

Cost of throughput is a lesser known but very important metric that represents the performance of the device in terms of overhead to the DUT, as well as to the overall system. In simple terms, cost of throughput is a reflection of the proportional use of medium – medium utilization – to transfer a given amount of data. There will always be some cost associated with throughput: the aim is to minimize it.

Being a shared medium, a higher cost of throughput implies a burden for all devices using the medium.

Objective

Validate that cost of throughput improves with increasing modulation rate.

Setup

2. Pick the “Simple” test for this exercise. It supports locking down Tx Power, MCS and Data rate for a given trial.

(58)

(59)

 Trial 3 – MCS=8, Tx Power=5, Data rate=50Mbps

(60)

4. Tx Power setting needs to correspond to optimum RSSI. In this case Tx Power = 5, results in RSSI of 30 dBm at the device.

Result Analysis

Observation 1 – MCS 3 @ 50Mbps

As the test execution starts, begin monitoring real-time stats.

First make sure the forwarding rate matches the target throughput

 Forwarding rate: 49.997 Mbps Next, check the Tx Flow Medium Utilization

 Tx Flow Medium Utilization %: 79.4

This represents the amount of resources taken up by of the DUT to transmit 50Mbps successfully. It includes retransmissions; as the retransmission rate is pretty low in this case, the medium utilization is mostly made up of cost of transmission of 50Mbps.

 Tx Failed ACK Frame rate (pps): 34 Observation 2 – MCS 5 @ 50Mbps

Same as above. Note down the Forwarding Rate, Medium Utilization and Tx Failed ACK Frame rate

(61)

 Tx Failed ACK Frame rate (pps): 58

Observation 3 – MCS 8 @ 50Mbps

Same as above, note down the Forwarding Rate, Medium Utilization and Tx Failed ACK Frame rate

 Forwarding rate: 49.992 Mbps

(62)

Observation 4 – MCS 9 @ 50Mbps

Same as above. Note down the Forwarding Rate, Medium Utilization and Tx Failed ACK Frame rate

 Forwarding rate: 49.997 Mbps

 Tx Flow Medium Utilization %: 73

(63)

Troubleshooting and Diagnostics

Symptom Diagnosis Comments

High Medium utilization

 Check L2 frame errors (or Tx failed ACK rate): if that’s high, medium utilization can be impacted

 Check the modulation rate: if that’s set low again medium utilization can be impacted

 Check target throughput: setting that high can result in high medium utilization

Test Variables

 Check results for different MCS rates

(64)

Conclusion

The expected result, in general, is that medium utilization goes down with higher order modulation rates, because high MCS results in more efficient encoding at the PHY layer, resulting in better overall performance.

Looking closer at the results, it becomes clear that medium utilization starts trending down from MCS3 to MCS 8. However, for MCS 9 it shoots back up. This is because of the amount of L2 errors and associated retransmission. It’s indeed quite high for MCS 9.

 Medium Utilization for MCS 3 = 79.4

 Medium Utilization for MCS 9 = 73

With this data we can conclude that MCS 9 performance for the given Tx Power is not on par with expectation. The overall cost of throughput is much higher for MCS 9.

(65)

Test Case 5: Roaming Validation

Overview

Roaming is the ability of a device to move from one AP to another while keeping an active network session. Roaming is now very common in most commercial deployments; users typically move within the campus and expect their connection to stay up.

When it comes to roaming function, the network only plays a small part (this is changing somewhat with 802.11r and k). The device makes the key decisions on when to roam and where to roam. The complexity arises from the fact that the active connection has to be maintained and serviced in parallel to completing the roaming process. The 802.11 standards don’t address roaming, so every vendor has their own implementation. This makes roaming function particularly susceptible to failures and interoperability issues.

Testing roaming should be at the top of a device vendor’s test plan, as it can impact the user’s experience quite significantly.

Objective

Validate the roaming success rate of a device roaming between APs in channels 48 and 44.

(66)

1. Launch IxVeriwave Golden AP WaveDevice. The workflow for configuring a test is outlined in the left frame of the GUI – System (chassis and port assignment), Access Points (AP configuration), Devices (Devices and Tests), and Analysis (Results analysis). Please refer to the user guide to familiarize yourself with WaveDevice GUI

2. Enter the IP address of the chassis that hosts the Golden AP card. Click on connect when done

3. Select and reserve the Golden AP cards (IxAP) to be used for simulating the roaming test. For this example, we select 2 IxAP cards.

4. Set the Channel information for the assigned cards. Set them to channels 44 and 48, as example.

Note: channel selection can have an impact on AP configuration parameters, such as AP Type and Bandwidth.

(67)

 General Tab

o Count - 5 (for each port)

o Port - <Make sure this matches, what has been reserved>

o Default Tx Power - <leave as default to start with; adjust based on RSSI feedback from device>

o This is the screen to set max. supported Data and Mgmt. PHY rates of the simulated AP. The DUT used for this exercise is an Apple iPhone 6, which is a SISO device that supports 256QAM. Therefore, the AP will be configured to support MCS 8 and 9.

(68)

o Under OFDM, leave the default settings as is, as it offers ultimate compatibility. Also set the Beacon PHY Rate to 6 Mbps for maximum compatibility. The Beacon PHY rate sets the PHY rate for management frames transmitted by AP.

 NSS 1 - MCS 0-9

 Advanced

(69)

Toolbar

Clicking Activate AP will begin beacon transmission from AP. The Beacons will be transmitted at Tx Power level set in AP config screen (default value 15dBm).

7. From DUT join the wireless network and start WaveAgent

8. Pick the “Roaming” test for this exercise. Roaming test is designed to simulate various roaming scenarios like

 2-AP roam back and forth

 Multi-AP roam

 Intra-channel

 Inter-channel

 Mix of Intra and Inter-channel roam

 Roam in presence of neighbor APs

(70)

configured in the test. When the device roams, the test engine determines if the roam was successful and then calculates a roam delay.

Select the device and configure roaming with following settings

 Path: Inter-channel Roam. This will automatically create a roam path of APs with alternating channels.

 Repeat: 3, to get sufficient trials. This will cycle through the roam path 3 times.

 Return to source AP: check

 Continue on fail: check. If roam fails, i.e. device doesn’t end up in the target AP, this will continue the test and try to recover from the failure for future trials.

 Min Power: Set this to -50dB. This will be lowest Tx Power level setting applied when stepping down power level in source AP. Currently -50 is a hardware limitation as well. External attenuators can be used if more attenuation is needed.

 Max Power: Set this to +15dB. This will be the maximum Tx Power setting applied to a target AP when stepping up power. This is also a hardware limitation.

 Power Step: 1 dB.

 Every: 1000 ms. Power step and every, are bets interpreted together. Power program will step up/down power in source/target AP based on these values.

 Neighbor APs Enable: Uncheck. This is an advanced configuration that is designed to validate the roaming algorithms performance with regards to picking the right AP. This will be covered in another test.

 Estimated Attenuation: This setting is only exposed for devices that don’t report an RSSI. For such devices, enter the approximate pathloss, the test engine will use this to work out the estimate RSSI based on changing Tx Power.

(71)

Result Analysis

When the test starts, roaming dashboard will begin populating. Roaming dashboard tracks BSSID, Channel, TxPower, Estimated RSSI and it will automatically calculate Roam Delay and Trial Status. Real-time Monitoring stats are also available in parallel for deeper analysis.

Observation 1 – Roam Summary

The roaming dashboard presents a run-time view of key statistics while the roam simulation is in progress. TxPower and Est. RSSI are updated every time they change, and they pause right at the moment the device roams. Soon after the roam, the status column indicates “Passed” or “Failed” based on a successful roam. A successful roam is when a device switches to the configured target AP is able to resume the active traffic session.

(72)

Based on the table above, this device roamed successfully in each of the 6 trials. The roam-delay was somewhere between 30ms and 60ms for each trial.

Observation 2 – Forwarding Rate Vs Packet Loss

Forwarding rate is the effective throughput that a device is able to achieve, Packet loss tracks number of packets lost during the last sampled period. These stats give a good insight into the impact on quality of experience for the user. High packet loss results in high medium utilization or in other words increased overhead to keep up with the same amount of forwarding rate. Based on the graph below, this device had overall very little impact on forwarding rate during the roam. This could also be because the intended load was not set high.

Observation 3 – TxPHYDataRate

Client Device’s TxPHYDataRate represents the link rate that device picks for transmission. A number of factors determine this selection, chief among which is quality of previous transmissions. Looking at the graph below the TxPHYDataRate remains mostly steady around 38Mbps, but it also fluctuates between a lower PHY rate quite often. Before we analyze this further, it will help to understand how the test system simulates roaming. The GoldenAP simulates roaming by controlling transmit power, which basically affects traffic in the downstream direction. The roaming test, however, only works in the upstream direction. Therefore, it shouldn’t have much of an impact on client’s TxPHYDataRate. Moreover, the fluctuating rate doesn’t make sense. This is an issue that requires further investigation.

0 2 4 6 8 10 12 14 16 18 20 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 ForwardingRate RxFlow1PacketLossNumber

(73)

Troubleshooting and Diagnostics

Symptom Diagnosis Comments

Roaming Failure

 Check call flow, and see if the device probes all active channels

 Check if the device sends out association request messages to the target AP

Test Variables

 Number of roam trials

 Inter channel and Intra channel

 Roam frequency

 Traffic types

 Traffic direction

Conclusion

Overall, the device under test performed well, as all the roams were successful and roam delay was under 100ms, which is the benchmark for voice traffic. However, the client’s TxPHYDataRate fluctuated quite a bit, which requires further investigat

0 10 20 30 40 50 60 1 ₁₆ ₃₁ ₄₆ ₆₁ ₇₆ ₉₁ 106 121 136 151 166 181 196 211 226 241 256 271 286 301 316 331 346 361 376 391 406 421

TxDataPHYRate

(74)

Test Case 6: Security Test

Overview

Wireless security covers two key functions, Authentication and Encryption. Today, most wireless networks operate with some form of 802.1x based authentication scheme and 802.11i based encryption scheme. It is therefore important to measure the impact of these settings on the performance of the device. For instance, a device’s performance with or without encryption might vary quite a bit. The same goes for other roaming, power save, and other functionalities.

Objective

Measure the impact of AES-CCMP encryption on effective performance

Setup

2. Select a GDPT test and setup a simple configuration to benchmark throughput. UDP traffic with 1518 byte packet size for both Upstream and Downstream direction.

(75)

3. Start test, collect results.

4. Re-run the same test configuration with Security turned ON. In Access Point configuration, enable security and set a password

5. Start and collect results.

Result Analysis

To analyze this test case, we will compare the results of the same test configuration with security turned ON and OFF.

(76)

Observation 1 – Security setup (Authentication & Encryption)

When a device associates with the AP, it goes through a security handshake to complete authentication and setup encryption. This can be analyzed by capturing and viewing a trace in wireshark.

WaveDevice supports packet capture in real-time.

Note: this step is not necessary, but it’s always good practice to validate that the security

Black Book dition. Critical Testing for Wireless Client Devices. Edition April September 2015 ADVANCED MPLS

Black Book

Table of Contents

How to Read this Book

Troubleshooting and Diagnostics

Typographic Conventions

Critical Testing for Wireless

Introduction

Wi-Fi Device Issues

Wi-Fi Device Test Challenges

Testing Methodology and Focus

MAC Layer:

Application Layer

Interoperability Testing – Stage 2

Field Testing – Stage 3

Device Test Methodology – Summary

Test Case 1: Throughput Benchmarking Test

 TCP/UDP

Objective

 NSS 1 - MCS 0-9

Result Analysis

Statistics Terminology Intended Load

Downstream

Observation 1 - Throughput

Trial 2 - UDP Upstream with packet size of 1518 bytes

Observation 3 – Aggregation

Observation 1 - Throughput

Trial 4 - TCP Upstream with packet size of 1518 bytes

Troubleshooting and Diagnostics

Symptom Diagnosis Comments

Test Variables

Test Case 2: Performance Characterization over Packet Sizes

Setup

Result Analysis

Observation 2 – Forwarding rate

Observation 4: Aggregation

Troubleshooting and Diagnostics

Symptom Diagnosis Comments

Test Variables

Test Case 3: Performance Characterization over Distance – Rate vs

Setup

 NSS 1 - MCS 0-9

Result Analysis

Observation 1 – Throughput Analysis

Observation 2 - L2 Frame Errors

Troubleshooting and Diagnostics

Symptom Diagnosis Comments

Test Variables

Test Case 4: Cost of Throughput Analysis

Objective

Result Analysis

Observation 1 – MCS 3 @ 50Mbps

Observation 3 – MCS 8 @ 50Mbps

Troubleshooting and Diagnostics

Symptom Diagnosis Comments

Test Variables

Test Case 5: Roaming Validation

Objective

 NSS 1 - MCS 0-9

Toolbar

Result Analysis

Observation 1 – Roam Summary

Observation 3 – TxPHYDataRate

Troubleshooting and Diagnostics

Symptom Diagnosis Comments

Test Variables

TxDataPHYRate

Setup

Result Analysis

Observation 1 – Security setup (Authentication & Encryption)