OVS Base Setup for Tests

Build OVS as described in section F.1.4, Building OVS with DPDK-netdev, reboot the system, and check the kernel boot line for 1 GB hugepage, iosolcpu setting for target cores (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39 for 14 core CPU with hyperthreading enabled), and the IOMMU disabled:

# dmesg |grep command

[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.18.5-201.fc21.x86_64 root=UUID=cc17be46-707c-414a-8c16-11ecda5905be ro default_hugepagesz=1GB

hugepagesz=1GB hugepages=16 hugepagesz=2M hugepages=2048

isolcpus=1,2,3,4,5,6,7,8,9,10,11,12,28,29,30,31,32,33,34,35,36,37,38,39 rhgb quiet 1. Load the OVS kernel module and check for it being present:

# modprobe openvswitch # lsmod |grep open

openvswitch 75016 0

geneve 13166 1 openvswitch gre 13535 1 openvswitch libcrc32c 12603 1 openvswitch vxlan 36832 2 i40e,openvswitch 2. Load the kernel UIO driver and DPDK UIO driver:

# modprobe uio

# insmod /usr/src/dpdk/x86_64-ivshmem-linuxapp-gcc/kmod/igb_uio.ko 3. Load the cuse and fuse kernel modules:

# modprobe cuse # modprobe fuse

4. Remove the vhost-net kernel module and delete any associated data: # rmmod vhost-net

# rm –f /dev/vhost-net

5. Load the event link kernel module for the user side vhost:

# insmod /usr/src/dpdk/lib/librte_vhost/eventfd_link/eventfd_link.ko 6. Find the target Ethernet interfaces:

# lspci -nn |grep Ethernet

03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller 10- Gigabit X540-AT2 [8086:1528] (rev 01)

03:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller 10- Gigabit X540-AT2 [8086:1528] (rev 01)

06:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)

06:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)

06:00.2 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)

71

7. Bind Ethernet interfaces to the UIO interface (# config xxxx down):

# /usr/src/dpdk/tools/dpdk_nic_bind.py --bind=igb_uio 06:00.0 # /usr/src/dpdk/tools/dpdk_nic_bind.py --bind=igb_uio 06:00.1 # /usr/src/dpdk/tools/dpdk_nic_bind.py --status #

8. Mount the hugepage file systems for both 1 GB and 2 MB hugepages: # mount -t hugetlbfs nodev /dev/hugepages

# mkdir /dev/hugepages_2mb

# mount -t hugetlbfs nodev /dev/hugepages_2mb -o pagesize=2MB

# cat /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages # echo 2048 > /sys/devices/system/node/node0/hugepages/hugepages-

2048kB/nr_hugepages

9. Terminate OVS and delete the previous OVS database (does not need to be done, if valid DB without a new OVS build):

# pkill -9 ovs

# rm -rf /usr/local/var/run/openvswitch/ # rm -rf /usr/local/etc/openvswitch/ # rm -f /tmp/conf.db

10. Set up directories for the new OVS database (does not need to be done, if valid DB without a new OVS build):

# mkdir -p /usr/local/etc/openvswitch # mkdir -p /usr/local/var/run/openvswitch

11. Initialize the new OVS database (does not need to be done, if valid DB without a new OVS build):

# cd /usr/src/ovs

# ./ovsdb/ovsdb-tool create /usr/local/etc/openvswitch/conf.db \ ./vswitchd/vswitch.ovsschema

12. Start the OVS database server: # cd /usr/src/ovs

# ./ovsdb/ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock \ --remote=db:Open_vSwitch,Open_vSwitch,manager_options \

--private-key=db:Open_vSwitch,SSL,private_\

key --certificate=db:Open_vSwitch,SSL,certificate \

--bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --pidfile --detach 13. Initialize the OVS database (only after creating a new DB):

# cd /usr/src/ovs

# ./utilities/ovs-vsctl --no-wait init

14. Start OVS with DPDK portion using 2 GB of node 0 memory: # cd /usr/src/ovs

# /vswitchd/ovs-vswitchd --dpdk -c 0x4 -n 4 --socket-mem 2048,0 -- unix:/usr/local/var/run/openvswitch/db.sock --pidfile --detach 15. Set the default flow timeout to 30 seconds:

# cd /usr/src/ovs/utilities

72

16. Set the default OVS PMD thread usage: # cd /usr/src/ovs/utilities

# ./ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=2

17. Create the OVS DPDK bridge and add the two physical NICs. This needs to be done sequentially one line at a time (or have a delay between each in a script) so that the port order is assigned sequentially. If previous ports were created in a different order or other ports were created, the port IDs of the ports might not be correct for the test. Either the flow set port IDs need to be changed in the OVS flow commands or the OVS tasks need to be stopped and the previous OVS database deleted, OVS tasks restarted, and new database created.

# cd /usr/src/ovs

# ./utilities/ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev # ./utilities/ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk # ./utilities/ovs-vsctl add-port br0 dpdk1 -- set Interface dpdk1 type=dpdk # ./utilities/ovs-vsctl show

18. Check OVS task affinitization (use your system PIDs): # ps -e |grep ovs

15544 ? 00:00:00 ovsdb-server 15901 ? 01:38:16 ovs-vswitchd # top -p 15901 -H -d1

Threads: 33 total, 1 running, 32 sleeping, 0 stopped, 0 zombie

%Cpu(s): 3.6 us, 0.0 sy, 0.0 ni, 96.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 65841244 total, 47628324 free, 17427804 used, 785116 buff/cache KiB Swap: 8388604 total, 8388604 free, 0 used. 48081896 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 15966 root 20 0 12.818g 7232 4644 R 99.9 0.0 21:06.58 pmd59 15901 root 20 0 12.818g 7232 4644 S 3.0 0.0 0:08.85 ovs- vswitchd 15902 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:12.28 dpdk_watchdog3 15903 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:00.02 cuse_thread2 15904 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:00.00 urcu1 15938 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:00.00 handler61 15939 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:00.00 handler60 15940 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:00.00 handler58 15941 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:00.00 handler57 15942 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:00.00 handler56 15943 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:00.00 handler55 15944 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:00.00 handler33 15945 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:00.00 handler34 15946 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:00.00 handler35 15947 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:00.00 handler36 15948 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:00.00 handler37 15949 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:00.00 handler38 15950 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:00.00 handler39 15951 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:00.00 handler40 15952 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:00.00 handler41

73 15954 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:00.00 handler43 15955 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:00.00 handler44 15956 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:00.00 handler45 15957 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:00.00 handler46 15958 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:00.24 revalidator47 15959 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:00.09 revalidator48 15960 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:00.08 revalidator49 15961 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:00.07 revalidator50 15962 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:00.09 revalidator51 15963 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:00.09 revalidator52 15964 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:00.08 revalidator53 15965 root 20 0 12.818g 7232 4644 S 0.0 0.0 0:00.08 revalidator54 # taskset -p 15901

pid 15901's current affinity mask: 4 # taskset -p 15902

pid 15902's current affinity mask: 4 # taskset -p 15903

pid 15903's current affinity mask: 4 # taskset -p 15904

pid 15904's current affinity mask: 4 # taskset -p 15938

pid 15938's current affinity mask: 4 # taskset -p 15939

pid 15939's current affinity mask: 4 # taskset -p 15940

pid 15940's current affinity mask: 4 # taskset -p 15941

pid 15941's current affinity mask: 4 # taskset -p 15942

pid 15942's current affinity mask: 4 # taskset -p 15943

pid 15943's current affinity mask: 4 # taskset -p 15944

pid 15944's current affinity mask: 4 # taskset -p 15945

pid 15945's current affinity mask: 4 # taskset -p 15946

pid 15946's current affinity mask: 4 # taskset -p 15947

pid 15947's current affinity mask: 4 # taskset -p 15948

pid 15948's current affinity mask: 4 # taskset -p 15949

74

pid 15949's current affinity mask: 4 # taskset -p 15950

pid 15950's current affinity mask: 4 # taskset -p 15951

pid 15951's current affinity mask: 4 # taskset -p 15952

pid 15952's current affinity mask: 4 # taskset -p 15953

pid 15953's current affinity mask: 4 # taskset -p 15954

pid 15954's current affinity mask: 4 # taskset -p 15955

pid 15955's current affinity mask: 4 # taskset -p 15956

pid 15956's current affinity mask: 4 # taskset -p 15957

pid 15957's current affinity mask: 4 # taskset -p 15958

pid 15958's current affinity mask: 4 # taskset -p 15959

pid 15959's current affinity mask: 4 # taskset -p 15960

pid 15960's current affinity mask: 4 # taskset -p 15961

pid 15961's current affinity mask: 4 # taskset -p 15962

pid 15962's current affinity mask: 4 # taskset -p 15963

pid 15963's current affinity mask: 4 # taskset -p 15964

pid 15964's current affinity mask: 4 # taskset -p 15965

pid 15965's current affinity mask: 4 # taskset -p 15966

pid 15966's current affinity mask: 2

Note: OVS task affiliation will change in the future.