------------------------------------------------------------------------

This project builds and integrates DPDK 16.07 and OVS-DPDK 2.6.90 from 
git master on Ubuntu 16.04 (LTS) / OpenStack Mitaka (LTS).

Oct 7, 2016 tikoehle

------------------------------------------------------------------------

Ubuntu 16.04 main apt repository provides a OVS-DPDK 2.5 package which
does not support jumbo frame size forwarding in the DPDK packet path.
This build integrates the OVS version with jumbo frame support on Ubuntu 
16.04 (LTS). Officially supported is this first with 16.10/Newton (EoS 
07/2017), 17.04/Ocata (EoS 01/2018), next LTS 18.04 (2018) but not with
16.04 (LTS) / Mitaka (LTS).

Also Canonical has added code to set the ownership and permissions of
vhost-user created sockets. This code does not exist in DPDK git master.
Also with OVS 2.6 there is a new way to configure OVS via the DPDK EAL.

The installation script in this repo shows how to build and integrate it
on Ubuntu 16.04. During the time of doing the work Canonicals OVS-DPDK 
source package was broken, otherwise I had updated the source package 
directly.

The following patches have been integrated.

------------------------------------------------------------------------
DPDK 16.07
------------------------------------------------------------------------
cd /usr/src
wget http://dpdk.org/browse/dpdk/snapshot/dpdk-16.07.zip
unzip dpdk-16.07.zip

1. Canonicals vhost-owner code

Patch the DPDK 16.07 source with this patch:
https://git.launchpad.net/~ubuntu-server/dpdk/tree/debian/patches/fix-vhost-user-socket-permission.patch?h=ubuntu-yakkety-dpdk16.07

2. Fix some socket issue with vhost-user sockets:

https://bugs.launchpad.net/ubuntu/+source/dpdk/+bug/1625542/+attachment/4748866/+files/dpdk_vhost_permission_tikoehle.patch


A patch for both issues is here [1]. It is based on Canonicals code [2].
I have added the code to my pilot OVS 2.6.90 + DPDK 16.07 deployment on 
Ubuntu 16.04. While testing I found issue #2 and created another patch. 
This code (one liner) may be different to what Canonical packages. For 
details see the corresponding bug-id.

The patch [1] implements ownership and permissions settings for vhost_user 
created sockets. The added DPDK EAL command line option is:

ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-extra="--vhost-owner libvirt-qemu:kvm --vhost-perm 0664"

To test it, first create a ovs vhost_user port. If everything works you 
should see the sockets created with the vhost-owner and vhost-perm as 
configured above. 

$ sudo ovs-vsctl show
    Bridge br-int
        fail_mode: secure
        Port "vhua689d816-bb"
            tag: 67
            trunks: [1120, 1121, 1122, 1123, 1124, 1129, 1130, 1140]
            Interface "vhua689d816-bb"
                type: dpdkvhostuser
        Port "vhu6dbe5673-10"
            tag: 63
            trunks: [1120, 1121, 1122, 1123, 1124, 1129, 1130, 1140]
            Interface "vhu6dbe5673-10"
                type: dpdkvhostuser

And here are the sockets created with libvirt-qemu:kvm and flags 0664.

localadmin@compute10:/var/run/openvswitch$ ls -la
::::
srw-rw-r--  1 libvirt-qemu kvm     0 Nov  6 22:48 vhu6dbe5673-10
srw-rw-r--  1 libvirt-qemu kvm     0 Nov  6 22:52 vhua689d816-bb


References:
[1] vhost_user_permissions.patch
[2] http://dpdk.org/dev/patchwork/patch/12222/
[3] https://github.com/openvswitch/ovs/blob/branch-2.6/INSTALL.DPDK.md


------------------------------------------------------------------------
OVS-DPDK 2.6.90
------------------------------------------------------------------------

Reference:
https://github.com/openvswitch/ovs/blob/branch-2.6/INSTALL.DPDK.md


-------------------------------------------------------------------------
How to configure OVS 2.6 via DPDK EAL
-------------------------------------------------------------------------
Jumbo frame mbuf size requires to increase socket mem from "2048,2048" to 
"4096,4096". The args to configure coremask, socket-mem, vhost-owner, .. 
are now with OVS 2.6 directly passed to the DPDK EAL via the ovs db and
/etc/default/openvswitch-switch is no longer needed. In fact you have to 
disable DPDK_OPTS in /etc/default/openvswitch-switch.

Disable it:
/etc/default/openvswitch-switch
#DPDK_OPTS

Here is the new ovs 2.6 way of configuring these parameters, formerly called 
DPDK_OPTS:

ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem="4096,4096"
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0x300
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-extra="--vhost-owner libvirt-qemu:kvm --vhost-perm 0664"

ovs-vsctl get Open_vSwitch . other_config

There are other important parameters. The above is the minimal set to
get it running. 


-------------------------------------------------------------------------
Binding pNIC interface to DPDK 
-------------------------------------------------------------------------
As usual, for the external packet path with OVS-DPDK you need to bind the 
pNIC interface to the dpdk driver. Check with:

dpdk_nic_bind --status

Should have the interface in /etc/dpdk/interfaces bound to dpdk driver, 
otherwise vfio_pci module is not loaded or modprobe it manually (modprobe 
vfio_pci).

systemctl restart dpdk.service


-------------------------------------------------------------------------
Test
-------------------------------------------------------------------------
Add dpdk br and port and configure port for jumbo frame:

ovs-vsctl add-br br-data -- set bridge br-data datapath_type=netdev
ovs-vsctl add-port br-data dpdk0 -- set interface dpdk0 type=dpdk -- set Interface dpdk0 mtu_request=9000

Check the bridge is netdev enabled and not system:
ovs-vsctl list bridge br-data

Check this copy of vswitchd is really dpdk enabled:
ovs-vsctl get Open_vSwitch . iface_types

Restart nova / neutron services:
service nova-compute start
service neutron-openvswitch-agent start


-------------------------------------------------------------------------
Mitaka Jumbo Frame Size
-------------------------------------------------------------------------
When nova booting a vm you need to manually set the mtu size for the vhu 
port, for example:

ovs-vsctl set Interface vhu6ef65d50-d7 mtu_request=9000

I did not find the feature in Mitaka.

Check that all vhost-user sockets were created with the correct ownership:
ls -la /var/run/openvswitch


-------------------------------------------------------------------------
Canonicals dpdk and ovs source packages
-------------------------------------------------------------------------
At the time of writing this there were some issues to build the openvswitch 
package with a dpdk enabled vswitchd:

https://bugs.launchpad.net/ubuntu/+source/openvswitch-dpdk/+bug/1629271

The dpdk 16.07 source package builds just fine, creates all .deb packages 
and installs correctly without any dependency issues.

The current latest Ubuntu dpdk and ovs source packages are:

https://launchpad.net/ubuntu/+source/dpdk/16.07-0ubuntu5
https://launchpad.net/ubuntu/+source/openvswitch/2.6.0-0ubuntu2