6WINDGate 5.0: High Speed Packet Processing in Containers with DPDK
We know the advantages of containers compared to virtual machines: fast to start, demand less resources, run on the host in a simplified architecture, etc. Container orchestration becomes widely used in production and also inside R&D to run the testing framework. This goes for 6WINDGate-based networking apps as well, and the good news is that 6WINDGate 5.0 features support for containers!
Here is a view of the software architecture for a deployment using SR-IOV and two instances of 6WINDGate:
Being DPDK-based and NFV-oriented, there are some considerations in terms of setup and security.
U Can’t Touch This
The efficient design of containers is based on sharing the Host Linux kernel, which is a reason to put in place multiple security rules on the application that you plan to run inside the container:
- You are not going to taint the host with a kernel module
- You may need some kernel services available in Linux version >= X
- You have to explicitly define the security capabilities you need to run in unprivileged mode
With regards to the kernel, 6WINDGate 5.0 does not bring out-of-tree kernel modules. Instead it relies on user space APIs developed by the Linux community, with some 6WIND contributions as well.
The exception path in 6WINDGate makes use of vhost-net, which is supported in the kernel for a long time, and also of Linux TC + eBPF, usable in Linux 4.9, and available back in 2016.
For IPsec, packets are captured using nftable “rt xfrm” targeted to offload them to the Fast Path. This feature has been introduced in 4.20, two years ago.
As a result, on the shelf distros like CentOS 8 / Red Hat 8, or Ubuntu 18.04 HWE are good for our 6WINDGate containers.
With Your Permission
With containers running so close to the host environment, it’s true the host is exposed. The reflex in terms of security is to grant the very minimal level of privileges to a container, and open some freedom in a case by case basis so that we do not “fear the unknown.” 😊
Fair enough: we need NET_ADMIN, which is required to give the necessary power to configure networking, and which is the destiny of 6WINDGate applications.
And to run the exception path with eBPF, SYS_ADMIN is needed for now, looking forward to the new CAP_BPF to be available, which will be enough to grant the privilege to load and run the eBPF program.
To debug and profile with ‘perf’ inside the container you might find it useful to add SYS_TRACE.
Let’s Get It Started
Some work has to be done first on the host, to give memory and a network port to the container. This mainly comes from the DPDK brick used by 6WINDGate:
- Memory allocation from huge pages, one mount point per container
- VF setup and VFIO binding done
Then you describe the startup configuration for the container:
- List of directories to share with the container: vfio groups, vhost-net and tun devices, hugepages filesystem e.g –device /dev/vfio/vfio
- List of capabilities e.g. –cap-add=NET_ADMIN
- List of default network settings that will appear read-only inside the container: forwarding, IPv6 enable, RPF disable… e.g. — sysctl net.ipv4.ip_forward=1
The container is ready to go.
Come As You Are
In the container world, it is possible to run a Linux distribution that is different than the distribution running on the host, knowing there is of course a single kernel, the host kernel.
For example, it means you can run your 6WINDGate application in an Ubuntu environment inside the container, while the host is running Red Hat. Or, run it inside a CentOS environment, while the host is a custom distro based on buildroot for an embedded system.
This lets you run the application in a known environment, and can also ease the integration with third party software.
Buckle up: start the container, install 6WINDGate from source or the .deb / .rpm packages stored on your local repository, which is the same way you work in a bare metal environment. Here’s an example with an RPM-like delivery:
# docker run -it centos bash … # cat /etc/centos-release CentOS Linux release 8.1.1911 (Core) [root@356665602a82 /]# uname -a Linux 356665602a82 5.3.0-24-generic #26~18.04.2-Ubuntu SMP Tue Nov 26 12:34:22 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux … # dnf install -e 0 -d 1 -y 6windgate-cp-routing 6windgate-cp-security-ike 6windgate-dpdk 6windgate-dpdk-crypto-base 6windgate-dpdk-crypto-intel-ipsec-mb 6windgate-dpdk-crypto-intel-mb 6windgate-dpdk-pmd-mellanox-rdma-core 6windgate-fp 6windgate-fpn-sdk-dpdk 6windgate-linux-fp-sync 6windgate-linux-fp-sync-fptun-ebpf 6windgate-mgmt-base-libyang 6windgate-product-base 6windgate-rtrlib 6windgate-tools-common-libs-daemonctl 6windgate-tools-common-libs-libconsole 6windgate-tools-common-libs-libvrf … Complete!
Then configure the Fast Path with two VFs, assign some CPUs then start:
cat > /etc/fast-path.env << EOF
: ${FP_MASK:=0x1000}
: ${FP_PORTS:= 0000:84:10.0 0000:84:10.1}
: ${CORE_PORT_MAPPING:=c12=0/c12=1}
EOF
# fast-path.sh start
Starting Fast Path...
And voilà, 6WINDGate is running inside the container.
Finally, the support of containers in 6WINDGate brings a smart way of running multiple instances of 6WINDGate-based applications, in the distribution of your choice. It opens the door to more efficient designs, by splitting the function into specialized containers.
Is your architecture ready for containers? Please Contact Us today to learn more.
Jean-Mickaël Guérin is 6WIND’s CTO and avid music fan.
