ligang/vpp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
vpp/docs/usecases/container_test.md
John DeNisco 98c2348b5c docs: Fix the Use Cases Index 
Signed-off-by: John DeNisco <jdenisco@cisco.com>
Change-Id: If4d5ef8ab93c905493af074fb4c2096f1ab222d4
2020-04-09 14:23:35 -04:00

641 lines
18 KiB
Markdown
Raw Blame History

Container-based network simulation
==================================
The "make test" framework provides a good way to test individual
features. However, when testing several features at once - or
validating nontrivial configurations - it may prove difficult or
impossible to use the unit-test framework.
This note explains how to set up lxc/lxd, and a 5-container testbed to
test a split-tunnel nat + ikev2 + ipsec + ipv6 prefix-delegation
scenario.
OS / Distro test results
------------------------
This setup has been tested on an Ubuntu 18.04 LTS system. If you're
feeling adventurous, the same scenario also worked on a recent Ubuntu
20.04 "preview" daily build.
Other distros may work fine, or not at all.
Proxy Server
------------
If you need to use a proxy server e.g. from a lab system, you'll
probably need to set HTTP_PROXY, HTTPS_PROXY, http_proxy and
https_proxy in /etc/environment. Directly setting variables in the
environment doesn't work. The lxd snap _daemon_ needs the proxy settings,
not the user interface.
Something like so:
```
HTTP_PROXY=http://my.proxy.server:8080
HTTPS_PROXY=http://my.proxy.server:4333
http_proxy=http://my.proxy.server:8080
https_proxy=http://my.proxy.server:4333
```
Install and configure lxd
-------------------------
Install the lxd snap. The lxd snap is up to date, as opposed to the
results of "sudo apt-get install lxd".
```
# snap install lxd
# lxd init
```
"lxd init" asks several questions. With the exception of the storage
pool, take the defaults. To match the configs shown below, create a
storage pool named "vpp." Storage pools of type "zfs" and "files" have
been tested successfully.
zfs is more space-efficient. "lxc copy" is infinitely faster with
zfs. The path for the zfs storage pool is under /var. Do not replace
it with a symbolic link, unless you want to rebuild all of your
containers from scratch. Ask me how I know that.
Create three network segments
-----------------------------
Aka, linux bridges.
```
# lxc network create dora
# lxc network create internet
# lxc network create swan
```
We'll explain the test topology in a bit. Stay tuned.
Set up the default container profile
------------------------------------
Execute "lxc profile edit default", and install the following
configuration. Note that the "shared" directory should mount your vpp
workspaces. With that trick, you can edit code from any of the
containers, run vpp without installing it, etc.
```
config: {}
description: Default LXD profile
devices:
eth0:
name: eth0
network: lxdbr0
type: nic
eth1:
name: eth1
nictype: bridged
parent: internet
type: nic
eth2:
name: eth2
nictype: bridged
parent: dora
type: nic
eth3:
name: eth3
nictype: bridged
parent: swan
type: nic
root:
path: /
pool: vpp
type: disk
shared:
path: /scratch
source: /scratch
type: disk
name: default
```
Set up the network configurations
---------------------------------
Edit the fake "internet" backbone:
```
# lxc network edit internet
```
Install the ip addresses shown below, to avoid having to rebuild the vpp
and host configuration:
```
config:
ipv4.address: 10.26.68.1/24
ipv4.dhcp.ranges: 10.26.68.10-10.26.68.50
ipv4.nat: "true"
ipv6.address: none
ipv6.nat: "false"
description: ""
name: internet
type: bridge
used_by:
managed: true
status: Created
locations:
- none
```
Repeat the process with the "dora" and "swan" networks, using these
configurations:
### dora network configuration
```
config:
ipv4.address: 10.166.14.1/24
ipv4.dhcp.ranges: 10.166.14.10-10.166.14.50
ipv4.nat: "true"
ipv6.address: none
ipv6.nat: "false"
description: ""
name: dora
type: bridge
used_by:
managed: true
status: Created
locations:
- none
```
### swan network configuration
```
config:
ipv4.address: 10.219.188.1/24
ipv4.dhcp.ranges: 10.219.188.10-10.219.188.50
ipv4.nat: "true"
ipv6.address: none
ipv6.nat: "false"
description: ""
name: swan
type: bridge
used_by:
managed: true
status: Created
locations:
- none
```
Create a "master" container image
---------------------------------
The master container image should be set up so that you can
build vpp, ssh into the container, edit source code, run gdb, etc.
Make sure that e.g. public key auth ssh works.
```
# lxd launch ubuntu:18.04 dora
<spew>
# lxc exec dora bash
dora# cd /scratch/my-vpp-workspace
dora# apt-get install make ssh
dora# make install-dep
dora# exit
# lxc stop dora
```
Mark the container image privileged. If you forget this step, you'll
trip over a netlink error (-11) aka EAGAIN when you try to roll in the
vpp configurations.
```
# lxc config set dora security.privileged "true"
```
Duplicate the "master" container image
--------------------------------------
To avoid having to configure N containers, be sure that the master
container image is fully set up before you help it have children:
```
# lxc copy dora dorahost
# lxc copy dora swan
# lxc copy dora swanhost
# lxc copy dora dhcpserver # optional, to test ipv6 prefix delegation
```
Install handy script
--------------------
See below for a handly script which executes lxc commands across the
current set of running containers. I call it "lxc-foreach," feel free
to call the script Ishmael if you like.
Examples:
```
$ lxc-foreach start
<issues "lxc start" for each container in the list>
```
After a few seconds, use this one to open an ssh connection to each
container. The ssh command parses the output of "lxc info," which
displays container ip addresses.
```
$ lxc-foreach ssh
```
Here's the script:
```
#!/bin/bash
set -u
export containers="dora dorahost swan swanhost dhcpserver"
if [ x$1 = "x" ] ; then
echo missing command
exit 1
fi
if [ $1 = "ssh" ] ; then
for c in $containers
do
inet=`lxc info $c | grep eth0 | grep -v inet6 | head -1 | cut -f 3`
if [ x$inet = "x" ] ; then
echo $c not started
else
gnome-terminal --command "/usr/bin/ssh $inet"
fi
done
exit 0
fi
for c in $containers
do
echo lxc $1 $c
lxc $1 $c
done
exit 0
```
Test topology
-------------
Finally, we're ready to describe a test topology. First, a picture:
```
===+======== management lan/bridge lxdbr0 (dhcp) ===========+===
| | |
| | |
| | |
v | v
eth0 | eth0
+------+ eth1 eth1 +------+
| dora | 10.26.88.100 <= internet bridge => 10.26.88.101 | swan |
+------+ +------+
eth2 / bvi0 10.166.14.2 | 10.219.188.2 eth3 / bvi0
| | |
| ("dora" bridge) | ("swan" bridge) |
| | |
v | v
eth2 10.166.14.3 | eth3 10.219.188.3
+----------+ | +----------+
| dorahost | | | dorahost |
+----------+ | +----------+
eth0 (management lan) <========+========> eth0 (management lan)
```
### Test topology discussion
This topology is suitable for testing almost any tunnel encap/decap
scenario. The two containers "dorahost" and "swanhost" are end-stations
connected to two vpp instances running on "dora" and "swan".
We leverage the Linux end-station network stacks to generate traffic
of all sorts.
The so-called "internet" bridge models the public internet. The "dora" and
"swan" bridges connect vpp instances to local hosts
End station configs
-------------------
The end-station Linux configurations set up the eth2 and eth3 ip
addresses shown above, and add tunnel routes to the opposite
end-station networks.
### dorahost configuration
```
ifconfig eth2 10.166.14.3/24 up
route add -net 10.219.188.0/24 gw 10.166.14.2
```
### swanhost configuration
```
sudo ifconfig eth3 10.219.188.3/24 up
sudo route add -net 10.166.14.0/24 gw 10.219.188.2
```
VPP configs
-----------
Split nat44 / ikev2 + ipsec tunneling, with ipv6 prefix delegation in
the "dora" config.
### dora configuration
```
set term pag off
comment { "internet" }
create host-interface name eth1
set int ip address host-eth1 10.26.68.100/24
set int ip6 table host-eth1 0
set int state host-eth1 up
comment { default route via swan }
ip route add 0.0.0.0/0 via 10.26.68.101
comment { "dora-private-net" }
create host-interface name eth2
bvi create instance 0
set int l2 bridge bvi0 1 bvi
set int ip address bvi0 10.166.14.2/24
set int state bvi0 up
set int l2 bridge host-eth2 1
set int state host-eth2 up
nat44 add interface address host-eth1
set interface nat44 in host-eth2 out host-eth1
nat44 add identity mapping external host-eth1 udp 500
nat44 add identity mapping external host-eth1 udp 4500
comment { nat44 untranslated subnet 10.219.188.0/24 }
comment { responder profile }
ikev2 profile add swan
ikev2 profile set swan udp-encap
ikev2 profile set swan auth rsa-sig cert-file /scratch/setups/doracert.pem
set ikev2 local key /scratch/setups/swankey.pem
ikev2 profile set swan id local fqdn swan.barachs.net
ikev2 profile set swan id remote fqdn broiler2.barachs.net
ikev2 profile set swan traffic-selector remote ip-range 10.219.188.0 - 10.219.188.255 port-range 0 - 65535 protocol 0
ikev2 profile set swan traffic-selector local ip-range 10.166.14.0 - 10.166.14.255 port-range 0 - 65535 protocol 0
create ipip tunnel src 10.26.68.100 dst 10.26.68.101
ikev2 profile set swan tunnel ipip0
comment { ipv6 prefix delegation }
ip6 nd address autoconfig host-eth1 default-route
dhcp6 client host-eth1
dhcp6 pd client host-eth1 prefix group hgw
set ip6 address bvi0 prefix group hgw ::2/56
ip6 nd address autoconfig bvi0 default-route
ip6 nd bvi0 ra-interval 5 3 ra-lifetime 180
set int mtu packet 1390 ipip0
set int unnum ipip0 use host-eth1
ip route add 10.219.188.0/24 via ipip0
```
### swan configuration
```
set term pag off
comment { "internet" }
create host-interface name eth1
comment { set dhcp client intfc host-eth1 hostname swan }
set int ip address host-eth1 10.26.68.101/24
set int state host-eth1 up
comment { default route via "internet gateway" }
comment { ip route add 0.0.0.0/0 via 10.26.68.1 }
comment { "swan-private-net" }
create host-interface name eth3
bvi create instance 0
set int l2 bridge bvi0 1 bvi
set int ip address bvi0 10.219.188.2/24
set int state bvi0 up
set int l2 bridge host-eth3 1
set int state host-eth3 up
nat44 add interface address host-eth1
set interface nat44 in bvi0 out host-eth1
nat44 add identity mapping external host-eth1 udp 500
nat44 add identity mapping external host-eth1 udp 4500
comment { nat44 untranslated subnet 10.166.14.0/24 }
comment { initiator profile }
ikev2 profile add dora
ikev2 profile set dora udp-encap
ikev2 profile set dora auth rsa-sig cert-file /scratch/setups/swancert.pem
set ikev2 local key /scratch/setups/dorakey.pem
ikev2 profile set dora id local fqdn broiler2.barachs.net
ikev2 profile set dora id remote fqdn swan.barachs.net
ikev2 profile set dora traffic-selector remote ip-range 10.166.14.0 - 10.166.14.255 port-range 0 - 65535 protocol 0
ikev2 profile set dora traffic-selector local ip-range 10.219.188.0 - 10.219.188.255 port-range 0 - 65535 protocol 0
ikev2 profile set dora responder host-eth1 10.26.68.100
ikev2 profile set dora ike-crypto-alg aes-cbc 256 ike-integ-alg sha1-96 ike-dh modp-2048
ikev2 profile set dora esp-crypto-alg aes-cbc 256 esp-integ-alg sha1-96 esp-dh ecp-256
ikev2 profile set dora sa-lifetime 3600 10 5 0
create ipip tunnel src 10.26.68.101 dst 10.26.68.100
ikev2 profile set dora tunnel ipip0
ikev2 initiate sa-init dora
set int mtu packet 1390 ipip0
set int unnum ipip0 use host-eth1
ip route add 10.166.14.0/24 via ipip0
```
IKEv2 certificate setup
-----------------------
In both of the vpp configurations, you'll see "/scratch/setups/xxx.pem"
mentioned. These certificates are used in the ikev2 key exchange.
Here's how to generate the certificates:
```
openssl req -x509 -nodes -newkey rsa:4096 -keyout dorakey.pem -out doracert.pem -days 3560
openssl x509 -text -noout -in doracert.pem
openssl req -x509 -nodes -newkey rsa:4096 -keyout swankey.pem -out swancert.pem -days 3560
openssl x509 -text -noout -in swancert.pem
```
Make sure that the "dora" and "swan" configurations point to the certificates.
DHCPv6 server setup
-------------------
If you need an ipv6 dhcp server to test ipv6 prefix delegation,
create the "dhcpserver" container as shown above.
Install the "isc-dhcp-server" Debian package:
```
sudo apt-get install isc-dhcp-server
```
### /etc/dhcp/dhcpd6.conf
Edit the dhcpv6 configuration and add an ipv6 subnet with prefix
delegation. For example:
```
subnet6 2001:db01:0:1::/64 {
range6 2001:db01:0:1::1 2001:db01:0:1::9;
prefix6 2001:db01:0:100:: 2001:db01:0:200::/56;
}
```
Add an ipv6 address on eth1, which is connected to the "internet"
bridge, and start the dhcp server. I use the following trivial bash
script, which runs the dhcp6 server in the foreground and produces
dhcp traffic spew:
```
#!/bin/bash
ifconfig eth1 inet6 add 2001:db01:0:1::10/64 || true
dhcpd -6 -d -cf /etc/dhcp/dhcpd6.conf
```
The "|| true" bit keeps going if eth1 already has the indicated ipv6
address.
Container / Host Interoperation
-------------------------------
Host / container interoperation is highly desirable. If the host and a
set of containers don't run the same distro _and distro version_, it's
reasonably likely that the glibc versions won't match. That, in turn,
makes vpp binaries built in one environment fail in the other.
Trying to install multiple versions of glibc - especially at the host
level - often ends very badly and is _not recommended_. It's not just
glibc, either. The dynamic loader ld-linux-xxx-so.2 is glibc version
specific.
Fortunately, it's reasonable easy to build lxd container images based on
specific Ubuntu or Debian versions.
### Create a custom root filesystem image
First, install the "debootstrap" tool:
```
sudo apt-get install debootstrap
```
Make a temp directory, and use debootstrap to populate it. In this
example, we create an Ubuntu 20.04 (focal fossa) base image:
```
# mkdir /tmp/myroot
# debootstrap focal /tmp/myroot http://archive.ubuntu.com/ubuntu
```
To tinker with the base image (if desired):
```
# chroot /tmp/myroot
<add packages, etc.>
# exit
```
Make a compressed tarball of the base image:
```
# tar zcf /tmp/rootfs.tar.gz -C /tmp/myroot .
```
Create a "metadata.yaml" file which describes the base image:
```
architecture: "x86_64"
# To get current date in Unix time, use `date +%s` command
creation_date: 1458040200
properties:
architecture: "x86_64"
description: "My custom Focal Fossa image"
os: "Ubuntu"
release: "focal"
```
Make a compressed tarball of metadata.yaml:
```
# tar zcf metadata.tar.gz metadata.yaml
```
Import the image into lxc / lxd:
```
$ lxc image import metadata.tar.gz rootfd.tar.gz --alias focal-base
```
### Create a container which uses the customized base image:
```
$ lxc launch focal-base focaltest
$ lxc exec focaltest bash
```
The next several steps should be executed in the container, in the
bash shell spun up by "lxc exec..."
### Configure container networking
In the container, create /etc/netplan/50-cloud-init.yaml:
```
network:
version: 2
ethernets:
eth0:
dhcp4: true
```
Use "cat > /etc/netplan/50-cloud-init.yaml", and cut-'n-paste if your
favorite text editor is AWOL.
Apply the configuration:
```
# netplan apply
```
At this point, eth0 should have an ip address, and you should see
a default route with "route -n".
### Configure apt
Again, in the container, set up /etc/apt/sources.list via cut-'n-paste
from a recently update "focal fossa" host. Something like so:
```
deb http://us.archive.ubuntu.com/ubuntu/ focal main restricted
deb http://us.archive.ubuntu.com/ubuntu/ focal-updates main restricted
deb http://us.archive.ubuntu.com/ubuntu/ focal universe
deb http://us.archive.ubuntu.com/ubuntu/ focal-updates universe
deb http://us.archive.ubuntu.com/ubuntu/ focal multiverse
deb http://us.archive.ubuntu.com/ubuntu/ focal-updates multiverse
deb http://us.archive.ubuntu.com/ubuntu/ focal-backports main restricted universe multiverse
deb http://security.ubuntu.com/ubuntu focal-security main restricted
deb http://security.ubuntu.com/ubuntu focal-security universe
deb http://security.ubuntu.com/ubuntu focal-security multiverse
```
"apt-get update" and "apt-install" should produce reasonable results.
Suggest "apt-get install make git".
At this point, you can use the "/scratch" sharepoint (or similar) to
execute "make install-dep install-ext-deps" to set up the container
with the vpp toolchain; proceed as desired.
Reference in New Issue View Git Blame Copy Permalink