ligang/vpp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add contiv vpp

Browse Source 
Change-Id: I92227fc4968fc6a478beb7f38707b91e9f0635ec
Signed-off-by: Scott Keeler <skeeler@cisco.com>
This commit is contained in:
Scott Keeler
2018-10-01 14:50:57 -04:00
committed by Dave Barach
parent 2d24cd0272
commit 25c4d396ea
19 changed files with 2632 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/usecases/contiv/BUG_REPORTS.md
+333
View File
File diff suppressed because it is too large Load Diff
docs/usecases/contiv/CORE_FILES.md
+73
View File
@@ -0,0 +1,73 @@
# Capturing VPP core dumps
In order to debug a crash of VPP, it is required to provide a coredump file, which allows backtracing of the VPP issue. The following items are the requirements for capturing a coredump:
#### 1. Disable k8s Probes to Prevent k8s from Restarting the POD with a Crashed VPP
As described in [BUG_REPORTS.md](BUG_REPORTS.html#collecting-the-logs-in-case-of-crash-loop).
#### 2. Modify VPP Startup config file
In `/etc/vpp/contiv-vswitch.conf`, add the following lines into the `unix` section:
```
unix {
...
coredump-size unlimited
full-coredump
}
```
#### 3. Turn on Coredumps in the Vswitch Container
After re-deploying Contiv-VPP networking, enter bash shell in the vswitch
container (use actual name of the vswitch POD - `contiv-vswitch-7whk7` in this case):
```
kubectl exec -it contiv-vswitch-7whk7 -n kube-system -c contiv-vswitch bash
```
Enable coredumps:
```
mkdir -p /tmp/dumps
sysctl -w debug.exception-trace=1
sysctl -w kernel.core_pattern="/tmp/dumps/%e-%t"
ulimit -c unlimited
echo 2 > /proc/sys/fs/suid_dumpable
```
#### 4. Let VPP Crash
Now repeat the steps that lead to the VPP crash. You can also force VPP to crash at the point where it is
running (e.g., if it is stuck) by using the SIGQUIT signal:
```
kill -3 `pidof vpp`
```
#### 5. Locate and Inspect the Core File
The core file should appear in `/tmp/dumps` in the container:
```
cd /tmp/dumps
ls
vpp_main-1524124440
```
You can try to backtrace, after installing gdb:
```
apt-get update && apt-get install gdb
gdb vpp vpp_main-1524124440
(gdb) bt
```
#### 6. Copy the Core File Out of the Container
Finally, copy the core file out of the container. First, while still inside the container,
pack the core file into an archive:
```
cd /tmp/dumps
tar cvzf vppdump.tar.gz vpp_main-1524124440
```
Now, on the host, determine the docker ID of the container, and then copy the file out of the host:
```
docker ps | grep vswitch_contiv
d7aceb2e4876 c43a70ac3d01 "/usr/bin/supervisor…" 25 minutes ago Up 25 minutes k8s_contiv-vswitch_contiv-vswitch-zqzn6_kube-system_9923952f-43a6-11e8-be84-080027de08ea_0
docker cp d7aceb2e4876:/tmp/dumps/vppdump.tar.gz .
```
Now you are ready to file a bug in [jira.fd.io](https://jira.fd.io/) and attach the core file.
docs/usecases/contiv/CUSTOM_MGMT_NETWORK.md
+26
View File
@@ -0,0 +1,26 @@
### Setting Up a Custom Management Network on Multi-Homed Nodes
If the interface you use for Kubernetes management traffic (for example, the
IP address used for `kubeadm join`) is not the one that contains the default
route out of the host, then you need to specify the management node IP address in
the Kubelet config file. Add the following line to:
(`/etc/systemd/system/kubelet.service.d/10-kubeadm.conf`):
```
Environment="KUBELET_EXTRA_ARGS=--fail-swap-on=false --node-ip=<node-management-ip-address>"
```
#### Example
Consider a 2 node deployment where each node is connected to 2 networks -
`10.0.2.0/24` and `192.168.56.0/24`, and the default route on each node points
to the interface connected to the `10.0.2.0/24` subnet. We want to use subnet
`192.168.56.0/24` for Kubernetes management traffic. Assume the addresses of
nodes connected to `192.168.56.0/24` are `192.168.56.105` and `192.168.56.106`.
On the `192.168.56.105` node you add the following line to `10-kubeadm.conf`:
```
Environment="KUBELET_EXTRA_ARGS=--fail-swap-on=false --node-ip=192.168.56.105"
```
On the `192.168.56.106` node you add the following line to `10-kubeadm.conf`:
```
Environment="KUBELET_EXTRA_ARGS=--fail-swap-on=false --node-ip=192.168.56.106"
```
docs/usecases/contiv/K8s_Overview.md
+109
View File
@@ -0,0 +1,109 @@
# Contiv/VPP Kubernetes Network Plugin
## Overview
Kubernates is a container orchestration system that efficiently manages Docker containers. The Docker containers and container platforms provide many advantages over traditional virtualization. Container isolation is done on the kernel level, which eliminates the need for a guest virtual operating system, and therefore makes containers much more efficient, faster, and lightweight. The containers in Contiv/VPP are referred to as PODs.
Contiv/VPP is a Kubernetes network plugin that uses [FD.io VPP](https://fd.io/)
to provide network connectivity between PODs in a k8s cluster (k8s is an abbreviated reference for kubernates).
It deploys itself as a set of system PODs in the `kube-system` namespace,
some of them (`contiv-ksr`, `contiv-etcd`) on the master node, and some
of them (`contiv-cni`, `contiv-vswitch`, `contiv-stn`) on each node in the cluster.
Contiv/VPP is fully integrated with k8s via its components,
and it automatically reprograms itself upon each change in the cluster
via k8s API.
The main component of the [VPP](https://fd.io/technology/#vpp) solution, which
runs within the `contiv-vswitch` POD on each node in the cluster. The VPP solution also provides
POD-to-POD connectivity across the nodes in the cluster, as well as host-to-POD
and outside-to-POD connectivity. This solution also leverages
VPP's fast data processing that runs completely in userspace, and uses
[DPDK](https://dpdk.org/) for fast access to the network IO layer.
Kubernetes services and policies are also a part of the VPP configuration,
which means they are fully supported on VPP, without the need of forwarding
packets into the Linux network stack (Kube Proxy), which makes them very
effective and scalable.
## Architecture
Contiv/VPP consists of several components, each of them packed and shipped as
a Docker container. Two of them deploy on Kubernetes master node only:
- [Contiv KSR](#contiv-ksr)
- [Contiv ETCD](#contiv-etcd)
The rest of them deploy on all nodes within the k8s cluster (including the master node):
- [Contiv vSwitch](#contiv-vswitch)
- [Contiv CNI](#contiv-cni)
- [Contiv STN](#contiv-stn-daemon)
The following section briefly describes the individual Contiv components, which are displayed
as orange boxes on the picture below:
![Contiv/VPP Architecture](../../_images/contiv-arch.png)
### Contiv KSR
Contiv KSR (Kubernetes State Reflector) is an agent that subscribes to k8s control plane, watches k8s resources and
propagates all relevant cluster-related information into the Contiv ETCD data store.
Other Contiv components do not access the k8s API directly, they subscribe to
Contiv ETCD instead. For more information on KSR, read the
[KSR Readme](https://github.com/contiv/vpp/blob/master/cmd/contiv-ksr/README.md).
### Contiv ETCD
Contiv/VPP uses its own instance of the ETCD database for storage of k8s cluster-related data
reflected by KSR, which are then accessed by Contiv vSwitch Agents running on
individual nodes. Apart from the data reflected by KSR, ETCD also stores persisted VPP
configuration of individual vswitches (mainly used to restore the operation after restarts),
as well as some more internal metadata.
### Contiv vSwitch
vSwitch is the main networking component that provides the connectivity to PODs.
It deploys on each node in the cluster, and consists of two main components packed
into a single Docker container: VPP and Contiv VPP Agent.
**VPP** is the data plane software that provides the connectivity between PODs, host Linux
network stack, and data-plane NIC interface controlled by VPP:
- PODs are connected to VPP using TAP interfaces wired between VPP, and each POD network namespace.
- host network stack is connected to VPP using another TAP interface connected
to the main (default) network namespace.
- data-plane NIC is controlled directly by VPP using DPDK. Note, this means that
this interface is not visible to the host Linux network stack, and the node either needs another
management interface for k8s control plane communication, or
[STN (Steal The NIC)](SINGLE_NIC_SETUP.html) deployment must be applied.
**Contiv VPP Agent** is the control plane part of the vSwitch container. It is responsible
for configuring the VPP according to the information gained from ETCD, and requests
from Contiv STN. It is based on the [Ligato VPP Agent](https://github.com/ligato/vpp-agent) code with extensions that are related to k8s.
For communication with VPP, it uses VPP binary API messages sent via shared memory using
[GoVPP](https://wiki.fd.io/view/GoVPP).
For connection with Contiv STN, the agent acts as a GRPC server serving CNI requests
forwarded from the Contiv CNI.
### Contiv CNI
Contiv CNI (Container Network Interface) is a simple binary that implements the
[Container Network Interface](https://github.com/containernetworking/cni)
API and is being executed by Kubelet upon POD creation and deletion. The CNI binary
just packs the request into a GRPC request and forwards it to the Contiv VPP Agent
running on the same node, which then processes it (wires/unwires the container)
and replies with a response, which is then forwarded back to Kubelet.
### Contiv STN Daemon
This section discusses how the Contiv [STN (Steal The NIC)](SINGLE_NIC_SETUP.html) daemon operation works. As already mentioned, the default setup of Contiv/VPP requires two network interfaces
per node: one controlled by VPP for data facing PODs, and one controlled by the host
network stack for k8s control plane communication. In case that your k8s nodes
do not provide two network interfaces, Contiv/VPP can work in the single NIC setup,
when the interface will be "stolen" from the host network stack just before starting
the VPP and configured with the same IP address on VPP, as well as
on the host-VPP interconnect TAP interface, as it had in the host before it.
For more information on STN setup, read the [Single NIC Setup README](./SINGLE_NIC_SETUP.html)
docs/usecases/contiv/MANUAL_INSTALL.md
+472
View File
File diff suppressed because it is too large Load Diff
docs/usecases/contiv/MULTI_NIC_SETUP.md
+21
View File
@@ -0,0 +1,21 @@
### Setting Up a Node with Multiple NICs
* First, configure hardware interfaces in the VPP startup config, as
described [here](https://github.com/contiv/vpp/blob/master/docs/VPP_CONFIG.md#multi-nic-configuration).
* For each interface owned by Linux, you need to provide individual
configuration for each interface used by VPP in the Node Configuration
for thenode in the `contiv-vpp.yaml`. For example, if both `ens3` and
`ens4` are known to Linux, then put the following stanza into the node's
NodeConfig:
```
...
NodeConfig:
- NodeName: "ubuntu-1"
StealInterface: "ens3"
StealInterface: "ens4"
...
```
If only `ens3` is known to Linux, you only put a line for `ens3` into the
above NodeConfig.
docs/usecases/contiv/NETWORKING.md
+137
View File
@@ -0,0 +1,137 @@
# Contiv/VPP Network Operation
This document describes the network operation of the Contiv/VPP k8s network plugin. It
elaborates the operation and config options of the Contiv IPAM, as well as
details on how the VPP gets programmed by Contiv/VPP control plane.
The following picture shows 2-node k8s deployment of Contiv/VPP, with a VXLAN tunnel
established between the nodes to forward inter-node POD traffic. The IPAM options
are depicted on the Node 1, whereas the VPP programming is depicted on the Node 2.
![Contiv/VPP Architecture](/_images/contiv-networking.png "contiv-networking.png")
## Contiv/VPP IPAM (IP Address Management)
IPAM in Contiv/VPP is based on the concept of **Node ID**. The Node ID is a number
that uniquely identifies a node in the k8s cluster. The first node is assigned
the ID of 1, the second node 2, etc. If a node leaves the cluster, its
ID is released back to the pool and will be re-used by the next node.
The Node ID is used to calculate per-node IP subnets for PODs
and other internal subnets that need to be unique on each node. Apart from the Node ID,
the input for IPAM calculations is a set of config knobs, which can be specified
in the `IPAMConfig` section of the [Contiv/VPP deployment YAML](../../../k8s/contiv-vpp.yaml):
- **PodSubnetCIDR** (default `10.1.0.0/16`): each pod gets an IP address assigned
from this range. The size of this range (default `/16`) dictates upper limit of
POD count for the entire k8s cluster (default 65536 PODs).
- **PodNetworkPrefixLen** (default `24`): per-node dedicated podSubnet range.
From the allocatable range defined in `PodSubnetCIDR`, this value will dictate the
allocation for each node. With the default value (`24`) this indicates that each node
has a `/24` slice of the `PodSubnetCIDR`. The Node ID is used to address the node.
In case of `PodSubnetCIDR = 10.1.0.0/16`, `PodNetworkPrefixLen = 24` and `NodeID = 5`,
the resulting POD subnet for the node would be `10.1.5.0/24`.
- **PodIfIPCIDR** (default `10.2.1.0/24`): VPP-internal addresses put the VPP interfaces
facing towards the PODs into L3 mode. This IP range will be reused
on each node, thereby it is never externally addressable outside of the node itself.
The only requirement is that this subnet should not collide with any other IPAM subnet.
- **VPPHostSubnetCIDR** (default `172.30.0.0/16`): used for addressing
the interconnect of VPP with the Linux network stack, within the same node.
Since this subnet needs to be unique on each node, the Node ID is used to determine
the actual subnet used on the node with the combination of `VPPHostNetworkPrefixLen`, `PodSubnetCIDR` and `PodNetworkPrefixLen`.
- **VPPHostNetworkPrefixLen** (default `24`): used to calculate the subnet
for addressing the interconnect of VPP with the Linux network stack, within the same node.
With `VPPHostSubnetCIDR = 172.30.0.0/16`, `VPPHostNetworkPrefixLen = 24` and
`NodeID = 5` the resulting subnet for the node would be `172.30.5.0/24`.
- **NodeInterconnectCIDR** (default `192.168.16.0/24`): range for the addresses
assigned to the data plane interfaces managed by VPP. Unless DHCP is used
(`NodeInterconnectDHCP = True`), the Contiv/VPP control plane automatically assigns
an IP address from this range to the DPDK-managed ethernet interface bound to VPP
on each node. The actual IP address will be calculated from the Node ID (e.g., with
`NodeInterconnectCIDR = 192.168.16.0/24` and `NodeID = 5`, the resulting IP
address assigned to the ethernet interface on VPP will be `192.168.16.5` ).
- **NodeInterconnectDHCP** (default `False`): instead of assigning the IPs
for the data plane interfaces, which are managed by VPP from `NodeInterconnectCIDR` by the Contiv/VPP
control plane, DHCP assigns the IP addresses. The DHCP must be running in the network where the data
plane interface is connected, in case `NodeInterconnectDHCP = True`,
`NodeInterconnectCIDR` is ignored.
- **VxlanCIDR** (default `192.168.30.0/24`): in order to provide inter-node
POD to POD connectivity via any underlay network (not necessarily an L2 network),
Contiv/VPP sets up a VXLAN tunnel overlay between each of the 2 nodes within the cluster. Each node needs its unique IP address of the VXLAN BVI interface. This IP address
is automatically calculated from the Node ID, (e.g., with `VxlanCIDR = 192.168.30.0/24`
and `NodeID = 5`, the resulting IP address assigned to the VXLAN BVI interface will be `192.168.30.5`).
## VPP Programming
This section describes how the Contiv/VPP control plane programs VPP, based on the
events it receives from k8s. This section is not necessarily for understanding
basic Contiv/VPP operation, but is very useful for debugging purposes.
Contiv/VPP currently uses a single VRF to forward the traffic between PODs on a node,
PODs on different nodes, host network stack, and DPDK-managed dataplane interface. The forwarding
between each of them is purely L3-based, even for cases of communication
between 2 PODs within the same node.
#### DPDK-Managed Data Interface
In order to allow inter-node communication between PODs on different
nodes and between PODs and outside world, Contiv/VPP uses data-plane interfaces
bound to VPP using DPDK. Each node should have one "main" VPP interface,
which is unbound from the host network stack and bound to VPP.
The Contiv/VPP control plane automatically configures the interface either
via DHCP, or with a statically assigned address (see `NodeInterconnectCIDR` and
`NodeInterconnectDHCP` yaml settings).
#### PODs on the Same Node
PODs are connected to VPP using virtio-based TAP interfaces created by VPP,
with the POD-end of the interface placed into the POD container network namespace.
Each POD is assigned an IP address from the `PodSubnetCIDR`. The allocated IP
is configured with the prefix length `/32`. Additionally, a static route pointing
towards the VPP is configured in the POD network namespace.
The prefix length `/32` means that all IP traffic will be forwarded to the
default route - VPP. To get rid of unnecessary broadcasts between POD and VPP,
a static ARP entry is configured for the gateway IP in the POD namespace, as well
as for POD IP on VPP. Both ends of the TAP interface have a static (non-default)
MAC address applied.
#### PODs with hostNetwork=true
PODs with a `hostNetwork=true` attribute are not placed into a separate network namespace, they instead use the main host Linux network namespace; therefore, they are not directly connected to the VPP. They rely on the interconnection between the VPP and the host Linux network stack,
which is described in the next paragraph. Note, when these PODs access some service IP, their network communication will be NATed in Linux (by iptables rules programmed by kube-proxy)
as opposed to VPP, which is the case for the PODs connected to VPP directly.
#### Linux Host Network Stack
In order to interconnect the Linux host network stack with VPP (to allow access
to the cluster resources from the host itself, as well as for the PODs with `hostNetwork=true`),
VPP creates a TAP interface between VPP and the main network namespace. The TAP interface is configured with IP addresses from the `VPPHostSubnetCIDR` range, with `.1` in the latest octet on the VPP side, and `.2` on the host side. The name of the host interface is `vpp1`. The host has static routes pointing to VPP configured with:
- A route to the whole `PodSubnetCIDR` to route traffic targeting PODs towards VPP.
- A route to `ServiceCIDR` (default `10.96.0.0/12`), to route service IP targeted traffic that has not been translated by kube-proxy for some reason towards VPP.
- The host also has a static ARP entry configured for the IP of the VPP-end TAP interface, to get rid of unnecessary broadcasts between the main network namespace and VPP.
#### VXLANs to Other Nodes
In order to provide inter-node POD to POD connectivity via any underlay network
(not necessarily an L2 network), Contiv/VPP sets up a VXLAN tunnel overlay between
each 2 nodes within the cluster (full mesh).
All VXLAN tunnels are terminated in one bridge domain on each VPP. The bridge domain
has learning and flooding disabled, the l2fib of the bridge domain contains a static entry for each VXLAN tunnel. Each bridge domain has a BVI interface, which
interconnects the bridge domain with the main VRF (L3 forwarding). This interface needs
a unique IP address, which is assigned from the `VxlanCIDR` as describe above.
The main VRF contains several static routes that point to the BVI IP addresses of other nodes.
For each node, it is a route to PODSubnet and VppHostSubnet of the remote node, as well as a route
to the management IP address of the remote node. For each of these routes, the next hop IP is the
BVI interface IP of the remote node, which goes via the BVI interface of the local node.
The VXLAN tunnels and the static routes pointing to them are added/deleted on each VPP,
whenever a node is added/deleted in the k8s cluster.
#### More Info
Please refer to the [Packet Flow Dev Guide](../dev-guide/PACKET_FLOW.html) for more
detailed description of paths traversed by request and response packets
inside Contiv/VPP Kubernetes cluster under different situations.
docs/usecases/contiv/Prometheus.md
+159
View File
@@ -0,0 +1,159 @@
# Prometheus Statistics
Each contiv-agent exposes statistics in Prometheus format at port `9999` by default.
Exposed data is split into two groups:
- `/stats` provides statistics for VPP interfaces managed by contiv-agent
Prometheus data is a set of counters with labels. For each interface,
the following counters are exposed:
* *inPackets*
* *outPackets*
* *inBytes*
* *outBytes*
* *ipv4Packets*
* *ipv6Packets*
* *outErrorPackets*
* *dropPackets*
* *inMissPackets*
* *inNobufPackets*
* *puntPackets*
Labels let you add additional information to a counter. The *interfaceName* and *node*
labels are specified for all counters. If an interface is associated with a particular
pod, then the *podName* and *podNamespace* labels are also specified for its counters;
otherwise, a placeholder value (`--`) is used (for example, for node interconnect
interfaces).
- `/metrics` provides general go runtime statistics
To access Prometheus stats of a node you can use `curl localhost:9999/stats` from the node. The output of contiv-agent running at k8s master node looks similar to the following:
```
$ curl localhost:9999/stats
# HELP dropPackets Number of dropped packets for interface
# TYPE dropPackets gauge
dropPackets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
dropPackets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 52
dropPackets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 9
dropPackets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 12
dropPackets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
# HELP inBytes Number of received bytes for interface
# TYPE inBytes gauge
inBytes{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
inBytes{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 24716
inBytes{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 726
inBytes{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 6113
inBytes{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
# HELP inErrorPackets Number of received packets with error for interface
# TYPE inErrorPackets gauge
inErrorPackets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
inErrorPackets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 0
inErrorPackets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 0
inErrorPackets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 0
inErrorPackets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
# HELP inMissPackets Number of missed packets for interface
# TYPE inMissPackets gauge
inMissPackets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
inMissPackets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 0
inMissPackets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 0
inMissPackets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 0
inMissPackets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
# HELP inNobufPackets Number of received packets ??? for interface
# TYPE inNobufPackets gauge
inNobufPackets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
inNobufPackets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 0
inNobufPackets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 0
inNobufPackets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 0
inNobufPackets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
# HELP inPackets Number of received packets for interface
# TYPE inPackets gauge
inPackets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
inPackets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 97
inPackets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 9
inPackets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 60
inPackets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
# HELP ipv4Packets Number of ipv4 packets for interface
# TYPE ipv4Packets gauge
ipv4Packets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
ipv4Packets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 68
ipv4Packets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 0
ipv4Packets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 52
ipv4Packets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
# HELP ipv6Packets Number of ipv6 packets for interface
# TYPE ipv6Packets gauge
ipv6Packets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
ipv6Packets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 26
ipv6Packets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 9
ipv6Packets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 8
ipv6Packets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
# HELP outBytes Number of transmitted bytes for interface
# TYPE outBytes gauge
outBytes{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
outBytes{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 5203
outBytes{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 0
outBytes{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 17504
outBytes{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
# HELP outErrorPackets Number of transmitted packets with error for interface
# TYPE outErrorPackets gauge
outErrorPackets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
outErrorPackets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 0
outErrorPackets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 0
outErrorPackets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 0
outErrorPackets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
# HELP outPackets Number of transmitted packets for interface
# TYPE outPackets gauge
outPackets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
outPackets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 49
outPackets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 0
outPackets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 45
outPackets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
# HELP puntPackets Number of punt packets for interface
# TYPE puntPackets gauge
puntPackets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
puntPackets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 0
puntPackets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 0
puntPackets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 0
puntPackets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
```
In order to browse stats in web UI Prometheus, it must be started locally by following the information in
the [Prometheus Getting Started Guide](https://prometheus.io/docs/prometheus/latest/getting_started/).
If you start Prometheus on a node, the following sample config can be used:
```yaml
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'contiv_stats'
metrics_path: '/stats'
static_configs:
- targets: ['localhost:9999']
- job_name: 'contiv_agent'
# metrics_path defaults to '/metrics'
static_configs:
- targets: ['localhost:9999']
```
Once Prometheus is started with the specified config, you should be able access its web UI at
`localhost:9090`.
```
tester@dev:~/Downloads/prometheus-2.1.0.linux-amd64$ ./prometheus --config.file=config.yml
```
If security features are enabled for the HTTP endpoint, then the config must be adjusted:
```yaml
- job_name: 'contiv_secured'
scheme: https
basic_auth:
username: user
password: pass
metrics_path: /stats
tls_config:
insecure_skip_verify: true
# CA certificate to validate API server certificate with.
#[ ca_file: <filename> ]
static_configs:
- targets: ['localhost:9191']
```
docs/usecases/contiv/SECURITY.md
+104
View File
@@ -0,0 +1,104 @@
# Security
There are two types of security that are utilized in Contiv, and are discussed in this section: [HTTP](#http-security) and [ETCD](#etcd-security).
## HTTP Security
By default, the access to endpoints (liveness, readiness probe, prometheus stats, ...) served by Contiv-vswitch and
Contiv-ksr is open to anybody. Contiv-vswitch exposes endpoints using port `9999` and contiv-ksr uses `9191`.
To secure access to the endpoints, the SSL/TLS server certificate and basic auth (username password) can be configured.
In Contiv-VPP, this can be done using the Helm charts in [k8s/contiv-vpp folder](https://github.com/contiv/vpp/tree/master/k8s/contiv-vpp).
To generate server certificate the approach described in [ETCD security](#etcd-security) can be leveraged.
## ETCD Security
By default, the access to Contiv-VPP ETCD is open to anybody. ETCD gets deployed
on the master node, on port `12379`, and is exposed using the NodePort service
on port `32379`, on each node.
To secure access to ETCD, we recommend using the SSL/TLS certificates to authenticate
both the client and server side, and encrypt the communication. In Contiv-VPP, this can be done using the Helm charts in [k8s/contiv-vpp folder](https://github.com/contiv/vpp/tree/master/k8s/contiv-vpp).
The prerequisite for that is the generation of SSL certificates.
### Generate Self-Signed Certificates
In order to secure ETCD, we need to create our own certificate authority,
and then generate the private keys and certificates for both the ETCD server and ETCD clients.
This guide uses CloudFlare's [cfssl](https://github.com/cloudflare/cfssl) tools to do this job.
It follows the steps described in this [CoreOS guide](https://github.com/coreos/docs/blob/master/os/generate-self-signed-certificates.md).
Perform the following steps to generate private keys and certificates:
##### 1. Install cfssl
```
mkdir ~/bin
curl -s -L -o ~/bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
curl -s -L -o ~/bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
chmod +x ~/bin/{cfssl,cfssljson}
export PATH=$PATH:~/bin
```
##### 2. Initialize a Certificate Authority
```
echo '{"CN":"CA","key":{"algo":"rsa","size":2048}}' | cfssl gencert -initca - | cfssljson -bare ca -
echo '{"signing":{"default":{"expiry":"43800h","usages":["signing","key encipherment","server auth","client auth"]}}}' > ca-config.json
```
##### 3. Generate Server Key + Certificate
Replace the IP address `10.0.2.15` below with the IP address of your master node:
```
export ADDRESS=127.0.0.1,10.0.2.15
export NAME=server
echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $NAME
```
##### 4. Generate Client Key + Certificate
```
export ADDRESS=
export NAME=client
echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $NAME
```
The above commands produce the following files that will be needed in order to secure ETCD:
- `ca.pem`: certificate of the certificate authority
- `server.pem`: certificate of the ETCD server
- `server-key.pem`: private key of the ETCD server
- `client.pem`: certificate for the ETCD clients
- `client-key.pem`: private key for the ETCD clients
### Distribute Certificates and Generate Contiv-VPP Deployment Yaml
There are two options for distributing the certificates to all nodes in a k8s cluster.
You can either distribute the certificates
[manually](#distribute-certificates-manually), or embed the certificates into the deployment yaml file and
distribute them as [k8s secrets](https://kubernetes.io/docs/concepts/configuration/secret/).
##### Distribute Certificates Manually
In this case, you need to copy the `ca.pem`, `client.pem` and `client-key.pem` files
into a specific folder (`/var/contiv/etcd-secrets` by default) on each worker node.
On the master node, you also need to add the `server.pem` and `server-key.pem` into that location.
Then you can generate the Contiv-VPP deployment YAML as follows:
```
cd k8s
helm template --name my-release contiv-vpp --set etcd.secureTransport=True > contiv-vpp.yaml
```
Then you can go ahead and deploy Contiv-VPP using this yaml file.
##### Embed the certificates into deployment the yaml and use k8s secret to distribute them {: #Embed-certificates }
In this case, you need to copy all 5 generated files into the folder with helm definitions
(`k8s/contiv-vpp`) and generate the Contiv-VPP deployment YAML as follows:
```
cd k8s
helm template --name my-release contiv-vpp --set etcd.secureTransport=True --set etcd.secrets.mountFromHost=False > contiv-vpp.yaml
```
Then just deploy Contiv-VPP using this yaml file.
Please note that the path of the mount folder with certificates, as well as the certificate
file names can be customized using the config parameters of the Contiv-VPP chart,
as described in [this README](https://github.com/contiv/vpp/blob/master/k8s/contiv-vpp/README.md).
docs/usecases/contiv/SINGLE_NIC_SETUP.md
+111
View File
@@ -0,0 +1,111 @@
### Setting up a Node with a Single NIC
#### Installing the STN Daemon
The STN (Steal the NIC) daemon must be installed on every node in the cluster that has only
one NIC. The STN daemon installation(*) should be performed before deployment
of the Contiv-VPP plugin.
\* Docker daemon must be present when installing STN. Also, Docker must be configured to allow shared mount.
On CentOS, this may not be the case by default. You can enable it by following the instructions at
[https://docs.portworx.com/knowledgebase/shared-mount-propagation.html](https://docs.portworx.com/knowledgebase/shared-mount-propagation.html).
Run as root (not using sudo):
```
bash <(curl -s https://raw.githubusercontent.com/contiv/vpp/master/k8s/stn-install.sh)
```
The install script should output the following:
```
Installing Contiv STN daemon.
Starting contiv-stn Docker container:
550334308f85f05b2690f5cfb5dd945bd9c501ab9d074231f15c14d7098ef212
```
Check that the STN daemon is running:
```
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
550334308f85 contivvpp/stn "/stn" 33 seconds ago Up 33 seconds contiv-stn
```
Check that the STN daemon is operational:
```
docker logs contiv-stn
```
The expected logs would look like the following excerpt:
```
2018/02/23 10:08:34 Starting the STN GRPC server at port 50051
```
For more details, please read the Go documentation for [contiv-stn](https://github.com/contiv/vpp/blob/master/cmd/contiv-stn/doc.go)
and [contiv-init](https://github.com/contiv/vpp/blob/master/cmd/contiv-init/doc.go).
#### Creating a VPP Interface Configuration
Create the VPP configuration for the hardware interface as described
[here](https://github.com/contiv/vpp/blob/master/docs/VPP_CONFIG.md#single-nic-configuration).
#### Configuring STN in Contiv-VPP K8s Deployment Files
The STN feature is disabled by default. It needs to be enabled either globally,
or individually for every node in the cluster.
##### Global Configuration:
Global configuration is used in homogeneous environments where all nodes in
a given cluster have the same hardware configuration, for example only a single
Network Adapter. To enable the STN feature globally, put the `StealFirstNIC: True`
stanza into the [`contiv-vpp.yaml`][1] deployment file, for example:
```
data:
contiv.yaml: |-
TCPstackDisabled: true
...
StealFirstNIC: True
...
IPAMConfig:
```
Setting `StealFirstNIC` to `True` will tell the STN Daemon on every node in the
cluster to steal the first NIC from the kernel and assign it to VPP. Note that
the Network Adapters on different nodes do not need to be of the same type. You
still need to create the respective vswitch configurations on every node in the
cluster, as shown [above](#creating-a-vpp-interface-configuration).
##### Individual Configuration:
Individual configuration is used in heterogeneous environments where each node
in a given cluster may be configured differently. To enable the STN feature
for a specific node in the cluster, put the following stanza into its Node
Configuration in the [`contiv-vpp.yaml`][1] deployment file, for example:
```
...
NodeConfig:
- NodeName: "ubuntu-1"
StealInterface: "enp0s8"
- NodeName: "ubuntu-2"
StealInterface: "enp0s8"
...
```
Note that you still have to create the vswitch configuration on the node as
shown [here](#creating-a-vpp-interface-configuration).
#### Uninstalling the STN Daemon
Run as root (not using sudo):
```
bash <(curl -s https://raw.githubusercontent.com/contiv/vpp/master/k8s/stn-install.sh) --uninstall
```
The install script should output the following:
```
Uninstalling Contiv STN daemon.
Stopping contiv-stn Docker container:
contiv-stn
contiv-stn
contiv-stn
```
Make sure that the STN daemon has been uninstalled:
```
docker ps -q -f name=contiv-stn
```
No containers should be listed.
[1]: ../k8s/contiv-vpp.yaml
docs/usecases/contiv/VMWARE_FUSION_HOST.md
+52
View File
@@ -0,0 +1,52 @@
### Preparing a VmWare Fusion Host
The *vmxnet3 driver* is required on a GigE Network Adapter used by VPP. On VmWare
Fusion, the default Network Adapter driver is an *Intel 82545EM (e1000)*, and there
is no GUI to change it to *vmxnet3*. The change must be done manually in the VM's
configuration file as follows:
- Bring up the VM library window: **Window -> Virtual Machine Library**
- Right click on the VM where you want to change the driver:
<*VM-Name*> **-> Show in Finder**. This pops up a new Finder window with a line
for each VM that Fusion knows about.
- Right click on the VM where you want to change the driver:
<*VM-Name*> **-> Show package contents**. This brings up a window with the
contents of the package.
- Open the file <*VM-Name*> **.vmx** with your favorite text editor.
- For each Network Adapter that you want used by VPP, look for the
Network Adapter's driver configuration. For example, for the VM's first
Network Adapter look for:
```
ethernet0.virtualDev = "e1000"
```
Replace `e1000` with `vmxnet3`:
```
ethernet0.virtualDev = "vmxnet3"
```
and restart the VM.
If you replaced the driver on your VM's primary Network Adapter, you will
have to change the primary network interface configuration in Linux.
First, get the new primary network interface name:
```
sudo lshw -class network -businfo
Bus info Device Class Description
========================================================
pci@0000:03:00.0 ens160 network VMXNET3 Ethernet Controller
```
Replace the existing primary network interface name in `/etc/network/interfaces`
with the above device name (ens160):
```
# This file describes the network interfaces available on your system,
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto ens160
iface ens160 inet dhcp
docs/usecases/contiv/VPPTRACE.md
+95
View File
@@ -0,0 +1,95 @@
## Using vpptrace.sh for VPP Packet Tracing
VPP allows tracing of incoming packets using CLI commands `trace add` and `show trace`
as explained [here](VPP_PACKET_TRACING_K8S.html), but it is a rather cumbersome process.
The buffer for captured packets is limited in size, and once it gets full the tracing stops. The user has to manually clear the buffer content, and then repeat the trace command to resume the packet capture, losing information about all packets received in the meantime.
Packet filtering exposed via the CLI command `trace filter` is also quite limited in what it can do. Currently there is just one available filter, which allows you to keep only packets that include a certain node in the trace or exclude a certain node in the trace.
It is not possible to filter the traffic by its content (e.g., by the source/destination IP address, protocol, etc.).
Last but not least, it is not possible to trace packets on a selected interface
like `tcpdump`, which allows tracing via the option `-i`. VPP is only able to capture packets
on the *RX side* of selected *devices* (e.g., dpdk, virtio, af-packet). This means
that interfaces based on the same device cannot be traced for incoming packets
individually, but only all at the same time. In Contiv/VPP all pods are connected
with VPP via the same kind of the TAP interface, meaning that it is not possible to
capture packets incoming only from one selected pod.
Contiv/VPP ships with a simple bash script [vpptrace.sh](https://github.com/contiv/vpp/blob/master/scripts/vpptrace.sh),
which helps alleviate the aforementioned VPP limitations. The script automatically
re-initializes buffers and traces whenever it is close to getting full, in order to
avoid packet loss as much as possible. Next it allows you to filter packets
by the content of the trace. There are two modes of filtering:
- *substring mode* (default): packet trace must contain a given sub-string in order to
be included in the output
- *regex mode*: packet trace must match a given regex in order to be printed
The script is still limited, in that capture runs only on the RX side of all interfaces that are built on top of selected devices. Using filtering, however, it is possible to limit
*traffic by interface* simply by using the interface name as a substring to match against.
#### Usage
Run the script with option `-h` to get the usage printed:
```
Usage: ./vpptrace.sh [-i <VPP-IF-TYPE>]... [-a <VPP-ADDRESS>] [-r] [-f <REGEXP> / <SUBSTRING>]
-i <VPP-IF-TYPE> : VPP interface *type* to run the packet capture on (e.g., dpdk-input, virtio-input, etc.)
- available aliases:
- af-packet-input: afpacket, af-packet, veth
- virtio-input: tap (version determined from the VPP runtime config), tap2, tapv2
- tapcli-rx: tap (version determined from the VPP config), tap1, tapv1
- dpdk-input: dpdk, gbe, phys*
- multiple interfaces can be watched at the same time - the option can be repeated with
different values
- default = dpdk + tap
-a <VPP-ADDRESS> : IP address or hostname of the VPP to capture packets from
- not supported if VPP listens on a UNIX domain socket
- default = 127.0.0.1
-r : apply filter string (passed with -f) as a regexp expression
- by default the filter is NOT treated as regexp
-f : filter string that packet must contain (without -r) or match as regexp (with -r) to be printed
- default is no filtering
```
`VPP-IF-TYPE` is a repeated option used to select the set of devices (e.g., virtio, dpdk, etc.)
to capture the incoming traffic. Script provides multiple aliases, which
are much easier to remember than the device names. For `dpdk-input` one can enter
just `dpdk`, or anything starting with `phys`, etc. For TAPs, the script is even
smart enough to find out the TAP version used, which allows to enter just `tap`
as the device name.
If `VPP-IF-TYPE` is not specified, then the default behaviour is to capture from both
`dpdk` (traffic entering the node from outside) and `tap` (preferred interface type
for pod-VPP and host-VPP interconnection, receiving node-initiated traffic).
vpptrace.sh can capture packets even from a VPP on a different host, provided that
VPP-CLI listens on a port, and not on a UNIX domain socket (for security reasons IPC
is the default communication link, see `/etc/vpp/contiv-vswitch.conf`). Enter the destination
node IP address via the option `-a`(localhost is the default).
The capture can be filtered via the `-f` option. The output will include only packets
whose trace matches contain the given expression/sub-string.
Option `-r` enables the regex mode for filtering.
#### Examples
- Capture all packets entering VPP via `tapcli-1` interface **AND** all packets
leaving VPP via `tapcli-1` that were sent from a pod, or the host on the *same node*
(sent from tap, not Gbe):
```
$ vpptrace.sh -i tap -f "tapcli-1"
```

- Capture all packets with source or destination IP address 10.1.1.3:
```
$ vpptrace.sh -i tap -i dpdk -f "10.1.1.3"
Or just:
$ vpptrace.sh "10.1.1.3"
```
- Capture all SYN-ACKs received from outside:
```
$ vpptrace.sh -i dpdk -f "SYN-ACK"
```
docs/usecases/contiv/VPP_CONFIG.md
+153
View File
@@ -0,0 +1,153 @@
## Creating VPP Startup Configuration
This document describes how to create the VPP startup configuration
file located at `/etc/vpp/contiv-vswitch.conf`.
### Hardware Interface Configuration
#### Single-NIC Configuration
You need to configure hardware interfaces for use by VPP. First, find out the PCI address of the host's network interface. On
Debian-based distributions, you can use `lshw`:
```
sudo lshw -class network -businfo
Bus info Device Class Description
========================================================
pci@0000:03:00.0 ens160 network VMXNET3 Ethernet Controller
```
In our case, it would be the `ens3` interface with the PCI address
`0000:00:03.0`
Now, add or modify the VPP startup config file (`/etc/vpp/contiv-vswitch.conf`)
to contain the proper PCI address:
```
unix {
nodaemon
cli-listen /run/vpp/cli.sock
cli-no-pager
coredump-size unlimited
full-coredump
poll-sleep-usec 100
}
nat {
endpoint-dependent
}
dpdk {
dev 0000:00:03.0
}
api-trace {
on
nitems 500
}
```
#### Multi-NIC Configuration
Similar to the single-NIC configuration, use command *lshw* to find the PCI
addresses of all the NICs in the system, for example:
```
$ sudo lshw -class network -businfo
Bus info Device Class Description
====================================================
pci@0000:00:03.0 ens3 network Virtio network device
pci@0000:00:04.0 ens4 network Virtio network device
```
In the example above, `ens3` would be the primary interface and `ens4` would
be the interface that would be used by VPP. The PCI address of the `ens4`
interface would be `0000:00:04.0`.
Make sure the selected interface is *shut down*, otherwise VPP
will not grab it:
```
sudo ip link set ens4 down
```
Now, add or modify the VPP startup config file in `/etc/vpp/contiv-vswitch.conf`
to contain the proper PCI address:
```
unix {
nodaemon
cli-listen /run/vpp/cli.sock
cli-no-pager
coredump-size unlimited
full-coredump
poll-sleep-usec 100
}
nat {
endpoint-dependent
}
dpdk {
dev 0000:00:04.0
}
api-trace {
on
nitems 500
}
```
If assigning multiple NICs to VPP you will need to include each NIC's PCI address
in the dpdk stanza in `/etc/vpp/contiv-vswitch.conf`.
##### Assigning all NICs to VPP
On a multi-NIC node, it is also possible to assign all NICs from the kernel for
use by VPP. First, you need to install the STN daemon, as described [here][1],
since you will want the NICs to revert to the kernel if VPP crashes.
You also need to configure the NICs in the VPP startup config file
in `/etc/vpp/contiv-vswitch.conf`. For example, to use both the primary and
secondary NIC, in a two-NIC node, your VPP startup config file would look
something like this:
```
unix {
nodaemon
cli-listen /run/vpp/cli.sock
cli-no-pager
coredump-size unlimited
full-coredump
poll-sleep-usec 100
}
nat {
endpoint-dependent
}
dpdk {
dev 0000:00:03.0
dev 0000:00:04.0
}
api-trace {
on
nitems 500
}
```
#### Installing `lshw` on CentOS/RedHat/Fedora
Note: On CentOS/RedHat/Fedora distributions, `lshw` may not be available
by default, install it by
```
sudo yum -y install lshw
```
### Power-saving Mode
In regular operation, VPP takes 100% of one CPU core at all times (poll loop).
If high performance and low latency is not required you can "slow-down"
the poll-loop and drastically reduce CPU utilization by adding the following
stanza to the `unix` section of the VPP startup config file:
```
unix {
...
poll-sleep-usec 100
...
}
```
The power-saving mode is especially useful in VM-based development environments
running on laptops or less powerful servers.
### VPP API Trace
To troubleshoot VPP configuration issues in production environments, it is
strongly recommended to configure VPP API trace. This is done by adding the
following stanza to the VPP startup config file:
```
api-trace {
on
nitems 500
}
```
You can set the size of the trace buffer with the <nitems> attribute.
docs/usecases/contiv/VPP_PACKET_TRACING_K8S.md
+510
View File
File diff suppressed because it is too large Load Diff
docs/usecases/contiv/Vagrant.md
+250
View File
@@ -0,0 +1,250 @@
## Contiv-VPP Vagrant Installation
### Prerequisites
The following items are prerequisites before installing vagrant:
- Vagrant 2.0.1 or later
- Hypervisors:
- VirtualBox 5.2.8 or later
- VMWare Fusion 10.1.0 or later or VmWare Workstation 14
- For VmWare Fusion, you will need the [Vagrant VmWare Fusion plugin](https://www.vagrantup.com/vmware/index.html)
- Laptop or server with at least 4 CPU cores and 16 Gig of RAM
### Creating / Shutting Down / Destroying the Cluster
This folder contains the Vagrant file that is used to create a single or multi-node
Kubernetes cluster using Contiv-VPP as a Network Plugin.
The folder is organized into two subfolders:
- (config) - contains the files that share cluster information, which are used
during the provisioning stage (master IP address, Certificates, hash-keys).
**CAUTION:** Editing is not recommended!
- (vagrant) - contains scripts that are used for creating, destroying, rebooting
and shutting down the VMs that host the K8s cluster.
To create and run a K8s cluster with a *contiv-vpp CNI* plugin, run the
`vagrant-start` script, located in the [vagrant folder](https://github.com/contiv/vpp/tree/master/vagrant). The `vagrant-start`
script prompts the user to select the number of worker nodes for the kubernetes cluster.
Zero (0) worker nodes mean that a single-node cluster (with one kubernetes master node) will be deployed.
Next, the user is prompted to select either the *production environment* or the *development environment*.
Instructions on how to build the development *contiv/vpp-vswitch* image can be found below in the
[development environment](#building-and-deploying-the-dev-contiv-vswitch-image) command section.
The last option asks the user to select either *Without StealTheNIC* or *With StealTheNIC*.
Using option *With StealTheNIC* has the plugin "steal" interfaces owned by Linux and uses their configuration in VPP.
For the production environment, enter the following commands:
```
| => ./vagrant-start
Please provide the number of workers for the Kubernetes cluster (0-50) or enter [Q/q] to exit: 1
Please choose Kubernetes environment:
1) Production
2) Development
3) Quit
--> 1
You chose Development environment
Please choose deployment scenario:
1) Without StealTheNIC
2) With StealTheNIC
3) Quit
--> 1
You chose deployment without StealTheNIC
Creating a production environment, without STN and 1 worker node(s)
```
For the development environment, enter the following commands:
```
| => ./vagrant-start
Please provide the number of workers for the Kubernetes cluster (0-50) or enter [Q/q] to exit: 1
Please choose Kubernetes environment:
1) Production
2) Development
3) Quit
--> 2
You chose Development environment
Please choose deployment scenario:
1) Without StealTheNIC
2) With StealTheNIC
3) Quit
--> 1
You chose deployment without StealTheNIC
Creating a development environment, without STN and 1 worker node(s)
```
To destroy and clean-up the cluster, run the *vagrant-cleanup* script, located
[inside the vagrant folder](https://github.com/contiv/vpp/tree/master/vagrant):
```
cd vagrant/
./vagrant-cleanup
```
To shutdown the cluster, run the *vagrant-shutdown* script, located [inside the vagrant folder](https://github.com/contiv/vpp/tree/master/vagrant):
```
cd vagrant/
./vagrant-shutdown
```
- To reboot the cluster, run the *vagrant-reload* script, located [inside the vagrant folder](https://github.com/contiv/vpp/tree/master/vagrant):
```
cd vagrant/
./vagrant-reload
```
- From a suspended state, or after a reboot of the host machine, the cluster
can be brought up by running the *vagrant-up* script.
### Building and Deploying the dev-contiv-vswitch Image
If you chose the optional development-environment-deployment option, then perform the
following instructions on how to build a modified *contivvpp/vswitch* image:
- Make sure changes in the code have been saved. From the k8s-master node,
build the new *contivvpp/vswitch* image (run as sudo):
```
vagrant ssh k8s-master
cd /vagrant/config
sudo ./save-dev-image
```
- The newly built *contivvpp/vswitch* image is now tagged as *latest*. Verify the
build with `sudo docker images`; the *contivvpp/vswitch* should have been created a few
seconds ago. The new image with all the changes must become available to all
the nodes in the K8s cluster. To make the changes available to all, load the docker image into the running
worker nodes (run as sudo):
```
vagrant ssh k8s-worker1
cd /vagrant/config
sudo ./load-dev-image
```
- Verify with `sudo docker images`; the old *contivvpp/vswitch* should now be tagged as
`<none>` and the latest tagged *contivvpp/vswitch* should have been created a
few seconds ago.
### Exploring the Cluster
Once the cluster is up, perform the following steps:
- Log into the master:
```
cd vagrant
vagrant ssh k8s-master
Welcome to Ubuntu 16.04 LTS (GNU/Linux 4.4.0-21-generic x86_64)
* Documentation: https://help.ubuntu.com/
vagrant@k8s-master:~$
```
- Verify the Kubernetes/Contiv-VPP installation. First, verify the nodes
in the cluster:
```
vagrant@k8s-master:~$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-master Ready master 22m v1.9.2 <none> Ubuntu 16.04 LTS 4.4.0-21-generic docker://17.12.0-ce
k8s-worker1 Ready <none> 15m v1.9.2 <none> Ubuntu 16.04 LTS 4.4.0-21-generic docker://17.12.0-ce
```
- Next, verify that all pods are running correctly:
```
vagrant@k8s-master:~$ kubectl get pods -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE
contiv-etcd-2ngdc 1/1 Running 0 17m 192.169.1.10 k8s-master
contiv-ksr-x7gsq 1/1 Running 3 17m 192.169.1.10 k8s-master
contiv-vswitch-9bql6 2/2 Running 0 17m 192.169.1.10 k8s-master
contiv-vswitch-hpt2x 2/2 Running 0 10m 192.169.1.11 k8s-worker1
etcd-k8s-master 1/1 Running 0 16m 192.169.1.10 k8s-master
kube-apiserver-k8s-master 1/1 Running 0 16m 192.169.1.10 k8s-master
kube-controller-manager-k8s-master 1/1 Running 0 15m 192.169.1.10 k8s-master
kube-dns-6f4fd4bdf-62rv4 2/3 CrashLoopBackOff 14 17m 10.1.1.2 k8s-master
kube-proxy-bvr74 1/1 Running 0 10m 192.169.1.11 k8s-worker1
kube-proxy-v4fzq 1/1 Running 0 17m 192.169.1.10 k8s-master
kube-scheduler-k8s-master 1/1 Running 0 16m 192.169.1.10 k8s-master
```
- If you want your pods to be scheduled on both the master and the workers,
you have to untaint the master node:
```
```
- Check VPP and its interfaces:
```
vagrant@k8s-master:~$ sudo vppctl
_______ _ _ _____ ___
__/ __/ _ \ (_)__ | | / / _ \/ _ \
_/ _// // / / / _ \ | |/ / ___/ ___/
/_/ /____(_)_/\___/ |___/_/ /_/
vpp# sh interface
Name Idx State Counter Count
GigabitEthernet0/8/0 1 up rx packets 14
rx bytes 3906
tx packets 18
tx bytes 2128
drops 3
ip4 13
...
```
- Make sure that `GigabitEthernet0/8/0` is listed and that its status is `up`.
- Next, create an example deployment of nginx pods:
```
vagrant@k8s-master:~$ kubectl run nginx --image=nginx --replicas=2
deployment "nginx" created
```
- Check the status of the deployment:
```
vagrant@k8s-master:~$ kubectl get deploy -o wide
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
nginx 2 2 2 2 2h nginx nginx run=nginx
```
- Verify that the pods in the deployment are up and running:
```
vagrant@k8s-master:~$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-8586cf59-6kx2m 1/1 Running 1 1h 10.1.2.3 k8s-worker1
nginx-8586cf59-j5vf9 1/1 Running 1 1h 10.1.2.2 k8s-worker1
```
- Issue an HTTP GET request to a pod in the deployment:
```
vagrant@k8s-master:~$ wget 10.1.2.2
--2018-01-19 12:34:08-- http://10.1.2.2/
Connecting to 10.1.2.2:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 612 [text/html]
Saving to: ‘index.html.1’
index.html.1 100%[=========================================>] 612 --.-KB/s in 0s
2018-01-19 12:34:08 (1.78 MB/s) - ‘index.html.1’ saved [612/612]
```
#### How to SSH into k8s Worker Node
To SSH into k8s Worker Node, perform the following steps:
```
cd vagrant
vagrant status
vagrant ssh k8s-worker1
```
docs/usecases/contiv/index.rst
+26
View File
@@ -0,0 +1,26 @@
.. _contiv:
##########
Contiv/VPP
##########
This section provides the following information about the Contiv function:
.. toctree::
:maxdepth: 2
K8s_Overview
SECURITY
Vagrant
MANUAL_INSTALL
VPP_CONFIG
VMWARE_FUSION_HOST
NETWORKING
SINGLE_NIC_SETUP
MULTI_NIC_SETUP
CUSTOM_MGMT_NETWORK
Prometheus
VPP_PACKET_TRACING_K8S
VPPTRACE
CORE_FILES
BUG_REPORTS
docs/usecases/index.rst
+1
View File
@@ -9,6 +9,7 @@ extensive list, but should give a sampling of the many features contained in FD.
.. toctree::
contiv/index
containers
vhost/index.rst
homegateway