041eacc816
This plugin provides load balancing for VPP in a way that is largely inspired from Google's MagLev: http://research.google.com/pubs/pub44824.html More info in the README.md Change-Id: I1223f495d5c2d5200808a398504119f2830337e9 Signed-off-by: Pierre Pfister <ppfister@cisco.com>
142 lines
5.0 KiB
Markdown
142 lines
5.0 KiB
Markdown
# Load Balancer plugin for VPP
|
|
|
|
## Version
|
|
|
|
The load balancer plugin is currently in *beta* version.
|
|
Both CLIs and APIs are subject to *heavy* changes.
|
|
Wich also means feedback is really welcome regarding features, apis, etc...
|
|
|
|
## Overview
|
|
|
|
This plugin provides load balancing for VPP in a way that is largely inspired
|
|
from Google's MagLev: http://research.google.com/pubs/pub44824.html
|
|
|
|
The load balancer is configured with a set of Virtual IPs (VIP, which can be
|
|
prefixes), and for each VIP, with a set of Application Server addresses (ASs).
|
|
|
|
Traffic received for a given VIP (or VIP prefix) is tunneled using GRE towards
|
|
the different ASs in a way that (tries to) ensure that a given session will
|
|
always be tunneled to the same AS.
|
|
|
|
Both VIPs or ASs can be IPv4 or IPv6, but for a given VIP, all ASs must be using
|
|
the same encap. type (i.e. IPv4+GRE or IPv6+GRE). Meaning that for a given VIP,
|
|
all AS addresses must be of the same family.
|
|
|
|
## Performances
|
|
|
|
The load balancer has been tested up to 1 millions flows and still forwards more
|
|
than 3Mpps per core in such circumstances.
|
|
Although 3Mpps seems already good, it is likely that performances will be improved
|
|
in next versions.
|
|
|
|
## Configuration
|
|
|
|
### Global LB parameters
|
|
|
|
The load balancer needs to be configured with some parameters:
|
|
|
|
lb conf [ip4-src-address <addr>] [ip6-src-address <addr>]
|
|
[buckets <n>] [timeout <s>]
|
|
|
|
ip4-src-address: the source address used to send encap. packets using IPv4.
|
|
|
|
ip6-src-address: the source address used to send encap. packets using IPv6.
|
|
|
|
buckets: the *per-thread* established-connexions-table number of buckets.
|
|
|
|
timeout: the number of seconds a connection will remain in the
|
|
established-connexions-table while no packet for this flow
|
|
is received.
|
|
|
|
|
|
### Configure the VIPs
|
|
|
|
lb vip <prefix> [encap (gre6|gre4)] [new_len <n>] [del]
|
|
|
|
new_len is the size of the new-connection-table. It should be 1 or 2 orders of
|
|
magnitude bigger than the number of ASs for the VIP in order to ensure a good
|
|
load balancing.
|
|
|
|
Examples:
|
|
|
|
lb vip 2002::/16 encap gre6 new_len 1024
|
|
lb vip 2003::/16 encap gre4 new_len 2048
|
|
lb vip 80.0.0.0/8 encap gre6 new_len 16
|
|
lb vip 90.0.0.0/8 encap gre4 new_len 1024
|
|
|
|
### Configure the ASs (for each VIP)
|
|
|
|
lb as <vip-prefix> [<address> [<address> [...]]] [del]
|
|
|
|
You can add (or delete) as many ASs at a time (for a single VIP).
|
|
Note that the AS address family must correspond to the VIP encap. IP family.
|
|
|
|
Examples:
|
|
|
|
lb as 2002::/16 2001::2 2001::3 2001::4
|
|
lb as 2003::/16 10.0.0.1 10.0.0.2
|
|
lb as 80.0.0.0/8 2001::2
|
|
lb as 90.0.0.0/8 10.0.0.1
|
|
|
|
|
|
|
|
## Monitoring
|
|
|
|
The plugin provides quite a bunch of counters and information.
|
|
These are still subject to quite significant changes.
|
|
|
|
show lb
|
|
show lb vip
|
|
show lb vip verbose
|
|
|
|
show node counters
|
|
|
|
|
|
## Design notes
|
|
|
|
### Multi-Threading
|
|
|
|
MagLev is a distributed system which pseudo-randomly generates a
|
|
new-connections-table based on AS names such that each server configured with
|
|
the same set of ASs ends up with the same table. Connection stickyness is then
|
|
ensured with an established-connections-table. Using ECMP, it is assumed (but
|
|
not relied on) that servers will mostly receive traffic for different flows.
|
|
|
|
This implementation pushes the parallelism a little bit further by using
|
|
one established-connections table per thread. This is equivalent to assuming
|
|
that RSS will make a job similar to ECMP, and is pretty useful as threads don't
|
|
need to get a lock in order to write in the table.
|
|
|
|
### Hash Table
|
|
|
|
A load balancer requires an efficient read and write hash table. The hash table
|
|
used by ip6-forward is very read-efficient, but not so much for writing. In
|
|
addition, it is not a big deal if writing into the hash table fails (again,
|
|
MagLev uses a flow table but does not heaviliy relies on it).
|
|
|
|
The plugin therefore uses a very specific (and stupid) hash table.
|
|
- Fixed (and power of 2) number of buckets (configured at runtime)
|
|
- Fixed (and power of 2) elements per buckets (configured at compilation time)
|
|
|
|
### Reference counting
|
|
|
|
When an AS is removed, there is two possible ways to react.
|
|
- Keep using the AS for established connections
|
|
- Change AS for established connections (likely to cause error for TCP)
|
|
|
|
In the first case, although an AS is removed from the configuration, its
|
|
associated state needs to stay around as long as it is used by at least one
|
|
thread.
|
|
|
|
In order to avoid locks, a specific reference counter is used. The design is quite
|
|
similar to clib counters but:
|
|
- It is possible to decrease the value
|
|
- Summing will not zero the per-thread counters
|
|
- Only the thread can reallocate its own counters vector (to avoid concurrency issues)
|
|
|
|
This reference counter is lock free, but reading a count of 0 does not mean
|
|
the value can be freed unless it is ensured by *other* means that no other thread
|
|
is concurrently referencing the object. In the case of this plugin, it is assumed
|
|
that no concurrent event will take place after a few seconds.
|
|
|